Description:
We are seeking a skilled Data Engineer with strong experience in designing, developing, and maintaining scalable data processing solutions. The ideal candidate should have expertise in Apache Spark, Scala, ETL development, MongoDB, REST APIs, and Snowflake. Experience with HBase is a plus. You will work closely with cross-functional teams to build robust data pipelines and support enterprise analytics and reporting initiatives.
Key Responsibilities
- Design, develop, and optimize scalable ETL pipelines for batch and real-time data processing.
- Build high-performance data processing applications using Apache Spark and Scala.
- Integrate data from multiple sources, including REST APIs, databases, and cloud platforms.
- Develop and maintain data ingestion and transformation workflows.
- Work with MongoDB for data storage, retrieval, and optimization.
- Design and optimize Snowflake data warehouse solutions for analytics and reporting.
- Develop reusable, maintainable, and efficient code following best practices.
- Monitor, troubleshoot, and optimize data pipelines to ensure reliability and performance.
- Collaborate with business analysts, data architects, and application teams to understand data requirements.
- Participate in code reviews, testing, deployment, and production support activities.
- Document technical designs, workflows, and operational procedures.
Required Skills
- Strong hands-on experience with Apache Spark.
- Proficiency in Scala programming.
- Experience developing and maintaining ETL pipelines.
- Strong knowledge of MongoDB.
- Experience developing and consuming REST APIs.
- Hands-on experience with Snowflake.
- Good understanding of SQL and data modeling concepts.
- Familiarity with Git and CI/CD practices.
- Strong analytical and problem-solving skills.
- Excellent communication and teamwork skills.
Preferred Skills
- Experience with HBase.
- Knowledge of distributed computing and big data ecosystems.
- Experience with cloud platforms such as AWS, Azure, or Google Cloud.
- Familiarity with workflow orchestration tools such as Apache Airflow.
- Experience working in Agile/Scrum environments.
Qualifications
- Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.
- 9+ years of experience in Data Engineering or Big Data development.
- Proven experience delivering enterprise-scale data integration and analytics solutions.