Description:
Big Data, Cloud, AirFlow, SnowFlake, ETL, Python, Scala
I am seeking a talented and experienced Big Data Engineer with expertise in cloud technology to join my clients team. As a Big Data Engineer, you will be responsible for designing, developing, and implementing data solutions using cloud-based technologies. Your role will involve working with large datasets and integrating them with cloud platforms to enable scalable and efficient data processing. You will collaborate with cross-functional teams to analyze business requirements and ensure the optimal use of cloud technology to meet data engineering needs. The successful candidate will have a strong understanding of big data concepts, cloud architecture, and data processing frameworks.
Duties and Responsibilities:
- Design and develop scalable data processing architectures using cloud technologies such as Amazon Web Services (AWS), Google Cloud Platform (GCP), or Microsoft Azure.
- Implement data pipelines and workflows to extract, transform, and load (ETL) large volumes of structured and unstructured data into cloud-based storage systems.
- Optimize data systems for performance, reliability, scalability, and cost-efficiency by leveraging cloud-native solutions.
- Collaborate with data scientists, software engineers, and other stakeholders to understand data requirements and design data solutions that meet business needs.
- Develop robust data governance frameworks and implement security measures to ensure data integrity and compliance with regulatory requirements.
- Conduct data profiling, data quality assessments, and data cleansing to ensure accuracy and consistency of data stored in the cloud.
- Identify opportunities for data optimization, performance tuning, and cost reduction in cloud-based data processing systems.
- Stay updated with the latest advancements in big data technologies and cloud platforms to propose innovative solutions to business problems.
- Conduct performance testing and optimization of data processes to ensure efficient execution of data transformations and analytics workflows.
- Collaborate with DevOps teams to establish automated processes for cloud infrastructure provisioning, monitoring, and continuous integration/continuous deployment (CI/CD).
Qualifications:
- Bachelor's degree in Computer Science, Engineering, or a related field. Master's degree preferred.
- Proven experience as a Big Data Engineer, Data Engineer, or similar role, with a focus on cloud-based technologies.
- In-depth knowledge of cloud platforms such as AWS, GCP, or Azure, including services like S3, EMR, Redshift, BigQuery, and Data Lake.
- Proficiency in programming languages like Python, Java, or Scala, and hands-on experience with big data processing frameworks such as Apache Spark or Apache Hadoop.
- Strong understanding of distributed computing, data management, and storage systems.
- Familiarity with data modeling techniques, data warehousing concepts, and database technologies.
- Experience with containerization technologies like Docker and orchestration tools like Kubernetes is a plus.
- Excellent problem-solving and analytical skills, with the ability to handle complex data engineering challenges.
- Strong communication and collaboration skills, with the ability to work effectively in cross-functional teams.
- Proven ability to work independently and manage multiple projects simultaneously.
-