Senior Data Engineer

Description:

V-Systems is looking for a Senior Java Spark Developer with strong hands-on experience in Java, Apache Spark, and the Cloudera Hadoop ecosystem to design and build scalable data processing applications. This role requires expertise in distributed computing, performance optimization, and large-scale data workload handling.

Key Responsibilities:

• Develop, test, and deploy Java-based Apache Spark applications to handle large-scale data processing tasks.

• Optimize Spark jobs to improve performance, scalability, and reliability.

• Build Java-based microservices and APIs for seamless data integration.

• Work with Cloudera Hadoop components including HDFS, Hive, Impala, HBase, Kafka, and Sqoop.

• Design and implement efficient data storage and retrieval solutions.

• Troubleshoot and resolve performance issues within Spark and Cloudera environments.

• Collaborate with data scientists, business analysts, and developers to gather and understand data requirements.

• Ensure data integrity, accuracy, and security best practices in all processing activities.

• Utilize tools like Kafka, Flume, Oozie, and Nifi for both real-time and batch data ingestion.

• Implement and manage version control using Git and CI/CD pipelines with Jenkins or GitLab.

• Deploy and maintain Spark applications in cloud or on-premises Cloudera clusters.

Required Skills & Experience:

• Strong application development skills with a focus on Java and Big Data processing.

• Extensive hands-on experience with Java, Apache Spark, and Spark SQL.

• Proficiency with Cloudera Hadoop ecosystem components: HDFS, Hive, Impala, HBase, Kafka, Sqoop.

• Experience building and optimizing ETL pipelines at scale.

• Knowledge of SQL and NoSQL databases like Hive, HBase, and PostgreSQL.

• Understanding of data warehousing, dimensional modeling, and data lake architectures.

• Proven ability to troubleshoot and optimize Spark applications.

• Familiarity with version control systems (Git, Bitbucket) and CI/CD tools (Jenkins, GitLab).

• Experience with real-time streaming technologies such as Kafka, Flume, Oozie, and Nifi.

• Excellent problem-solving skills, detail orientation, and ability to thrive in fast-paced environments.

Application Instructions:

Please share your updated resume and include the following details:

1. Years of Experience

2. Visa Status

3. Current Location

4. LinkedIn Profile URL

5. Updated Resume attachment