Lead II - Senior Data Engineer (AWS, Pyspark, SQL)

Job Title: Big Data Engineer
Experience: 7+ Years
Location: Any UST India Location
Employment Type: Full-time

Must-Have – Required Skills & Qualifications:

• Experience:

  • 7+ years of experience in Big Data architecture and engineering.

• AWS Expertise:

  • Strong hands-on experience with AWS services such as DMS, Kinesis, Athena, Glue, Lambda, S3, EMR, Redshift, etc.

• Real-Time Data Streaming & Synchronization:

  • Proficiency with Debezium and Kafka for real-time data streaming and synchronization.

• Spark Optimization:

  • Expertise in Spark optimization for batch processing improvements.

• Query Optimization:

  • Strong skills in SQL and Oracle query optimization.

• Big Data Frameworks:

  • Experience with Big Data frameworks like Hadoop, Spark, Hive, Presto, Athena, etc.

• CI/CD Integration:

  • Hands-on experience in CI/CD automation and integrating AWS services with DevOps pipelines.

• Problem-Solving Skills:

  • Strong problem-solving skills with the ability to work in an Agile environment.

Good-to-Have – Preferred Skills:

• Dremio to Athena Migrations:

  • Experience migrating solutions from Dremio to Athena.

• Cloud-Native DR Solutions on AWS:

  • Exposure to cloud-native disaster recovery solutions on AWS.

• Performance Improvement Documentation:

  • Strong analytical skills to document and implement performance improvements effectively.

Data Architecture,AWS,Spark,SQL