Big Data Engineer

Job Title: Big Data Engineer (Apache Spark)

Experience: 5-8 Years

Location: Bengaluru/Hyderabad

We are seeking a skilled and detail-oriented Big Data Engineer with strong expertise in Apache Spark to join our dynamic data team. The ideal candidate will have proven experience in designing and implementing large-scale data processing solutions and a strong understanding of distributed computing concepts.

Key Responsibilities:

• Design and develop scalable data processing solutions using Apache Spark (Core, SQL, Streaming, MLlib).
• Build, optimize, and maintain robust data pipelines and ETL processes for structured and unstructured data.
• Collaborate with data scientists, analysts, and software engineers to integrate Spark-based data products into production workflows.
• Ensure data quality, integrity, and security throughout the data lifecycle
• Monitor, troubleshoot, and improve the performance of Spark jobs in production environments.
• Integrate Spark with cloud platforms such as AWS, Azure, or GCP for scalable deployment.

Required Skills and Qualifications:

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
• 3+ years of hands-on experience with Apache Spark in a large-scale data environment.
• Strong proficiency in Scala, Python, or Java (preference for Scala).
• Experience with data storage technologies such as HDFS, Hive, HBase, S3, etc.
• Familiarity with SQL, Kafka, Airflow, and NoSQL databases.
• Understanding of distributed systems and parallel computing concepts.
• Experience with containerization and orchestration tools (e.g., Docker, Kubernetes) is a plus.

Preferred Qualifications:

• Certification in Big Data or Spark (e.g., Databricks Certified Developer).
• Experience working in a DevOps/CI-CD environment.
• Knowledge of data warehousing concepts and tools like Snowflake or Redshift.