Lead Data Engineer

Overview

This role is for one of Zepcruit's clients.

Job Title: Lead/Senior Data Engineer

Job Description:

We are looking for a Senior Data Engineer with strong expertise in Azure Databricks, PySpark, and distributed computing to develop and optimize scalable ETL pipelines for manufacturing analytics. The role involves working with high-frequency industrial data to enable real-time and batch data processing.

Location: Pune

Job Type: Full-Time

Key Responsibilities:

• Build scalable real-time and batch processing workflows using Azure Databricks, PySpark, and Apache Spark.
• Perform data pre-processing, including cleaning, transformation, deduplication, normalization, encoding, and scaling to ensure high-quality input for downstream analytics.
• Design and maintain cloud-based data architectures, including data lakes, lakehouses, and warehouses, following Medallion Architecture.
• Deploy and optimize data solutions on Azure (preferred), AWS, or GCP with a focus on performance, security, and scalability.
• Develop and optimize ETL/ELT pipelines for structured and unstructured data from IoT, MES, SCADA, LIMS, and ERP systems.
• Automate data workflows using CI/CD and DevOps best practices, ensuring security and compliance with industry standards
• Monitor, troubleshoot, and enhance data pipelines for high availability and reliability.
• Utilize Docker and Kubernetes for scalable data processing.
• Collaborate with automation team, data scientists and engineers to provide clean, structured data for AI/ML models.

Desired Skills and Qualifications:

• Bachelor’s or master’s degree in computer science, Information Technology, or a related field.
• 7+ years of experience in core data engineering, with a strong focus on cloud platforms such as Azure (preferred), AWS, or GCP
• Proficiency in PySpark, Azure Databricks, Python and Apache Spark, etc.
• 2 years of team handling experience.
• Expertise in relational databases (e.g., SQL Server, PostgreSQL), time series databases (e.g. Influx DB), and NoSQL databases (e.g., MongoDB, Cassandra)
• Experience in containerization (Docker, Kubernetes).
• Strong analytical and problem-solving skills with attention to detail.
• Good to have MLOps, DevOps including model lifecycle management
• Excellent communication and collaboration skills, with a proven ability to work effectively as a team player.
• Comfortable working in a dynamic, fast-paced startup environment, adapting quickly to changing priorities and responsibilities.