Lead Data Engineer

Job Title: Sr Data Engineering Lead
Location: Indore, MP (Hybrid 3 Days a Week)
Employment Type: Full-Time

About the Role
Ccube is seeking a visionary Senior Data Engineering Lead to spearhead and lead our entire Data
Management practice. In this pivotal role, you will architect, build, and scale our enterprise-grade data
infrastructure, enabling data-driven decision-making across the organization. This position calls for a
strong technical foundation in distributed data systems, deep hands-on experience with modern big data
and cloud technologies, and a proven track record of leading and mentoring high-performing data
engineering teams. The ideal candidate will define the data engineering roadmap, drive innovation, and
deliver robust, scalable, and efficient data solutions that power Ccube’s growth and success.

Key Responsibilities

● Data Strategy & Architecture
○ Define the end-to-end data engineering strategy and technical vision aligned with
Ccube’s business goals.
○ Architect scalable, high-performance, and cost-efficient data platforms Preferably on
(AWS, Azure) or GCP.
○ Design and optimize data Lakehouse medallion architectures integrating batch and
streaming pipelines (Spark, Kafka, Delta Lake, Iceberg, Hudi, etc.).
○ Build reusable frameworks for data ingestion, transformation, and orchestration
across heterogeneous systems.

● Data Engineering Execution
○ Lead the development and optimization of ETL/ELT pipelines using PySpark, Scala,
SQL, and Airflow or equivalent orchestration tools.
○ Oversee the implementation of real-time streaming solutions leveraging Kafka,
Kinesis, or Pub/Sub.
○ Guide the team in integrating structured, semi-structured, and unstructured data
sources.
○ Drive adoption of DataOps and DevOps best practices — CI/CD for data pipelines,
automated testing, and monitoring
.
● Cloud & Infrastructure
○ Design and manage cloud-native data solutions (AWS Glue, EMR, Redshift,
Snowflake, BigQuery, Databricks, etc.)
○ Lead initiatives for data platform modernization and cloud migration strategies.

● Governance, Security & Observability
○ Define and enforce standards for data quality, lineage, governance, and metadata
management (e.g., Great Expectations, Apache Atlas, or Collibra).
○ Implement robust data security, compliance, and privacy frameworks aligned with
industry standards (GDPR, HIPAA, etc.).
○ Establish observability frameworks for data pipelines — logging, monitoring, and
anomaly detection.

● Leadership & Collaboration
○ Lead and mentor a team of senior and mid-level data engineers, fostering a culture of
excellence, ownership, and innovation.
○ Collaborate cross-functionally with AI/ML, Analytics, and Product Engineering teams to
enable data-driven decision-making.
○ Evaluate emerging technologies (e.g., VectorDBs, GraphDBs, RAG frameworks) and
drive their adoption for advanced data-driven use cases.
○ Represent the data engineering practice in architecture reviews and executive technology
forums.

Required Qualifications

● Experience:
○ 10+ years in data engineering and architecture roles, with 3+ years in technical
leadership or data platform lead roles.
● Technical Expertise:
○ Deep proficiency in Spark, PySpark, Scala, SQL, and distributed data processing.
○ Strong experience with big data ecosystems (Hadoop, Hive, Presto, Trino, Delta Lake,
etc.).
Proven hands-on work in cloud data platforms – AWS (Glue, EMR, Redshift), GCP
(Dataflow, BigQuery), or Azure (Synapse, Data Factory).
○ Experience with workflow orchestration (Airflow, Dagster, Prefect) and
containerization (Docker, Kubernetes).
○ Expertise in modern data storage systems – Snowflake, Databricks, Iceberg, Hudi, or
Delta Lake.
○ Exposure to data observability, data mesh, and feature store frameworks.

● Leadership:
○ Strong people management, mentorship, and cross-functional collaboration skills.
○ Demonstrated success in building or scaling a data engineering function or CoE (Center
of Excellence).

● Certifications (Preferred):
○ AWS Certified Data Analytics – Specialty / GCP Professional Data Engineer / Databricks
Certified Data Engineer / Snowflake Architec