Lead Data Engineer

Client of Talentegrity is looking for a Lead Data Engineer. To lead and oversee the architecture, development, and maintenance of data pipelines, databases, and ETL processes, ensuring the efficient and effective handling of data at scale. To collaborate with cross-functional teams and stakeholders to create solutions that drive data-driven decisions across the organization.
CTC budget - Max of 40 L/annum

Design and implement scalable, reliable, and efficient data architectures to support business needs.

Define data models, schemas, and storage solutions (e.g., data lakes, data warehouses, NoSQL databases).

Evaluate and select appropriate technologies, tools, and frameworks for data processing, storage, and analytics.

Ensure the architecture supports both batch and real-time data processing based on client needs.

Knowledge on data security, and compliance with regulations (e.g., GDPR, CCPA).

Build and maintain robust ETL/ELT pipelines to ingest, transform, and load data from various sources.

Optimize data pipelines for performance, scalability, and cost-efficiency.

Implement data quality checks and monitoring to ensure data accuracy and reliability.

Automate data workflows and integrate with CI/CD pipelines for seamless deployment.

Design and manage cloud-based data infrastructure (e.g., AWS, Azure, GCP).

Implement infrastructure-as-code (IaC) for provisioning and managing resources.

Collaborate with business stakeholders, product managers, and analytics teams to translate business requirements into technical solutions.

10+ years of experience in Data Engineering, with at least 3+ years in an architectural role

Strong expertise in data engineering tools and technologies (e.g., Apache Spark, 1 Cloud (AWS or GCP or Azure), SQL, Python).

Proficiency in any cloud platforms (e.g., AWS, Azure, GCP) and their data services.

Experience with data modeling, ETL/ELT processes, and data warehousing solutions.

Knowledge of distributed systems, big data technologies, and real-time data processing.

Strong leadership, communication, and problem-solving skills.

Familiarity with DevOps practices and tools (e.g., Docker, Kubernetes, Terraform).

Understanding of data governance, security, and compliance requirements.