Lead- Data Engineer

Job Summary:

We are seeking an experienced Azure Data Engineer with deep expertise in Databricks and Delta Lake to join our growing data engineering team. This role will focus on building robust, scalable data pipelines and delivering high-quality data solutions using the Medallion architecture in a Delta Lakehouse environment. You will work closely with cross-functional teams to support data-driven decision-making across the organization.

Responsibilities:

• Design, implement, and maintain scalable ETL/ELT pipelines using Azure Data Factory, Azure Databricks, and Azure Synapse Analytics.
• Develop and optimize data workflows using Delta Lake following the Medallion architecture (Bronze, Silver, Gold layers).
• Build and manage Databricks notebooks, clusters, and job workflows for data ingestion, transformation, and analytics.
• Integrate data from diverse sources (structured/unstructured) into the data lake and ensure data quality and consistency across layers.
• Implement data governance, quality checks, and lineage tracking using Databricks features and other Azure services.
• Collaborate with data Architects, Business Super Users, and BI Developers to support analytics and reporting requirements.
• Apply performance tuning, partitioning strategies, and caching for optimized query performance.
• Work with CI/CD pipelines using Azure DevOps or equivalent tools to ensure seamless deployment of data solutions.
• Ensure data security and compliance with company and regulatory standards.

Qualification:

Bachelor's or Master's degree in Computer Science, Data Engineering, or a related field.

Required Skills:

• 3+ years of experience as a Data Engineer with strong skills in the Azure Ecosystem.
• Expertise in Azure Databricks and Delta Lake with practical experience implementing Medallion architecture (Bronze/Silver/Gold layers).
• Strong programming skills in Python and/or Scala, and advanced SQL knowledge.
• Experience with Azure Data Lake Storage Gen2, Azure Data Factory, and Azure Synapse Analytics.
• Solid understanding of data modeling, big data concepts, and data lakehouse architectures.
• Familiarity with performance tuning and job orchestration in large-scale distributed systems.
• Experience working with version control (e.g., Git) and CI/CD pipelines (e.g., Azure DevOps).
• Strong problem-solving skills and attention to detail.

Preferred Skills:

• Experience with real-time data processing (e.g., using Structured Streaming).
• Familiarity with Power BI or other reporting/visualization tools.
• Certification in Azure (e.g., DP-203: Data Engineering on Microsoft Azure) is a plus.
• Exposure to data governance and data cataloging tools (e.g., Purview, Unity Catalog).