Senior/Lead Azure Data Engineer - Python/PySpark

Company : Material Plus

Role : Senior/ Lead Data Engineer (Azure, Databricks, Data Factory, Data Fabric)?

Location : Gurgaon (Hybrid)

Job Title : Senior/ Lead Data Engineer (Microsoft Azure, Synapse Data Engineering (Fabric))

Job Responsibilities

Design and Develop Data Pipelines :

• Hands on with development, and optimisation of scalable and reusable - data pipelines in Azure Microsoft Fabric Synapse Data Engineering, leveraging both batch and real-time processing techniques.
• Ensure smooth integration with Azure Data Factory for orchestration and workflow management.
  
  

Cloud Data Architecture

• Collaborate with the Data Architecture team to design and implement robust data architectures in the Azure environment, ensuring they align with business needs while optimising performance, scalability, and cost-efficiency.
  
  

Pipeline Optimisation

• Continuously monitor and optimise the performance, cost, and reliability of data pipelines, ensuring efficient processing, storage, and management of large datasets.
  
  

Cross-functional Collaboration

• Work closely with data engineering teams, analysts, and business stakeholders to understand data requirements, developing solutions that enable self-service analytics and support the decision-making process.
  
  

Documentation & Knowledge Sharing

• Contribute to internal documentation, fostering a culture of knowledge-sharing.
• Provide mentorship and guidance to junior engineers, helping to elevate team skills and improve overall team performance.
  
  

Microsoft Fabric Experience: Apply your knowledge of Azure Tech Stack on Data Engineering (or your willingness to learn) Fabric-based development to manage end-to-end data orchestration, governance, and security across cloud and on-premises systems, ensuring seamless data movement and integration across hybrid environments.

Data Modelling Expertise: Leverage your deep expertise in Azure to design and implement data models, create processing pipelines, and integrate with other Azure services like Data Lake and Synapse to support data storage and analytics needs.

Required Skills And Qualifications

Experience with Azure Ecosystem (Preferably Synapse) :

• 5+ years of hands-on experience with Azure Ecosystem, including Synapse, Spark, OneLake, and other Fabric tools.
• Expertise in optimising Fabric notebooks and efficiently managing large-scale data workloads.
  
  

Proficiency In Azure Data Factory

• Strong experience with designing and orchestrating complex data pipelines using Azure Data Factory, with an emphasis on seamless data flow integration across various Azure services.
  
  

Familiarity With Microsoft Fabric

• A working knowledge or eagerness to learn Azure Data Fabric, focusing on cross-platform data orchestration, governance, and security.
  
  

Advanced Data Engineering Skills

• Extensive experience in data engineering, including the design and implementation of ETL processes and working with large datasets.
• Proven expertise in data quality, monitoring, and testing practices.
  
  

Cloud Architecture Design Expertise

• Experience designing and implementing data architectures in the Azure ecosystem, including tools such as Data Lake, Synapse, and Azure Storage.
  
  

SQL And Data Modelling Expertise

• Strong skills in SQL and data modelling, with the ability to design optimised data structures, tables, and views. Knowledge of both transactional and analytical data modelling.
  
  

Collaboration And Communication Skills

• Strong ability to work cross-functionally with teams from various domains.
• Ability to communicate complex technical concepts to both technical and non-technical stakeholders.
  
  

Cost Optimisation

• Proven experience optimising data engineering processes and Azure resources for both performance and cost, particularly in large-scale cloud environments.
  
  

Preferred Skills

Data Lakehouse Experience :

• Familiarity with Data Lakehouse architectures, particularly with tools like Delta Lake, OneLake, etc.
  
  

Azure Ecosystem Familiarity

• Knowledge of Azures full ecosystem for end-to-end data integration and ETL processes.
  
  

Proficiency In PySpark And Python

• Expertise in PySpark for data processing tasks, with a solid foundation in Python.
  
  

Fabric Integration

• Familiarity with Fabric and how it integrates with other services within the Azure ecosystem.
• Databricks Experience : Experience with Databricks is a plus
  
  

(ref:hirist.tech)