Lead Data Engineer - Databricks

Position: Lead Data Engineer (Databricks)   

Location: Pune, Ahmedabad  

Required Experience: 7 to 10 years   

Preferred Immediate Joiners 

Job Overview:   

We are looking for an accomplished Lead Data Engineer with expertise in Databricks to join our dynamic team. This role is crucial for enhancing our data engineering capabilities, and it offers the chance to work with advanced technologies, including Generative AI.   

 Key Responsibilities:   

• Lead the design, development, and optimization of data solutions using Databricks, ensuring they are scalable, efficient, and secure.   
• Collaborate with cross-functional teams to gather and analyse data requirements, translating them into robust data architectures and solutions.   
• Develop and maintain ETL pipelines, leveraging Databricks and integrating with Azure Data Factory as needed.   
• Implement machine learning models and advanced analytics solutions, incorporating Generative AI to drive innovation.   
• Ensure data quality, governance, and security practices are adhered to, maintaining the integrity and reliability of data solutions.   
• Provide technical leadership and mentorship to junior engineers, fostering an environment of learning and growth.   
• Stay updated on the latest trends and advancements in data engineering, Databricks, Generative AI, and Azure Data Factory to continually enhance team capabilities.   

Qualifications:   

• Bachelor’s or master’s degree in computer science, Information Technology, or a related field.   
• 7+ to 10 years of experience in data engineering, with a focus on Databricks.   
• Proven expertise in building and optimizing data solutions using Databricks and integrating with Azure Data Factory/AWS Glue.   
• Proficiency in SQL and programming languages such as Python or Scala.   
• Strong understanding of data modelling, ETL processes, and Data Warehousing/Data Lakehouse concepts.   
• Familiarity with cloud platforms, particularly Azure, and containerization technologies such as Docker.   
• Excellent analytical, problem-solving, and communication skills.   
• Demonstrated leadership ability with experience mentoring and guiding junior team members.   

Preferred Skills:   

• Experience with Generative AI technologies and their applications.   
• Familiarity with other cloud platforms, such as AWS or GCP.   
• Knowledge of data governance frameworks and tools.