Lead Python Data Engineer - Databricks

Overview

Job Summary:
We are seeking a highly skilled Data Engineer with a strong background in Databricks and a
passion for data engineering. The ideal candidate will have extensive experience in
designing and implementing data solutions using various technologies, including PySpark,
Spark SQL, and Power BI. This role requires a proactive individual who can work
independently and collaboratively in a fast-paced environment.

Required Technical Skills:
• 7+ years of hands-on experience in software development, with a strong emphasis
on Python programming.
• Deep expertise in PySpark (Spark SQL, DataFrames, and Datasets) for building and
optimizing scalable data pipelines. (PySpark)
• Solid experience with Azure Databricks (Unity Catalog), including working with Delta
tables, workflows, and scripting in Python.
• Understanding of data governance, data management, security, and compliance in
cloud environments.
• Familiar with Azure cloud services, especially Azure Data Lake, Azure Data Factory,
• Experience with Power BI, including DAX and Power Query.
• Modeling data for Data Warehousing and Visualization projects through Semantic
Layers.
• Skilled in writing complex SQL queries and building views involving joins across
multiple tables.
• Proficient in tools like DevOps (project management), and version control and
CI/CD pipelines.

Nice to have Skills:
• Experience Azure AI, Azure Copilot Studio and Azure Fabric.
• Experience building workflows with Power Automate.
• EventHub or Kafka for streaming data ingestion.
• Proficient in designing and implementing real-time data streaming solutions.
• Experience across the Software Development Lifecycle (SDLC), with a strong grasp
of Agile Scrum methodologies.

Soft Skills & Collaboration:
• Strong collaboration mindset with enthusiasm for mobbing sessions and cross
team coordination.
• Excellent problem-solving and code review skills.
• Strong communication skills with the ability to understand business requirements
and translate them into technical solutions.
• Proactive and responsive during business hours for any ad-hoc calls or meetings.
• Demonstrated ability to prioritize effectively, think creatively, and adapt quickly to
new technologies and feedback.