Lead Data Engineer

We are seeking a highly skilled Data Engineer with expertise in building scalable data pipelines and working with Azure Cloud services. The ideal candidate will be proficient in designing and implementing end-to-end data pipelines, ensuring high performance and scalability. You will leverage Azure services like Azure Data Factory, Azure SQL Database, and Azure Databricks, as well as PySpark for data transformations. This is an excellent opportunity to make a significant impact in an innovative environment.

Key Responsibilities:

• Data Pipeline Development: Design, implement, and optimize end-to-end data pipelines on Azure, focusing on scalability, performance, and reliability.

• ETL Workflow Development: Develop and maintain ETL workflows to ensure seamless and efficient data processing.

• Azure Cloud Expertise: Utilize Azure services such as Azure Data Factory, Azure SQL Database, and Azure Databricks for effective data engineering solutions.

• Data Storage Solutions: Implement and manage data storage solutions on Azure, ensuring optimal performance and scalability.

• Data Transformation: Use PySpark to perform advanced data transformations, ensuring high-quality and well-structured data outputs.

• Data Cleansing and Enrichment: Implement data cleansing, enrichment, and validation processes using PySpark. • Performance Optimization: Optimize data pipelines, queries, and PySpark jobs to enhance overall system performance and scalability.

• Bottleneck Identification: Identify and resolve performance bottlenecks within data processing workflows, ensuring efficiency. Must Have Skills:

• Azure Expertise: Proven experience with Azure Cloud services, especially Azure Data Factory, Azure SQL Database, and Azure Databricks.

• PySpark Proficiency: Expertise in PySpark for data processing and analytics.

• Data Engineering: Strong background in building and optimizing data pipelines and workflows.

• ETL Processes: Solid experience with data modeling, ETL processes, and data warehousing.

• Performance Tuning: Ability to optimize data pipelines and jobs to ensure scalability and performance.

• Problem-Solving: Experience troubleshooting and resolving performance bottlenecks in data workflows.

• Collaboration: Strong communication skills and ability to work collaboratively in cross-functional teams.

Good to Have Skills:

• Azure Certifications: Any relevant Azure certifications will be considered a plus.

• Advanced Data Transformation: Experience with advanced data transformations using PySpark and other tools.

• Data Governance: Knowledge of data governance best practices in the context of data engineering and cloud solutions.

• Big Data Technologies: Familiarity with other big data technologies and tools such as Hadoop, Spark, and Kafka.

• Data Warehousing: Experience in designing and implementing data warehousing solutions in the Azure environment.

Qualification:

• Bachelor’s degree in Computer Science, Engineering, Information Technology, or a related field.

• 5-8 years of professional experience as a Data Engineer with a focus on Azure technologies.