Senior Data Engineer (Fabric)

Job Purpose:

We are looking for a Senior Data Engineer with extensive experience in developing ETL processes using PySpark Notebooks and Microsoft Fabric, and supporting existing legacy SQL Server environments. The ideal candidate will possess a strong background in Spark-based development, demonstrate a high proficiency in SQL, and be comfortable working independently, collaboratively within a team, or leading other developers when required, coupled with strong communication skills.

Requirements:

We are looking for a Senior Data Engineer with extensive experience in developing ETL processes using PySpark Notebooks and Microsoft Fabric, and supporting existing legacy SQL Server environments. The ideal candidate will possess a strong background in Spark-based development, demonstrate a high proficiency in SQL, and be comfortable working independently, collaboratively within a team, or leading other developers when required, coupled with strong communication skills. The ideal candidate will possess Experience with Azure Data Services, including Azure Data Factory, Azure Synapse or similar tools,Experience of creating DAG's, implementing activities, and running Apache Airflow and Familiarity with DevOps practices, CI/CD pipelines and Azure DevOps.

The ideal candidate should have:

Key Responsibilities:

• Design, develop, and maintain ETL Notebook orchestration pipelines using PySpark and Microsoft Fabric.
• Working with Apache Delta Lake tables, Change Data Feed (CDF), Lakehouses and custom libraries
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver efficient data solutions.
• Migrate and integrate data from legacy SQL Server environments into modern data platforms.
• Optimize data pipelines and workflows for scalability, efficiency, and reliability.
• Provide technical leadership and mentorship to junior developers and other team members.
• Troubleshoot and resolve complex data engineering issues related to performance, data quality, and system scalability.
• Debugging of code, breaking down to test components, identify issues and resolve
• Develop, maintain, and enforce data engineering best practices, coding standards, and documentation.
• Conduct code reviews and provide constructive feedback to improve team productivity and code quality.
• Support data-driven decision-making processes by ensuring data integrity, availability, and consistency across different platforms.

Qualifications:

• Bachelor’s or Master’s degree in Computer Science, Data Science, Engineering, or a related field.
• 10+ years of experience in data engineering, with a strong focus on ETL development using PySpark or other Spark-based tools.
• Proficiency in SQL with extensive experience in complex queries, performance tuning, and data modeling.
• Experience with Microsoft Fabric or similar cloud-based data integration platforms is a plus.
• Strong knowledge of data warehousing concepts, ETL frameworks, and big data processing.
• Familiarity with other data processing technologies (e.g., Hadoop, Hive, Kafka) is an advantage.
• Experience working with both structured and unstructured data sources.
• Excellent problem-solving skills and the ability to troubleshoot complex data engineering issues.
• Experience with Azure Data Services, including Azure Data Factory, Azure Synapse, or similar tools.
• Experience of creating DAG's, implementing activities, and running Apache Airflow
• Familiarity with DevOps practices, CI/CD pipelines and Azure DevOps.

About Aspire:

Aspire Systems is a global technology services firm serving as a trusted technology partner for more than 275 customers across the globe. Aspire works with the world's most innovative enterprises in Banking, Insurance, Retail and ISVs, helping them leverage technology to transform their business in the current digital landscape. Its proven approach to Software Engineering & Digital Technologies helps companies run smart business models. Aspire’s core philosophy of ‘Attention. Always.’ communicates its belief in lavishing care and attention to its customers and employees. The company currently has over 4900 employees globally and is CMMI Level 3 certified. It operates across North America, LATAM, Europe, Middle East, and Asia Pacific.

Aspire Systems has been recognized for the 12th time in a row as one of the Top 100 ‘Best Companies to Work For’ by the Great Place to Work® Institute. To know more about Aspire Systems, visit https://www.aspiresys.com/.