Expert Data Engineer(Azure)

Overview

About the Company

At Barry Callebaut, we integrate global expertise with the strength of global talent to drive innovation and efficiency across our global operations. Our Global Business Services (GBS) center, located in Hyderabad, plays a pivotal role in ensuring seamless delivery across 147 facilities worldwide, contributing to annual revenues of CHF 10.5 billion. As a distinct global capability hub, GBS India is uniquely positioned to enhance strategic & operational excellence. In India, we are cultivating a high-performing team that brings specialized capabilities and forward-thinking solutions to our global network. Our professionals lead strategic, transformative initiatives that are shaping the future of the chocolate industry while advancing global careers. By leveraging the promising talent pool, leadership, and business insights, we drive digital innovation, process excellence, and sustainability - contributing meaningfully to our mission of making sustainable chocolate the norm.

About the Role

Barry Callebaut Digital (BC Digital) is on a mission to lead the digital revolution in the chocolate industry, and we're looking for an Expert Data Engineer for the Data Pipelines, to help us build a foundation for actionable insights, based on robust, scalable, and efficient data pipelines. Reporting to the Head of Data Engineering, your focus will lie on ensuring data availability for analytical needs, through the transformation and provisioning of data from various source systems and domains across Barry Callebaut. As part of the central Data Engineering unit, you will be responsible for managing and coordinating external and distributed engineers, while providing community support, training, and governance to drive best practices across the engineering community within BC.

Responsibilities

• Design and build robust and scalable data pipelines to load and transform data from various source systems
• Lead the design, development, and deployment of scalable data solutions using Microsoft Fabric and other Azure Data Services.
• Build and maintain data pipelines to integrate structured and unstructured data for advanced analytics and business intelligence.
• Collaborate with architects and business stakeholders to create logical and physical data models.
• Design and build robust and scalable platform capabilities
• Leverage Microsoft Fabric capabilities for seamless data integration, transformation, and visualization.
• Ensure efficient use of Fabric’s unified analytics platform, connecting OneLake, Synapse, and Power BI across the organization.
• Design and manage ETL/ELT pipelines for large-scale data ingestion using MS Fabric Dataflows.
• Implement transformations using Notebooks, or Spark-based solutions
• Optimize and manage data lakes and data warehouses in Azure Data Lake Storage (ADLS) and Microsoft Fabric’s Lakehouse.
• Implement partitioning, caching, and indexing strategies to ensure optimal data performance and query execution.
• Design and build performance monitoring and troubleshooting
• Monitor and improve pipeline performance, identifying and resolving bottlenecks
• Implement validation processes to ensure data is accurate, consistent, and accessible
• Build automations for repetitive tasks related to data analytics
• Perform root cause analysis and implement corrective measures
• Identify opportunities to optimize pipelines for performance and cost efficiency
• Collaborate across stakeholders
• Collaborate with data scientists, analysts, and business stakeholders to understand their data requirements and build the right solutions
• Document data engineering processes, systems, and procedures
• Manage and coordinate BC’s external and distributed engineers, ensuring effective collaboration and integration
• Provide community support, training, and governance to drive best practices across the data engineering community
• Lead the development and implementation of global frameworks and patterns for data ingestion and pipelines, working closely with all other internal and external Data Engineer roles

Qualifications

• Degree in IT/technology, data analytics, or a comparable field
• Proficiency in English

Required Skills

• Advanced knowledge of data engineering processes, with a minimum of 7 years of experience (senior)
• Comprehensive understanding of data engineering patterns and best practices for pipeline orchestration
• Extensive experience in developing data analytics solutions on Azure
• Strong background in designing and building efficient, reliable, and automated data pipelines, ETL workflows, data warehousing, and Big Data processing, with experience in respective technologies like Airflow, dbt, etc.
• Proficiency in Spark and Databricks Lakehouse technologies
• Hands-on expertise with Python or PySpark
• In-depth knowledge of Azure Stack components like Azure Databricks, Azure Data Factory, Azure Synapse, and ADLS Gen2 from a data engineering perspective
• Familiarity with utilizing DevOps and DataOps methodologies to enhance development and deployment practices (e.g., CI/CD)
• Experience in managing and coordinating data engineers, providing community support, training, and governance to drive best practices
• Expertise in leading the development and implementation of global frameworks and patterns
• Excellent problem-solving skills and structured way of working

Preferred Skills

• Collaborates well across diverse and globally distributed teams, with the ability to build and maintain positive relationships across different levels and functions of the organization
• Is a true team player, supporting their colleagues by sharing knowledge and experience and commits to the teams’ joint success
• Is capable of effective communication, conveying complex problems in a clear and persuasive manner to internal and external stakeholders
• Is open to try and learn new technologies and skills
• Has an independent and self-driven personality, taking responsibility and owning tasks
• Can guide diverse teams and drive change in a collaborative way on a global scale
• Can create internal and external partnerships/networks across the organization with technical and non-technical stakeholders

At Barry Callebaut, we are committed to Diversity & Inclusion. United by our strong values, we thrive on the diversity of who we are, where we come from, what we’ve experienced and how we think. We are committed to nurturing an inclusive environment where people can truly be themselves, grow to their full potential and feel they belong. #oneBC - Diverse People, Sustainable Growth.