Data Engineer - Databricks

Overview

We are seeking a highly skilled and motivated Databricks Data Engineer with at least 4+ years of experience in the data engineering domain to join our dynamic data engineering team. The ideal candidate will have hands-on experience with Databricks, Spark, and other cloud technologies, along with a strong background in data pipeline development, ETL processes, and working with large datasets in a cloud environment.

As a Senior Databricks Data Engineer, you will play a key role in designing, building, and maintaining scalable data pipelines, optimizing data workflows, and working closely with data scientists, analysts, and other engineers to provide actionable insights.

Key Responsibilities

• Design, implement, and optimize data pipelines using Databricks and Apache Spark to handle large-scale datasets.
• Develop and maintain efficient ETL workflows and perform data integration from a variety of data sources into cloud-based environments (AWS, Azure, or GCP).
• Collaborate with data scientists, analysts, and business stakeholders to understand requirements and deliver reliable data solutions.
• Optimize Databricks notebooks, workflows, and data processing jobs for performance and scalability.
• Implement best practices in data security, governance, and performance optimization within the Databricks ecosystem.
• Work closely with the DevOps team to automate deployment pipelines and ensure robust monitoring and alerting.
• Troubleshoot performance issues and optimize data processing to meet service level agreements (SLAs).
• Stay up to date with the latest technologies in data engineering, particularly Databricks, Spark, and related frameworks.