Databricks Data Engineer/Senior Engineer

This position will primarily be responsible for designing, developing, and maintaining robust ETL/ELT pipelines for data ingestion, transformation, and storage. The role also involves designing and developing scalable data solutions.

The person will work with a team responsible for ensuring the availability, reliability, and performance of data systems and requires a good understanding of infrastructure management, cost optimization, and performance tuning in Databricks environments. The candidate will design, develop, and maintain scalable data pipelines of Databricks on cloud platforms (Databricks, Azure, AWS, or GCP).

The person will also work with a collaborative team responsible for driving client performance by combining data-driven insights and strategic thinking to solve business challenges. The candidate should have strong organizational, critical thinking, and communication skills to interact effectively with stakeholders.

Responsibilities:

• Design, develop, and manage end-to-end data pipelines on Databricks using Spark, Delta Lake, and related technologies
• Implement and optimize ETL/ELT workflows for ingesting, transforming, and storing large volumes of structured and semi-structured data
• Manage and monitor Databricks infrastructure, including cluster configuration, auto-scaling, job execution, and resource utilization
• Identify and implement cost-saving strategies across Databricks workspaces through efficient pipeline design, job scheduling, and infrastructure tuning
• Orchestrate workflows using tools like Apache Airflow, Azure Data Factory, or similar orchestration frameworks
• Collaborate with data architects and platform engineers to ensure efficient data architecture and performance tuning
• Monitor data quality, integrity, and lineage across all pipelines and proactively troubleshoot data issues
• Develop, maintain, and optimize data models and storage layers that support BI/reporting and analytics use cases
• Partner with business stakeholders to translate business needs into technical specifications and scalable data solutions
• Maintain comprehensive documentation of pipeline architecture, configurations, and operational best practices

Technical Qualifications:

• 3+ years of experience in Data Engineering with at least 2 years of hands-on experience with Databricks (Azure, AWS, or GCP)
• Strong proficiency in Apache Spark and Delta Lake within the Databricks ecosystem
• Experience in infrastructure management on Databricks: cluster setup, scaling, security roles, workspace configuration
• Demonstrated ability in cost optimization techniques in Databricks (e.g., job cluster vs. all-purpose cluster usage, cost/performance tradeoffs)
• Proficient in Python and SQL for data processing and pipeline development
• Experience with orchestration tools such as Apache Airflow, Azure Data Factory, or AWS Glue Workflows
• Familiarity with data warehousing, data lakes, and distributed data processing frameworks
• Exposure to BI tools like Power BI, Tableau, or Looker is a plus
• Experience with relational (e.g., PostgreSQL, SQL Server) and NoSQL (e.g., MongoDB) databases

Personal Skills:

• Strong analytical skills: ability to read business requirements, analyze problems, and propose solutions
• Ability to identify alternatives and find optimal solutions
• Ability to follow through and ensure logical implementation
• Quick learner with the ability to adapt to new concepts and software
• Ability to work effectively in a team environment
• Strong time management skills, capable of handling multiple tasks and competing deadlines
• Effective written and verbal communication skills

Education and Work Experience:

• Background in Computer Science, Information Technology, Data Science, or a related field preferred
• Minimum 3 years of total experience, with at least 2 years of relevant experience in data engineering and data pipeline development
• Certification in Databricks, Azure Data Engineering, or any related data technology is an added advantage