Databricks Developer / Data Engineer - DDDE 0724 SS #2

Job Title: Databricks Developer / Data Engineer

Duration - 12 Months with Possible Extension

Location: Hyderabad, Telangana (Hybrid) 1-2 days onsite at client location

Job Summary:

We are seeking a highly skilled Databricks Developer / Data Engineer with 5+ years of experience in building scalable data pipelines, managing large datasets, and optimizing data workflows in cloud environments. The ideal candidate will have hands-on expertise in Azure Databricks, Azure Data Factory, and other Azure-native services, playing a key role in enabling data-driven decision-making across the organization.

Key Responsibilities:

• Design, develop, and maintain scalable ETL/ELT pipelines for data ingestion, transformation, and integration
• Work with both structured and unstructured data from a variety of internal and external sources
• Collaborate with data analysts, scientists, and engineers to ensure data quality, integrity, and availability
• Build and manage data lakes, data warehouses, and data models (Azure Databricks, Azure Data Factory, Snowflake, etc.)
• Optimize performance of large-scale batch and real-time processing systems
• Implement data governance, metadata management, and data lineage practices
• Monitor and troubleshoot pipeline issues; perform root cause analysis and proactive resolution
• Automate data validation and quality checks
• Ensure compliance with data privacy, security, and regulatory requirements
• Maintain thorough documentation of architecture, data workflows, and processes

Mandatory Qualifications:

• 5+ years of hands-on experience with:

• Azure Blob Storage, Azure Data Lake Storage, Azure SQL Database

• Azure Logic Apps, Azure Data Factory, Azure Databricks, Azure ML

• Azure DevOps Services, Azure API Management, Webhooks
  

  • Intermediate-level proficiency in Python scripting and PySpark
  • Basic understanding of Power BI and visualization functionalities

Technical Skills & Experience Required:

• Proficient in SQL and working with both relational and non-relational databases (e.g., SQL, PostgreSQL, MongoDB, Cassandra)
• Hands-on experience with Apache Spark, Hadoop, Hive for big data processing
• Proficiency in building scalable data pipelines using Azure Data Factory and Azure Databricks
• Solid knowledge of cloud-native tools: Delta Lake, Azure ML, Azure DevOps
• Understanding of data modeling, OLAP/OLTP systems, and data warehousing best practices
• Experience with CI/CD pipelines, version control with Git, and working with Azure Repos
• Knowledge of data security, privacy policies, and compliance frameworks
• Excellent problem-solving, troubleshooting, and analytical skills