Senior Data Engineer- Databricks

Decision Minds is a leader in Data Cloud, Big Data, Cloud Analytics, AI/ML, and Multi-Cloud deployments. We are a team of passionate thought leaders and industry experts who strive for excellence and clarity in creating innovative solutions and products. Our goal is to revolutionize Data Analytics, Artificial Intelligence, Cloud Computing, and Robotic Process Automation, and make a positive impact on people's lives.

Know more about us at www.decisionminds.com | LinkedIn

Location: Bangalore/ Chennai/ Hyderabad (preferred)

Job Title: Senior Data Engineer - Databricks
Skills: Databricks, Oracle, Hadoop, AWS Cloud, PySpark, Data Warehousing, ETL/ELT, Cloud Migration
Education: B.E/B.Tech/Master’s

Key Responsibilities:

• Develop and maintain scalable data pipelines in Databricks to support the migration of data from Oracle-based Data Warehouse (DWH) and Operational Data Store (ODS) systems.
• 
  Analyze and understand existing Oracle schemas, stored procedures, and data transformation logic.
• 
  Translate PL/SQL logic into PySpark/Databricks SQL.
• 
  Develop and maintain Delta Lake-based datasets with appropriate partitioning, indexing, and optimization strategies.
• 
  Perform data validation and reconciliation post-migration, including row count, data integrity, and accuracy checks.
• 
  Build reusable and modular data ingestion and transformation frameworks using Databricks Notebooks, Jobs, and Workflows.
• 
  Collaborate with DevOps teams for CI/CD pipeline integration and efficient deployment.
• 
  Optimize performance of existing pipelines and queries in Databricks.
• 
  Document architecture, transformation logic, and data lineage clearly for operational transparency.

Mandatory Skills & Experience:

• 6 to 8 years of experience in data engineering, with strong experience in DWH/ODS environments.
• 
  Minimum 3+ years hands-on experience in Databricks (including PySpark, SQL, Delta Lake, Workflows).
• 
  Strong understanding of Lakehouse architecture, cloud data platforms, and big data processing.
• 
  Proven experience in migrating data warehouse and ETL workloads from Oracle to cloud platforms.
• 
  Experience with PL/SQL, query tuning, and reverse engineering legacy systems.
• 
  Experience with CI/CD pipelines, version control (e.g., Git), and automated testing.
• 
  Familiarity with data governance, security policies, and compliance in cloud environments.
• 
  Strong communication and documentation skills.

Preferred Skills (Advantage):

• Experience in cloud migration projects (AWS/Azure).
  
• Knowledge of Delta Lake, Unity Catalog, and Databricks workflows.
  
• Exposure to Pentaho and/or TIBCO Data Virtualization/Integration tools will be an added advantage.
• 
  AWS/Azure/Databricks certifications

Tools & Technologies:

• Databricks, Oracle, Hadoop (HDFS, Hive, Sqoop), AWS (S3, RDS)
  
• PySpark, SQL, Python
• 
  CI/CD (Jenkins, GitHub Actions), Orchestration (Airflow, Control-M)
  
• JIRA, Confluence, Git (GitHub/Bitbucket)
  

Cloud Certifications (Preferred):

• Databricks Certified Data Engineer
  
• AWS Certified Solutions Architect/Developer