Senior Data Engineer-DataBricks

Job Title: Senior Data Engineer

Location: Hyderabad, Chennai & Bangalore

Experience: 5+ Years

Job Summary

We are looking for a highly skilled and experienced Senior Data Engineer to join our growing data engineering team. The ideal candidate will have a strong background in building and optimizing data pipelines and data architecture, as well as experience with big data technologies and Azure cloud services. You will work closely with cross-functional teams to ensure data is accessible, reliable, and ready for analytics and business insights.

Mandatory Skills

• Advanced SQL
• Python or Scala for data engineering
• ETL pipeline development
• Cloud platforms (AWS/GCP/Azure)
• Azure 1st party services (ADF, Azure Databricks, Synapse, etc.)
• Big data tools (Spark, Hadoop)
• Data warehousing (Redshift, Snowflake, Big Query)
• Workflow orchestration tools (Airflow, Prefect, or similar)
  
  

Key Responsibilities

• Design, develop, and maintain scalable and reliable data pipelines
• Demonstrate experience and leadership across two full project cycles using Azure Data Factory, Azure Databricks, and PySpark
• Collaborate with data analysts, scientists, and software engineers to understand data needs
• Implement data quality checks and monitoring systems
• Optimize data delivery and processing across a wide range of sources and formats
• Ensure security and governance policies are followed in all data handling processes
• Evaluate and recommend tools and technologies to improve data engineering capabilities
• Lead and mentor junior data engineers as needed
• Work with cross-functional teams in a dynamic and fast-paced environment
  
  

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Engineering, or related field
• Certification in cloud platforms or big data technologies (preferred)
• 5+ years of data engineering experience
  
  

Technical Skills

• Programming: Python, Scala, SQL
• Big Data: Spark, Hadoop, Hive
• Stream Processing: Storm, Spark Streaming
• Data Warehousing: Snowflake, Big Query, Redshift
• Cloud: AWS (S3, Lambda, Glue), GCP, Azure (ADF, Azure Databricks)
• Orchestration: Apache Airflow, Prefect, Luigi
• Databases: PostgreSQL, MySQL, NoSQL (MongoDB, Cassandra)
• Tools: Git, Docker, Kubernetes (basic), CI/CD
  
  

Soft Skills

• Strong problem-solving and analytical thinking
• Excellent verbal and written communication
• Ability to manage multiple tasks and deadlines
• Collaborative mindset with a proactive attitude
• Strong analytic skills related to working with unstructured datasets
  
  

Good to Have

• Experience with real-time data processing (Kafka, Flink)
• Knowledge of data governance and privacy regulations (GDPR, HIPAA)
• Familiarity with ML model data pipeline integration
  
  

Work Experience

• Minimum 5 years of relevant experience in data engineering roles
• Experience with Azure 1st party services across at least two full project lifecycles
  
  

Compensation & Benefits

• Competitive salary and annual performance-based bonuses
• Comprehensive health and optional Parental insurance.
• Retirement savings plans and tax savings plans.
• Work-Life Balance: Flexible work hours
  
  

Key Result Areas (KRAs)

• Timely development and delivery of high-quality data pipelines
• Implementation of scalable data architectures
• Collaboration with cross-functional teams for data initiatives
• Compliance with data security and governance standards
  
  

Key Performance Indicators (KPIs)

• Uptime and performance of data pipelines
• Reduction in data processing time
• Number of critical bugs post-deployment
• Stakeholder satisfaction scores
• Successful data integrations and migrations
  
  

Contact: hr@bigtappanalytics.com