Data Engineer

Overview

• Job Description :
• We are seeking a skilled
• Data Engineer
• to design, build, and optimize scalable data pipelines for processing large datasets. The ideal candidate will work with modern data tools and cloud platforms to develop robust, high-performance data systems that support
• analytics and AI-driven applications

Requirements

• Design and architect scalable ETL/ELT pipelines using Apache Airflow, dbt, and Kafka for real-time and batch data processing.
• Build and maintain data lakes and data warehouses on platforms such as AWS S3, Snowflake, or Databricks, leveraging Delta Lake for ACID-compliant data management.
• Develop and implement data quality frameworks using Great Expectations and monitor data systems using Prometheus and Grafana.
• Optimize data infrastructure for performance and cost using Kubernetes, Terraform (Infrastructure as Code), and serverless services such as AWS Lambda and AWS Glue.
• Collaborate with cross-functional teams to implement CI/CD pipelines using GitHub Actions or GitLab CI, integrating with dbt Cloud for data transformations.
• Develop and manage real-time data processing solutions using Apache Spark (PySpark/Databricks).Work with vector databases such as Pinecone to support AI and machine learning applications.

Required Skills

• Strong programming skills in Python and SQL.
• Hands-on experience with cloud platforms such as AWS, GCP, or Azure (e.g., EMR, Redshift, BigQuery).
• Experience with modern data tools:
• Kafka or Flink for streaming
• dbt for data transformation
• Airflow for workflow orchestration
• Familiarity with containerization technologies like Docker and Kubernetes.
• Understanding of observability tools and frameworks such as ELK Stack.

Preferred Skills

• Experience with machine learning pipelines using MLflow.
• Knowledge of data governance tools such as Collibra.
• Experience working with HIPAA-compliant data environments.
• Exposure to Scala or Java for Apache Spark development.

Benefits
5 Days working
Flexible Timing
12 Paid Leaves
12 Official Holidays
Festival Celebrations
Birthday Celebrations
Friendly work Culture

Pay: ₹509,356.57 - ₹1,166,829.63 per year

Work Location: In person