Data Engineer

Overview

Job Description

• We are looking for a Data Engineer who has working knowledge of building and

maintaining scalable data pipelines on-premises and on the cloud. This includes
understanding the input and output data sources, upstream downstream dependencies
and ensuring data quality. A key aspect of this role will be focusing on the deprecation
of migrated workflows and migration of workflows into new systems (if needed).

• The ideal candidate should be experienced with tools and technologies such as Git, Apache

Airflow, Apache Spark, SQL, data migration, and data validation.

Key Responsibilities:

Workflow Deprecation

Plan and execute the deprecation of migrated workflows by evaluating current workflows; dependencies and consumption.

Utilize tools and best practices to identify, mark, and communicate deprecated workflows to stakeholders.

Data Migration

Plan and execute data migration tasks to move data between different storage systems or formats.
Ensure the accuracy and completeness of data during migration processes.
Implement strategies to accelerate the pace of data migration by backfilling, validating, and making new data assets ready for use.

Data Validation

Define and implement data validation rules to ensure data accuracy, completeness, and reliability.
Utilize data validation solutions and anomaly detection methods to monitor data quality.

Workflow Management

Use Apache Airflow to schedule, monitor, and automate data workflows.
Develop and manage DAGs (Directed Acyclic Graphs) in Airflow to orchestrate complex data processing tasks.

Data Processing

Develop and maintain data processing scripts using SQL and Apache Spark.
Optimize data processing for performance and efficiency.

Version Control
Use Git for version control, collaborating with the team to manage the codebase and track changes.
Ensure best practices in code quality and repository management.

Continuous Improvement

Keep up to date with the latest developments in data engineering and related technologies.
Continuously improve and refactor data pipelines, tooling, and processes to enhance performance and reliability.

Skills and Qualifications:

• Bachelor's degree in Computer Science, Engineering, or a related field.
• Proficient in Git for version control and collaborative development.
• Proficiency in SQL and experience with database technologies.
• Experience in data pipeline tools such as Apache Airflow.
• Strong knowledge of Apache Spark for data processing and transformation.
• Experience with data migration and validation techniques.
• Knowledge of data governance and security practices.
• Strong problem-solving skills and the ability to work independently and in a team.
• Ability to communicate with global team
• Ability to work as a team in high performing environment.

Experience

• 4 – 6 years

Location

• Bangalore

Job Types: Full-time, Permanent

Pay: ₹900,000.00 - ₹1,000,000.00 per year

Benefits:

• Provident Fund

Schedule:

• Day shift

Application Question(s):

• We are looking for immediate joiner. Can you join immediately?
• What's your current CTC?

Location:

• Bengaluru, Karnataka (Preferred)

Work Location: In person