Data Engineer

Overview

1. About the Role :

As a Data Engineer for the Data Science team, you will play a pivotal role in enriching and maintaining the organizations central repository of datasets. This repository serves as the backbone for advanced data analytics and machine learning applications, enabling actionable insights from financial and market data. You will work closely with cross-functional teams to design and implement robust ETL pipelines that automate data updates and ensure accessibility across the organization.

This is a critical role requiring technical expertise in building scalable data pipelines, ensuring data quality, and supporting data analytics and reporting infrastructure for business growth.

2. Key Responsibilities :

ETL Development :

- Design, develop, and maintain efficient ETL processes for handling multi-scale datasets.

- Implement and optimize data transformation and validation processes to ensure data accuracy and consistency.

- Collaborate with cross-functional teams to gather data requirements and translate business logic into ETL workflows.

Data Pipeline Architecture :

- Architect, build, and maintain scalable and high-performance data pipelines to enable seamless data flow.

- Evaluate and implement modern technologies to enhance the efficiency and reliability of data pipelines.

- Build pipelines for extracting data via web scraping to source sector-specific datasets on an ad hoc basis.

Data Modeling :

- Design and implement data models to support analytics and reporting needs across teams.

- Optimize database structures to enhance performance and scalability.

Data Quality and Governance :

- Develop and implement data quality checks and governance processes to ensure data integrity.

- Collaborate with stakeholders to define and enforce data quality standards across the organization.

Documentation and Communication :

- Maintain detailed documentation of ETL processes, data models, and other key workflows.

- Effectively communicate complex technical concepts to non-technical stakeholders and business users.

Cross-Functional Collaboration :

- Work closely with the Quant team and developers to design and optimize data pipelines.

- Collaborate with external stakeholders to understand business requirements and translate them into technical solutions.

3. Essential Requirements :

Basic Qualifications :

- Bachelors degree in Computer Science, Information Technology, or a related field.

- Proven experience as a Data Engineer with expertise in ETL techniques (minimum 2 years).

- 36 years of strong programming experience in languages such as Python, Java, or Scala.

- Hands-on experience in web scraping to extract and transform data from publicly available web sources.

- Proficiency with cloud-based data platforms such as AWS, Azure, or GCP.

- Strong knowledge of SQL and experience with relational and non-relational databases.

- Deep understanding of data warehousing concepts and architectures.

- Familiarity with big data technologies like Hadoop, Spark, and Kafka.

- Experience with data modeling tools and techniques.

- Excellent problem-solving, analytical, and communication skills.

Preferred Qualifications :

- Masters degree in Computer Science or Data Science.

- Knowledge of data streaming and real-time processing frameworks.

- Familiarity with data governance and security best practices.

Job Type: Full-time

Pay: Up to ₹6,000,000.00 per year

Schedule:

• Day shift
• Monday to Friday

Work Location: In person