Data Engineer

Overview

Data Engineer

Project Role Description : Design, develop and maintain data solutions for data generation, collection, and processing. Create data pipelines, ensure data quality, and implement ETL (extract, transform and load) processes to migrate and deploy data across systems.

Must have skills : Python (Programming Language)

Good to have skills : MySQL, Kubernetes, Data Engineering, DuckDB, GIT

Minimum 5 Year(s) Of Experience Is Required

Educational Qualification : 15 years full time education

Summary

As a Python Data Engineer, You will be responsible for designing, developing, and maintaining Python-based data pipelines, applications and services. You will collaborate with cross-functional teams to deliver high-quality software solutions that meet the needs of our business divisions.

Roles & Responsibilities

Design, develop, and maintain robust data pipelines using Python.

Implement ETL (Extract, Transform, Load) processes to ingest data from various sources.

Optimize and manage data storage solutions, ensuring data integrity and performance.

Collaborate with other engineers and analysts to understand data requirements and deliver solutions.

Monitor and troubleshoot data pipeline issues, ensuring timely resolution.

Develop and maintain documentation for data engineering processes and systems

Ensure data security and compliance with relevant regulations and Standards.

Stay updated with the latest industry trends and technologies in data engineering.

Professional & Technical Skills

Proven experience as a Data Engineer, with a focus on Python.

Strong proficiency in Python programming and relevant libraries (e.g., Pandas, NumPy)

Experience with SQL and database management systems (e.g., PostgresoL, MySQL)

Familiarity with data pipeline orchestration tools such as Dagster.

Experience with analytical databases like DuckDB.

Knowledge of cloud storage solutions such as Amazon S3 or Azure Blob Storage.

Familiarity with big data technologies (e.g., Hadoop, Spark) is a plus.

Knowledge of cloud platforms (e.g., AWS, Azure, GCP) and related services.

Experience with containerization and orchestration tools (e.g., Docker, Kubernetes)

Familiarity with data visualization tools (e.g., Tableau, Power BI) Understanding of machine learning concepts and frameworks.

Knowledge of version control systems (e.g., Git)

Additional Information

• The candidate should have minimum 6.5 years of experience in Python programming
• A 15 years full time education is required.