Senior Data Engineer

Overview

4+ years of experience in Data Engineering with at least 2+ years working on Databricks.

AWS Cloud Services:

Hands-on experience with AWS ecosystem, including S3, Glue, Redshift, DynamoDB, Lambda, and other AWS data services.

Programming Languages:

Proficiency in Python (PySpark), Scala, or SQL for data processing and transformation.

Databricks:

Extensive experience with Databricks Workspace, Delta Lake, and managing

Databricks jobs and pipelines.

Big Data Frameworks:

Strong knowledge of Apache Spark for distributed data processing.

Data Warehousing:

Experience with modern data warehouse solutions, including Redshift, Snowflake, or

Databricks SQL.

Version Control & CI/CD:

Familiarity with Git, Terraform, and CI/CD pipelines for deploying data solutions.

Monitoring & Debugging:

Experience with tools like CloudWatch, Datadog, or equivalent for pipeline monitoring and troubleshooting.

Preferred Qualifications:

Certification in AWS Data Analytics or Databricks.

Experience with real-time data streaming tools like Kafka, Kinesis, or AWS MSK.

Knowledge of data governance and data security best practices.

Exposure to machine learning workflows and integration with Databricks.

Key Responsibilities:

Data Pipeline Development:

Design, implement, and manage scalable ETL/ELT pipelines using AWS services and Databricks.

Data Integration:

Ingest and process structured, semi-structured, and unstructured data from multiple sources into AWS Data Lake or Databricks.

Data Transformation:

Develop advanced data processing workflows using PySpark, Databricks SQL, or

Scala to enable analytics and reporting.

Databricks Management:

Configure and optimize Databricks clusters, notebooks, and jobs for performance and cost efficiency.

AWS Architecture:

Design and implement solutions leveraging AWS-native services like S3, Glue, Redshift, EMR, Lambda, Kinesis, and Athena.

Collaboration:

Work closely with Data Analysts, Data Scientists, and other Engineers to understand business requirements and deliver data-driven solutions.

Performance Tuning:

Optimize data pipelines, storage, and queries for performance, scalability, and reliability.

Monitoring and Security:

Ensure data pipelines are secure, robust, and monitored using CloudWatch, Datadog, or equivalent tools.

Documentation:

Maintain clear and concise documentation for data pipelines, workflows, and architecture.

Note: Please send Cv only, who can attend the interview weekdays, with short notice period (max 15 days only)

Job Location: Chennai (Work From Office, working hours are 1:00 PM to 10:00 PM (IST), Monday to Friday)

Job Type: Full-time

Pay: ₹1,400,000.00 - ₹2,000,000.00 per year

Benefits:

• Provident Fund

Schedule:

• Day shift

Supplemental Pay:

• Yearly bonus

Application Question(s):

• If you are a immediate joiner and able to attend the interview on a short notice, Please send your resume to nagesh@global-tech.co.in

Experience:

• ETL: 4 years (Required)
• AWS: 4 years (Required)
• Databricks: 4 years (Required)
• Data Engineering: 4 years (Required)

Location:

• Chennai, Tamil Nadu (Required)

Work Location: In person