Bangalore, Karnataka, India
Information Technology
Full-Time
Virtusa
Overview
Experience
8 years of experience in data engineering, specifically in cloud environments like AWS.
Proficiency in PySpark for distributed data processing and transformation.
Solid experience with AWS Glue for ETL jobs and managing data workflows.
Hands-on experience with AWS Data Pipeline (DPL) for workflow orchestration.
Strong experience with AWS services such as S3, Lambda, Redshift, RDS, and EC2.
Technical Skills
Proficiency in Python and PySpark for data processing and transformation tasks.
Deep understanding of ETL concepts and best practices.
Familiarity with AWS Glue (ETL jobs, Data Catalog, and Crawlers).
Experience building and maintaining data pipelines with AWS Data Pipeline or similar orchestration tools.
Familiarity with AWS S3 for data storage and management, including file formats (CSV, Parquet, Avro).
Strong knowledge of SQL for querying and manipulating relational and semi-structured data.
Experience with Data Warehousing and Big Data technologies, specifically within AWS.
Additional Skills
Experience with AWS Lambda for serverless data processing and orchestration.
Understanding of AWS Redshift for data warehousing and analytics.
Familiarity with Data Lakes, Amazon EMR, and Kinesis for streaming data processing.
Knowledge of data governance practices, including data lineage and auditing.
Familiarity with CI/CD pipelines and Git for version control.
Experience with Docker and containerization for building and deploying applications.
Design and Build Data Pipelines: Design, implement, and optimize data pipelines on AWS using PySpark, AWS Glue, and AWS Data Pipeline to automate data integration, transformation, and storage processes.
ETL Development: Develop and maintain Extract, Transform, and Load (ETL) processes using AWS Glue and PySpark to efficiently process large datasets.
Data Workflow Automation: Build and manage automated data workflows using AWS Data Pipeline, ensuring seamless scheduling, monitoring, and management of data jobs.
Data Integration: Work with different AWS data storage services (e.g., S3, Redshift, RDS) to ensure smooth integration and movement of data across platforms.
Optimization and Scaling: Optimize and scale data pipelines for high performance and cost efficiency, utilizing AWS services like Lambda, S3, and EC2.
8 years of experience in data engineering, specifically in cloud environments like AWS.
Proficiency in PySpark for distributed data processing and transformation.
Solid experience with AWS Glue for ETL jobs and managing data workflows.
Hands-on experience with AWS Data Pipeline (DPL) for workflow orchestration.
Strong experience with AWS services such as S3, Lambda, Redshift, RDS, and EC2.
Technical Skills
Proficiency in Python and PySpark for data processing and transformation tasks.
Deep understanding of ETL concepts and best practices.
Familiarity with AWS Glue (ETL jobs, Data Catalog, and Crawlers).
Experience building and maintaining data pipelines with AWS Data Pipeline or similar orchestration tools.
Familiarity with AWS S3 for data storage and management, including file formats (CSV, Parquet, Avro).
Strong knowledge of SQL for querying and manipulating relational and semi-structured data.
Experience with Data Warehousing and Big Data technologies, specifically within AWS.
Additional Skills
Experience with AWS Lambda for serverless data processing and orchestration.
Understanding of AWS Redshift for data warehousing and analytics.
Familiarity with Data Lakes, Amazon EMR, and Kinesis for streaming data processing.
Knowledge of data governance practices, including data lineage and auditing.
Familiarity with CI/CD pipelines and Git for version control.
Experience with Docker and containerization for building and deploying applications.
Design and Build Data Pipelines: Design, implement, and optimize data pipelines on AWS using PySpark, AWS Glue, and AWS Data Pipeline to automate data integration, transformation, and storage processes.
ETL Development: Develop and maintain Extract, Transform, and Load (ETL) processes using AWS Glue and PySpark to efficiently process large datasets.
Data Workflow Automation: Build and manage automated data workflows using AWS Data Pipeline, ensuring seamless scheduling, monitoring, and management of data jobs.
Data Integration: Work with different AWS data storage services (e.g., S3, Redshift, RDS) to ensure smooth integration and movement of data across platforms.
Optimization and Scaling: Optimize and scale data pipelines for high performance and cost efficiency, utilizing AWS services like Lambda, S3, and EC2.
Similar Jobs
View All
5 Hours ago
Software Engineer - C#/Javascript
Information Technology
- 1 - 5 Yrs
- Anywhere in India/Multiple Locations
Job Description :
We are searching for a Software Engineer who is experienced in C#, .NET technologies, and web technologies (Angular, Aurelia). Join our collaborative team, where you'll work together on exciting projects, designing, building, and...
5 Hours ago
Node.js Developer
Information Technology
- 2 - 6 Yrs
- Anywhere in India/Multiple Locations
Job Description:
We are looking for a skilled Node.js Developer to join our backend engineering team. The ideal candidate will be responsible for building scalable server-side applications, working on API development, and integrating with database...
6 Hours ago
Frontend Developer
Information Technology
- 2 - 7 Yrs
- Anywhere in India/Multiple Locations
Experience Level: 2+ Years
Employment Type: Full-Time
About the Role: We are looking for a talented Frontend Developer with expertise in React.js to build interactive and responsive web interfaces. You will work closely with backend developers,...
1 Day ago
Business Analyst (PDM) (H/F) - AIRBUS INDIA PRIVATE LIMITED
Information Technology
Job Description: Role: Business Analyst (PDM) Description: The Business Analyst - PDM is part of IIDPPT siglum located in India. He/she is responsible for the PDM and PDM related legacy products here in India. Success of this position will requir...
1 Day ago
Automation Test Lead - Senior Associate - Operate
Information Technology
At PwC, our people in software and product innovation focus on developing cutting-edge software solutions and driving product innovation to meet the evolving needs of clients. These individuals combine technical experience with creative thinking to ...
1 Day ago
Canvas App, Power Apps, Power automation / Senior Software Engineer
Information Technology
Job DescriptionSome careers shine brighter than others.If you’re looking for a career that will help you stand out, join HSBC and fulfil your potential. Whether you want a career that could take you to the top, or simply take you in an exciting new ...
1 Day ago
CCaaS Business Analyst-Senior Associate
Information Technology
A career within Data and Analytics services will provide you with the opportunity to help organisations uncover enterprise insights and drive business results using smarter data analytics. We focus on a collection of organisational technology capabi...
1 Day ago
Sustainability- Technology & Innovation - Full Stack Developer (Gen AI & Cloud) - Senior Associate
Information Technology
At PwC, our people in software and product innovation focus on developing cutting-edge software solutions and driving product innovation to meet the evolving needs of clients. These individuals combine technical experience with creative thinking to ...
Talk to us
Feel free to call, email, or hit us up on our social media accounts.
Email
info@antaltechjobs.in