Vishakhapatnam, Andhra Pradesh, India
Information Technology
Full-Time
Virtusa
Overview
Experience
8 years of experience in data engineering, specifically in cloud environments like AWS.
Proficiency in PySpark for distributed data processing and transformation.
Solid experience with AWS Glue for ETL jobs and managing data workflows.
Hands-on experience with AWS Data Pipeline (DPL) for workflow orchestration.
Strong experience with AWS services such as S3, Lambda, Redshift, RDS, and EC2.
Technical Skills
Proficiency in Python and PySpark for data processing and transformation tasks.
Deep understanding of ETL concepts and best practices.
Familiarity with AWS Glue (ETL jobs, Data Catalog, and Crawlers).
Experience building and maintaining data pipelines with AWS Data Pipeline or similar orchestration tools.
Familiarity with AWS S3 for data storage and management, including file formats (CSV, Parquet, Avro).
Strong knowledge of SQL for querying and manipulating relational and semi-structured data.
Experience with Data Warehousing and Big Data technologies, specifically within AWS.
Additional Skills
Experience with AWS Lambda for serverless data processing and orchestration.
Understanding of AWS Redshift for data warehousing and analytics.
Familiarity with Data Lakes, Amazon EMR, and Kinesis for streaming data processing.
Knowledge of data governance practices, including data lineage and auditing.
Familiarity with CI/CD pipelines and Git for version control.
Experience with Docker and containerization for building and deploying applications.
Design and Build Data Pipelines: Design, implement, and optimize data pipelines on AWS using PySpark, AWS Glue, and AWS Data Pipeline to automate data integration, transformation, and storage processes.
ETL Development: Develop and maintain Extract, Transform, and Load (ETL) processes using AWS Glue and PySpark to efficiently process large datasets.
Data Workflow Automation: Build and manage automated data workflows using AWS Data Pipeline, ensuring seamless scheduling, monitoring, and management of data jobs.
Data Integration: Work with different AWS data storage services (e.g., S3, Redshift, RDS) to ensure smooth integration and movement of data across platforms.
Optimization and Scaling: Optimize and scale data pipelines for high performance and cost efficiency, utilizing AWS services like Lambda, S3, and EC2.
8 years of experience in data engineering, specifically in cloud environments like AWS.
Proficiency in PySpark for distributed data processing and transformation.
Solid experience with AWS Glue for ETL jobs and managing data workflows.
Hands-on experience with AWS Data Pipeline (DPL) for workflow orchestration.
Strong experience with AWS services such as S3, Lambda, Redshift, RDS, and EC2.
Technical Skills
Proficiency in Python and PySpark for data processing and transformation tasks.
Deep understanding of ETL concepts and best practices.
Familiarity with AWS Glue (ETL jobs, Data Catalog, and Crawlers).
Experience building and maintaining data pipelines with AWS Data Pipeline or similar orchestration tools.
Familiarity with AWS S3 for data storage and management, including file formats (CSV, Parquet, Avro).
Strong knowledge of SQL for querying and manipulating relational and semi-structured data.
Experience with Data Warehousing and Big Data technologies, specifically within AWS.
Additional Skills
Experience with AWS Lambda for serverless data processing and orchestration.
Understanding of AWS Redshift for data warehousing and analytics.
Familiarity with Data Lakes, Amazon EMR, and Kinesis for streaming data processing.
Knowledge of data governance practices, including data lineage and auditing.
Familiarity with CI/CD pipelines and Git for version control.
Experience with Docker and containerization for building and deploying applications.
Design and Build Data Pipelines: Design, implement, and optimize data pipelines on AWS using PySpark, AWS Glue, and AWS Data Pipeline to automate data integration, transformation, and storage processes.
ETL Development: Develop and maintain Extract, Transform, and Load (ETL) processes using AWS Glue and PySpark to efficiently process large datasets.
Data Workflow Automation: Build and manage automated data workflows using AWS Data Pipeline, ensuring seamless scheduling, monitoring, and management of data jobs.
Data Integration: Work with different AWS data storage services (e.g., S3, Redshift, RDS) to ensure smooth integration and movement of data across platforms.
Optimization and Scaling: Optimize and scale data pipelines for high performance and cost efficiency, utilizing AWS services like Lambda, S3, and EC2.
Similar Jobs
View All
1 Day ago
Travelxp - Senior Software Developer - Backend Technologies
Information Technology
- Mumbai, Maharashtra, India
About The JobJob Title : Senior Software DeveloperLocation : Mumbai OnsiteExperience : 2-3 yearsRole OverviewWe are looking for a Senior Software Developer - Backend Focus who's passionate about building robust, scalable systems and pushing the boun...
1 Day ago
Technical Lead
Information Technology
- Mumbai, Maharashtra, India
Tech Lead(Fullstack) - Nexa (Conversational Voice AI Platform)Location : Bangalore Type: Full-timeExperience : 4+ years (preferably in early-stage startups)Tech Stack : Python (core), Node.js, React.jsAbout NexaNexa is a new venture by the founders ...
1 Day ago
Sr. Software Engineer - .NET C# Job
Information Technology
- Mumbai, Maharashtra, India
We use cookies to offer you the best possible website experience. Your cookie preferences will be stored in your browser’s local storage. This includes cookies necessary for the website's operation. Additionally, you can freely decide and change any...
1 Day ago
Zinnia - Senior Data Scientist - Analytics/Python
Information Technology
- Mumbai, Maharashtra, India
Who We AreZinnia is the leading technology platform for accelerating life and annuities growth.With innovative enterprise solutions and data insights, Zinnia simplifies the experience of buying, selling, and administering insurance products.All of w...
1 Day ago
Senior Data/BI Engineer - Tableau/Power BI
Information Technology
- Mumbai, Maharashtra, India
What You'll Do Data Analytics & Modeling : Apply strong Data Analytics and Analytical Skills to understand complex business requirements and translate them into effective Data Modeling solutions. Data Pipeline Development : Design, develop, and ma...
1 Day ago
BigStep Technologies - Artificial Intelligence/Machine Learning Engineer - LLM Models
Information Technology
- Mumbai, Maharashtra, India
Responsibilities Research, design, develop, and test Python code. Collaborate with the Product and Data Science teams to define core AI/ML platform features. Understand business requirements, functional specifications, and research pre-trained mo...
1 Day ago
TIFIN - IT System Administrator
Information Technology
- Mumbai, Maharashtra, India
ResponsibilitiesWe are looking for a proactive and detail-oriented Systems Administrator - IT Operations to join our team in Mumbai. In this role, you will be responsible for managing IT infrastructure, ensuring system uptime, supporting end-user de...
1 Day ago
Markovate - Senior Data Engineer - Spark/Hadoop
Information Technology
- Mumbai, Maharashtra, India
About Markovate.At Markovate, we dont just follow trendswe drive them.We transform businesses through innovative AI and digital solutions that turn vision into reality.Our team harnesses breakthrough technologies to craft bespoke strategies that ali...
Talk to us
Feel free to call, email, or hit us up on our social media accounts.
Email
info@antaltechjobs.in