Free cookie consent management tool by TermsFeed AWS Data Engineer - Python/PySpark | Antal Tech Jobs
Back to Jobs
3 Days ago

AWS Data Engineer - Python/PySpark

decor
Bangalore, Karnataka, India
Information Technology
Full-Time
AKIRA Insights

Overview

Key Responsibilities

  • Hands on experience in data related activities such as data parsing, cleansing quality definition data pipelines, storage and ETL scripts.
  • Expert knowledge in AWS Data Lake implementation and support (S3, Glue, DMS Athena, Lambda, API Gateway, Redshift).
  • Experiences in programming language Python/PySpark/SQL.
  • Experience with data migration with hands-on experience.
  • Experiences in consuming rest API using various authentication options with in AWS Lambda architecture.
  • Orchestrate triggers, debug and schedule batch job using a AWS Glue, Lambda and step functions.
  • Hands-on experience with Redshift, including data models, storage, and writing effective queries.
  • Understanding of AWS security features such as IAM roles and policies.
  • Knowledge of the Devops tools and CI/CD process.
  • AWS certification in AWS is highly Skills & Expertise :

Cloud (AWS) Expertise

S3 (Simple Storage Service) :

  • Data storage and partitioning, lifecycle policies, data replication, encryption at rest and transit, and data versioning, Data archive and Data sharing :
  • Creation of Lambdas, configuring event-driven functions, monitoring, and integration with other AWS services of s3,Glue,API Gateway, Redshift etc.

Glue

  • Creating & Managing ETL pipelines, Glue crawlers, job scheduling, and integration with S3, Redshift, and :
  • Creating Tables, Views and Stored procedures, parameter tuning, workload management, configuring triggers, Redshift Spectrum usage for querying S3 data.

API Gateway

  • Designing Restful APIs, securing endpoints using IAM or Cognito, throttling and logging API usage.

VPC (Virtual Private Cloud)

  • Aware of existing VPC design, subnets, NAT gateways, peering, routing, and network ACLs for services creation.

ELB (Elastic Load Balancer)

  • Configuring ALB/NLB for load distribution, health checks, sticky sessions, and SSL termination. CloudTrail :
  • Enabling auditing and compliance logging, managing trails, integrating with CloudWatch and third-party SIEM :
  • Knowledge about Sagemaker ML model training and deployment workflows, managing notebooks, endpoints,

and model & DevOps Tools : / GitHub Actions :

  • Managing version control, branching strategies, and automating workflows for testing and deployment.

Jenkins / CloudBees

  • Building pipelines for build-test-deploy automation, plugin management, parallel execution, and integrations with SCM and artifact :
  • Static code analysis, security vulnerability checks, technical debt reporting, integration into CI/CD.

DevOps/MLOps & AI/ML Awareness

  • Understanding of ML model lifecycle : data preprocessing, training, evaluation, deployment, and monitoring.
  • Experience supporting Data Scientists and ML Engineers in deploying models to production.
  • Familiarity with tools and workflows like SageMaker, MLflow, Airflow (optional), and pipelines integration

(ref:hirist.tech)
Share job
Similar Jobs
View All
5 Hours ago
PHP Developer
Information Technology
  • 3 - 7 Yrs
  • Anywhere in India/Multiple Locations
We are looking for Senior PHP Developer, capable of developing high-end web applications using PHP ,MY SQL & CodeIgniter. Working knowledge in API Integrations for E-commerce portals preferred.   Required Candidate profile 3+ years Exp. with P...
decor
5 Hours ago
Java Developer
Information Technology
  • 2 - 5 Yrs
  • Anywhere in India/Multiple Locations
Role - Software Engineer (Java Developer) Years of Experience - 2 to 5 Work Mode - Remote Bachelors degree in computer science, Engineering, or a related field. 2 to 5 years of experience in Java development. Proficiency in Java, with...
decor
1 Day ago
IN_Associate_Data Scientist_D&A _Advisory_Mumbai
Information Technology
  • Bangalore, Karnataka, India
Line of ServiceAdvisory Industry/SectorNot Applicable SpecialismData, Analytics & AI Management LevelAssociate Job Description & SummaryAt PwC, our people in data and analytics focus on leveraging data to drive insights and make informed business de...
decor
1 Day ago
IN-Senior Associate_Azure Data Engineer_Data and Analytics_Advisory_Hyderabad
Information Technology
  • Bangalore, Karnataka, India
Line of ServiceAdvisory Industry/SectorNot Applicable SpecialismData, Analytics & AI Management LevelSenior Associate Job Description & SummaryAt PwC, our people in data and analytics engineering focus on leveraging advanced technologies and techniq...
decor
1 Day ago
Senior Data Scientist
Information Technology
  • Bangalore, Karnataka, India
Senior Advisor Data ScienceData Science is all about breaking new ground to enable businesses to answer their most urgent questions. Pioneering massively parallel data-intensive analytic processing, our mission is to develop a whole new approach to ...
decor
1 Day ago
IN_Associate_Data Scientist_D&A _Advisory_Mumbai
Information Technology
  • Bangalore, Karnataka, India
Line of ServiceAdvisoryIndustry/SectorNot ApplicableSpecialismData, Analytics & AIManagement LevelAssociateJob Description & SummaryAt PwC, our people in data and analytics focus on leveraging data to drive insights and make informed business decisi...
decor
1 Day ago
IT Technician
Information Technology
Job description: Long Description Mange Active Directory, Users & Computers in windows 2019 2022 server. Configuration Microsoft Outlook 2016, 2019 O365. Configuration and Maintained Cisco Managed Switches, WLCs, Routers. Basic Level Troubleshooti...
decor
1 Day ago
Business Analyst/Project Manager
Information Technology
  • Bangalore, Karnataka, India
We are seeking a dynamic Business Analyst / Project Manager with 3+ years of experience to lead end-to-end software project delivery. The ideal candidate will be skilled in gathering business requirements, managing stakeholders, and coordinating cro...
decor

Talk to us

Feel free to call, email, or hit us up on our social media accounts.
Social media