Bangalore, Karnataka, India
Information Technology
Full-Time
Country Delight
Overview
Position Overview
We are seeking a talented and experienced Data Engineer with expertise in Apache Spark, Python / Java and distributed systems. The ideal candidate will be skilled in creating and managing data pipelines using AWS.
Key Responsibilities
We are seeking a talented and experienced Data Engineer with expertise in Apache Spark, Python / Java and distributed systems. The ideal candidate will be skilled in creating and managing data pipelines using AWS.
Key Responsibilities
- Design, develop, and implement data pipelines for ingesting, transforming, and loading data at scale.
- Utilize Apache Spark for data processing and analysis.
- Utilize AWS services (S3, Redshift, EMR, Glue) to build and manage efficient data pipelines.
- Optimize data pipelines for performance and scalability, considering factors like partitioning, bucketing, and caching.
- Write efficient and maintainable Python code.
- Implement and manage distributed systems for data processing.
- Collaborate with cross-functional teams to understand data requirements and deliver optimal solutions.
- Ensure data quality and integrity throughout the data :
- Proven experience with Apache Spark and Python / Java.
- Strong knowledge of distributed systems.
- Proficiency in creating data pipelines with AWS.
- Excellent problem-solving and analytical skills.
- Ability to work independently and as part of a team.
- Bachelor's degree in Computer Science, Information Technology, or a related field (or equivalent experience).
- Proven experience in designing and developing data pipelines using Apache Spark and Python.
- Experience with distributed systems concepts (Hadoop, YARN) is a plus.
- In-depth knowledge of AWS cloud services for data engineering (S3, Redshift, EMR, Glue).
- Familiarity with data warehousing concepts (data modeling, ETL) is preferred.
- Strong programming skills in Python (Pandas, NumPy, Scikit-learn are a plus).
- Experience with data pipeline orchestration tools (Airflow, Luigi) is a plus.
- Excellent problem-solving and analytical skills.
- Strong communication and collaboration Qualifications :
- Experience with additional AWS services (e.g., AWS Glue, AWS Lambda, Amazon Redshift).
- Familiarity with data warehousing and ETL processes.
- Knowledge of data governance and best practices.
- Have a good understanding of the oops concept.
- Hands-on experience with SQL database design
- Experience with Python, SQL, and data visualization/exploration tools
Similar Jobs
View All
1 Day ago
Java Full Stack Developer
Information Technology
- Bangalore, Karnataka, India
Primary Skill - Java, Springboot, Angular, React.js Responsibilities: Lead and mentor a team of Java & Springboot Developers in the design, development, and maintenance of applications. Work with business stakeholders and technical teams to gather...
1 Day ago
Python Developer, Analyst, Firm Risk Management
Information Technology
- Bangalore, Karnataka, India
Morgan StanleyPython Developer - AnalystProfile DescriptionWe’re seeking someone to join our team as (Analyst - Python Developer) in (Credit Risk) division.Firm Risk ManagementIn the Firm Risk Management division, we advise businesses across the Fir...
1 Day ago
Junior SQL Developer/Software Integration Specialist
Information Technology
- Bangalore, Karnataka, India
Position : Junior SQL Developer / Software Integration SpecialistLocation : MumbaiExperience : Fresher or 1 Year of Relevant ExperienceJob Type : Full-timeDepartment : IT / DevelopmentKey skills : Sybase Power Builder/PB 6.5 /POWER BUILDERJob Overvi...
1 Day ago
Agile Coach
Information Technology
- Bangalore, Karnataka, India
Job PurposeResponsible to ensure a successful Agile adoption within the organization and thereby within the scrum teams and within the different organization disciplines. Lead the way to better deliver done-high-quality products, through continuous ...
1 Day ago
Python Developer - Data Platform
Information Technology
- Bangalore, Karnataka, India
Key Responsibilities Develop, maintain, and optimize robust data ingestion and ETL/ELT pipelines using Python on the Databricks platform. Work collaboratively with the architecture team to align implementation with long-term data strategies. Leve...
1 Day ago
AWS Admin And AWS Architect
Information Technology
- Bangalore, Karnataka, India
Primary skills:Technology->IOT Platform->AWS IOT A day in the life of an Infoscion As part of the Infosys consulting team, your primary role would be to get to the heart of customer issues, diagnose problem areas, design innovative solutions and fa...
1 Day ago
Machine Learning Engineer - Data Modeling
Information Technology
- Bangalore, Karnataka, India
Role : Machine Learning and AI EngineerExp : 4+ YearsType : HybridLocation : Bangalore We are looking for a skilled Mid-Level AI Engineer with extensive experience in building full-stack AI solutions. The ideal candidate will have a strong foundat...
1 Day ago
Interesting Job Opportunity: Target Integration - Principal Software Engineer - Node.js/Python
Information Technology
- Chennai, Tamil Nadu, India
Key Responsibilities Drive the end-to-end architecture, design, and implementation of robust, scalable, and secure software systems. Define technical standards, frameworks, and best practices, ensuring delivery excellence. Collaborate with stakeh...
Talk to us
Feel free to call, email, or hit us up on our social media accounts.
Email
info@antaltechjobs.in