Mumbai, Maharashtra, India
Information Technology
Full-Time
Sigmoid
Overview
We are looking for a skilled Data Engineer with 6+ years of experience in big data technologies, particularly Python, PYSpark, SQL, and data lakehouse architectures. The ideal candidate will have a strong background in building scalable data pipelines and experience with modern data storage formats, including Apache Iceberg. You will work closely with cross-functional teams to design and implement efficient data solutions in a cloud-based environment.
Data Pipeline Development
The core responsibilities for the job include the following:
Data Pipeline Development
The core responsibilities for the job include the following:
- Design, build, and optimize scalable data pipelines using Apache Spark.
- Implement and manage large-scale data processing solutions across data lakehouses.
- Work with modern data lakehouse platforms (e. g. Apache Iceberg) to handle large datasets.
- Optimize data storage, partitioning, and versioning to ensure efficient access and querying.
- Write complex SQL queries to extract, manipulate, and transform data.
- Develop performance-optimized queries for analytical and reporting purposes.
- Integrate various structured and unstructured data sources into the lakehouse environment.
- Work with stakeholders to define data needs and ensure data is available for downstream consumption.
- Implement data quality checks and ensure the reliability and accuracy of data.
- Contribute to metadata management and data cataloging efforts.
- Monitor and optimize the performance of Spark jobs, SQL queries, and overall data infrastructure.
- Work with cloud infrastructure teams to optimize costs and scale as needed.
- Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.
- 8+ years of experience in data engineering, with a focus on Java/Python, Spark, and SQL Programming languages.
- Hands-on experience with Apache Iceberg, Snowflake, or similar technologies.
- Strong understanding of data lakehouse architectures and data warehousing principles.
- Proficiency in AWS data services.
- Experience with version control systems like Git and CI/CD pipelines.
- Excellent problem-solving and analytical skills.
- Strong communication and collaboration skills.
- Experience with containerization (Docker, Kubernetes) and orchestration tools like Airflow.
- Certifications in AWS cloud technologies.
Similar Jobs
View All
11 Hours ago
SAP Functional Architect
Information Technology
- 4000000,00,000 - 4500000,00,000 INR - Annual
- 12 - 15 Yrs
- Bangalore, Chennai
We are seeking an experienced SAP Pre-Sales Architect with a strong functional background and deep expertise in at least one SAP functional area. The ideal candidate will have extensive knowledge of cross-module integrations and a proven track record...
13 Hours ago
SDE
AI & Machine Learning Advancement
- 4 - 8 Yrs
- Bangalore
We are seeking a highly skilled and motivated Data Engineer to join our dynamic AI/Analytics team.
More specifically, you will responsible for:
4 + Years of track record of relevant work experience and a computer Science or related technic...
14 Hours ago
Product Technical Architect - InsurTech SaaS
Information Technology
- 50,00,000 - 60,00,000 INR - Annual
- 12 - 18 Yrs
- Hyderabad
Summary role description:
Hiring for a Product Technical Architect (Node.js) for a global InsurTech SaaS platform provider.
Company description:
Our client is a Silicon Valley HQ, venture-funded, tenured provider of the Ins...
1 Day ago
Technical Business Analyst_Director _Business Analysis
Information Technology
- Bangalore, Karnataka, India
Technical Business Analyst - Director - Software EngineeringProfile DescriptionWe’re seeking someone to join our team as (Director) Technical Business Analyst Will work on technical and functional analysis of platforms/applications and documentation...
1 Day ago
Senior Software Engineer - Ruby on Rails
Information Technology
- Bangalore, Karnataka, India
Job DescriptionOur team is responsible for the core User model for Zendesk. The team’s mission is to optimize the in-product, customer and developer experience for representing our customers’ Users at Zendesk.This role will be joining our team based...
1 Day ago
Data Engineer
Information Technology
- Bangalore, Karnataka, India
Project Role : Data EngineerProject Role Description : Design, develop and maintain data solutions for data generation, collection, and processing. Create data pipelines, ensure data quality, and implement ETL (extract, transform and load) processes...
1 Day ago
Mainframe z/OS System Administrator
Information Technology
- Bangalore, Karnataka, India
Who We AreAt Kyndryl, we design, build, manage and modernize the mission-critical technology systems that the world depends on every day. So why work at Kyndryl? We are always moving forward – always pushing ourselves to go further in our efforts to...
1 Day ago
Test Engineer - L3
Information Technology
- Bangalore, Karnataka, India
Job description:Job Description Role PurposeThe purpose of this role is to prepare test cases and perform testing of the product/ platform/ solution to be deployed at a client end and ensure its meet 100% quality assurance parameters. ͏ Do Instrume...
Talk to us
Feel free to call, email, or hit us up on our social media accounts.
Email
info@antaltechjobs.in