Overview
Data Engineer will be responsible for designing, implementing, and maintaining robust data pipelines on the Databricks platform. The ideal candidate will have a deep understanding of big data technologies, ETL processes, and data integration. This role requires collaboration with data scientists, analysts, and other stakeholders to ensure data availability and integrity for analytics and reporting.
Key Responsibilities
Key Responsibilities
- Design and Develop Data Pipelines: Create and maintain scalable and efficient data pipelines using Databricks and Apache Spark.
- Data Integration: Integrate data from various sources, including databases, APIs, and external datasets, ensuring data quality and consistency.
- ETL Processes: Develop and optimize ETL (Extract, Transform, Load) processes to support data analytics and business intelligence.
- Performance Optimization: Optimize data processing performance, including tuning Spark jobs and managing resource allocation.
- Collaboration: Work closely with data scientists, analysts, and other stakeholders to understand data requirements and provide solutions.
- Data Quality: Implement data validation and cleansing procedures to ensure data accuracy and reliability.
- Documentation: Document data pipelines, processes, and architecture for maintenance and knowledge sharing.
- Monitoring and Troubleshooting: Monitor data pipelines for performance and reliability, and troubleshoot issues as they arise.
Similar Jobs
View All
1 Day ago
Head of AI Cybersecurity
Information Technology
- 8 - 13 Yrs
- Chennai, Mumbai (All Areas)
Location: Mumbai & Chennai- Hybrid (1 week WFO in a month) | Experience: 8+ years
Type: Full-time
Seeking a visionary and hands-on Head of AI Cybersecurity to lead the end-to-end security of AI/ML systems. This role will drive strategy, implement...
1 Day ago
Intern-AI Engineer
Finance & Banking
- Pune, Maharashtra, India
Scopely is looking for an Intern - AI Engineer to join our Gen AI team in Bangalore!At Scopely, we care deeply about what we do and want to inspire play, every day - whether in our work environments alongside our talented colleagues, or through our ...
1 Day ago
Software Developer - PHP/Laravel
Finance & Banking
- Pune, Maharashtra, India
Software Developer (Laravel/PHP)Location : MumbaiWorking Mode : On-siteExperience : 1-5 YearsAbout The RoleKey Responsibilities : Collaborate with cross-functional teams to understand project requirements, define scope, and determine best practices...
1 Day ago
GenAI Data Scientist - Manager
Finance & Banking
- Pune, Maharashtra, India
At PwC, our people in data and analytics focus on leveraging data to drive insights and make informed business decisions. They utilise advanced analytics techniques to help clients optimise their operations and achieve their strategic goals. In data...
1 Day ago
Java Developer - C11
Finance & Banking
- Pune, Maharashtra, India
The Applications Development Intermediate Programmer Analyst is an intermediate level position responsible for participation in the establishment and implementation of new or revised application systems and programs in coordination with the Technolo...
1 Day ago
Trademo - Lead Data/AI Engineer
Finance & Banking
- Pune, Maharashtra, India
About TrademoAt Trademo, we are transforming global trade and supply chains by leveraging cutting-edge AI technology to provide businesses with unparalleled visibility, compliance, and intelligence solutions.Our AI-driven platform simplifies the com...
1 Day ago
Senior Web Developer- C11 - CHENNAI
Finance & Banking
- Pune, Maharashtra, India
The Digital S/W Engineer Intmd Analyst is a developing professional role. Deals with most problems independently and has some latitude to solve complex problems. Integrates in-depth specialty area knowledge with a solid understanding of industry sta...
1 Day ago
GenAI Data Scientist - Manager
Finance & Banking
- Pune, Maharashtra, India
At PwC, our people in data and analytics focus on leveraging data to drive insights and make informed business decisions. They utilise advanced analytics techniques to help clients optimise their operations and achieve their strategic goals. In data...
Talk to us
Feel free to call, email, or hit us up on our social media accounts.
Email
info@antaltechjobs.in