Bangalore, Karnataka, India
Information Technology
Full-Time
Srijan Now Material
Overview
Position Overview
We are seeking a highly skilled and motivated Data Engineer to join our dynamic team. The ideal candidate will have extensive experience with AWS Glue, Apache Airflow, Kafka, SQL, Python and DataOps tools and technologies. Knowledge of SAP HANA & Snowflake is a plus. This role is critical for designing, developing, and maintaining our clients data pipeline architecture, ensuring the efficient and reliable flow of data across the organization.
Key Responsibilities
Design, Develop, and Maintain Data Pipelines :
Technical Expertise :
We are seeking a highly skilled and motivated Data Engineer to join our dynamic team. The ideal candidate will have extensive experience with AWS Glue, Apache Airflow, Kafka, SQL, Python and DataOps tools and technologies. Knowledge of SAP HANA & Snowflake is a plus. This role is critical for designing, developing, and maintaining our clients data pipeline architecture, ensuring the efficient and reliable flow of data across the organization.
Key Responsibilities
Design, Develop, and Maintain Data Pipelines :
- Develop robust and scalable data pipelines using AWS Glue, Apache Airflow, and other relevant technologies.
- Integrate various data sources, including SAP HANA, Kafka, and SQL databases, to ensure seamless data flow and processing.
- Optimize data pipelines for performance and reliability.
- Design and implement data transformation processes to clean, enrich, and structure data for analytical purposes.
- Utilize SQL and Python for data extraction, transformation, and loading (ETL) tasks.
- Ensure data quality and integrity through rigorous testing and validation processes.
- Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions that meet their needs.
- Collaborate with cross-functional teams to implement DataOps practices and improve data life cycle management.
- Monitor data pipeline performance and implement improvements to enhance efficiency and reduce latency.
- Troubleshoot and resolve data-related issues, ensuring minimal disruption to data workflows.
- Implement and manage monitoring and alerting systems to proactively identify and address potential issues.
- Maintain comprehensive documentation of data pipelines, transformations, and processes.
- Adhere to best practices in data engineering, including code versioning, testing, and deployment procedures.
- Stay up-to-date with the latest industry trends and technologies in data engineering and DataOps.
Technical Expertise :
- Extensive experience with AWS Glue for data integration and transformation.
- Proficient in Apache Airflow for workflow orchestration.
- Strong knowledge of Kafka for real-time data streaming and processing.
- Advanced SQL skills for querying and managing relational databases.
- Proficiency in Python for scripting and automation tasks.
- Experience with SAP HANA for data storage and management.
- Familiarity with DataOps tools and methodologies for continuous integration and delivery in data engineering.
- Knowledge of Snowflake for cloud-based data warehousing solutions.
- Experience with other AWS data services such as Redshift, S3, and Athena.
- Familiarity with big data technologies such as Hadoop, Spark, and Hive.
- Strong analytical and problem-solving skills.
- Excellent communication and collaboration abilities.
- Detail-oriented with a commitment to data quality and accuracy.
- Ability to work independently and manage multiple projects simultaneously.
Similar Jobs
View All
11 Hours ago
Program Manager
Information Technology
- 15 - 18 Yrs
- Gurgaon / Gurugram
We’re Hiring | Program Manager
Location: Gurugram, India
Domain: Program Management | Digital Systems | Java/.NET | Agile | Fintech/Supply Chain
Are you a strategic thinker with deep experience in program management of digital products? We’re lo...
11 Hours ago
Technical Fullstack Architect - Node.js
Information Technology
- 50,00,000 - 60,00,000 INR - Annual
- 12 - 18 Yrs
- Hyderabad
About the Role:
We are seeking a Fullstack Technical Architect with deep expertise in backend development using Node.js and proficiency in frontend technologies like React or any modern JavaScript framework. You will play a key role in building an...
11 Hours ago
Principal Engineer - Fullstack
Information Technology
- 30,00,000 - 40,00,000 INR - Annual
- 8 - 12 Yrs
- Mumbai
Looking for candidate who is enthusiastic to work in a Startup environment and build things from Scratch individually
Candidate has past experience in scalable consumer facing applications managing latency and traffic FullStack Individual Contribu...
12 Hours ago
Asst. Manager / Dy. Manager – Talent Acquisition
Automotive
- 4,00,000 - 8,00,000 INR - Yearly
- 4 - 8 Yrs
- West Bengal
We are looking for dynamic and experienced professionals for the Talent Acquisition team at our Kharagpur Plant location. The incumbent will be responsible for managing the full-cycle recruitment process for both technical and functional roles across...
1 Day ago
Data Analyst (Kannada Speakers)
AI & Machine Learning Advancement
- 1 - 1 Yrs
- Karnataka, India
For thousands of years, maps have provided humans with the knowledge they need to make decisions. As a Maps Evaluator, you will have the opportunity to provide ground truth for your town, city or country.
At Peroptyx, we are looking for Data Ana...
1 Day ago
Data Analyst (Telugu Speakers)
AI & Machine Learning Advancement
- 1 - 1 Yrs
- Andhra Pradesh, Telangana, India
For thousands of years, maps have provided humans with the knowledge they need to make decisions. As a Maps Evaluator, you will have the opportunity to provide ground truth for your town, city or country.
At Peroptyx, we are looking for Data Ana...
1 Day ago
Solution Architect – Databricks
Information Technology
- 40,00,000 - 50,00,000 INR - Annual
- 9 - 15 Yrs
- Mumbai
Summary role description:
Hiring for a Solution Architect – Databricks for global technology consulting and system integration firm specializing in data engineering, AI and ML.
Company description:
Our client is a US-headquarte...
1 Day ago
Interesting Job Opportunity: Utilli - Full Stack Developer - Node.js/React.js
Information Technology
- Bangalore, Karnataka, India
Job SummaryWe are seeking a highly skilled Full Stack Engineer to lead a team of developers in building and maintaining scalable web applications. This role requires a hands-on leader with deep expertise in JavaScript frameworks and modern developme...
Talk to us
Feel free to call, email, or hit us up on our social media accounts.
Email
info@antaltechjobs.in