Free cookie consent management tool by TermsFeed Associate Architect-Data Engineer & AI | Antal Tech Jobs
Back to Jobs
16 Weeks ago

Associate Architect-Data Engineer & AI

decor
Indore, Madhya Pradesh, India
Finance & Banking
Full-Time
Fission Labs

Overview

Hyderabad, India
10+ Years
Full Time

Roles and Responsibilities

Associate Architect- Data Engineer is responsible for overseeing the design, development, and management of data infrastructure and pipelines within an organization. This role involves a mix of technical leadership, project management, and collaboration with other teams to ensure the efficient collection, storage, processing, and analysis of large datasets. The Lead Data Engineer typically manages a team of data engineers, architects, and analysts, ensuring that data workflows are scalable, reliable, and meet the business’s requirements.

Responsibilities:

  • Lead the design, development, and maintenance of data pipelines and ETL processes

architect and implement scalable data solutions using Databricks and AWS.

  • Optimize data storage and retrieval systems using Rockset, Clickhouse, and CrateDB.
  • Develop and maintain data APIs using FastAPI.
  • Orchestrate and automate data workflows using Airflow.
  • Collaborate with data scientists and analysts to support their data needs.
  • Ensure data quality, security, and compliance across all data systems.
  • Mentor junior data engineers and promote best practices in data engineering.
  • Evaluate and implement new data technologies to improve the data infrastructure.
  • Participate in cross-functional projects and provide technical leadership.
  • Manage and optimize data storage solutions using AWS S3, implementing best practices for data lakes and data warehouses.
  • Implement and manage Databricks Unity Catalog for centralized data governance and access control across the organization.

Qualifications Required

  • Bachelor's or Master's degree in Computer Science, Engineering, or related field
  • 10+ years of experience in data engineering, with at least 6 years in a lead role
  • Strong proficiency in Python, PySpark, and SQL
  • Extensive experience with Databricks and AWS cloud services
  • Hands-on experience with Airflow for workflow orchestration
  • Familiarity with FastAPI for building high-performance APIs
  • Experience with columnar databases like Rockset, Clickhouse, and CrateDB
  • Solid understanding of data modeling, data warehousing, and ETL processes
  • Experience with version control systems (e.g., Git) and CI/CD pipelines
  • Excellent problem-solving skills and ability to work in a fast-paced environment
  • Strong communication skills and ability to work effectively in cross-functional teams
  • Knowledge of data governance, security, and compliance best practices
  • Proficiency in designing and implementing data lake architectures using AWS S3
  • Experience with Databricks Unity Catalog or similar data governance and metadata

management tools

Skills and Experience Required

Preferred Qualifications:

  • Experience with real-time data processing and streaming technologies
  • Familiarity with machine learning workflows and MLOps
  • Certifications in Databricks, AWS
  • Experience implementing data mesh or data fabric architectures
  • Knowledge of data lineage and metadata management best practices

Tech Stack

Databricks, Python, PySpark, SQL, Airflow, FastAPI, AWS (S3, IAM, ECR, Lambda), Rockset, Clickhouse, CrateDB

Why you'll love working with us:

  • Opportunity to work on business challenges from top global clientele with high impact.
  • Vast opportunities for self-development, including online university access and sponsored certifications.
  • Sponsored Tech Talks, industry events & seminars to foster innovation and learning.
  • Generous benefits package including health insurance, retirement benefits, flexible work hours, and more.
  • Supportive work environment with forums to explore passions beyond work.
  • This role presents an exciting opportunity for a motivated individual to contribute to the development of cutting-edge solutions while advancing their career in a dynamic and collaborative environment.

Share job
Similar Jobs
View All
1 Day ago
Lead Python Developer
Information Technology
  • 8 - 12 Yrs
  • Anywhere in India/Multiple Locations
My Client is revolutionizing the hospitality industry by transforming how businesses manage their finances through cutting-edge Accounts Payable automation. Our mission? To eliminate manual work from business expenses and empower operators with real-...
decor
1 Day ago
Business Analyst I, DeviceOS
Information Technology
  • Chennai, Tamil Nadu, India
This job is with Amazon, an inclusive employer and a member of myGwork – the largest global platform for the LGBTQ+ business community. Please do not contact the recruiter directly. DescriptionAmazon is looking for a data-savvy professional to crea...
decor
1 Day ago
IT Support Mgr II
Information Technology
  • Chennai, Tamil Nadu, India
This job is with Amazon, an inclusive employer and a member of myGwork – the largest global platform for the LGBTQ+ business community. Please do not contact the recruiter directly. DescriptionAre you passionate about helping people solve IT proble...
decor
1 Day ago
EverestDX - Senior Python Developer
Information Technology
  • Chennai, Tamil Nadu, India
Job Title : Sr. Python DeveloperJob Location : Hyderabad Type : Full : 8+ Period : Immediate to 15 days joiners are highly : Design, develop, and maintain scalable Python applications and AI-powered solutions. Collaborate with cross-functional tea...
decor
1 Day ago
.NET Application Developer
Information Technology
  • Chennai, Tamil Nadu, India
🌟 We're Hiring: .NET Application Developer! 🌟We are looking for a skilled .NET Application Developer with at least 5 years of experience to join our dynamic team. The ideal candidate will have a strong background in developing robust applications us...
decor
1 Day ago
Full Stack Developer - Node.js/React.js
Information Technology
  • Chennai, Tamil Nadu, India
Designation : Full Stack Developer( React.js+ Node.js)Job Location : PrabhadeviJob Day & Timing : Mon - Fri , 9 AM -6 PMJob Description Perform hands-on computer program improvement, counting composing code and unit tests, doing verification of co...
decor
1 Day ago
Sr. Test Engineer - QA Job
Information Technology
  • Chennai, Tamil Nadu, India
We use cookies to offer you the best possible website experience. Your cookie preferences will be stored in your browser’s local storage. This includes cookies necessary for the website's operation. Additionally, you can freely decide and change any...
decor
1 Day ago
QA Engineer
Information Technology
  • Chennai, Tamil Nadu, India
Flowserve is a world-leading manufacturer and aftermarket service provider of comprehensive flow control systems. Join a company whose people are committed to building a more sustainable future to make the world better for everyone. With 16,000+ emp...
decor

Talk to us

Feel free to call, email, or hit us up on our social media accounts.
Social media