Galileo Data Engineer with AI

We are looking for a highly skilled and dedicated Data Engineer to join our AI solutions development squad. This team will be responsible for building cutting-edge applications leveraging Large Language Models (LLMs), covering the entire development lifecycle from concept to deployment and operations. The Data Engineer will design, build, and maintain robust data infrastructure to support AI applications. The ideal candidate will have expertise in structured and unstructured data, vector databases, real-time data processing, and cloud-based AI solutions using AWS or Azure. Experience with Galileo is essential to ensure seamless data integration and optimization.

Your tasks

• Working closely with AI engineers, data experts, and product teams to use large language models (LLMs) in applications that are reliable, scalable, and ethical
• Creating and managing systems that handle large amounts of data, making sure the data is collected, cleaned, stored, and accessed efficiently for AI projects
• Handling vector databases (like AWS OpenSearch or Azure AI Search) to store and find complex data used in Generative AI
• Building cloud-based data tools using platforms like AWS or Azure to support AI and data needs
• Creating data pipelines that move and process data in real-time or in batches (ETL/ELT)
• Improving how data is stored and processed to make sure it's fast, scalable, and cost-effective

Requirements

• 5 years of experience working in AI/ML, contributing to the development and deployment of intelligent systems
• Skilled in integrating Large Language Models (LLMs) into generative AI applications with a focus on scalability and user experience
• Proficient in designing and implementing scalable ETL/ELT data pipelines to support real-time and batch processing
• Hands-on experience with vector databases such as AWS OpenSearch and Azure AI Search for high-dimensional data storage and retrieval
• Strong background in cloud data engineering using AWS and Azure, including data warehousing and integration with Snowflake
• Experience in working within Agile teams, collaborating closely with AI engineers, data scientists, and product owners