Sr. Data Engineer

Overview

Sr. Engineering Developer

Apply on this job link

https://tinyurl.com/5bdk7z78

With Following must have skills:

• Minimum of 12 plus years’ experience as a data engineer
• Hands-on experience with Azure Cloud eco-system.
• Hands-on experience using Python for data manipulation.
• Deep understanding of vectors and vector databases.
• Hands-on experience scaling POC to production.
• Hands-on experience using tools such as Document Intelligence, Snowflake, function app. Azure AI Search
• Experience working with PII/PHI
• Hands-on experience working with unstructured data.

Experience within the Azure ecosystem, including Azure AI Search, Azure Storage Blob, Azure Postgres, with expertise in leveraging these tools for data processing, storage, and analytics tasks.

• Proficiency in data preprocessing and cleaning large datasets efficiently using Azure Tools, Python, and other data manipulation tools.
• Strong background in Data Science/MLOps, with hands-on experience in DevOps, CI/CD, Azure Cloud computing, and model monitoring.
• Expertise in healthcare data standards, such as HIPAA and FHIR, with a deep understanding of sensitive data handling and data masking techniques to protect PII and PHI.
• In-depth knowledge of search algorithms, indexing techniques, and retrieval models for effective information retrieval tasks.
• Experience with chunking techniques and working with vectors and vector databases like Pinecone.
• Ability to design, develop, and maintain scalable data pipelines for processing and transforming large volumes of structured and unstructured data, ensuring performance and scalability.
• Implement best practices for data storage, retrieval, and access control to maintain data integrity, security, and compliance with regulatory requirements.
• Implement efficient data processing workflows to support the training and evaluation of solutions using large language models (LLMs), ensuring that models are reliable, scalable, and performant.
• Proactively identify and resolve data quality issues, pipeline failures, or resource contention to minimize disruption to systems.
• Experience with large language model frameworks, such as Langchain, and the ability to integrate them into data pipelines for natural language processing tasks.
• Familiarity with Snowflake for data management and analytics, with the ability to work within the Snowflake ecosystem to support data processes.
• Knowledge of cloud computing principles and hands-on experience with deploying, scaling, and monitoring AI solutions on platforms like Azure, AWS, and Snowflake.
• Ability to communicate complex technical concepts effectively to both technical and non-technical stakeholders, and collaborate with cross-functional teams.
• Analytical mindset with attention to detail, coupled with the ability to solve complex problems efficiently and effectively.
• Knowledge of cloud cost management principles and best practices to optimize cloud resource usage and minimize costs.
• Experience with ML model deployment, including testing, validation, and integration of machine learning models into production systems.
• Knowledge of model versioning and management tools, such as MLflow, DVC, or Azure Machine Learning, for tracking experiments, versions, and deployments.
• Model monitoring and performance optimization, including tracking model drift and addressing performance issues to ensure models remain accurate and reliable.
• Automation of ML workflows through CI/CD pipelines, enabling smooth model training, testing, validation, and deployment.
• Monitoring and logging of AI/ML systems post-deployment to ensure consistent reliability, scalability, and performance.
• Collaboration with data scientists and engineering teams to facilitate model retraining, fine-tuning, and updating.
• Familiarity with containerization technologies, like Docker and Kubernetes, for deploying and scaling machine learning models in production environments.
• Ability to implement model governance practices to ensure compliance and auditability of AI/ML systems.
• Understanding of model explainability and the use of tools and techniques to provide transparent insights into model behavior.

Job Type: Contractual / Temporary
Contract length: 3 months

Pay: ₹120,000.00 - ₹140,000.00 per month

Schedule:

• US shift

Experience:

• Data science: 10 years (Required)

Work Location: Remote

Application Deadline: 27/04/2025
Expected Start Date: 01/05/2025