Senior Apache Spark Technical Lead - Scala, Python

Overview

Chennai, Tamil Nadu
Job Summary

DATA ENGINEER Role Overview We are looking for a highly skilled Data Engineer to join our Machine Learning Model Development & Integration stream within Financial Crime Technology. In this role, you will design, build, and optimise data pipelines, feature engineering workflows, and model integration components that enable scalable ML solutions for AML, Fraud, and Transaction Monitoring use cases. You will work closely with Data Scientists, ML Engineers, Solution Architects, and Platform Engineers to deliver production‑ready ML capabilities that are robust, compliant, and cloud‑native. Key Responsibilities 1. Data Engineering & Pipeline Development Design and implement scalable ETL/ELT pipelines for ML model training, inference, and monitoring. Build data ingestion frameworks using tools such as EMR, Kafka, Python, Spark, PySpark, MongoDB. Develop feature engineering pipelines to support model experimentation and productionisation. Ensure data quality, lineage, versioning, and reproducibility across ML workflows. 2. ML Model Integration & Deployment Integrate ML models into real-time and batch applications using custom APIs, or microservices. Build model inference pipelines, scoring engines, and real-time streaming integrations. Automate model deployment, CICD, and configuration using GitLab, AWS CodePipeline, Docker, Terraform. 3. Cloud Architecture & Platform Engineering Design cloud-native architecture patterns aligned with enterprise standards and regulatory expectations. Use services such as S3, Lambda, Step Functions, AppConfig. Optimise cost, performance, and reliability across ML workloads. 4. Cross-functional Collaboration Partner with Data Scientists to understand model input needs, feature dependencies, and execution flows. Collaborate with Platform teams to onboard, scale, and monitor ML workloads. Work with Compliance, Security, and Risk teams to ensure regulatory alignment (e.g., PRA SS2/21, model governance). 5. Operational Excellence Build monitoring, alerting, and observability for data pipelines and model endpoints. Implement automated lineage, auditability, and compliance controls. Enable A/B testing, model comparison workflows, and shadow mode deployments. Skills & Experience Required Technical Skills Strong experience in Python, SQL, PySpark, MongoDB and distributed data processing. Hands-on expertise with AWS cloud services—particularly EMR, Lambda, Step Functions, S3 Experience with Kafka or other event streaming technologies. Solid understanding of data modelling, feature stores, and ML pipeline orchestration. Understanding of ML lifecycle concepts:

Key Responsibilities
1. Lead technical teams in the design and implementation of solutions using apache spark, scala, and python
2. Provide technical expertise and guidance to team members in resolving complex technical issues
3. Collaborate with stakeholders to gather requirements and define project scope
4. Ensure adherence to best practices in coding, testing, and deployment processes
5. Conduct code reviews and performance optimization activities
6. Troubleshoot and debug technical issues to ensure seamless project delivery
7. Stay updated with the latest trends and advancements in apache spark, scala, and python technologies
8. Mentor team members and facilitate knowledge sharing within the team
Skill Requirements
1. Strong proficiency in apache spark, scala, and python
2. Experience in leading technical teams and projects
3. Excellent problem-solving skills and ability to think critically
4. Solid understanding of software development life cycle
5. Good communication and interpersonal skills
6. Ability to work effectively in a collaborative team environment
7. Strong organizational and leadership abilities
8. Experience in performance tuning and optimization techniques
9. Knowledge of big data concepts and tools is a plus
Other Requirements
1.Relevant certifications in apache spark, scala, or python are a plus
#body.unify div.unify-button-container .unify-apply-now: focus, #body.unify div.unify-button-container .unify-apply-#body.unify div.unify-button-container .unify-apply-now: focus, #body.unify div.unify-button-container .unify-apply-