6861 - Senior Cloud Infrastructure Engineer for Data Platform

Senior Cloud Infrastructure Engineer for Data Platform
Experience : 6 to 12 yrs
work location : Remote must be ready to work at Bangalore / Pune / Hyderabad locations.

Job Title : Senior Cloud Infrastructure Engineer for Data Platform

The ideal candidate will play a critical role in designing, implementing, and maintaining cloud infrastructure and CI/CD pipelines to support scalable, secure, and efficient data and analytics solutions. This role requires a strong understanding of cloud-native technologies, DevOps best practices, and hands-on experience with Azure and Databricks.

Key Responsibilities
1. Cloud Infrastructure Design & Management
o Architect, deploy, and manage scalable and secure cloud infrastructure on Microsoft Azure.
o Implement best practices for Azure Resource Management, including resource groups, virtual networks, and storage accounts.
o Optimize cloud costs and ensure high availability and disaster recovery for critical systems.
2. Databricks Platform Management
o Set up, configure, and maintain Databricks workspaces for data engineering, machine learning, and analytics workloads.
o Automate cluster management, job scheduling, and monitoring within Databricks.
o Collaborate with data teams to optimize Databricks performance and ensure seamless integration with Azure services.
3. CI/CD Pipeline Development
o Design and implement CI/CD pipelines for deploying infrastructure, applications, and data workflows using tools like Azure DevOps, GitHub Actions, or similar.
o Automate testing, deployment, and monitoring processes to ensure rapid and reliable delivery of updates.
4. Monitoring & Incident Management
o Implement monitoring and alerting solutions using tools like Dynatrace, Azure Monitor, Log Analytics, and Databricks metrics.
o Troubleshoot and resolve infrastructure and application issues, ensuring minimal downtime.
5. Security & Compliance
o Enforce security best practices, including identity and access management (IAM), encryption, and network security.
o Ensure compliance with organizational and regulatory standards for data protection and cloud operations.
6. Collaboration & Documentation
o Work closely with cross-functional teams, including data engineers, software developers, and business stakeholders, to align infrastructure with business needs.
o Maintain comprehensive documentation for infrastructure, processes, and configurations.

Required Qualifications

• Education: Bachelor’s degree in Computer Science, Engineering, or a related field.
• Experience:

o 6+ years of experience in DevOps or Cloud Engineering roles.
o Proven expertise in Microsoft Azure services, including Azure Data Lake, Azure Databricks, Azure Data Factory (ADF), Azure Functions, Azure Kubernetes Service (AKS), and Azure Active Directory.
o Hands-on experience with Databricks for data engineering and analytics.

Technical Skills:
o Proficiency in Infrastructure as Code (IaC) tools like Terraform, ARM templates, or Bicep.
o Strong scripting skills in Python, or Bash.
o Experience with containerization and orchestration tools like Docker and Kubernetes.
o Familiarity with version control systems (e.g., Git) and CI/CD tools (e.g., Azure DevOps, GitHub Actions).

• Soft Skills:

o Strong problem-solving and analytical skills.
o Excellent communication and collaboration abilities.