Data Engineer (Azure/AWS)-Pyspark

Rockwell Automation is a global technology leader focused on helping the world’s manufacturers be more productive, sustainable, and agile. With more than 28,000 employees who make the world better every day, we know we have something special. Behind our customers - amazing companies that help feed the world, provide life-saving medicine on a global scale, and focus on clean water and green mobility - our people are energized problem solvers that take pride in how the work we do changes the world for the better.

We welcome all makers, forward thinkers, and problem solvers who are looking for a place to do their best work. And if that’s you we would love to have you join us!

Job Description

• As a professional services provider, our people are our greatest asset.
• We would rather have a conversation and determine if you are a good fit than never meet at all.
• Consulting is an apprenticeship-based business, and many of the skills you need to master are acquired on the job
• You will report to Chief Architect of the function and will work from our Electronics City, Bengaluru Office in Hybrid work model.
  
  

Your Responsibilities

• Build, deploy, and maintain scalable ETL pipelines on Databricks, ensuring high performance and reliability.
• Process and transform large datasets using Pandas, PySpark, or equivalent libraries.
• Integrate data from multiple sources such as APIs, RDBMS, and file systems into data lakes or warehouses.
• Perform data cleaning, transformation, and enrichment to support analytics and our requirements.
• Write and optimize advanced SQL queries for complex data transformations and extractions.
• Design efficient database schemas and implement data partitioning for improved query performance.
• Develop containerized workflows using Docker.
• Orchestrate applications using Kubernetes in production environments.
• Leverage Azure (preferred) or AWS cloud services for data storage, processing, and orchestration.
• Utilize tools like Azure Data Factory, Databricks, or Glue for end-to-end data pipeline implementation.
• Manage CI/CD pipelines for automated deployment and testing of data engineering solutions.
• Work with tools such as Azure DevOps, GitLab, or Jenkins for continuous integration.
• Optimize workflows for scalability, speed, and cost-effectiveness.
• Monitor pipeline performance and troubleshoot production issues efficiently.
• Write clean, modular, and well-documented code adhering to best coding practices.
• Conduct code reviews to ensure high-quality deliverables.
• Work with data scientists, analysts, and business stakeholders to understand data requirements.
• Participate in agile workflows, sprint planning, and retrospectives.
  
  

The Essentials - You Will Have

• 2 to 4 years of experience in data engineering or a similar role.
• Advanced proficiency in Python, including libraries like Pandas.
• Hands-on experience with Databricks, Pyspark, or other big data frameworks.
• Expertise in SQL, relational databases, and query optimization techniques.
• Strong experience in Azure (preferred) or AWS, with knowledge of cloud-native data tools.
  
  

The Preferred - You Might Also Have

• Experience with Azure DevOps, GitLab, or Jenkins for continuous integration and delivery
• Proficiency in Docker and Kubernetes for container orchestration.
  
  

What We Offer

Our benefits package includes …

• Comprehensive mindfulness programs with a premium membership to Calm
• Volunteer Paid Time off available after 6 months of employment for eligible employees.
• Company volunteer and donation matching program – Your volunteer hours or personal cash donations to an eligible charity can be matched with a charitable donation.
• Employee Assistance Program
• Personalized wellbeing programs through our OnTrack program
• On-demand digital course library for professional development
• At Rockwell Automation we are dedicated to building a diverse, inclusive and authentic workplace, so if you're excited about this role but your experience doesn't align perfectly with every qualification in the job description, we encourage you to apply anyway. You may be just the right person for this or other roles.
  
  

Rockwell Automation’s hybrid policy aligns that employees are expected to work at a Rockwell location at least Mondays, Tuesdays, and Thursdays unless they have a business obligation out of the office.