IN-Senior Associate _Pyspark Data Engineer_Data and Analytics_ Advisory_Bangalore

Line of Service

Advisory

Industry/Sector

Not Applicable

Specialism

Data, Analytics & AI

Management Level

Senior Associate

Job Description & Summary

A career within…. A career within Data and Analytics services will provide you with the opportunity to help organisations uncover enterprise insights and drive business results using smarter data analytics. We focus on a collection of organisational technology capabilities, including business intelligence, data management, and data assurance that help our clients drive innovation, growth, and change within their organisations in order to keep up with the changing nature of customers and technology. We make impactful decisions by mixing mind and machine to leverage data, understand and navigate risk, and help our clients gain a competitive edge.

• Why PWC
  
  
  

At PwC, you will be part of a vibrant community of solvers that leads with trust and creates distinctive outcomes for our clients and communities. This purpose-led and values-driven work, powered by technology in an environment that drives innovation, will enable you to make a tangible impact in the real world. We reward your contributions, support your wellbeing, and offer inclusive benefits, flexibility programmes and mentorship that will help you thrive in work and life. Together, we grow, learn, care, collaborate, and create a future of infinite experiences for each other. Learn more about us.

At PwC, we believe in providing equal employment opportunities, without any discrimination on the grounds of gender, ethnic background, age, disability, marital status, sexual orientation, pregnancy, gender identity or expression, religion or other beliefs, perceived differences and status protected by law. We strive to create an environment where each one of our people can bring their true selves and contribute to their personal growth and the firm’s growth. To enable this, we have zero tolerance for any discrimination and harassment based on the above considerations. "

Job Description & Summary: A career within Data and Analytics services will provide you with the opportunity to help organisations uncover enterprise insights and drive business results using smarter data analytics. We focus on a collection of organisational technology capabilities, including business intelligence, data management, and data assurance that help our clients drive innovation, growth, and change within their organisations in order to keep up with the changing nature of customers and technology. We make impactful decisions by mixing mind and machine to leverage data, understand and navigate risk, and help our clients gain a competitive edge. 

Responsibilities

• Manage the development of end-to-end data ingestion, transformation, and ETL workflows using Hadoop, PySpark, and related big data technologies.
• Design and implementation of distributed computing frameworks and real-time data processing solutions using Spark and PySpark.
• Ensure the effective management and operation of Hadoop clusters (HDFS, YARN, Hive, HBase) and related big data tools, ensuring high availability, security, and performance.
• Develop and maintain data pipelines to process large volumes of structured and unstructured data, ensuring data consistency, integrity, and efficiency.
• Provide technical guidance, mentorship, and career development for junior engineers and team members in the Hadoop and PySpark ecosystem.
• Design and implementation of data models and data storage strategies, including integration with cloud platforms such as AWS, Google Cloud, or Azure.
• Optimize performance of data pipelines and queries, troubleshooting any issues related to data processing and performance bottlenecks.
• Ensure adherence to best practices in code quality, data governance, and security standards.
• Communicate complex technical challenges and solutions effectively to both technical and non-technical stakeholders.
• Drive innovation by staying up-to-date with the latest big data technologies and incorporating relevant advancements into the platform.
• 3 years of experience in Big Data Engineering, with a strong background in Hadoop and PySpark (at least 2-3 years in a leadership or managerial role).
• Proven experience in designing and implementing data pipelines using PySpark for large-scale data processing.
• Strong understanding and hands-on experience with the Hadoop ecosystem, including HDFS, MapReduce, Hive, HBase, and YARN.
• Proficiency in Python and PySpark, including working with distributed computing frameworks and data processing workflows.
• Expertise in working with ETL processes, data lakes, data warehousing, and cloud platforms (AWS, GCP, or Azure).
  
  
  

Mandatory Skill Sets

Pyspark, Python, Hadoop, Sql

Preferred Skill Sets

Pyspark, Python, Hadoop, Sql

Years Of Experience Required

3-6

Education Qualification

B.Tech / M.Tech / MBA / MCA

Education (if blank, degree and/or field of study not specified)

Degrees/Field of Study required: Master of Business Administration, Master of Engineering, Bachelor of Engineering

Degrees/Field Of Study Preferred

Certifications (if blank, certifications not specified)

Required Skills

Python (Programming Language)

Optional Skills

Accepting Feedback, Accepting Feedback, Active Listening, Agile Scalability, Amazon Web Services (AWS), Analytical Thinking, Apache Hadoop, Azure Data Factory, Communication, Creativity, Data Anonymization, Database Administration, Database Management System (DBMS), Database Optimization, Database Security Best Practices, Data Engineering, Data Engineering Platforms, Data Infrastructure, Data Integration, Data Lake, Data Modeling, Data Pipeline, Data Quality, Data Transformation, Data Validation {+ 18 more}

Desired Languages (If blank, desired languages not specified)

Travel Requirements

Not Specified

Available for Work Visa Sponsorship?

No

Government Clearance Required?

No

Job Posting End Date