Lead/ Sr. Pyspark Data Engineer

Overview

This strategic role that -

• Demonstrate lead level capabilities in conceptual analysis, problem solving, and data analysis. Demonstrate ability to take ownership or initiatives or projects with the ability to lead and facilitate business requirements and harness data to meet the requirements.
• Must be able to collaborate with subject matter experts, business analysts, data analysts and other data stakeholders for data integration and alignment.
• Provides support in the use of enterprise data, including guidance in understanding and exploitation of the underlying data and business models, and identifying optimal data sourcing and mapping.
• Assists in developing and publishing data management standards, policies and procedures, including the management and maintenance of related tools and environments.
• Create and review standard data deliverables, ensuring adherence to defined corporate standards and definitions.
• Demonstrate ability to understand, produce, and extend large and complex data models for operational and dimensional reporting data stores.
• Experience coordinating information requirements of multiple departments, including assessment of usefulness, reliability and cleanliness of data from several sources.
• Ability to effectively present complex technical information to non-technical audiences and to senior decision-makers
  
  

• 4 – 8 years of experience in established data technologies viz. Data ingestion, ETL, and data warehouse, CDC and Event Streaming technologies.
• Strong hands on experience in AWS cloud services for data management and governance.
• Develop pipelines to ingest data from different sources such as Database, delimited files etc, do complex transformation and loading data into the S3.
• develop Glue mapping and create Glue data catalogue.
• Hands on experience in design and implementation of data pipelines, ETL and data warehouse technologies following industry standards.
• Experienced or conversant in emerging tools like parquet or iceberg and unstructured data. Must understand them sufficiently to guide the organization in understanding and adopting them
  
  

• Strong sense of ownership of design solution
• Advocate for the user’s perspective on the Agile team and promote the importance of design
• Embraces a culture of trust and complete transparency
• Open to learning new ideas outside scope or knowledge skillset
• Ability to identify innovative business solutions and implement
• Demonstrate success working in a team-based environment.
• Information systems project management or leadership experience preferred.
• Requires strong decision-making, organizational, planning and problem solving skills.
• Excellent analytical skills required.
• Strong facilitation and negotiation skills required.
• Strong communication skills required; technical and business writing skills are required.
• Strong interpersonal skills, with competence in interfacing with business users required.
• Ability to summarize and present design alternatives.
• Ability to influence and sell soundness of design and deliverables.
  
  

• Strong communication skills with ability to work with business and Technology stakeholders
• Utilizes team collaboration to create innovative solutions efficiently
• Comfortable working with quick turnaround times and deadlines
• B.E/B.Tech/BCA/MCA from a reputed organization
  
  

• 4 – 8 years of experience in established data technologies viz. Data ingestion, ETL, and data warehouse, CDC and Event Streaming technologies.
• Strong hands on experience in AWS cloud services for data management and governance.
• Develop pipelines to ingest data from different sources such as Database, delimited files etc, do complex transformation and loading data into the S3.
• develop Glue mapping and create Glue data catalogue.
• Hands on experience in design and implementation of data pipelines, ETL and data warehouse technologies following industry standards.
• Experienced or conversant in emerging tools like parquet or iceberg and unstructured data. Must understand them sufficiently to guide the organization in understanding and adopting them