Lead Data Engineer - Python/PySpark

Job Description

The Lead Data Engineer will provide technical expertise in analysis, design, development, rollout and maintenance of data integration initiatives. This role will contribute to implementation methodologies and best practices, as well as work on project teams to analyze, design, develop and deploy business intelligence / data integration solutions to support a variety of customer needs. This position oversees a team of Data Integration Consultants at various levels, ensuring their success on projects, goals, trainings and initiatives though mentoring and coaching.

Provides technical expertise in needs identification, data modelling, data movement and transformation mapping (source to target), automation and testing strategies, translating business needs into technical solutions with adherence to established data guidelines and approaches from a business unit or project perspective whilst leveraging best fit technologies

(e.g., cloud, Hadoop, NoSQL, etc.) and approaches to address business and

environmental challenges

Works with stakeholders to identify and define self-service analytic solutions, dashboards, actionable enterprise business intelligence reports and business intelligence best practices.

Responsible for repeatable, lean and maintainable enterprise BI design across organizations. Effectively partners with client team.

• Leadership not only in the conventional sense, but also within a team we expect people to be leaders.
• Candidate should elicit leadership qualities such as Innovation, Critical thinking, optimism/positivity, Communication, Time Management, Collaboration, Problem-solving, Acting Independently, Knowledge sharing and :
• Design, develop, test, and deploy data integration processes (batch or real-time)
  
  

using tools such as Microsoft SSIS, Azure Data Factory, Databricks,

Matillion, Airflow, Sqoop, etc.

• Create functional & technical documentation e.g. ETL architecture
  
  

documentation, unit testing plans and results, data integration

specifications, data testing plans, etc.

• Provide a consultative approach with business users, asking questions to
  
  

understand the business need and deriving the data flow, conceptual,

logical, and physical data models based on those needs. Perform data

analysis to validate data models and to confirm ability to meet business

needs.

• May serve as project or DI lead, overseeing multiple consultants from Stays current with emerging and changing technologies to best recommend and
  
  

implement beneficial technologies and approaches for Data Integration

• Ensures proper execution/creation of methodology, training, templates, resource
  
  

plans and engagement review processes

• Coach team members to ensure understanding on projects and tasks, providing
  
  

effective feedback (critical and positive) and promoting growth

opportunities when appropriate.

• Coordinate and consult with the project manager, client business staff, client
  
  

technical staff

and project developers in data architecture best practices

and anything else that is data related at the project or business unit

levels

• Architect,design, develop and set direction for enterprise self-service analytic
  
  

solutions, business intelligence reports, visualisations and best practice

standards. Toolsets include but

not limited to : SQL Server Analysis and

Reporting Services, Microsoft Power BI, Tableau and

Qlik.

• Work with report team to identify, design and implement a reporting user
  
  

experience that is consistent and intuitive across environments, across

report methods, defines security and meets usability and scalability have :

• Writing code in programming language & working experience in Python, Pyspark, Databricks, Scala or Similar Data Pipeline Development & Management
• Design, develop, and maintain ETL (Extract, Transform, Load) pipelines using AWS services like AWS Glue, AWS Data Pipeline, Lambda, and Step Functions.
• Implement incremental data processing using tools like Apache Spark (EMR), Kinesis, and Kafka.
• Work with AWS data storage solutions such as Amazon S3, Redshift, RDS, DynamoDB, and Aurora.
• Optimize data partitioning, compression, and indexing for efficient querying and cost optimization.
• Implement data lake architecture using AWS Lake Formation & Glue Catalog.
• Implement CI/CD pipelines for data workflows using Code Pipeline, Code Build, and GitHub to have :
• Enterprise Data Modelling and Semantic
  
  

Modelling & working experience in ERwin, ER/Studio, PowerDesigner or Similar

• Logical/Physical model on Big Data sets or modern data warehouse & working experience in ERwin, ER/Studio, PowerDesigner or Similar
• Agile Process (Scrum cadences, Roles, deliverables) & basic understanding in either Azure
  
  

DevOps, JIRA or Similar.

(ref:hirist.tech)