Pyspark Data Engineer

Minimum 5+ years of development and design experience in experience as Data Engineer

Experience on Big Data platforms and distributed computing (e.g. Hadoop, Map/Reduce, Spark, HBase, Hive)

Experience in data pipeline software engineering and best practice in python (linting, unit tests, integration tests, git flow/pull request process, object-oriented development, data validation, algorithms and data structures, technical troubleshooting and debugging, bash scripting )

Experience in Data Quality Assessment (profiling, anomaly detection) and data documentation (schema, dictionaries)

Experience in data architecture, data warehousing and modelling techniques (Relational, ETL, OLTP) and consider performance alternatives

Used SQL, PL/SQL or T-SQL with RDBMSs production environments, no-SQL databases nice to have

Linux OS configuration and use, including shell scripting.

Well versed with Agile, DevOps and CI/CD principles (GitHub, Jenkins etc.), and actively involved in solving, troubleshooting issues in distributed services ecosystem

Experience in Agile methodology.

Ensure quality of technical and application architecture and design of systems across the organization.

Effectively research and benchmark technology against other best in class technologies.

Experience in Banking, Financial and Fintech experience in an enterprise environment preferred Able to influence multiple teams on technical considerations, increasing their productivity and effectiveness,

by sharing deep knowledge and experience.

Self-motivator and self starter, Ability to own and drive things without supervision and works collaboratively with the teams across the organization.

Have excellent soft and interpersonal skills to interact and present the ideas to team. The engineer should've good listening skills and speaks clearly in front of team, stakeholders and management. The engineer should always carry positive attitude towards work and establishes effective team relations and builds a climate of trust within the team. Should be enthusiastic and passionate and creates a motivating environment for the team.