Data Engineer

Key Responsibilities

• Design and develop high-volume, data engineering solutions for mission-critical systems with quality.
• Making enhancements to various applications that meets business and auditing requirements.
• Research and evaluate alternative solutions and make recommendations on improving the product to meet business and information risk requirements.
• Evaluate service level issues and suggested enhancements to diagnose and address underlying system problems and inefficiencies.
• Participate in full development lifecycle activities for the product (coding, testing, release activities).
• Support Release activities on weekends as required.
• Support any application issues reported during weekends.
Coordinating day-To-day activities for multiple projects with onshore and offshore team members. Ensuring the availability of platform in lower environments

Required Qualifications

3-4 years of overall IT experience, which includes hands on experience in Big Data technologies.

• Mandatory - Hands on experience in Python and PySpark.

• Build pySpark applications using Spark Dataframes in Python using Jupyter notebook and PyCharm(IDE).

• Worked on optimizing spark jobs that processes huge volumes of data.

• Hands on experience in version control tools like Git.

• Worked on Amazon’s Analytics services like Amazon EMR, Amazon Athena, AWS Glue.

• Worked on Amazon’s Compute services like Amazon Lambda, Amazon EC2 and Amazon’s Storage service like S3 and few other services like SNS.

• Experience/knowledge of bash/shell scripting will be a plus.

• Has built ETL processes to take data, copy it, structurally transform it etc. involving a wide variety of formats like CSV, TSV, XML and JSON.

• Experience in working with fixed width, delimited , multi record file formats etc.

• Good to have knowledge of datawarehousing concepts – dimensions, facts, schemas- snowflake, star etc.

• Have worked with columnar storage formats- Parquet,Avro,ORC etc. Well versed with compression techniques – Snappy, Gzip.

• Good to have knowledge of AWS databases (atleast one) Aurora, RDS, Redshift, ElastiCache, DynamoDB.

• Hands on experience in tools like Jenkins to build, test and deploy the applications

• Awareness of Devops concepts and be able to work in an automated release pipeline environment.

• Excellent debugging skills.
  

Preferred Qualifications

• Experience working with US Clients and Business partners.
• Knowledge on Front end frameworks.
• Exposure to BFSI domain is a good to have.
• Hands on experience on any API Gateway and management platform.