Software Engineer II - Data Engineer

Job responsibilities

• Organizes, updates, and maintains gathered data that will aid in making the data actionable. Assist in the design, development, and implementation of scalable data pipelines and ETL batches using Python/PySpark on AWS.
• Demonstrates basic knowledge of the data system components to determine controls needed to ensure secure data access. Use infrastructure as code to build applications to orchestrate and monitor data pipelines, create and manage on-demand compute resources on cloud programmatically, create frameworks to ingest and distribute data at scale.
• Be responsible for making custom configuration changes in one to two tools to generate a product at the business or customer request
• Updates logical or physical data models based on new use cases with minimal supervision. Optimize and maintain data infrastructure on cloud platform, ensuring scalability, reliability, and performance.
• Adds to team culture of diversity, equity, inclusion, and respect

Required qualifications, capabilities, and skills

• Formal training or certification on data engineering concepts and 2+ years applied experience
• Experience in software development and data engineering, with demonstrable hands-on experience in Python and PySpark.
• Proven experience with cloud platforms such as AWS, Azure, or Google Cloud.
• Good understanding of data modeling, data architecture, ETL processes, and data warehousing concepts.
• Experience with big data technologies and services like AWS EMRs, Redshift, Lambda, S3.
• Excellent communication skills to work effectively with stakeholders, partner teams, and to translate technical concepts into business terms. Proven experience with efficient Cloud DevOps practices and CI/CD tools like Jenkins/Gitlab, for data engineering platforms.
• Good knowledge of SQL and NoSQL databases, including performance tuning and optimization.
• Strong analytical skills to troubleshoot issues and optimize data processes, working independently and collaboratively.
• Experience in working within a team of engineers, with a proven track record of successful project support.

• Knowledge of machine learning concepts, language models, and cloud-native MLOps pipelines and frameworks is a big plus.
• AWS Certifications in data engineering and machine learning is a plus.
• Familiarity with data visualization tools and data integrations.