Data Engineer - Pyspark, Databricks and SQL

Overview

Optum is a global organization that delivers care, aided by technology to help millions of people live healthier lives. The work you do with our team will directly improve health outcomes by connecting people with the care, pharmacy benefits, data and resources they need to feel their best. Here, you will find a culture guided by inclusion, talented peers, comprehensive benefits and career development opportunities. Come make an impact on the communities we serve as you help us advance health optimization on a global scale. Join us to start Caring. Connecting. Growing together.

Primary Responsibilities

• Design, build, and maintain scalable data infrastructure and pipelines
• Oversee data collection, transformation, and storage strategies to ensure data integrity and efficiency
• Collaborate with cross-functional teams including data science, analytics, product management, and IT
• Ensure optimal data pipeline architecture and troubleshoot issues as they arise
• Establish processes for data quality, validation, monitoring, and reporting
• Stay updated with the latest technologies and implement best practices in data engineering
• Comply with the terms and conditions of the employment contract, company policies and procedures, and any and all directives (such as, but not limited to, transfer and/or re-assignment to different work locations, change in teams and/or work shifts, policies in regard to flexibility of work benefits and/or work environment, alternative work arrangements, and other decisions that may arise due to the changing business environment). The Company may adopt, vary or rescind these policies and directives in its absolute discretion and without any limitation (implied or otherwise) on its ability to do so
  
  

• Bachelors or Masters degree in Computer Science, Engineering, or related field
• 3+ years of experience in data engineering
• 3+ years experience in SQL, Python, PySpark, and/or Scala programming languages
• 3+ years hands-on experience with Databricks or similar big data processing platforms
• Experience in implementing data governance frameworks and security best practices
• Solid understanding of data warehousing, ETL (Extract, Transform, Load) processes, and data modeling
• Familiarity with big data technologies (e.g. Spark, Kafka)
• Proven excellent problem-solving skills and attention to detail