Data Engineer (Lakehouse Architecture, PySpark)

Overview

Job Title: Data Engineer (Lakehouse Architecture, PySpark)
Number of Positions: 2
Experience: 8-10+ years
Duration: 6 months to 1 year
Location: Offshore Support (Client Location: US)
Mode: Remote/Offshore

Job Description:

We are looking for experienced Data Engineers with strong expertise in Lakehouse Architecture implementations (Medallion), complex metadata-driven data ingestion frameworks, and CI/CD in Microsoft Fabric. The ideal candidate should have deep knowledge of Python and PySpark and be able to design, implement, and optimize scalable data solutions.

Key Responsibilities:

• Design and implement Lakehouse Architecture following the Medallion framework.
• Develop and maintain metadata-driven data ingestion pipelines to support large-scale data processing.
• Work with Microsoft Fabric to build CI/CD pipelines for seamless data operations.
• Optimize ETL/ELT processes using PySpark and Python to handle large datasets efficiently.
• Collaborate with cross-functional teams to ensure data integrity, security, and scalability.
• Troubleshoot and resolve issues in data pipelines to ensure high availability and performance.
• Work closely with onshore teams to align project goals and deliverables.

Required Skills & Experience:

• 8-10+ years of experience in data engineering and big data technologies.
• Expertise in Lakehouse Architecture implementation with Medallion principles.
• Strong knowledge of Python and PySpark for data processing.
• Experience in building and managing CI/CD pipelines in Microsoft Fabric.
• Hands-on experience with metadata-driven data ingestion frameworks.
• Strong knowledge of data modeling, ETL processes, and performance optimization.
• Ability to work independently and collaborate with onshore teams in the US.
• Excellent problem-solving and analytical skills.

Preferred Skills:

• Experience with Azure Data Lake, Databricks, Delta Lake, or Snowflake.
• Familiarity with cloud-based data architectures and data governance.

Interested candidates can share their profiles at hr@cssinfotech.in

Job Types: Full-time, Permanent

Work Location: In person