Senior Data Engineer - 100% Remote

Job Title: Senior Data Engineer

Experience: 8–12 Years

Location: Remote

Key Responsibilities:

Design, build, and maintain scalable data pipelines using Databricks and Apache Spark.

Develop and maintain robust data architecture and ETL/ELT frameworks for structured and unstructured data.

Collaborate with Data Scientists and ML Engineers to enable end-to-end ML model pipelines - from data ingestion and feature engineering to model deployment and monitoring.

Ensure data quality, integrity, and governance across various data sources and systems.

Optimize data processing jobs for performance and cost-effectiveness in cloud environments (preferably Azure or AWS).

Implement best practices for CI/CD, data versioning, and workflow orchestration (e.g., using Airflow, MLflow, Delta Lake, etc.).

Required Skills and Qualifications:

8–12 years of experience in data engineering, with a strong background in big data technologies.

Expertise in Databricks, Apache Spark, Delta Lake, and associated cloud data services.

Solid experience in designing and implementing data lakehouses, data warehouses, and real-time/streaming data pipelines.

Proficient in Python, SQL, and at least one cloud platform (Azure preferred; AWS/GCP also acceptable).

Strong understanding of ML lifecycle, including feature stores, model training, and inference pipelines.

Hands-on experience integrating and scaling AI/ML models in production environments.

Experience with CI/CD tools, infrastructure as code, and version control (e.g., Git, Terraform).