Sr Data Engineer

Key Responsibilities

• Design and Develop Data Pipelines: Create and maintain scalable and efficient data pipelines using Databricks and Apache Spark.
• Data Integration: Integrate data from various sources, including databases, APIs, and external datasets, ensuring data quality and consistency.
• ETL Processes: Develop and optimize ETL (Extract, Transform, Load) processes to support data analytics and business intelligence.
• Performance Optimization: Optimize data processing performance, including tuning Spark jobs and managing resource allocation.
• Collaboration: Work closely with data scientists, analysts, and other stakeholders to understand data requirements and provide solutions.
• Data Quality: Implement data validation and cleansing procedures to ensure data accuracy and reliability.
• Documentation: Document data pipelines, processes, and architecture for maintenance and knowledge sharing.
• Monitoring and Troubleshooting: Monitor data pipelines for performance and reliability, and troubleshoot issues as they arise.