Senior Data Engineer

Overview

We are looking for a skilled and driven Data Engineer to join our dynamic technology team. As a Data Engineer, you will be responsible for building and maintaining robust, scalable, and high-performance data pipelines and platforms. This role involves working with big data technologies and cloud infrastructure, ensuring efficient data flow and integrity across our systems.

Skills Required: Python, PySpark, AWS, SQL, Databricks, Data Structures & Algorithms (DSA)

Key Responsibilities:

• Design, build, and maintain efficient and scalable data pipelines using PySpark.
• Work with AWS cloud services to support and enhance the organization’s data infrastructure
• Design and develop high-performance queries and processes using SQL.
• Leverage Databricks for data engineering workflows and advanced analytics solutions.
• Implement best practices in data management, governance, quality, and security.
• Collaborate with cross-functional teams including Data Scientists, Analysts, and Business Stakeholders to understand data requirements.
• Optimize data processing systems for performance, scalability, and reliability.
• Troubleshoot and resolve data-related issues in a timely manner.

Required Skills:

• Strong hands-on experience with PySpark and distributed data processing
• Proficiency in AWS (S3, Glue, Lambda, Redshift, etc.).
• Excellent command over SQL and data querying techniques.
• Experience working with Databricks for data pipeline development and deployment.
• Strong understanding of Data Structures and Algorithms (DSA).
• Familiarity with version control systems (e.g., Git), and CI/CD pipelines.
• Ability to work in a fast-paced, agile environment.

Qualification:

• Minimum 4 years of relevant experience is mandatory
• B.E/B.Tech or MCA graduates are preferred.
• Strong academic background in Computer Science, Information Technology, or a related field.

Preferred Qualifications:

• Certification in AWS or Big Data technologies.
• Knowledge of data warehousing concepts and ETL frameworks.
• Experience in Agile/Scrum methodologies.