Azure Data Engineer

Company:IT Services Organization

Key Skills: Spark, Azure Databricks, Azure, Python, Pyspark

Roles and Responsibilities:

• Develop and maintain scalable data processing systems using Apache Spark and Azure Databricks.
• Implement data integration from various sources including RDBMS, ERP systems, and files.
• Design and optimize SQL queries, stored procedures, and relational schemas.
• Build stream-processing systems using technologies such as Apache Storm or Spark-Streaming.
• Utilize messaging systems like Kafka or RabbitMQ for data ingestion.
• Ensure performance tuning of Spark jobs for optimal efficiency.
• Collaborate with cross-functional teams to deliver high-quality data solutions.
• Lead and mentor a team of data engineers, fostering a culture of continuous improvement and Agile practices.

Skills Required:

• Proficient in Apache Spark and Azure Databricks
• Strong experience with Azure ecosystem and Python
• Working knowledge of Pyspark (Nice-to-Have)
• Experience in data integration from varied sources
• Expertise in SQL optimization and stream-processing systems
• Familiarity with Kafka or RabbitMQ
• Ability to lead and mentor engineering teams
• Strong understanding of distributed computing principles

Education: Bachelor's degree in Computer Science, Information Technology, or a related field.