Bangalore, Karnataka, India
Information Technology
Full-Time
Wolters Kluwer
Overview
Job Description
The Cloud Data Engineer will be responsible for designing, implementing end to end solutions in Azure Enterprise Data Lake and optimizing data pipelines, ensuring the availability, performance, scalability and security of large-scale data processing systems. This role requires deep understanding of big data technologies, data architecture, infrastructure, CI/CD and data engineering best practices. Experience with Unity Catalog is a bonus. The Cloud Data Engineer will work closely with architects, leads and other stakeholders to support data driven decision making processes.
Experience Required
- Minimum 8+ years of experience with strong hands-on experience as a senior Data Engineer or related role
- 5+ years of demonstrated experience in developing Big Data solutions that support business analytics and data science teams
- 3-5 years of proficient Data ingestion end-to-end implementation of projects in Azure Enterprise Data Lake, Azure Functions, Databricks, Blob Storage, Cosmos DB, Azure stream analytics, Python, SQL
- Extensive hands-on experience implementing Lake house architecture using Data bricks Data Engineering platform, SQL, Unix shell scripting, SQL Analytics, Delta Lake, and Unity Catalog
- Good understanding of spark architecture with Databricks structured streaming, setting up Azure with Databricks, managing clusters in Databricks
- Experienced in DevOps and deployment automations with Azure DevOps - ARM, YAML, Terraform
- Ability to research the latest trends and propose advanced tooling/solutions for Cloud Data Lake & Data Science platforms
- Experience with business intelligence and analytics tools such as OBIEE, PowerBI or Tableau
- Collaborate applications teams/Business users to develop new pipelines with Cloud data migration methodologies and processes including tools like Azure Data Factory, Event Hub, etc
Roles & Responsibilities
- Drive and implement design of data schemas, drive cloud data lake platform design decisions and development standards and maintain data pipelines for data ingestion, processing, and transformation in Azure.
- Drive analysis, architecture, design, governance and development of data warehouse, data lake, and business intelligence solutions
- Using a combination of Azure Data factory, Azure Blob Storage, T-SQL, Pyspark and Azure Databricks should be able to Extract, transform and load from sources system to Azure Data Storage services
- By ensuring data quality, consistency and reliability, Integrate data from various sources,
- Define data requirements, gather and mine large scale structured and unstructured data, and validate data using various tools in a cloud environment.
- Manage and optimize Azure Enterprise data Lake to achieve efficient data storge and processing.
- Develop and optimize ETL processes using Databricks and related tools like Apache Spark
- Implementing data validation and cleansing procedures will ensure the quality, integrity, and dependability of the data.
Similar Jobs
View All
1 Day ago
TrueFan - Senior Machine Learning Engineer
Information Technology
- Thiruvananthapuram, Kerala, India
About UsTrueFan is at the forefront of AI-driven content generation, leveraging cutting-edge generative models to build next-generation products. Our mission is to redefine content generation space through advanced AI technologies, including deep ge...
1 Day ago
Salesforce commerce cloud consultant
Information Technology
- Thiruvananthapuram, Kerala, India
Salesforce Commerce Cloud consultant 5+ Years of Experience 6 to 12 months Mode - Remote 1.1LPM - 1.2LPM Max Key Responsibilities Translate business requirements into scalable Salesforce Service Cloud solutions, in collaboration with CAE's technic...
1 Day ago
Cloud Infrastructure Engineer
Information Technology
- Thiruvananthapuram, Kerala, India
DescriptionInvent the future with us. Recognized by Fast Company’s 2023 100 Best Workplaces for Innovators List, Ampere is a semiconductor design company for a new era, leading the future of computing with an innovative approach to CPU design focuse...
1 Day ago
Devops Engineer- Intermetiate
Information Technology
- Thiruvananthapuram, Kerala, India
BackJD: Dev ops Engineer:As a DevOps Specialist- should be able to take ownership of the entire DevOps process, including Automated CI/CD pipelines and deployment to production.They should also be comfortable with risk analysis and prioritization.Le...
1 Day ago
Sr Data Scientist (London)
Information Technology
- Thiruvananthapuram, Kerala, India
AryaXAI stands at the forefront of AI innovation, revolutionizing AI for mission-critical, highly regulated industries by building explainable, safe, and aligned systems that scale responsibly. Our mission is to create AI tools that empower research...
1 Day ago
Software Test Engineer
Information Technology
- Thiruvananthapuram, Kerala, India
By clicking the “Apply” button, I understand that my employment application process with Takeda will commence and that the information I provide in my application will be processed in line with Takeda’s Privacy Notice and Terms of Use. I further att...
1 Day ago
Software Developer 5 (Java Fullstack)
Information Technology
- Thiruvananthapuram, Kerala, India
Job DescriptionBuilding off our Cloud momentum, Oracle has formed a new organization - Oracle Health Applications & Infrastructure. This team focuses on product development and product strategy for Oracle Health, while building out a complete platfo...
1 Day ago
Java Developer - Spring Frameworks
Information Technology
- Thiruvananthapuram, Kerala, India
Java DescriptionWe are looking for a passionate and talented Java Developer with 2-3 years of hands-on experience to join our growing development team.The ideal candidate should have a strong foundation in Java technologies and the ability to develo...
Talk to us
Feel free to call, email, or hit us up on our social media accounts.
Email
info@antaltechjobs.in