Chennai, Tamil Nadu, India
Information Technology
Full-Time
R Systems
Overview
Your Role and Impact
We are seeking a skilled Data Engineer to lead the migration from Hive Catalog to Databricks Unity Catalog on Azure. The Data Engineer will own the end-to-end migration of metadata and access controls from Hive Catalog to Unity Catalog within the Azure cloud environment. The role demands strong expertise in data cataloging, metadata management, Azure cloud infrastructure, and security best practices.
Your Contribution
We are seeking a skilled Data Engineer to lead the migration from Hive Catalog to Databricks Unity Catalog on Azure. The Data Engineer will own the end-to-end migration of metadata and access controls from Hive Catalog to Unity Catalog within the Azure cloud environment. The role demands strong expertise in data cataloging, metadata management, Azure cloud infrastructure, and security best practices.
Your Contribution
- Analyze the existing Hive Catalog metadata, schema, and security configurations.
- Design and execute a robust migration plan to Unity Catalog with minimal disruption and data integrity assurance.
- Collaborate with Data Governance, Security, and Cloud Infrastructure teams to implement access controls and policies leveraging Azure Active Directory (AAD).
- Develop automation scripts and tools to support migration, validation, and ongoing management.
- Troubleshoot migration challenges and provide post-migration support.
- Document migration processes and train stakeholders on Unity Catalog capabilities.
- Integrate Unity Catalog with Azure native services such as Azure Data Lake Storage Gen2, Azure Key Vault, and Azure Active Directory for security and identity management.
- Optimize Azure resource utilization during migration and production workloads.
- Keep current with Azure Databricks Unity Catalog enhancements and Azure cloud best practices.
- trong knowledge of metadata management, data governance frameworks, and data cataloging.
- Proficient in SQL, Python, and scripting for automation.
- Hands-on experience with Azure Databricks, Apache Spark, and Azure cloud services including Azure Data Lake Storage Gen2, Azure Key Vault, and Azure Active Directory.
- In-depth understanding of Azure cloud infrastructure: compute (VMs, Azure Databricks clusters), storage, networking, and security components.
- Experience integrating data catalog solutions with Azure identity and access management (Azure AD, RBAC).
- Strong grasp of data security, IAM policies, and access control in Azure environments.
- Excellent analytical, problem-solving, and communication skills.
Similar Jobs
View All
1 Day ago
Software Developer 4
Information Technology
- Bangalore, Karnataka, India
Job DescriptionDesign, develop, troubleshoot and debug software programs for databases, applications, tools, networks etc.ResponsibilitiesAs a member of the software engineering division, you will take an active role in the definition and evolution ...
1 Day ago
Software Developer (ReactJS & DotNet) in Faridabad, Noida, Palwal, Ballabgarh
Information Technology
- Bangalore, Karnataka, India
As a Software Developer at Digital Shahi Pvt. Ltd., you will have the opportunity to work on cutting-edge technologies such as ReactJS, DotNet, MongoDB, MERN stack, and Node.js. Join our dynamic team and be a part of revolutionizing the digital land...
1 Day ago
iOS Developer - Objective C/SWIFT
Information Technology
- Bangalore, Karnataka, India
Job DescriptionWe are looking for a skilled and enthusiastic iOS Developer to join our team. The candidate will have experience developing and maintaining mobile applications on the iOS platform and possess a passion for creating user-friendly and h...
1 Day ago
Manual/Automation Tester
Information Technology
- Thiruvananthapuram, Kerala, India
Job DescriptionThis is a full-time on-site role located in Hyderabad. As a Testing professional with 0-2 year of experience, you will be responsible for conducting testing activities, including developing and executing test plans, test cases, and te...
1 Day ago
Sr. Software Engineer - iOS Development
Information Technology
- Thiruvananthapuram, Kerala, India
Position Description: As a Senior Software Engineer, iOS Developer, you will contribute to the development of intuitive, high-performance iOS applications for enterprise use. This role focuses on implementing features and improving the product exper...
1 Day ago
Sr. Software Engineer
Information Technology
- Thiruvananthapuram, Kerala, India
What you will do: Design, develop, enhance, and test software / embedded software for medical and connectivity products for R&D department. Closely work with peers in software engineering, software quality assurance and project & product management ...
1 Day ago
PHP Developer - Laravel Frameworks
Information Technology
- Thiruvananthapuram, Kerala, India
Location: Pune, Bengaluru, Chennai, Hyderabad, GurugramWork Model: Hybrid (3 days from office)Experience Required: 10+ yearsRole Type: Individual ContributorClient: US-based multinational banking institutionNotice Period: Immediate to 21 daysRole Su...
1 Day ago
Data Engineer - Generative AI
Information Technology
- Thiruvananthapuram, Kerala, India
About The RoleWe are seeking a skilled and passionate Data Engineer to join our team and drive the development of scalable data pipelines for Generative AI (GenAI) and Large Language Model (LLM)-powered applications. This role demands hands-on exper...
Talk to us
Feel free to call, email, or hit us up on our social media accounts.
Email
info@antaltechjobs.in