Hyderabad, Telangana, India
Information Technology
Full-Time
Sedin Technologies
Overview
Databricks Data Engineer
Key Responsibilities :
Key Responsibilities :
- Design, develop, and maintain high-performance data pipelines using Databricks and Apache Spark.
- Implement medallion architecture (Bronze, Silver, Gold layers) for efficient data processing.
- Optimize Delta Lake tables, partitioning, Z-ordering, and performance tuning in Databricks.
- Develop ETL/ELT processes using PySpark, SQL, and Databricks Workflows.
- Manage Databricks clusters, jobs, and notebooks for batch and real-time data processing.
- Work with Azure Data Lake, AWS S3, or GCP Cloud Storage for data ingestion and storage.
- Implement CI/CD pipelines for Databricks jobs and notebooks using DevOps tools.
- Monitor and troubleshoot performance bottlenecks, cluster optimization, and cost management.
- Ensure data quality, governance, and security using Unity Catalog, ACLs, and encryption.
- Collaborate with Data Scientists, Analysts, and Business Teams to deliver Skills & Experience :
- 5+ years of hands-on experience in Databricks, Apache Spark, and Delta Lake.
- Strong SQL, PySpark, and Python programming skills.
- Experience in Azure Data Factory (ADF), AWS Glue, or GCP Dataflow.
- Expertise in performance tuning, indexing, caching, and parallel processing.
- Hands-on experience with Lakehouse architecture and Databricks SQL.
- Strong understanding of data governance, lineage, and cataloging (e.g., Unity Catalog).
- Experience with CI/CD pipelines (Azure DevOps, GitHub Actions, or Jenkins).
- Familiarity with Airflow, Databricks Workflows, or orchestration tools.
- Strong problem-solving skills with experience in troubleshooting Spark jobs.
- Hands-on experience with Kafka, Event Hubs, or real-time streaming in Databricks.
- Certifications in Databricks, Azure, AWS, or GCP.
Similar Jobs
View All
6 Hours ago
Lead Python Developer
Information Technology
- 8 - 12 Yrs
- Anywhere in India/Multiple Locations
My Client is revolutionizing the hospitality industry by transforming how businesses manage their finances through cutting-edge Accounts Payable automation. Our mission? To eliminate manual work from business expenses and empower operators with real-...
23 Hours ago
US Voice Process (Banking Domain)
Telecommunications
- 40000 - 45000 INR - Monthly
- 1 - 1 Yrs
- Greater Noida
Process: US Voice – Banking (International)
Eligibility: Graduate / Undergraduate with 6 months international voice process experience
Experience: 6 months – 2+ years (in International BPO – Voice)
Salary:
₹40,000 CTC (6 months+ experienc...
1 Day ago
Software/Web Developer - AngularJS/Javascript
Information Technology
- Hyderabad, Telangana, India
We are hiring for women who are on career break in Hyderabad.Company : MicronRole : Software Developer - Application Development and SupportSkills : Web Development, JavaScript & Angular, C# & SQL, Agile methodologies, API, Cloud experience Azure, A...
1 Day ago
OculusIT - Network Administrator
Information Technology
- Hyderabad, Telangana, India
Company DescriptionOculusIT is higher education's high-touch, customer-centric IT services partner of choice.They specialize in cost-effective, responsive, and flexible IT Leadership; Managed ERP, Security, and Infrastructure Services; and 24x7 Oper...
1 Day ago
Software Engineer - Backend Technologies
Information Technology
- Pune, Maharashtra, India
Job DescriptionOur engineering team is hiring a backend software engineer to contribute to the development of our Warehouse Management System (WMS) and its companion Handy Terminal device, both of which are integral to our logistics product suite.Th...
1 Day ago
DevOps Engineer - Cloud Infrastructure
Information Technology
- Hyderabad, Telangana, India
Key Responsibility Areas (KRAs) CI/CD Pipeline Management : Design, build, and maintain robust, automated CI/CD pipelines for various applications, ensuring seamless integration, testing, and deployment processes. Infrastructure as Code (IaC) : De...
1 Day ago
Software Engineer - Frontend Internship in Noida, Gurgaon
Information Technology
- Pune, Maharashtra, India
Selected Intern's Day-to-day Responsibilities Include Develop web apps and landing pages as provided by the designer Work on challenging projects involving heavy rendering Demonstrate team spirit and collaborate with peers About Company: Nicola ...
1 Day ago
Snowflake Data Engineer - SQL
Information Technology
- Pune, Maharashtra, India
About The JobExp : 8 - 14 YrsWork Mode : HybridLocation : Bangalore, Chennai, Kolkata, Pune and GurgaonPrimary Skills : Architect, Snowflake, SQL, DWH, Power BI, ETL and Informatica.We are seeking a skilled Snowflake Developer with a strong backgrou...
Talk to us
Feel free to call, email, or hit us up on our social media accounts.
Email
info@antaltechjobs.in