Kolkata, West Bengal, India
Information Technology
Full-Time
BrightEdge
Overview
The Staff Software Engineer, Data Ingestion will be a critical individual contributor responsible for designing collection strategies, developing, and maintaining robust and scalable data pipelines. This role is at the heart of our data ecosystem, deliver new analytical software solution to access timely, accurate, and complete data for insights, products, and operational efficiency.
Key Responsibility
Key Responsibility
- Design, develop, and maintain high-performance, fault-tolerant data ingestion pipelines using Python.
- Integrate with diverse data sources (databases, APIs, streaming platforms, cloud storage, etc.).
- Implement data transformation and cleansing logic during ingestion to ensure data quality.
- Monitor and troubleshoot data ingestion pipelines, identifying and resolving issues promptly.
- Collaborate with database engineers to optimize data models for fast consumption.
- Evaluate and propose new technologies or frameworks to improve ingestion efficiency and reliability.
- Develop and implement self-healing mechanisms for data pipelines to ensure continuity.
- Define and uphold SLAs and SLOs for data freshness, completeness, and availability.
- Participate in on-call rotation as needed for critical data pipeline issues.
- 6+ years experience in software development industry from computer science background
- Extensive Python Expertise Extensive experience in developing robust, production-grade applications with Python.
- Data Collection & Integration Proven experience collecting data from various sources (REST APIs, OAuth, GraphQL, Kafka, S3, SFTP, etc.).
- Distributed Systems & Scalability Strong understanding of distributed systems concepts, designing for scale, performance optimization, and fault tolerance.
- Cloud Platforms Experience with major cloud providers (AWS or GCP) and their data-related services (e.g., S3, EC2, Lambda, SQS, Kafka, Cloud Storage, GKE).
- Database Fundamentals Solid understanding of relational databases (SQL, schema design, indexing, query optimization). OLAP database experience is a plus (Hadoop)
- Monitoring & Alerting Experience with monitoring tools (e.g., Prometheus, Grafana) and setting up effective alerts.
- Version Control Proficiency with Git.
- Containerization (Plus) Experience with Docker and Kubernetes.
- Streaming Technologies (Plus) Experience with real-time data processing using Kafka, Flink, Spark Streaming
Similar Jobs
View All
5 Hours ago
Frontend Developer - Remote
Information Technology
- 1 - 7 Yrs
- Anywhere in India/Multiple Locations
Location: Remote (India only)
Experience: 1+ Years
Job Type: Full-Time
Salary: Competitive (based on experience)
About the Role
We’re hiring a Frontend Developer with 1+ years of experience to join our remote team. You’ll build responsi...
22 Hours ago
Senior Talent Acquisition Specialist – Tech Hiring
Information Technology
- 5 - 12 Yrs
- Bangalore, Mumbai (All Areas), Hyderabad, Pune
Position: Senior Talent Acquisition Specialist – Tech Hiring
Experience Required: 5+ years
Location: Mumbai, Pune, Bangalore, Hyderabad (Hybrid)
Industry: Technology, IT Services, Product Companies
We’re looking for an experienced Talent Ac...
1 Day ago
Senior Vice President of Engineering
Information Technology
- 12 - 16 Yrs
- India
We are hiring a Senior Vice President (SVP) of Engineering for a high-growth, innovation-led cybersecurity company specializing in AI-powered security solutions. The company secures over 500 million+ devices, is backed by top-tier investors, and is t...
1 Day ago
.Net Developer - C#/ASP/Entity Framework
Information Technology
- Kolkata, West Bengal, India
We are looking for a skilled and detail-oriented .NET Developer to join our dynamic software development team. The successful candidate will be responsible for designing, developing, testing, and deploying .NET applications, with a strong focus on s...
1 Day ago
Astrotalk - Junior iOS Developer - SWIFT/CocoaPods
Manufacturing & Industrial
- Pune, Maharashtra, India
Responsibilities Design, develop, and maintain scalable, high-performance iOS applications using Swift Implement Apple's Human Interface Guidelines to create intuitive and seamless user experiences Leverage frameworks such as CoreData, CoreGraphi...
1 Day ago
[x]cube LABS - Java/Golang Developer
Manufacturing & Industrial
- Pune, Maharashtra, India
Job DescriptionWe are seeking a skilled and motivated Java + GoLang Developer with over 3 years of hands-on experience in backend development. The ideal candidate should have a strong academic background, a solid understanding of software engineerin...
1 Day ago
Senior Associate - Business Analyst
Information Technology
- Kolkata, West Bengal, India
Job PurposeGathering, analyzing, and writing requirements, data analysis, report creation, system flow diagrams, PowerPoints, as well as assisting with project delivery of in-house/vendor application implementation. Work with cross-functional teams ...
1 Day ago
Lead Software Engineer - Java
Information Technology
- Kolkata, West Bengal, India
Top 3 Mandatory Skills Java Microservices Spring Boot Were Looking For a Lead Software EngineerOur Team is looking for a highly skilled and experienced Lead Software Engineer to lead the development and enhancement of a key management system.Th...
Talk to us
Feel free to call, email, or hit us up on our social media accounts.
Email
info@antaltechjobs.in