Bangalore, Karnataka, India
Information Technology
Full-Time
OSI Digital
Overview
We are looking for a skilled Web Crawling Engineer with strong expertise in Python and hands-on experience building scalable crawlers and scrapers. The ideal candidate should have deep knowledge of crawling frameworks, anti-blocking techniques, and proxy management, along with experience solving challenges like captchas and rate-limiting.
Key Responsibilities
Key Responsibilities
- Design, develop, and maintain scalable and efficient web crawlers for data extraction.
- Work with Scrapy, Requests, and other Python libraries to implement crawling solutions.
- Handle anti-scraping measures, including IP blocks, rate limits, and captchas.
- Implement and manage proxy rotation and session management strategies.
- Ensure high-quality and structured data extraction, cleaning, and storage.
- Monitor crawler performance, troubleshoot issues, and optimize for reliability and efficiency.
- Collaborate with the data engineering team to integrate crawled data into pipelines.
- Stay updated with the latest tools, techniques, and best practices in web crawling.
- Strong programming skills in Python.
- Proven experience with Scrapy, Requests, BeautifulSoup, Selenium (if required).
- Hands-on experience solving blocking issues, handling
- Good understanding of proxy usage, rotation, and fingerprinting avoidance techniques.
- Knowledge of HTTP, cookies, headers, sessions, and request/response cycles.
- Experience with databases (SQL/NoSQL) for storing crawled data.
- Strong debugging, problem-solving, and analytical skills.
- Experience with cloud environments (AWS, GCP, Azure).
- Knowledge of data pipelines and ETL processes.
- Bachelor’s degree in Computer Science, Information Technology, or equivalent practical experience.
- 3+ years of experience in web crawling or related fields
Similar Jobs
View All
1 Day ago
Data Scientist
Information Technology
- Bangalore, Karnataka, India
All Posting Locations: Ahmedabad, Gujarat, IN Job Functions: Global Business Services Date Published: May 14, 2025 Ref#: R-92872 ABOUT THE ROLE Job Description Job Title: Data Scientist – Operations Location: Kraft Heinz Global Capability Center (GCC...
1 Day ago
Associate Data Scientist , Global GBS
Information Technology
- Bangalore, Karnataka, India
All Posting Locations: Ahmedabad, Gujarat, IN Job Functions: Global Business Services Date Published: May 28, 2025 Ref#: R-89892 ABOUT THE ROLE Job Description About Us: We're a global leader in the food and beverage industry, dedicated to making lif...
1 Day ago
System Administrator
Information Technology
- 8805.59 - 37085.9 INR - Monthly
- Bangalore, Karnataka, India
A System Administrator is responsible for the installation, maintenance, and reliable operation of an organization's computer systems, servers, and networks to ensure security, efficiency, and high availability. Key duties include configuring and m...
1 Day ago
Cyber Security Software Developer
Information Technology
- Bangalore, Karnataka, India
IntroductionIBM is a global technology and innovation company. It is the largest technology and consulting employer in the world, with presence in 170 countries. The diversity and breadth of the entire IBM portfolio of research, consulting, solution...
1 Day ago
IT Administrator
Information Technology
- 30000 - 40000 INR - Monthly
- Bangalore, Karnataka, India
Role Summary· Own day-to-day IT operations for a 37 member firm covering user support, Microsoft 365 administration, device and network management, security & backup, and vendor coordination.· Engage stakeholders to understand business needs; desig...
1 Day ago
Cloud Architect
Information Technology
- 1200000 - 1440000 INR - Yearly
- Bangalore, Karnataka, India
Overview:We are seeking an experienced Cloud Architect with over 2 years of experience to lead the design and implementation of our cloud infrastructure, specifically tailored for our advanced AI and machine learning initiatives. You will be the ar...
1 Day ago
System Administrator
Information Technology
- 15000 - 20000 INR - Monthly
- Bangalore, Karnataka, India
Contact - 9150917569 Walk-In Drive on Saturday 20-09-2025 Job Summary:We are seeking a skilled and proactive System Administrator to join our IT team. The ideal candidate will be responsible for maintaining, configuring, and ensuring reliable ope...
1 Day ago
QA Manual Tester
Information Technology
- 15000 - 30000 INR - Monthly
- Bangalore, Karnataka, India
Software Tester duties and responsibilitiesanalyzing users stories and/use cases/requirements for validity and feasibilitycollaborate closely with other team members and departmentsKnowledge of API Testing.Execute all levels of testing (System, Fun...
Talk to us
Feel free to call, email, or hit us up on our social media accounts.
Email
info@antaltechjobs.in