Back to search
Python Web Scrapping Engineer - Competetive salary - Remote
Overview
We are looking for a Senior Python Scraping Engineer to design, build, and operate large-scale, resilient web scraping systems, with a strong focus on Google and Google-like search environments.
This role sits at the intersection of data engineering, reverse engineering, and systems reliability, and is critical to delivering accurate, timely, and trustworthy data at scale.
You will work on complex scraping problems where nave approaches do not work-dynamic rendering, sophisticated anti-bot defences, frequent markup changes, and high-volume data collection.
We are leaders in adopting AI-assisted and agentic coding workflows. This role is ideal for someone who believes in leveraging AI tools to accelerate development and wants to deepen their skills in this area.
What You'll Do
Design, build, and maintain large-scale Python scraping systems that operate reliably against Google and other highly defended websites
Develop scrapers for dynamic, JavaScript-heavy pages using browser automation and hybrid scraping approaches
Continuously adapt to changes in page structure, request flows, and anti-automation mechanisms
Engineer robust extraction pipelines that prioritise data accuracy, consistency, and observability
Implement and optimise proxy, fingerprinting, and request-routing strategies to maximise stability and throughput
Monitor scrape health, detect failures early, and debug production issues quickly and methodically
Optimise performance, cost, and latency across scraping infrastructure
Collaborate closely with data engineers, data scientists, and product teams to ensure scraped data is usable and trusted
Document scraping logic, architectural decisions, and operational runbooks for long-term maintainability
Required Skills and Experience
Strong professional experience with Python, including production-grade scraping systems
Deep, hands-on experience scraping Google or similarly protected platforms
Solid understanding of HTTP, TLS, cookies, headers, redirects, and browser networking behaviour
Experience with browser automation frameworks such as Playwright, Selenium, Puppeteer, or equivalents
Strong knowledge of HTML parsing, DOM traversal, and data extraction techniques
Proven experience handling rate limiting, CAPTCHAs, IP rotation, and bot detection systems
Comfortable designing and operating asynchronous and concurrent scraping architectures
Experience running scraping systems at scale in cloud environments
Excellent debugging skills and the ability to reason about complex, non-obvious failure modes
Hands-on experience with headless browser fingerprinting and anti-bot evasion techniques
To apply for this role please submit your CV or contact Dillon Blackburn on (phone number removed) or at (url removed).
Tenth Revolution Group are the go-to recruiter for Data & AI roles in the UK offering more opportunities across the country than any other recruitment agency. We're the proud sponsor and supporter of SQLBits, Power Platform World Tour, and the London Fabric User Group. We are the global leaders in Data & AI recruitment.
