Web Scraping Engineer
You've built scrapers that ran perfectly for three months, then broke overnight because a target site shipped new bot detection on a Tuesday. You've written retry logic, rotated proxies, swapped fingerprints, and still ended up staring at a CAPTCHA wall at midnight. You know the difference between a scraper that works in a demo and one that survives in production.
We're looking for an engineer who has lived inside this problem. Someone who understands why requests fail, how anti-bot systems fingerprint traffic, and what it takes to collect web data reliably across thousands of targets. You won't be scraping for one company anymore. You'll be building and testing against the infrastructure that powers scraping at scale for hundreds of them.
职责
✔️ Build, maintain, and optimize web scraping systems that test and validate Massive's proxy and browser automation infrastructure against real-world targets.
✔️ Investigate anti-bot detection systems, fingerprinting techniques, and evasion strategies. Understand what's blocking requests and why.
✔️ Design reproducible benchmarking frameworks that measure proxy performance across geographies, ASNs, and target difficulty levels.
✔️ Collaborate with product and infrastructure teams to translate scraping failures into engineering requirements.
要求
✔️ 3+ years of hands-on experience building and running web scrapers in production. Not tutorials. Production systems with uptime expectations and data quality requirements.
✔️ Strong Python skills. Experience with at least one of: Scrapy, Playwright, Puppeteer, Selenium, or raw HTTP clients like httpx/curl_cffi.
✔️ Working knowledge of how anti-bot systems operate: TLS fingerprinting, header analysis, JavaScript challenges, behavioral detection, IP reputation scoring.
✔️ Familiarity with proxy infrastructure (residential, ISP, datacenter) and how proxy selection impacts success rates on different target categories.
✔️ You've debugged a scraper that worked on one ISP and failed on another, and you figured out why.