Automation Breaking Fixes: Rate Limits, CAPTCHA, and Brittle Selectors
Identify which failure type is stopping your automation, apply the right fix for each cause, and build in the safeguards that keep scrapers running reliably.
Fast Diagnosis
Rate limit, CAPTCHA, or selector drift: which one broke your run?
Most failures trace to one of three causes. Check your error logs before changing any configuration.
Root Causes
4 failure modes and how to confirm each one
| Root cause | How to confirm | Urgency |
|---|---|---|
| Rate limiting (HTTP 429) | Logs show 429 responses or timeout clusters after rapid sequential requests from a fixed IP | High |
| CAPTCHA detection | Response body contains a Cloudflare challenge page or reCAPTCHA markup instead of expected content | High |
| Brittle CSS or XPath selectors | Run completes with HTTP 200 but extracted data fields return null, empty strings, or wrong values | Medium |
| IP-level block | All requests from a fixed IP return 403 or redirect to a block page regardless of timing | High |
Retrying on a blocked IP can turn a single-address block into a subnet ban. Test from a clean proxy before running any retry logic.
The Fix
5 steps to fix rate limits, CAPTCHA blocks, and broken selectors
Confirm the failure type from logs first, apply the targeted fix, then run a single-request test before restoring full volume.
- Read the logs before touching any config
Pull the most recent run logs and filter by HTTP status code. A 429 means rate limiting, a 403 with challenge markup means CAPTCHA or IP block, and an empty 200 means selector drift.
- Fix rate limiting: add randomized request delays
Set a 2-second minimum delay with a random 1 to 6 second offset. Reduce concurrent workers: single-threaded scrapers trigger throttling far less than parallel runs on the same domain.
- Fix CAPTCHA blocks: rotate proxies and enable stealth mode
Switch from datacenter to residential proxies for targets running Cloudflare or PerimeterX. Enable stealth mode in your headless browser and rotate the user agent header per session.
- Fix brittle selectors: target stable attributes, not positional classes
Compare current page source against your selector and replace class-name or position-based selectors with data-test-id attributes or ARIA roles. Add one fallback selector per extraction field to prevent silent null returns after page updates.
- Fix IP blocks: verify on a clean IP before retrying
Test the same request from a clean residential proxy. If it succeeds, rotate to a fresh IP and pause all retries from the blocked address for at least 24 hours.
Run a single-request test after any fix. Scale to 5% of normal volume and monitor logs for 15 minutes before restoring the full run.
Prevention
4 defaults that prevent rate limits, CAPTCHA blocks, and selector drift
Randomized delays and proxy rotation cost 2 hours to configure at build time. Retrofitting them after a live block costs days of lost data.
A zero-result run is the earliest signal of breaking, often before HTTP errors appear. Configure a webhook alert on any run returning zero extracted records.
- Randomize all request delays from the start
Never use fixed intervals. Apply a base delay plus a random offset so the server cannot fingerprint your traffic as machine cadence.
- Rotate proxy at the session level
Rotate IP on every new session, not every N requests. Session-level rotation reduces fingerprint correlation across requests the server sees as a single visit.
- Validate selectors before the extraction loop begins
Run a selector check at the start of each run and fail fast on any mismatch. This catches page-structure changes before a full run of null data appears successful in your history.
- Set a hard concurrent request ceiling per domain
Define a maximum concurrent request limit inside your scraper config. Infrastructure constraints like server RAM are not a substitute for explicit domain-level throttle budgets.
Scraping Tools
Apify, PhantomBuster, TexAu: built-in protection at the platform level
Common Questions
5 questions: rate limits, CAPTCHAs, and broken selectors
Consistent request patterns get flagged by server-side rate-limit detection after repeated runs from the same IP. Add randomized delays and rotate IPs between sessions to break the detectable cadence.
Rarely with consistent reliability. Free and datacenter proxies are widely blacklisted by Cloudflare and PerimeterX. Residential proxies use real ISP addresses that are significantly harder to block at scale.
Open Chrome DevTools and run your selector via document.querySelectorAll in the Console. If it returns zero nodes, start at the outermost container and work inward to locate the structural change.
A rate limit is temporary: the server resumes after a cooldown of minutes to hours. An IP block is persistent: test from a different IP immediately to confirm which you have.
Cookie-based LinkedIn scraping carries high account risk, with soft bans appearing before a full block. Keep daily connection requests below 20 to 30 and use tools with randomized action timing.
Fixed the automation? Now make it break-proof.
The Scraping SOP covers selector strategy, proxy configuration, ethics, and run monitoring in a single step-by-step reference for operators.
Related Reading