The most expensive scraping failures do not crash. The job finishes, the log is full of 200 OK, a file appears, and someone builds a report on it. Only later does anyone notice the data is wrong: duplicates, a gap where a run of pages returned nothing, and a “company” named Username.

That last one is worth pausing on, because it explains the whole problem. Picture a scraper reading a company directory. Partway through, its login session expires, so the server stops returning the directory and returns the sign-in page instead, with status 200. The scraper doesn’t see a login page; it sees HTML. It finds the first labelled field, which is the form’s Username box, and dutifully saves “Username” as a company name. The scraper didn’t crash. It successfully collected the login form. (That example is illustrative, not a specific client incident, and the exact shape depends on the site; the failure itself is common.)

A 200 means the HTTP request was handled successfully at the transport level. It says nothing about whether the body contains the records you asked for, or whether they are complete and correct. A server can answer 200 with a sign-in form, a bot-detection check, an empty application shell or the wrong page, and every one of those parses without an error. This article is about catching that, and it comes from collecting data from sites that are deliberately hard to collect from.

Five responses that say 200 and mean something else

  1. A verification or challenge page. Bot-detection services such as Akamai and DataDome judge the whole session: browser behaviour, cookies, request order and timing. When a check fails they often return a small page with a script instead of the data, sometimes with status 200.
  2. A sign-in page. The session expired somewhere in the run. The server redirects to the login form, which arrives as 200.
  3. An application shell. The HTML is a container; the records are loaded afterwards by JavaScript from the application’s own, usually undocumented, requests.
  4. The previous page again. Stateful paging ignores ?page=7 and serves page 1, because the real next page is a postback that carries hidden state.
  5. A partial or different layout. Ten cards instead of twenty, or a second template where fields appear in a different order, so position-based parsing shifts a price into a rating.

Check what you received, not how it arrived

The fix is not a smarter parser. It is a short checklist that runs on every response before parsing, and decides whether the response is data at all.

ResponseIs it adata page?Is it new?Is eachrecord valid?Save +checkpoint
  • Shape: the container that holds records exists, and the page is roughly the size real data pages are. A 2 KB answer where pages are normally 48 KB is a signal on its own.
  • Identity: no password field, no challenge script, no block-page reference ID. Look for the markers of the pages you do not want, as well as the ones you do.
  • Novelty: the record IDs on this page have not been seen before. A content hash identical to the previous page means paging is broken.
  • Validity: every record has the fields that must exist, mapped by label rather than position. Missing values are stored as missing, with a reason, never as zero.
  • Totals: at the end, the count is reconciled with what the source says it holds. Differences are explained in the run report.

Then decide what each failure means

Once a response is classified, the right reaction differs. A sign-in page means sign in again with the client’s own account. A rate limit means wait as long as the server asks, with one shared pace across workers. Repeated pages mean carry the state forward. A verification step is different: the first move is usually to reduce pressure (slower pacing, longer-lived sessions) and to see whether an official API or export exists. If checks persist, the honest result is a report, not a guess.

The responses below are synthetic, written for this article, but each one is a pattern I have met on real projects. Pick one to see the signals and the options with their trade-offs.

Diagnose it

What the response is really telling you

Seven synthetic responses a collector might receive. Pick one to see the signals an engineer reads, what is really happening, and the realistic options, each with its honest trade-off rather than a score.

200 OK · 2 KB · Looks like success

Status
200 OK
Size
2 KB (data pages are ~48 KB)
Body
no results container; a script and a "checking your browser" message
Cookies
a short-lived cookie is set
Timing
answered in 40 ms, faster than real pages

What it really is. A verification step from the site's bot detection, not data. Bot-detection products judge the whole session (browser behaviour, cookies, request order, timing) and can answer with status 200.

Options, and what each one costs

  • Slow down and keep sessions longer-livedOften stops the checks from firing at all, and it is the most durable fix. Costs throughput, so a large job takes longer.
  • Use a real browser where the site expects oneReliable when the site truly needs a browser, but slow and heavy to run at scale, and more to keep working as the site changes. Appropriate only within what the site permits.
  • Ask for an official API, export or data agreementThe most reliable and lowest-maintenance route when it exists; depends on the site offering one and on commercial terms.
  • If the checks persist, stop and reportNot a workaround, a decision: better to hand back a clear gap than invented data. Always appropriate.

200 OK · sign-in page · Session expired

Redirects
/results → 302 → /login → 200
Body
a form with a password field
Cookies
session cookie cleared
When
about 25 minutes into the run

What it really is. The login session expired. A parser looking for "rows" will happily extract the sign-in form (this is where a "company" called Username comes from).

Options, and what each one costs

  • Verify each response is a data page before parsingCheap and catches the whole class of problem; needs a reliable signal for "this is really data", which you design per source.
  • Re-authenticate with the client's own account, then retryRestores the run without losing progress; only for access the client is entitled to, using their credentials, never anyone else's.
  • Refresh the session before it expiresAvoids the interruption entirely; needs you to know or measure the session lifetime.

200 OK · empty list · JavaScript-rendered

Body
an empty application container and large script bundles
Network tab
the page itself calls a JSON endpoint after loading
Size
48 KB of HTML, zero records

What it really is. The HTML is a shell; the data arrives through the application's own requests, often undocumented.

Options, and what each one costs

  • Collect from the same JSON request the page makesFast, stable and light once you understand its parameters and session needs; it is an undocumented endpoint, so it can change without notice.
  • Render the page in a headless browserWorks without reverse-engineering anything, but is slow, resource-heavy and more to maintain. A sensible fallback, rarely the first choice.

200 OK · page 7 · Same as page 1

Content hash
identical to page 1
Form
hidden state fields that change with every page
URL
?page=7 is ignored by the server

What it really is. Stateful paging. The next page is a postback that carries state from the current one, so asking by page number repeats page 1.

Options, and what each one costs

  • Carry the state forward from each responseThe correct way to page this kind of site; a little more code than a page-number loop, and worth it. Verify each page is new by its record IDs so a silent repeat is caught.

429 · Retry-After: 30 · Rate limited

Status
429 Too Many Requests
Header
Retry-After: 30
Pattern
appears after bursts from parallel workers

What it really is. The server is explicitly asking the client to slow down.

Options, and what each one costs

  • Honour the wait and share one pace across workersThe respectful, reliable answer; it caps how fast the job can go, which is usually the right trade.
  • Schedule the heavy part off-peakEases the limit without more infrastructure; only helps if the deadline allows it.
  • Add IP capacityRaises throughput, but adds cost and operational complexity, and only makes sense when the use is allowed and pacing alone is not enough. Not a way around a block.

200 OK · 10 records · Two incomplete

Records
10, of which 2 have no price
Layout
a second card layout with fields in a different order
Totals
source says 120 results

What it really is. Partial or variant content. Position-based parsing shifts columns, so a price lands in the rating field and missing values quietly become zeros.

Options, and what each one costs

  • Map fields by label, validate, reconcile with the source totalTurns a silent corruption into a visible, explained gap; costs a validation pass per record, which pays for itself on the first delivery.

403 · reference ID · Hard block

Status
403 Forbidden
Body
a block page with an incident reference
Scope
every request from this session and address

What it really is. Access was denied outright. Retrying harder makes it worse and can affect others who share the address.

Options, and what each one costs

  • Stop, diagnose, and agree the next step with the clientThe only responsible first move; costs time but prevents a bad situation getting worse.
  • Reconsider the route: official access, a partner feed, a smaller scopeOften the real fix. Slower to arrange, far more reliable, and keeps the work on the right side of the line.

Illustrative responses written for this page. They describe general patterns; no protection product's internals or bypass methods are shown.

Why this matters more now

A first scraper is quick to write, and quicker still to generate. What takes experience is knowing which of these failures a particular source produces, designing checks that catch them, and keeping the collector correct when the site changes. That is the part clients actually pay for: data they can trust, and a clear report when something could not be collected.

If you want to see the whole pattern run, the difficult-source lab puts a quick script and an engineered collector side by side against the same awkward site.