Simulation · advanced

Difficult-source lab

Run a quick script and an engineered collector against the same awkward website, and see why "it ran without errors" is not "the data is complete".

Interactive demo

Simulation. A synthetic website with 120 records on 12 pages, built for this demo. No real site is contacted, and no access-protection technique is shown: when the engineered collector meets a bot check, it slows down, retries later or stops and asks a person.
Turn on the problems real sources have

A typical quick script

Requests each page by URL, parses the table, appends rows.

Rows saved
0
Correct
0
Duplicates
0
Junk or corrupted
0
Flagged for review
0

An engineered collector

Checks what each response really is before trusting it.

Rows saved
0
Correct
0
Duplicates
0
Junk or corrupted
0
Flagged for review
0

Diagnose it

What the response is really telling you

Seven synthetic responses a collector might receive. Pick one to see the signals an engineer reads, what is really happening, and the realistic options, each with its honest trade-off rather than a score.

200 OK · 2 KB · Looks like success

Status
200 OK
Size
2 KB (data pages are ~48 KB)
Body
no results container; a script and a "checking your browser" message
Cookies
a short-lived cookie is set
Timing
answered in 40 ms, faster than real pages

What it really is. A verification step from the site's bot detection, not data. Bot-detection products judge the whole session (browser behaviour, cookies, request order, timing) and can answer with status 200.

Options, and what each one costs

  • Slow down and keep sessions longer-livedOften stops the checks from firing at all, and it is the most durable fix. Costs throughput, so a large job takes longer.
  • Use a real browser where the site expects oneReliable when the site truly needs a browser, but slow and heavy to run at scale, and more to keep working as the site changes. Appropriate only within what the site permits.
  • Ask for an official API, export or data agreementThe most reliable and lowest-maintenance route when it exists; depends on the site offering one and on commercial terms.
  • If the checks persist, stop and reportNot a workaround, a decision: better to hand back a clear gap than invented data. Always appropriate.

200 OK · sign-in page · Session expired

Redirects
/results → 302 → /login → 200
Body
a form with a password field
Cookies
session cookie cleared
When
about 25 minutes into the run

What it really is. The login session expired. A parser looking for "rows" will happily extract the sign-in form (this is where a "company" called Username comes from).

Options, and what each one costs

  • Verify each response is a data page before parsingCheap and catches the whole class of problem; needs a reliable signal for "this is really data", which you design per source.
  • Re-authenticate with the client's own account, then retryRestores the run without losing progress; only for access the client is entitled to, using their credentials, never anyone else's.
  • Refresh the session before it expiresAvoids the interruption entirely; needs you to know or measure the session lifetime.

200 OK · empty list · JavaScript-rendered

Body
an empty application container and large script bundles
Network tab
the page itself calls a JSON endpoint after loading
Size
48 KB of HTML, zero records

What it really is. The HTML is a shell; the data arrives through the application's own requests, often undocumented.

Options, and what each one costs

  • Collect from the same JSON request the page makesFast, stable and light once you understand its parameters and session needs; it is an undocumented endpoint, so it can change without notice.
  • Render the page in a headless browserWorks without reverse-engineering anything, but is slow, resource-heavy and more to maintain. A sensible fallback, rarely the first choice.

200 OK · page 7 · Same as page 1

Content hash
identical to page 1
Form
hidden state fields that change with every page
URL
?page=7 is ignored by the server

What it really is. Stateful paging. The next page is a postback that carries state from the current one, so asking by page number repeats page 1.

Options, and what each one costs

  • Carry the state forward from each responseThe correct way to page this kind of site; a little more code than a page-number loop, and worth it. Verify each page is new by its record IDs so a silent repeat is caught.

429 · Retry-After: 30 · Rate limited

Status
429 Too Many Requests
Header
Retry-After: 30
Pattern
appears after bursts from parallel workers

What it really is. The server is explicitly asking the client to slow down.

Options, and what each one costs

  • Honour the wait and share one pace across workersThe respectful, reliable answer; it caps how fast the job can go, which is usually the right trade.
  • Schedule the heavy part off-peakEases the limit without more infrastructure; only helps if the deadline allows it.
  • Add IP capacityRaises throughput, but adds cost and operational complexity, and only makes sense when the use is allowed and pacing alone is not enough. Not a way around a block.

200 OK · 10 records · Two incomplete

Records
10, of which 2 have no price
Layout
a second card layout with fields in a different order
Totals
source says 120 results

What it really is. Partial or variant content. Position-based parsing shifts columns, so a price lands in the rating field and missing values quietly become zeros.

Options, and what each one costs

  • Map fields by label, validate, reconcile with the source totalTurns a silent corruption into a visible, explained gap; costs a validation pass per record, which pays for itself on the first delivery.

403 · reference ID · Hard block

Status
403 Forbidden
Body
a block page with an incident reference
Scope
every request from this session and address

What it really is. Access was denied outright. Retrying harder makes it worse and can affect others who share the address.

Options, and what each one costs

  • Stop, diagnose, and agree the next step with the clientThe only responsible first move; costs time but prevents a bad situation getting worse.
  • Reconsider the route: official access, a partner feed, a smaller scopeOften the real fix. Slower to arrange, far more reliable, and keeps the work on the right side of the line.

Illustrative responses written for this page. They describe general patterns; no protection product's internals or bypass methods are shown.

What this shows

JavaScript rendering, stateful paging, expired sessions, bot checks, rate limits, incomplete records and completeness checks.

Most failed scraping projects don't fail loudly. The script runs, exits with code 0 and produces a file. The problem is what is in the file: duplicates from a page that ignored its page number, a "row" that is really the sign-in form, blank pages where a bot check was served, columns shifted by a layout change. Nobody notices until the data is used.

JavaScript-rendered pages

Why scripts break: the HTML arrives empty and the data loads afterwards. What I do: find the request the page itself makes (often a clean JSON endpoint) and collect from that, or render with a real browser only where it is needed.

Stateful paging

Why scripts break: "page 2" is a postback that carries hidden state from page 1; asking for ?page=2 quietly returns page 1. What I do: carry the state forward from each response and verify every page is new.

Expired sessions

Why scripts break: the server answers 200 with a sign-in page, and the parser happily extracts junk. What I do: check that each response is what it should be before parsing, and sign in again with the client's own account.

Bot checks

Why scripts break: an interstitial page replaces the data, again with status 200. What I do: detect it, slow down, retry later, and when it persists, stop and report instead of guessing. The approach is agreed with the client per source and stays within what they are allowed to access.

Rate limits

Why scripts break: a 429 either crashes the run or gets ignored. What I do: honour the server's requested wait, keep a shared pause across workers, and resume from a checkpoint.

Incomplete records

Why scripts break: a different layout shifts columns, so a price lands in the rating field. What I do: map fields by label, validate every record and flag what is missing with a reason, then reconcile the total against the source.

Close to: protected directory collection, stateful permit portal, listings past a result cap.

Related service: Data extraction from difficult and protected websites

Next step

Need this for real?

The demos run on sample data. Tell me about your source, your workflow or your data, and I will tell you what a real version would involve.