Open a results page in a browser, click “Next”, and it simply works. That’s because the browser quietly carries things from one request to the next, and the website expects them back. A scraper that asks for “page 2” as if it were an ordinary address is a bit like walking into a shop and asking for “the second item the assistant was about to show me”. Nobody knows what you mean, but they’re too polite to say so.

This article is about those quiet extras: what they are, why dropping one almost never produces an error, and how a collector should carry them. It’s about being a well-behaved client of sites you’re allowed to collect from, not about getting past anyone’s defences.

What a browser carries for you

  • Cookies. Small labels the site gave you earlier, sent back with every request. The important one is usually a session cookie, which means “this is the same visitor as before”.
  • Hidden state. Many forms contain fields you never see. In older ASP.NET sites they’re called __VIEWSTATE and friends; elsewhere they’re CSRF or “state” tokens. The server puts a fresh value in every page and expects the next request to send it back.
  • Order. Pages are often meant to be visited in sequence: the search form, then the results, then page two. Jumping straight to the end skips the moment the site hands out what that step needs.
  • Identity. A browser presents itself in many small ways: its headers, how it negotiates the connection, how it behaves over time. Some sites tie a session to that identity, so the same cookie from a different-looking client is treated differently. In the protected-directory project, a cookie earned by one browser identity was only honoured from that same identity and address.

Stateful paging, in one picture

“Page 2” isn’t an address on these sites. It’s a form submission that includes state from page 1, and page 3 needs state from page 2. That’s how the member list in the directory project worked: a grid where page N+1 is a postback carrying hidden state from page N. You can’t jump to page 40. You walk there, one page at a time, handing each page’s state to the next.

Load thesearch formSubmit thesearch (page 1)Submit with page1's state (page 2)Submit with page2's state (page 3)

Try it: take things away

Below is a made-up portal (nothing real is contacted) and a four-step run. It starts as a quick script that sends none of the extras. Switch them on one at a time and watch what each one fixes. Keep an eye on the status column.

Simulation. A made-up portal. Nothing real is contacted.

What the collector sends

Simplified on purpose: real sites differ in what they check and in what order. The point is the pattern, and that every one of these failures looks like success.

Why nobody notices

Every failure in that table came back as 200 OK. The portal isn’t being hostile. It’s answering the question it was asked, which was the wrong question. A sign-in page is a perfectly good page, and a parser that finds the first labelled box on it will happily save “Username” as a company name (more on that in why scrapers fail even when the server says 200). The server isn’t being rude. It’s keeping track of a conversation you forgot you were in.

What a careful collector does

  • Treats the session as an object with an age, not a global variable. It knows when the session started and refreshes it before it expires.
  • Carries the newest hidden state from each response, never a copy from three pages ago.
  • Keeps one identity per session: the same client, address and cookie jar from the first request to the last.
  • Checks every response before parsing it: is this a data page or a sign-in page, and are these records new? A page whose records it has already seen is a warning, not a success.
  • Recovers deliberately. When the session dies, it signs in again with the client’s own account and continues from a checkpoint. If it can’t, it stops and says so, rather than saving a login form.

Two things are worth saying plainly. If a site keeps showing a verification step, the answer isn’t a cleverer trick. It’s slowing down, asking for an official feed, or agreeing a smaller scope with the client. And some sessions can’t be automated at all, such as a login that needs a one-time code. There the honest design puts a person in the loop once (see the dealer-portal story) and automates everything around them.