Collecting a member directory behind Cloudflare

A client needed the full contents of a professional directory, including each member’s sub-accounts. The site sits behind Cloudflare and pages through a stateful WebForms grid. I built a collector that uses the cheapest working way past the protection, resumes after a crash, and then checked the output against the listing to see what it missed.

Role

Sole engineer: collector design, run, output checks

Stack

Python, browser-matched HTTP client, Playwright (fallback only), WebForms postbacks

Type

Protected sources

Status

Client work (2026). Client and site details are confidential.

access ladder · cheapest working rung first1 · plain HTTP clientfast, cheap2 · browser-like TLS clientmatched fingerprint3 · real browserPlaywright, fallback onlyWebForms gridstateful pagingrecords~3,300members +sub-accountsgap check vslistingresumable crawl · output verified against the public listing
  1. Problem

    A complete member directory, including nested sub-accounts, as structured data.

  2. Constraints

    Cloudflare on every page, a postback-only paging grid, and a run long enough to be interrupted.

  3. Approach

    A cheapest-first access ladder, state-carrying pagination, a resumable crawl and validation against the source.

  4. Result

    About 3,300 records with about 530 sub-accounts and zero duplicate member IDs, checked against listing counts.

  • ~3,300records collected
  • ~530of them sub-accounts
  • 0duplicate member IDs
  • Listing checksprofiles and sub-accounts verified against the source

What the client needed

The client wanted every member of a professional directory as structured data: name, company, address, phone, website, custom profile fields, and the list of sub-accounts each member manages. Sub-accounts have their own profile pages and can have sub-accounts of their own.

Three problems that stack

Cloudflare in front of every page. Requests need a valid clearance cookie, and that cookie is only honoured from the same IP and the same browser identity that earned it. Running a real browser for thousands of pages works, but it’s slow and expensive.

Stateful paging. The member list is an ASP.NET WebForms grid. Page N+1 is a postback carrying hidden state from page N, so you can’t jump to a page by URL.

A long run. Thousands of profile pages take hours, and something will interrupt it.

Decisions

A cheapest-first access ladder. The collector tries three ways in, in order of cost, and only moves down when the cheaper one fails:

Access ladder and crawl, as the collector runs it
[load progress + saved session]
        |
  A  reuse saved clearance cookie ---- valid --> SESSION OK
        | rejected (403/429 or challenge page)
  B  plain HTTP with a browser TLS fingerprint (up to 5 tries) --> SESSION OK
        | still challenged
  C  real Chrome, once, to earn a fresh clearance --> SESSION OK
        |                   (then back to plain HTTP with its cookie + user agent)
        v
SESSION OK -> list page 1 -> read "Page X of Y"
   -> postback pages 2..Y, collect member IDs, drop duplicates
        (checkpoint every 5 pages)
   -> fetch each member profile -> queue its sub-accounts
        (checkpoint every 20 profiles, fresh session every 50)
   -> breadth-first over the sub-account queue
   -> write JSON + CSV

When tier C is used, the browser only earns the cookie. The collector then switches back to plain HTTP and copies the browser’s user agent and network route, because the clearance is tied to both. Getting that detail wrong is the most common reason a “working” Cloudflare scraper starts failing a few minutes in.

Rebuild postbacks, don’t click. Each list page is a POST that carries __VIEWSTATE, __EVENTVALIDATION, and the pager link’s target. If the session drops mid-crawl, the collector refreshes it, loads page 1, and replays postbacks up to where it was. An earlier version skipped a page during that recovery; the fix was to consume the recovered page instead of jumping past it.

The member ID is the identity. Every record is keyed on the site’s numeric member ID. That’s what lets a restart skip finished profiles and keeps the same person from appearing twice when they’re listed under two parents.

The output contract

Synthetic values · real field names One output record (JSON; the CSV has the same fields plus one column per custom field)
{"id": "1000001", "title": "Member A, Example Lending Co",
 "member_type": "Example Type", "company": "Example Lending Co",
 "street_address_lines": ["1 Example St"], "city": "Exampleville",
 "state": "EX", "zip_code": "00000", "phone": "000-000-0000",
 "email": "[email protected]", "website": "https://example.org",
 "custom_fields": {"Region": "Region X"},
 "sub_account_count": 1,
 "sub_accounts": [{"id": "1000002", "name": "Member B"}],
 "is_sub_account": false, "profile_type": "main"}

Sub-account records carry parent_id and parent_name. The site hides email addresses behind a simple character-shift encoding in the page script; the collector decodes them back to plain text.

Output validation

I checked the delivered structure against the source listing, using member IDs and sub-account counts to identify duplicates and gaps:

  • Listing member IDs were compared with collected profiles; the check identified one profile requiring follow-up.
  • Duplicate IDs in the output: none.
  • Each member’s stated sub-account count, compared with the sub-account links actually collected: the totals matched.

The run report makes coverage visible: collected records, duplicate checks, and any profile that needs follow-up. The client can assess the dataset against the source rather than rely on a success message.

Proof

Delivered client work

  • Ran to completion on a protected, multi-page source and was checked against the site's own listing counts.
  • Delivered as structured data with a documented output format and validation counts.
  • Client and site details are confidential.

Good to know

  • If the site changes its protection or layout, the collector needs a quick re-tune. It reports failures clearly instead of guessing.
  • A real browser is used only when nothing lighter works, because it is slower and costlier at scale.
  • Built for the client's authorised use of their own data source.

See why naive collectors fail on sources like this →

Next step

Is your source behind Cloudflare or a login?

Send the URL and the fields you need. I'll tell you which access approach is likely to work and what it costs to run.