- ~3,300records collected
- ~530of them sub-accounts
- 0duplicate member IDs
- Listing checksprofiles and sub-accounts verified against the source
What the client needed
The client wanted every member of a professional directory as structured data: name, company, address, phone, website, custom profile fields, and the list of sub-accounts each member manages. Sub-accounts have their own profile pages and can have sub-accounts of their own.
Three problems that stack
Cloudflare in front of every page. Requests need a valid clearance cookie, and that cookie is only honoured from the same IP and the same browser identity that earned it. Running a real browser for thousands of pages works, but it’s slow and expensive.
Stateful paging. The member list is an ASP.NET WebForms grid. Page N+1 is a postback carrying hidden state from page N, so you can’t jump to a page by URL.
A long run. Thousands of profile pages take hours, and something will interrupt it.
Decisions
A cheapest-first access ladder. The collector tries three ways in, in order of cost, and only moves down when the cheaper one fails:
[load progress + saved session]
|
A reuse saved clearance cookie ---- valid --> SESSION OK
| rejected (403/429 or challenge page)
B plain HTTP with a browser TLS fingerprint (up to 5 tries) --> SESSION OK
| still challenged
C real Chrome, once, to earn a fresh clearance --> SESSION OK
| (then back to plain HTTP with its cookie + user agent)
v
SESSION OK -> list page 1 -> read "Page X of Y"
-> postback pages 2..Y, collect member IDs, drop duplicates
(checkpoint every 5 pages)
-> fetch each member profile -> queue its sub-accounts
(checkpoint every 20 profiles, fresh session every 50)
-> breadth-first over the sub-account queue
-> write JSON + CSV
When tier C is used, the browser only earns the cookie. The collector then switches back to plain HTTP and copies the browser’s user agent and network route, because the clearance is tied to both. Getting that detail wrong is the most common reason a “working” Cloudflare scraper starts failing a few minutes in.
Rebuild postbacks, don’t click. Each list page is a POST that carries __VIEWSTATE, __EVENTVALIDATION, and the pager link’s target. If the session drops mid-crawl, the collector refreshes it, loads page 1, and replays postbacks up to where it was. An earlier version skipped a page during that recovery; the fix was to consume the recovered page instead of jumping past it.
The member ID is the identity. Every record is keyed on the site’s numeric member ID. That’s what lets a restart skip finished profiles and keeps the same person from appearing twice when they’re listed under two parents.
The output contract
{"id": "1000001", "title": "Member A, Example Lending Co",
"member_type": "Example Type", "company": "Example Lending Co",
"street_address_lines": ["1 Example St"], "city": "Exampleville",
"state": "EX", "zip_code": "00000", "phone": "000-000-0000",
"email": "[email protected]", "website": "https://example.org",
"custom_fields": {"Region": "Region X"},
"sub_account_count": 1,
"sub_accounts": [{"id": "1000002", "name": "Member B"}],
"is_sub_account": false, "profile_type": "main"}
Sub-account records carry parent_id and parent_name. The site hides email addresses behind a simple character-shift encoding in the page script; the collector decodes them back to plain text.
Output validation
I checked the delivered structure against the source listing, using member IDs and sub-account counts to identify duplicates and gaps:
- Listing member IDs were compared with collected profiles; the check identified one profile requiring follow-up.
- Duplicate IDs in the output: none.
- Each member’s stated sub-account count, compared with the sub-account links actually collected: the totals matched.
The run report makes coverage visible: collected records, duplicate checks, and any profile that needs follow-up. The client can assess the dataset against the source rather than rely on a success message.
Proof
Delivered client work
- Ran to completion on a protected, multi-page source and was checked against the site's own listing counts.
- Delivered as structured data with a documented output format and validation counts.
- Client and site details are confidential.
Good to know
- If the site changes its protection or layout, the collector needs a quick re-tune. It reports failures clearly instead of guessing.
- A real browser is used only when nothing lighter works, because it is slower and costlier at scale.
- Built for the client's authorised use of their own data source.
See why naive collectors fail on sources like this →