Exporting 2,478 permit records from a stateful county portal

A client needed every roofing, siding, and window permit a US county issued in 2026, with contractor details from each record page. The portal only pages through results one postback at a time. I delivered a 2,478-row workbook and the script that produced it, built so an interrupted run resumes without redoing finished records.

Role

Sole engineer: scoping, build, run, delivery

Stack

Python, browser-matched HTTP client, BeautifulSoup, openpyxl

Type

Data extraction

Status

Completed client contract (2026). Client and data are confidential.

p1p2p3p4p5p6county portal · one postback per page__VIEWSTATE carried page to pagecheckpoint.jsonresume here · finished records skippedpermits.xlsx2,478 rows · contractor details
  1. Problem

    Every permit record from a county portal as a clean export.

  2. Constraints

    A stateful portal: sessions, server-side paging state and records reachable only through the UI flow.

  3. Approach

    Checkpoint after every record, resume without repeats, and a Run Info sheet recording the search, date range and record count.

  4. Result

    2,478 records delivered, 30 columns each, with zero duplicate record numbers and zero detail errors.

  • 2,478permit records delivered
  • 30columns per record
  • 0duplicate record numbers
  • 0records with a detail error

What the client needed

A construction lead-generation client wanted every roofing, siding, and window permit issued by a US county between January and late September 2026. For each permit they needed the list fields (date, record number, type, address, status) plus details that only appear on the record’s own page: the project’s cost, roof slope, window count, and the licensed contractor’s name, licence, and phone.

The deliverable was a workbook and the script, so they could rerun it later for new dates. The final file had 2,478 rows. Its Run Info sheet records the search, the date range, and the record count, so the client could see exactly what was asked of the portal.

Why the obvious approach doesn’t work

The portal runs on Accela, an ASP.NET WebForms application. Results pages have no page number in the URL. Each “Next” click is a form postback that carries hidden state (__VIEWSTATE, __EVENTVALIDATION) from the page before it. A browser handles that for you, but driving a browser through hundreds of list pages and 2,478 detail pages is slow and breaks easily.

The second problem is length. A run takes hours. If it dies at record 1,300, starting over wastes the morning, and resuming carelessly can skip or duplicate records.

Decisions

Requests instead of a browser. I used an HTTP client with a Chrome-like TLS fingerprint and rebuilt each postback by hand: read every live form control from the current page, find the Next link’s __doPostBack target, and post it back with the right __EVENTTARGET. The run ends when there is no Next link.

Record number as the identity. Every row is keyed on the permit’s record number. Duplicates seen across pages are dropped during the run, so a permit that shows up twice in the portal’s results appears once in the workbook.

Checkpoint completed details, not page positions. This is the decision that matters most, and the one I’d make again. Saving “I was on page 132” is fragile, because ViewState from an old session is useless after a restart. So the checkpoint stores finished records instead, each with a _detail_complete flag. On restart the script re-walks the list pages, which is cheap, and skips the detail fetch for any record already complete, which is the expensive part.

Synthetic values · real field names Checkpoint file: a list of finished rows, rewritten after every detail page
[
  {"Date": "03/02/2026", "Record Number": "SYN-2026-000101",
   "Record Type": "Roofing Siding Window", "Status": "Issued",
   "Details URL": "https://example.invalid/CapDetail.aspx?id=SYN101",
   "Application Information - ROOFING - Roof Slope": "4/12",
   "Licensed Professional - Name": "Example Roofing LLC",
   "Detail Error": "", "_detail_complete": true},
  {"Record Number": "SYN-2026-000102", "...": "...",
   "Detail Error": "Application Information section not found",
   "_detail_complete": false}
]

What an interruption looks like

Here is the restart path, reconstructed from the script’s own log messages. The first run is stopped during page 132. The rerun reloads 1,311 finished records, walks the list again, and only fetches details for what’s missing.

Synthetic values · real field names Interrupted run and resume, reconstructed from the script’s log format
List page 132: 10 records; processing details now
Details 1/10: SYN-2026-001311
Details 2/10: SYN-2026-001312
^C
Interrupted; latest checkpoint retained.              # exit code 130

$ python permits_scraper.py ...                 # same command
Checkpoint loaded: 1311 completed records available for resume.
List page 1: 10 records; processing details now
  Checkpoint: 10 completed records on this page will be skipped.
...
List page 132: 10 records; processing details now
  Checkpoint: 1 completed records on this page will be skipped.
Details 2/10: SYN-2026-001312
...
Collected and processed 2478 unique records
Saved permits_roofing_siding_window_2026.xlsx         # checkpoint deleted, exit 0

The checkpoint is written to a temporary file and then renamed over the old one, so a crash during the write leaves the previous good checkpoint in place, never half-written JSON. The script also exits with a non-zero code if any record ended incomplete, and keeps the checkpoint in that case, so a scheduled run can’t report success with gaps in it.

The output

Synthetic values · real field names Workbook sample, selected columns
Date Record Number Status Roof Slope Cost of Project Contractor Detail Error
03/02/2026 SYN-2026-000101 Issued 4/12 12000 Example Roofing LLC
03/03/2026 SYN-2026-000102 Finaled 8500 Sample Exteriors Inc

Every row links back to its record page on the portal, so any value can be checked at the source. The detail-error column was empty for all 2,478 rows in the delivered file.

Proof

Delivered client work

  • Delivered export with a Run Info sheet recording the search, date range and record count.
  • Client and data are confidential.

Good to know

  • Built around the portal's current flow; if the portal is redesigned, the navigation steps are updated.
  • Paced to what the portal handles comfortably.

Try the resumable job runner →

More work

Other case studies

access ladder · cheapest working rung first1 · plain HTTP clientfast, cheap2 · browser-like TLS clientmatched fingerprint3 · real browserPlaywright, fallback onlyWebForms gridstateful pagingrecords~3,300members +sub-accountsgap check vslistingresumable crawl · output verified against the public listing
Protected sources

Protected Directory Collector

About 3,300 member and sub-account records from a Cloudflare-protected directory, with a three-step access ladder and a resumable crawl.

TLS fingerprintingCloudflareresumable
Read the case study
clientsHTTP / SOCKSHAProxyadmin socket · livedrain → rotate → addLTE modem 1serving · new carrier IPLTE modem 2serving · new carrier IPLTE modem 3rebooting · route restoreLTE modem 4serving · new carrier IPRaspberry Pi · Go scheduler rotates one modem at a time
Infrastructure

Mobile Proxy Farm

LTE dongles on a Raspberry Pi behind HAProxy, with ordered recovery after each reboot and a scheduler that rotates one modem at a time.

GoHAProxypolicy routing
Read the case study
account 1account 2account 3account 4account 5booking flowdate · section prioritypaired seatsverification steppayment form filledstops herea person confirmsparallel sessions · Telegram status updates · human hand-off at payment
Protected sources

Protected Booking Automation

Five accounts in parallel through login, date and section priority, paired seat selection, and a verification challenge, stopping at payment for a human to confirm.

Seleniumseat selectionhuman handoff
Read the case study
Next step

Need records out of a portal like this?

Send the portal and the search you'd run by hand. I'll check whether it can be exported with plain requests and reply within two working days.