Websites change without telling you. A designer renames a CSS class. Someone in marketing adds a crossed-out “was $39” price next to the real one. A developer swaps two columns. Meanwhile your collector keeps running and keeps returning rows. The only open question is whether you find out before your client does.
This article is about the dull, valuable part of a scraping project: noticing when something has quietly gone wrong.
Three ways to find a field, three ways to break
- By position (“the second item in the card”). Fastest to write, and the first to break: add, remove or reorder anything and every value after it shifts.
- By class name (“the element called
price“). Reads well. It breaks when the class is renamed, which happens in every redesign and on some Tuesdays. It breaks worse when the class is reused for something else. - By label (“the value next to the word Price”). Usually the sturdiest, because words change less often than markup. It breaks when the words change (“Price” becomes “Now”).
No method is immune. I lean towards labels where a page has them (it’s how I kept output rows aligned in the property-tax and metrics projects), but the real protection is what comes next.
Checking the shape isn’t enough
A shape check asks whether a value at least looks right: does the price have a currency symbol and digits, is the rating a number from 0 to 5? That catches the obvious breakage, such as price and rating swapping places.
It doesn’t catch the sneaky kind. Suppose a redesign adds a crossed-out “was” price and reuses the class name price for it. A collector asking for “the first element called price” now returns $39.00. That’s a perfectly respectable price. It’s also wrong, and nothing about it looks wrong.
Canaries: a record you already know the answer to
The fix is old and cheap. On every run, collect one reference item whose true values you know, and compare. If the collector reads it differently, something changed, whatever the shape check says. (The name comes from the coal-mine bird, minus the sad ending. This one is there to be cross-examined, not to suffer.)
Choose the reference carefully. A price is a poor canary because prices change for good reasons. Better: a fixed directory entry, a catalogue item that never changes, or a test record you control. I recommend this technique for any collector that has to stay correct for a long time.
Try it: change the website
Here’s a made-up product page and three ways of reading it. Switch on some changes and see which collector survives, which is stopped by the shape check, and which one keeps delivering wrong data that looks fine.
Simulation. A made-up product page. Nothing real is contacted.
Make failure loud, and cheap to fix
- Validate every record: required fields present, values in range, formats plausible.
- Reconcile the total against what the source says it holds, and explain any difference.
- Run a canary on every run, as above.
- Alert on thresholds, not blips. For example: more than a small percentage of records failing validation, or the canary mismatching, stops the run and tells someone.
- Save the raw page for every failure, so a fix takes minutes instead of an investigation.
- Report plainly. “Collected 4,812 of 4,830; 18 flagged: price missing” beats a green tick every time.
Maintenance is part of the job
Websites change, and a source that changes often costs more to keep working than one that doesn’t. That’s normal, and it’s better to plan for than to be surprised by. I’d rather tell you on Tuesday that the collector needs a small tweak than have you discover it in a board meeting. If a collector matters to your business, monitoring and maintenance is worth deciding up front, and every handover I do comes with a short runbook so you aren’t dependent on me remembering how it works.