Getting text out of a web page is usually the easy part of a scraping project. The harder part is that websites display values for humans, not for spreadsheets. The same catalogue can show $1,299.00, 1.049,50 € and £89; a rating can be four star characters or “4.5 out of 5”; stock can be “In stock”, “Only 2 left!” or “Pre-order”. If those strings go straight into a CSV, the client cannot sort, filter or compare anything.
Here is the normalisation layer I put between the parser and the output in most extraction projects. The Python below, the Page to data demo and the interactive tool at the end of this article are separate implementations of the same rules, checked against one shared set of edge-case tests. The code here is simplified to show the idea; the point is the decisions, not the regexes.
Prices: decide the decimal separator, don’t guess the country
Europe often writes 1.049,50 and the US writes 1,049.50. Looking up the country is fragile. Looking at the string itself is reliable enough for prices: if the last separator is a comma followed by exactly two digits, the comma is the decimal point.
# An explicit 3-letter code wins. A bare "$" is shared by USD, CAD, AUD and more,
# so flag it rather than silently calling it USD.
CODES = {"USD", "CAD", "AUD", "NZD", "SGD", "EUR", "GBP", "INR", "BDT", "JPY"}
SYMBOLS = {"€": "EUR", "£": "GBP", "₹": "INR", "৳": "BDT", "¥": "JPY"}
def parse_price(text):
if not text:
return None, None
# "$12–$15" is a range, not the number 1215. Detect two amounts first.
if re.search(r"ds*(?:–|—|-|to)s*D*d", text):
return "range", None
code = next((c for c in CODES if re.search(rf"b{c}b", text, re.I)), None)
symbol = next((cur for sym, cur in SYMBOLS.items() if sym in text), None)
currency = code or symbol or ("$?" if "$" in text else None) # "$?" = unconfirmed dollar
digits = re.sub(r"[^d.,]", "", text)
last_comma, last_dot = digits.rfind(","), digits.rfind(".")
if last_comma > last_dot and len(digits) - last_comma - 1 == 2:
digits = digits.replace(".", "").replace(",", ".") # 1.049,50 -> 1049.50
else:
digits = digits.replace(",", "") # 1,299.00 -> 1299.00
return (round(float(digits), 2) if digits else None), currency
Two decisions matter more than the regex. First, CAD $99 is Canadian dollars, not US: an explicit code always beats a bare symbol, and a lone $ is kept as “unconfirmed” so nobody mistakes it for USD. Second, $12–$15 is a range, not 1215; it is caught before the number parser runs. Keep the currency as its own column, and never convert during scraping, which would bake in one day’s exchange rate.
Abbreviated numbers: 1.2K, 3M and “n/a”
SEO and social metrics are often shown as 1.2K or 3M. In a domain-metrics enrichment job for a client, these came back mixed with blanks and placeholder text, so the parser maps suffixes to multipliers and turns placeholders into empty values instead of zeros:
SUFFIX = {"K": 1_000, "M": 1_000_000, "B": 1_000_000_000}
def parse_count(text):
t = (text or "").strip().upper().replace(",", "")
if t in {"", "N/A", "ND", "-", "—"}:
return None # unknown, not zero
if t[-1] in SUFFIX:
return int(float(t[:-1]) * SUFFIX[t[-1]])
return int(float(t))
The difference between None and 0 matters. “We don’t know this site’s traffic” and “this site has no traffic” lead to different business decisions.
Ratings and stock: map display text to a small vocabulary
def parse_rating(text):
if not text:
return None
stars = text.count("★")
if stars:
return stars
m = re.search(r"([d.]+)s*out ofs*5", text, re.I)
return float(m.group(1)) if m else None
def parse_stock(text):
t = (text or "").lower()
qty = re.search(r"d+", t)
# Negatives first. "Not in stock" contains "in stock"; order decides the answer.
if re.search(r"out of stock|sold out|unavailable|not in stock|not available", t):
return "out_of_stock", 0
if re.search(r"pre-?order|back ?order|backordered", t): # orderable, delayed delivery
return "preorder", None
if "only" in t or "low" in t: return "low_stock", int(qty.group()) if qty else None
if re.search(r"bin stockb|bavailableb", t): return "in_stock", None
return None, None
Order is the whole trick here. Not in stock contains the substring “in stock”, so a positive check that runs first marks a sold-out product as available. Checking the negatives first fixes it. A small, fixed vocabulary (in_stock, low_stock, out_of_stock, preorder) is what makes alerts and dashboards possible later. Note that backorder and pre-order usually mean orderable, with delayed delivery (not out of stock), so they get their own status rather than being lumped in. Unknown wording returns None, which a data-quality check catches, rather than an invented status.
Dates: be explicit about day-first or month-first
03/04/2024 can be the 4th of March or the 3rd of April depending on where the data came from; the order varies by source, not neatly by geography. There is no safe automatic answer, so the date order is a setting decided per source, confirmed once by finding a date whose first number is above 12. And a parser must reject dates that cannot exist: 2024-02-30 is not a real day, and storing it is worse than storing nothing.
from calendar import monthrange
def valid(y, mo, d):
return 1 <= mo <= 12 and 1 <= d <= monthrange(y, mo)[1] # handles leap years
def parse_date(text, day_first):
t = text.strip()
if m := re.match(r"(d{4})-(d{1,2})-(d{1,2})$", t):
y, mo, d = map(int, m.groups())
elif m := re.match(r"(d{1,2})[/.-](d{1,2})[/.-](d{4})$", t):
a, b, y = map(int, m.groups())
d, mo = (a, b) if day_first else (b, a)
else:
return None # unknown format: flag, don't guess
if not valid(y, mo, d):
return None # 2024-02-30 and 2023-02-29 are rejected here
return f"{y}-{mo:02d}-{d:02d}"
A strict parser such as Python's dateutil raises an error on 2024-02-30, which is the safe behaviour. The trap is a permissive custom parser, or a lenient date mode, that silently accepts or shifts an impossible date. That is what an explicit calendar check prevents.
Deduplicate on a stable key, after normalising
Sponsored listings and "featured" blocks often repeat a product that also appears in the normal results. Deduplicate after normalisation and on a stable identifier such as a SKU or listing ID, never on the raw title: "Stoneware mug (set of 4)" and "Stoneware Mug (Set of 4)" are the same product.
Keep the raw value too
On real projects I usually keep the original text in a price_raw column alongside the parsed number, at least during the first deliveries. When a client asks why a value looks wrong, the raw text answers the question in seconds, and it makes improving the parser safe.
You can try all of this on a deliberately messy sample catalogue in the Page to data demo, and switch normalisation off to see what you would get without it. For your own spreadsheets, the CSV cleaner applies similar rules to names, emails, phone numbers and dates.
Try it: normalise a value
Type any value you might scrape. The rules from this article run in your browser. When a value is genuinely ambiguous, the tool says so instead of guessing.
Examples: