mojibake.wtf
A test suite and demo for web scrapers and HTML parsers.
The … cases are HTTP responses that test how browsers and scrapers decode and parse a page. Most make at least one stack disagree with Chrome. A few are controls, kept because the divergence they were written for didn't reproduce. The site serves each one byte for byte, with the selector to apply and what Chrome extracted. Fetch them with your stack and compare.
GET /cases.json- Every case: its URL, selector,
Content-Type, fingerprint, Chrome's answer, and the answers of … measured stacks. GET /c/<id>- The response itself. The body and the
Content-Typeheader are exactly the ones measured, andX-Mojibake-Shacarries the fingerprint.
import requests
from parsel import Selector
site = "https://mojibake.wtf"
cases = requests.get(site + "/cases.json").json()["cases"]
for case in cases:
r = requests.get(site + case["url"])
got = Selector(text=r.text).css(case["selector"]).getall()
want = case["browser"]["values"]
print("ok " if got == want else "FAIL", case["id"])
Example: /c/fake-price
A product page whose price comes out differently in Chrome and in Scrapy on an older w3lib. Both report no error.
| Chrome renders | … |
|---|---|
…, after re.sub |
…
re.sub(r"[^\d,]", "", _) on what response.css(".price::text").get() returned, …
|
The laptop has not become cheaper.
…
scrapy shell https://mojibake.wtf/c/fake-price
response.css(".price::text").get()
import re; re.sub(r"[^\d,]", "", _)
……
Why
More cases
Six more cases. Most common other answer
is the answer given most often by the … current scraping stacks when they disagreed with Chrome.
| Case | Chrome extracted | Most common other answer | Scrapers agreeing |
|---|
Driving a real browser
| Stack | Engine | How it reads the page | Extracted |
|---|
Agreement with the Chrome reference on this corpus
| # | Stack | Language | How it reads the page | Agreed | Share |
|---|
How the cases work
Every case is at /c/<id>. Byte for byte means the response body and the Content-Type header are exactly the ones measured, and the X-Mojibake-Sha header carries the case's fingerprint. /cases.json lists every case with its selector and what Chrome extracted. Selectors use parsel's meaning, where E::text is the text nodes directly inside E.
The test at the top of the page measures Requests' decoding and Parsel's parsing together. r.text is already decoded by Requests, so it doesn't isolate Parsel's own byte decoding. To test a parser on its own, hand it r.content and the Content-Type header.