The WindfallIntelligence crawler
What it fetches, how it identifies itself, and how to allow or block it.
Windfall Intelligence Ltd publishes a daily research brief for UK private bankers, built from public records. One part of that work reads the news pages that professional advisers publish about the transactions they have acted on, so that a sale recorded at Companies House can be read alongside the adviser’s own announcement of it. This page describes the crawler that does the reading.
How it identifies itself
Every request carries the user agent WindfallIntelligence/1.0 (research; contact hello@joinwindfall.com). The agent token for robots.txt is WindfallIntelligence. Requests are also signed under Web Bot Auth (HTTP Message Signatures, RFC 9421, Ed25519), and the public key is published at /.well-known/http-message-signatures-directory on this host, so a site can verify that a request claiming to be ours is ours. The crawler is submitted to Cloudflare’s Verified Bots programme under the same name; this sentence will say “listed” once it is.
What it fetches, and how often
Only the newsroom, press and deal pages of professional-services firms (law firms, accountants, corporate-finance advisers), found through the site’s own sitemap. It reads HTML and XML only. It never submits forms, never logs in, never fetches files behind a paywall, and never reads a page that robots.txt has disallowed. It waits at least two seconds between requests to the same host, or longer if robots.txt states a crawl-delay, and it reads a bounded number of pages a night. The text it reads is used to extract facts about a transaction (the parties, the advisers, the date) with a link back to the page; it is never republished.
How to allow or block it
The crawler reads your robots.txt before anything else and honours a group addressed to WindfallIntelligence before the general group. To block it entirely:
User-agent: WindfallIntelligence Disallow: /
A site behind Cloudflare that blocks unverified bots can allow this one by name in its bot settings without changing anything else. A 403 on robots.txt itself is treated as a refusal and the host is not fetched again. If you would rather write to a person, the address below reaches one, and a request to stop is honoured the same day without argument.
Contact
hello@joinwindfall.com. Operated by Windfall Intelligence Ltd, registered in England and Wales. What the company does with public records more broadly is set out on the provenance page.