fcascan

Extracts the UK FCA Financial Services Register into a database, enriches it with Companies House data, and serves a small lofigui site to navigate the result. Built in the gobank style: one Go module, PostgreSQL in production, pglike (SQLite with PostgreSQL syntax) for development and tests.
Documentation https://fcascan.docs.bytestone.uk/
Source (NAS Forgejo, LAN only) https://git.lan.drummonds.net/hum3/fcascan

Stage 1: the extractor

The register API is a per-record lookup service: Firm/{FRN} plus sub-resources such as Names, Address, Individuals, Permissions and AR (appointed representatives). It has no "list every firm" call and is limited to 50 requests per 10 seconds. The only complete, free enumeration is therefore to probe every Firm Reference Number: six-digit FRNs start at 100000 and seven-digit ones (issued since July 2023) at 1000000, so the range is about one million numbers.

fcascan turns that into a work queue in the database (scan_item):

  • seed enqueues one probe per FRN (or the FRNs from a file or a search).
  • run drains the queue under a token-bucket rate limiter. Each fetch is committed in its own transaction together with its queue-row update, so you can stop the process at any moment and run again to resume. A firm found enqueues its sub-resources; an FRN with no firm is marked absent and costs nothing further.
  • Every response is stored verbatim as JSON (fca_firm.raw, fca_firm_resource.raw). A few columns are parsed out for navigation and for joining to Companies House (name, status, business type, Companies House number, names, addresses). reparse rebuilds those from the stored JSON without touching the API, so the parsers can improve later.
  • refresh --older-than 720h re-queues items fetched more than 30 days ago; refresh --absent also re-probes empty FRNs to pick up newly issued ones.
  • Failed fetches back off (attempts² minutes) and give up after five tries; refresh --reset-errors puts them back.

How long it takes

At the default 45 requests per 10 seconds (leaving headroom under the limit), using the 64% hit rate measured in the seven-digit range by the test pass (the six-digit range, allocated over 25 years, is likely sparser):

Work Calls Time
Probe 100000–1100000 1,000,001 ~2.6 days
Sub-resources, 5 per firm, at a 64% hit rate ~3.2M ~8.2 days
Full first pass, all resources ~4.2M ~11 days
Full first pass, --resources Names,Address ~2.3M ~6 days
Probe only (firm record, no resources) 1.0M ~2.6 days

The dashboard and status use the measured hit rate for the ETA as soon as probing starts. fcascan estimate --hit-rate 0.3 --resources 2 runs the sums for other assumptions. Ways to shorten a first pass:

  • --resources Names,Address (the two needed for Companies House matching) instead of the default five.
  • --details-status Authorised,Registered to skip sub-resources for firms that are no longer authorised.
  • seed --search "bank" or seed --file frns.txt to start from a known subset and probe the whole range later.

A monthly refresh of the firm records alone is one call per firm. If the register holds around 300,000 firm records past and present, that is ~19 hours; refreshing only currently authorised firms is less.

The test pass

Before committing days of API time, a 30-minute pass over the seven-digit FRNs starting at 1000000 answers the questions the estimate depends on. These are the newest firms (FRNs issued since July 2023), so the window is dense and the sample is current.

task test:pass                       # or, by hand:
./fcascan init
./fcascan seed --from 1000000 --to 1020000
./fcascan run --frn-from 1000000 --frn-to 1020000 --duration 30m
./fcascan status --frn-from 1000000 --frn-to 1020000
./fcascan fields

--frn-from/--frn-to confine the run to that window of the queue, so a full-range seed can sit alongside untouched. --duration stops the run cleanly; everything fetched is committed and a rerun resumes at the next FRN.

What the pass should tell you (at 45 requests per 10 s, about 8,100 calls):

Question Where to look What to expect
Real throughput status: last 10 min items/s; run log req/s close to 4.5/s; lower means latency-bound, raise --workers
Hit rate in the 7-digit range status: hit rate replaces the 15% prior in every ETA
Probes vs resources status: found/absent vs resources done ~4,600 probes and ~700 firms if the hit rate is 15%
Exact wire format fields: every JSON key, count, example confirms or corrects the parsers; then reparse
Error behaviour status: deferred/errors; /queue page zero 429s at 45/10 s; any 5xx retried and deferred
Data quality serve → Firms → a firm page Companies House number, names, addresses populated

The pass is repeatable: run it again and it continues from the first unprobed FRN in the window. Set PASS_MINUTES, PASS_FROM or PASS_TO to vary it (task test:pass PASS_MINUTES=10).

Results of the first pass (2026-09-19, FRNs 1000000–1001936)

Measure Result
Throughput 4.50 req/s for the whole 30 minutes, no 429s
API calls 8,103
FRNs probed 1,936: 1,232 firms, 704 absent
Hit rate 63.6% in the seven-digit range (the prior was 15%)
Resources fetched 6,160 (Names and Address for every firm; Individuals 29%; AR 0%)

What it changed:

  • Sub-resources dominate. At a 64% hit rate each FRN costs about 4.2 calls, not 1.75. The estimates below use the measured figure.
  • Permissions is an object, not an array. The API returns Data keyed by activity name; the decoder now treats that as one record and a fca_firm_permission table holds one row per activity with its limitations.
  • Many seven-digit FRNs are sole traders with an empty Companies House number, so stage 2 will need name matching, not just number joins.
  • The wire format otherwise matched the parsers: "Organisation Name", "Companies House Number", "Current Names[].Name", "Address Line 1" (and the API's own typo "Address LIne 3", absorbed by the case-insensitive lookup).
  • A run stopped by --duration used to record the in-flight batch as errors; it now stops cleanly. refresh --clear-deferred lifts any leftover backoff.

Usage

go build -o fcascan ./cmd/fcascan     # or: task build

export FCA_API_EMAIL=you@example.com  # from https://register.fca.org.uk/Developer/s/
export FCA_API_KEY=...
export FCASCAN_DSN=postgres://user:pass@host/fcascan   # omit for ./fcascan.db (SQLite, WAL)

./fcascan init                        # create schema
./fcascan estimate                    # what a full range costs
./fcascan seed                        # queue FRNs 100000..1100000
./fcascan run                         # Ctrl-C to stop; run again to resume
./fcascan status                      # counts, throughput, ETA
./fcascan serve                       # http://localhost:8090 dashboard and firm browser
./fcascan refresh --older-than 720h   # re-queue stale items, then run again

run flags: --rate (requests per 10 s), --workers, --resources, --details-status, --max-attempts, --once, --frn-from/--frn-to (window), --duration (time limit).

Running the extractor and the site together

They are separate processes sharing the database. With the default SQLite file in WAL mode the site's reads never block the extractor's writes; with Postgres this is ordinary. A multi-day run is best left in the background:

task run:bg -- --resources Names,Address   # nohup, log in run.log, tokens via tp secrets
task serve:bg                              # site on :8090, log in serve.log
task ps                                    # what is running
task run:log                               # follow progress
task run:stop                              # SIGINT: finishes in-flight items, commits, exits
task serve:stop

The dashboard's status tag reads "Extracting" while items have been committed in the last ten minutes and "Idle" otherwise. tp secrets fetches the tokens once at start, so the run does not depend on the Bitwarden session staying unlocked.

With the FCA credentials stored in Bitwarden and mapped in ~/.config/bw-session/items.conf, tp secrets ./fcascan run supplies them without putting keys in the environment.

Stage 2: Companies House enrichment

The same queue design (ch_item, one transaction per fetch) against the Companies House REST API, which allows 600 requests per five minutes (the client uses 580).

export CH_API_KEY=...            # or map it in items.conf and use tp secrets
./fcascan ch seed                # companies for every firm with a Companies House number
./fcascan ch seed --search       # also a name search for firms without one (1 call each)
./fcascan ch run                 # profiles → officers + PSCs → people matching → appointments
./fcascan ch status

What it builds:

  • Firm → company (ch_match): by the Companies House number on the FCA record (method number), or by a company search whose normalised name equals the firm's registered or current trading name (name, active company preferred). Firms with no confident match keep the search candidates for review (none).
  • Company profile, officers, PSCs (ch_company, ch_officer, ch_psc) for every matched company.
  • People → officers (ch_person_match): each FCA individual at a firm is compared by name with the officers of the matched company. Companies House gives "SURNAME, Forenames", the FCA "Forenames Surname"; both are normalised. Score 1.0 when surname and all forenames agree, 0.85 when surname and first forename agree, 0.7 on a matching initial. The default threshold is 0.85 (--min-score).
  • Appointments (ch_appointment): every appointment, at any company, of each matched person. This is what links an FCA-approved person to their other directorships.

Cost: two calls per matched company (profile and officers) plus one for PSCs, one per matched person for appointments, one per firm searched by name. At 580 per five minutes that is about 7,000 calls an hour, so 10,000 matched companies with their people take roughly six hours.

Getting a Companies House API key

  1. Register at the Companies House Developer Hub (a free account).
  2. Under Manage applications, create an application for the live environment and add a REST API key to it.
  3. The key is sent as the basic-auth user name; there is no email. Export it as CH_API_KEY, or store it in Bitwarden and map CH_API_KEY in ~/.config/bw-session/items.conf so tp secrets supplies it.

The site shows the match on each firm page (profile, officers with links to the matched FCA individuals, PSCs), the officer match and appointments on each individual page, and has /companies, /company/{number} and /officer/{id} pages.

Layout

fca/     rate-limited FCA API client, envelope decoding, forgiving field lookup
ch/      rate-limited Companies House API client (basic auth, paging)
db/      Open (pgx or pglike), schema.sql, migration
scan/    stage 1: Store (queue, one-transaction writes, refresh, reparse,
         counts/ETA), Runner (worker pool), parsers
enrich/  stage 2: Store (ch_item queue, matching, one-transaction writes),
         Runner, parsers, name normalisation
web/     lofigui site: dashboard with filters and facets, firms, individuals,
         companies, officers, queue, DB explorer, about
cmd/fcascan/  the CLI

DB Explorer

The site's DB Explorer page is gobank's, via go-dbexplorer: every table with row counts, then per-table schema, indexes and a paginated, sortable, filterable browser (/explorer/fca_firm?sort=frn&dir=desc, /explorer/fca_firm_permission?filter=frn&value=1000005). It works on both backends. go-dbexplorer is not yet published, so go.mod has a replace pointing at ../../../git/hum3/go-dbexplorer, as gobank does.

Development

task check        # fmt, vet, test (tests use pglike in memory)
FCASCAN_TEST_DSN=postgres:///fcascan_test task test   # same tests on Postgres
task serve        # dashboard against ./fcascan.db

Roadmap

See ROADMAP.md: stage 2 is Companies House enrichment, stage 3 the navigation site.

License

MIT