1. Overview

This dataset provides longitudinal financial data — total income, total expenditure, and detailed breakdowns — for registered charities across the United Kingdom. It was produced by the UK Third and Civil Society Sector Database project, which collects, processes, and links public administrative data on civil society organisations across the United Kingdom.

The data is drawn from the annual returns and financial reports filed by charities with their respective regulators: the Charity Commission for England and Wales (CCEW), the Office of the Scottish Charity Regulator (OSCR), and the Charity Commission for Northern Ireland (CCNI). These regulatory filings are publicly available and provide a consistent, official record of charity finances over time.

The dataset consolidates these three sources into a single standardised format, covering over 308,000 charities and spanning financial years from 2000 to 2026. Each record represents one charity’s financial return for a single financial year, enabling longitudinal analysis of the UK charity sector’s financial health at scale.

2. What are Charity Financial Records?

Registered charities in the United Kingdom are required by law to file annual financial information with the charity regulator in their jurisdiction. The nature and detail of these filings varies by regulator and by charity size, but all returns include at minimum the charity’s total income and total expenditure for a given financial year.

England and Wales (CCEW)

Charities registered with the Charity Commission for England and Wales submit annual returns that include headline financial data (Part A) and, for charities with income exceeding £500,000, a detailed financial breakdown (Part B). Part B provides granular information on sources of income (donations, trading, investments, etc.) and categories of expenditure (raising funds, charitable activities, governance, etc.). The source data is available from the CCEW full register download.

Scotland (OSCR)

Charities registered with the Office of the Scottish Charity Regulator submit annual monitoring returns. OSCR publishes a rolling five-year extract of the Scottish Charity Register that includes total income and total expenditure, as well as a subset of detailed income and expenditure categories (donations and legacies, charitable activities, other trading activities, investments, other income; raising funds spending, charitable activities spending, and other spending). The source data is available from the OSCR register download.

Northern Ireland (CCNI)

Charities registered with the Charity Commission for Northern Ireland submit annual reports that include headline financial figures. Some CCNI records include detailed income and expenditure categories similar to the CCEW Part B data. CCNI publishes a current snapshot of the register and historical snapshots are archived periodically. The source data is available from the CCNI charity search.

3. Dataset Contents

The dataset contains 4,059,863 records across 23 fields. Each row represents one charity’s financial return for a single financial year. The fields are organised into four groups: identifiers, aggregate financials, detailed breakdowns, and metadata.

Identifiers and Time Period

Field	Description	Type	Coverage
uid	Unique organisation identifier (e.g., GB-CHC-1000000, GB-SC-SC012345, GB-NIC-100001)	Text	100%
fy	Financial year (integer, based on year in which the financial period ends)	Numeric	100%
fys	Financial year start date (format: ddmonyyyy, e.g., 06apr2023)	Date	99.9%
fye	Financial year end date (format: ddmonyyyy, e.g., 05apr2024)	Date	100%

Aggregate Financials

Field	Description	Type	Coverage
inc	Total gross income (£)	Numeric	100%
exp	Total gross expenditure (£)	Numeric	100%

Income Breakdown

Field	Description	Type	Coverage
incomefromdonationsandlegacies	Income from donations and legacies	Numeric	6.1%
incomefromothertradingactivities	Income from other trading activities	Numeric	6.1%
incomefromcharitableactivities	Income from charitable activities	Numeric	6.1%
incomefrominvestments	Income from investments	Numeric	6.1%
incomefromother	Other income not classified above	Numeric	6.1%
incomefromlegacies	Income from legacies (subset of donations and legacies)	Numeric	5.0%
incomefromendowments	Income from endowments	Numeric	5.0%

Expenditure Breakdown

Field	Description	Type	Coverage
expenditureonraisingfunds	Expenditure on raising funds	Numeric	3.9%
expenditureoncharitableactivitie	Expenditure on charitable activities	Numeric	6.1%
expenditureonother	Other expenditure not classified above	Numeric	6.1%
expenditureoninvestment	Expenditure on investment management	Numeric	5.0%
expenditureongrants	Expenditure on grants to institutions	Numeric	5.0%
expenditureongovernance	Expenditure on governance	Numeric	5.1%
expenditureonsupportcosts	Expenditure on support costs	Numeric	5.0%
expenditureondepreciation	Expenditure on depreciation	Numeric	5.0%

Metadata

Field	Description	Type	Coverage
source	Source regulator (ccew, oscr, or ccni)	Text	99.8%
extract	Extraction batch identifier (e.g., ccew sep2024, oscr 27jan2026)	Text	100%

4. Coverage & Completeness

Core Field Coverage

The core fields — uid, fy, fye, inc, and exp — are present in virtually all records. These fields are required for inclusion in the dataset; records missing a valid charity number, financial year end date, or income value are excluded during processing.

Jurisdiction Breakdown

The dataset draws from all three UK charity regulators. England and Wales accounts for the large majority of records, reflecting the greater number of charities registered with CCEW.

Jurisdiction	Regulator	Records	Organisations	Share
England & Wales	CCEW	3,619,195	264,390	89.1%
Scotland	OSCR	400,144	36,103	9.9%
Northern Ireland	CCNI	40,524	8,214	1.0%
Total		4,059,863	308,707	100%

Year-by-Year Coverage

The dataset covers financial years from 2000 to 2025, with peak coverage between 2015 and 2022 (170,000–181,000 records per year). Coverage in recent years declines due to reporting lag — charities have up to 10 months after their financial year end to file their annual return, so the most recent years will always be incomplete at any given extraction date.

Financial Year	Records	Financial Year	Records
2000	123,613	2013	168,589
2001	115,174	2014	169,604
2002	129,881	2015	170,202
2003	136,138	2016	172,237
2004	139,198	2017	174,202
2005	131,470	2018	176,541
2006	134,536	2019	177,770
2007	163,067	2020	180,826
2008	162,892	2021	180,879
2009	163,279	2022	177,502
2010	165,430	2023	168,707
2011	165,711	2024	134,338
2012	166,815	2025	75,398

Detailed Financial Data Availability

The 15 detailed income and expenditure fields are populated for approximately 5–6% of all records. Detailed breakdowns are available from:

• CCEW Part B filers — charities with annual income exceeding £500,000, which are required to submit detailed financial breakdowns as part of their annual return. This provides the fullest set of detailed fields (all 15 columns).
• OSCR — the public extract includes a subset of detailed categories under different column names: 5 income categories (donations and legacies, charitable activities, other trading activities, investments, other) and 3 expenditure categories (raising funds, charitable activities, other). The remaining detailed fields (legacies, endowments, investment management, grants, governance, support costs, depreciation) are not available from OSCR. In practice only around 10% of Scottish charity-years carry a genuine breakdown: OSCR’s export records zeros where no breakdown was supplied, and from the July 2026 release those all-zero placeholders alongside positive headline income are recorded as missing rather than kept as spurious zeros (see Limitations & Caveats).
• Some CCNI records — certain CCNI filings include detailed income and expenditure categories.

Although the detailed fields cover a small proportion of all records, the charities that do report detailed breakdowns tend to be the largest in the sector. The roughly 245,000 records with detailed data represent the most financially significant charities in England, Wales, Scotland, and Northern Ireland.

5. What Can You Learn?

This dataset enables a wide range of analyses of the UK registered charity sector. Below are some illustrative use cases, several of which have already been demonstrated using this data in published research.

Sector-Level Trends

Track aggregate income and expenditure across the entire UK charity sector over 25 years. Analyse how the sector responded to economic shocks such as the 2008 financial crisis, the period of austerity, and the COVID-19 pandemic. Identify long-term growth or contraction trends in charity finances. When combined with CPI deflators, the data supports both nominal and real-terms (inflation-adjusted) time-series analysis.

Year	Charities	Total Income (£bn)	Total Expenditure (£bn)	Median Income (£)
2005	131,470	62.0	58.1	15,834
2006	134,536	65.1	61.9	15,628
2007	163,066	87.0	78.9	15,156
2008	162,892	88.6	81.2	15,685
2009	163,279	91.4	85.9	16,505
2010	165,427	94.6	87.6	16,823
2011	165,709	92.5	86.3	17,148
2012	166,813	93.8	90.2	17,719
2013	168,580	97.2	93.4	17,915
2014	169,586	101.8	97.3	18,600
2015	170,180	107.4	102.9	19,414
2016	172,197	111.7	107.8	19,882
2017	174,107	113.3	108.2	20,251
2018	176,429	114.9	110.8	21,236
2019	177,632	118.3	116.4	21,569
2020	180,663	117.6	113.6	19,217
2021	180,727	116.7	111.7	17,865
2022	177,237	115.1	112.2	18,975
2023	168,221	116.4	114.8	23,389
2024	132,940	119.9	117.9	48,526

Source: Charity Financial Records dataset. All values in 2023 prices, adjusted using ONS CPI (series D7BT). 2023–2024 figures are provisional due to reporting lag; the sharp rise in 2024 median income reflects incomplete filing by smaller charities.

Financial Health Analysis

Identify charities with declining income over consecutive years, or those where expenditure consistently exceeds income. For example, deriving a “loss-making” indicator (where expenditure exceeds income in a given year) enables analysis of financial distress rates across regions and over time. Annual relative growth rates can be calculated to track the year-on-year trajectory of individual charities or cohorts.

Jurisdictional Comparisons

Compare financial patterns across England and Wales, Scotland, and Northern Ireland. Examine whether charities in different jurisdictions show different income levels, growth rates, or financial resilience during economic downturns.

Subsector Analysis

When linked to the TCSS Organisation Register, analyse finances by charity type, activity area, or geographic region. For example, the data can be joined with the UK Charity Activity Tags (UKCAT) ICNPTSO classification to compare median income and expenditure trends across charity sectors such as social services, health, education, culture and recreation, and environment.

Donation Dependency

For charities with detailed income breakdowns, the proportion of income from donations and legacies can be tracked over time and compared across jurisdictions. This enables analysis of how reliant different parts of the charity sector are on voluntary income, and whether that reliance has changed in response to policy or economic shifts.

Geographic Analysis

When linked to the Organisation Register’s geographic data (postcodes, local authorities, regions), financial data can be mapped at the local authority level. This supports analysis of regional variation in charity finances, the relationship between local deprivation and charity financial health, and per-capita measures of charitable activity.

Income Composition

For the larger charities with detailed breakdowns, examine how income sources are distributed — for example, the relative importance of donations, trading income, charitable activities, and investment returns. Track whether charities have diversified their income sources over time or become more dependent on a single source.

6. Limitations & Caveats

Reporting Lag

Charities file their annual returns after their financial year ends, with deadlines varying by jurisdiction (typically 10 months for CCEW). As a result, the most recent financial years in the dataset will always be incomplete at any given extraction date. The sharp decline in records for 2024 and 2025 reflects this reporting lag, not a decline in the number of active charities.

Detailed Breakdowns

The 15 detailed income and expenditure fields are only available for a subset of records (approximately 5–6%). CCEW Part B provides the fullest detail (all 15 fields), while OSCR provides a subset of 8 categories and some CCNI records include partial detail. The remaining detailed fields (e.g., legacies, endowments, governance, support costs) are only available from CCEW. In addition, a charity’s headline income and expenditure (taken from its Charity Commission Part A return) can occasionally disagree with its own detailed breakdown (taken from Part B). Earlier releases attributed this wholly to inconsistency between the two parts of the filed return. That was only partly correct: a further cause was in this dataset’s own processing. Where a charity changed its accounting year-end it filed two genuine periods ending in the same calendar year, and the breakdown was matched to the headline by year rather than by accounting period, so a headline from one set of accounts could be published beside the breakdown of another. From the July 2026 (v1.6) release the breakdown is matched by accounting period, which makes the mismatch impossible, and the 35 affected charity-years already published were corrected — 19 given the breakdown for the correct period and 16 left blank because no return for that period is available. Genuine inconsistency between a charity’s own Part A and Part B returns does still occur, and the dataset continues to reproduce both as filed. Analyses using these fields cannot be considered representative of the full sector.

Scottish Breakdown Availability

Scottish detailed breakdowns are particularly sparse, and their absence is structured rather than random. OSCR’s export records zeros — not blanks — in every breakdown field when a charity supplied no breakdown, which affects roughly 90% of Scottish charity-years. Because an all-zero breakdown alongside positive headline income is self-contradictory (the income must arise in some category), those placeholder zeros are recorded as missing from the July 2026 release onwards; genuine all-zero breakdowns for zero-income charity-years are retained.

Two structural points help interpret the gap among larger Scottish charities:

• Cross-border charities are not affected — charities registered with both OSCR and CCEW (around 1,400, including the largest charities operating in Scotland) appear in this dataset under their England-and-Wales record only, following the same deduplication used for the TCSS Organisation Register. Their financial records, including Part B breakdowns where income exceeds £500,000, are found under their GB-CHC identifier; the Register’s matches file links the two registrations. A small number — around 17 — have incomplete financial histories, either because their two registrations are not yet linked or because the charity has stopped filing with one regulator while continuing with the other; these are queued for upstream repair to the Register.
• The gap concentrates in domestic registrants — 4,635 Scottish charities in the dataset have at least one financial year with income above £250,000 but no breakdown. Most (3,605) are standard OSCR registrants; all 131 Registered Social Landlords fall in this group (their audited accounts are filed with the Scottish Housing Regulator rather than OSCR); the remainder are mostly charities absent from the current OSCR export. For these organisations the breakdown is simply not published by OSCR, and users needing it must consult the charities’ published accounts.

Analyses of the detailed fields for Scotland should therefore treat the available subset (around 10% of Scottish charity-years) as unrepresentative, particularly of the largest domestically-registered charities.

Cross-Border Double-Counting

The deduplication described above depends on the Organisation Register correctly linking a charity’s OSCR and CCEW registrations. Where that link is missing, a charity can appear twice — once under each regulator — with the same financial year reported at identical amounts under both. A quality audit in July 2026 identified 82 such charities, accounting for 188 charity-years and roughly £296 million of income counted twice. The July 2026 release resolves 59 of them (some £282 million, including the largest single case): the updated Organisation Register now links their two registrations, and the dataset removes previously captured rows under a Scottish identifier once the Register deduplicates that record under the charity’s England-and-Wales identifier. The remaining 23 charities — 69 charity-years and roughly £14 million of income counted twice — are queued for a register correction and will be removed the same way once linked. Analysts computing sector totals, or comparing Scotland with England and Wales, should be aware of this residual overlap.

The wider matching gap means a further 250–300 dual registrations exist that mostly do not yet file in both countries. Because these will be linked in the Register as they begin to file — and because each data refresh now removes previously captured rows for any Scottish record the Register has since deduplicated — the overlap is corrected upstream rather than growing unchecked.

Implausibly Low Single-Year Figures

A small number of charity-years in this dataset carry an income figure that is implausibly low — roughly a thousand times smaller than the same charity reports in its neighbouring years. These cases originate in the Office of the Scottish Charity Regulator’s own published five-year export, not in our processing: every case we were able to check was already wrong in the regulator’s source data, and we have chosen to reproduce the source faithfully rather than silently alter reported figures. The affected years number 364 charity-years across 141 charities (one further case left the dataset with the cross-border deduplication described above). We publish the full list of them alongside the data (tcss-charity-finhist-1000x-flags.csv) so that analysts can identify and handle these years themselves — for example by excluding them, treating them as missing, or checking the charity’s filings directly.

Append-Only Records and Later Corrections

The dataset is deliberately append-only: once a charity-year has been captured it is never revised, so corrections a regulator publishes afterwards are not pulled back into the existing row. This keeps the historical record stable and reproducible, but it has four consequences worth noting.

• Removal of deduplicated Scottish records — the one case where rows leave the dataset. When the Organisation Register links a Scottish registration to its England-and-Wales record, previously captured rows under the Scottish identifier are removed, because they duplicate the same charity’s CCEW-side filings (see Cross-Border Double-Counting above). In the July 2026 release this removed 425 rows across 88 Scottish records. Figures within surviving rows are still never revised.
• Scottish breakdown backfill — From the July 2026 release, a limited backfill rule fills in Scottish income and expenditure breakdown detail that OSCR published after a row was first captured, but only where the charity-year’s headline income and expenditure are unchanged and the breakdown genuinely decomposes them. In the July 2026 release 41 rows were filled this way.
• Breakdowns left blank on purpose — A further 42 rows that would otherwise qualify were deliberately left unfilled because OSCR’s breakdown fields for them carry an apparent units error — the categories sum to roughly one hundred times the agreed headline figure. These breakdowns remain blank rather than introduce a known inconsistency.
• One known headline erratum — Queens Cross Housing Association (SC036434), financial year 2025, is preserved as originally filed. OSCR’s first publication was affected by a £000s slip (recorded here as income 27,700 and spending 27,042) and was later corrected by OSCR to income 32,321,000 and spending 49,613,000. The dataset keeps the original figures; analysts using this charity-year should prefer the corrected values.

OSCR Rolling Window

OSCR publishes a rolling five-year extract of the Scottish Charity Register. Historical OSCR data beyond the most recent five-year window is only available if it was captured in a previous extraction. The dataset’s OSCR coverage for earlier years depends on which baseline snapshots were available during data collection.

Variable Financial Year Periods

Charities do not all operate on the same financial year. While many use an April–March cycle, others use calendar years (January–December) or other reporting periods. The fy field represents the calendar year in which the financial period ends, enabling comparison across charities, but users should be aware that the actual reporting periods may not align precisely.

Spine Filtering

The dataset only includes charities that appear in the TCSS Organisation Register (the “Spine”). Some recently registered or very small charities may not yet have been added to the Spine and will therefore be absent from this dataset. The exclusion log records all filtered records and the reason for their exclusion.

Currency and Inflation

All financial values are reported in nominal terms (current prices) as filed by the charity. No inflation adjustment has been applied. Users conducting time-series analysis should apply their own deflators if real-terms comparison is needed.

What is NOT in the Data

The dataset does not include: balance sheet items (assets, liabilities, reserves), staff numbers or employment data, trustee or director details, geographic location, or activity classifications. Many of these gaps can be addressed by linking to the wider data resources of the UK Third and Civil Society Sector Database, which provides organisation registers, CIC 36 forms, and procurement data for the same organisations.

7. Citation & Licence

Suggested Citation

McDonnell, D. et al. (2026). Charity Financial Records. UK Third and Civil Society Sector Database. Available at: https://uk-third-sector-database.github.io/data/. Licensed under CC BY 4.0.

If you would like to learn more about this dataset and how it can be applied to your project or research programme, please contact research@brawdata.com.

8. Changelog

Version	Date	Description
1.6	20 July 2026	Correction to the England-and-Wales detailed breakdowns, and a fuller download package. Where a charity changed its accounting year-end it files two genuine periods ending in the same calendar year; the detailed breakdown (Part B) was previously matched to the headline (Part A) by financial year rather than by accounting period, so the two could be resolved to different sets of accounts. The breakdown is now matched by accounting period, and the 35 already-published charity-years affected have been corrected: 19 now carry the breakdown for the period their headline describes, and 16 have been left blank because the Charity Commission publishes no Part B return for that period. No headline income or expenditure figure changed, no record was added or removed, and the dataset still contains 4,059,863 records across 308,707 charities. The affected charity-years are listed in the release’s repair log. The download archive now also contains this guidance (HTML and PDF), the licence, and a MANIFEST.json recording the schema version, column order, row counts and a SHA-256 checksum for every file, so a download can be verified without reference to any external record
1.5	20 July 2026	Data re-released, rebuilt against Organisation Register v1.1 (July 2026): 4,059,863 records across 308,707 charities (FY 2000–2026), with 96,127 new charity-years added over the November 2025 baseline. A new register-deduplication step now removes previously captured rows for Scottish records the Register links to an England-and-Wales record, and re-runs on every rebuild: this release removed 425 rows across 88 Scottish records, resolving 59 of the 82 audited cross-border double-counting cases (some £282 million); the remaining 23 cases (roughly £14 million) stay queued for a register correction. The Scottish breakdown backfill announced in v1.4 is applied (41 rows filled); the implausibly-low-year flag list is published alongside the data (364 charity-years across 141 charities after deduplication)
1.4	20 July 2026	Guidance update following a two-wave quality audit of the Scottish (OSCR) data. New caveats added on cross-border double-counting (82 charities dual-registered with the Charity Commission and OSCR but not yet linked, giving 188 charity-years and roughly £296 million of income counted twice, to be corrected upstream in the next release) and on implausibly low single-year income figures reproduced faithfully from OSCR’s own export (365 charity-years across 142 charities, now flagged in tcss-charity-finhist-1000x-flags.csv); notes added on the dataset’s append-only vintage, the forthcoming Scottish breakdown backfill (41 rows qualifying, 42 held back for an apparent units error), the Queens Cross Housing Association (SC036434) FY 2025 headline erratum, the internal inconsistency that can arise between CCEW Part A and Part B returns, and incomplete histories for around 17 cross-border charities; wording clarified so that unmatched Scottish charities are described as absent from the current OSCR export rather than deregistered
1.3	July 2026	Re-extracted with July 2026 data: 4,060,076 records across 308,710 charities (FY 2000–2026). OSCR all-zero breakdown placeholders alongside positive income are now recorded as missing rather than zero (14,885 rows); OSCR placeholder dates (1970/2027) excluded; consistent integer formatting and baseline column order restored; scripted source downloads and post-build validation added to the pipeline
1.2	March 2026	Re-extracted with March 2026 data: 4,025,042 records across 307,229 charities; updated OSCR (20 Mar 2026 export) and CCNI (20 Mar 2026 snapshot)
1.1	March 2026	OSCR detailed income and expenditure fields now populated (8 categories); coverage percentages updated
1.0	January 2026	Initial release: 3,998,722 records across 305,465 charities (FY 2000–2025)

A1. Pipeline Architecture

The charity financial records pipeline processes publicly available regulatory data from three UK charity regulators and consolidates it into a single standardised dataset. The pipeline is implemented in Python and is orchestrated through a command-line interface (run_pipeline.py) that supports running individual jurisdiction processors or the full workflow.

The pipeline operates in four phases:

1. Jurisdiction Processing — Each regulator’s source data is processed by a dedicated module (process_ccew.py, process_oscr.py, process_ccni.py). These modules parse the raw files, validate required fields, standardise dates and financial values, construct UIDs, and deduplicate within-source records. Each module produces a jurisdiction-specific extract CSV.
2. Combination — The three jurisdiction extracts are concatenated into a single dataset. Records are filtered against the TCSS Organisation Register (“Spine”) to retain only charities that appear in the cross-register lookup. This ensures consistency with other datasets in the TCSS ecosystem.
3. Baseline Append — New records are compared against an existing baseline dataset using a composite key of uid + fy. Only genuinely new records are appended, preventing duplicate accumulation across pipeline runs.
4. Output Validation — The final dataset is validated against the defined output schema (23 columns), sorted by uid and fy, and written to CSV.

All excluded records are logged to an exclusion log (charity-exclusion-log.csv) with a timestamp, source, reason code, and descriptive detail, providing a full audit trail of data processing decisions.

A2. Source Data

The pipeline draws on publicly available data from three regulatory sources. Each source has distinct file formats, date conventions, and field structures that are handled by dedicated processing modules.

CCEW (England & Wales)

Property	Detail
Source	CCEW Full Register Download
Format	TAB-delimited text files
Files	Part A (headline financials) + Part B (detailed breakdown, charities >£500k)
Date format	YYYY-MM-DD HH:MM:SS.NNNNNNN
UID prefix	GB-CHC-
Dedup strategy	Part A by (uid, fy) and Part B by (uid, accounting period end), each keeping the most recent by ar_received_date
Part A/Part B join	By (uid, accounting period end), so a breakdown can only describe the same accounts as the headline beside it

OSCR (Scotland)

Property	Detail
Source	OSCR Scottish Charity Register Download
Format	CSV
Files	Single 5-year rolling extract
Date format	DD/MM/YYYY
UID prefix	GB-SC-
Detailed fields	5 income categories (Donations and legacies income, Charitable activities income, Other trading activities income, Investments income, Other income) and 3 expenditure categories (Raising funds spending, Charitable activities spending, Other spending)
Note	Financial year start (fys) derived from FYE minus 1 year plus 1 day. OSCR column names differ from CCEW and are mapped to the standardised schema during processing.

CCNI (Northern Ireland)

Property	Detail
Source	CCNI Charity Search
Format	CSV (UTF-8 and Latin-1 variants)
Files	Current snapshot + historical snapshots
Date formats	DD/MM/YYYY, DD Month YYYY, YYYY-MM-DD
UID prefix	GB-NIC-
Dedup strategy	By (uid, fy), keeping most recent extract

A3. Deduplication & Validation

Data quality is maintained through a multi-stage validation and deduplication process. Each stage logs excluded records to the exclusion log with a reason code.

Required Field Validation

Every record must have:

• A valid charity registration number (used to construct the uid)
• A parseable financial year end date (used to derive fy and fye)
• A non-null income value (inc)

Records failing any of these checks are excluded with reason code missing_required or invalid_date.

Within-Source Deduplication

Some source files contain duplicate records for the same charity and financial year. The deduplication strategy varies by source:

• CCEW Part A: When multiple records exist for the same (uid, fy) pair, the record with the most recent ar_received_date is retained, as it reflects the latest submission.
• OSCR and CCNI: The first occurrence for each (uid, fy) pair is retained.

Excluded duplicates are logged with reason code duplicate_source.

Spine Filtering

After combining the three jurisdiction extracts, records are filtered against the TCSS Organisation Register (Spine). Only charities with a UID present in the Spine are retained. This ensures the dataset is consistent with other TCSS datasets and that all records can be linked using the common uid field. Excluded records are logged with reason code not_in_spine.

Baseline Deduplication

When appending new records to the existing baseline dataset, a composite key of uid + fy is used to identify records that already exist. Only genuinely new records are appended, preventing duplicate accumulation across pipeline runs. Duplicates are logged with reason code duplicate_baseline.

Register Deduplication

Before appending, previously captured rows are removed for organisations the current Organisation Register deduplicates under another record: the row’s identifier is absent from the Register’s spine file but present in its matches file as a constituent of another organisation. The removal is scoped to Scottish (GB-SC-) and Northern Irish (GB-NIC-) identifiers — the surviving record for a deduplicated cross-border charity is its England-and-Wales registration — and re-runs on every rebuild, so Register improvements propagate to the dataset automatically. Removals are logged with reason code register_dedup.

Exclusion Reason Codes

Code	Description
missing_required	Missing charity registration number, income, or expenditure
invalid_date	Could not parse financial period dates
duplicate_source	Duplicate within source file (most recent kept)
not_in_spine	Charity not found in TCSS Organisation Register
duplicate_baseline	Record already exists in baseline dataset (same uid + fy)
implausible_year	Financial year before 1990, or period end after the source snapshot date (placeholder row)
register_dedup	Organisation deduplicated under another Register record (identifier absent from spine, present in matches)

A4. Reproducibility

The pipeline code is available in the project’s repository, with the charity financial records pipeline located in the code/charities/ directory.

Scripts

Script	Purpose
run_pipeline.py	CLI entry point; orchestrates the full workflow with subcommands (ccew, oscr, ccni, combine, all, coverage)
config.py	Central configuration: paths, field mappings, output schema, exclusion reason codes
process_ccew.py	Processes CCEW Part A + Part B data
process_oscr.py	Processes OSCR 5-year rolling extract
process_ccni.py	Processes CCNI current and historical snapshots
combine_and_append.py	Combines jurisdiction extracts, validates against Spine, appends to baseline
coverage_analysis.py	Generates variable, organisation, and year-by-year coverage reports
dependencies.py	Dependency management; ensures required packages are installed

Dependencies

The pipeline requires Python 3.11 or later, with dependencies managed using uv. The only external dependency is:

• pandas — for data manipulation and CSV I/O

Running the Pipeline

The full pipeline can be run with a single command:

python run_pipeline.py all

Individual stages can also be run independently:

python run_pipeline.py ccew      # Process CCEW data only
python run_pipeline.py oscr      # Process OSCR data only
python run_pipeline.py ccni      # Process CCNI data only
python run_pipeline.py combine   # Combine and append
python run_pipeline.py coverage  # Generate coverage reports

Input Data

Running the pipeline requires the source data files to be downloaded from each regulator’s website and placed in the expected directory structure under data/input/financial-history/raw/. The baseline dataset and Spine lookup file must also be present. All input paths are defined in config.py.