API reference

The entry point

compare

compare(reference, new_data, *, match_threshold=0.5, verified_threshold=0.75, name_weight=0.4, sample_size=2000, random_state=0, exact_first=True)

Compare new data against a reference and report what changed.

Parameters:

Name	Type	Description	Default
reference		Whatever you have; the more it contains, the more can be verified. A DataFrame (or path to one) compares names AND contents. A list of expected column names (or anything with feature_names_in_, like a fitted sklearn model) compares names only, and the report says so.	required
new_data		The data to check: a DataFrame or a path readable by :func:read_table.	required
match_threshold	float	Minimum score in [0, 1] to propose a match at all; higher means fewer, more confident matches. Default 0.5.	0.5
verified_threshold	float	Minimum score in [0, 1] for a same-name match to count as verified; below it the pair is listed as suspect. Higher sends more borderline same-name matches to suspect. Default 0.75.	0.75
name_weight	float	How the score is split between names and contents: 0 judges only by the values (used automatically for headerless data), 1 only by the names. Default 0.4.	0.4
sample_size	int	Rows sampled from large files before content comparison.	2000
random_state	int	Seed for sampling, for reproducible scores.	0
exact_first	bool	Pair identically named, same-type columns immediately instead of comparing them against everything (faster, and namesakes cannot be stolen by fuzzy lookalikes). Default True.	True

Returns:

Type	Description
ComparisonReport

ComparisonReport dataclass

Everything :func:compare found, summary and evidence together.

The five category fields answer "what happened":

• verified: matched under the same name with contents that agree.
• renamed: matched confidently, but under a different name.
• suspect: the name exists on both sides but the pair does not line up (different content types, drifted contents, or each side matched something else).
• dropped: reference columns with no acceptable match.
• added: new columns nothing claimed.

print(report) shows the counts; report.summary() adds the column names. The evidence behind every decision is in report.scores and report.candidates(column).

Attributes:

Name	Type	Description
verified	list of FieldMatch	Matched under the same name with contents that agree.
renamed	list of FieldMatch	Matched confidently, but under a different name.
suspect	list of Suspect	Same name on both sides, but the pair does not line up; each entry carries a plain-language reason.
dropped	list of str	Reference columns with no acceptable match.
added	list of str	New columns nothing claimed.
suggestions	list of Candidate	For each dropped column, its closest below-threshold candidate, if any.
matches	list of FieldMatch	Every accepted match: verified, renamed, and drifted namesakes.
mapping	dict of str to str	{new_column: reference_column} for every accepted match.
scores	DataFrame	The full evidence table, one row per compared pair (see :func:field_match.matching.similarity_scores).
notes	list of str	Warnings about conditions that affect what could be verified (headerless data, duplicate names, empty columns).
reference_columns	list of str	Every column name from the reference side.
new_columns	list of str	Every column name from the new data.
row_counts	tuple of (int or None, int)	Reference row count (None for a names-only comparison) and new data row count.
match_threshold	float	The match_threshold this report was generated with.
verified_threshold	float	The verified_threshold this report was generated with.
name_only	bool	Whether the reference was column names only, so contents were not checked.

apply

apply(new_df)

Return a copy of new_df renamed to the reference's names.

candidates

candidates(column, limit=5)

Ranked candidates for one column, from the evidence table.

Looks the column up on the reference side first, then the new side, and returns its best-scoring counterparts.

rename_snippet

rename_snippet()

A reviewable rename_dict, for pasting into your own script.

Same-name matches are listed in a leading comment (verified, nothing to rename); actual renames go in the dict, in the new file's own column order so the snippet reads alongside the file. Only the dict is emitted - apply it with your own DataFrame's variable name, as shown in the trailing comment.

to_dict

to_dict()

A JSON-friendly version of the report, for logs and alerts.

summary

summary(show_columns=True, max_columns=8)

The report as readable text.

show_columns=False gives just the counts and notes; True (default) also lists what is in each category, capped at max_columns entries per category.

Suspect dataclass

A column name both datasets share whose contents do not line up.

Candidate dataclass

The closest available candidate for an otherwise unmatched column.

Conveniences

align_to_model

align_to_model(model, data, *, match_threshold=0.5, auto_apply=True)

Align data's columns to a fitted sklearn model's expected input.

Thin wrapper around :func:compare using the model's feature_names_in_ as the expected schema. Defaults to auto_apply=True (returns a ready-to-predict DataFrame, raising if any expected column has no match) since this is normally the last step before calling model.predict(). Pass auto_apply=False to get the :class:ComparisonReport and review first.

generate_column_rename

generate_column_rename(data1, data2, *, match_threshold=0.5, **kwargs)

Generate a reviewable rename_dict for data2's columns.

Convenience for compare(data1, data2).rename_snippet(): only actual renames go in the dict, same-name matches are noted in a leading comment, and entries follow data2's own column order so the snippet reads alongside the file it renames.

read_table

read_table(path, sheet_name=0, file_format=None, **kwargs)

Read a data file into a DataFrame, dispatching on file extension.

Parameters:

Name	Type	Description	Default
path	str | Path	Path to a data file. Extension-recognized formats: .csv, .tsv, .xlsx/.xls (Excel), .parquet, .json, .dta (Stata), .sav (SPSS), .sas7bdat/.xpt (SAS), .rds (R), .fwf (fixed-width text).	required
sheet_name	str | int	Sheet to read for Excel files (name or 0-based index). Ignored for other formats. Use :func:list_sheets to see what's available.	0
file_format	str | None	Override format detection - one of the values in _FILE_FORMATS ("csv", "excel", "fwf", "json", "parquet", "rds", "sas", "spss", "stata", "tsv"). Required for ambiguous extensions like .txt/.dat that don't imply a single format.	None
**kwargs		Passed through to the underlying pandas reader. For fixed-width files, pass colspecs or widths if you know the layout; otherwise pandas infers column boundaries from the first 100 rows (colspecs="infer", the default here) - works for cleanly aligned files, but worth spot-checking against a codebook. For CSV/TSV/fixed-width files, a non-UTF-8 file (common in older government data) is retried automatically with encoding="latin-1" unless you pass encoding= yourself.	{}

Raises:

Type	Description
ValueError	If the extension isn't recognized and no file_format is given, or an unrecognized file_format is passed.
ImportError	If the format needs an optional dependency that isn't installed: Excel needs openpyxl (field-match[excel]), Parquet needs pyarrow (field-match[parquet]), SPSS needs pyreadstat (field-match[spss]), R data needs pyreadr (field-match[r]). SAS needs no extra dependency.

list_sheets

list_sheets(path)

List sheet names in an Excel file. Returns [] for non-Excel files.

Machinery

match_fields

match_fields(data1, data2, *, match_threshold=0.5, name_weight=0.4, sample_size=2000, random_state=0, assignment='optimal', exact_first=True)

Find the best column matching between two DataFrames.

The raw match list with no report around it - most users want :func:field_match.compare instead. This is the home of the assignment knob, including "all", which a report cannot represent (one column, many candidates).

Parameters:

Name	Type	Description	Default
match_threshold	float	Minimum combined score in [0, 1] to accept a match; higher means fewer, more confident matches.	0.5
assignment	str	How to resolve competing matches - "optimal" (default, globally best one-to-one pairing via the Hungarian algorithm), "greedy" (faster one-to-one approximation, no scipy required), or "all" (every candidate pair scoring at or above match_threshold, no picking at all).	'optimal'
exact_first	bool	If True (default), columns whose names match exactly (ignoring case) and whose contents are the same type family are paired immediately and skip the all-pairs comparison. Much faster when the two files mostly share a schema. Ignored for assignment="all", which by definition reports every candidate pair. Set False to force the full comparison for every column.	True

Returns:

Type	Description
list of FieldMatch	Matches with score >= match_threshold, sorted by descending score.

similarity_scores

similarity_scores(data1, data2, *, name_weight=0.4, sample_size=2000, random_state=0)

Score every comparable column pair between two DataFrames.

This is the raw evidence table with no matching applied: one row per candidate pair, scored but not picked. Use it when you want scores without a full :func:field_match.compare run - checking one column against another dataset (similarity_scores(df1[["age"]], df2)), hunting near-duplicate columns within one file (similarity_scores(df, df)), or feeding custom logic. After a compare() call, the same table is available as report.scores.

Columns are compared when they belong to the same inferred type family (numeric, datetime, boolean, text). Object columns whose values overwhelmingly parse as numbers/dates/booleans are coerced first, so "2021-06-01" strings can match a datetime column.

Parameters:

Name	Type	Description	Default
data1	DataFrame	The reference DataFrame (e.g. last year's cleaned data).	required
data2	DataFrame	The new DataFrame whose columns you want to identify.	required
name_weight	float	Weight in [0, 1] given to column-name similarity; the remainder goes to column-content similarity. 0 judges only by the values, 1 only by the names.	0.4
sample_size	int	Columns longer than this are sampled before content comparison.	2000
random_state	int	Seed for sampling, for reproducible scores.	0

Returns:

Type	Description
DataFrame	One row per compared pair with columns source, target, family, name_score, content_score, score, sorted by source then descending score.

FieldMatch dataclass

A proposed match between one column in each DataFrame.

Similarity functions

name_similarity

name_similarity(name_1, name_2)

Fuzzy similarity between two column names.

Tokenizes both names (snake_case, camelCase, spaces) and greedily pairs tokens. A token pair scores 1.0 for an exact match, 0.8 when one is an ordered-subsequence abbreviation of the other (qty/quantity, hh/household), or its character ratio for near-typos. Unrelated words score 0 - incidental shared letters don't count.

numeric_similarity

numeric_similarity(series_1, series_2)

Similarity between two numeric columns.

Compares the full empirical distributions via the two-sample Kolmogorov-Smirnov statistic, both raw and median-centered so that location shifts between annual releases still match. Unlike comparing means/ranges, this is robust to outliers, negative values, and those shifts.

Note

This keys on the shape of the distribution, not on meaning: two unrelated numeric columns with similar distributions (e.g. any two roughly-normal columns on comparable scales) can score high. In compare this is mitigated by name_weight and one-to-one assignment, but a match resting on content alone (name_score near 0) is worth an eye.

datetime_similarity

datetime_similarity(series_1, series_2)

Similarity between two datetime columns.

KS distance on epoch values, both raw and median-centered - so this year's survey dates still match last year's survey_date column even though the two ranges never overlap.

boolean_similarity

boolean_similarity(series_1, series_2)

Similarity between two boolean columns.

Simply 1 - |p1 - p2| where p is the proportion of True values. Always bounded in [0, 1].

text_similarity

text_similarity(series_1, series_2, sample_size=500, random_state=0)

Similarity between two text/categorical columns.

Blends two signals:

• exact-value Jaccard overlap of the unique values (strong signal for categorical codes, IDs, state names, ...), and
• character 3-gram Jaccard of the pooled values (catches formatting drift, e.g. "VA" vs "Virginia" scores low here but free-text fields about the same topic score high).

Long columns are sampled for performance.