Rank 94 at the Amazon ML Challenge, or: how not to compare 6.7 trillion pairs

The Amazon ML Challenge 2026 handed 19,900+ registrants 72 hours and a deceptively polite task: entity resolution. Take a clean list of businesses, take two noisy sources full of their misspelt cousins, and decide which records are secretly the same business. Our team, Team Businessmen, finished 94th. (Yes, the team name was a choice. No, I will not be taking questions.)

• rank

  94

• registrations

  19,900+

• LB F0.5

  0.9897

• records

  10M

"Acme Plumbing LLC, 120 Main Street" turns up in one source as "ACME PLUMBING, 120 Main St". Easy. It also turns up as "Acme Plumbing, 126 Main St", which is a different business wearing the same name. Not easy. Now multiply that by three countries, several scripts, and every creative way humanity has found to abbreviate "Street".

The metric was F0.5, averaged per business. F0.5 values precision twice as much as recall, which is a polite way of saying that a wrong match hurts far more than a missed one. Almost every decision below is just that sentence, taken seriously.

The obvious approach compares every noisy record with every clean business. That comes to roughly 6.7 trillion pairs, at which point even an optimistic GPU files a complaint. So the pipeline is retrieve, then classify: cheap searches pull a shortlist for each record, and the expensive model only ever sees the shortlist.

The catch is that retrieval quietly sets your ceiling. A true pair that no search finds is gone for good, and no amount of downstream brilliance will resurrect it. So I tracked candidate recall after every stage, with the energy of a nervous accountant:

• Searching on name words alone kept about 85% of the true pairs. Respectable, and nowhere near enough.
• A union of several searches (sparse TF-IDF, dense embeddings, transliterated names, name-only) kept 99.45%.
• Trimming each record to its best 3 candidates (6 for the awkward ones) still kept 99.15%, at 2.2 pairs per record.

Net result: 6.7 trillion pairs became about 23 million, which a gradient-boosted model can chew through without drama.

The pair scorer is a LightGBM model. Text similarity matters, obviously, but several of the strongest signals are not text at all:

• How the house numbers relate: identical, one digit off, or a few doors down.
• How many clean businesses share this address or this name. A crowded address makes a name match far less trustworthy.
• How the pair ranks against the other candidates for the same record.

Gradient boosting is gloriously unsentimental about mixing signals like these. It trains in a couple of hours on millions of rows, and its probabilities calibrate cleanly, which the final step depends on.

The reflexive last step is "keep everything above 0.5". Under F0.5, that is the wrong question asked with great confidence. Whether a 0.7 candidate deserves a place depends on what else that business already has. With a near-certain first match, adding a likely second one raises the expected score. With nothing strong, it mostly buys you a false positive.

So for each business we kept the set of candidates with the highest expected F0.5, computed from the calibrated probabilities. Out of fold it beat four threshold rules, from one global cut-off to per-country ones. Not by a landslide. But leaderboards are settled in the fourth decimal place, and so, it turns out, was my sleep schedule.

Competitions have a rare talent for making you believe your own nonsense. The usual culprit is leakage: the model looks brilliant because it has quietly peeked at the answers. Two rules kept us out of that ditch:

• Split by business, not by row. Every clean business got exactly one role (training, tuning, or a 15% holdout), decided by its ID. All of its candidate pairs inherit that role, so no business is ever half in training and half in the exam hall.
• The holdout is read once. Calibration, the decision rule and early stopping were all tuned on out-of-fold predictions. The holdout existed purely to tell us the truth at the end.

The holdout (US and India) scored 0.991 macro F0.5, and the final leaderboard, France and all, said 0.9897. Numbers that agree are the most boring thing in machine learning, and the most reassuring.

• In a retrieve-then-rank system, measure recall at every stage. The classifier cannot fix what retrieval threw away.
• Read the metric before you design the model. F0.5 rewrote our decision step from scratch.
• Most late gains came from error analysis, not shinier models. Our most common mistake was one record scoring high against two businesses at the same address, and fixing that pattern beat any hyperparameter search I could have run.