Your deal pipeline's domain tags are stale, and refreshing them is cheap

Post 2 of 4 on the economics of decision models: one of many.

Roofr went through Y Combinator in 2017 as a marketplace for homeowners: type in an address, get a satellite estimate for a new roof in seconds, get matched with a contractor within a day. YC filed it under Consumer, Home and Personal, and at the time that was the right call. Today Roofr sells measurements, proposals, a CRM, invoicing and payments to roofing contractors, and its own YC one-liner reads "Sales software for roofers". The free-form tags on its profile have caught up (SaaS, Construction, Proptech), but the industry field, the one a directory filters on, still says Consumer. YC assigns that field around batch time and rarely revisits it, so a company that pivots keeps its original label. Nobody got this wrong. The 2017 label is accurate for 2017. For anything that describes the market today (a market map, comps, deal sourcing) it needs refreshing, and so does every label like it. We kept Roofr in the sample on purpose, as the clearest case of a pattern that runs through it.

The funds we work with have the same problem at a different scale. On one engagement the fund's own domains are assigned zero-shot, from company descriptions, to tens of thousands of companies by a frontier language model with thinking on. That pass is expensive enough to run once in a long while, so every tag is a Roofr in the making: right on the day of the run, drifting from then on as companies pivot, add a product line or move from consumers to businesses. With a decision model the same pass costs a small fraction, so it can run on every refreshed description, as often as descriptions change, and the sourcing pipeline downstream reflects the market as it is. For a portfolio company, the dated classification in the investment memo is worth keeping as it was, so that later measurement has a fixed starting point. The live tag is the one that should move. That is the business case, and this post measures it on public data.

Post 1 asked one yes or no question about 100 shell commands. This post asks a harder kind of question on lower stakes. It asks which of seven investing domains (YC's six plus Other) each of 500 real YC companies belongs to. A decision model answers with a probability for every domain at once, and that spread tells you more than post 1's single number could.

The decision models are again TypeSafe's Jev and OpenAI's Luna (gpt-6-luna on the Decisions API). Sonnet 5.5 is the language model, and the one they hand the hard cases to. The question for this post is what that spread of probabilities buys when a wrong answer is cheap.

How a decision model picks one of many

Post 1 asked a yes or no question and got back one number. Here the question is a list, and the answer is a number for every item on it, adding up to 1. Jev's answer for Roofr, added up by domain: B2B 0.56, Real Estate and Construction 0.44.

What matters is how high the top pick is and how far it leads the runner-up. A company at 0.95 with nothing close is settled. Roofr is close to a coin toss, and the model says so.

That is what the band in this post reads. If the top pick is below 0.6, or leads the runner-up by less than 0.2, the company goes to Sonnet. The band applies those two tests to the 51 subindustries the model actually answers, which the next section shows.

The question, as code

The question is one Pydantic model, like post 1's. The comments are the fund's definitions, and the decision models read them literally. A fund would rewrite them in its own words; that file is the whole configuration. The list runs to 51 subindustries, each tagged with its domain, trimmed here to a few per domain.

class SubindustryChoice(BaseModel):
    """Assign the company to the single investing subindustry that fits it best, given the fund's definitions."""
    subindustry: Literal[
        'Analytics',                    # B2B: Sells software that collects, models, or visualizes business data...
        'Sales',                        # B2B: Sells software to businesses for managing sales pipelines, commissions...
        'Content',                      # Consumer: Companies that sell media, editorial, or informational content...
        'Home and Personal',            # Consumer: Companies that sell services, software, or products to individual consumers for managing their homes...
        'Payments',                     # Fintech: Companies that sell transaction processing, payment acceptance, fraud prevention...
        'Banking and Exchange',         # Fintech: Companies that sell deposit accounts, currency exchange, crypto trading infrastructure, or core banking and compliance software...
        'Healthcare IT',                # Healthcare: Sells software, data infrastructure, or AI systems to healthcare providers, payers...
        'Manufacturing and Robotics',   # Industrials: Companies that sell industrial automation systems, robots, engineering software...
        'Construction',                 # Real Estate and Construction: Sells software, AI tools, or payment and compliance infrastructure to construction companies, contractors...
        # ... 41 more
        'Other',                        # Education, government, nonprofits, or anything none of the six domains above describes.
    ]
    """Which one subindustry best describes what this company sells and to whom?"""

We ask for the subindustry and read the domain off it, because the finer question gave the better domain answer for both decision models. Post 4 has why.

Each request carries the company name, YC's one-liner and its long description, under one instruction: "You classify companies into a venture fund's investing domains." Jev and Luna return a probability for every one of the 51 options. Sonnet gets the same definitions and runs with thinking off, a request setting that here makes it five times cheaper and, on a 100-company check, no worse against the reference.

500 companies, three models

We drew 500 active YC companies, 2019 batches onward, from a public mirror of YC's directory with seed 7, each with a description of at least 200 characters. The sample is balanced, not YC's mix: 84 or 85 companies per domain, all 47 eligible Real Estate and Construction companies, 30 Other, and Roofr added by hand. The models saw the name and the descriptions, never YC's label.

We grade against two answer keys, because YC's labels were assigned around batch time and many of these companies have moved on since. The first is YC's own label, assigned around batch time: the "Agreement with YC" column. The second, "Against reference", replaces it wherever any model disagreed with YC. Opus 5.5, Anthropic's largest model, read those 242 companies' current descriptions without seeing YC's label or any model's pick, and its answer became the reference, closer to a label for the company as it is now. Opus read the domain differently from YC's label on 65 of them. Opus is Claude, so the reference likely flatters Sonnet; the engineers' section has the detail. The decision models ran from Warsaw with 32 requests in flight and Sonnet with 6, so the latency columns are not like for like; costs come from recorded tokens.

Each company is 0.2 of a point, and a gap of a few points between the two decision models is inside the noise. Against YC the three 500-company rows are within two points of each other, because the ceiling there is the batch-time labelling, not the models. Against the reference the spread opens to four points, Sonnet ahead. The last row is 100 companies, so its agreement with YC is left blank rather than set against the 500-company rows; on those same 100, thinking on and off both score 97% against the reference and pick the same label for 94 companies.

On cost, Jev at $0.15 per 1,000 companies is a twentieth of Sonnet's $3.01 and Luna at $0.28 a tenth; Sonnet with thinking on costs $15.30, for reasons in the engineers' section. Sonnet's median latency was 7 times Luna's and 2.9 times Jev's, under the concurrency noted above. Sonnet's price is Anthropic's listed 5.5 rate on 9 October 2026.

Where Jev and Luna disagree

Jev and Luna pick a different domain for 35 of the 500 companies. On those 35 Jev matched the reference 22 times and Luna 13, too few to rank the two. What differs is how they miss. Luna's misses are confident: payments companies filed under B2B (Volopay at 1.00) and sector companies sent to Other (EARTH AI at 0.75). Jev's misses are more often hesitant, and 17 of its 35 would have been routed by the band, against 12 of Luna's. A confident miss is one no band can catch.

What the probability is worth

With seven labels the probability can be checked. A company put at 0.7 should have the right domain about 70% of the time. Here the probability is for the domain: the model's probabilities for the subindustries under it, added up. Accuracy is against the reference, per 0.1 bucket, Jev and Luna side by side.

No company fell below 0.3 for either model. Both are close to calibrated against the reference: the expected calibration error, the average gap between the stated probability and how often the model is right, is 0.018 for Jev and 0.023 for Luna, Jev slightly under-confident and Luna slightly over. Against YC's labels the same numbers are 0.090 and 0.108, so both models would look overconfident if YC were the truth; most of that gap is in the labels, not the models.

The rows under 0.6 hold 27 Jev companies and the model is right on 16 of them. That is what makes the list useful. It is short, the model is honest about being unsure on it, and the long tail outside it is nearly always right. Post 1 found Luna sharper and Jev more hedged. That does not show at the domain level: Jev puts 80% of companies at or above 0.9 and Luna 81%. Jev does park more companies in the 0.5 to 0.6 bucket, 21 against 10, and that is where its routed set comes from.

The band

The design is post 1's, adapted to a list. The decision model answers every company. Where it is unsure, Sonnet with thinking off answers instead. With many labels, unsure means one of two things: the top pick is below 0.6, or it leads the runner-up by less than 0.2. The thresholds are one rule shared across the series, not tuned to this data; the table below shows what other settings would do.

With Jev the rule sends 66 companies (13%) to Sonnet, and the combined system matches Sonnet alone at 97% against the reference (which likely flatters Sonnet) for $0.54 per 1,000 against $3.01. With Luna the rule routes 43 companies (9%) and reaches 94% against the reference; the two routed sets overlap on only 17 companies, so each model's band tracks its own uncertainty.

You can only see what routing buys if your labels are good enough to show it. Against YC's own labels Jev alone agrees on 84.8% of companies, the combined system on 85.4%, and Sonnet alone on 85.4%, so Jev is within a point of Sonnet at a twentieth of the cost and there is nothing to route for. Against the adjudicated reference, routing 13% closes a two-point gap at a sixth of Sonnet's cost.

The rule sends 66 companies from Jev to Sonnet. On those 66 Jev had the domain right for 50 and Sonnet for 62. Routing changed the domain on 14 of them, 13 to the reference label and 1 away from it. Five of the 66 are below. Probabilities are Jev's, summed to domain level; the rationale is the opening of Sonnet's; a check mark means Sonnet matched the reference.

Roofr is the opening anecdote and the band's best case: Jev leans B2B, Sonnet reads it as construction, Opus settled on Construction with B2B Sales as a secondary. It sits in two domains, and Jev's rerun minutes later put Construction first at 0.51. Blueshoe is the clean fix: a law firm that sells to individuals, filed as business software by YC at batch time and by Jev now, which Sonnet read as consumer from the description. FLOWER CO. is the most common case, a band that fires and changes nothing: Jev and Sonnet both say Consumer, and the disagreement is one level down.

Cardamon is not Sonnet erring. YC, every decision-model run and Sonnet say B2B; only Opus says Fintech, and the reference follows Opus. The fund's own definitions overlap here: Banking and Exchange includes "core banking and compliance software" sold to "financial institutions", and B2B Legal lists "compliance" too. A company that sells compliance software to banks fits both as written. Volopay went the other way. Jev had Fintech right at 0.56, the band fired on the margin, and Sonnet moved it to B2B.

Post 1 said the band's width depends on what a wrong answer costs. Two experiments in, that claim can be sharper. The data decides where the points are. Against the reference the curve is flat past 13% routed, 97.0% at 13% and 97.4% at 38% for 2.4 times the cost. The cost of being wrong decides how much each of those points is worth to you. On the guardrail every point was worth paying for, and there were six commands' worth. Here a mis-filed company costs a missed look, so a point is worth little, and at the domain level the points run out at 13%. That is still a wider band than post 1's, 13% of companies against 6% of commands. At the subindustry level the curve keeps climbing to 38% routed; that is post 4's.

For a fund the recipe is short. Write the domain definitions in your own words. Label a sample as the companies are today, because against dated labels routing shows no gain. Run the decision model on everything and send what it is unsure about to a language model. On this data that was 13% of companies, at 97% against a reference that likely flatters Sonnet, for $0.54 per 1,000 against $3.01 for Sonnet on everything. At that price the whole pipeline can be re-tagged every time a description changes.

A nightly re-tag depends on stability. Between two identical Jev runs minutes apart, four companies changed domain, Roofr among them, all four inside the band; Luna returned identical output on all 500.

For engineers

Sampling

The sample is balanced by domain, not drawn at random. The companies come from the yc-oss mirror of YC's public directory, which refreshes daily and declares no licence, so the decisions-demo repository publishes only each company's slug, name, YC label, batch and a hash of its description, and rebuilds the descriptions locally. The draw, on 3 October 2026 with seed 7, takes active companies from the 2019 batches onward with a usable YC label and a description long enough to classify, then spreads them as evenly as the pools allow: 30 Other and 84 or 85 per domain, with subindustries in proportion to their pool. Real Estate and Construction was small enough that every eligible company is in. Roofr, Summer 2017, is the one company kept regardless of batch.

That balance is why the per-domain mix matters when reading the scoreboard. YC's eligible pool is more than half B2B, so the sample takes a small share of the B2B companies and most of the Consumer ones. Luna does better on the domains YC files as B2B, Consumer and Other, Jev on Fintech, Healthcare, Industrials and Real Estate and Construction. Reweighted to the pool's domain mix, the two decision models tie at the domain level, with Sonnet still ahead. The ordering in the scoreboard is a property of this balanced sample, not a general ranking of the two models.

Why Sonnet runs with thinking off

Thinking off is the like-for-like setting. On 5.5, "off" means between_tools, not disabled. With thinking off, Pydantic AI sends the schema as a tool definition, so Sonnet receives the same 51 options and the same definitions Jev does, and it still writes a short visible rationale before the tool call on nearly every company. That is the Sonnet the band escalates to.

It is also the setting that keeps Sonnet's cost column fair. The schema runs to over 6,000 tokens, and as a tool definition it is prompt-cached across calls. With thinking on, the same schema goes as uncached strict-mode structured output on every request, and that alone is the gap between $3.01 and $15.30 per 1,000 in the scoreboard. The difference is in how the request is shaped, not in what the model is asked.

What thinking on changed

Nothing measurable in the answers. On the first 100 companies of the shuffled sample, thinking on and off score the same against the reference at the domain level and pick the same label on all but six companies. Against YC's own labels the picture on those 100 is the same: thinking on, thinking off and Jev land within two companies of each other.

What it changed was latency, in the opposite direction from what the name suggests. Thinking on was faster at the median on those 100, because with structured output it writes only the answer and no rationale, so it emits a fraction of the output tokens. The setting moves cost and latency more than the answers, which is why the main runs use thinking off and thinking on appears only as the 100-company check.

Latency and cost details

Two caveats on the columns. The routing table prices a routed company at Sonnet's average cost, but the companies the rule actually routes are the long, ambiguous ones, which cost more than an average Sonnet call; the honest figure for the default rule is closer to $0.58 per 1,000 than $0.54, which changes nothing in the argument. On latency, Luna's tail at 32 requests in flight is partly queueing: its p95 fell by more than half on a rerun minutes later, so read that column as what a busy afternoon looks like rather than as a property of the model.

On stability, the flips sit where the band already is. Jev's four domain flips on the rerun all had the winning domain near a coin toss in both passes, and across all 500 companies its confidences moved by about a hundredth on average. A deterministic model gives a reproducible band; a model with small run-to-run noise moves the band's edge slightly between runs without touching the confident answers. Both decision models' mean confidence sits within two points of their accuracy at the domain level, which is the calibration table in one line.

How the reference was built

The reference replaces YC's label only where a model disagreed with it. A company is disputed when any main run (the Choice shapes on Jev and Luna, Sonnet, or Sonnet with thinking) disagrees with YC on domain or subindustry; that is 242 of the 500. For each, Opus 5.5 with thinking at high effort read the company text only, never YC's label or any run's pick, and returned a primary domain and subindustry, an optional secondary pair and one sentence of reason, for under $9 in total. Where a company was not disputed, YC's label stands.

Opus and YC split the disputes roughly down the middle, and the disagreements mostly moved companies into B2B. Opus kept YC's primary on 112 of the 242 and read 130 differently, half of those at the domain level; the net effect is more companies in B2B and fewer in Other than YC files. The 41 disputes added after the Luna runs were a check on this: Opus sided with YC on all but one. None of the 242 labels has been hand-corrected, including the companies named in this post, so a round of corrections would shift every figure here by a few points.

Opus judging Sonnet

Opus is Claude, so Sonnet's accuracy against this reference is likely flattered, and the design cannot say by how much. On the adjudicated companies Sonnet agrees with Opus's domain more often than Jev does, by 13 companies; whether that is Sonnet reading better or two Claude models reading alike cannot be separated here.

Two things bound the effect. Against YC alone, where no Claude model is involved, Sonnet and Jev agree with YC on the same share of domains, so the gap only appears once Opus is the judge. And Luna gets no family advantage: where Luna and Opus read a company one way and Sonnet another, the reference counts for Luna.

Two axes, and the shape of the question

The band and the calibration table read different quantities, and the difference is small. The band reads the top probability and the margin from the 51-option answer; the calibration table sums those probabilities to the picked domain. Reading the domain as the parent of the top subindustry, rather than as the domain with the highest summed probability, changes the pick on a handful of companies for each model, and a rule written on the summed domain probabilities lands within a point of the default at a similar routed share. Post 1's guardrail run had not stored Jev's full probability vector for action, which is why that post could not apply this rule; every run here stored it.

The finer question gives the better coarse answer. Asked the seven-domain question directly, both decision models score several points lower against the reference than when asked the 51-option question and read off the parent. That is why the flat 51-option question is the one reported here, and why post 4 compares the shapes.

Where the numbers are

uv run report domains in the decisions-demo repository prints every table in this post from the committed result files, without an API key: the scoreboard, calibration, routing fit, all 66 routed companies with their domain-level view, the adjudication counts, the same-100 comparison and the stability table. The result files for every run are under data/results/, with the adjudicated labels and the sample. The full research report, with the other two experiments, is docs/report.md in the same repository.

Next

The taxonomy has seven labels, a full distribution per company and cheap mistakes, so the band pays for exactly as many points as the labels can show. Post 3 puts the same shape inside a triage pipeline for inbound deal flow, where the decision model reads every email and the language model writes only for the top of the list.

The harness, every result file above and the research report are in the decisions-demo repository. The figures were computed on 9 October 2026 from the recorded result files; the company descriptions are YC's and the companies' and are rebuilt locally rather than redistributed.

Models and prices used: Jev, TypeSafe's decision model, typesafe:jev-latest, which resolved to jev-1.13.0 on 9 October 2026 (the runs were on 4 October and the responses do not record the version), at $0.042 per million input tokens, output free; OpenAI gpt-6-luna through the Decisions API, public beta from 6 October 2026, at $0.10 per million input tokens, no output or cache charges; Claude Sonnet 5.5 with thinking off, and thinking on for the 100-company check, at input $2, output $10, cache write $2.50 and cache read $0.10 per million tokens, per Anthropic's price page on 9 October 2026; Claude Opus 5.5 as judge, at input $4, output $20, cache write $5 and cache read $0.20 per million tokens, per Anthropic's price page on 13 October 2026. Python 3.12.13, pydantic-ai-slim 2.52.0, typesafe-sdk 0.7.2, openai 3.26.1; Luna through the repo's own adapter with every request parameter at its default. Jev and Sonnet runs 4 October, Luna runs 9 October, adjudication before 5 October with 41 companies added 9 October.