Following the Preference, Missing the Optimum: Compliance Without Optimization in AI Housing Recommendation

The Question

More renters now describe what they need to an AI assistant and read the five addresses it returns. The renter sees what the system surfaced and never sees what it left out. If a cheaper apartment with a shorter commute was in the same inventory and got skipped, nothing looks wrong. The renter just pays the difference every month for a year.

Earlier audits showed that AI models steer housing seekers by perceived identity. None could say what a user actually loses when a recommender overlooks a better option, because none had a complete inventory to score the omissions against. This study builds that inventory and puts a dollar figure on the miss.

How the Audit Works

For each of 150 renter scenarios in New York City, the model is handed a pool of 120 real listings drawn from 3,885 RentCast records, each with known rent, bedrooms and a transit commute computed from the MTA timetable. Because every candidate is known in advance, the exact set of suitable listings and its best options can be computed for every scenario.

• Strict dominance as the main outcome. A recommendation is strictly dominated if the same pool holds a listing that is cheaper, has a shorter commute and has no fewer bedrooms. This needs no weights and no assumption about what the renter values. It is a statement about the choice set alone.

• A random-selection floor built from 200 random draws per scenario, so every rate has something to be compared against.

• A custom transit router. A backward RAPTOR search over the MTA GTFS feed, solving arrive-by commutes and validated against published travel times from Grand Central to Tottenville.

• A matched identity grid. The same request and the same pool, changing only an identity cue (name or housing voucher), so any shift in recommendations can be attributed to that cue.

In total, 9,945 model calls (96.5% parsed) across three models from OpenAI and Anthropic, for $57.01 in API cost.

What I Found

• Compliance is near-perfect. Models broke a stated hard requirement on only 1.8% of recommendations, against a 66.6% random floor.

• Optimization is poor, and it has a price. 39.0% of recommendations were strictly dominated. When they were, the better listing on the same screen was a median $900 a month cheaper and 3.5 minutes closer.

• On rent, the models did worse than random. The average gap to the best option was +$498 a month for the model versus +$261 for random selection.

• Preferences are honored, but not optimized. Changing one sentence about what matters moved median recommended rent by $646 a month in the right direction. Yet under "rent matters most", recommendations still sat $606 a month above the five cheapest suitable listings in the pool.

• Spelling the rule out does not help. An explicit "minimize rent first" instruction produced no material improvement, established by equivalence testing against a pre-specified $50 bound (effect +$3.48, 90% CI of −$11 to +$18) rather than read from a non-significant p-value.

More Choice, Worse Ranking

With filtering removed and only suitable listings shown, the cheapest listing was selected in 93.7% of answers at 10 candidates, 72.4% at 20, 57.1% at 40 and 53.5% at 80. The obvious product instinct, retrieving more listings to improve recall, makes the ranking step worse.

The pattern replicates across vendors. Three models spanning a 45x range in price per token returned rent-first gaps of +$700, +$699 and +$702. Paying more for a model does not fix it. I report this as a shared failure mode, not a model ranking.

Identity and Fair Housing

Holding the request and the inventory fixed, I found almost no identity-based disparity. 47 of 48 pre-specified contrasts were null after Benjamini-Hochberg correction, and 105 further contrasts on the census-tract characteristics of recommended listings found none. This null applies to re-ranking a fixed candidate set. It does not license claims about open-ended search, where prior work does find steering.

Four Design Faults Caught Before They Became Findings

• The walk radius was deleting cheap apartments. At the standard 800 m subway-access radius, the excluded 11% of listings were $700 a month cheaper at the median and concentrated in Queens and Staten Island. Widening to 1,200 m kept 94.6% of listings.

• Dominance rate depended on pool size. In the full universe 99.5% of one-bedrooms are dominated, versus about 55% in a 120-listing pool. I split the measure so model behavior is not confused with my own sampling choice.

• Studio pools were structurally undersized, which tied pool size to bedroom count. The slots were redistributed.

• Missing data was about to correlate with the treatment. A pilot had 26% parse failures from long answers, and length may vary with the identity cue. I capped justifications and discarded the pilot rather than merge it. The final parse rate was 96.7% with zero invented listing IDs.

My first draft also concluded that these models ignore stated priorities. That comparison was confounded. I retracted the claim and rebuilt the experiment, and everything above is the corrected version.

Limitations

• This audits in-context ranking, not deployed products. The model is handed its candidate pool, so nothing here describes ChatGPT, Perplexity or any shipped search pipeline.

• The stress-test pools are not a market. Most listings violate a constraint by design, so violation rates are not prevalence estimates for real housing search.

• The protocol was written and version-controlled before data collection but was not filed with a registry.

• Three models and two vendors are enough to rule out a single-lab explanation, not enough to claim universality.

Why It Matters

Standard AI evaluation asks whether an answer was relevant. This design asks whether something strictly better was on the same screen, and prices the answer in dollars and minutes. I propose dominance-rate instrumentation as a diagnostic that housing platforms and AI developers can deploy on their own inventories. All code, prompts and per-call results are released.