The Most Trustworthy AI Output Is the Least Reliable

The signals you use to judge whether AI output is trustworthy are the same signals fabrication produces.

More citations. More confidence. More specific numbers. More professional structure. Longer output with more detail. These are what make an AI response feel reliable. They're also what the model generates when it's fabricating, because fabrication has no constraint on specificity. Real data has limits. Fabricated data doesn't. The model can cite as many sources, produce as many numbers, and assert as confidently as the output requires. Sourced output has to work with what's available, which often means acknowledging gaps.

Tested this with blinded evaluation. Three AI outputs with real data sourced from named studies. Three with fabricated citations and invented numbers. The reader rated the fabricated versions as equivalent or more trustworthy. The fabricated versions cited more sources, used more specific numbers, and asserted more confidently. The sourced versions acknowledged limitations. The acknowledgment of limitations, which is the signal of honesty, cost the output credibility in the reader's evaluation.

The reader was a domain expert with 87 experiments in AI evaluation. They still couldn't distinguish sourced from fabricated based on the output alone.

The mechanism: RLHF trains models to produce output humans rate highly. Humans rate confident, well-cited, specific output highly. Fabrication produces all three without constraint because there's no external anchor. Sourced output is constrained by what the source actually says, which is often more limited, more qualified, and less impressive than what unconstrained generation produces. The training that makes AI output sound trustworthy is the same training that makes fabrication sound more trustworthy than truth.

This is measurable in the output itself. AI-generated citations are more numerous (the model cites as many as the argument needs). AI-generated numbers are more precise (the model generates exact values, not ranges). AI-generated conclusions are more confident (no hedging, no "this is unclear"). Each of these properties is what trust evaluation rewards. Each is more prevalent in fabricated output than in sourced output.

The practical implication: the feeling that AI output is trustworthy is not evidence that it's correct. Especially when the output is detailed, well-cited, and confident. Those properties correlate with fabrication, not with accuracy. The outputs that deserve the most scrutiny are the ones that feel the most trustworthy. The ones that acknowledge limits and gaps are more likely to be honest, even though they feel less reliable.