Hallucinations Are an Operating Risk

Hallucination is not a trivia bug. It is an operating risk when someone acts on invented detail.

Fluent wrong is more dangerous than an obvious error. People forward it.

A CSM at a SaaS firm pasted an AI summary of a renewal thread into a QBR pack. The summary invented a discount the account had never been offered. The customer asked for it on the next call. Trust took a hit that no model card could repair.

The claim

Treat model text like an unverified draft until a person checks the facts that can create commitments, money movement, or policy claims.

Where risk concentrates

• Customer-facing numbers and dates.
• Legal and security language.
• Internal policy answers used in live decisions.
• Anything that will be copied into a deck without citations.

They added a simple rule. Any AI draft that touched commercial terms needed a human initials check before it left the team. Volume of drafts stayed high. Bad promises dropped. That is the point.

Controls that fit real work

• Require sources for policy and customer facts.
• Block send on high-risk fields until review.
• Log who accepted a draft that became external.
• Coach managers to ask what was checked, not only what was generated.

If your risk talk stops at model temperature, you are missing the operating path where the wrong sentence becomes a commitment.