Chapter 2: Fluent, Wrong, Expensive
A senior sales engineer showed me something in a workshop last year. Quietly, the way people show you things they're not sure they're allowed to have.
It was a proposal draft, and it was good. Clean structure, confident language, the right technical vocabulary. He had built it with ChatGPT from the customer's requirements email, and in his words, it had saved the deal. The customer was impressed by the turnaround. His boss was impressed by the polish.
One problem. The motor it specified would have tripped the protection system at startup. Not a matter of opinion, not an edge case: a machine that would have failed on commissioning day.
Fluent. Wrong. Risk disguised as speed.
Here is the uncomfortable statistic from my own conversations: half the sales engineers I talk to are already quietly using ChatGPT to help configure quotes. There is a shadow CPQ operating inside your company right now, and it hallucinates. Before you reach for the policy hammer, understand what that actually means. Your people are not being reckless. They are telling you, in the clearest way possible, what they need: a faster way to get from a messy customer request to a credible draft. The shadow CPQ is not the enemy. It's your spec.
But the tool they've chosen cannot do the job safely, and it's worth being precise about why.
What the intern doesn't know
A large language model is a prediction machine. It has read approximately everything, and it produces the most plausible next words given what it has seen. For language, that's a superpower. For configuration, it's a trap, because plausible and valid are different things.
The LLM knows what a truck is. It just doesn't know your trucks. It doesn't know that the sleeper cab requires the high-horsepower motor, that the gearbox ratio it just proposed isn't manufactured, or that the voltage it confidently selected fails regional compliance where the customer operates. Those facts live in your engineering department, and the model has never met your engineering department.
Worse, it fails charmingly. When a junior employee doesn't know something, they hesitate, and you can see it. A language model produces the same confident prose whether it's right or wrong. There's a word for this that I've adopted: workslop. Fluent, confident nonsense at scale.
And there's a subtler failure mode. Ask an LLM to assemble a quote and it reaches for the typical: the bundle that looks like the past quotes it has seen. That's how outdated accessories sneak into new proposals and how last year's pricing logic gets a second life. An AI trained on your history learns your history, including the parts you've been trying to retire.
I watched a vendor pilot where the AI stitched together a genuinely clever bundle with a tidy discount ladder. Legal flagged it a week later: it violated the company's own pricing policy. Nobody had taught the model the policy, because the policy lived in a PDF and three people's heads. When you evaluate AI for quoting, evaluate the brakes, not the engine.
The arithmetic of almost right
The demos will tell you the model is 95 percent correct, as if that settles it. Run the numbers in the other direction: a 95 percent correct engine is a 5 percent revenue and trust leak. One quote in twenty carries an error into contract, into engineering, into the factory, or into a customer's building. In the last chapter we counted what a single wrong elevator costs. Now multiply by your quote volume.
The core misunderstanding is about what kind of task quoting is. Writing the proposal narrative is a creativity task, and LLMs are wonderful at it. Deciding what goes in the machine and what it costs is a verification task. A quote is a contract. "Probably right" is wrong.
Anti-pattern: The Ventriloquist Bot. A chat interface bolted onto quoting that produces configurations directly from the language model, with no validation layer underneath. It looks like your configurator is talking. It's actually improvising, in your brand voice, with your margin.
None of this is an argument against the technology. It's an argument about seating arrangements. The language model belongs in the conversation, drafting, translating, explaining. It does not belong anywhere near the commit button.
Your sales engineers have already voted on whether they want an assistant. The only question is whether you give them one with rails.
Which raises the obvious question: what do the rails look like? The answer has been sitting in plain sight, unadvertised, for twenty-five years.
An LLM is a brilliant intern with no access to your engineering department, and no fear of being wrong.