There's a moment every AI product team knows well. The demo goes great. The stakeholders nod. Someone says “this is going to change everything.” And then the thing ships to real users, and within a week you're in an incident channel at 9pm trying to figure out why it's confidently wrong about something it was right about in every rehearsal.
I've now shipped four AI-powered products into production: Sustainability Intelligence, Carbon Pre-Audit, Carbon Credit Management, and Sustainability Benchmarking. All are being used by real customers making real decisions with the output. Every one of them taught me some version of the same lesson. The demo lied to you, and it wasn't even trying to.
Demos are curated. Production is not.
When you build a demo, you unconsciously curate the inputs. You pick the customer's cleanest emissions dataset. You pick the supplier record that actually has a GHG Protocol-aligned activity factor attached to it. You ask the model the question in the way you know it answers well.
Real users don't do any of that.
Sustainability Intelligence is where I learned this the hard way. The product's job is to tell a company which regulations and standards actually apply to it. That means the specific set of obligations triggered by its industry, geography, and size, rather than sustainability in general. That's a genuinely hard problem, because applicability is never one clean rule. It combines thresholds (revenue, headcount, balance sheet size), scope (does it apply at the entity level or the consolidated group level), phase-in timelines that differ by company size, and geography-specific transpositions of the same underlying directive. In the demo, we asked it about companies whose profile sat cleanly inside or outside a given regulation's thresholds, and it got the applicability call right every time. It felt like a genuine expert in the room.
In production, a real user asked a version of a question no one had scripted: “does CSRD apply to us next reporting year?” The company's standalone revenue and headcount sat just under the threshold, and the model answered confidently that it did not apply. That was wrong. CSRD applicability is assessed at the consolidated group level, and the company had acquired a mid-sized EU subsidiary that quarter, which pushed the combined group over the threshold. The model wasn't being careless about the rule itself. It knew the thresholds. It just answered the question using the numbers it had in front of it and never surfaced that the scope of assessment (entity vs. consolidated group) was the actual variable that mattered here. Nothing about the output looked wrong. It read like something a sustainability analyst would write.
That's the moment “the demo lied to you” stopped being an abstraction for me. A model that knows a regulation's thresholds cold can still get the applicability call wrong if it doesn't correctly identify which figures, at which level of the corporate structure, those thresholds are supposed to be applied to. In a domain where a wrong “you're not in scope” could mean a company misses a real filing obligation, “the thresholds were technically right” is nowhere near good enough. It reshaped how we thought about the whole product. The job was bigger than knowing the regulations. The system also had to know exactly what triggers each one, and show its work on why a company is or isn't in scope, rather than just assert the conclusion.
The failure modes nobody demos
A few patterns showed up across all four products that never once showed up in a demo environment.
1. Confidently wrong beats visibly broken. Sustainability Benchmarking exposed a quieter version of the same problem, buried not in language but in arithmetic. To benchmark companies against each other, the underlying metrics have to be normalized onto the same basis, and that normalization is full of silent assumptions. One customer reported waste in units that needed a volume-to-weight conversion before it could be compared against peers who reported straight in tonnes. The right conversion factor depends on the material, so the model would pick a plausible default rather than flag that it was guessing. Another metric, training hours, showed up from different companies in genuinely incompatible shapes: some reported total training hours per employee, others reported the plain aggregate with no employee count attached at all. Dividing one by headcount and leaving the other alone produces a benchmark that looks perfectly clean and is comparing two different things. A crashed API call gets noticed immediately. A benchmark quietly built on a bad unit conversion or an unnormalized metric does not, until a customer's sustainability lead presents it to their board and someone asks why their ranking looks off.
2. “Just put it in the LLM” is not a strategy. With Carbon Pre-Audit, the first version leaned on the model for almost everything, including judgment calls that are really just rules, like which emission factor applies to which activity type, how to flag a missing data field, or when a value falls outside a plausible range for that industry. Those are deterministic questions. They have a correct answer that doesn't change based on how the question is phrased. Handing them to an LLM meant the same input could get evaluated slightly differently depending on framing, and worse, it meant every “why did the system flag this” question led back to “because the model said so,” which is not an answer you can put in front of an auditor.
The instinct after that was to swing hard the other way and pull the validation logic out into hand-written rules engineered by the team. That fixed consistency, but it doesn't scale. Every new emission factor table, every new industry's plausibility range, every edge case a customer surfaces becomes a ticket for an engineer to hand-encode a rule.
The approach that actually worked was the third option. Use the AI to generate and maintain the rules (draft the plausibility ranges, propose the emission-factor mappings, surface the edge cases worth a rule), then apply those rules deterministically, every time, through regular code, not through a fresh LLM judgment call on every request. The model gets used for what it's actually good at, synthesizing domain knowledge into a first draft a human can review, and the system gets to behave the same way twice in a row, which is the bar for anything that ends up in front of an auditor.
3. Building for your first few customers quietly builds a product that only works for them. Carbon Credit Management made this trap obvious. The earliest customers were concentrated in oil and gas, and their retirement reasons, vintage-year conventions, and registry quirks were specific to that industry: a particular way of documenting why a credit was retired, particular vintage-year edge cases tied to project timelines common in that sector. The model learned those patterns fast, because they showed up constantly in the data it saw. It got very good, very quickly, at handling oil and gas credits.
The problem is that it got good at it in a way that reinforced the pattern rather than generalized past it. When a customer from a completely different industry showed up with different retirement conventions and different vintage structures, the model's fluency with the oil-and-gas shape of the problem made it more confident about interpretations that didn't actually apply, not less. It was not neutral about unfamiliar patterns. It tried to fit them into the shape it already knew.
Getting the AI to be genuinely generic, to handle a new industry's conventions as a first-class case rather than a deviation from the pattern it learned first, turned out to be a much harder problem than getting it to work well for the customers already in front of us. It's the difference between building something that works and building a platform, and it's tempting to mistake the first for the second because your early customers will tell you it's working great.
None of these are “the AI is bad.” They're what happens when narrow, well-behaved test conditions meet the actual messiness of production data, production users, and production customer mixes, which, in regulated domains like carbon accounting and sustainability disclosure, is a much higher-stakes gap than in a typical consumer app.
What actually closes the gap
A few things consistently mattered more than better prompts.
Build your eval set from real production data, not your own test cases, as early as possible. The moment you have even ten real customer inputs, your eval set should include them: messy units, missing fields, and all.
Design for “confidently wrong” as the default failure mode, not the exception. In an audit or disclosure context, an AI system needs to surface its own uncertainty, flagging thin sample sizes, missing data, or low-confidence extrapolations, rather than paper over them with a clean-looking answer.
Separate “knows the rule” from “knows how to apply the rule.” A model can recite a regulation's thresholds all day long. Whether this specific company, assessed at the correct level (entity or consolidated group), actually crosses them is a structured determination, not a language task, and the system needs to show that determination rather than just assert a conclusion.
Treat the first two weeks of real usage as a second discovery phase, not a launch victory lap. The most valuable eval cases I've ever had came from watching real customers use the product in week one, not from anything my team dreamed up beforehand.
Assume the data will be worse than your best pilot customer's data. Design the ingestion and validation layer for your messiest realistic customer, not your best one.
Ask “is this actually a rules problem?” before reaching for the model. If the correct answer to a question doesn't change based on how it's phrased, it's a rules problem wearing an AI costume. The best use of the model in that situation is often generating and maintaining the rules, not applying them fresh every time.
Interrogate whether your product works generically, or just works for the customers currently in the room. Fast early traction with a handful of similar customers can quietly train the system, and your own intuition about what “working” looks like, around their specific conventions, which is a debt you pay down later at a much higher cost.
The takeaway
A demo proves the model can do the task under good conditions. Production proves whether your system can do the task under real conditions: incomplete data, ambiguous questions, deterministic judgment calls dressed up as AI problems, and a customer base that's more varied than your first few logos, all while still being something a customer can trust enough to put their name on. That gap is where the actual engineering work lives, and it's almost never visible until real users show up.
That's the first lesson from the front line. Next up: why prompting is genuinely the easy 20% of building these systems, and where the other 80% actually goes.
