If you've never shipped an AI product past the pilot stage, it's easy to assume the work is mostly prompt engineering. Find the right words, get the right output, ship it. I believed some version of that too, right up until the first time one of our products had to survive real usage for more than a few weeks.
Across Sustainability Intelligence, Carbon Pre-Audit, Carbon Credit Management, and Sustainability Benchmarking, the prompt itself has consistently been the smallest piece of the actual engineering effort. Getting the model to produce a good answer under good conditions is maybe 20% of the work. The other 80% is the infrastructure that makes a good answer reliable, catchable when it's wrong, and affordable at real usage volume, and none of that shows up in a demo.
The prompt is the part that ships fastest and breaks the least
This isn't a knock on prompt design. Getting instructions, examples, and context right matters, and a bad prompt will absolutely sink a product. But once the prompt is decent, it tends to stay decent. It's not where the surprises come from in month two or month six. The surprises come from everything wrapped around it.
Where the other 80% actually goes
1. Evals are a program, not a step. Early on, “evaluation” meant running a handful of test questions before a release and eyeballing the answers. That worked fine when the product did one thing. It stopped working the moment Sustainability Intelligence had to handle regulatory applicability questions across a growing list of frameworks, geographies, and company sizes. A change to how the model reasoned about one regulation could silently change its behavior on an unrelated one. Building a real eval set, one that grows every time production surfaces a new edge case, that runs automatically before anything ships, and that someone actually owns, turned out to be a bigger and more permanent engineering investment than any single prompt ever was. It is never finished. You maintain it for the life of the product.
2. “Did it reconcile correctly” is the wrong question. “Can you show me every step, and who took it” is the real one. With Carbon Credit Management, the first version of “guardrails” meant checking whether the final reconciliation number was right. That's necessary, but it's not what actually matters when a customer's auditor asks about a specific credit six months later. What they need is a full audit trail: which fields the model inferred versus which a human confirmed, when a human overrode a model suggestion and why, which version of the rules and which model were in play at the time, and a timestamp on every single one of those steps. “The model reconciled the credit” isn't an answer an auditor accepts. “Here is exactly what the model proposed, when, what a human changed, when, and why” is. Building that, not as a log dump but as something a non-technical reviewer can actually read back and trust, took considerably longer than getting the reconciliation logic itself right. It's the piece that made the product genuinely usable in an audit context instead of just operationally useful.
3. Data residency requirements mean your context budget isn't the same everywhere. Carbon Pre-Audit handles emissions data for multinational customers, and some of them operate in geographies with data residency rules, where the data legally can't leave the jurisdiction, which means it can't go to a hosted frontier model running elsewhere. The workaround is a locally deployed open-source model, and that's where we got caught. Those models often ship with a much smaller context window than the hosted model we'd designed and tested against. The prompt and the surrounding context we'd built assumed a context budget that simply didn't exist in the restricted-region deployment. Supporting documents, prior audit notes, and reference tables that fit comfortably in one environment got silently truncated or rejected in the other. “Works in production” turned out to mean “works in the specific production environment I tested,” not production generally. The fix had nothing to do with a smarter prompt. It came from making the application itself aware of which model and deployment it was talking to, and adapting how much context it sent accordingly, trimming, summarizing, or chunking rather than assuming the same budget everywhere.
4. The worst-case user action is a design input, not an edge case. Sustainability Intelligence tracks roughly 2,400 regulations across geographies and sectors. Individually, analyzing any one regulation against a company's profile is fast and cheap. The problem showed up the first time a real user selected “all” in the regulation list and pressed refresh. That single click fanned out into thousands of model calls at once. One customer, one action, and a cost and latency spike that had nothing to do with how many customers we had and everything to do with what one customer's UI let them do. Nothing about that request was malicious or even unreasonable from the user's side. Selecting everything and refreshing is exactly what a “select all” checkbox invites you to do. We just hadn't designed for it, because it never came up in testing where nobody had a reason to refresh 2,400 regulations at once. The fix had nothing to do with a cheaper model call. It came from rethinking the interaction: batching and queuing bulk requests instead of firing them all at once, giving the user real progress feedback instead of a hung screen, and, more fundamentally, asking whether a “refresh all” needs to re-run every regulation from scratch or only the ones where something relevant, the company's profile or the regulation itself, had actually changed since the last run.
5. Handling model or provider changes without breaking anything. Every one of these products has had to absorb a model version change at some point. Sometimes a genuine upgrade, sometimes a provider-side behavior change that altered output formatting or tone just enough to break a downstream parser that had been quietly assuming a specific shape of response. The prompt didn't need to change. The plumbing around it, the eval suite that catches regressions before a customer does, the parsing layer that doesn't fall over when the format changes slightly, is what made those transitions non-events instead of incidents.
None of this is a complaint
I want to be careful not to make this sound like a case against using AI for this work. It isn't. The model is genuinely doing something none of these products could do without it: reasoning across regulatory frameworks, drafting plausibility ranges, synthesizing benchmarks across messy inputs. The point is narrower. The model call is the part of the system that looks the most impressive and takes the least engineering time, relative to everything that has to surround it before that capability is something a paying customer can depend on every day.
What actually deserves the time
Budget your project plan like the prompt is 20% of the work, because it is. If evals, audit trails, fallback behavior, and cost engineering aren't separate line items in your plan, they'll get built anyway, just later, under worse conditions, as incident response instead of design.
Build the eval set as a living asset from day one, not a pre-launch checklist. It should grow every time production surfaces something your original test cases didn't cover.
Log the process, not just the outcome. “The answer was correct” isn't enough in a context anyone might audit later. Capture what the model proposed, what a human changed, and when, every time, so the trail is reconstructable months after the fact.
Don't assume your context budget is portable. If any customer's deployment could route to a smaller or differently-hosted model, for data residency or any other reason, the application needs to know it and adapt what it sends, not assume the same headroom everywhere.
Design for the worst-case click, not just the typical one. A “select all” button, a bulk refresh, a retry-everything action. Anything your UI allows will eventually get used exactly as allowed, at whatever scale that implies.
Assume the model underneath you will change. Build the layer around it so a version change is something your eval suite catches, not something a customer reports.
The takeaway
Prompting is the part of building an AI product that looks the most like the finished thing, which is exactly why it's easy to overweight. The real work, the part that determines whether a product survives six months of real usage instead of just a good demo day, is almost entirely in the infrastructure around the model call: how you make every step reconstructable after the fact, how you stay aware of what environment you're actually running in, how you design for what your UI allows rather than what you expect, and how you make sure it still works when the model underneath it inevitably changes.
That's the second lesson from the front line. Next up: who do you actually blame when the AI is wrong, and how you build trust and accountability into a product a client is putting their name on.
