I got a peek at our AI operations dashboard last week, care of Neal Iyer (thanks Neal). Like many of us, we are all trying to get a handle on the economics of AI. We already know that Tanya Brennan can spend $1,000 on Fable in one week without thinking about it. And we know that if we're not careful, our Amazon Web Services (AWS) Bedrock bill can double in a month. These cost components are the easiest to at least make visible. The total cost of AI, and the extent to which these costs are creating a return, continues to be a frustrating topic of conversation.
There is no one size fits all model for evaluating the cost of AI solutions and vendor pricing methodologies are changing. What is not changing (at least for the next 42 seconds) are the ten cost dimensions. I've grouped these into four buckets, and each dimension should be looked at for every major AI investment. A subset should be looked at for individual use cases, or small groups of use cases.
The most obvious cost elements are, of course, the application itself and the any use charges. We are pretty familiar with these as they are part of the traditional software-as-a-service (SaaS) model. This can be convenient pricing model with a good partner as it creates a predictable cost pattern. Whether you buy or build, you will likely have to integrate, which is your cost to deliver.
The choice of buy versus build has an impact on the downstream cost dimensions like talent. When you build, you have to get more of a very specialized type of talent that is difficult to find. Growing this talent internally is expensive and time consuming, partnering to acquire the talent is an option. Regardless, it is essential that every mortgage organization have some subset of W2 talent for AI and generative AI.
Oh the tokens. So much controversy, so much variability. A token is a word or part of a word, and it is the currency of large language models (LLMs). You pay for tokens as they go in, "thinking tokens", and the tokens that come out. Even if they are wrong.
The good news is that the cost of foundation AI has decreased significantly since "the early days" of 2023 and 2024. The price of a fixed level of AI capability has fallen 9x to 900x per year depending on the task (Epoch AI, 2025). GPT-4-level output that cost ~$30 per million tokens in 2023 now costs well under $1 (industry pricing analyses, 2026).
The bad news is that the cost of LLM usage for most organizations is still going up. Why? Because we typically use the newer bigger models for everything, without optimizing, and the price for those models has not gone down.
We recently implemented our AI ops dashboard for cost visibility, and have a range of techniques we use for cost optimization, including model distillation, and intelligent design of our innovation pipeline. Model distillation is the process of learning from a higher horsepower model and then training a lower cost model to act like the bigger one. Intelligent design in the innovation pipeline is the process of dynamically optimizing model choices and context management in the pipeline relative to the tasks they need to perform.
The next big challenge for us, and really for everyone, is how to connect token use to outcomes. We get the big bill from AWS, and we can connect that bill to people and to models, but we don't know if the spend was "worth it". This comes down to LLM observability and monitoring, and it's not easy but it can be done.
The first step is to find all the spend - subscriptions, cloud charges, vendor products, API charges - finding and gathering the cost for every person or team using generative AI. This sounds easy, but really isn't. For example, this may mean combing through credit card statements for that pesky Synthesia subscription you forgot you had. And then you have to do this across all credit cards.
This step requires you to connect the dollars spent to the people who spent them. You found that $2,500 charge for Replit in May - now you have to connect that to the people who have access. You have to find the individuals who are using the product compared to ilde seats and users. This requires you to go into yet another platform, and connect that platform to you AI operations solutions. This can make your team nervous. People generally do not like to be monitored. And the point is not to monitor the people, it is to understand how AI is being used for the purpose of optimizing cost. You may even need some teams or people to spend MORE.
This is the hardest part, and requires an even lower level of monitoring as you want to see not only THAT the people are using AI but WHAT they are using it to create. If you only use one platform, say Microsoft Copilot, this won't be that hard. But if you use a lot of different platforms, this can be a challenge. We have recently implemented Langfuse for LLM observability and have started to connect prompts to tokens, models, tool use, and agent invocation.
So now you found all the spend, connected it to your AI ops console, and are able to observe the relationship between the dollars and the purpose. Next, and most difficult, you elevate that to the outcome level. Full transparency, we are still working on this. We have pockets of use that we understand well, and pockets that we still have to connect. Spend + person + intent = the basis for understanding the outcome.
This is your data, your evaluation process, and your compliance approach. Much has been said about the important of good data, I won't dwell on this here. But I will talk about the other costs of quality - compliance and evaluation. Cost of compliance is the cost of the program and processes you have in place to govern your AI use - and especially your high risk uses.
Step one is to understand what you are doing with AI and does any of it qualify as what Colorado is calling a consequential decision. I use this standard because it's the best one I can find, and it works well with what has been defined by the Administration for compliance by the federal government. The decision to place something in the catalog as well as the decision to designate a use case as consequential or high risk should be documented.
I am advising everyone to put in place the governance model proposed by the Administration in M-25-21, I think it's a solid framework that is straightforward. The hardest elements are the logging and monitoring, both of which needs to be in place for any AI use case in my opinion.
One thing to note, a human-in-the-loop who just reviews AI generated content will ultimately be insufficient to meet what I think examiners and regulators will look for. In addition, you will want to understand, document, and be able to explain and demonstrate effectiveness of your context specific guardrails.
In addition to guardrails, another key cost of compliance is your evaluation program. These are the tests, preferably performed at scale by automation, that you perform to test that AI systems are performing in accordance with your expectations - including standards for quality. If you do not have an evaluation program, start small, with core metrics that matter.
There really is not substitute for high quality benchmark data sets. These are sometimes also called QA (question and answer) pairs. QA pairs are are structured data sets documenting representative test questions along with their verified, correct reference answer (often called "ground truth"). They are used as a standard benchmark to test how accurately and safely an AI model or retrieval system responds. These are time consuming to do well but will be a critical part of your ongoing quality process.
You want to run your evals on a schedule, anytime you need to test prompt changes, and then before and after model upgrades. Depending on what information your vendor makes available to you, you might be able to rely on their evaluation program but at the end of the day, you are accountable. As always, I recommend the open source framework by Promptfoo
As you run your evals, you might see that your eval scores are getting worse, you will need to take action if this is the case. Your eval program will be a critical investment in the robustness of your overall AI ecosystem and should not be skimped on. It may end up being a significant cost to consider.
And finally, the cost of innovation. This article is already way too long so I'll summarize. Talent, time, change, and opportunity cost. These are the hardest to quantify and the most important to do well. I promise to write another article on this specifically as part of this series.
