AI Strategy Through the Lens of Costs

It may seem surprising, but understanding the different types of costs also provides a strategic vision for deploying AI intelligently and progressively within your organization.

Personally, I believe that for a strategy to be truly efficient, you need to think in terms of concentric circles spreading outward -- like when you throw a stone into water and the ripples propagate.

Identifying the highest-impact area first, then creating momentum that gradually reaches every part of the business, is a sound approach to building a sustainable organization that leverages the power of these new tools.

That said, this comes at a cost, and the costs associated with GenAI are an integral part of strategic and tactical AI implementation.

Managing AI through costs

We often hear the misguided advice telling company executives not to worry about implementation costs because AI is inherently profitable. At Reboot, we disagree with this idea.

We have seen companies launch sporadic, unstructured initiatives that produce one or two projects but remain confined to a corner of the organization without impacting the business as a whole.

The payoff is worth it, but it is an investment -- and not just a financial one. Like anything else, it requires a solid understanding of what's involved.

This is why, by identifying the different costs, we also understand the mechanics behind AI -- its possibilities as well as its constraints. Here is what I propose we explore, cost by cost.

Inference cost

The inference cost is the cost incurred when using an LLM-type model to generate a response.

Naturally, costs vary depending on the large model being used.

Broadly speaking, to generate text output from an LLM, there is what you input and what the LLM responds with. These inputs and outputs are managed through tokens, and each transaction has a token cost.

For small transactions, we're talking cents. But when you think in terms of volume, the cost becomes significantly larger.

The more computing power is required, the higher the cost.

Generating an image requires more computing power than generating text.

Another option for controlling costs would be to host an open-source LLM that you can train yourself to optimize the inference process.

The notable advantage is that you have full control over costs. The downside is the setup cost and, potentially, maintaining and updating the model and/or training the LLM to improve its performance.

Of course, here we're only addressing the financial aspects, but there are other factors to consider, such as security and cybersecurity concerns.

Fine-tuning cost

Fine-tuning is the process of adapting a pre-trained generative AI model to a specific task or domain.

You need to provide the data, train the model, account for the complexity of the chosen LLM, and iterate through training cycles.

These training iterations are called epochs.

According to OpenAI, the specific fine-tuning cost is:

Base cost per 1,000 tokens x Number of tokens in the input file x Number of epochs trained

Prompt engineering cost

Once your model is trained, you need to be able to query it by conditioning it according to your needs. This is achieved through the process of structuring text that can be interpreted and understood by a generative AI model, known as prompt engineering.

It is both an art and a profession to condition AI so that it delivers quality, high-performance results.

The challenge is finding the right balance between fine-tuning and prompt engineering.

If the task demands a high degree of accuracy and precision, fine-tuning will be more important, more costly, and more time-consuming than prompt engineering.

Cloud cost

Nothing new here -- data needs to be hosted, as does the AI, and this inevitably incurs costs.

As the AI strategy rolls out, you will likely need to make legacy systems coexist with new ones while ensuring access to all data.

Typically, companies opt for a "lift and shift" cloud approach where legacy systems are maintained as long as necessary while new AI-powered systems are deployed alongside them.

This is not a permanent solution, but rather a transitional one.

Talent cost

No surprise here -- to orchestrate all of this, you need skilled talent, like the kind we have at Reboot. And naturally, these skill sets are still quite rare.

Beyond operational skills, there is also a need for continuous training -- and not just on technical topics. These professionals are often multidisciplinary, tackling subjects from an essentially non-technical angle.

The impact of AI is significant, so company culture must be equally strong. Don't underestimate this point, because a company that uses AI will inevitably evolve and rethink its business for the better.

Finally, operational cost

Yes, you need to maintain and continue to enrich your AI systems through MLOps (Machine Learning Operations) while integrating with existing systems to ensure reliable and efficient model deployment and maintenance.

This involves automation and processes that fit into the workflow: data retrieval, feature engineering, training, testing, model storage, and deployment.

Underestimated costs

You need proper infrastructure and must progressively equip it with the resources needed to handle current challenges and the evolution of IT systems driven by AI.

You need to secure data thoroughly and prevent data leaks and intellectual property theft, counter the risks associated with malicious content distribution and targeted attacks.

You need to continuously train and raise awareness about technology-related risks and the risks posed by shadow IT (software not approved by the IT department).

You also need the ability to manage side effects such as biases:

• Machine bias reflects the biases embedded in training data,
• Availability bias favors easily accessible content, which reinforces existing prejudices,
• Confirmation bias keeps users entrenched in their existing beliefs,
• Selection bias comes from unrepresentative training data,
• Group attribution bias also reinforces prejudices by generalizing behaviors based on the actions of a few insufficiently representative individuals,
• Contextual, linguistic, and anchoring biases influence content relevance, cultural belonging, and dependence on initial information,
• Automation bias encourages placing unwarranted trust in AI-generated content

Conclusion

Our recommendation is to establish cost controls to make informed decisions, taking into account the various aspects we have just covered -- but also and above all, your company culture.

Whether you prefer using off-the-shelf solutions or building your own AI in-house has a completely different impact. This is why a financial arbiter can be valuable in certain situations.

Build an AI Factory or outsource? Create a cross-functional AI team that evaluates on a case-by-case basis, or opt for absolute governance and centralized AI? All of these questions become relevant the moment you enter this world.