Understanding the Financial Implications of LLM Deployment

Table of Links

Abstract and 1. Introduction

2. Basic premises for our models

   2.1. How LLMs are charged

   2.2 An anedotal model from industry

   2.3 Choice of Costs in the Model

   2.4 Why multiple scenarios

3. A Decision Theoretic Model

   3.1 How to model success

4. A Model for Commercial Operations Based on a Single Transaction

   4.1 Example of Using the Model

   4.2 Model Analysis

   4.3 Discussion of the RoI vs. Earning dillema

5. Modelling of a Binary Classification Problem

   5.1 Local Sensitivity Analysis

   5.2 Global Sensitivity Analysis by Sobol Method

6. Related Work

7. Future work

8. Conclusion, References, and Acknowledgements

2 Basic premises for our models

According to the PMBoK, 7th edition, “value is the ultimate indicator of project success”. For every planned project, it is important to analyze the value of the project in different qualitative and quantitative terms. Among other indicators, financial results are key performance measures for project success. When establishing a business case for a project, it is crucial to understand its costs, revenues, and return on investment [Project Management Institute, 2021].

Costs are the total amount of money spent on a project, while benefits are the total amount of money gained from a new project. Earnings are the difference between benefits and costs [1], the total amount of money the project will bring to the company, and return of investment (RoI), is the relation between earnings and costs [Project Management Institute, 2021].

2.1 How LLMs are charged

There are different ways in which LLMs are charged. Our models assume that they are charged by token, as OpenAI does [Shekhar et al., 2024]. However, AWS charges specific engines by the hour of the instance used [Kodali et al.,2024]. Moreover, the costs of the use of proprietary machines can be calculated using traditional capital and operating expenses methods [Park, 2013].

2.2 An anedotal model from industry

In this subsection, we use our experience to discuss how most companies try to calculate the expected earnings from using an LLM in a project.

We bring attention to the fact that although one is actually making a prediction, probabilistic terms such as “expected value” and “probability of success” are not commonly used in a business case for a project.

This model represents the idea that the introduction of the LLM allows the company to convert some lost revenue into new revenue; however, it disregards the fact that the result of an operation using the LLM can be bad for the business. Moreover, usually it does not consider business tasks, that is, it considers that if the LLM task is a success, then the business task will also be a success.

A better model should include losses from the overall results of using LLMs, defined as L and the earnings would be the following:

Where Q, Q ≪ M, is the number of transactions that would result in a negative result.

With the same reasoning, RoI can be recalculated as:

This model is yet too simplistic, and we improve it in the next subsections, showing how it can be made more representative of the problem.

[1] We will use earnings E in this paper to avoid using P for profits, using P for probability