Artificial IntelligenceArchive edition6 min read

Managing inference cost in a small application

Estimate operating cost from the requests a feature really performs.

Luminous geometric nodes arranged in a layered lattice — Managing inference cost in a small application
Original ARTware computational illustration.

The first working version of an AI-powered tool might process only one note at a time and produce a short response. At that scale, the amount spent on each request can seem negligible. After the feature is released, users may submit longer material, ask for several variations and repeat an operation when an incomplete result appears. The financial behavior of the application is then determined by the real mix of requests and outcomes. Estimating cost starts with observing that mix. A small system does not require a complicated forecasting model, but it does require more than multiplying the price of a single demonstration by a projected number of users.

Measure the work a request actually requires

For a service billed by input and output units of text, find out how many units are charged for a complete operation. Include the application instructions, user material, retrieved context and other components that are sent with the request. Count the resulting text using the applicable accounting information. A feature that summarizes notes may send a comparatively large input and return a few paragraphs. A drafting feature may have the opposite emphasis. Different products and configurations can have different billing arrangements, so use the applicable rate and recorded usage rather than assuming every system accounts for the workload identically.

Suppose a request typically involves 1,500 input tokens and 400 output tokens. Treat that as a starting observation rather than a fixed characteristic of the feature. Some notes may contain fewer than a hundred words, while the minutes of a lengthy planning session may occupy thousands of tokens. Instructions sent with each request might remain approximately constant or grow when additional examples are included. A second attempt could repeat much of the first input. Record the distribution of request sizes, including exceptional cases. An ordinary average alone does not say whether a small portion of traffic accounts for a substantial part of spending.

Use simple arithmetic with transparent assumptions

A straightforward illustrative calculation is enough to check the scale of an early estimate. Imagine hypothetical rates of two currency units per million input tokens and eight per million output tokens. With 1,500 input and 400 output tokens, a successful single request would cost 0.003 plus 0.0032, or 0.0062 currency units. Ten thousand such requests would cost 62 units. These numbers are an arithmetic example, not a quotation from a real service. Substitute the actual applicable rates and accounting rules when preparing an operating forecast. Version the assumptions so that a later revision to pricing is distinguishable from a change in workload.

If each completed task requires an average of 1.2 requests of that same size, the simplified estimated expenditure becomes 74.4 units for ten thousand completed tasks. The multiplication is appropriate only when the retry requests really have similar usage. In practice the first request and a correction may differ in size, and some attempts may never produce a completed task. Count those failed tasks in the cost of operating the feature, even if they are absent from the number of successful outcomes. A cost per completed task describes the expense of producing useful work more directly than the price of an isolated request.

Related reading: Handling sensitive input in an AI-enabled product.

Look at the upper end of the workload

An ordinary user may generate one summary from a short note near the end of a meeting. Another might submit an entire project transcript, return several times to request changes and perform the operation repeatedly over a busy afternoon. Neither scenario is adequately represented by one average. Inspect a distribution of charged usage per request and per account where the application has a lawful reason to associate those measurements. Estimate an ordinary day, a busy day and a reasonable peak scenario. A high-percentile request size can help identify whether an input limit would affect only exceptional material or ordinary use.

Forecast at the level where a financial decision will be made. If an application has a monthly operating allowance, translate representative tasks into a monthly number of completed operations and expenses. If one user performs many tasks on the same material, consider whether the workload is attributable to text generation, document retrieval or repeated review. A rough estimate with a visible range is often better than a precise total built on invented traffic. State what is still unknown and collect measurements that reduce that uncertainty first. A two-week pilot with ordinary users might reveal that long inputs are rarer, or more common, than an early assumption suggested.

Layered flowing curves with fine network-like intersections
ARTware visual study. A complementary computational composition.

Reduce costs without damaging the task

Limit context to what the feature needs. An editing tool asked to revise a selected paragraph may not need the full document for every small wording adjustment. A summarizer preparing a project-level overview may actually require multiple sections, including the conditions in which decisions were made. Reducing text indiscriminately could make the feature faster and cheaper while producing unusable work. Compare the narrower input against a representative evaluation set. If removing a recurring introductory passage leaves the output unchanged, it may be a sensible optimization. If removing an earlier discussion causes a decision to be misrepresented, the reduction is not an improvement.

Caching may avoid repeating work that has already been done. A summary of a fixed publicly available document, generated with an unchanged configuration, might be suitable for reuse. A response based on a changed document is not equivalent, and material subject to access restrictions needs appropriate isolation. Define what makes a stored result valid: the content version, task parameters and relevant configuration may all contribute to the decision. Invalidate or bypass cache entries when those conditions change. A cache that returns a cheap but outdated result to the wrong task offers no useful saving. Measure the successful reuse rate and the expense of maintaining the stored material.

Manage retries and limits as product behavior

A response may be structurally incomplete or interrupted. Establish when another attempt is useful and set a bounded retry policy. Repeating exactly the same request indefinitely because a required condition is difficult to satisfy creates a rising bill and leaves the user waiting. If the application knows a document is too long for the chosen operation, explain the supported options before starting costly work. Limits on input length, output size or the frequency of expensive operations should correspond to the intended task. Communicate the limit clearly, including what the user can do when their material exceeds it.

Set up aggregate monitoring that lets a maintainer notice unusual consumption. Record the numbers of requests, completed tasks, failed requests and retries, together with relevant usage sizes. Avoid collecting entire notes merely to support cost accounting. Link expenditure to feature categories or operations rather than recording private content that does not contribute to the calculation. Use a notification or operational limit appropriate to the application's budget and decide what will happen if it is reached. Degrading an optional feature gracefully is preferable to allowing unexpected expenditure without visibility or making unrelated core functions depend on the inference allowance.

Include the expenses beyond generated text

An inference bill is one component of operating an AI feature. Retrieving and storing documents, processing uploads, reviewing questionable output and maintaining evaluations all consume resources. A feature that costs very little per request may still be inefficient if most outputs require substantial manual correction. Document the expense of the other work in the same operating forecast, even if early estimates are ranges. Review both spending and quality after a limited release. A configuration that lowers the request bill while increasing the number of rejected drafts may cost more per accepted result. The useful financial metric connects expenditure to work the application has completed to an acceptable standard.

Explore the implementation area: AI solutions.