IT & TechnologyArchive edition6 min read

Setting up useful operational metrics

Measure workload and user-visible behavior that informs decisions about reliability.

Connected infrastructure nodes and concentric paths — Setting up useful operational metrics
Original ARTware computational illustration.

A monitoring dashboard can show a healthy processor, a comfortable memory margin and steady network traffic while a surprising proportion of application tasks fail. Those graphs tell a maintainer something about the resources in use, but they do not explain what the application delivers. An operational metric becomes useful when it helps answer a question about workload, outcome or delay and supports a decision someone may actually make. Begin with the service's essential activities. Think about what a satisfactory outcome looks like for a user, what things can go wrong between the request and the result, and which observations would distinguish those situations. The technical resource graphs can then provide context rather than substitute for evidence of normal operation.

Name activities before naming measurements

Imagine an application that accepts documents and produces a normalized output. Its primary activity is not simply receiving a web request: it must accept suitable input, record the work, process the document and make the result available. The initial operation may finish before the processing itself begins. Counting accepted requests alone would leave the team unaware of documents that never complete. Define the stages in terms that the application can observe. An incoming submission can be accepted or rejected according to its input requirements. Accepted work enters a pending state, then an execution state, and reaches a completed or unsuccessful outcome. These distinctions allow the metrics to describe whatever is happening rather than merging substantially different stages.

Choose a small set of measurements associated with those stages. A count of accepted submissions indicates incoming workload. Completed and failed processing counts describe outcomes. A count of currently pending work shows whether the system is keeping up. A processing duration indicates how much time the relevant operation takes. If a user must wait before processing can begin, queue delay may deserve its own measurement or an end-to-end duration may be more persuasive. Each metric answers a different question. For example, an increased queue does not always mean the processor is broken. It could reflect a temporary increase in arrivals, a reduced processing rate or a failure that keeps a subset of work from reaching a final state.

Choose units and comparisons that make sense

A cumulative count grows over time and is useful when comparing how many operations have taken place between two observations. Its rate over a defined interval can describe how quickly work is arriving or being completed. A gauge of current pending jobs answers a different question because it can rise or fall. A duration measurement records how long a particular operation took. Decide what each instrument represents before constructing charts. Label units clearly and make sure that charts being compared use compatible time intervals. Otherwise an impressive-looking difference can arise simply because one graph shows the total count since startup and another shows the number of events observed during a recent minute.

Related reading: Choosing storage for application files.

Keep denominators visible when calculating outcome proportions. Twenty failures among forty attempted operations suggest a different situation from twenty among two thousand. If the count of incoming work includes requests that have not yet completed, comparing it directly with current failures can also be misleading. For the document processor, define whether an outcome proportion is based on the group of jobs that reached a final state during an interval or the work accepted during a particular interval after allowing sufficient time for completion. These calculations support different analyses. Choose the interpretation appropriate to the decision and explain it in the dashboard or its operational notes. A percentage without a defined population is difficult to use confidently.

Give the slowdown a meaningful shape

A single average processing time may be dominated by the most common small documents. Larger but entirely legitimate submissions could have a very different pattern. Record a latency distribution where the measurement system supports it, and inspect ordinary and slower portions of that distribution. Segment work according to distinctions that genuinely affect its expected behavior, such as a small set of size ranges or operation types. A high processing duration for an unusually large input may be less concerning than an equal duration for a routine request. The segmentation should remain limited enough for maintainers to compare results. Avoid attaching unbounded identifiers or user-provided labels to every measurement, since that can make aggregated observations expensive and difficult to interpret.

Layered flowing curves with fine network-like intersections
ARTware visual study. A complementary computational composition.

A task can also be slow for reasons other than the processor's execution time. During a busy period, the processing duration might remain steady while pending work accumulates. In that case a chart of average execution time would show little change even though users wait longer. Compare arrivals, completions, backlog and end-to-end timing over the same period. A growing backlog with steady arrivals might warrant inspecting the processing capacity, while growing arrivals and proportionally growing completions may simply explain a brief increase in demand. The graphs need not reveal every cause by themselves. They should indicate which part of the workflow warrants closer examination. Detailed diagnostics can then be collected for particular representative operations without overwhelming the general dashboard.

Establish what ordinary behavior looks like

Collect observations during normal use before deciding which changes should prompt a response. A small system may receive nearly all its requests during business hours and little traffic overnight. A scheduled operation can produce a temporary rise in resource use. If neither pattern is understood, automatic warnings may appear during expected quiet or busy periods. Examine the application's actual operating rhythm and note useful comparisons. A weekday afternoon may be a better reference for another weekday afternoon than an overnight period. Workload can also change gradually as a service attracts more use. Review the baselines when application behavior, capacity or the kinds of tasks being performed change materially.

Set notifications around conditions that merit a human action. A sustained increase in unsuccessful final outcomes could require examination of the causes. A backlog that continues to grow beyond an acceptable delay might warrant inspecting processing capacity or a stuck dependency. Specify the observation period and minimum relevant activity so a single unusual data point does not create a persistent stream of irrelevant notifications. Provide enough context for the recipient to take the next step. A message stating that a number exceeded a threshold is less useful than one naming the affected activity, the measurement interval and the related workload. Establish who responds and what reference material helps that person investigate.

Use an exercise to check the dashboard

In a test environment, run successful document submissions, deliberately unsuitable input and a controlled period of slower processing using representative non-sensitive material. Observe whether the measurements can distinguish an input rejection from an execution failure and a slow queue from a slow task. Look at resource graphs at the same time. Ask a maintainer to explain the likely cause from the available observations and identify what additional evidence is needed. This is also a useful time to check whether an apparently missing result is due to an instrument that was attached to the wrong stage. Adjust the definitions before deciding that the graphs provide reliable evidence. Keep the exercise bounded to the approved test environment.

When the dashboard is useful, it becomes possible to describe an operational problem in terms of the service: the accepted workload increased, completed tasks did not keep pace and people waited longer than expected. Resource measurements can help explain that observation, but they are no longer carrying the entire argument. As the application evolves, remove charts that nobody uses to make decisions and add new observations only when a real question cannot currently be answered. The aim is a coherent picture of normal work and its exceptions. A compact collection of well-defined measurements will often help a small maintenance team more than an impressive surface filled with numbers of uncertain significance.

Explore the implementation area: IT consulting.