IT & TechnologyArchive edition6 min read

Understanding latency in distributed systems

Find the delay that matters instead of optimizing one component in isolation.

Connected infrastructure nodes and concentric paths — Understanding latency in distributed systems
Original ARTware computational illustration.

An application that feels immediate on a developer's machine may feel slow when its components run in different locations. The code can be executing just as efficiently as before. What has changed is the journey required to complete an ordinary request. A database lookup that takes a small fraction of a second locally may also involve connection setup, waiting for another operation to finish and transferring a response across a network. Repeating that journey several times turns modest delays into a visible pause. Improving distributed latency begins with timing the whole operation and following its constituent parts. It is easy to make one component look faster while changing very little about the time a user actually waits.

Measure the complete activity

Choose a task that matters and define its boundaries. For a visitor opening a populated catalog view, the task might end when the item details appear, not when the first incomplete document has been delivered. For a processing service, it might end when the requested output is available rather than when an initial request is accepted. Both measurements could be useful, but they describe different behavior. Record timing at the point where the activity begins and where it reaches the intended result. Then collect the information needed to explain that interval. If a screen looks ready while the data is still being retrieved, consider measuring its initial display and the arrival of essential content separately.

Imagine that a product screen requests information about an item, its availability, related items, media and a category name. The individual requests might each finish in eighty milliseconds under ordinary conditions. If they must be performed in sequence, their cumulative duration is roughly four hundred milliseconds before allowing for other work involved in preparing the screen. Several independent requests launched together may complete in a time closer to the slowest of the group, although they share processing and connection capacity. The numbers here are illustrative. Their purpose is to show why reviewing an isolated endpoint's timing is not sufficient to explain a screen that depends on multiple exchanges.

Account for connection and waiting time

The duration reported by a requesting service is not the same as the amount of work carried out by the receiving application. A request may wait for an available connection. It may require establishing a network route and negotiating the connection before the receiving process can accept its message. Work can sit in a queue while a shared resource is occupied. The receiving application performs its operation, prepares the result and sends it back. All these intervals may contribute to the timing recorded by the caller. Where measurements are available, record connection setup, time waiting before processing, actual execution and transfer as distinguishable observations. Do not assume that the largest number visible in an application log necessarily describes the largest part of the user's delay.

Related reading: Setting up useful operational metrics.

A convenient way to follow a request across components is to attach an ordinary correlation identifier to the operation and include it in the relevant diagnostic records. This lets a maintainer connect observations belonging to the same request without recording unnecessary content. Compare durations calculated within each process rather than interpreting timestamps from separate machines as perfectly synchronized. If the outer operation takes eight hundred milliseconds and an inner service reports sixty milliseconds of processing, there is a substantial interval to explain. Look for other steps, waits and repeated calls before concluding that increasing processing power at the inner service would help.

Look for journeys that do not need to be repeated

A common source of cumulative delay is an interface that retrieves a list and then asks a remote service for the detail belonging to every entry. Locally, this pattern may seem an uncomplicated way of keeping code organized. Across a network, a page containing twenty entries can produce twenty additional exchanges. Review the information required by the screen as a whole. Perhaps the service that returns the original list can include the small amount of summary data the caller needs. Alternatively, a separate operation can accept a collection of identifiers and return the matching information together. These approaches change the shape of the data exchange rather than merely tuning the time spent searching for an individual record.

Layered flowing curves with fine network-like intersections
ARTware visual study. A complementary computational composition.

Do not automatically replace all separate operations with a single enormous request. Sending a larger collection of data can consume more time if most of it is not used. One large response may also make partial results harder to handle. For the catalog screen, a batch of summary information could be appropriate while a detailed history remains a separate operation that is performed only when someone requests it. Decide which data naturally belongs to one user-visible activity and which can reasonably be loaded later. If operations are independent, consider overlapping them within sensible limits. Track whether the change reduces waiting without moving an unreasonable burden to the service that handles the combined work.

Identify where a reused result is acceptable

Caching avoids repeating some operations when a previously obtained result is still suitable. Category descriptions that are rarely edited may be candidates for reuse, whereas a stock value might have a much shorter useful lifetime. Before introducing a cache, define what the cached value represents and what happens when the authoritative information changes. A delay in updating an image caption may be acceptable, while a delay in reflecting the availability of a limited item may not be. The refresh strategy should reflect that difference. Measure the time for a cached response and a response that requires obtaining fresh information. A cache that improves frequent requests may leave an occasional uncached request just as slow as before.

Also consider what happens when many entries cease to be reusable at about the same moment. A system may suddenly need to perform a large amount of previously deferred work. Avoid measuring only a conveniently warmed application with all the desired values already present. Test a representative cold start and a sustained period of normal use. Examine how the service recovers after an ordinary interruption or expiration boundary. Be clear about which user-visible values can remain temporarily older than the source if the originating system is slow or unavailable. Preserving a prior nonessential description may keep part of a page useful, but representing an unknown observation as a current fact can be misleading.

Examine the requests at the slower end

An arithmetic mean can conceal an important range of behavior. A service completing most operations in a predictable time may still have a recurring group of unusually slow requests. Collect timings for enough representative activities to examine several parts of the distribution. Compare a typical duration with a high-percentile duration, and inspect individual traces that contribute to that slower range. Separate activities with substantially different workloads, such as opening a small catalog page and generating a large report. Otherwise changes in the proportion of these activities can alter the combined measurements even though neither operation has changed. Examine results under ordinary workload and at a busy but realistic period.

After changing the data exchanges or caching policy, repeat the same observations with comparable inputs. Check that the expected data still arrives, including an incomplete or delayed result from a dependency. Look at total duration alongside network intervals, queue time and the execution time of significant operations. Use those findings to decide whether more work is warranted. A design that consistently completes the relevant user task within its required time can be preferable to a complex one constructed to make the individual components exceptionally fast. Latency optimization is worthwhile when it removes real waiting from a real activity, not simply when it improves the smallest number that happens to be available in a dashboard.

Explore the implementation area: IT consulting.