IT & TechnologyArchive edition6 min read

Preparing an application for data recovery

Treat recovery as an observable operation rather than a checkbox for having backups.

Connected infrastructure nodes and concentric paths — Preparing an application for data recovery
Original ARTware computational illustration.

A nightly backup job can run for months without proving that the application can be recovered. It may copy the expected data to a safe location and still omit a configuration needed to interpret those records. A file reference in the database might point to an object that was stored elsewhere and never included in the backup. The current application version may not understand the older data format. These gaps are easiest to discover during a planned recovery exercise, when an unsuccessful attempt can be investigated without a production interruption. Treat preparation for recovery as the design of a complete, verifiable operation. The important result is a usable service and its required information, not the presence of a folder named backups.

Work back from the expected service

List the activities that need to work after a recovery. A publishing service might need to display existing entries, associate the proper images with them and allow a new draft to be prepared. A processing application may need to show previously accepted work and provide a consistent account of operations that were unfinished at the time of the saved state. Use these activities to identify the required material rather than beginning with a list of directories. Application data is an obvious requirement, but there may also be durable user files, configuration, information describing the application release and other dependencies. Distinguish what is authoritative from material that can be regenerated once the service is running.

For example, an application database might record the details of documents submitted by users while the documents themselves reside in separate storage. The database notes the stored object's identifier and an associated processing result. Restoring the records without locating the corresponding objects could produce pages full of inaccessible documents. Saving the objects without their relationships would make it difficult to establish which entries they belong to. Some processing results may be regenerated, while others reflect decisions made by the user that cannot be reconstructed automatically. Identify the required relationships. A recovery plan should state which collections need to correspond to one another and how the restored application will recognize them.

Decide how much interruption and change can be tolerated

Two constraints are worth discussing with the people who depend on the service. First, how far back can recovery reasonably move the stored state? Second, how long can normal operation remain unavailable while restoration is carried out? These questions address different concerns. A backup produced every night may contain information from before the beginning of a busy workday even if the restoration itself completes quickly. Saving changes frequently does not ensure a fast restoration if assembling the environment takes many hours. Set practical objectives using the consequences of missing work and the organization's ability to provide alternatives during disruption. There is little value in adopting demanding targets that cannot be supported by the available system.

Related reading: A practical approach to technical consulting.

For a low-traffic informational site that is updated weekly, saving the entire approved publication after each change may cover much of the necessary content recovery. A rapidly changing application needs a design matched to the rate and importance of committed changes. The storage mechanism may have capabilities for creating snapshots or preserving a sequence of changes. Understand their behavior and the requirements for reconstruction before relying on them. Determine how the snapshot relates to activity that is taking place at the same time. Different components captured independently may describe different moments. Define acceptable consistency and use the storage systems' supported methods to obtain it; merely starting each copy at around the same time does not establish a coherent result.

Make prerequisites part of the recovery record

A maintainer should be able to tell which application release can read a particular saved state. Record the version identifier, important configuration requirements and any relevant data format or migration state with the backup inventory. Provide a controlled way to recover essential access information and secrets through the organization's legitimate secret management process. Do not copy confidential configuration indiscriminately into an unprotected backup archive or place it in the written procedure. The goal is to make the authorized recovery path explicit without weakening the protection of those values. Document who may retrieve the necessary material and how it should be applied to the isolated recovery environment.

Layered flowing curves with fine network-like intersections
ARTware visual study. A complementary computational composition.

Consider which recovery dependencies may themselves be unavailable during the expected failure. If the only instructions for restoring a service are stored inside that service, they cannot be consulted when needed most. If all retained copies depend on the same failed storage location, their number alone does not create independence. Keep a suitable inventory and the recovery procedure available through a separate controlled location. Review how copies are protected, who can access them and how long they are retained. An older copy may be useful when a problem is discovered well after a mistaken change, but retaining every version indefinitely may introduce unnecessary costs and exposure. Set a retention schedule that addresses the actual application requirements.

Conduct a restoration without putting production at risk

Choose a representative saved state and restore it in an appropriately isolated test environment. The environment should not accidentally send ordinary production messages, process real submissions or present restored records as current information to other applications. Use the documented configuration process and application version rather than improvising a replacement from the latest source. Time the stages separately: locating the retained material, preparing the environment, restoring the data, reconnecting required stored objects and checking the service. A long preparation step might account for more interruption than the actual copying operation. If a necessary prerequisite is difficult to obtain, record it and revise the plan rather than silently using an undocumented shortcut.

Verify that the restored content is meaningful. Select known records from different parts of the application and inspect their associations. Open a representative stored document, verify an associated result and check that an optional missing file is handled as expected. Compare appropriate counts and other known properties to those recorded with the saved state. A basic check that the database responds to ordinary requests is not enough to establish that the information needed for actual work is available. Exercise essential application activities using controlled test input. Include one ordinary change to confirm that the restored environment can accept new work where that is part of the recovery objective. Keep test-generated material separate from the saved state.

Improve the next exercise using what was learned

Document the result in practical terms. How much time did the exercise take? Which planned steps worked without adjustment? Was any required information absent or unusually difficult to retrieve? Which application activities were verified and which were outside the scope of the test? Avoid recording a successful backup as a successful recovery if the restoration has not been exercised. Set a schedule for repeating the trial and identify changes that should trigger a new one, such as a storage migration or a significant change to the application's data representation. Assign responsibility for checking the saved material and maintaining the recovery procedure.

Recovery readiness is an ongoing property of the system, because the system and its dependencies will change. A concise, accurate procedure that has been used successfully is more useful than a detailed document that no longer matches the running application. Frequent copies may improve the amount of work that can be preserved, while efficient environment preparation may improve the time it takes to resume service. Both improvements should be based on observed constraints. When recovery has been rehearsed, the team has concrete evidence of what it can restore and which limitations remain. That evidence supports sensible investment instead of leaving a critical operational question unanswered until the service is already unavailable.

Explore the implementation area: IT consulting.