# ADR: Kept awake
Status: accepted, 2026-09-28. Steve's goal for it: no visitor ever meets a cold read, on either site, at any hour. Shipped as 1.0.3.35.
## In plain words
This page decides that each running copy of the site sends itself every public read a visitor can make once every four minutes. That keeps the databases and their caches from going cold between visitors (a keep-warm loop). It runs inside the app on both sites, and the health card shows when it last ran.
What that is worth: a developer can trust that the first visitor after a quiet spell gets a warm read instead of waiting two seconds, and the organization pays nothing extra for it on the plans it already has.
## Context
Read from Missouri on the Cosmos DB site on the morning of 28 September, every read that had been quiet for a while was slow the first time and fast after it:
| The read | First after a quiet spell | Warm |
| --- | --- | --- |
| `GET /api/admin/activity` | 2.0 to 2.2 s | about 260 ms |
| `GET /api/health` | 736 ms | about 190 ms |
| The activity card's four chunks through the edge | about 700 ms on a miss | 33 ms on a hit |
The chunks are the edge's (ADR: Cache headers, the addendum of the same day). The reads are the store's: a query's connections and its plan go cold when nobody asks for a while, and so do the caches in front of the stores. The first visitor after the quiet pays for waking them. The activity report had a second cost of its own: counting every visitor row of its window. That count is now kept and rebuilt behind the reader (ADR: Site activity, and the line an address does not cross, the addendum of the same day). What was left was simply that nobody had asked in a while.
## Decision
**The container keeps its own reads warm.** `KeepWarm` in `api/TheYard.Api/KeepWarm.cs` is a loop inside the process that, every four minutes for as long as the container runs, sends every public read a visitor can open through the app's own HTTP pipeline, once for each store (the `X-Yard-Store` header): the listing, the filter values, one vehicle from that listing, the health report, the stores, the version, the tests summary, the recommendations, the activity report for each of its three windows, the machines, the page sweep, the metrics, the SQL and store logs, the log, Azure's view of the container, the peer, the tests, the proof, the experiment, the telemetry and the errors. Each read carries the site's own mark on its agent, `TheYard-SelfRead/1 (keep-warm)`, so the activity card counts it as the site reading itself and never as a person. It also carries a header of its own, which keeps it out of the request ring and the kept log: two hundred slots of the container reading itself every four minutes would be the speed tile timing the loop, not a visitor.
- **In the process, not on a schedule outside it.** A scheduled workflow starts when a runner is free, not when it was asked (Steve's experience is that scheduled runners start late and are not dependable). A loop inside the container runs exactly as long as there is something to keep warm, including the minutes after a roll.
- **Staggered.** The first pass waits a random part of a minute, so the two sites' containers do not fire together on the one plan.
- **Never overlapping, never throwing.** A pass is awaited before the next is timed, so a pass that runs long pushes the next back rather than running beside it. A read that fails or answers anything but 200 is counted and logged as a warning, and the pass goes on; a pass that throws is logged and the next one still comes.
- **On where it is deployed, off everywhere else.** It runs where App Service runs the site (`WEBSITE_SITE_NAME` is App Service's own setting) and nowhere else, the test host included. `KeepWarm:Enabled` turns it off on a site, or on anywhere, as an app setting.
- **Visible.** The health card says when the last pass ran and how many reads it sent, with the slowest of them: "Kept warm: last pass 12:04, 46 reads, slowest 312 ms".
## What it costs
Every read goes to the container's own loopback address, so none of it passes the edge or counts against the edge's allowance. What a pass spends is store work, priced before shipping against what Azure itself said on 28 September (`az` reads, 12:47 CDT):
- **Azure Cosmos DB.** The account `cosmos-theyard-ss` is on the free tier, and its database is provisioned at 1,000 request units a second; it is not serverless and not autoscaled. On provisioned throughput the bill is the rate and not the use, and the free tier covers the first 1,000 a second, so reads under the rate add $0. What a pass uses: one read of the activity report for each of its three windows cost 312 request units when measured that morning, and a pass sets off at most one rebuild of each window's report per container; the health check's two point reads and one vehicle add a few more, as do the store's own reads. Two containers, one pass each every four minutes, is 720 passes a day: about 225,000 request units a day, about 2.6 a second on average against the 1,000 provisioned, a quarter of one per cent. A pass is a burst of about 312 over a few seconds, which is under the rate. The SDK retries a throttled read on its own. The figure per pass is read off the live site once the loop runs.
- **Azure SQL Database.** The sites point at `sqldb-theyard-ss-basic`: Basic and five DTUs, online. It has no auto-pause delay and no minimum capacity, because a Basic database has neither. It is priced by the day whatever it does ($4.90 a month, ADR: One plan, two sites), so the loop adds $0, and it cannot pause. The serverless database beside it, `sqldb-theyard-ss`, is paused and no site points at it.
- **App Service.** `PLAN-THEYARD-SS` is Basic B1: one worker for two sites. It has no autoscale setting and no elastic scaling. Both web apps are running with Always On set, and a health-check path of `/healthz`. A Basic plan is priced by the hour whatever it runs ($12.41 a month, ADR: One plan, two sites) and never scales to zero, so the loop runs inside a processor already paid for and adds $0. Always On is what keeps the process itself alive between visitors; the loop keeps what is inside it warm.
- **Nothing else can go to sleep.** The two container groups the sites ran on before the plan, `aci-theyard-ss` and `aci-theyard-cosmos-ss`, are still in the resource group and serve nothing; the edge's rules no longer point at them.
## What it does not do
Steve's plan also asked each pass to fetch the build's hashed files through the public address, so the edge never lets a chunk go between visitors. That part is not built, for two reasons read before shipping; it waits on his call. The edge keeps a proxied file at the node that served it, and a read from the container in West US 3 warms the node nearest West US 3, not the one a visitor in Missouri reaches (ADR: Cache headers, the addendum of 28 September). And the edge charges for it: about forty files a pass from both containers every four minutes is close to a million requests and five gigabytes a month, about 300 of the edge's credits a month at its April 2026 prices (2 credits per 10,000 requests, 20 per gigabyte), the whole of the free allowance, to keep warm a node that serves nobody in particular. The deploy's warm-up, and the push from Missouri, cover the first visitor after a roll instead.
## Turning it off
Set `KeepWarm__Enabled` to `false` in a site's app settings; the site restarts without the loop and the health card says "Kept warm: off on this container". Nothing else depends on it: the reads it sends are reads any visitor can send.
## What holds it
- `KeepWarmTests`: with a clock the test moves by hand: the first pass waits the stagger and then one pass runs every four minutes; a pass that runs past the interval delays the next and two never run at once; a pass that throws does not stop the next; inside a pass a read that fails is counted, and every other read is still sent, on both stores, with the site's own mark.
- `keepWarm.test.ts`: the health card's line, when the loop has run, has not yet, and is off.
## Addendum, 2026-10-04: both catalogues stay loaded, on purpose
Warming both stores through the request pipeline also loads both catalogues and keeps them loaded,
and the catalogue keeper (ADR: One plan, two sites) was there to let the second one go after ten idle
minutes. Four minutes is less than ten, so the two features cancelled each other and the Admin tab
showed it. Both stores stay warm, which is this record's goal: a visitor who switches stores never
meets a cold read. The cost is the second catalogue's memory on each site, working sets of 350 to
475 MB a site with both held, and the keeper is retired.
## Addendum, 2026-10-17: the first screen's reading is kept warm too
`/api/pulse` joined the loop with the band on the landing page (ADR: One reading for the first screen). It belongs here for the reason the rest of the list does: it is a read a visitor opens on the first screen, it asks the two sources the Admin tab asks, and a cold one costs the visitor the wait. The loop's own count follows the list rather than a number written down, so nothing else moved.
## Where it sits
KeepWarm and KeepWarmState live in the host Api, with one line on the health card in the front end, since sending reads to the app's own pipeline is a hosting concern that no inner ring should know about. Single responsibility, one reason to change, separates the schedule in KeepWarm from the reads in KeepWarmReads, and the loop takes its clock as a TimeProvider through the constructor, the dependency inversion habit that lets KeepWarmTests move time by hand. It cost a pass of reads every four minutes per container, inside capacity the plans already pay for. A plan billed per request or able to scale to zero would make the loop cost money and reopen the choice.
## Files
- [`api/TheYard.Api/KeepWarm.cs`](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Api/KeepWarm.cs): the loop, the reads and the pass.
- [`api/TheYard.Api/Composition/Startup.cs`](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Api/Composition/Startup.cs): the wiring (region keep-warm-wiring).
- [`api/TheYard.Api/Composition/ObservabilityRegistration.cs`](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Api/Composition/ObservabilityRegistration.cs): the request hook that leaves its reads out of the ring and the kept log.
- [`api/TheYard.Api/Endpoints/HealthEndpoints.cs`](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Api/Endpoints/HealthEndpoints.cs): the health report that carries its last pass.
- [`api/TheYard.Tests/KeepWarmTests.cs`](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Tests/KeepWarmTests.cs): the schedule, the overlap and the failures, on a clock the test moves.
- [`src/lib/keepWarm.ts`](https://github.com/SteveStout/TheYard/blob/main/src/lib/keepWarm.ts) and [`src/components/admin/HealthCard/HealthCard.tsx`](https://github.com/SteveStout/TheYard/blob/main/src/components/admin/HealthCard/HealthCard.tsx): the health card's line.
```csharp
///
/// Keeps every read a visitor can open warm, on both stores, for as long as the
/// container runs (ADR: Kept awake). Without it the first read after a quiet
/// spell is the slow one, seconds against a few hundred milliseconds warm; the
/// record carries the numbers. Every four minutes this sends each public read through the app's own HTTP
/// pipeline, once per store, so the stores' connections, the caches in front of
/// them and the report cache never go cold between visitors.
///
/// In process on purpose: a scheduled workflow starts when a runner is
/// free, not when it was asked, and a loop that lives in the container runs
/// exactly as long as there is something to keep warm. The first pass waits a
/// random part of a minute, so the two containers never fire together; a pass
/// that runs long delays the next rather than overlapping it; and a read that
/// fails is a warning, never an exception, so the next pass always comes.
///
public sealed class KeepWarm : BackgroundService
{
/// Carried on every read, so the request ring and the kept log leave it out; the agent is what the activity card reads.
public const string Header = "X-Yard-Keep-Warm";
/// The site's own mark (Hits.IsSelf), so the activity card counts these as the site reading itself and never as people.
public const string Agent = "TheYard-SelfRead/1 (keep-warm)";
public static readonly TimeSpan Interval = TimeSpan.FromMinutes(4);
public static readonly TimeSpan MostStagger = TimeSpan.FromSeconds(60);
private readonly Func> _pass;
private readonly TimeProvider _clock;
private readonly TimeSpan _stagger;
private readonly ILogger _logger;
private KeepWarmPass? _last;
private int _passes;
public KeepWarm(Func> pass, TimeProvider clock, TimeSpan stagger, ILogger logger)
{
_pass = pass;
_clock = clock;
_stagger = stagger;
_logger = logger;
}
/// The last pass that finished, for the health report; null until one has.
public KeepWarmPass? Last => Volatile.Read(ref _last);
/// How many passes have started.
public int Passes => Volatile.Read(ref _passes);
/// A random wait before the first pass, up to .
public static TimeSpan RandomStagger() => TimeSpan.FromMilliseconds(Random.Shared.Next((int)MostStagger.TotalMilliseconds));
///
/// The wait before the next pass: what is left of the interval after the one
/// that just ran, and nothing when it took longer, so a slow pass pushes the
/// next one back and two passes never run at once.
///
public static TimeSpan Next(DateTimeOffset started, DateTimeOffset ended)
{
TimeSpan left = Interval - (ended - started);
return left > TimeSpan.Zero ? left : TimeSpan.Zero;
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
try
{
await Task.Delay(_stagger, _clock, stoppingToken);
while (!stoppingToken.IsCancellationRequested)
{
DateTimeOffset started = _clock.GetUtcNow();
Interlocked.Increment(ref _passes);
try
{
Volatile.Write(ref _last, await _pass(stoppingToken));
}
catch (Exception ex) when (ex is not OperationCanceledException || !stoppingToken.IsCancellationRequested)
{
// The type, never the message: a message can name a server.
_logger.LogWarning("A keep-warm pass failed: {Type}", ex.GetType().Name);
}
await Task.Delay(Next(started, _clock.GetUtcNow()), _clock, stoppingToken);
}
}
catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
{
// The container is stopping; so is the loop.
}
}
}
/// One pass: when it ran, how many reads it sent, how many did not answer 200, and the slowest.
/// When the pass started.
/// How many reads the pass sent.
/// How many of those reads did not answer 200.
/// The slowest read, in milliseconds.
/// The path and store of the slowest read, or null when nothing was read.
public sealed record KeepWarmPass(DateTimeOffset At, int Reads, int Failed, long SlowestMs, string? Slowest);
/// What a pass reads, and the pass itself.
public static class KeepWarmReads
{
///
/// Every public read a visitor can open, the Admin tab's cards with the
/// windows they open on. One vehicle is added per store from the listing
/// itself, so the vehicle read is a real one.
///
public static readonly string[] Paths =
[
"/api/vehicles",
"/api/facets",
"/api/health",
"/api/stores",
"/api/version",
"/api/pulse",
"/api/tests/summary",
"/api/recommendations",
"/api/admin/activity?window=24h",
"/api/admin/activity?window=7d",
"/api/admin/activity?window=30d",
"/api/admin/machines",
"/api/admin/pages",
"/api/admin/metrics",
"/api/admin/sql",
"/api/admin/store",
"/api/admin/logs",
"/api/admin/azure",
"/api/admin/peer",
"/api/admin/tests",
"/api/admin/proof",
"/api/admin/experiment",
"/api/admin/telemetry",
"/api/errors",
];
/// Every read in , once per store, through the client given; a read that fails is counted and the pass goes on.
public static async Task RunAsync(HttpClient self, IReadOnlyList stores, TimeProvider clock, ILogger logger, CancellationToken cancellation)
{
DateTimeOffset at = clock.GetUtcNow();
int reads = 0;
int failed = 0;
long slowest = 0;
string? slowestPath = null;
foreach (string store in stores)
{
foreach (string path in Paths)
{
var (status, ms, body) = await ReadAsync(self, path, store, logger, cancellation);
reads++;
if (status != 200)
{
failed++;
}
if (ms > slowest)
{
slowest = ms;
slowestPath = $"{path} ({store})";
}
if (path == "/api/vehicles" && FirstVehicle(body) is { } id)
{
var (vehicleStatus, vehicleMs, _) = await ReadAsync(self, $"/api/vehicles/{Uri.EscapeDataString(id)}", store, logger, cancellation);
reads++;
failed += vehicleStatus == 200 ? 0 : 1;
if (vehicleMs > slowest)
{
slowest = vehicleMs;
slowestPath = $"/api/vehicles/{{id}} ({store})";
}
}
}
}
return new KeepWarmPass(at, reads, failed, slowest, slowestPath);
}
private static async Task<(int Status, long Ms, string? Body)> ReadAsync(HttpClient client, string path, string store, ILogger logger, CancellationToken cancellation)
{
var clock = Stopwatch.StartNew();
try
{
using var request = new HttpRequestMessage(HttpMethod.Get, path);
request.Headers.TryAddWithoutValidation("User-Agent", KeepWarm.Agent);
request.Headers.TryAddWithoutValidation(KeepWarm.Header, "1");
request.Headers.TryAddWithoutValidation(Backends.HeaderName, store);
using var response = await client.SendAsync(request, cancellation);
string? body = path == "/api/vehicles" ? await response.Content.ReadAsStringAsync(cancellation) : null;
if (path != "/api/vehicles")
{
// Read to the end, so the read is what a visitor's is, then thrown away.
await response.Content.CopyToAsync(Stream.Null, cancellation);
}
if ((int)response.StatusCode != 200)
{
logger.LogWarning("Keep-warm read {Path} on {Store} answered {Status}", path, store, (int)response.StatusCode);
}
return ((int)response.StatusCode, clock.ElapsedMilliseconds, body);
}
catch (Exception ex) when (ex is not OperationCanceledException || !cancellation.IsCancellationRequested)
{
logger.LogWarning("Keep-warm read {Path} on {Store} failed: {Type}", path, store, ex.GetType().Name);
return (0, clock.ElapsedMilliseconds, null);
}
}
/// The first vehicle's id in a listing, or null when the listing has none or cannot be read.
public static string? FirstVehicle(string? listing)
{
if (string.IsNullOrEmpty(listing))
{
return null;
}
try
{
using var document = JsonDocument.Parse(listing);
foreach (var property in document.RootElement.EnumerateObject())
{
if (property.Value.ValueKind == JsonValueKind.Array)
{
foreach (var item in property.Value.EnumerateArray())
{
if (item.ValueKind == JsonValueKind.Object && item.TryGetProperty("id", out var id) && id.ValueKind == JsonValueKind.String)
{
return id.GetString();
}
}
}
}
}
catch (JsonException)
{
// Not a listing this can read; the pass goes on without a vehicle read.
}
return null;
}
}
```
*Live from [`api/TheYard.Api/KeepWarm.cs`](https://github.com/SteveStout/TheYard/blob/1d6b50b/api/TheYard.Api/KeepWarm.cs#L7-L246), region `keep-warm`, read from this build at 1d6b50b.*