# ADR: What the machines are doing
Status: accepted, 2026-09-19, shipped as 1.0.0.149. Asked for in one sentence: the SQL and memory
performance of the relational store, the document store and the container, on the Admin tab, kept
the way the site activity is kept.
## In plain words
This page decides how the Admin tab shows what the machines under the site are doing: the container's memory and processor sampled on a timer, and each database asked for the reading it keeps about itself. Each resource gets its own chart, with every minute kept for a month so a restart cannot empty it.
What that is worth: a developer can see whether a container's memory is climbing toward its limit, and the organization can check the claim that one small container serves the whole site, with nothing added to the bill.
## Context
The Admin tab could already say what the application did: every SQL statement with its milliseconds,
every document-store operation with its request charge, the timing of every path, and eight paired
rounds proving the two stores answer the same. What it could not say is what any of it cost the
machines underneath. A container with 1.5 GB and one processor serving a hundred thousand vehicles
out of its own memory is the claim the Performance overview makes, and nothing on the site showed
the memory it was talking about.
The reason is in the shape of every other reading here: they are all taken when something happens.
A request is timed because a request arrived. A statement is recorded because a statement ran.
Memory is not an event. Nothing was going to show it until something asked on a clock.
The three machines also owe three different answers, and pretending otherwise would be the easy
mistake here:
- **The container** knows its own memory and its own processor time exactly, from the runtime.
- **Azure SQL Database** keeps its own reading of itself, `sys.dm_db_resource_stats`, fifteen seconds
at a time for the last hour, on every tier including the Basic one this site pays $4.90 a month
for. It costs one SELECT to read and nothing to keep.
- **Azure Cosmos DB** has no memory reading to give, and no processor reading either. It is sold by
request unit, and what it will tell you is what each operation charged.
## What was considered
| Option | What it buys | What it costs |
| --- | --- | --- |
| **Sample the container on a timer, read each store the way that store reports itself** | Every number comes from the thing it is about, at no charge, and the card can say plainly where each one came from | Three sources and three shapes on one card, and a timer that has to be kept cheap |
| Azure Monitor metrics for all three | One shape for everything, and history that outlives the container | A metrics read is an Azure API call with its own permission and its own bill, and the container's managed identity would need a role it does not have today |
| Application Insights | Already configured, already collecting | It samples what the application does, not what the machine has; the memory counters it does carry are the ones this sampler would be reading anyway, an hour late |
| Leave it to the Performance overview | Nothing to build | The page quotes numbers a reader cannot check, which is the one thing this project does not do |
**Decision: sample the container, and ask each store for the reading it actually keeps.**
## The three readings, and what each one is worth
**The container.** A hosted service takes a sample every fifteen seconds and keeps the last two
hundred and forty, which is an hour in about twenty kilobytes. Each sample carries the working set
as the operating system sees it, the managed heap as the collector sees it, the share of the
container's processors this process used since the previous sample, the thread count and the
collection counts. The first sample reports no processor share: there is nothing to subtract from,
and a percentage averaged over the whole life of the process wearing an interval's label is a wrong
number rather than a missing one. The limit the percentages are read against is the runtime's own
`TotalAvailableMemoryBytes`, which is what the container group's definition granted.
**The relational store.** One SELECT against `sys.dm_db_resource_stats`, which returns the last hour
as processor, data, log, memory and worker percentages of what the tier allows. On the Basic tier
those are percentages of five DTUs, which is the honest way to read them: a number here is a share
of a very small machine. The view exists on Azure SQL Database and nowhere else, so on SQLite, on a
containerized SQL Server, and on a database that did not come up, the card says which of those it is
looking at instead of drawing a zero.
**The document store.** No memory, no processor, by the nature of the product: this card says so in
those words rather than leaving a gap a reader has to explain to themselves. What it shows instead is
the operations ring folded into request units a minute, beside the thousand request units a second
the free tier allows, so the one number that could ever cost money is the one on the card.
## Addendum, 2026-09-19, shipped as 1.0.0.150: the same readings, drawn
Steve, on the card as it first shipped: graphs of the processor and memory of every Azure resource
this site runs, one chart per resource, the way the site activity is drawn. Three tables of numbers
are a reading; a line is a shape, and the shape is what says whether a container is climbing.
One chart per resource, and the unit is the resource's own. The container's two lines are
percentages: memory as a share of the limit the container group granted, and the processor share
already measured that way, so both sit on one axis. The relational store's two are percentages
because that is what its own view reports, of what the tier allows. The document store has no
processor or memory to draw, so its chart is what it does have, request units a minute, on an axis
of its own, and the sentence beside it still says why the other two lines are missing.
A percentage chart keeps a full axis whatever the hour held: a quiet hour drawn against its own
maximum looks like a busy one, which is the chart lying with true numbers. The arithmetic lives in
`src/lib/machineChart.ts` with the activity chart's, and a missing reading breaks the line rather
than being drawn across, because the first processor share of a process has nothing to compare
against and a line through it would be a number nobody measured.
### What the first live read found
Two things, and both are in this version rather than in a note. The limit the container's memory is
read against is the runtime's own, `TotalAvailableMemoryBytes`, which on this container group is
1,057 MB against the 1,536 MB the group granted: .NET works to about seven tenths of a container's
memory by default, and a process that passes its own limit is the one that gets collected. The card
says which limit it is showing rather than letting a reader assume the other one.
And the relational store's resource view answered with a `SqlException` on both sites, so the card
showed its absent reading exactly as designed. Reading the view needs `VIEW DATABASE STATE`, which
this container's identity may not hold; the reason now goes to the container's log, where an
operator can read it, while the public card still says only the type. A reading that fails politely
and says where the reason is beats a card that shows a zero.
That last sentence was half wrong when it was written, and 1.0.0.151 is the correction: the log ring
this site serves keeps an exception's **type** and not its message, so "the container's log carries
the reason" was true of the container's stdout and not of anything a reader could open. The card now
carries the database's own error number instead, and for the two numbers this view answers
permission with, and 262, the one both live sites answered with when this shipped, a sentence with
the answer in it: the view needs `VIEW DATABASE STATE`,
and `db_datareader` and `db_datawriter`, the two roles this container's identity holds
(ADR: The SQL Server backend), do not carry it. A number names no server, which is why it can be on
a public page when a message cannot.
### The cast, and the four minutes it took
`sys.dm_db_resource_stats` returns its percentages as `decimal(5,2)`. The record read them into
doubles, which is an `InvalidCastException`, and the endpoint answered 500 on both live sites from
the minute the `VIEW DATABASE STATE` grant let the query run at all until the fix rolled. It could
not have failed earlier: before the grant the query never got as far as a row, so the shape of a
row was never tested against the real view. That is the class of defect a suite on SQLite cannot
catch and a live read can, and the live read is what caught it, in Steve's own Admin tab.
Two things changed. The statement casts each percentage to float, in the statement rather than in
the type, because these are percentages a chart draws and not money. And the page sweep asks for
the Admin tab's own readings now, so an endpoint that throws shows up as a page that is down at the
next roll rather than in a card somebody happens to be looking at.
## Addendum, 2026-09-20: a day, a week and a month, kept where a roll cannot empty them
Steve, on the card a day after it shipped: twenty-four hours, seven days and thirty days on the
machine charts, beside the hour. The hour is the sampler's ring, this process's own memory, and the
first roll after a bad night empties exactly the picture somebody wanted. On the move to one plan
(ADR: One plan, two sites) every roll that morning emptied the hour, and the memory readings that
decided the plan's size had to be taken by a script asking from outside every fifteen seconds.
**What is kept.** One document a minute from each site, in a `machines` container in Azure Cosmos DB
beside the log and the activity rows, partitioned on the UTC day, expiring after thirty-one days by
the container's own time-to-live, one day more than the longest window. Twenty seconds after a
minute ends, `MachineRecorder` folds it out of what the process already measures: the working set,
managed heap and processor share of the four samples the sampler took, the mean and the peak both
kept; the relational store's own rows for that minute; the request units the operations ring holds
for it; and the traffic the request ring holds for it, how many requests, their median and
ninety-fifth, and how many answered 4xx and 5xx, which is what the next charts are drawn from. Nothing new is measured for it except one read of the resource view a minute, on the
quiet context the activity counters use, so a read a minute outside any request is not the newest
line on the SQL card for ever.
**A figure nobody read stays absent.** The first processor share of a process, a relational reading
on a site that has none: these are null in a minute and left out of its document, not written as
zero and not written as null. The store's average skips a field a document does not carry, which
is the same rule the chart follows when it breaks its line over a gap, and a test against the real
account holds the grouped query to the folding this repository does itself.
**How a window is read.** Every minute is written with three bucket keys, five minutes, an hour and
four hours, so a day is 288 points, a week 168 and a month 180, and each is one `GROUP BY` over a key
the index holds rather than tens of thousands of documents read back to be averaged in the process. The
answer is cached for a minute, because a minute is how often it can change and the Admin tab is a
page somebody leaves open. `/api/admin/machines?window=24h`, `7d` or `30d`; with no window it is the
hour, as before.
**How a window is drawn.** Slot by slot from the window's start to now, with a gap wherever the store
holds nothing. Drawing only the buckets that exist would close every gap up and make a site that was
down for a day look like one that never was; the card says how many of the window's buckets hold a
reading. A process with no document store beside it keeps nothing and says so in words.
**What it costs.** One upsert a minute a site: 2,880 writes a day between them, against a free
allowance of a thousand request units a second, and about forty-five thousand small documents a site
at any one time against 25 GB. The bill does not move.
## Addendum, 2026-09-20 (1.0.0.166): a window is drawn from its first reading
The addendum above draws a kept window whole, "with null wherever the store holds nothing: an hour
the site was down, or the weeks before anything was kept". The first half of that holds. The second
was changed the day it shipped: the weeks before anything was kept are no longer drawn, because on
that day they were all there was to see. The reasoning and the rule are in
ADR: The Admin tab, as a product, the addendum on where a window starts. What a window holds is still
counted against the whole of it.
## What it costs
Nothing on the bill. The sampler is a timer in a process that is already running; the resource view
is one SELECT against a database already connected, excluded from the SQL card for the same reason
the health check is, so the card cannot fill the page with the act of reading it; the document
reading is arithmetic over a ring the container already keeps. No tier, no resource, no metrics API.
## Where it sits
The readings span four rings: MachineSampler and MachineRecorder sit in the host Api, the kept minute and the IMachineHistory port are in Application, CosmosMachineHistory in Infrastructure.Cosmos stores the minutes, and the SQL resource view now lives in Infrastructure as ResourceStats. Dependency inversion (the inner ring owns the interface and the outer ring implements it) means the recorder writes through IMachineHistory and a host with no document store gets NullMachineHistory instead. It cost a VIEW DATABASE STATE grant on Azure SQL and one upsert a minute per site, both inside what the bill already covers. Azure Monitor metrics would make sense once the site's identity holds a reader role and a second line on the bill is acceptable.
## Files
- [`api/TheYard.Api/Machines.cs`](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Api/Machines.cs): the sampler and the document reading.
- [`api/TheYard.Infrastructure/ResourceStats.cs`](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Infrastructure/ResourceStats.cs): the relational store's resource view, read by the adapter that owns the database.
- [`api/TheYard.Application/MachineHistory.cs`](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Application/MachineHistory.cs): a kept minute, a bucket, the windows, the folding and the port.
- [`api/TheYard.Infrastructure.Cosmos/CosmosMachineHistory.cs`](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Infrastructure.Cosmos/CosmosMachineHistory.cs) and [`infra/cosmos/machines.json`](https://github.com/SteveStout/TheYard/blob/main/infra/cosmos/machines.json): the minutes in the document store, the grouped query, and the container they live in.
- [`api/TheYard.Tests/MachineHistoryTests.cs`](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Tests/MachineHistoryTests.cs): the folding, the document, the cache, the endpoint, and the grouped query against the real account.
- [`api/TheYard.Api/Composition/ActivityRegistration.cs`](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Api/Composition/ActivityRegistration.cs): the sampler registered (region machine-sampler-wiring).
- [`api/TheYard.Api/Endpoints/AdminEndpoints.cs`](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Api/Endpoints/AdminEndpoints.cs): the endpoint (region machines-endpoint).
- [`api/TheYard.Api/Composition/StoreRegistration.cs`](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Api/Composition/StoreRegistration.cs): the quiet context factory that keeps the statement off the SQL card.
- [`src/components/admin/MachinesCard/MachinesCard.tsx`](https://github.com/SteveStout/TheYard/blob/main/src/components/admin/MachinesCard/MachinesCard.tsx): the card, three readings with their three honesties, and the chart each one is drawn in.
- [`src/lib/machineChart.ts`](https://github.com/SteveStout/TheYard/blob/main/src/lib/machineChart.ts): the chart arithmetic, React-free and tested on its own.
- [`api/TheYard.Tests/MachinesTests.cs`](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Tests/MachinesTests.cs): the ring, the first sample, the folding, and the shape the endpoint answers with.
- [`docs/decisions/ADR-077-every-page-checked.md`](https://github.com/SteveStout/TheYard/blob/main/docs/decisions/ADR-077-every-page-checked.md): the other card this morning added, and the sweep whose requests this one's numbers include.
- [`docs/performance/PERFORMANCE.md`](https://github.com/SteveStout/TheYard/blob/main/docs/performance/PERFORMANCE.md): the claim about one small container that this card is the running proof of.
```csharp
///
/// Keeps a minute at a time (ADR: What the machines are doing). The sampler's
/// ring is an hour of this process's memory and a roll empties it, so a day,
/// a week and a month could not be drawn from it.
/// Twenty seconds after each minute ends, this folds the minute that just
/// finished, out of what the process already measures, and hands it to the
/// port: the four samples, the relational store's own rows for that minute,
/// and what the document store charged in it.
///
/// Nothing new is measured for it except one read of the resource view a
/// minute, on the quiet context. With no store to keep minutes in it does
/// nothing at all, and asks again every ten minutes whether there is one.
///
public sealed class MachineRecorder(
MachineSampler sampler,
IMachineHistory history,
string site,
Func> relational,
Func> operations,
Func> requests,
ILogger logger) : BackgroundService
{
/// How long after a minute ends it is folded: long enough for its last sample and the store's last row to exist.
public static readonly TimeSpan Settle = TimeSpan.FromSeconds(20);
///
/// One minute out of the rings, or null when the sampler took no sample in
/// it, which is a process that was not running: there is nothing to keep,
/// and a gap on the chart is the true picture of a gap.
///
public static MachineMinute? Fold(
DateTimeOffset minute,
string site,
double memoryLimitMb,
IReadOnlyList samples,
IReadOnlyList rows,
IReadOnlyList operations,
IReadOnlyList? requests = null)
{
var start = minute.UtcMinute;
var end = start.AddMinutes(1);
var taken = samples.Where(sample => sample.At >= start && sample.At < end).ToList();
if (taken.Count == 0)
{
return null;
}
// The view's end_time is UTC with no kind on it.
var read = rows
.Where(row =>
{
var at = new DateTimeOffset(DateTime.SpecifyKind(row.At, DateTimeKind.Utc));
return at >= start && at < end;
})
.ToList();
var charged = operations.Where(operation => operation.At >= start && operation.At < end).ToList();
var traffic = TrafficMinutes.From((requests ?? []).Where(request => request.At >= start && request.At < end).ToList()).SingleOrDefault();
return new MachineMinute(
start,
site,
memoryLimitMb,
Math.Round(taken.Average(sample => sample.WorkingSetMb), 1),
Math.Round(taken.Max(sample => sample.WorkingSetMb), 1),
Math.Round(taken.Average(sample => sample.ManagedMb), 1),
MachineFolding.MeanOf(taken.Select(sample => sample.CpuPercent)),
MachineFolding.MaxOf(taken.Select(sample => sample.CpuPercent)),
MachineFolding.MeanOf(read.Select(row => (double?)row.CpuPercent)),
MachineFolding.MeanOf(read.Select(row => (double?)row.MemoryPercent)),
MachineFolding.MeanOf(read.Select(row => (double?)row.DataIoPercent)),
Math.Round(charged.Sum(operation => operation.RequestCharge), 2),
charged.Count,
traffic?.Requests ?? 0,
traffic?.P50Ms,
traffic?.P95Ms,
traffic?.ServerErrors ?? 0,
traffic?.ClientErrors ?? 0);
}
///
/// The loop: wait for a minute to end and settle, fold it, and hand it to
/// the history port; with no store to keep minutes in, wait ten minutes
/// and ask again. A failure is logged by type and the loop goes on.
///
protected override async Task ExecuteAsync(CancellationToken stopping)
{
while (!stopping.IsCancellationRequested)
{
try
{
if (!(await history.AvailabilityAsync(stopping)).Available)
{
await Task.Delay(TimeSpan.FromMinutes(10), stopping);
continue;
}
var now = DateTimeOffset.UtcNow;
await Task.Delay(now.UtcMinute.AddMinutes(1).Add(Settle) - now, stopping);
var minute = DateTimeOffset.UtcNow.UtcMinute.AddMinutes(-1);
var load = await relational(stopping);
var folded = Fold(minute, site, MachineSampler.MemoryLimitMb, sampler.Snapshot(), load.Rows, operations(), requests());
if (folded is not null)
{
await history.AppendAsync(folded, stopping);
}
}
catch (OperationCanceledException) when (stopping.IsCancellationRequested)
{
return;
}
catch (Exception ex)
{
// The type, never the message, and the loop goes on: a minute
// that could not be kept is a gap on a chart, not an outage.
logger.LogWarning("A minute of the machines could not be kept ({Exception})", ex.GetType().Name);
await Task.Delay(TimeSpan.FromMinutes(1), stopping).ContinueWith(_ => { }, TaskScheduler.Default);
}
}
}
}
///
/// The kept windows, read back for the card, each cached for a minute: the
/// Admin tab is a page somebody leaves open, a month is a grouped query over
/// forty thousand documents, and the answer cannot change more often than a
/// minute is written.
///
public sealed class MachineHistoryReader(IMachineHistory history, string site)
{
/// The lock that guards the cache, since requests read and fill it at the same time.
private readonly object _gate = new();
/// The last answer for each window, by the window's name, with when it was read.
private readonly Dictionary _cached = new(StringComparer.Ordinal);
///
/// One window as the card reads it: an unknown or hourly window answers
/// empty (the hour comes from the sampler's ring, not from here), a store
/// that keeps nothing answers with its reason, and anything else is the
/// window's buckets and totals, served from the cache when under a minute old.
///
public async Task