# ADR: Backends, side by side
Status: accepted, 2026-09-08. One card at the top of the Admin tab, on both
sites, with this container and its peer on the same rows: the store, the cold
start, the seed and what it cost, and how long the things a visitor does take
on each, with the request charge beside every number the document store can put
one on. Parent: ADR: A second store on Cosmos DB, and what it costs.
## In plain words
One card on the Admin tab puts the two backends side by side (the SQL database and the document database, Azure Cosmos DB) and shows how long each thing a visitor does takes on each. Each site reads its partner's numbers through its own API, so either tab shows both.
What that is worth: a developer can compare the two stores row by row on live numbers, and the organization can show the comparison on its own site instead of in a pasted document.
## Context
Steve wanted the comparison built into the product rather than pasted into a
document, and he wanted the same Admin section on both sites. One component, one
image, deployed twice: whichever tab he opens, he sees both backends.
## Decision
**The peer is read through this container's API, not from the browser.**
`/api/admin/peer` fetches the peer's `/api/admin/metrics` server side and
returns it. That keeps CORS closed on both origins and keeps the peer's address
out of the client bundle: one configuration value per container, `Peer:Url`,
holds the peer's public origin, which is a URL and not a credential, and the
browser only ever learns the peer's host name.
```csharp
///
/// The other container, read through this one's API rather than from the
/// browser (ADR: Backends, side by side). Two reasons. Keeping the read server
/// side keeps CORS closed on both origins, and it keeps the peer's address out
/// of the client bundle: one configuration value per container holds the
/// peer's public FQDN, which is a URL and not a credential.
///
/// The peer is a thing that can be down. No peer configured, no answer,
/// a slow answer and a wrong answer are four different sentences on the card,
/// and none of them is an exception: the rest of the Admin tab renders whatever
/// this returns. Two and a half seconds, not a hang, because the card refreshes
/// every thirty and a peer that takes longer than that is a peer that is
/// down for the purposes of a comparison.
///
public sealed class PeerReader(string? configuredUrl, HttpClient client)
{
public static readonly TimeSpan Patience = TimeSpan.FromSeconds(2.5);
/// The peer's origin, or null when none is configured or the deploy left its placeholder behind.
public Uri? Url { get; } =
string.IsNullOrWhiteSpace(configuredUrl) || configuredUrl.StartsWith("__", StringComparison.Ordinal)
? null
: Uri.TryCreate(configuredUrl.TrimEnd('/'), UriKind.Absolute, out var url) ? url : null;
public async Task ReadAsync()
{
if (Url is null)
{
return new PeerView(false, false, "no peer is configured on this container", null, DateTimeOffset.UtcNow, null, null);
}
string host = Url.Host;
try
{
using var timeout = new CancellationTokenSource(Patience);
using var response = await client.GetAsync(new Uri(Url, "/api/admin/metrics"), timeout.Token);
if (!response.IsSuccessStatusCode)
{
return new PeerView(true, false, $"the peer answered {(int)response.StatusCode}", host, DateTimeOffset.UtcNow, null, null);
}
var metrics = await response.Content.ReadFromJsonAsync(cancellationToken: timeout.Token);
string? version = await VersionAsync(timeout.Token);
return new PeerView(true, true, null, host, DateTimeOffset.UtcNow, metrics, version);
}
catch (OperationCanceledException)
{
return new PeerView(true, false, $"the peer did not answer within {Patience.TotalSeconds:0.#} seconds", host, DateTimeOffset.UtcNow, null, null);
}
catch (Exception ex) when (ex is HttpRequestException or JsonException or InvalidOperationException)
{
// The type, never the message: a message here carries a host name
// and a port, and this is a public page.
return new PeerView(true, false, $"the peer could not be read ({ex.GetType().Name})", host, DateTimeOffset.UtcNow, null, null);
}
}
///
/// The peer's build, from its public /api/version, inside the same patience as its metrics.
/// The metrics answer carries no version, and two sites on different builds explain most
/// differences between them, so the card asks. A version it cannot read is null, never a
/// failure: the comparison stands without it.
///
private async Task VersionAsync(CancellationToken token)
{
try
{
using var response = await client.GetAsync(new Uri(Url!, "/api/version"), token);
if (!response.IsSuccessStatusCode)
{
return null;
}
var body = await response.Content.ReadFromJsonAsync(cancellationToken: token);
return body.TryGetProperty("version", out var version) && version.ValueKind == JsonValueKind.String
? version.GetString()
: null;
}
catch (Exception ex) when (ex is HttpRequestException or JsonException or InvalidOperationException or OperationCanceledException)
{
return null;
}
}
}
/// What the comparison card learns about the peer: whether there is one, whether it answered, and if so its metrics as it reported them.
/// True when this container has a peer to compare with.
/// True when the peer answered with its metrics.
/// Why the peer could not be read, or null when it was.
/// The peer's host name, or null when none is configured.
/// When the read was attempted, UTC.
/// The peer's metrics as it reported them, or null when it could not be read.
/// The peer's build from its /api/version, or null when it could not be read.
public sealed record PeerView(bool Configured, bool Reachable, string? Reason, string? Host, DateTimeOffset FetchedAt, JsonElement? Metrics, string? Version);
```
*Live from [`api/TheYard.Api/Peer.cs`](https://github.com/SteveStout/TheYard/blob/1d6b50b/api/TheYard.Api/Peer.cs#L8-L99), region `peer`, read from this build at 1d6b50b.*
**The peer is a thing that can be down.** ADR: The relational store established
that the site comes up when the database does not, and the same rule holds
here: no peer configured, no answer, a slow answer and a wrong answer are four
different sentences on the card, and none of them is an exception. Two and a
half seconds of patience, not a hang, because the card refreshes every thirty
seconds and a peer that takes longer than that is down for the purposes of a
comparison. The rest of the Admin tab renders whatever the peer says.
The tests were written in the order the record asks for, the failures first:
```csharp
[Fact]
public async Task With_no_peer_configured_the_endpoint_says_so_and_the_rest_of_admin_is_untouched()
{
await using var api = Api(null);
var client = api.CreateClient();
var peer = await client.GetFromJsonAsync("/api/admin/peer");
Assert.False(peer.GetProperty("configured").GetBoolean());
Assert.False(peer.GetProperty("reachable").GetBoolean());
Assert.Contains("no peer", peer.GetProperty("reason").GetString());
var health = await client.GetAsync("/api/health");
Assert.Equal(HttpStatusCode.OK, health.StatusCode);
}
[Fact]
public async Task A_placeholder_the_deploy_left_behind_is_no_peer()
{
await using var api = Api("__PEER_URL__");
var peer = await api.CreateClient().GetFromJsonAsync("/api/admin/peer");
Assert.False(peer.GetProperty("configured").GetBoolean());
}
[Fact]
public async Task An_unreachable_peer_is_a_sentence_within_the_patience_not_a_hang()
{
// Port 9 is discard; nothing listens there and the connection is refused.
await using var api = Api("http://127.0.0.1:9");
var client = api.CreateClient();
var clock = Stopwatch.StartNew();
var peer = await client.GetFromJsonAsync("/api/admin/peer");
clock.Stop();
Assert.True(peer.GetProperty("configured").GetBoolean());
Assert.False(peer.GetProperty("reachable").GetBoolean());
Assert.False(string.IsNullOrWhiteSpace(peer.GetProperty("reason").GetString()));
// A refused connection answers in milliseconds on an idle machine. The
// margin is for the suite, not the code: with two stores booting in
// every application at once, a timer's callback and a refused socket
// both wait their turn on the thread pool, and this took 4.6 seconds
// once (ADR: One container, both stores). The claim is "not a hang",
// which ten seconds still holds against a peer that never answers.
Assert.True(clock.Elapsed < PeerReader.Patience + TimeSpan.FromSeconds(8), $"took {clock.Elapsed}");
// The type of the failure, never its message: a message names a host and a port.
Assert.DoesNotContain("127.0.0.1:9", peer.GetProperty("reason").GetString());
}
[Fact]
public async Task A_peer_that_answers_slowly_is_reported_as_slow_and_nothing_waits_for_it()
{
// Twenty seconds, against a patience of two and a half. The assertion
// below is generous on purpose: what it proves is that nothing waits
// for the peer's full answer, and a tight number would measure the
// machine, which under the whole suite in parallel once took four
// seconds to fire a timer (ADR: The SQL Server backend, the addendum on
// the CI failure that could not explain itself).
using var slow = new CannedPeer(delay: TimeSpan.FromSeconds(20));
await using var api = Api(slow.Origin);
// The client first, so the clock measures the peer read and not the
// host coming up: creating the client is what boots Program.cs, and
// that migrates, seeds and indexes a hundred thousand vehicles.
var client = api.CreateClient();
var clock = Stopwatch.StartNew();
var peer = await client.GetFromJsonAsync("/api/admin/peer");
clock.Stop();
Assert.False(peer.GetProperty("reachable").GetBoolean());
Assert.Contains("did not answer", peer.GetProperty("reason").GetString());
Assert.True(clock.Elapsed < TimeSpan.FromSeconds(15), $"took {clock.Elapsed}");
}
```
*Live from [`api/TheYard.Tests/PeerTests.cs`](https://github.com/SteveStout/TheYard/blob/1d6b50b/api/TheYard.Tests/PeerTests.cs#L29-L100), region `peer-down`, read from this build at 1d6b50b.*
and the happy path last, against a canned peer on the loopback:
```csharp
[Fact]
public async Task A_peer_that_answers_is_relayed_whole_with_its_host_and_nothing_else_about_its_address()
{
using var canned = new CannedPeer(delay: TimeSpan.Zero);
await using var api = Api(canned.Origin);
var peer = await api.CreateClient().GetFromJsonAsync("/api/admin/peer");
Assert.True(peer.GetProperty("reachable").GetBoolean());
Assert.Equal("127.0.0.1", peer.GetProperty("host").GetString());
Assert.Equal("Azure Cosmos DB", peer.GetProperty("metrics").GetProperty("store").GetProperty("store").GetString());
// Two reads: the metrics, then the build, so the card can name both sites' versions.
Assert.Equal("1.0.3.999", peer.GetProperty("version").GetString());
Assert.Equal(2, canned.Hits);
}
```
*Live from [`api/TheYard.Tests/PeerTests.cs`](https://github.com/SteveStout/TheYard/blob/1d6b50b/api/TheYard.Tests/PeerTests.cs#L104-L118), region `peer-up`, read from this build at 1d6b50b.*
**The rows are routes, not paths.** A bid on one vehicle and a bid on another
are the same operation, and a card that listed a hundred thousand paths would
say nothing. Timing is grouped by route template, with the identifiers taken
out, and the store log's operations are grouped the same way so the document
column can say what a route costs in request units as well as milliseconds:
```csharp
///
/// Timing by route rather than by path, for the rows the comparison card puts
/// side by side: a bid on one vehicle and a bid on another are the same
/// operation, and a card that listed a hundred thousand paths would say
/// nothing (ADR: Backends, side by side).
///
public static partial class Routes
{
[GeneratedRegex("^/api/vehicles/[^/]+")]
private static partial Regex VehicleId();
[GeneratedRegex("^/api/docs/diagrams/[^/]+")]
private static partial Regex DiagramName();
[GeneratedRegex("^/api/docs/[^/]+")]
private static partial Regex DocSlug();
[GeneratedRegex("^/api/images/.+")]
private static partial Regex ImageFile();
public static string TemplateOf(string path)
{
string template = VehicleId().Replace(path, "/api/vehicles/{id}");
template = DiagramName().Replace(template, "/api/docs/diagrams/{name}");
if (!template.StartsWith("/api/docs/diagrams/", StringComparison.Ordinal))
{
template = DocSlug().Replace(template, "/api/docs/{slug}");
}
return ImageFile().Replace(template, "/api/images/{file}");
}
public static string Of(string method, string path) => $"{method} {TemplateOf(path)}";
/// Per-route timings, busiest first.
public static IReadOnlyList ByRoute(IReadOnlyList requests) =>
requests
.GroupBy(entry => Of(entry.Method, entry.Path), StringComparer.Ordinal)
.Select(group =>
{
long[] durations = group.Select(entry => entry.DurationMs).ToArray();
return new RouteTiming(group.Key, durations.Length, Percentiles.Of(durations, 50), Percentiles.Of(durations, 95), durations.Max());
})
.OrderByDescending(timing => timing.Count)
.ThenBy(timing => timing.Route, StringComparer.Ordinal)
.ToArray();
///
/// What each route costs the document store, from the operations the store
/// log attributes to it: how many operations a request runs and the request
/// units they add up to, as the median over the requests seen. Requests are
/// told apart by their trace identifier, so twenty sign-ins are twenty
/// samples and not one sum. Grouping by the path string instead would
/// report twenty sign-ins as one request costing the sum of all twenty.
///
public static IReadOnlyList ChargesByRoute(IReadOnlyList operations) =>
operations
.Where(operation => operation.Request is not null)
.GroupBy(operation => (operation.RequestId ?? operation.Request!, operation.Request!))
.Select(perRequest =>
{
var parts = perRequest.Key.Item2.Split(' ', 2);
string route = parts.Length == 2 ? Of(parts[0], parts[1]) : perRequest.Key.Item2;
return (Route: route, Operations: perRequest.Count(), Charge: perRequest.Sum(operation => operation.RequestCharge), CrossPartition: perRequest.Count(o => o.Partition.StartsWith("cross", StringComparison.Ordinal)));
})
.GroupBy(sample => sample.Route, StringComparer.Ordinal)
.Select(group =>
{
double[] charges = group.Select(sample => sample.Charge).OrderBy(charge => charge).ToArray();
return new RouteCharge(
group.Key,
group.Count(),
Math.Round(group.Select(sample => (double)sample.Operations).Average(), 1),
Math.Round(charges[(int)Math.Ceiling(charges.Length * 0.5) - 1], 2),
Math.Round(charges[^1], 2),
group.Sum(sample => sample.CrossPartition));
})
.OrderByDescending(charge => charge.Requests)
.ThenBy(charge => charge.Route, StringComparer.Ordinal)
.ToArray();
}
/// One route's request timings for the comparison card.
/// The method and route template, such as GET /api/vehicles/{id}.
/// How many requests to the route are in the window.
/// The median duration, in milliseconds.
/// The 95th percentile duration, in milliseconds.
/// The slowest duration, in milliseconds.
public sealed record RouteTiming(string Route, int Count, long P50Ms, long P95Ms, long MaxMs);
/// What one route costs the document store per request, for the comparison card.
/// The method and route template, such as GET /api/vehicles/{id}.
/// How many requests to the route were seen.
/// The average number of store operations per request, to one decimal place.
/// The median request units per request.
/// The most request units any one request spent.
/// How many of the route's operations crossed partitions, across all its requests.
public sealed record RouteCharge(string Route, int Requests, double OperationsPerRequest, double RuP50, double RuMax, int CrossPartition);
```
*Live from [`api/TheYard.Api/Peer.cs`](https://github.com/SteveStout/TheYard/blob/1d6b50b/api/TheYard.Api/Peer.cs#L103-L199), region `routes`, read from this build at 1d6b50b.*
The request-unit figure per route is exact per request: every operation the
store log records carries the trace identifier ASP.NET Core gives the request
that caused it, so twenty sign-ins are twenty samples of 2 RU and not one sum
of 40. The first version of this card grouped by method and path instead and
reported twenty sign-ins as a single 34 RU request, which the first
measurement session caught (ADR: Measuring both stores).
**Cold start is measured on each container, at its own start.** How long the
store took to answer, how long the catalogue and the bids took to load, and
when the container was ready to serve, all counted from the process's own
start time and exposed in the metrics, so the two columns compare two real
cold starts rather than one stopwatch and one memory:
```csharp
///
/// How long each part of coming up took, and when the container was ready to
/// serve, measured from the process's own start. The comparison card shows
/// these for both containers on the same rows (ADR: Backends, side by side).
///
public sealed class StartupTimings
{
/// When this process started, UTC, which every figure here is measured from.
private readonly DateTime _processStart = System.Diagnostics.Process.GetCurrentProcess().StartTime.ToUniversalTime();
/// How long each named step took, in milliseconds.
private readonly Dictionary _steps = new(StringComparer.Ordinal);
/// Milliseconds from process start to the point the host finished warming, or null until then.
public long? ReadyMs { get; private set; }
///
/// Runs one step of coming up, records how long it took under , and
/// passes its result on.
///
public async Task Time(string step, Func> work)
{
var clock = System.Diagnostics.Stopwatch.StartNew();
var result = await work();
_steps[step] = clock.ElapsedMilliseconds;
return result;
}
///
/// Runs one step of coming up that returns nothing and records how long it took under .
///
public async Task Time(string step, Func work)
{
var clock = System.Diagnostics.Stopwatch.StartNew();
await work();
_steps[step] = clock.ElapsedMilliseconds;
}
/// Marks the host warm: becomes the time since the process started.
public void Ready() => ReadyMs = (long)(DateTime.UtcNow - _processStart).TotalMilliseconds;
/// How long a step took in milliseconds, or null when it has not run.
public long? Ms(string step) => _steps.TryGetValue(step, out long ms) ? ms : null;
}
```
*Live from [`api/TheYard.Api/AdminObservability.cs`](https://github.com/SteveStout/TheYard/blob/1d6b50b/api/TheYard.Api/AdminObservability.cs#L214-L258), region `startup-timings`, read from this build at 1d6b50b.*
**The card.** Two columns, this container first, the peer second, on rows for
the store, the cold start, the store check, the seed with its charge, the
catalogue and bid loads, the requests window, the seven routes a visitor
actually takes, and the store operations window:
```tsx
/** The rows the comparison card puts side by side: the paths a visitor actually takes (ADR: Backends, side by side). */
const COMPARED_ROUTES: { route: string; label: string }[] = [
{ route: 'GET /api/vehicles', label: 'Listing page' },
{ route: 'GET /api/vehicles/{id}', label: 'Vehicle page' },
{ route: 'GET /api/facets', label: 'Filter values' },
{ route: 'POST /api/vehicles/{id}/bids', label: 'Bid write' },
{ route: 'POST /api/auth/login', label: 'Sign in' },
{ route: 'POST /api/auth/register', label: 'Register' },
{ route: 'POST /api/market/tick', label: 'Room tick' },
];
/**
* One column of the comparison: a store, wherever it runs, on the rows the card
* draws. A timing is null when the store's window is empty, which the column
* says in words rather than as a zero.
*/
type StoreColumn = {
title: string;
store: string;
startup: Startup;
requests: {
window: number;
p50_ms: number | null;
p95_ms: number | null;
by_route: RouteTiming[];
};
charges: RouteCharge[];
storeOps: string;
};
function storeOpsLine(store: StoreSummary | null, sql: SqlSummary | null): string {
if (store !== null) {
return `${percentileWords(store.p50_ms, store.p95_ms, 'no operations yet')}, ${store.ru_total} RU over ${store.window}, ${store.cross_partition} cross-partition`;
}
if (sql !== null) {
return `${percentileWords(sql.p50_ms, sql.p95_ms, 'no statements yet')} over ${sql.window} statements`;
}
return '';
}
/** A whole container's metrics as one column, which is what a peer answers with. */
function columnOf(title: string, m: Metrics): StoreColumn {
const cosmos = m.store.store === 'Azure Cosmos DB';
return {
title,
store: m.store.store,
startup: m.startup,
requests: m.requests,
charges: cosmos ? m.store_by_route : [],
storeOps: storeOpsLine(cosmos ? m.store : null, cosmos ? null : m.sql),
};
}
/** One store of a container that runs more than one (ADR: One container, both stores). */
function columnOfBackend(backend: BackendMetrics, current: boolean): StoreColumn {
return {
title: current ? `${backend.store}, serving this visit` : backend.store,
store: backend.store,
startup: backend.startup,
requests: backend.requests,
charges: backend.store_by_route,
storeOps: storeOpsLine(backend.store_metrics, backend.sql),
};
}
/**
* The columns the card compares. A container running both stores compares
* them with each other, in one process, the current one first; a container
* running one compares itself with its peer, whatever /api/admin/peer relayed,
* and when it relayed nothing the column says why and the rest of the card
* stands (ADR: Backends, side by side).
*/
function columns(
mine: Metrics,
peer: Peer | null
): { columns: StoreColumn[]; note: string | null } {
const backends = mine.backends ?? [];
if (backends.length > 1) {
const current = backends.find((b) => b.store === mine.store.store) ?? backends[0];
const ordered = [current, ...backends.filter((b) => b !== current)];
return {
columns: ordered.map((b) => columnOfBackend(b, b === current)),
note: 'Both stores run in this container, so the two columns share a process, a region and a request ring; only the store differs.',
};
}
const theirs = peer !== null && peer.reachable ? peer.metrics : null;
const peerTitle =
peer === null
? 'The peer could not be read'
: !peer.configured
? 'No peer is configured on this container'
: peer.reachable
? `The other site (${peer.metrics?.store.store ?? 'store unknown'})`
: `The other site is not answering`;
const peerNote = peer !== null && peer.configured && !peer.reachable ? peer.reason : null;
const empty: StoreColumn = {
title: peerTitle,
store: '',
startup: {
...mine.startup,
ready_ms: null,
schema_ms: 0,
seed_ms: 0,
seed_ru: null,
catalogue_ms: null,
bids_ms: null,
},
requests: { window: 0, p50_ms: null, p95_ms: null, by_route: [] },
charges: [],
storeOps: '',
};
return {
columns: [
columnOf(`This site (${mine.store.store})`, mine),
theirs === null ? empty : columnOf(peerTitle, theirs),
],
note: peerNote,
};
}
/** A line of the comparison: what is measured, and its reading in each store's column. */
type ComparedRow = { label: string; cells: (column: StoreColumn) => string };
function Comparison({ mine, peer }: { mine: Metrics; peer: Peer | null }) {
const { columns: cols, note } = columns(mine, peer);
const blank = (column: StoreColumn) => column.store === '';
const routeCell = (column: StoreColumn, route: string): string => {
if (blank(column)) return '';
const timing = column.requests.by_route.find((r) => r.route === route);
if (timing === undefined) return 'not seen yet';
const charge = column.charges.find((r) => r.route === route);
const time = `${percentileWords(timing.p50_ms, timing.p95_ms)} (${timing.count})`;
return charge === undefined ? time : `${time} ยท ${charge.ru_p50} RU`;
};
const rows: ComparedRow[] = [
{ label: 'Store', cells: (c) => c.store },
{
label: 'Cold start, process start to ready',
cells: (c) => (blank(c) ? '' : ms(c.startup.ready_ms)),
},
{
label: 'Store check (schema or containers)',
cells: (c) => (blank(c) ? '' : ms(c.startup.schema_ms)),
},
{
label: 'Seed, first boot only',
cells: (c) => (blank(c) ? '' : msAndRu(c.startup.seed_ms, c.startup.seed_ru)),
},
{ label: 'Catalogue load', cells: (c) => (blank(c) ? '' : ms(c.startup.catalogue_ms)) },
{ label: 'Bids load', cells: (c) => (blank(c) ? '' : ms(c.startup.bids_ms)) },
{
label: 'Requests, all paths',
cells: (c) =>
blank(c)
? ''
: `${percentileWords(c.requests.p50_ms, c.requests.p95_ms)} (${c.requests.window})`,
},
...COMPARED_ROUTES.map((entry) => ({
label: entry.label,
cells: (c: StoreColumn) => routeCell(c, entry.route),
})),
{ label: 'Store operations, this window', cells: (c) => c.storeOps },
];
return (
row.label}
columns={[
{ name: 'Measured', cell: (row) => row.label },
...cols.map((column): Column => ({
name: column.title,
mono: true,
cell: (row) => row.cells(column),
})),
]}
note={note === null ? undefined : { content: note, testId: 'peer-note' }}
/>
);
}
/** This site's build beside the other site's, the first thing to rule out when two columns differ. */
function versionsLine(own: string | null, peer: Peer | null): string {
const mine = own === null ? 'this site: version not read' : `This site runs v${own}`;
if (peer === null || !peer.configured) return `${mine}; no other site to compare.`;
if (!peer.reachable || peer.version === undefined || peer.version === null) {
return `${mine}; the other site's version could not be read.`;
}
return `${mine}; the other site runs v${peer.version}${own === peer.version ? ', the same build' : ''}.`;
}
/** The process under both columns: when it started, and how much it compiled for itself. */
function runtimeLine(m: Metrics): string {
const started = `This process started ${clockLabel(m.startup.started_at)}`;
const prepared = m.startup.prepare_ms === null ? '' : `, prepared in ${ms(m.startup.prepare_ms)}`;
if (m.runtime === undefined) return `${started}${prepared}.`;
const r = m.runtime;
return `${started}${prepared}; the runtime compiled ${r.jit_methods.toLocaleString('en-US')} methods itself in ${ms(r.jit_ms)}, ${r.gc} garbage collection${r.gc_concurrent ? ', concurrent' : ''}, ${r.processors} processor${r.processors === 1 ? '' : 's'}.`;
}
```
*Live from [`src/components/admin/BackendsCard/BackendsCard.tsx`](https://github.com/SteveStout/TheYard/blob/1d6b50b/src/components/admin/BackendsCard/BackendsCard.tsx#L24-L224), region `comparison`, read from this build at 1d6b50b.*
## What it looks like
Both cards on 1.0.0.92, taken a minute apart after six paired rounds of the
measurement script had given every row on both sides something to show. The
live site first, reading its peer:
And the same card on the Cosmos DB container, where the columns swap and the
request charge sits in the first column:
Read across any row and the two containers agree with each other about the
other one, which is the property the peer endpoint exists to give: each site
reads the other's metrics rather than remembering them. The bid write row is
the one to look at twice. Server side the document store spends 11 ms on a
bid and the relational one 86, and the measurement record shows what is left
of that difference by the time it reaches a visitor in Missouri
(ADR: Measuring both stores).
## Consequences
- Both sites carry the same card, and opening either shows both backends.
- The comparison is live, not a screenshot: every number is the last few
hundred requests each container has seen, and the cold start is the one it
actually had.
- `/api/admin/peer` and `/api/admin/store` are excluded from the timing rings
like the other observability reads, or the card would fill with itself.
- A peer address is one more environment variable on each container group,
set in the deploy, never in the repository.
- The one thing this card cannot do is compare on equal traffic. The live SQL
site has visitors and the Cosmos one has whoever is comparing; the measurement
record runs the same requests against both in one session for that reason
(ADR: Measuring both stores).
## Addendum, 2026-09-08: the two columns moved into one process
The evening's ask was a toggle at the top of the page, and the answer to it
(ADR: One container, both stores) changed what this card compares. A
container that runs both stores now puts them on these same rows with each
other, the one serving the visit first, and says in a note that the two
columns share a process, a region and a request ring: only the store
differs. The request ring records which store served each request, so one
ring is split two ways rather than two rings being read from two places.
Nothing above is withdrawn. The peer endpoint, its patience and its four
sentences are what a container running one store still uses, and the second
container still runs the same image with the other default, so the two-tab
comparison this record was written for still works. What the in-process
comparison adds is the one thing the two-container one could not have: a
measurement with no network in it.
## Addendum, 2026-09-09: parity, every number for both stores
The morning after both stores went into every container, Steve asked for the
Admin tab to show every figure for Cosmos DB that it shows for SQL Server.
The Admin tab was inventoried card by card on both sites before
anything changed, from the page's source and from each site's own answers
to the endpoints behind it, and each number marked as belonging to one
store, to both, or to the container:
- **Backends, side by side**: both, by construction; the columns are the
backends, and the store-operations row already put the relational window
and the document window on the same row.
- **Same performance, proven**: both, by construction.
- **The partition key, live**: the document store's alone, and it stays so;
the ask is that the relational store never has a statistic the document
store lacks, not the other way round.
- **Application health**: both, one check per store with its duration.
- **Azure's view of the container**, **Traffic, last hour**, **Recent
errors**: the container's, not a store's.
- **Timing**: the relational store's alone. The card said "SQL: p50, p95,
slowest" from the SQL ring, and on the site whose default store is the
document one that line was about the other store while the store serving
the visit got nothing. The window sentence named the request ring and the
SQL ring and not the store ring. This was the gap.
- **The SQL this application ran** and **What the document store ran**:
both, on both sites. The document store's card was already drawn whenever
the container had a document store, whichever store served the visit; its
rule compared the log's label to a name, and the SQL card's rule read the
backends list, so the two rules were the same rule written two ways.
- The page's opening paragraph described the SQL statements and the log and
did not mention the operations at all.
What changed. The Timing card has a document store line under the SQL line,
with the same three numbers and the two only that store can put a figure on:
the request units the window cost, and how many of its operations fanned out
across partitions rather than reading one. Its window sentence names all
three rings. On a container with no document store, a developer's machine
or CI, the line says so in words rather than going missing, which is the
standard the comparison card already held. The words are arithmetic in
`src/lib/metrics.ts`, read from the backend that has a document store and not
from the top-level block, whose label is the store serving the visit while
its numbers are always the document ring's; a peer on an older build with no
backends list falls back on the label. The document store card's rule reads
the same backends list the SQL card's does, with the label as the fallback.
The opening paragraph names both logs and states the standard.
```ts
/**
* The document store's window on this container, or null when it runs none.
* The per-backend list is the authority: the top-level `store` block is
* labelled with the store serving the visit, which on a container running
* both is the relational one half the time, while its numbers are always
* the document store ring's. A peer on an older build has no list, and for
* it the label is all there is.
*/
export function documentStore(metrics: TimingMetrics): StoreWindow | null {
if (metrics.backends !== undefined) {
const backend = metrics.backends.find((candidate) => candidate.store_metrics !== null);
return backend?.store_metrics ?? null;
}
return metrics.store.store === 'Azure Cosmos DB' ? metrics.store : null;
}
/** The sentence that says what the card's numbers are measured over: every ring the container keeps. */
export function timingWindow(metrics: TimingMetrics): string {
const store = documentStore(metrics);
const rings = [
`${metrics.requests.window} requests`,
`${metrics.sql.window} SQL statements`,
store === null
? 'no document store on this container'
: `${store.window} document store operations`,
];
return `Measured in this process, over the last ${rings[0]}, ${rings[1]} and ${rings[2]}, with the endpoints this page reads left out so it does not fill with the act of being read.`;
}
/** The relational store's line: the same three numbers it has always shown, or that it has none yet. */
export function sqlLine(metrics: TimingMetrics): string {
const sql = metrics.sql;
if (sql.p50_ms === null || sql.p95_ms === null) {
return 'SQL: no statements in the ring yet, so there is nothing to time.';
}
return `SQL: ${percentileWords(sql.p50_ms, sql.p95_ms)}, slowest ${timingWords(sql.max_ms, 'none')}.`;
}
/**
* The document store's twin of the SQL line, plus the two numbers only it
* can put a figure on: what the window cost in request units, and how many
* of its operations fanned out across partitions instead of reading one.
*/
export function documentStoreLine(metrics: TimingMetrics): string {
const store = documentStore(metrics);
if (store === null) {
return 'Document store: none on this container, so there is no operation to time.';
}
const fanned =
store.window === 0
? 'no operations yet'
: `${store.cross_partition} of ${store.window} operations fanned out across partitions`;
const timing =
store.p50_ms === null || store.p95_ms === null
? 'nothing to time'
: `${percentileWords(store.p50_ms, store.p95_ms)}, slowest ${timingWords(store.max_ms, 'none')}`;
return `Document store: ${timing}, ${store.ru_total} RU over the window, ${fanned}.`;
}
```
*Live from [`src/lib/metrics.ts`](https://github.com/SteveStout/TheYard/blob/1d6b50b/src/lib/metrics.ts#L63-L120), region `document-store-window`, read from this build at 1d6b50b.*
What holds it: `src/lib/metrics.test.ts` for the words, including the
container with no document store and the quiet window;
`AdminObservabilityTests` for the shape the card draws from, every store a
container runs answering a window of its own with the fields the lines need,
both kinds at once on a two-store container; and one assertion in
`tests/e2e/admin.spec.ts` on the two lines, which the gate runs on SQLite
alone and again with both stores.
**Before and after, from both sites.** The Timing card on 1.0.0.100, at
08:15 CDT on 2026-09-09, on the live site and on the second site: one store's
line, and on the second site that line is about the store not serving the
visit:
The same card on 1.0.0.101, at 08:47 CDT, both sites: the window sentence
names all three rings, and the document store has its line under the SQL
line with the request units and the fan-out count beside the percentiles:
## Where it sits
The comparison is a host and front end job: `Peer.cs` in the Api reads the other site's `/api/admin/metrics` server side and groups timings by route template, `AdminObservability.cs` records each container's own cold start, and `BackendsCard` with `src/lib/metrics.ts` draws the two columns in the browser. Single responsibility, one reason to change per class, shows in `Peer.cs` owning the peer and nothing else, including the four ways a peer can fail to answer, so the rest of the Admin tab never has to know a second site exists. It cost a `Peer:Url` setting on each site, a two and a half second wait before a slow peer counts as down, and a comparison that is never on equal traffic because the two sites have different visitors. Since both stores now run in one process (the 2026-09-08 addendum), a team starting fresh could rely on the in-process columns alone and keep the peer read only for sites that run one store.
## Files
- [`api/TheYard.Api/Peer.cs`](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Api/Peer.cs): the peer reader and the route grouping.
- [`api/TheYard.Api/AdminObservability.cs`](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Api/AdminObservability.cs): the startup timings and the store window.
- [`api/TheYard.Api/Endpoints/AdminEndpoints.cs`](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Api/Endpoints/AdminEndpoints.cs): the endpoint (region peer-endpoint).
- [`api/TheYard.Api/MetricsReport.cs`](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Api/MetricsReport.cs): the metrics it relays.
- [`src/components/admin/BackendsCard/BackendsCard.tsx`](https://github.com/SteveStout/TheYard/blob/main/src/components/admin/BackendsCard/BackendsCard.tsx) and [`src/components/admin/TimingCard/TimingCard.tsx`](https://github.com/SteveStout/TheYard/blob/main/src/components/admin/TimingCard/TimingCard.tsx): the card, and the Timing card's two store lines.
- [`src/lib/metrics.ts`](https://github.com/SteveStout/TheYard/blob/main/src/lib/metrics.ts): the Timing card's words, with [`src/lib/metrics.test.ts`](https://github.com/SteveStout/TheYard/blob/main/src/lib/metrics.test.ts) beside it.
- [`api/TheYard.Tests/PeerTests.cs`](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Tests/PeerTests.cs): down first, up last.
- [`infra/aci-theyard-cosmos.yaml`](https://github.com/SteveStout/TheYard/blob/main/infra/aci-theyard-cosmos.yaml): the second container group, with its peer setting.