# ADR: Observability, the Admin tab

Status: roughed in 2026-09-01, the evening it was asked for. A polish pass
is planned for the Thursday build day; this records the shape and the
reasoning so the rough-in is a decision, not an accident.

## In plain words

The site has a public Admin tab that shows its own condition: the health checks with how long each took, and what Azure reports about the running container. A third card lists recent server errors, kept in memory and cleared on each deploy.

What that is worth: a developer sees a slow check or a restart without opening the Azure portal, and the organization can show that the site reads its own state but can change nothing (Reader scope).

## Context

The site could report its version but not its condition. The ask, verbatim:
health checks and errors, viewable from the website like an admin tab,
reaching through Azure.

## Decision

An Admin tab in the header, a real screen with three cards that fetch and
fail independently:

- Application health. Hand-rolled probes behind /healthz (liveness),
  /readyz (readiness), and /api/health (structured JSON: overall status,
  each probe, uptime, version, commit). The container healthcheck now hits
  /healthz. The framework health-check library is the planned upgrade; the
  hand-rolled version shipped the same evening it was asked for.
- Azure's view of the container. The container group runs as a
  user-assigned identity, that identity now holds Reader on its own
  resource group, and the API trades the identity for a management-plane
  token and reads its own container group: group state, container state,
  restart count, and the image Azure believes it is running. Cached 60
  seconds. Anywhere the identity endpoint does not exist, local dev
  included, the card degrades to app-level health and says so.
- Recent server errors. Middleware records unhandled exceptions and 5xx
  responses into a fixed 50-entry in-memory buffer served at /api/errors.
  No stack traces are exposed. The buffer resets on every deploy and the
  screen says that plainly.

The tab is public, no login. On this site the observability is the
exhibit, the same reasoning that serves the Bicep file from a menu. A real
product gates this behind authentication and ships errors to a persistent
sink; both are documented here and deliberately not built, the same
pattern as the undeployed production design.

## Consequences

- The client id and resource path baked in as defaults are identifiers,
  not secrets; a token for them can only be minted from inside the
  container itself.
- Reader scope means the site can see itself but change nothing.
- The error buffer is honest about its limits, which is the demo working
  as intended: the limits are part of the story.

## Addendum, 2026-09-02: second pass, shipped as 1.0.0.17

After the first pass, Steve said he liked a screen where the health checks
are easy to see. The second pass makes the screen say more without
saying it louder.

- **Every health check reports its duration.** Each probe is timed with a
  stopwatch and the Admin tab prints the milliseconds beside the check in
  the muted style, so a slow disk or a slow lookup is visible before it
  turns into a failure. The readiness endpoint uses the same probes and is
  unchanged.
- **Azure's card lists the container's recent events.** The management
  read already returned the container's instance view; the card now shows
  its last three events, newest first, with the name, how many times it
  happened, when it last happened, and the message trimmed to a line. A
  restart story (pulled, started, killed, started again) reads from the
  tab without opening the portal. Unavailable degrades exactly as before.
- **The sweep.** Five comments in Program.cs carried a double-encoded em
  dash, and one of them a mangled "resume", from an encoding-blind edit on
  the first day; they are plain ASCII now, changed byte by byte with
  everything else in the file untouched.

The samples below are read from this build's source each time the page is
served (ADR: Live code samples). The timed checks
([`api/TheYard.Api/Endpoints/HealthEndpoints.cs`](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Api/Endpoints/HealthEndpoints.cs)):

```csharp
// Each probe is timed and each answer is a value, never an exception: a
// health endpoint that throws tells an orchestrator nothing. The checks are
// deliberately about the files this app cannot run without.
private static async Task<HealthCheckEntry[]> RunChecksAsync(HostPaths paths, Backends backends, bool readinessOnly = false)
{
    // Each probe is timed: the Admin tab shows the milliseconds beside the check,
    // so a slow disk or a slow lookup shows up before it fails (ADR-010, second pass).
    // Awaited, because the document store's probe is two point reads over the
    // network and a health check that blocked a thread on them would be the
    // defect the ports record describes (ADR: The ports learn to wait).
    async Task<HealthCheckEntry> Check(string name, Func<Task<bool>> probe, string detail, bool gatesReadiness = true)
    {
        if (readinessOnly && !gatesReadiness)
        {
            // Not asked, so not run. The caller filters these out anyway; this
            // is what stops the probe behind them from happening at all.
            return new HealthCheckEntry(name, "pass", "not asked", 0, gatesReadiness);
        }

        var clock = Stopwatch.StartNew();
        try { return new HealthCheckEntry(name, await probe() ? "pass" : "fail", detail, clock.ElapsedMilliseconds, gatesReadiness); }
        catch (Exception ex) { return new HealthCheckEntry(name, "fail", ex.GetType().Name, clock.ElapsedMilliseconds, gatesReadiness); }
    }
    var checks = new List<HealthCheckEntry>
    {
        await Check("dataset file", () => Task.FromResult(File.Exists(paths.DataPath)), "data/vehicles.json present"),
        await Check("docs", () => Task.FromResult(File.Exists(Path.Combine(paths.RepoRoot, "docs", "hosting", "HOSTING.md"))), "served documents findable"),
        await Check("photo manifest", () => Task.FromResult(File.Exists(paths.ManifestPath)), "image manifest present"),
        // The process listens before its catalogue is in (Startup.cs, listen-first),
        // so ready means the default store's catalogue has loaded: the thing a
        // deploy, and a blue-green swap, must wait for before sending a visitor.
        await Check("catalogue", () => Task.FromResult(backends.Default.CatalogueLoaded), "the default store's catalogue is loaded"),
    };
    // One check per store, named by the store, so a container running both
    // says which one is unavailable. The default store's check is called
    // "database", because the deploy's Verify step and the Admin tab's card both
    // read that name.
    foreach (var backend in backends.All)
    {
        checks.Add(await Check(
            ReferenceEquals(backend, backends.Default) ? "database" : $"database ({backend.Key})",
            backend.Probe,
            // The reason is in the log, not in this response. A health endpoint
            // is public on purpose, and an exception message from a storage
            // failure is typically a filesystem path: exactly the map of the
            // inside of the process that ProblemHandler refuses to draw.
            backend.Database.Ready
                ? $"the seed catalogue is in the store ({backend.Name})"
                : $"{backend.Name} is unavailable, serving the catalogue from files; "
                    + "the reason is in the log",
            // The one check that does not gate readiness, which is the whole
            // point of the fallback. A container with no database still serves
            // the catalogue, the filters, the photos and the bidding; the only
            // thing it loses is bids outliving the process. Reporting itself
            // not ready would take a working site out of service: the deploy's
            // `curl -fsS /readyz` would fail against a site serving every
            // vehicle perfectly well.
            gatesReadiness: false));
    }
    return checks.ToArray();
}
```
*Live from [`api/TheYard.Api/Endpoints/HealthEndpoints.cs`](https://github.com/SteveStout/TheYard/blob/1d6b50b/api/TheYard.Api/Endpoints/HealthEndpoints.cs#L106-L166), region `health-checks`, read from this build at 1d6b50b.*

The events, read from the same management response (the observability
types moved to
[`api/TheYard.Api/Observability.cs`](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Api/Observability.cs)
in ADR: The staff review):

```csharp
// The last three events Azure recorded for the container (pulls, starts,
// kills), newest first, each message trimmed: enough to read a restart
// story from the Admin tab without opening the portal (ADR-010, second pass).
if (civ.TryGetProperty("events", out var evs) && evs.ValueKind == JsonValueKind.Array)
{
    events = evs.EnumerateArray()
        .Select(e => new
        {
            name = e.TryGetProperty("name", out var n) ? n.GetString() ?? "" : "",
            count = e.TryGetProperty("count", out var c) && c.TryGetInt32(out int ci) ? ci : 1,
            last_at = e.TryGetProperty("lastTimestamp", out var lt) ? lt.GetString() ?? "" : "",
            message = Trim(e.TryGetProperty("message", out var msg) ? msg.GetString() ?? "" : "", 140),
        })
        .OrderByDescending(e => e.last_at, StringComparer.Ordinal)
        .Take(3)
        .Cast<object>()
        .ToList();
}
```
*Live from [`api/TheYard.Api/Observability.cs`](https://github.com/SteveStout/TheYard/blob/1d6b50b/api/TheYard.Api/Observability.cs#L219-L236), region `azure-events`, read from this build at 1d6b50b.*

The proof is one API test that every check carries a non-negative
duration and two end-to-end checks that the Admin tab prints it, on a
laptop and on a phone, in
[`api/TheYard.Tests/AdminEndpointTests.cs`](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Tests/AdminEndpointTests.cs),
[`tests/e2e/admin.spec.ts`](https://github.com/SteveStout/TheYard/blob/main/tests/e2e/admin.spec.ts)
and [`tests/e2e/mobile.spec.ts`](https://github.com/SteveStout/TheYard/blob/main/tests/e2e/mobile.spec.ts).
The events list is proven by the live site, since only the container on
Azure can ask about itself.

## Where it sits

Everything this record built lives in the host Api and the front end: the probes in Endpoints/HealthEndpoints.cs, the ErrorRingBuffer and the AzureSelf reader in Observability.cs, and the cards under src/components/admin. Each card fetches and fails on its own, so one broken reading never blanks the tab. No port was needed because the container reads its own memory and its own Azure resource, and the price is an error buffer that empties on every deploy. Errors that must survive a roll are the reason to choose differently, and that is the job ILogStore, an Application port, now does for logs that outlive the container.

## Files

- [`api/TheYard.Api/Endpoints/HealthEndpoints.cs`](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Api/Endpoints/HealthEndpoints.cs): the probes (region health-checks),
  `/healthz`, `/readyz` and `/api/health`.
- [`api/TheYard.Api/Endpoints/ErrorEndpoints.cs`](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Api/Endpoints/ErrorEndpoints.cs) and [`api/TheYard.Api/Endpoints/AdminEndpoints.cs`](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Api/Endpoints/AdminEndpoints.cs): `/api/errors` and
  `/api/admin/azure`.
- [`api/TheYard.Api/Composition/RequestPipeline.cs`](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Api/Composition/RequestPipeline.cs): the middleware that records server errors (region error-log).
- [`api/TheYard.Api/Observability.cs`](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Api/Observability.cs): the health record, the error
  ring buffer and the Azure reader (region azure-events).
- [`src/components/admin/HealthCard/HealthCard.tsx`](https://github.com/SteveStout/TheYard/blob/main/src/components/admin/HealthCard/HealthCard.tsx), [`src/components/admin/ErrorsCard/ErrorsCard.tsx`](https://github.com/SteveStout/TheYard/blob/main/src/components/admin/ErrorsCard/ErrorsCard.tsx) and
  [`src/components/admin/AdminPanel/AdminPanel.module.css`](https://github.com/SteveStout/TheYard/blob/main/src/components/admin/AdminPanel/AdminPanel.module.css): the three cards.
- [`api/TheYard.Tests/AdminEndpointTests.cs`](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Tests/AdminEndpointTests.cs): the API proof;
  [`tests/e2e/admin.spec.ts`](https://github.com/SteveStout/TheYard/blob/main/tests/e2e/admin.spec.ts) and [`tests/e2e/mobile.spec.ts`](https://github.com/SteveStout/TheYard/blob/main/tests/e2e/mobile.spec.ts):
  the browser proof.
- [`Dockerfile`](https://github.com/SteveStout/TheYard/blob/main/Dockerfile): the container health check that hits `/healthz`.
- [`.github/workflows/deploy.yml`](https://github.com/SteveStout/TheYard/blob/main/.github/workflows/deploy.yml): the deploy's Verify step asks
  `/readyz` before it trusts a roll (ADR: The staff review).

## The look, from the live site

![The Admin tab at 1.0.0.22: three passing health checks with their durations, Azure reporting the container group Running with its last three events, and no server errors recorded](/api/docs/images/app-admin.jpg#1280x800)

The health card, from the fetch to the rows
([`src/components/admin/HealthCard/HealthCard.tsx`](https://github.com/SteveStout/TheYard/blob/main/src/components/admin/HealthCard/HealthCard.tsx), a chunk of its own since the workbench):

```tsx
<article className={`${styles.card} op-glass`} data-testid="health-card">
  <h2 className={styles.cardTitle}>Application health</h2>
  <Range of="keptWarm" />
  {health === null ? (
    <CardState state="loading" />
  ) : health === 'failed' ? (
    failed('the health report')
  ) : (
    <>
      <p className={styles.statusRow}>
        <span className={pill(health.status === 'healthy')}>{health.status}</span>
        <span className={styles.muted}>
          v{health.version} · {health.commit} · up {formatUptime(health.uptime_seconds)}
        </span>
      </p>
      <ul className={styles.checkList}>
        {health.checks.map((check) => (
          <li key={check.name} className={styles.healthRow}>
            <span className={pill(check.status === 'pass')}>{check.status}</span>
            <span>{check.name}</span>
            <span className={styles.muted}>{check.detail}</span>
            <span className={styles.duration} data-testid="check-duration">
              {check.duration_ms} ms
            </span>
          </li>
        ))}
      </ul>
      {/* The keep-warm loop's last pass (ADR: Kept awake). */}
      <p className={styles.muted} data-testid="kept-warm">
        {keptWarmLine(health.kept_warm)}
      </p>
    </>
  )}
</article>
```
*Live from [`src/components/admin/HealthCard/HealthCard.tsx`](https://github.com/SteveStout/TheYard/blob/1d6b50b/src/components/admin/HealthCard/HealthCard.tsx#L16-L49), region `health-card`, read from this build at 1d6b50b.*

## Addendum, 2026-10-07 (1.0.3.89): ready means the catalogue is in

The process listens before its catalogue loads (ADR: The ports learn to wait, the addendum on listening first), so `/readyz` gained one check that gates it, `catalogue`, which passes once the default store's hundred thousand vehicles are in memory. `/healthz` still answers the moment the process is up. `/api/health` shows the new check beside the others, and since it is an API read it waits for the load like every other, so the Admin tab sees the check pass; it is `/readyz` that says 503 while a fresh container loads. The deploy waits on `/readyz` before it reads the filters, and a blue-green swap will wait on the same thing.