# ADR: The box reports itself

Status: accepted, 2026-11-07, at 1.0.4.156. WickedAwsome, the Linux box that runs the job hunt's morning program and the Claude sessions that build this site, gets a card under Cloud infrastructure. Its own agent posts a snapshot once a minute and a live sample every two seconds to the site, behind the operator's key; the site holds them in memory and keeps one minute a snapshot in the plan container.

## Reading list

1. [ADR: One frame for every resource](https://theyard.stevenstout.biz/?doc=adr-one-frame-for-every-resource): the frame the card is drawn in, its dials, its rows and its readout.
2. [ADR: The Nightly Function on the Admin tab](https://theyard.stevenstout.biz/?doc=adr-the-nightly-function-on-the-admin-tab): the first card fed by a push with the operator's key, whose route this one copies.
3. [ADR: Every Claude lane reports its usage](https://theyard.stevenstout.biz/?doc=adr-every-claude-lane-reports-its-usage): a document held in this process and kept in the plan container under kinds of its own.
4. [ADR: Site activity, and the line an address does not cross](https://theyard.stevenstout.biz/?doc=adr-activity): why the box's posts stay out of the activity card, and why no address reaches the wire.

## In plain words

The job hunt does not only run in Azure. Every morning at six a program on a Linux box at home drafts and files, and the sessions that write this site run there too. Until now the Admin tab watched every Azure resource and said nothing about the machine the work actually runs on. Now the box sends the site what Linux's own System Monitor shows (each core, the memory and swap, the network, every disk and the array under them, both GPUs, every temperature and the services the job hunt needs) and the card draws it in the same frame as every other resource. If the box goes quiet for three minutes, its row turns red.

What that is worth: the one machine the job hunt cannot run without is watched the way the cloud is, and a failed disk in its array, a stopped service or a box that stopped reporting shows on the same tab, in the same colour, as a throttled database.

## Context

On 4 November Steve asked for "a module similar to this for wicked awsomse for our server because it dose support the yard's job hunt", then "also I want all information on the resources" and "any way we can make the monitors live like on linux resources", with a picture of System Monitor's Resources tab. The box is an i3-8350K with four cores, 32 GB, two RTX 2080s and a RAID0 of six SSDs with no redundancy (J:\server\DONE-2026-10-26.md): any one failing loses the system.

## Decision

1. **Push, not pull.** The box posts to the site; the site never reaches into a home network. One user unit, `theyard-box-agent.service`, runs a Python script with the standard library only, reading /proc, /sys, `df`, `nvidia-smi` and `systemctl`, with no package installed and no sudo.
2. **Two cadences, one connection.** Every two seconds a small live sample (each core, memory, swap, each interface's rates, each GPU's load) goes to `POST /api/admin/cloud/box/live` and lands in a ring of sixty in this process's memory; every thirtieth tick the full snapshot goes to `POST /api/admin/cloud/box/snapshot`. Both are behind the operator's key in `X-Admin-Key`, read from a file of mode 600 in the box owner's home, never in the repository, never logged. The agent posts to the web app's own name, not through the edge, so the stream costs no edge credits.
3. **Held in memory, kept a minute at a time.** The newest snapshot and the ring live in this process, which serves both names. The recorder keeps one minute per snapshot as `box-minute` and the snapshot's slow facts as `box-hour` in the plan container, which expires every document after thirty-one days, so there is no new container and no new kind of expiry.
4. **The red rule, with its guard.** No snapshot for three minutes turns the row "Is the box reporting?" red, which turns the card and its rail dot red. The silence is measured from the later of the newest snapshot (held or kept) and this process's start, so a roll, two to five minutes with nothing listening, never reads as the box being down. A live sample older than ten seconds reads blue on the live panels only.
5. **The thresholds are the box's own.** All cores amber at 70 % and red at 90 %, held fifteen minutes (the local gate takes the i3 to 100 % for about eight); memory 85 and 95; any file system 80 and 90; each GPU's memory 85 and 95; every temperature amber ten degrees under its sensor's own high mark and red at it, the marks read from the sensors and `nvidia-smi`, never typed; the array red at once when a member is missing or failed; a service that must run (the remote-control session, the J: mount, Tailscale, ssh and the 6 AM timer) red at once when it is not running, and amber when the morning run's last result failed. Sunshine is known to be off and reads as information.
6. **The box's posts are not visitors.** An accepted post under `/api/admin/cloud/box/` stays out of the request ring, the activity card and the kept log; a refused one stays in, because the operator's key was tried. Every request under that prefix is left out of Application Insights' request telemetry, so forty-three thousand posts a day spend nothing of its 0.1 GB daily cap.
7. **No address and no port on the wire.** The snapshot may carry where the box sits on a network; the card names interfaces by name only, and every word from the box passes one scrub (`BoxWords.Scrub`) that replaces any address and any port before it reaches a public page.

What was considered: Azure Arc (a sign-in on the box, more on the box, and Log Analytics ingestion billed), streaming the samples (the edge buffers a streamed answer, the option ADR-100 rejected) and a Cosmos DB kind of its own (the plan container already expires at thirty-one days).

## What holds it

`BoxSnapshotRulesTests` holds every rule's sentence, the list limits and the three handlers over the in-memory ports; `BoxCardTests` the eight rows waiting before a snapshot, the three minutes and the guard, each red case and the scrub; `BoxRoutesTests` the two posts' 401, 400 and 200, the anonymous live read, and both exclusions; `cloud.spec.ts` the card's eight rows in the one frame.

## Where it sits

The records in `TheYard.Data`, the rules in the Domain's Admin feature, the ports and handlers in the Application's Cloud feature, and the source, the card, its rows, its words and its routes in the host's Cloud slice beside the network log's. The agent and its unit are in `scripts/box-agent/`.

```csharp
/// <summary>The box's card's rows.</summary>
public static class BoxRows
{
    /// <summary>How long the box may be silent before its reporting row turns red.</summary>
    public static readonly TimeSpan Silent = TimeSpan.FromMinutes(3);

    /// <summary>The services that must run: a stopped one reads red at once.</summary>
    public static readonly IReadOnlyList<string> MustRun = ["claude-remote-control.service", "jdrive.service", "tailscaled.service", "ssh.service", "jobhunt-6am.timer"];

    /// <summary>The service whose last result reads amber when it failed: the job hunt's morning run.</summary>
    public const string MorningRun = "jobhunt-6am.service";

    /// <summary>The eight rows; every one waits in info before the first snapshot.</summary>
    public static IReadOnlyList<CloudRow> All(BoxSnapshot? snapshot, DateTimeOffset? newestKept, DateTimeOffset started, DateTimeOffset now, int keptLastHour)
    {
        var reporting = Reporting(snapshot, newestKept, started, now, keptLastHour);
        if (snapshot is null)
        {
            string[] keys = ["cores", "memory", "network", "disks", "gpus", "temps", "services"];
            string[] questions = ["What is each core doing?", "Is the memory and swap fine?", "Is the network moving?", "Are the disks and file systems healthy?", "What are the GPUs doing?", "How hot is it?", "Are the job-hunt services up?"];
            return [.. keys.Select((key, i) => new CloudRow(key, questions[i], BoxSource.Waiting, [], "info")), reporting];
        }
        return [Cores(snapshot), Memory(snapshot), Network(snapshot), Disks(snapshot), Gpus(snapshot), Temps(snapshot), Services(snapshot, now), reporting];
    }

    /// <summary>Each core's load, clock and temperature.</summary>
    public static CloudRow Cores(BoxSnapshot snapshot)
    {
        var loads = snapshot.Cores.Select(core => core.Load).DefaultIfEmpty(0).ToList();
        string spread = $"{loads.Min().ToString("0", CultureInfo.InvariantCulture)} to {loads.Max().ToString("0", CultureInfo.InvariantCulture)} %";
        return new CloudRow("cores", "What is each core doing?", $"{snapshot.Cores.Count} cores",
            [new($"all {snapshot.Cores.Count} at {spread}; the box idles under 10 % and the local gate takes every core to 100 % for about 8 minutes", null)], "good")
        {
            Table = Table(["Core", "Load", "Clock", "Temp"], snapshot.Cores.Select(core => new[] { $"CPU{core.Index + 1}", BoxWords.Percent(core.Load), $"{(core.Mhz / 1000).ToString("0.00", CultureInfo.InvariantCulture)} GHz", BoxWords.Degrees(core.Temp) })),
        };
    }

    /// <summary>Memory, cache and swap; amber when swap is in use or memory passes its amber line.</summary>
    public static CloudRow Memory(BoxSnapshot snapshot)
    {
        double share = snapshot.Memory.TotalB > 0 ? 100.0 * snapshot.Memory.UsedB / snapshot.Memory.TotalB : 0;
        double swapShare = snapshot.Swap.TotalB > 0 ? 100.0 * snapshot.Swap.UsedB / snapshot.Swap.TotalB : 0;
        var (amber, red) = CloudThresholds.BoxMemory;
        string tone = share >= red ? "alarm" : share >= amber || snapshot.Swap.UsedB > 0 ? "warning" : "good";
        return new CloudRow("memory", "Is the memory and swap fine?", $"{BoxWords.Bytes(snapshot.Memory.UsedB)} of {BoxWords.Bytes(snapshot.Memory.TotalB)}",
            [new($"cache {BoxWords.Bytes(snapshot.Memory.CacheB)}, free {BoxWords.Bytes(snapshot.Memory.FreeB)} · swap {BoxWords.Bytes(snapshot.Swap.UsedB)} of {BoxWords.Bytes(snapshot.Swap.TotalB)} · amber {amber} %, red {red} %, swap in use is amber", null)], tone)
        {
            Table = Table(["Pool", "Used", "Of", "Share"],
            [
                ["Memory", BoxWords.Bytes(snapshot.Memory.UsedB), BoxWords.Bytes(snapshot.Memory.TotalB), BoxWords.Percent(Math.Round(share, 1))],
                ["Cache", BoxWords.Bytes(snapshot.Memory.CacheB), "", BoxWords.Percent(snapshot.Memory.TotalB > 0 ? Math.Round(100.0 * snapshot.Memory.CacheB / snapshot.Memory.TotalB, 1) : null)],
                ["Swap", BoxWords.Bytes(snapshot.Swap.UsedB), BoxWords.Bytes(snapshot.Swap.TotalB), BoxWords.Percent(Math.Round(swapShare, 1))],
            ]),
        };
    }

    /// <summary>Each interface's rates and totals, by name only: no address reaches the card.</summary>
    public static CloudRow Network(BoxSnapshot snapshot)
    {
        var real = snapshot.Nets.Where(net => net.Name != "lo").ToList();
        return new CloudRow("network", "Is the network moving?", $"{BoxWords.Rate(real.Sum(net => net.InBps))} in, {BoxWords.Rate(real.Sum(net => net.OutBps))} out",
            [new($"totals since boot {BoxWords.Bytes(real.Sum(net => (double)net.InTotalB))} received, {BoxWords.Bytes(real.Sum(net => (double)net.OutTotalB))} sent", null)], "good")
        {
            Table = Table(["Interface", "In now", "Out now", "Received", "Sent"], real.Select(net => new[] { BoxWords.Scrub(net.Name), BoxWords.Rate(net.InBps), BoxWords.Rate(net.OutBps), BoxWords.Bytes(net.InTotalB), BoxWords.Bytes(net.OutTotalB) })),
        };
    }

    /// <summary>Each file system and the array: red when the array is not whole or a file system passes its red line.</summary>
    public static CloudRow Disks(BoxSnapshot snapshot)
    {
        var (amber, red) = CloudThresholds.BoxFileSystem;
        double Fill(BoxDisk disk) => disk.SizeB > 0 ? 100.0 * disk.UsedB / disk.SizeB : 0;
        var array = snapshot.Disks.FirstOrDefault(disk => disk.RaidState is not null);
        bool broken = array is not null && (array.MembersActive < array.Members || array.RaidState!.Contains("(F)", StringComparison.Ordinal)
            || array.RaidState.Contains("inactive", StringComparison.OrdinalIgnoreCase) || array.RaidState.Contains("degraded", StringComparison.OrdinalIgnoreCase));
        double fullest = snapshot.Disks.Select(Fill).DefaultIfEmpty(0).Max();
        string tone = broken || fullest >= red ? "alarm" : fullest >= amber ? "warning" : "good";
        string answer = array is null ? "no array" : broken ? $"the array is not whole: {array.MembersActive} of {array.Members} active" : $"{BoxWords.Scrub(array.RaidState)}, {array.MembersActive} of {array.Members} active";
        return new CloudRow("disks", "Are the disks and file systems healthy?", answer,
            [new($"red when the array reports a missing or failed member, or any file system passes {red} %; amber from {amber} %", null)], tone)
        {
            Table = Table(["Mount", "Type", "Size", "Used", "Share", "State"], snapshot.Disks.Select(disk => new[] { $"{disk.Mount} ({BoxWords.Scrub(disk.Device)})", disk.Fstype, BoxWords.Bytes(disk.SizeB), BoxWords.Bytes(disk.UsedB), BoxWords.Percent(Math.Round(Fill(disk), 1)), disk.RaidState is null ? "" : $"{BoxWords.Scrub(disk.RaidState)}, {disk.MembersActive} of {disk.Members} active" })),
        };
    }

    /// <summary>Each GPU: amber at its memory's amber line or ten degrees under its slowdown mark, red at either red.</summary>
    public static CloudRow Gpus(BoxSnapshot snapshot)
    {
        var (amber, red) = CloudThresholds.BoxGpuMemory;
        string ToneOf(BoxGpu gpu)
        {
            double memory = gpu.MemTotalB > 0 ? 100.0 * gpu.MemUsedB / gpu.MemTotalB : 0;
            return memory >= red || gpu.SlowdownTemp is { } hot && gpu.Temp >= hot ? "alarm"
                : memory >= amber || gpu.SlowdownTemp is { } warm && gpu.Temp >= warm - CloudThresholds.BoxTemperatureUnder ? "warning" : "good";
        }
        string tone = Worst(snapshot.Gpus.Select(ToneOf));
        string answer = snapshot.Gpus.Count == 0 ? "no GPU read" : $"{snapshot.Gpus.Count} x {snapshot.Gpus[0].Name.Replace("NVIDIA GeForce ", "", StringComparison.Ordinal)}";
        return new CloudRow("gpus", "What are the GPUs doing?", answer,
            [new($"amber at {amber} % of a GPU's memory or {CloudThresholds.BoxTemperatureUnder} °C under its own slowdown mark, red at {red} % or the mark", null)], tone)
        {
            Table = Table(["GPU", "Load", "Memory", "Temp", "Power", "State"], snapshot.Gpus.Select(gpu => new[]
            {
                $"GPU {gpu.Index} {gpu.Name}", BoxWords.Percent(gpu.Load), $"{BoxWords.Bytes(gpu.MemUsedB)} of {BoxWords.Bytes(gpu.MemTotalB)}",
                gpu.SlowdownTemp is { } mark ? $"{BoxWords.Degrees(gpu.Temp)}, slowdown {BoxWords.Degrees(mark)}" : BoxWords.Degrees(gpu.Temp),
                gpu.PowerW is { } watts ? $"{watts.ToString("0", CultureInfo.InvariantCulture)} W of {gpu.PowerLimitW?.ToString("0", CultureInfo.InvariantCulture) ?? "?"} W" : "not read", gpu.Pstate,
            })),
        };
    }

    /// <summary>Every sensor against its own marks: amber ten degrees under its high mark, red at it; no mark, no tone.</summary>
    public static CloudRow Temps(BoxSnapshot snapshot)
    {
        static string ToneOf(BoxTemp temp) =>
            temp.High is not { } high ? "good" : temp.Now >= high ? "alarm" : temp.Now >= high - CloudThresholds.BoxTemperatureUnder ? "warning" : "good";
        var package = BoxSource.Package(snapshot);
        return new CloudRow("temps", "How hot is it?", package is null ? "no sensor read" : $"{BoxWords.Degrees(package.Now)} {BoxWords.Scrub(package.Sensor)}",
            [new($"amber at {CloudThresholds.BoxTemperatureUnder} °C under each sensor's own high mark, red at the mark; the marks come from the sensors, never typed here", null)], Worst(snapshot.Temps.Select(ToneOf)))
        {
            Table = Table(["Sensor", "Now", "Marks"], snapshot.Temps.Select(temp => new[]
            {
                BoxWords.Scrub(temp.Sensor), BoxWords.Degrees(temp.Now),
                string.Join(", ", new[] { temp.High is { } h ? $"high {BoxWords.Degrees(h)}" : null, temp.Critical is { } c ? $"critical {BoxWords.Degrees(c)}" : null }.OfType<string>()),
            })),
        };
    }

    /// <summary>The services: red when one that must run is not active, amber when the morning run's last result failed.</summary>
    public static CloudRow Services(BoxSnapshot snapshot, DateTimeOffset now)
    {
        bool Active(BoxService service) => service.ActiveState == "active";
        var stopped = MustRun.Where(unit => snapshot.Services.FirstOrDefault(service => service.Unit == unit) is not { } found || !Active(found)).ToList();
        bool morningFailed = snapshot.Services.FirstOrDefault(service => service.Unit == MorningRun)?.Note is { } note && note.Contains("last result", StringComparison.Ordinal) && !note.Contains("success", StringComparison.Ordinal);
        string tone = stopped.Count > 0 ? "alarm" : morningFailed ? "warning" : "good";
        int running = snapshot.Services.Count(Active);
        return new CloudRow("services", "Are the job-hunt services up?", $"{running} of {snapshot.Services.Count} active",
            [new(stopped.Count > 0 ? $"not running: {string.Join(", ", stopped)}" : $"red when {string.Join(", ", MustRun)} is not running; amber when the morning run's last result failed", null)], tone)
        {
            Table = Table(["Service", "State", "Up for", "Note"], snapshot.Services.Select(service => new[]
            {
                $"{service.Unit} ({service.Scope})", $"{service.ActiveState}, {service.SubState}",
                service.Since is { } since && Active(service) ? BoxWords.Span(now - since) : "", BoxWords.Scrub(service.Note),
            })),
        };
    }

    /// <summary>
    /// Whether the box is reporting: red after three minutes with no snapshot, measured from the later of the newest
    /// snapshot (held or kept) and this process's start, so a roll never reads as the box being down.
    /// </summary>
    public static CloudRow Reporting(BoxSnapshot? snapshot, DateTimeOffset? newestKept, DateTimeOffset started, DateTimeOffset now, int keptLastHour)
    {
        DateTimeOffset? newest = new[] { snapshot?.At, newestKept }.Where(at => at is not null).Max();
        var from = newest is { } at && at > started ? at : started;
        var silence = now - from;
        string count = $"{keptLastHour} snapshots kept in the last hour; the agent adds nothing to the bill, one small write a minute on the free tier";
        if (newest is null)
        {
            return silence > Silent
                ? new CloudRow("reporting", "Is the box reporting?", $"no: no snapshot in the {BoxWords.Span(silence)} since this site started", [new(count, null)], "alarm")
                : new CloudRow("reporting", "Is the box reporting?", BoxSource.Waiting, [new(count, null)], "info");
        }
        return silence > Silent
            ? new CloudRow("reporting", "Is the box reporting?", $"no: silent for {BoxWords.Span(silence)}", [new(count, null)], "alarm")
            : new CloudRow("reporting", "Is the box reporting?", $"yes, the newest snapshot {BoxWords.Span(now - newest.Value)} ago", [new(count, null)], "good");
    }

    /// <summary>The worst of a set of tones; good when there are none.</summary>
    public static string Worst(IEnumerable<string> tones) =>
        tones.OrderByDescending(tone => tone switch { "alarm" => 3, "warning" => 2, "info" => 1, _ => 0 }).FirstOrDefault() ?? "good";

    /// <summary>A table from rows of words.</summary>
    private static CloudTable Table(IReadOnlyList<string> columns, IEnumerable<IEnumerable<string>> rows) =>
        new(columns, [.. rows.Select(row => (IReadOnlyList<CloudLink>)[.. row.Select(cell => new CloudLink(cell, null))])]);
}
```
*Live from [`api/TheYard.Api/Features/Admin/Cloud/BoxRows.cs`](https://github.com/SteveStout/TheYard/blob/bafcdbe/api/TheYard.Api/Features/Admin/Cloud/BoxRows.cs#L11-L184), region `box-rows`, read from this build at bafcdbe.*

## Files

- [`api/TheYard.Data/BoxSnapshot.cs`](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Data/BoxSnapshot.cs) - what the box posts
- [`api/TheYard.Domain/Features/Admin/BoxSnapshotRules.cs`](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Domain/Features/Admin/BoxSnapshotRules.cs) - what it may hold
- [`api/TheYard.Application/Features/Admin/Cloud/BoxPorts.cs`](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Application/Features/Admin/Cloud/BoxPorts.cs) and [`BoxHandlers.cs`](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Application/Features/Admin/Cloud/BoxHandlers.cs) - the ring, the snapshots and the three handlers
- [`api/TheYard.Api/Features/Admin/Cloud/BoxCard.cs`](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Api/Features/Admin/Cloud/BoxCard.cs), [`BoxRows.cs`](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Api/Features/Admin/Cloud/BoxRows.cs), [`BoxSource.cs`](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Api/Features/Admin/Cloud/BoxSource.cs) and [`BoxRoutes.cs`](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Api/Features/Admin/Cloud/BoxRoutes.cs) - the card and its routes
- [`scripts/box-agent/`](https://github.com/SteveStout/TheYard/tree/main/scripts/box-agent) - the agent, its unit and its installer

## Addendum, 2026-11-07 (1.0.4.157): the live panels

The card draws four live panels above its chart, as System Monitor's Resources tab draws them: each core, the memory and swap, the network and the GPUs, sixty seconds wide with now at the right, the cores, memory and GPUs as shares and the network scaled to the window's peak with the loopback left out. They read `GET /api/admin/cloud/box/live` every two seconds while the card is open and the tab is in view, and never while the tab is hidden; past ten seconds without a sample they say "no live sample" in blue. The poll's cost: an answer of about 2 KB every two seconds, about 3.5 MB an hour while the card is open, a few hundredths of the edge's hourly credits, and nothing while it is closed. The layout is `src/lib/box/boxLive.ts`, held by its Vitest file; `box.spec.ts` holds the four panels, the blue state where no agent posts, two to four reads in six seconds, and none while hidden.