# Memory, held by the build

TheYard is a used-vehicle auction platform built in the open: a .NET 10 API, a React front end, and two databases underneath it. Four apps share one small App Service plan, a B2 with 3.5 GB of memory between them, so a megabyte one app keeps is a megabyte the other three cannot have. This page is the eight rules the apps are held to, each with why it matters, the number this site moved when it applied it, and the code or test that holds it.

## In plain words

A program's memory is a shared room. Anything it keeps "just in case" takes space from everything else on the same machine, and a collection that only ever grows takes all of it eventually. The rules here say what a .NET app on a small shared machine should keep, for how long, and how to make the build notice when one slips.

What that is worth: the site stopped holding a second copy of data its databases already keep, and every rule below is checked on every build instead of remembered.

## The request path, before and after

![A listing request before and after: process memory in the middle, then the store in the middle](/api/docs/images/memory-request-path.svg#960x430)

Before, each site loaded all hundred thousand vehicles into its own memory, for both databases, and answered every search by walking that list. The databases held the 200 starting vehicles. After, the databases hold all hundred thousand and answer the search themselves, and the site asks for one page and keeps nothing but a five-second cache (ADR: The store does the work).

Measured before the change, on 1.0.3.176: the plan at 82 per cent of its 3.5 GB, each .NET site about 750 MB of working set and 420 to 460 MB of heap, each holding two hundred thousand vehicles. The same readings after the change are in ADR: The store does the work, read once the fill had finished.

## The eight rules

### a. A store is not a cache

Why: data that already lives in a database and is also held whole in memory is paid for twice, and the memory copy has none of the database's guarantees: no bound, no expiry, rebuilt on every start.

Here: the catalogue was held whole in each process, about 750 MB a site. Now the store holds it and the process holds none once the store answers, which is a number the Admin tab shows and a test reads.

```csharp
/// <summary>Rule a. The test host's relational store answers its catalogue, so the count it holds in memory reads zero.</summary>
[Fact]
public async Task Once_the_store_answers_the_process_holds_no_catalogue()
{
    var client = factory.CreateClient();
    // On the relational store by name, so the suite booted on Cosmos DB asks the same question.
    client.DefaultRequestHeaders.Add(Backends.HeaderName, "sql");
    await client.GetAsync("/api/vehicles?limit=1");

    using var machines = JsonDocument.Parse(await client.GetStringAsync("/api/admin/machines"));
    var catalogues = machines.RootElement.GetProperty("container").GetProperty("catalogues").EnumerateArray().ToList();
    var relational = catalogues.Single(c => c.GetProperty("store").GetString()!.Contains("SQL", StringComparison.OrdinalIgnoreCase)).GetProperty("held");

    Assert.Equal("store", relational.GetProperty("mode").GetString());
    Assert.Equal(0, relational.GetProperty("held_in_memory").GetInt32());
    Assert.Equal(TestCatalogue.Vehicles, relational.GetProperty("filled").GetInt32());
    Assert.True(machines.RootElement.GetProperty("container").GetProperty("gc").GetProperty("heap_size_mb").GetDouble() > 0);
}
```
*Live from [`api/TheYard.Tests/MemoryRulesTests.cs`](https://github.com/SteveStout/TheYard/blob/7830983/api/TheYard.Tests/MemoryRulesTests.cs#L20-L37), region `rule-a`, read from this build at 7830983.*

### b. Every cache is bounded, and every cache expires

Why: a dictionary that only grows is a memory leak on a delay. Bound it by a count, by bytes or both, and give every entry an expiry.

Here: one cache type, `BoundedCache`, used for every cache a request fills. The reset-mail limit forgets stale addresses and stops at 10,000. A miss loads once a key, so a cache whose entry has just expired costs the store one query rather than one per visitor who asked in the same moment. A test names every collection field in the six production projects with what bounds it, and fails on a new one until somebody writes its bound down.

```csharp
// Bounded by count and bytes, each entry expiring: the shape every cache here takes.
var pages = new BoundedCache<string, IReadOnlyList<CatalogueEntry>>(
    maxCount: 64, maxBytes: 16L << 20, expiry: TimeSpan.FromSeconds(5),
    bytesOf: page => 64 + page.Count * 1_500);
```

```csharp
// Bounded by count and bytes, expiring after a bucket of the clock: a store's answer is reused
// for the five seconds a page of auctions cannot visibly change in, and never grows past the
// numbers here (ADR: The store does the work, rule b).
private readonly BoundedCache<string, SegmentCounts> _counts = new(512, 1L << 20, TimeSpan.FromSeconds(5), _ => 96);
private readonly BoundedCache<string, IReadOnlyList<CatalogueEntry>> _pages = new(64, 16L << 20, TimeSpan.FromSeconds(5), page => 64 + (page.Count * ExpectedVehicleBytes));
private readonly BoundedCache<string, CatalogueEntry> _vehicles = new(4_096, 8L << 20, TimeSpan.FromMinutes(10), _ => ExpectedVehicleBytes, comparer: StringComparer.Ordinal);
```
*Live from [`api/TheYard.Application/InventoryService.cs`](https://github.com/SteveStout/TheYard/blob/7830983/api/TheYard.Application/InventoryService.cs#L73-L78), region `listing-cache`, read from this build at 7830983.*

### c. Page, never materialise

Why: `ToList()` over a table brings every row into memory to keep a page of them. Put the filter, the order and the page in the query (`Where`, `OrderBy`, `Skip`, `Take`) so the database reads only as far as the page, and read with `AsNoTracking` when nothing is written back, so EF Core keeps no change-tracking copy of each row.

```csharp
// The database orders by an index and stops at the page's last row.
var page = await db.Catalogue.AsNoTracking()
    .Where(row => row.Make == make)
    .OrderBy(row => row.Price).ThenBy(row => row.Seq)
    .Skip(offset).Take(limit)
    .ToListAsync();
```

```csharp
/// <summary>
/// The filter as a WHERE clause: the store's own spellings compared with equality, the
/// segment as a comparison of the slot with now, every typed word as a substring of the
/// searchable text or a status word that names the row's segment, and the ids somebody's
/// bid moved left out as one JSON parameter, however many there are.
/// </summary>
private IQueryable<CatalogueRow> Query(YardDbContext db, CatalogueFilter filter)
{
    long d = filter.SinceAnchorMs;
    var rows = db.Catalogue.AsNoTracking();
    if (_beyond)
    {
        rows = rows.Where(row => row.Seq < target);
    }
    if (filter.Make is { } make)
    {
        rows = rows.Where(row => row.Make == make);
    }
    if (filter.BodyStyle is { } bodyStyle)
    {
        rows = rows.Where(row => row.BodyStyle == bodyStyle);
    }
    if (filter.TitleStatus is { } titleStatus)
    {
        rows = rows.Where(row => row.TitleStatus == titleStatus);
    }
    if (filter.Province is { } province)
    {
        rows = rows.Where(row => row.Province == province);
    }
    // A typed word is looked for in every row, so on a SQLite file the schedule's columns are
    // then compared as "+ 0", which no index serves: SQLite's planner, which has no statistics,
    // reads the narrow words index alone rather than ranging over a schedule index and reading
    // every row in the range for its text. SQL Server's planner has statistics and is left to them.
    rows = (filter.Segment, local && filter.Tokens.Count > 0) switch
    {
        (CatalogueSegment.Live, false) => rows.Where(row => row.StartOffsetMs <= d && row.EndOffsetMs > d),
        (CatalogueSegment.Upcoming, false) => rows.Where(row => row.StartOffsetMs > d),
        (CatalogueSegment.Ended, false) => rows.Where(row => row.EndOffsetMs <= d),
        (CatalogueSegment.Live, true) => rows.Where(row => row.StartOffsetMs + 0 <= d && row.EndOffsetMs + 0 > d),
        (CatalogueSegment.Upcoming, true) => rows.Where(row => row.StartOffsetMs + 0 > d),
        (CatalogueSegment.Ended, true) => rows.Where(row => row.EndOffsetMs + 0 <= d),
        _ => rows,
    };
    if (filter.MinCondition is { } grade)
    {
        rows = rows.Where(row => row.ConditionGrade >= grade);
    }
    if (filter.PriceMin is { } low)
    {
        rows = rows.Where(row => row.Price >= low);
    }
    if (filter.PriceMax is { } high)
    {
        rows = rows.Where(row => row.Price <= high);
    }
    foreach (var token in filter.Tokens)
    {
        string text = token.Text;
        bool live = token.AlsoMatches.Contains(CatalogueSegment.Live);
        bool upcoming = token.AlsoMatches.Contains(CatalogueSegment.Upcoming);
        bool ended = token.AlsoMatches.Contains(CatalogueSegment.Ended);
        rows = rows.Where(row => row.SearchText.Contains(text)
            || (live && row.StartOffsetMs <= d && row.EndOffsetMs > d)
            || (upcoming && row.StartOffsetMs > d)
            || (ended && row.EndOffsetMs <= d));
    }
    if (filter.Exclude.Count > 0)
    {
        string[] moved = [.. filter.Exclude];
        rows = rows.Where(row => !EF.Parameter(moved).Contains(row.Id));
    }
    return rows;
}

/// <summary>The order as an ORDER BY that one index serves, the place breaking every tie.</summary>
private static IQueryable<CatalogueRow> Ordered(IQueryable<CatalogueRow> rows, CatalogueOrder order) => order switch
{
    CatalogueOrder.EndAscending => rows.OrderBy(row => row.EndOffsetMs).ThenBy(row => row.Seq),
    CatalogueOrder.StartAscending => rows.OrderBy(row => row.StartOffsetMs).ThenBy(row => row.Seq),
    CatalogueOrder.EndDescending => rows.OrderByDescending(row => row.EndOffsetMs).ThenBy(row => row.Seq),
    CatalogueOrder.PriceAscending => rows.OrderBy(row => row.Price).ThenBy(row => row.Seq),
    CatalogueOrder.PriceDescending => rows.OrderByDescending(row => row.Price).ThenBy(row => row.Seq),
    CatalogueOrder.ConditionDescending => rows.OrderByDescending(row => row.ConditionGrade).ThenBy(row => row.Seq),
    CatalogueOrder.BidsDescending => rows.OrderByDescending(row => row.BidCount).ThenBy(row => row.Seq),
    _ => rows.OrderBy(row => row.Seq),
};

/// <summary>
/// The three segments' counts in one statement: one scan of the narrow counts index, three
/// conditional counts. A filter kept to one segment is counted without the segment in its WHERE
/// and the other two counts set to nothing here, because the conditional counts already split
/// the rows by segment, and a segment in the WHERE draws the planner to a range over a schedule
/// index that then reads every row in it for the other columns the filter names.
/// </summary>
public async Task<SegmentCounts> CountAsync(CatalogueFilter filter)
{
    await using var db = await contexts.CreateDbContextAsync();
    long d = filter.SinceAnchorMs;
    var counts = await Query(db, filter.In(CatalogueSegment.Any))
        .GroupBy(_ => 1)
        .Select(all => new
        {
            Live = all.Count(row => row.StartOffsetMs <= d && row.EndOffsetMs > d),
            Upcoming = all.Count(row => row.StartOffsetMs > d),
            Ended = all.Count(row => row.EndOffsetMs <= d),
        })
        .FirstOrDefaultAsync();
    if (counts is null)
    {
        return SegmentCounts.None;
    }
    return filter.Segment switch
    {
        CatalogueSegment.Live => new SegmentCounts(counts.Live, 0, 0),
        CatalogueSegment.Upcoming => new SegmentCounts(0, counts.Upcoming, 0),
        CatalogueSegment.Ended => new SegmentCounts(0, 0, counts.Ended),
        _ => new SegmentCounts(counts.Live, counts.Upcoming, counts.Ended),
    };
}

/// <summary>One page: the ORDER BY, then OFFSET and FETCH in the database, so it reads only as far as the page.</summary>
public async Task<IReadOnlyList<CatalogueEntry>> PageAsync(CatalogueFilter filter, CatalogueOrder order, int skip, int take)
{
    if (take <= 0)
    {
        return [];
    }
    await using var db = await contexts.CreateDbContextAsync();
    if (!local || filter.Tokens.Count == 0)
    {
        var rows = await Ordered(Query(db, filter), order).Skip(skip).Take(take).ToListAsync();
        return [.. rows.Select(row => row.ToEntry())];
    }
    // With a typed word on a SQLite file, the page's places first, off the words index alone and in an order no
    // index serves, so the planner cannot walk an order's index reading every row for its text;
    // then the page's rows by their place.
    var places = await Unindexed(Query(db, filter), order).Skip(skip).Take(take).Select(row => row.Seq).ToListAsync();
    if (places.Count == 0)
    {
        return [];
    }
    var found = (await db.Catalogue.AsNoTracking().Where(row => EF.Parameter(places).Contains(row.Seq)).ToListAsync())
        .ToDictionary(row => row.Seq);
    return [.. places.Select(place => found[place].ToEntry())];
}
```
*Live from [`api/TheYard.Infrastructure/EfVehicleCatalogue.cs`](https://github.com/SteveStout/TheYard/blob/7830983/api/TheYard.Infrastructure/EfVehicleCatalogue.cs#L70-L215), region `ef-catalogue-query`, read from this build at 7830983.*

### d. Streams over byte arrays past 85 KB

Why: an object of 85,000 bytes or more goes on the large object heap, which is collected only with the oldest generation and is not compacted by default, so a stream of large allocations fragments memory and makes the expensive collections run more often.

Here: reading a picture's size used to load the whole file, up to 941 KB, on every page that asked. It reads the header now: 24 bytes for a PNG, a seek to the frame marker for a JPEG, the first 32,768 characters of an SVG.

```csharp
// Only the header is read. Reading the whole file to look at its first bytes put every
// photograph of a document, up to 941 KB, on the large object heap on every page that
// asked its size (ADR: The store does the work, rule d): a PNG's size is in its first 24
// bytes, a JPEG's in a frame marker the walk below seeks to, and an SVG's in its opening
// tag, which is in the first 32,768 characters of every drawing here.
using var stream = File.OpenRead(path);
if (extension == ".svg")
{
    char[] buffer = ArrayPool<char>.Shared.Rent(SvgHeadChars);
    try
    {
        using var reader = new StreamReader(stream);
        int read = reader.ReadBlock(buffer, 0, SvgHeadChars);
        return SvgSize(new string(buffer, 0, read));
    }
    finally
    {
        ArrayPool<char>.Shared.Return(buffer);
    }
}
Span<byte> head = stackalloc byte[24];
if (extension == ".png")
{
    return stream.ReadAtLeast(head, 24, throwOnEndOfStream: false) < 24
        ? null
        : ((head[16] << 24) | (head[17] << 16) | (head[18] << 8) | head[19],
            (head[20] << 24) | (head[21] << 16) | (head[22] << 8) | head[23]);
}
if (extension is ".jpg" or ".jpeg")
{
    Span<byte> marker = head[..9];
    long at = 2;
    while (at + 9 < stream.Length)
    {
        stream.Position = at;
        stream.ReadExactly(marker);
        if (marker[0] != 0xFF)
        {
            at++;
            continue;
        }
        if (marker[1] is >= 0xC0 and <= 0xCF and not 0xC4 and not 0xC8 and not 0xCC)
        {
            return ((marker[7] << 8) | marker[8], (marker[5] << 8) | marker[6]);
        }
        at += 2 + ((marker[2] << 8) | marker[3]);
    }
}
return null;
```
*Live from [`api/TheYard.Api/DocImages.cs`](https://github.com/SteveStout/TheYard/blob/7830983/api/TheYard.Api/DocImages.cs#L74-L122), region `header-only`, read from this build at 7830983.*

### e. Workstation GC, concurrent, and a heap limit

Why: Server GC gives each core its own heap and is tuned for a machine of one's own; on a small shared plan, Workstation GC with concurrent collection keeps the footprint small. A heap limit makes a regression the runtime's problem inside its own share rather than the neighbours'.

Here: Workstation and concurrent since ADR-094; the limit is `DOTNET_GCHeapHardLimitPercent=0x1E` on each .NET site. Settings read from the environment are hexadecimal, so `0x1E` is thirty per cent, and a plain `30` would have been forty-eight.

```csharp
/// <summary>
/// Rule e. Both .NET sites carry a heap limit of thirty per cent, written in the hexadecimal the
/// runtime reads GC settings from the environment in, and keep their catalogues in the store.
/// </summary>
[Fact]
public void Each_site_has_a_heap_limit_of_thirty_per_cent_and_keeps_its_catalogue_in_the_store()
{
    string template = File.ReadAllText(Path.Combine(Repo.Root(), "infra", "appservice.bicep"));

    Assert.Contains("{ name: 'DOTNET_GCHeapHardLimitPercent', value: '0x1E' }", template, StringComparison.Ordinal);
    Assert.Equal(30, Convert.ToInt32("1E", 16));
    Assert.Contains("{ name: 'Catalogue__Keep', value: 'true' }", template, StringComparison.Ordinal);
}
```
*Live from [`api/TheYard.Tests/MemoryRulesTests.cs`](https://github.com/SteveStout/TheYard/blob/7830983/api/TheYard.Tests/MemoryRulesTests.cs#L254-L266), region `rule-e`, read from this build at 7830983.*

### f. Pool what is expensive

Why: a new `HttpClient` per call opens a new connection per call and can run a machine out of sockets; one per class is a pool per class. Microsoft's guideline for .NET 5 and later is one long-lived handler with `PooledConnectionLifetime` set, or `IHttpClientFactory` where clients are built by dependency injection. One `CosmosClient` per process, for the same reason.

Here: seven readers held seven static clients. They share one handler now, which retires each connection after two minutes so a changed DNS answer is picked up.

```csharp
/// <summary>
/// One <see cref="SocketsHttpHandler"/> for the process, and a cheap client over it per call. This
/// is the pooling IHttpClientFactory gives code that is built by dependency injection, in the
/// form Microsoft documents for code that is not: the handler holds the connections and is never
/// disposed, each connection is retired after two minutes so a changed DNS answer is picked up,
/// and a client is a wrapper that carries one caller's timeout. Seven readers each held a static
/// client of their own before this, seven pools for the same few hosts.
/// </summary>
public static class OutboundHttp
{
    private static readonly SocketsHttpHandler Handler = new()
    {
        PooledConnectionLifetime = TimeSpan.FromMinutes(2),
        PooledConnectionIdleTimeout = TimeSpan.FromMinutes(1),
    };

    /// <summary>
    /// A handler that answers in place of the network for the flow that set it, and only that
    /// flow: a test routes its own reader's calls to recorded answers while every other test,
    /// running beside it, still sees the real pool (and, having no credential, reads nothing).
    /// </summary>
    private static readonly AsyncLocal<HttpMessageHandler?> Routed = new();

    /// <summary>A client over the shared pool with one caller's timeout; disposing it leaves the pool alone.</summary>
    public static HttpClient Client(TimeSpan timeout) => new(Routed.Value ?? Handler, disposeHandler: false) { Timeout = timeout };

    /// <summary>Sends this flow's outbound calls to <paramref name="handler"/> until the returned scope is disposed. The suite's seam.</summary>
    public static IDisposable RouteTo(HttpMessageHandler handler)
    {
        var before = Routed.Value;
        Routed.Value = handler;
        return new Restore(before);
    }

    private sealed class Restore(HttpMessageHandler? before) : IDisposable
    {
        public void Dispose() => Routed.Value = before;
    }
}
```
*Live from [`api/TheYard.Api/OutboundHttp.cs`](https://github.com/SteveStout/TheYard/blob/7830983/api/TheYard.Api/OutboundHttp.cs#L8-L46), region `outbound-http`, read from this build at 7830983.*

### g. No static collection that grows with traffic

Why: a static field lives as long as the process, so a static list that every request adds to is a leak that no collection can fix. Logs and recent errors are ring buffers of a fixed size.

Here: none was found; the request, log, error and SQL rings were already rings. The collection test above holds that.

### h. Every app measured the same way

Why: a rule nobody measures is a guess. The Node rendering service runs under `--max-old-space-size=192`; whether that is tight or padded is a reading, not an opinion.

Here: the rendering service's `/readyz` answers its heap used against its limit, and each .NET site's `/api/admin/machines` answers the collector's own reading.

```csharp
/// <summary>
/// The garbage collector's own account of this process, read once: the mode in effect, the
/// heap against the limit it may grow to, the two generations that hold long-lived and large
/// objects, how much of the heap is pinned or fragmented, and the share of time spent in
/// collections. The numbers ADR: The store does the work records before and after.
/// </summary>
public static object GcReading()
{
    var info = GC.GetGCMemoryInfo(GCKind.Any);
    static double Mb(long bytes) => Math.Round(bytes / 1024d / 1024d, 1);
    var generations = info.GenerationInfo;
    return new
    {
        mode = System.Runtime.GCSettings.IsServerGC ? "server" : "workstation",
        concurrent = AppContext.TryGetSwitch("System.GC.Concurrent", out bool concurrent) ? concurrent : true,
        heap_size_mb = Mb(info.HeapSizeBytes),
        committed_mb = Mb(info.TotalCommittedBytes),
        limit_mb = Mb(info.TotalAvailableMemoryBytes),
        gen2_mb = generations.Length > 2 ? Mb(generations[2].SizeAfterBytes) : 0,
        large_object_heap_mb = generations.Length > 3 ? Mb(generations[3].SizeAfterBytes) : 0,
        fragmented_mb = Mb(info.FragmentedBytes),
        pinned_objects = info.PinnedObjectsCount,
        pause_percent = Math.Round(info.PauseTimePercentage, 2),
        allocated_mb = Mb(GC.GetTotalAllocatedBytes(precise: false)),
        collections = new { gen0 = GC.CollectionCount(0), gen1 = GC.CollectionCount(1), gen2 = GC.CollectionCount(2) },
    };
}
```
*Live from [`api/TheYard.Api/Machines.cs`](https://github.com/SteveStout/TheYard/blob/7830983/api/TheYard.Api/Machines.cs#L47-L73), region `gc-reading`, read from this build at 7830983.*

## Reading the collector's own numbers

`GET /api/admin/machines` answers, for the process that served it, the GC mode in effect, the heap against the limit it may grow to, generation 2 and the large object heap after the last collection, how much is pinned or fragmented, the share of time spent in collections, and how many vehicles the process holds. The two numbers worth watching first are the heap against the limit and the catalogue held, which reads zero.

## Further reading

- Microsoft Learn, [Fundamentals of garbage collection](https://learn.microsoft.com/en-us/dotnet/standard/garbage-collection/fundamentals)
- Microsoft Learn, [The large object heap on Windows](https://learn.microsoft.com/en-us/dotnet/standard/garbage-collection/large-object-heap)
- Microsoft Learn, [Runtime configuration options for garbage collection](https://learn.microsoft.com/en-us/dotnet/core/runtime-config/garbage-collector)
- Microsoft Learn, [Guidelines for using HttpClient](https://learn.microsoft.com/en-us/dotnet/fundamentals/networking/http/httpclient-guidelines)
- Microsoft Learn, [Tracking vs. no-tracking queries](https://learn.microsoft.com/en-us/ef/core/querying/tracking)
- ADR: The store does the work, the record behind this page