# Who comes

**Every page and API request the site serves is counted once, by the store that served it, as a visit that names nobody.** The files a page is made of (scripts, styles, fonts, photographs, and the text and XML files at the root) are not counted, because one visit is one page and not the forty files behind it, and neither are the Admin tab's own reads. The Admin tab draws the count day by day, split three ways: people, scanners and crawlers, and the site reading itself.

## In plain words

This page says how the site counts who visits it. Every page or API request is counted once as a visit that names nobody. People are counted apart from bots, and the site's own checks are counted apart from both.

What that is worth: a developer sees a visitor count that groups a day's requests without ever storing a full address, and the organization gets a count of real readers that its own tools cannot inflate.

## What a visit carries

A visit is handed to the counter as a record with six fields, and a record with no field for a thing cannot carry it:

```csharp
// Site activity, kept by the store that served the request (ADR: Site
// activity, and the line an address does not cross). One port, two adapters,
// the same shape as bids: the relational side keeps two tables, the document
// side keeps two kinds of document in one container, and the Admin tab reads
// either through this.
//
// What a hit carries is the whole privacy decision, so it is decided here and
// not in an adapter. Seven fields: when, a visitor token that is a keyed hash
// of the address and rotates daily, the address cut to its first three octets,
// the path with no query string, the store, whether the request looked like a
// bot, and on a page load the host of the page that linked here (the
// host only, never its path or query, and not personal information, so it is
// as public as the rest of the tab). No user agent, no
// account, no email, no full address, no query string. A row that has no
// field for a thing cannot leak it.

/// <summary>One request as the activity feature sees it. The type has no room for anything a person could be named by.</summary>
/// <param name="At">When the request arrived.</param>
/// <param name="Visitor">A keyed hash of the address that rotates daily, so one visitor can be counted without being named.</param>
/// <param name="Network">The address cut to its first three octets.</param>
/// <param name="Path">The path requested, with no query string.</param>
/// <param name="Store">The store that served the request.</param>
/// <param name="Bot">True when the request looked like a bot.</param>
/// <param name="Source">On a page load, the host of the page that linked here; null otherwise.</param>
public sealed record ActivityHit(DateTimeOffset At, string Visitor, string Network, string Path, string Store, bool Bot, string? Source = null);

/// <summary>One store's requests in one UTC hour, with the paths it served most, for the graph.</summary>
/// <param name="Store">The store that served the requests.</param>
/// <param name="Hour">The start of the UTC hour.</param>
/// <param name="Requests">How many requests the store served in the hour.</param>
/// <param name="Bots">How many of those requests looked like bots.</param>
/// <param name="Paths">The most requested paths in the hour, with a request count for each.</param>
public sealed record ActivityHour(string Store, DateTimeOffset Hour, int Requests, int Bots, IReadOnlyDictionary<string, int> Paths);

/// <summary>One visitor token on one store on one UTC day, for the table nobody anonymous can read.</summary>
/// <param name="Store">The store the visitor reached.</param>
/// <param name="Day">The UTC day as yyyy-MM-dd.</param>
/// <param name="Visitor">The visitor token, a keyed hash of the address that rotates daily.</param>
/// <param name="Network">The address cut to its first three octets.</param>
/// <param name="FirstSeen">The visitor's first request that day.</param>
/// <param name="LastSeen">The visitor's last request that day.</param>
/// <param name="Requests">How many requests the visitor made that day.</param>
/// <param name="Bots">How many of those requests looked like bots.</param>
/// <param name="Paths">The visitor's most requested paths, with a request count for each.</param>
/// <param name="Sources">The hosts that linked the visitor here, with a page-load count for each; null when none were kept.</param>
public sealed record ActivityVisitor(
    string Store,
    string Day,
    string Visitor,
    string Network,
    DateTimeOffset FirstSeen,
    DateTimeOffset LastSeen,
    int Requests,
    int Bots,
    IReadOnlyDictionary<string, int> Paths,
    IReadOnlyDictionary<string, int>? Sources = null);

/// <summary>
/// Whether a store can keep activity right now, and if not, why, in one
/// sentence a page can show. <paramref name="Retention"/> is how long a kept
/// row lasts, said by the store that keeps it ("kept in Azure Cosmos DB with no
/// expiry"); null when the store is down or does not know.
/// </summary>
/// <param name="Available">True when the store can keep activity right now.</param>
/// <param name="Reason">One sentence a page can show saying why activity is or is not kept.</param>
/// <param name="Retention">How long a kept row lasts, as the store says it; null when the store is down or does not know.</param>
public sealed record ActivityAvailability(bool Available, string Reason, string? Retention = null);

/// <summary>What keeping activity has cost a store: request units, operations, and the operations that failed.</summary>
/// <param name="RequestUnits">The request units spent keeping activity.</param>
/// <param name="Operations">How many store operations keeping activity took.</param>
/// <param name="Failures">How many of those operations failed.</param>
public sealed record ActivityCost(double RequestUnits, int Operations, int Failures);

/// <summary>
/// What keeping activity has cost a store since the process started, for a
/// store that has a unit to count it in: request units on the document store.
/// Beside the port rather than on it, because the relational store has no such
/// unit and a fake in a test should not have to invent one.
/// </summary>
public interface IActivityCost
{
    ActivityCost Cost { get; }
}

/// <summary>Port: where activity is kept. Writes arrive in batches, never from a request thread.</summary>
public interface IActivityStore
{
    Task<ActivityAvailability> AvailabilityAsync(CancellationToken cancellation);

    Task RecordAsync(IReadOnlyList<ActivityHit> hits, CancellationToken cancellation);

    Task<IReadOnlyList<ActivityHour>> HoursAsync(DateTimeOffset since, CancellationToken cancellation);

    Task<IReadOnlyList<ActivityVisitor>> VisitorsAsync(DateTimeOffset since, CancellationToken cancellation);
}

/// <summary>The port wired to nothing: a store that did not come up keeps no activity and says so.</summary>
public sealed class NullActivityStore(string reason) : IActivityStore
{
    public static readonly NullActivityStore Instance = new("this store keeps no activity");

    public Task<ActivityAvailability> AvailabilityAsync(CancellationToken cancellation) =>
        Task.FromResult(new ActivityAvailability(false, reason));

    public Task RecordAsync(IReadOnlyList<ActivityHit> hits, CancellationToken cancellation) => Task.CompletedTask;

    public Task<IReadOnlyList<ActivityHour>> HoursAsync(DateTimeOffset since, CancellationToken cancellation) =>
        Task.FromResult<IReadOnlyList<ActivityHour>>([]);

    public Task<IReadOnlyList<ActivityVisitor>> VisitorsAsync(DateTimeOffset since, CancellationToken cancellation) =>
        Task.FromResult<IReadOnlyList<ActivityVisitor>>([]);
}
```
*Live from [`api/TheYard.Application/Activity.cs`](https://github.com/SteveStout/TheYard/blob/1d6b50b/api/TheYard.Application/Activity.cs#L4-L116), region `activity-port`, read from this build at 1d6b50b.*

- **When**, to the hour.
- **The visitor token**, a keyed hash of the day and the address. Within a day the same address is the same token on both sites, so one person's requests group together; the next day it is a different token, so nothing joins one day to the next. The key is the signing key the containers share and nobody outside has, which is what stops anybody turning a token back into an address by hashing every IPv4 address there is.
- **The network**, the address cut to its first three octets (`203.0.113.x`): enough to see a network, a country or an obvious scanner range, and not enough to name a machine. A full address is never stored, on either store.
- **The path**, with any at sign percent-encoded so an address pasted into a URL cannot travel. The query string is never recorded.
- **The store** that served the request (Azure SQL Database or Azure Cosmos DB), so the graph keeps a line for each site.
- **Whether it looked like a bot**, read from the request's user agent and what it asked for. The agent itself is read and forgotten.

```csharp
/// <summary>
/// The address, made into something the page can group by and nobody can
/// reverse. A keyed hash of the day and the address: keyed with the signing
/// key, which every container shares and nobody outside has, so the same
/// address is the same token on both sites within a day and a different one
/// tomorrow; the day is in the hash rather than in a salt table, so there is
/// no table of salts to keep or to lose. The address itself is kept only as
/// its first three octets, which is enough to see a network, a country and an
/// obvious scanner range, and not enough to name a machine. A full address is
/// never stored, and a full address is never sent anywhere from here.
/// </summary>
public sealed class VisitorTokens(string key)
{
    /// <summary>The signing key as bytes, the key of every hash this makes.</summary>
    private readonly byte[] _key = Encoding.UTF8.GetBytes(key);

    /// <summary>Thirty-two hex characters: the first sixteen bytes of the keyed hash.</summary>
    public string TokenFor(string address, DateTimeOffset at)
    {
        byte[] hash = HMACSHA256.HashData(_key, Encoding.UTF8.GetBytes(at.UtcDay + "|" + address));
        return Convert.ToHexStringLower(hash.AsSpan(0, 16));
    }

    /// <summary>203.0.113.7 becomes 203.0.113.x; an IPv6 address keeps its first three groups and ends in x; anything unreadable is x.</summary>
    public static string NetworkOf(string? address)
    {
        if (string.IsNullOrWhiteSpace(address))
        {
            return "x";
        }

        string[] octets = address.Split('.');
        if (octets.Length == 4 && octets.All(octet => int.TryParse(octet, out int value) && value is >= 0 and <= 255))
        {
            return $"{octets[0]}.{octets[1]}.{octets[2]}.x";
        }

        if (address.Contains(':', StringComparison.Ordinal))
        {
            string[] groups = address.Split(':');
            return string.Join(':', groups.Take(3)) + ":x";
        }

        return "x";
    }

    /// <summary>
    /// The visitor's address. The forwarded-headers middleware has already written
    /// it onto the connection by the time anything reads it, counted back from the
    /// right of X-Forwarded-For past the proxies this app runs behind, so for a visitor
    /// who comes through the edge, a value they typed into the header is never the one
    /// used (ADR: The order of the request pipeline). Until then this read the edge's
    /// own header, and without it the leftmost X-Forwarded-For entry, which is the one
    /// a visitor writes. An address is what feeds a count, never a permission.
    /// </summary>
    public static string AddressOf(HttpContext context) =>
        WithoutPort(context.Connection.RemoteIpAddress?.ToString() ?? "");

    /// <summary>
    /// The address without the port a forwarding hop may have written after
    /// it. Since the forwarded-headers middleware resolves the address, the
    /// connection's address never carries a port and this is a guard rather than
    /// a fix; it stays because the rule it holds still matters. App Service's own
    /// requests reached the container as 127.0.0.1 with a port, and with the port
    /// left on, it went into the token and the
    /// network: every one of those requests was a new visitor, hundreds a day,
    /// and the network showed the whole address. 203.0.113.7:443 is
    /// 203.0.113.7; [2001:db8::1]:443 is 2001:db8::1; an address with no port
    /// is itself.
    /// </summary>
    public static string WithoutPort(string address)
    {
        if (address.StartsWith('['))
        {
            int close = address.IndexOf(']', StringComparison.Ordinal);
            return close > 1 ? address[1..close] : address;
        }

        int colon = address.IndexOf(':', StringComparison.Ordinal);
        return colon > 0 && colon == address.LastIndexOf(':') && address[..colon].Count(c => c == '.') == 3
            ? address[..colon]
            : address;
    }
}
```
*Live from [`api/TheYard.Api/Activity.cs`](https://github.com/SteveStout/TheYard/blob/1d6b50b/api/TheYard.Api/Activity.cs#L25-L108), region `visitor-token`, read from this build at 1d6b50b.*

A port after an address is dropped before the token is made, so one machine is one visitor whatever port it came from.

## Three kinds of visitor

Each visitor-day is exactly one of three kinds, and the three add up to the day's total:

- **People.** Everybody who made a request that did not look like a bot.
- **Scanners and crawlers.** A token whose every request looked like a bot (by its agent or by what it asked for): the probes for WordPress, `.env` and `.git` that every public address gets, and the search engines, whose agents name themselves.
- **The site's own reads.** The tools that check the site on its owner's behalf, the page sweep, the release's own readers and App Service's health check among them, carry `TheYard-SelfRead` in their user agent or come from the machine itself, and are counted apart so they never pass for people. A stranger could claim the mark and hide from the count; that costs a count of visitors and reaches nothing else.

The chart stacks the three day by day, people at the bottom, in three colours chosen as a set that stays apart under colour blindness ([Colour and style](https://theyard.stevenstout.biz/?doc=color-style#chart-colours)). Visitors only is the default view; the lines by store are a click away.

## Where they came from

On a page load, and only then, a visit also keeps the lowercased host of the page that linked here and nothing else of it, so neither a path nor a query that could say something about the reader reaches a row. The hosts are grouped on the card as LinkedIn, GitHub, search, another site, or typed and unknown. A link opened from a PDF, the resume among them, sends no referrer, so it reads as typed unless the link carries a tag of its own.

## The path through

The card also counts, in visitor-days, how many readers took each step of the path a reader takes through a portfolio: the landing page, the inventory, the page about the author, and the resume. Each step is a set of paths, and the resume counts both addresses it is served from.

## The decisions behind it

- [ADR-071, Site activity, and the line an address does not cross](https://theyard.stevenstout.biz/?doc=adr-activity), with its addenda on the three kinds, the referrers and the path.