# Site traffic

**Who comes to this site, what it keeps about them, and how it gets found: 34 activity tests and 8 kept-log tests hold every rule on these pages.**

The site counts its own traffic and shows it on the Admin tab, to anybody. That is only safe because of what a visit is allowed to leave behind: a daily token, three octets of a network, the page asked for and the store that answered, and nothing that names a person. This section explains how visits are counted and kept and how the site is found. Every rule on it is one a test enforces.

## In plain words

This section explains how the site counts its own visitors and shows the totals to anyone on the Admin tab. That is safe because a visit keeps nothing that names a person, and a test holds every rule here. It also covers how search engines find the site.

What that is worth: a developer sees how to measure traffic without keeping personal data, and the organization can publish its numbers openly because nothing in them names a person.

## In this section

```tiles
traffic-who | Who comes | People, scanners and crawlers, and the site reading itself, told apart and counted. | author
traffic-kept | What is kept | What a visit leaves behind, where it is kept, for how long, and what is never kept. | records
traffic-found | Being found | Robots, the sitemap, the page about the author and the data search engines read. | about
traffic-search-console | Search Console, step by step | The site put into Google Search Console one screen at a time, with the code behind each step. | practices
```

## The standing rules

- No email address, no full IP address, no user agent and no query string is ever kept ([`Activity.cs`](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Application/Activity.cs), the hit's six fields, held by [ActivityTests](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Tests/ActivityTests.cs)).
- A visitor is a keyed hash of the day and the address, so nothing joins one day to the next.
- No request waits on the store: a visit is handed to a queue and written in the background.
- The totals are public; the per-visitor rows and the kept log answer only to the operator's key, and are a 404 without it.
- The site's own reads are marked and counted apart, so they never pass for people.
- Every document the site serves is in the sitemap, and a test keeps the two lists the same ([PublicFaceTests](https://github.com/SteveStout/TheYard/blob/main/api/TheYard.Tests/PublicFaceTests.cs)).

## Glossary of traffic terms

The words these pages use, in plain language, each linked to the section that shows it in use.

```glossary
Visitor token |  | A keyed hash of the day and the visitor's address. The same address is the same token on both sites for a day, and a new one tomorrow. | traffic-who#what-a-visit-carries | Who comes
Network |  | The address cut to its first three octets, enough to see a network or a scanner range and not enough to name a machine. | traffic-who#what-a-visit-carries | Who comes
Visitor-day |  | One visitor on one day, the unit every count on the card is made of. | traffic-who#three-kinds-of-visitor | Who comes
Self-read | TheYard-SelfRead | The mark the site's own tools carry on their requests, so a check the site runs on itself is not counted as a person. | traffic-who#three-kinds-of-visitor | Who comes
Referrer |  | The page that linked here. Only its host is kept, on a page load, and it is grouped as LinkedIn, GitHub, search, another site, or typed. | traffic-who#where-they-came-from | Who comes
Time to live | ttl | How long a document stays in Azure Cosmos DB before the database removes it by itself. | traffic-kept#how-long | What is kept
Ring |  | A fixed-size list held in memory that drops its oldest entry when it is full, which is what the Admin tab read before anything was kept. | traffic-kept#the-kept-log | What is kept
Operator's key | Admin__Key | The one secret that opens the per-visitor rows and the kept log. Without it both are a 404. | traffic-kept#public-and-keyed | What is kept
Sitemap | sitemap.xml | The list of every address the site wants found, read by search engines from the root of the domain. | traffic-found#the-sitemap | Being found
Structured data | application/ld+json | Facts about a page written for machines in schema.org's vocabulary: who the author is and where the code lives. | traffic-found#structured-data | Being found
llms.txt | /llms.txt | A plain index of the site for AI tools, pointing at the documents as markdown they can read without running the app. | traffic-found#for-ai-tools | Being found
Canonical | rel="canonical" | The one address a page wants to be listed under, so the two sites do not compete for it. | traffic-found#the-page-about-the-author | Being found
Search Console |  | Google's report on what its crawler sees: what is indexed, under which address, and what people searched to find it. | traffic-search-console | Search Console
Property |  | The unit Search Console reports on: a whole domain, proven by DNS, or one address and everything under it. | traffic-search-console#step-1-add-a-property | Search Console
Verification token | GOOGLE_SITE_VERIFICATION | The value Google issues to prove ownership. Here it is a repository variable the image build writes into the home page. | traffic-search-console#step-2-prove-ownership | Search Console
URL inspection |  | Search Console's answer for one address: whether it is indexed, how Google found it, and which canonical it chose. | traffic-search-console#step-4-inspect-a-url-and-ask-for-indexing | Search Console
```

## How it is enforced

```readouts
Activity tests | 34
Kept log tests | 8
Documents in the sitemap | 135
Crawler and head tests | 11
```

Numbers read from the build when the page is opened. The facts, by name:

```text
Hits_fold_into_one_delta_per_store_and_hour
Hits_fold_into_one_delta_per_store_day_and_visitor_with_first_and_last_seen
A_delta_keeps_the_top_paths_only_and_a_merge_keeps_twenty
The_token_is_thirty_two_hex_characters_the_same_within_a_day_and_different_across_days_and_keys
A_recorded_path_has_no_at_sign_and_no_tail
A_read_by_the_sites_own_tools_is_its_own_row_under_the_mark_and_not_a_person
The_address_is_the_one_the_forwarded_headers_middleware_resolved_and_never_a_raw_header
A_visitors_sources_are_folded_by_host_and_a_hit_with_none_adds_nothing
Every_visitor_day_is_one_kind_and_the_loopback_rows_of_a_day_are_one_machine
The_resume_opens_are_counted_by_door_people_only
A_resume_open_through_a_door_counts_in_the_resume_step
The_path_to_the_resume_counts_the_visitor_days_that_reached_each_step
The_hosts_that_linked_here_are_counted_in_visitor_days
A_visitor_day_is_counted_once_per_day_so_the_window_sums_the_days
The_admin_key_admits_the_whole_value_and_nothing_else
Every_hit_goes_to_the_keeper_and_keeps_the_store_that_served_it
A_full_channel_counts_the_hit_it_drops
The_report_carries_the_drop_count_the_cost_and_the_retention
A_keeper_with_no_unit_puts_no_cost_but_keeps_its_retention
A_keeper_that_is_down_puts_no_retention
The_public_report_has_a_series_per_store_and_no_at_sign_after_an_address_was_offered
A_window_that_is_not_one_of_the_three_is_refused
The_visitor_rows_are_a_404_without_the_key_and_carry_no_at_sign_with_it
The_same_batch_twice_adds_rather_than_overwrites
A_referring_host_is_kept_on_the_visitor_row_in_either_store
```

## The decisions behind it

- [ADR-071, Site activity, and the line an address does not cross](https://theyard.stevenstout.biz/?doc=adr-activity)
- [ADR-073, Logs that outlive the container](https://theyard.stevenstout.biz/?doc=adr-kept-logs)
- [ADR-053, The public face](https://theyard.stevenstout.biz/?doc=adr-public-face)
- [ADR-020, Every diagram opens on its own page](https://theyard.stevenstout.biz/?doc=adr-diagrams), the addendum on the page about the author