Xperience.Telemetry 0.2.0

Xperience.Telemetry

Intended for Xperience by Kentico SaaS environments, to get more robust statistics than the Xperience Portal provides.

dotnet add package Xperience.Telemetry
dotnet add package Xperience.Telemetry.Kentico

Using it

Required: set CustomApplicationInsights:ConnectionString to the connection string of your own Application Insights resource (see CustomApplicationInsights). It is what turns the telemetry on. Without it AddXperienceTelemetry logs a warning at startup and registers only the watchdog and the probe: no metrics, no AppStarted / AppStopping events, no forwarding to your resource and no scheduled captures, and AddXperienceTelemetryKentico adds no cache metrics.

// Every environment: the probe behind POST gc/pinned, which does nothing until it is called.
services.AddXperienceTelemetryProbe();

// Where the watchdog and the telemetry should run (SaaS), after Application Insights is registered.
// Needs CustomApplicationInsights:ConnectionString to be set (see above).
services.AddXperienceTelemetry(configuration);
services.AddXperienceTelemetryKentico(configuration);     // after the line above

// Once the app is built. The delegate decides who gets in; it is required so the endpoints can't be left open.
app.MapXperienceTelemetryEndpoints(MyAuth.AuthorizeAsync);
app.MapXperienceTelemetryKenticoEndpoints(MyAuth.AuthorizeAsync);

Calling AddXperienceTelemetry more than once has no effect. The two Map methods take an optional second argument, the prefix, api/telemetry unless given. The Kentico endpoints sit under cache in the same prefix, so give both calls the same one. The default is specific to this package, so it doesn't collide with an app's own endpoints.

The delegate is a Func<HttpContext, ValueTask<bool>>, so whatever sign-in the app already has can be used. Returning false answers 401. It may also throw, which leaves the response to the app's exception handling. The POST endpoints change the state of the process (a forced GC), so unless the app protects against cross-site requests the delegate shouldn't rely on credentials a browser attaches by itself, such as cookies. Every response carries Cache-Control: no-store.

To put an application's own diagnostics endpoints behind the same check, MapXperienceTelemetryAuthorizedGroup returns the route group the endpoints above sit in:

app.MapXperienceTelemetryAuthorizedGroup(MyAuth.AuthorizeAsync, "api/telemetry/mine").MapGet("ping", () => "pong");

Configuration

Everything has a default, so nothing has to be configured: a section only needs to exist to change values, and a key left out keeps its default. It is plain .NET configuration, so appsettings.json, environment variables (GcMaintenance__RestartPrivateMB) or any other provider works. This is every key at its default:

{
  "GcMaintenance": {
    "Enabled": true,
    "CompactionEnabled": true,
    "CompactionFragmentedMB": 1000,
    "CompactionMinIntervalMinutes": 10,
    "CompactionMaxIntervalMinutes": 60,
    "RestartEnabled": false,
    "RestartPrivateMB": 2600,
    "RestartQuietHoursPrivateMB": 2000,
    "QuietHoursStart": 1,
    "QuietHoursEnd": 6,
    "QuietHoursTimeZone": "UTC",
    "MinUptimeMinutes": 60,
    "CheckIntervalSeconds": 30
  },
  "PinnedObjectsSnapshots": {
    "Enabled": true,
    "FirstAfterMinutes": 20,
    "IntervalMinutes": 30,
    "MaxPerProcess": 24,
    "MaxPauseMs": 1500,
    "CheckIntervalSeconds": 30
  },
  "DatabaseTelemetry": {
    "Enabled": true,
    "IntervalSeconds": 60,
    "CommandTimeoutSeconds": 10
  },
  "CustomApplicationInsights": {
    "ConnectionString": "required: the connection string of your own Application Insights resource",
    "EnableDependencyTracking": true
  }
}

The memory defaults assume a host limit of 3 GB (3,072 MB). Private bytes run about 120 MB above what the GC has committed, and a process is killed at around 3,130 MB of them, so the restart thresholds leave a margin below that. On a host with a different limit, scale CompactionFragmentedMB, RestartPrivateMB and RestartQuietHoursPrivateMB. Restarting stays off until RestartEnabled is turned on (see below).

GcMaintenance

The watchdog (GcMaintenanceOptions). By default it forces a compacting GC when gen2 holds 1,000 MB of free space. Restarting is up to you: it stops the app gracefully so that the host starts a new process, which only helps where the host does that, so it stays off until you set RestartEnabled. Once it is on, the app restarts at 2,600 MB of private bytes (2,000 MB during the quiet hours, 01:00 to 06:00 UTC), never before the process has been up an hour, and always after trying a compaction first.

PinnedObjectsSnapshots

Scheduled pinned objects captures (PinnedObjectsSnapshotOptions), reported as PinnedObjectsReport events with the trigger Scheduled, so the series can be read from Application Insights without calling the endpoint. They only run where CustomApplicationInsights:ConnectionString is set. Each capture stops the app for about as long as one full GC.

DatabaseTelemetry

How much of its limits the database used (DatabaseTelemetryOptions, Kentico package), reported as the Database.* metrics. It reads sys.dm_db_resource_stats, which only Azure SQL Database has: on any other database the service logs one line and stops. It only runs where CustomApplicationInsights:ConnectionString is set, and reads through Kentico's own connection, so it needs no connection string of its own.

CustomApplicationInsights

Read by AddXperienceTelemetry and AddXperienceTelemetryKentico when they register services.

When a change takes effect

GcMaintenance and PinnedObjectsSnapshots are read through IOptionsMonitor, so a configuration that reloads (a changed appsettings.json on a host that watches it) takes effect at the next check. The exception is CheckIntervalSeconds, which is read once when the service starts, as are DatabaseTelemetry:Enabled and IntervalSeconds; CommandTimeoutSeconds is read at every query. The CustomApplicationInsights keys decide what gets registered, so they are read once, at startup. The GcMaintenanceStarted event records the GcMaintenance values a process started with.

What it sends

The names are the contract: dashboards and queries depend on them, so changing one is a breaking change.

Custom events

• AppStarted, AppStopping: process id, machine, process start, and the version and BuildCommit / BuildBranch / BuildNumber of the entry assembly (the last three come from AssemblyMetadata attributes with those keys, which the app's build has to stamp in). A process killed for running out of memory doesn't send AppStopping; its absence before the next AppStarted is the signal.
• GcConfig (once, at startup): the GC settings that reached the process, which the host may have overridden.
• GcMaintenanceStarted (its options), GcCompaction (what a forced compaction achieved) and MemoryWatchdogRestart (why the app was restarted).
• PinnedObjectsReport (trigger Manual or Scheduled) and PinnedObjectsSnapshotsStopped.

Metrics

• Memory.* and GC.* once a minute: working set, private bytes, limit, committed, fragmentation, generation sizes, collection counts, pause per minute, allocation rate, the last full blocking and background GC (GC.FullBlocking.*, GC.Background.*), thread pool.
• GC.Events.*, GC.PinnedObjects.Max and GC.GcHandles.Max, counted from the runtime's own GC events.
• GC.Compaction.PauseMs and GC.Compaction.ReclaimedMB.
• Cache.Entries, Cache.KeyFamilies and Cache.EntriesByFamily (Kentico package).
• Database.CpuPercent, Database.DataIoPercent, Database.LogWritePercent and Database.MemoryPercent (the average over each 15 second interval), Database.WorkerPercent and Database.SessionPercent (the highest in it), all as a percentage of the limit of the service tier, and Database.DtuLimit (Kentico package; Azure SQL Database only). One value per interval, stamped with the time the database gave it. At 100 Database.WorkerPercent the database refuses new requests.

Requests get a UserAgent property. With a custom resource, requests, slow or failed dependencies, exceptions, traces, events and metrics are also forwarded to it, before sampling.

Endpoints

Under the prefix (api/telemetry by default), so GET api/telemetry/gc and so on:

Decisions worth knowing

• Pinned objects are read from the runtime's events. PinnedObjectsProbe listens to the GC keyword at verbose level for the length of one forced gen2 GC. It stops the app for as long as that collection takes (roughly 60 to 200 ms on a heap of a few hundred MB), at most one at a time and one every two minutes. The runtime filters events per listener, so the verbose session doesn't reach the other listeners (GcEventListener stays at informational).

• The forwarder has to be the last initializer. It copies what the others have filled in (user agent, role name, operation id). It is added when the TelemetryConfiguration is created, which is after the Application Insights registration has added the rest, and TelemetryForwardingService, the first hosted service, moves it to the end if the registrations were made in the other order. That service also registers the flush on ApplicationStopping first, so, as those callbacks run in reverse, it runs after AppStopping and MemoryWatchdogRestart have been sent. The SDK reads IConfigureOptions<TelemetryConfiguration> but ignores IPostConfigureOptions, which is why Configure is used.

• A compaction is always tried before a restart, and doesn't repeat for a while (doubling up to an hour) if it reclaimed little, which is what happens when pinned objects are in the way.

• Restarting is opt-in. It ends the process with StopApplication and relies on the host to start another one. Where the host doesn't, turning it on is an outage; where the limit is higher than the defaults assume, it restarts a healthy process.

• ObjectSizeEstimator skips what it can't walk safely: delegates, reflection types, tasks, threads, DI service objects and anything in Microsoft.AspNetCore.*. It sizes IntPtr and UIntPtr as pointers, because their only field is of their own type, so asking for their layout would never end.

• Database samples are read from the database's own history. sys.dm_db_resource_stats keeps a sample every 15 seconds for about an hour, so each poll asks for everything newer than the last sample it reported instead of taking a reading. A poll that fails (a database that has run out of workers can refuse this query too) leaves that point where it is, and the next poll that works reports what the failed ones missed, under the times the database gave the samples. Nothing from before the process started is reported, because an earlier process already did. Every instance of an application reports the same database-wide samples, so chart them by maximum, not sum.

Tests

dotnet test Xperience.Telemetry.slnx

They use the real GC, the real runtime events and a test server. The GC-sensitive ones share one process, so the assembly turns off parallel execution. One test allocates hundreds of megabytes and takes seconds, so it only runs when asked to:

XPERIENCE_TELEMETRY_STRESS=1 dotnet test Xperience.Telemetry.slnx

What the services report, under which names, is pinned by tests that call the reports directly. Not covered: the timers that call them on a schedule, and anything that needs a running Kentico site.

Releases

Both packages are published to nuget.org by .github/workflows/publish.yml when a v*.*.* tag is pushed, after the tests pass.

License

MIT