Juice.Observability.Agent 1.2.0

Juice Observability Platform

Turns telemetry from Juice services into a resource model, health, rules, alerts and investigation workflows.

Build and test

Needs the .NET 10 SDK. Integration tests need the local stack running with --profile test and are skipped otherwise.

deploy/local/scripts/init.sh && (cd deploy/local && docker compose --profile test up -d --wait)
dotnet test
deploy/local/scripts/run-api.sh          # API on http://localhost:5180, migrates and provisions the tenant
(cd console && npm ci && npm start)      # Console on http://localhost:4200, proxying /api to :5180

docker compose --profile platform up -d --build serves the Console and the API together on http://localhost:5180. console/e2e/journey.mjs signs in through Keycloak and walks the investigation journey (Console requirements §21) in a headless browser.

Build output goes to build/. Add a migration with dotnet tool restore && dotnet ef migrations add <Name> --project src/Juice.Observability.Data --output-dir Migrations; MigrationTests fails when the model changes without one.

Packages and CI

Besides the image, the platform ships as libraries for a host app of the customer's own:

src/Juice.Observability.Host and console/src are the reference host and app: they use the packages the way a customer would.

var builder = WebApplication.CreateBuilder(args);
builder.Configuration.AddJuiceObservabilityFiles();
builder.Services.AddJuiceObservability(builder.Configuration);
var app = builder.Build();
app.UseCorrelationId();
app.UseAuthentication();
app.UseAuthorization();
app.MapJuiceObservability();
app.Run();

Publishing (.github/workflows): the backend and the console library are versioned and released independently.

Each next version is the one to release: raise it after a release so alphas sort above it. Tags have no v.

The console library calls the API over HTTP, so record which versions work together:

API

All routes need a bearer token from the deployment's OIDC provider, with audience juice-platform and a platform role (viewer, operator or admin) in the roles claim. Errors are RFC 9457 problem details with correlationId and traceId; send X-Correlation-Id to set your own. Lists take limit (1–200, default 50) and cursor (the previous page's nextCursor).

A resource's detail also carries its health per environment and each instance's state. With Integrations:Grafana:BaseUrl set, a service and an alert also carry a link to the optional Grafana overview; without it, links is empty. Telemetry queries are platform-owned templates, tenant-filtered and bounded; services, instances, resources and dependencies are named by platform id, never by telemetry names from the caller.

{tenantId} must be the caller's tenant; any other id, existing or not, gets the same 403. /health/live and /health/ready are anonymous.

A local token, for example for victor (viewer):

source deploy/local/.env
curl -s -d grant_type=password -d client_id=juice-platform-cli -d username=victor \
  -d "password=$KEYCLOAK_DEMO_PASSWORD" http://localhost:8080/realms/juice/protocol/openid-connect/token

juice-platform-cli exists only in the local realm, for development and tests.

Discovery

Every 30 s the worker reads telemetry from its checkpoint to one minute ago, in windows of at most 15 minutes, as platform_reader. Every query filters on the Collector-stamped tenant. It reconciles each window and moves the checkpoint in one transaction.

Each window also re-reads the 10 minutes before the checkpoint (Discovery:Overlap). Telemetry is stored by its own time, so a row can land after its window was read: a long span is written when it ends but timed by its start, and the Collector holds data back while ClickHouse is slow. Upserts make the re-read harmless. A dependency's span count adds only spans after the checkpoint, unless the dependency is first seen in the overlap. Telemetry later than one minute plus the overlap is still missed.

• Services are keyed by namespace and name, across environments. Each namespace is one application.

• Instances are keyed by service.instance.id, or by host and process id (marked derived). An instance goes inactive after 15 minutes without telemetry and is deleted 30 days later.

• Resources:

  Resource	Key
  Host	host.id, else host.name
  Container	container.id
  Pod	k8s.pod.uid
  Messaging destination	system + destination
  Database	system + server + namespace

• Dependencies:

  • A client or producer span whose child server or consumer span is in another service gives a dependency between the two services, confidence observed.
  • A span naming a destination or a database gives a dependency on that resource, confidence inferred, since the resource itself never reports.

• Renamed services map through Discovery:Aliases.

• Limits. A window that fills Discovery:RowLimit rows is split, down to 10 s. Values are cut to 256 characters and attributes to 64. Both are counted in juice.observability.discovery.truncated.

Health

Every 30 s the health worker reads a 5-minute evidence window ending 10 s ago. It evaluates:

• each active instance;
• each service in every environment it was seen in;
• each application per environment;
• each host.

It writes one current snapshot per subject, and a transition only when the state changes.

• Instances. The error rate of server and consumer spans: 1 % is Degraded and 5 % is Unhealthy, counted only once there are at least 3 requests. With less traffic, recent telemetry alone is a heartbeat.

• Services:

  • a failed critical dependency (Health:CriticalDependencies) makes the service Critical;
  • if half or more of its fresh instances are unhealthy, it is Unhealthy;
  • any partial failure, or an elevated service error rate, makes it Degraded.

  Instances that went silent after a restart don't count.

• Applications. Children are weighted, optionally critical (Health:ServiceWeights), with a minimum coverage of 50 %. Missing optional services never make an application unhealthy.

• Reasons are stable codes, such as partial_instance_failure or critical_dependency_failed:shop/billing.

• Policy version. Each snapshot and transition carries the policy version, a hash of the rule settings.

• Replays. A transition is unique per subject, evidence window and policy version, so re-evaluating the same evidence never adds one.

• ClickHouse unavailable. The pass fails and the states are kept. The API marks a state stale once it is older than three intervals.

• Freshness. A new failure is visible after about the evidence lag (10 s) plus ingestion; tested at 10–11 s with a 1 s evaluation loop. With the 30 s interval the bound is about 42 s, against the 60 s target. juice.observability.health.evidence_age records it in operation.

Rules and alerts

Rules are deployment configuration. Rules:File names a JSON file (deploy/local/rules.json) that is validated and loaded at startup:

• an invalid rule stops the host;
• each create, update or disable writes an audit record (actor deployment-config);
• the API only reads rules.

Each rule is scoped by Service and Environment (or Host). It also has a schedule (default 1 min), a window, a threshold and a recovery threshold (hysteresis), minimum samples, ForDuration, RecoveryDuration and Cooldown.

• Lifecycle. A breach opens a Pending alert. It becomes Firing once the breach has held for ForDuration, and Resolved after RecoveryDuration at or below the recovery threshold.
  • A Pending alert that stops breaching resolves without ever firing, so a transient breach never fires.
  • Values between the two thresholds keep a Firing alert firing.
  • After an alert that fired resolves, a new one can't fire until Cooldown has passed.
• Deduplication. There is one open alert per rule and subject, enforced by a unique index.
• No data. Too few samples, or no rows, means no data. It is recorded on the alert and holds the alert's state; it is never read as zero.
• Scheduler. It runs 4 evaluations at once, each cut off after 20 s. A rule still running when due again is skipped, never stacked.
  • Metrics: juice.observability.rules.evaluations, .duration, .lag, .skipped and juice.observability.alerts.transitions.
• Breach-to-alert. Firing comes at most lag (10 s) plus one schedule after ForDuration: 70 s for a 1-minute rule, against the 2-minute target. Measured in tests at 22 s with a 5 s schedule and a 10 s ForDuration.

Notifications and incidents

Rules, channels, the allowlist and escalation can be changed through the API and the Console, by admins, with an audit record. The deployment's files still work: a row belongs to whichever side created it, and the runbook has the details.

A channel chooses what it takes by event, rule, environment and severity, and can group what arrives in a window into one notification. A silence or maintenance window holds notifications, not alerts, and what it held goes out when it ends if it still matters. An escalation policy tells more channels as an alert stays open, and stops when the alert resolves or, if the policy says so, when it's acknowledged. An incident groups alerts into a piece of work with its own record, and doesn't change them.

Status: see Phase 6 acceptance, including what hasn't been tried against real Slack, Teams and TLS mail servers.

Changes and correlation

A change event comes from one of three places:

• Telemetry: a new service.version from a new instance is recorded with no actor, since telemetry shows that it changed, not who changed it. A service's first version isn't a change, and old instances still reporting during a rollout don't read as going back.
• A pipeline: POST /changes (operator or admin). The actor comes from the token. Reporting the same source and externalKey again returns the first event.
• The platform: its own rules and webhooks when configuration loads them.

curl -X POST "$PLATFORM_URL/api/v1/tenants/$TENANT_ID/changes" -H "Authorization: Bearer $PLATFORM_TOKEN" -H 'Content-Type: application/json' -d '{
  "kind": "deployment", "summary": "Deployed orders 1.4.2", "service": "shop/orders", "environment": "production", "toVersion": "1.4.2",
  "source": "github-actions", "externalKey": "run-4211", "attributes": { "commit": "abc123", "url": "https://ci.example.com/runs/4211" }
}'

Secrets: only summaries are stored. Attributes are allowlisted (commit, branch, tag, pipeline, build, image, ticket, url, reason); anything else is dropped, name included. Text is masked for passwords, tokens, API keys, URL credentials and connection strings before it's stored. That catches common shapes, not every secret, so send a summary, not a command or a log.

Correlation: an alert's page lists what happened from 2 hours before its breach to 5 minutes after, on its subject, on what the subject directly depends on, and to its rule. Each item's score is the sum of points shown with their reasons:

This is a reading order, not a diagnosis. Being close in time is correlation, and the API and the Console say so wherever these are shown.

Load test runs

The platform generates no load. A load test marks its window, and the Console's Load runs page reads each service's telemetry from it, compares runs, and shows where a service started to struggle, which everyday traffic can't reveal. Marking needs a token with the operator or admin role. The load testing guide explains how to scope and write tests. With k6:

import http from 'k6/http';

const runs = `${__ENV.PLATFORM_URL}/api/v1/tenants/${__ENV.TENANT_ID}/load-runs`;
const headers = { Authorization: `Bearer ${__ENV.PLATFORM_TOKEN}`, 'Content-Type': 'application/json' };

export const options = { stages: [{ duration: '5m', target: 50 }, { duration: '5m', target: 200 }] };

export function setup() {
  const body = JSON.stringify({ name: 'checkout, up to 200 users', environment: 'staging' });
  return http.post(runs, body, { headers }).json();
}

export default function () { http.get(__ENV.TARGET_URL); }

export function teardown(run) { http.post(`${runs}/${run.id}/end`, null, { headers }); }

A run with no end, because the test died, is read up to now. A limit is the first time step of two in a row where one of these holds:

Figures come from spans, so they undercount if traces are sampled. Steps the window cuts short are left out, and a step runs from 15 s to an hour by the run's length. The broker's own queue depth and the database aren't measured.

Webhooks

When an alert fires, or resolves after firing, one delivery per matching endpoint is written in the same transaction as the alert change. A webhook can therefore never roll an alert back, and replaying an evaluation never notifies twice. The dispatcher sends every 2 s.

• Signing. HMAC-SHA256, Standard Webhooks form: webhook-id, webhook-timestamp, webhook-signature: v1,<base64> over {id}.{timestamp}.{body}.
• Destinations. HTTPS only, no redirects. DNS is checked on every connection: one internal answer blocks the destination. Private addresses need an allowlist entry. Loopback, link-local and cloud metadata are never reachable.
• Retries. Transient failures retry with exponential backoff and jitter (10 s to 10 min, 8 attempts). Other 4xx fail at once. Every attempt is recorded.

Configuration, secrets and receiver guidance are in docs/deployment.md.

Juice Agent

A Juice Agent runs on a host and answers read-only diagnostics: host.summary, disk.usage and tcp.reachability. It connects only outbound, to the platform's agent listener (Agents:Port, 8443; 5181 on the local stack), over mTLS. It enrolls once with a one-time token from an admin, sending a CSR for a key it generates and keeps, and renews its 7-day certificate itself.

Every command is signed by the platform's command key and checked again by the agent before it runs: the signature, that it names this agent and tenant, its time window and lifetime, its nonce (a replay is refused), the action and its parameter bounds, and this host's own allowlist, which wins over a valid signature. There is no action that runs a command, a script or an executable.

An admin can update an agent with agent.update to a package signed by a release key, kept offline. The host must trust that key in its own configuration, and the platform can't supply one. The agent's launcher runs the new version on trial: it keeps the version if it reaches the platform within 2 minutes, and goes back to the version before otherwise. The platform's command key and agent CA can be rotated without enrolling agents again. See Phase 7 acceptance, the runbook and deployment.

License

MIT