Documentation.AdaptivePolling.Sources.AzureDevOps 1.3.0

AdaptivePolling

A tiny, dependency-free adaptive polling scheduler for .NET (targets net8.0, net9.0, net10.0).

When you have to poll thousands of resources for changes — repos for new commits, blobs for new versions, endpoints for new data — and push notifications / webhooks aren't an option (security, infra, third-party limits), a flat poll rate is wasteful: most resources rarely change, yet you check them all at the same cadence.

AdaptivePolling learns each resource's change cadence and polls hot resources often and cold ones rarely — and adapts automatically as activity rises and falls.

flat polling:     every resource, every 15 min  ──────────────►  huge, mostly-wasted load
adaptive polling: hot repos  ~minutes
                  warm repos ~hours      self-tunes per resource, bounded by a daily safety net
                  cold repos ~daily

How it works

Two cooperating mechanisms — the same feedback ideas behind TCP's RTT estimator and congestion control:

1. EWMA estimator — an exponentially weighted moving average of the gap between successive changes predicts the expected time to the next change. O(1) per resource, no training, no model.
2. Multiplicative controller — on a hit (change found) the interval ramps down toward the floor (÷ RampUpFactor) so a newly-active resource is polled more often; on a miss (no change) the interval ramps up (× BackoffFactor, exponential backoff). Bursty resources pull themselves to the front of the queue; dormant ones drift to the back.

Why the controller drives the live interval (not the EWMA directly): when you poll slower than a resource changes, the measured gap between observed changes is dominated by your own polling cadence (censoring), which would inflate the interval for exactly the bursty resources you most want to watch. The hit/miss controller is immune to this. The EWMA is still learned and is used to warm-start from real, uncensored history (SeedFromHistory) and for diagnostics.

Everything is clamped to [MinInterval, MaxInterval], so even a long-dormant resource is still checked at least once per MaxInterval (default 24h) — you never miss a change, you just check less often.

Why not ML? Most resources are low-volume and noisy, so a heavy statistical/ML model has little signal and a brutal cold-start. A decaying average plus a feedback controller degrades gracefully and is trivial to operate.

Install

dotnet add package Documentation.AdaptivePolling

Just want the answer? (zero-config)

If all you want is "how often does this repo get commits, and how often should I poll it?", install the batteries-included package and hand it a URL. Provider (GitHub vs Azure DevOps), default branch, authentication, and tuning are all inferred — the only required argument is the URL.

dotnet add package Documentation.AdaptivePolling.Repositories

using Documentation.AdaptivePolling.Repositories;

var p = await CommitFrequency.PredictAsync("https://github.com/dotnet/runtime");

Console.WriteLine($"~{p.ExpectedCommitsPerDay:0.##}/day; poll every {p.RecommendedPollInterval:g}");

Everything beyond the URL is optional:

var p = await CommitFrequency.PredictAsync(
    "https://dev.azure.com/org/project/_git/repo",
    token: pat,            // optional; else GITHUB_TOKEN / AZURE_DEVOPS_PAT env var, else anonymous
    branch: "release",     // optional; else the repo's default branch is resolved automatically
    path: "docs/");        // optional path filter

Need to reuse the wiring (e.g. for the live monitor) without naming a provider? CommitFrequency.CreateSource() returns one ICommitSource that handles both GitHub and Azure DevOps.

Managed identity / Azure AD

For Azure DevOps with a managed identity (or any Azure.Core TokenCredential), pass an IAccessTokenProvider instead of a string — no PAT required. Add Documentation.AdaptivePolling.Auth.AzureAd and:

using Azure.Identity;
using Documentation.AdaptivePolling.Auth.AzureAd;
using Documentation.AdaptivePolling.Repositories;

var credential = new ManagedIdentityCredential(clientId); // or DefaultAzureCredential, etc.
var auth = AzureAdTokenProvider.ForAzureDevOps(credential);

var p = await CommitFrequency.PredictAsync(
    "https://dev.azure.com/org/project/_git/repo",
    auth);   // branch/path still optional

Bring your own auth (token delegate)

Don't want to take the Auth.AzureAd dependency, or need to choose the credential per repository (e.g. a different PAT/identity per Azure DevOps org)? Pass a delegate that returns an AccessToken — AccessToken.FromAzureDevOpsPat(pat) for a PAT (Basic) or AccessToken.Bearer(jwt) for an Azure AD / managed-identity token. The delegate is handed the repository Uri, so you can mint the right credential for each repo:

var p = await CommitFrequency.PredictAsync(
    "https://dev.azure.com/org/project/_git/repo",
    async (repoUri, ct) => usePat
        ? AccessToken.FromAzureDevOpsPat(await GetPatFor(repoUri, ct))
        : AccessToken.Bearer(await GetAadTokenFor(repoUri, ct)));

Quick start

Use the keyed scheduler with your timer "heartbeat":

using Documentation.AdaptivePolling;

var poller = new AdaptivePoller(new PollOptions
{
    MinInterval = TimeSpan.FromMinutes(15),
    MaxInterval = TimeSpan.FromHours(24),
});
var scheduler = new AdaptivePollScheduler<string>(poller);

// Track resources (cold-start, or warm-start from known history).
scheduler.EnsureTracked("repoA", DateTime.UtcNow);
scheduler.EnsureTrackedFromHistory("repoB", pastCommitTimesUtc, DateTime.UtcNow);

// On every timer tick (e.g. every 15 min):
var now = DateTime.UtcNow;
foreach (var key in scheduler.GetDue(now))
{
    var latest = await CheckForChangeAsync(key);          // your ADO/GitHub call - only for DUE resources
    var changed = latest.Sha != lastKnownSha[key];
    scheduler.Report(key, changed, latest.CommitTimeUtc, now);
}

Persisting state across restarts

State is in memory; snapshot it to any store and reload on startup:

IReadOnlyDictionary<string, PollState> snapshot = scheduler.Snapshot();
// serialize snapshot -> blob/table/Cosmos/SQL ...

// on startup:
scheduler.Load(loadedSnapshot);

Stateless core (bring your own store)

If you already persist per-resource state, skip the scheduler and use IAdaptivePoller directly. PollState is a plain serializable POCO:

if (poller.IsDue(state, now))
{
    var (sha, commitTime, changed) = await CheckAsync(...);
    state = poller.Observe(state, changed, commitTime, now);   // returns updated state
    await SaveAsync(id, state);
}

Tuning (PollOptions)

Watching Git repositories (batteries included)

The core above is storage- and source-agnostic. On top of it ships a small, decoupled component that pulls commit history for a branch + path from GitHub or Azure DevOps and turns it into the headline output: how often a new commit is expected.

Package layout

All packages multi-target net8.0, net9.0, and net10.0.

Pick only what you need. The two source packages and the two auth packages are independent.

1. One-shot prediction (stateless)

Ask "how often does this branch get a commit?" — pull history, compute, done. Nothing persisted. The wiring below is the explicit route; for the zero-config one-liner see Just want the answer? above.

using Documentation.AdaptivePolling.Sources;
using Documentation.AdaptivePolling.Sources.GitHub;

var source = new GitHubCommitSource(TokenProviders.GitHubToken(pat)); // or Anonymous() for public
var predictor = new CommitFrequencyPredictor(source);

var target = new RepoTarget("https://github.com/dotnet/runtime", "main", path: "src/libraries");
CommitFrequencyPrediction p = await predictor.PredictAsync(target);

Console.WriteLine($"~{p.ExpectedCommitsPerDay:0.##} commits/day");
Console.WriteLine($"expected every {p.ExpectedInterval:g}");
Console.WriteLine($"recommended poll interval {p.RecommendedPollInterval:g}");
Console.WriteLine($"next commit ~{p.NextExpectedCommitUtc:u} (confidence {p.Confidence:0.00}, n={p.SampleSize})");

Already have the commit timestamps from your own store? Skip the source entirely and call the static helper — no ICommitSource to construct:

CommitFrequencyPrediction p = CommitFrequencyPredictor.FromHistory(
    target,
    history,        // IReadOnlyList<CommitInfo>
    DateTimeOffset.UtcNow,
    options);       // optional PollOptions

Only have bare timestamps (no SHAs)? There's an overload that takes them directly, so you don't have to fabricate CommitInfo instances:

CommitFrequencyPrediction p = CommitFrequencyPredictor.FromHistory(
    target,
    commitTimestamps,   // IEnumerable<DateTimeOffset>
    DateTimeOffset.UtcNow,
    options);

2. Live adaptive monitor (stateful)

RepoChangeMonitor watches a mixed fleet, checks only repos that are due, detects new commits by SHA, feeds the hit/miss outcome into the controller, and refreshes each prediction.

using Documentation.AdaptivePolling.Monitor;
using Documentation.AdaptivePolling.Sources;
using Documentation.AdaptivePolling.Sources.AzureDevOps;
using Documentation.AdaptivePolling.Sources.GitHub;

var source = new CompositeCommitSource(
    new GitHubCommitSource(TokenProviders.GitHubToken(ghPat)),
    new AzureDevOpsCommitSource(TokenProviders.AzureDevOpsPat(adoPat)));

var monitor = new RepoChangeMonitor(source);
monitor.PredictionUpdated += (_, e) =>
    Console.WriteLine($"{e.Target}: new commit, now ~{e.Prediction.ExpectedCommitsPerDay:0.##}/day");

// Warm-start from real history so cadence is right from the first tick:
await monitor.TrackWithHistoryAsync(new RepoTarget("https://github.com/org/repo", "main"));
monitor.Track(new RepoTarget("https://dev.azure.com/org/proj/_git/repo", "main")); // or cold-start

// On each heartbeat (your timer), poll only what's due:
foreach (var result in await monitor.PollDueAsync())
{
    if (result.Changed) { /* enqueue work for result.Target */ }
}

State survives restarts via monitor.Snapshot() / monitor.Load(...).

3. Dependency injection + hosted heartbeat

services.AddGitHubSource(TokenProviders.GitHubToken(ghPat));
services.AddAzureDevOpsSource(TokenProviders.AzureDevOpsPat(adoPat));
services.AddAdaptivePollingMonitor(o => o.Heartbeat = TimeSpan.FromMinutes(1));

This registers a singleton RepoChangeMonitor (composing every registered ICommitSource) and a BackgroundService that calls PollDueAsync every heartbeat. Resolve the monitor to Track/TrackWithHistoryAsync your targets and subscribe to PredictionUpdated.

Authentication

Auth is one interface — IAccessTokenProvider — returning an AccessToken { Scheme, Value } fetched per request (so rotation and short-lived tokens just work). GitHub and Azure AD use Bearer; Azure DevOps PATs use Basic.

Azure AD / managed identity (e.g. for Azure DevOps without a PAT):

using Azure.Identity;
using Documentation.AdaptivePolling.Auth.AzureAd;

var provider = AzureAdTokenProvider.ForAzureDevOps(new DefaultAzureCredential());
var source = new AzureDevOpsCommitSource(provider);

GitHub App (mints and caches installation tokens, BCL-only):

using Documentation.AdaptivePolling.Auth.GitHubApp;

var provider = new GitHubAppTokenProvider(new GitHubAppOptions
{
    AppId = "123456",
    InstallationId = 7891011,
    PrivateKeyPem = File.ReadAllText("app.private-key.pem"),
});
var source = new GitHubCommitSource(provider);

CommitFrequencyPrediction

Design notes

• Pure & testable. Pass nowUtc explicitly; inject an IJitterSource (NoJitterSource for deterministic tests). The estimation math lives in the standalone Ewma helper.
• Thread-safe scheduler. AdaptivePollScheduler<TKey> guards its state with a lock and hands out deep copies.
• No dependencies. netstandard2.0, nothing but the BCL.

License

MIT — see LICENSE.