AspNetCoreSharedServer 4.1.1

AspNetCoreSharedServer

An ASP.NET Core server for Linux, macOS & Windows that starts Kestrel on demand and stops it when idle. For use when hosting ASP.NET Core in a shared hosting environment behind a proxy like Apache or Nginx, or as an alternative to IIS when serving net.tcp endpoints, where Kestrel should not run as a permanent system service using resources, but should be started on demand and shutdown when idle.

Usage

First install AspNetCoreSharedServer using the comman sudo dotnet tool install AspNetCoreSharedServer --global. Next, execute sudo aspnet-server install to install the server as a service. You might need to add the /root/.dontet/tools path to your PATH environment variable for root first for this to work. This works on Linux using Systemd or OpenRC (like on Alpine) and on macOS. You can then manage the service with your system's service commands and the name of the service aspnet-server. To uninstall the system service run sudo aspnet-server uninstall. To show help on usage, execute sudo aspnet-server -? or sudo aspnet-server -help.

Configure AspNetCoreSharedServer

All the configuration is stored in /etc/aspnet-server/configuration.json, or in %ProgramData%\aspnet-server\configuration.json on Windows, and in json files named after the application name in /etc/aspnet-server or %ProgramData%\aspnet-server. When the configuration is changed, changes are applied on the fly. The configuration.json file is defined as follows:

{
  "IdleTimeout": 300,
  "Recycle": 1200,
  "Offline": false,
  "User": "www-data",
  "Group": "www-data"
  "FailureLimit": 5,
  "FailureInterval": "00:05:00",
  "MemoryLowThreshold": 0.95,
  "IdleTimeoutOnLowMemory": "00:01:00"
  "Syslog": {
    "Host": "localhost",
    "Port": 514,
    "Protocol": "Udp"
  },
  "Command": "None",

  "Applications": [
    {
      "Name": "MyApp",
      "Assembly": "/Path/To/ASPNETCoreApp.dll",
      "WorkingDirectory": "/Path/To",
      "Arguments": "",
      "Urls": "http://original-domain.org",
      "ListenUrls": "http://localhost:10000",
      "Environment": {
        "ASPNETCORE_ENVIRONMENT": "Production"
      }
      "IdleTimeout": 300,
      "Recycle": 1200,
      "Offline": false,
      "User": "www-data",
      "Group": "www-data",
      "Status": "Running"
    }, 
    ... more applications can be defined here ...
  ]
}

• Name specifies the name of the application. It must be unique.
• Assembly denotes the path to the ASP.NET Core application DLL.
• WorkingDirectory an optional working directory for the Kestrel process. If not specified, the path of Assembly will be used.
• Arguments are optional command line arguments passed to the application.
• Urls is the original URL that the application will respond to. This are the original urls the main server (Apache or NGINX) is listening on. This value will be passed to the Kestrel process in the environment variable ORIGINAL_URLS.
• ListenUrls are the URLs that this Application will listen on. This is where the proxy server (Apache or NGINX) can forward requests to. You can omit the port in this urls and AspNetCoreSharedServer will automatically assign a port.
• IdleTimeout is the time in seconds or JSON time value after which an application will be stopped when it is idle.
• Recycle is the time in seconds or JSON time value after which the application will be restarted, regardless of activity.
• Environment is a dictionary of environment variables that will be set for the application when it is started. The server also sets ASPNET_SERVER to its own version (the one aspnet-server -v prints, for example 4.0.2), so that an application can tell that it runs behind AspNetCoreSharedServer and which version. It cannot be overridden here.
• Offline if set to true, the application or the server will be disabled and set offline.
• Syslog optional Syslog configuration.
• FailureLimit number of failures within FailureInterval in order to disable app pool.
• FailureInterval time span within to count the number of failures.
• MemoryLowThreshold a double value between 0 and 1 indicating the percentage of occupied memory when IdleTimeout should switch to the IdleTimeoutOnLowMemory value. So you can lower IdleTimeout when memory is low, so the server uses fewer RAM.
• EnableProxyHeader and EnableProxyV2Header (per application, default off): The server will send a PROXY V1/V2 header to Kestrel, so that the application can see the real address of the client. Note that Kestrel does not support this header natively, and you must use a middleware like Scintillating.ProxyProtocol.Middleware; the NuGet package AspNetCoreSharedServer.Middleware turns it on for you when you call builder.WebHost.UseAspNetCoreSharedServer() (see PROXY protocol). That is an environment variable the server sets for the application to v1 or v2, so that it knows that every TCP connection starts with that header and which version it is (v2 if both options are set). The variable is not set when no header is sent.
• EnableHttp3 (global or per application, default true) forwards HTTP/3 (QUIC over UDP) for applications that have an https URL in ListenUrls. See HTTP/3. It is switched off automatically for an application that has EnableProxyHeader or EnableProxyV2Header set.

The individual applications can also be placed in separate json files, named after the applications name like so:

{
  "Name": "MyApp",
  "Assembly": "/Path/To/ASPNETCoreApp.dll",
  "WorkingDirectory": "/Path/To",
  "Arguments": "",
  "Urls": "http://original-domain.org",
  "ListenUrls": "http://localhost:10000",
  "Environment": {
    "ASPNETCORE_ENVIRONMENT": "Production"
  }
  "IdleTimeout": 300,
  "Recycle": 1200,
  "Offline": false,
  "User": "www-data",
  "Group": "www-data",
  "Status": "Running"
} 

This would be placed in /etc/aspnet-server/MyApp.json. Note that aspnet-server install sets the permissions of /etc/aspnet-server to user only access and the owner & group of /etc/aspnet-server to root & www-data. That means that only root can create applications. If you want all users of group www-data to be able to create applications, change access to /etc/aspnet-server to user & group.

After you have defined your applications in the configuration.json file, you can proxy to the sockets specified in ListenUrls from Apache or Nginx. The original Urls that Apache or Nginx serve will be passed to Kestrel as a environment varibale ORIGINAL_URLS from the Urls parameter in configuration.json.

HTTP/3

For an application with an https URL in ListenUrls, AspNetCoreSharedServer also listens on the same port for UDP and forwards HTTP/3 (QUIC) to the application. The application has to do two things:

1. Kestrel must listen on the HTTPS port it is given in ASPNETCORE_URLS with HttpProtocols.Http1AndHttp2AndHttp3.

2. The application must tell clients where to find HTTP/3. Kestrel only knows its own internal port, so on its own it would advertise a port that clients cannot reach. AspNetCoreSharedServer therefore passes the correct Alt-Svc header value, for example h3=":443"; ma=86400, in the environment variable HTTP3_ALT_SVC_HEADER. The NuGet package AspNetCoreSharedServer.Middleware sends it for you with one line:

   app.UseAspNetCoreSharedServer();
   

   Without the package, send it yourself with every response:

   var altSvc = Environment.GetEnvironmentVariable("HTTP3_ALT_SVC_HEADER");
   if (!string.IsNullOrEmpty(altSvc))
   {
       app.Use(async (context, next) =>
       {
           context.Response.Headers["Alt-Svc"] = altSvc;
           await next();
       });
   }
   

   HTTP3_ALT_SVC_HEADER is empty, and HTTP3_ENABLED is false, when HTTP/3 is not forwarded for the application.

Things to know:

• A UDP datagram never starts an application. Without a running application HTTP/3 requests are dropped, and clients fall back to TCP, which starts it.
• HTTP/3 traffic counts as activity for IdleTimeout.
• The PROXY protocol only exists for TCP. Over HTTP/3 the application would see 127.0.0.1 as the client, so HTTP/3 is switched off for applications that have EnableProxyHeader or EnableProxyV2Header set.
• Every HTTP/3 client gets its own session, which ends after 60 seconds without traffic. At most 4096 sessions are kept, and a new session only starts with a datagram of at least 1200 bytes, the size of a QUIC client's first packet.
• Set "EnableHttp3": false in configuration.json, or for a single application, to switch HTTP/3 off.

PROXY protocol

With EnableProxyHeader or EnableProxyV2Header, AspNetCoreSharedServer starts every TCP connection to the application with a PROXY header that carries the real address of the client. Kestrel does not read that header by itself, so the application has to be told to. Add the NuGet package AspNetCoreSharedServer.Middleware and call UseAspNetCoreSharedServer on the web host, before the Kestrel endpoints are created:

var builder = WebApplication.CreateBuilder(args);

builder.WebHost.UseAspNetCoreSharedServer();   // reads the PROXY header; must come before UseKestrel / ConfigureKestrel
builder.WebHost.UseKestrel(options => { ... });

var app = builder.Build();
app.UseAspNetCoreSharedServer();               // sends the Alt-Svc header, see HTTP/3 above

• Without this call Kestrel does not understand the header: the application then sees the connection as invalid, and HttpContext.Connection.RemoteIpAddress is not the client.
• The call does nothing unless the server sets the environment variable PROXY_PROTOCOL (v1 or v2), so it is safe to leave in when the application also runs without AspNetCoreSharedServer or without a PROXY header.
• It turns on Scintillating.ProxyProtocol.Middleware (a dependency of the package, .NET 8 or newer) for every Kestrel endpoint, and it also works together with UseHttps.
• When PROXY_PROTOCOL is set, every connection must carry the header. That is always the case behind AspNetCoreSharedServer, but it means the same port cannot also serve direct connections.
• HTTP/3 is switched off for an application that gets a PROXY header, because the PROXY protocol only exists for TCP.

API Library

Windows: one local user per app

On Windows an app that has a User (per application or in configuration.json) runs as that local user, the counterpart of the sudo -u launch on Linux and macOS. The server logs the user on with S4U (LsaLogonUser, "service for user"), which needs no password, and starts the app with that token. The app is put in a job object, so it ends with the server.

• The server has to run as LocalSystem, or as an account that holds the privileges Act as part of the operating system (SeTcbPrivilege), Replace a process level token and Adjust memory quotas for a process. Such an account can become any local user, so it is as powerful as LocalSystem. Without them the app does not start, and the status says which privileges are missing.
• The user must exist. The server never creates accounts by itself; run aspnet-server create-user <name> as administrator. The account gets a random password that nobody knows, the right to log on as a batch job (what S4U needs) and the denial of every interactive, remote desktop and network logon, so nobody can log on as it.
• Windows has no primary group to switch to, so Group only decides who may run the app: the user has to be a member of it (directly or through another group), otherwise the app does not start.
• The token of an S4U logon has no network credentials: the app cannot reach network shares as that user.
• Give each app's folder to its own user (and read access to the server's account); that is what keeps the apps apart.
• The app's own json file (/etc/aspnet-server/<Name>.json on Linux, <Name>.json next to configuration.json on Windows) may only be changed by the app's User and by administrators. On Linux and macOS that is "owned by the user or root, and not writable by group or other"; on Windows it is read from the file's permissions: the owner must be the user, an administrator or SYSTEM, and no other account or group may have a right to write, append, delete or change the permissions of the file (read access for others is fine). An app whose file does not meet that is taken offline with an error, and the server sets the permissions itself when it saves the file: no inherited rules, full control for SYSTEM and administrators, read and write for the user. A file that the AspNetCoreSharedServer.Api created is checked the first time the server loads it.
• The same goes for configuration.json and its folder, which on Linux and macOS belong to root. On Windows the folder gets full control for SYSTEM and administrators only (plus modify rights for a local group aspnet-server, if it exists, for the people who may create applications), and configuration.json is owned by the administrators and can only be changed by SYSTEM and administrators. The server sets this when it saves, if it runs as an administrator or SYSTEM (anybody else would lock themselves out), and when it loads a configuration.json that someone else can change it refuses it and writes the error into it, like on Linux. AspNetCoreSharedServer.Api does the same on Windows when it saves.
• aspnet-server install creates the group aspnet-server if it does not exist, every time it runs: on Linux and macOS with groupadd or dscl, on Windows as a local group. The server gives its configuration folder to that group, which fails without it. The group starts empty: add the people who may create applications to it. Uninstalling leaves it, because it holds people.
• The machine wide locks (the lock of the configuration and the lock that lets only one server run) are files inside the configuration folder (*.lock), not in the folder that AsyncMutexLock would use for every program on the machine. A lock is held by whoever can open its file, so a lock file that every user can open is one that every user can hold, and anybody could stop the server from starting or from saving its configuration. The configuration folder belongs to root (SYSTEM, the administrators) and the group aspnet-server, so exactly the people who may use the configuration can use the locks; anybody else is told that they have no permission, and not that another instance is running. On Linux and macOS the lock files are mode 660 for that group. A server started with --config-dir has its locks in that folder, so it does not compete with the real one, and a member of the group (or root) is all it takes to run it.
• The server reads configuration.json and the app files through one open file and checks the owner and permissions of that file, not of its path, so nobody who can write to the folder can swap a checked file for another one between the check and the read. On Linux and macOS the file is opened with O_NOFOLLOW, on Windows it is locked against writing, renaming and deleting while it is read. A symbolic link is never trusted: a link as configuration.json is not loaded, and an app file that is a link is skipped. When the server saves, a link at the path is removed and never written through, and on Linux and macOS the content goes to a new file (created with O_EXCL | O_NOFOLLOW, its owner set before anything is written) that is renamed over the old one, so neither a symbolic link nor a hard link at the path can send the write into another file. A file that is not secure is written back with the owner it has, so saving never makes it trusted.
• The configuration lives in %ProgramData%\aspnet-server on Windows and in /etc/aspnet-server on Linux and macOS. The command line option --config-dir <folder> (or --config-dir=<folder>) names another folder on every platform, for the server and for the commands (aspnet-server stop MyApp --config-dir <folder>); the tests use it, so that they never touch the real one. The server protects and checks that folder like the default one. It is deliberately not an environment variable: the environment of a process that runs as root or SYSTEM is handed down by whatever started it, a command line is what the person who starts the server writes. A service never has the option, so it always uses the fixed folder. The old applications.json import (from /etc/aspnetcore) never happens when the old file is in the folder of the current configuration, because the import deletes the old folder.
• A user of a domain is not supported, only local users (name, .\name or MACHINE\name).
• When User is the account that the server runs as, or empty, the app is started directly, as before.

You can also manage the configuration.json file programmatically by using AspNetCoreSharedServer.Api: Import AspNetCoreSharedServer.Api via nuget to your project.

using AspNetCoreSharedServer;
...
var app = new Application()
{
  Name = "Name of the ASP.NET Core application",
  Assembly = "/Path/To/StartupAssembly.dll",
  WorkingDirectory = "Path/To",
  Arguments = "Startup Arguments",
  Environment = new Dictionary<string, string>()
  {
    { "ASPNETCORE_ENVIRONMENT", "Production" }
  },
  ListenUrls = "http://localhost:10000",
  Urls = "http://original-domain.org",
};
AspServer.Configuration.Update(app);
or
AspServer.Configuration.Add(app);
or
AspServer.Configuration.Remove(app);
or
AspServer.Configuration.Remove("App Name");
or 
await AspServer.Configuration.UpdateAsync(app);
or
await AspServer.Configuration.AddAsync(app);
or
await AspServer.Configuration.RemoveAsync(app);
or
await AspServer.Configuration.RemoveAsync("App Name");

Or to lookup an application:

using AspNetCoreSharedServer;
...
AspServer.Configuration.Load();
or
await AspServer.Configuration.LoadAsync();

var app = AspServer.Configuration.Applications["Name of Application"];
...

To find a free IP port:

var port = AspServer.FindFreePort();
or
var port = await AspServer.FindFreePortAsync();

If you do non atomic stuff with AspServer.Configuration, you must enclose it in a mutex lock, so always only one process can modify configuration.json:

using (var mutex = AspServer.Lock()) {
    AspServer.Configuration.Load();

    code that manipulates AspServer.Configuration ...
    
    AspServer.Configuration.Save();
}

or

using (var mutex = await AspServer.LockAsync()) {
    await AspServer.Configuration.LoadAsync();

    code that manipulates AspServer.Configuration ...
    
    await AspServer.Configuration.SaveAsync();
}