Elastic.Serilog.Sinks 9.0.0

Elastic.Serilog.Sinks

A Serilog sink that writes logs directly to Elasticsearch or Elastic Cloud using the Elastic Common Schema.

NOTE: To use version 9.0.0 of the sink you need atleast version 8.15.0 of the Elastic Stack.

Example

There's a few ways that you can extend a Serilog LoggerConfiguration:

Log.Logger = new LoggerConfiguration()
	.MinimumLevel.Debug()
	.Enrich.FromLogContext()

NOTE: Don't forget we also publish an Elastic.Apm.SerilogEnricher for the Elastic APM Agent!

Writing to Elasticsearch

.WriteTo.Elasticsearch(new [] { new Uri("http://localhost:9200" )}, opts =>
{
	opts.DataStream = new DataStreamName("logs", "console-example", "demo");
	opts.BootstrapMethod = BootstrapMethod.Failure;
	opts.ConfigureChannel = channelOpts =>
	{
		channelOpts.BufferOptions = new BufferOptions 
		{ 
			ConcurrentConsumers = 10 
		};
	};
})

Writing to Elastic Cloud:

.WriteTo.ElasticCloud("cloudId", "cloudUser", "cloudPass", opts =>

opts is an instance of ElasticsearchSinkOptions with the following options

Note that you can also pass ElasticsearchSinkOptions directly

.WriteTo.Elasticsearch(new ElasticsearchSinkOptions(client.Transport)

This allows you to reuse the Transport used by the Elasticsearch Client for instance.

ECS Aware Message Templates

This sink by proxy of its formatter allows you to set ECS fields directly from the message template using properties that adhere to the https://messagetemplates.org/ format.

The available ECS message template properties are listed under LogTemplateProperties.* e.g LogTemplateProperties.TraceId

Log.Information("The time is {TraceId}", "my-trace-id");

Will override trace.id on the resulting ECS json document.

Application Settings Configuration

This sink can be configured through appsettings.json when used in combination with Serilog.Settings.Configuration.

Elasticsearch appsettings configuration

When configuring through appsettings only the bootstrapMethod configuration is required

{
  "Serilog": {
    "Using": [ "Elastic.Serilog.Sinks" ],
    "MinimumLevel": { "Default": "Information" },
    "WriteTo": [
      {
        "Name": "Elasticsearch",
        "Args": {
          "bootstrapMethod": "Silent",
          "nodes": [ "http://elastichost:9200" ],
          "useSniffing": true,
          "apiKey": "<apiKey>",
          "username": "<username>",
          "password": "<password>",

          "ilmPolicy" : "my-policy",
          "dataStream" : "logs-dotnet-default",
          "includeHost" : true,
          "includeUser" : true,
          "includeProcess" : true,
          "includeActivity" : true,
          "filterProperties" : [ "prop1", "prop2" ],
          "proxy": "http://localhost:8200",
          "proxyUsername": "x",
          "proxyPassword": "y",
          "debugMode": false,

          //EXPERT settings, do not set unless you need to 
          "maxRetries": 3,
          "maxConcurrency": 20,
          "maxInflight": 100000,
          "maxExportSize": 1000,
          "maxLifeTime": "00:00:05",
          "fullMode": "Wait"
        }
      }
    ]
  }
}

Elastic Cloud appsettings configuration

When configuring through appsettings only the bootstrapMethod configuration is required

You can specify either endpoint or cloudId, cloudId will take precedence.

You'll need to specify either apiKey or username and password.

{
  "Serilog": {
    "Using": [ "Elastic.Serilog.Sinks" ],
    "MinimumLevel": { "Default": "Information" },
    "WriteTo": [
      {
        "Name": "ElasticCloud",
        "Args": {
          "bootstrapMethod": "Silent",
          "endpoint": "https://<redacted>.es.us-central1.gcp.cloud.es.io",
          "cloudId": "<cloudId>",
          "apiKey": "<apiKey>",
          "username": "<username>",
          "password": "<password>",
          
          "ilmPolicy" : "my-policy",
          "dataStream" : "logs-dotnet-default",
          "includeHost" : true,
          "includeUser" : true,
          "includeProcess" : true,
          "includeActivity" : true,
          "filterProperties" : [ "prop1", "prop2" ],
          "proxy": "http://localhost:8200",
          "proxyUsername": "x",
          "proxyPassword": "y",
          "debugMode": false,

          //EXPERT settings, do not set unless you need to 
          "maxRetries": 3,
          "maxConcurrency": 20,
          "maxInflight": 100000,
          "maxExportSize": 1000,
          "maxLifeTime": "00:00:05",
          "fullMode": "Wait"
        }
      }
    ]
  }
}

Comparison with Serilog.Sinks.Elasticsearch

• Serilog.Sinks.Elasticsearch is an amazing community led sink that has a ton of options and works against older Elasticsearch versions < 8.0.
• Serilog.Sinks.Elasticsearch is unofficially supported by Elastic with some of the .NET team helping to maintain it.
• Elastic.Serilog.Sinks is officially supported by Elastic and was purposely build to adhere to newer best practices around logging, datastreams and ILM.
• Elastic.Serilog.Sinks is purposely build to have fewer configuration options and be more prescriptive than Serilog.Sinks.Elasticsearch.
  • That is not to say there aren't plenty of configuration hooks in Elastic.Serilog.Sinks

Notable absent features:

• Elastic.Serilog.Sinks only works with Elasticsearch 8.x and up.
  • This is because the bootrapping (BootstrapMethod) attempts to load templates build for Elasticsearch 8.0 and up.
• Elastic.Serilog.Sinks has only one way it emits data to Elasticsearch confirming to the ecs-logging specification
  • That doesn't mean you can not introduce your own additional properties though.
• Elastic.Serilog.Sinks has no durable mode.
  • If you need higher guarantees on log delivery use Serilog.Sinks.File with our ECS log formatter for Serilog and use filebeat to ship these logs.
  • Check out Elastic Agent & Fleet to simplify collecting logs and metrics on the edge.

If you miss a particular feature from Serilog.Sinks.Elasticsearch in Elastic.Serilog.Sinks please open a feature request! We'd love to grow this sink organically moving forward.