Skip to main content
Version: 8

Logging Configuration

Ed-Fi API v8 uses Serilog for structured logging in both the Ed-Fi API and the Configuration Service. Logs are written to the console in structured format, suitable for ingestion by log-monitoring platforms such as Splunk, CloudWatch, Datadog, or the ELK stack.

The log level for each service is controlled independently:

  • Ed-Fi API: LOG_LEVEL environment variable (maps to Serilog__MinimumLevel__Default in appsettings.json)
  • Configuration Service: CONFIGURATION_SERVICE_LOG_LEVEL environment variable

Log Levels​

LevelWhen usedAction required
FATALApplication cannot start or must shut downInvestigate immediately. Check for missing required configuration or resource exhaustion. Report bugs through the Ed-Fi Community Hub
ERRORUnexpected error that interrupts service, or an external failure (e.g., database unavailable after retries)Investigate and resolve the external issue. Report application bugs through the Ed-Fi Community Hub
WARNUnexpected condition the system recovered from automatically (e.g., a database retry that succeeded)Monitor frequency. Frequent warnings may indicate an underlying issue worth investigating
INFORMATIONNormal request lifecycle events — method, path, response code, duration, trace IDGenerally no action required
DEBUGDetailed diagnostics, including anonymized request payloads (see PII Protection)Use for integration troubleshooting. Do not leave enabled in production

Default Configuration​

Production (INFORMATION)​

The recommended log level for production deployments. Logs request lifecycle events — method, path, response code, duration, and trace ID — without including request or response body content.

LOG_LEVEL=Information
CONFIGURATION_SERVICE_LOG_LEVEL=Information

Both services write structured JSON to the console — one JSON object per log entry. Sample output at INFORMATION level (formatted for readability):

{
"Timestamp": "2026-06-29T14:23:45.123Z",
"Level": "Information",
"MessageTemplate": "{EventName}: DMS request completed: {Method} {Path} responded {StatusCode} in {DurationMs} ms with TraceId {TraceId}",
"RenderedMessage": "HttpRequestCompleted: DMS request completed: POST /data/ed-fi/schools responded 201 in 18 ms with TraceId 0HN...",
"Properties": {
"Application": "EdFi.DataManagementService",
"EventName": "HttpRequestCompleted",
"EventId": { "Id": 1228001, "Name": "HttpRequestCompleted" },
"SourceContext": "EdFi.DataManagementService.Frontend.AspNetCore.Infrastructure.LoggingMiddleware",
"RequestLayer": "Frontend",
"TraceId": "0HN...",
"ActivityTraceId": "4bf92f3577b34da6a3ce929d0e0e4736",
"SpanId": "00f067aa0ba902b7",
"Method": "POST",
"Path": "/data/ed-fi/schools",
"StatusCode": 201,
"DurationMs": 18,
"PathBase": ""
}
}

ActivityTraceId and SpanId are present only when an ASP.NET Activity is active (e.g., when OpenTelemetry tracing is enabled). Failure events use EventId 1228002 and Level Error. The file sink writes the same structured properties but omits RenderedMessage.

warning

Do not set Serilog:WriteTo:*:Args:outputTemplate in a local appsettings.Development.json override. Serilog's configuration array merging causes outputTemplate to silently replace the JSON formatter with plain text, breaking log collection. Override MinimumLevel only.

Local Development (DEBUG)​

The environment files that ship with the Docker Compose setup start both services at Information. To troubleshoot an integration locally, set the log level to Debug in the .env file you start the stack with. At this level, the Ed-Fi API logs anonymized request payloads. See PII Protection for how anonymization works.

LOG_LEVEL=Debug
CONFIGURATION_SERVICE_LOG_LEVEL=Debug

Production Troubleshooting​

When diagnosing a difficult-to-reproduce issue in production, temporarily raise the log level to DEBUG. Return to INFORMATION as soon as the issue is captured — DEBUG logging generates significantly more output and should not be left on indefinitely.

warning

DEBUG logging includes anonymized request bodies. Although sensitive field values are replaced with "*", ensure your log storage and access policies are appropriate before enabling DEBUG in a production environment.

Container Log Retention​

The Docker Compose setup limits how much log output Docker keeps for each container, so logs can't fill the host's disk:

VariableDefaultDescription
DOCKER_LOG_MAX_SIZE50mMaximum size of one log file before Docker starts a new one
DOCKER_LOG_MAX_FILE5Maximum number of log files Docker keeps for each container

With the defaults, each container keeps up to 250 MB of logs, and the oldest entries are discarded as new ones arrive. Under heavy load at Debug level, that can amount to only a few seconds of history.

For a longer investigation, raise these values temporarily and restore the defaults afterward, or export logs with OpenTelemetry so they're kept outside the container.

caution

Docker rejects "unlimited" values such as -1 or 0. A very large value, such as DOCKER_LOG_MAX_SIZE=1000g, effectively removes the limit and can let a log file fill the host's disk, which affects every container on the host.

PII Protection at DEBUG Level​

At DEBUG level, the Ed-Fi API logs anonymized HTTP request payloads. Before logging, every scalar value in the request body (strings, numbers, booleans, null) is replaced with "*", leaving only the JSON structure and property names visible. This provides enough information to diagnose request shape issues (missing fields, incorrect nesting, wrong property names) without exposing student or staff data.

Example: a POST /api/data/ed-fi/students request body is logged as:

{
"studentUniqueId": "*",
"birthDate": "*",
"firstName": "*",
"lastSurname": "*",
"sexDescriptor": "*"
}

This masking behavior is the default (MaskRequestBodyInLogs=true). To log the full unmasked request body — for example, when debugging a parsing issue in a controlled environment — set MaskRequestBodyInLogs=false:

MASK_REQUEST_BODY_IN_LOGS=false

Correlation IDs​

Every HTTP request is assigned a correlation ID that appears in request log entries and in error response bodies that carry a correlationId or traceId field. The correlation ID makes it possible to trace a specific failed request across multiple log lines and across service boundaries.

By default the Ed-Fi API ignores client-supplied correlation IDs and uses the request's trace identifier. To accept one, set AppSettings:CorrelationIdHeader (CORRELATION_ID_HEADER in Docker Compose) to the header name:

CORRELATION_ID_HEADER=correlationid
info

Requests that result in an HTTP 5xx response are logged at ERROR level. Requests that result in an HTTP 4xx response (including validation errors and not-found responses) are logged at Information level.

Example 1: Error response body containing a Correlation ID​

{
"detail": "Data validation failed. See 'validationErrors' for details.",
"type": "urn:ed-fi:api:bad-request:data-validation-failed",
"title": "Data Validation Failed",
"status": 400,
"correlationId": "b9567d36-91c1-4bae-aff5-3fde8367b969",
"validationErrors": {
"$.gradeLevels": ["gradeLevels is required."]
},
"errors": []
}

Example 2: Log entry for the same request​

The log entry for the request in Example 1 contains the same value as TraceId, linking the structured log event to the error response body:

{
"Timestamp": "2026-06-29T14:23:45.123Z",
"Level": "Information",
"MessageTemplate": "{EventName}: DMS request completed: {Method} {Path} responded {StatusCode} in {DurationMs} ms with TraceId {TraceId}",
"RenderedMessage": "HttpRequestCompleted: DMS request completed: POST /data/ed-fi/schools responded 400 in 12 ms with TraceId b9567d36-91c1-4bae-aff5-3fde8367b969",
"Properties": {
"Application": "EdFi.DataManagementService",
"EventName": "HttpRequestCompleted",
"EventId": { "Id": 1228001, "Name": "HttpRequestCompleted" },
"TraceId": "b9567d36-91c1-4bae-aff5-3fde8367b969",
"Method": "POST",
"Path": "/data/ed-fi/schools",
"StatusCode": 400,
"DurationMs": 12
}
}

Specifying the Correlation ID for a Request​

Clients send their correlation ID in the header named by AppSettings:CorrelationIdHeader (Docker Compose env var: CORRELATION_ID_HEADER). If the setting is empty or a request doesn't send the header, the Ed-Fi API uses the request's trace identifier instead.

Using a client-supplied correlation ID is particularly useful when the API sits behind a gateway or proxy that logs its own correlation IDs — matching them allows a single request to be traced end-to-end.

tip

Assign a unique correlation ID per request as early as possible in your client code. A GUID per request is a simple and reliable approach.

Example 3: Correlation ID supplied as an HTTP header​

This example assumes CORRELATION_ID_HEADER=correlationid.

POST /api/data/ed-fi/schools HTTP/1.1
Host: localhost:8080
Content-Type: application/json
Authorization: Bearer <access_token>
correlationid: b9567d36-91c1-4bae-aff5-3fde8367b969

{ ... }

Correlation ID Normalization​

Before the Ed-Fi API uses a correlation ID in a log entry or an error response, it normalizes the value. This applies to client-supplied values and to trace identifiers alike:

  • Values longer than the maximum length are truncated. The maximum is set by CORRELATION_ID_MAX_LENGTH (AppSettings:CorrelationIdMaxLength), which defaults to 255 and accepts 64 through 1024, inclusive. Length is measured in UTF-16 code units; a non-BMP character, such as many emoji, uses two units. If truncation would split a surrogate pair, that character is omitted.
  • After truncation, Unicode control characters (category Cc, including carriage return, line feed, tab, and null), invisible format characters (category Cf), Unicode line and paragraph separators (U+2028 and U+2029), and unpaired UTF-16 surrogates are removed. All other characters, including punctuation, non-ASCII characters, emoji, and spaces, are kept. Filtering after truncation can leave a value shorter than the configured maximum.
  • If the result is empty or contains only whitespace, the request's server-generated trace identifier is used instead.

The Ed-Fi API never rejects a request because of its correlation ID. Because the same normalized value is used everywhere, the correlation ID in an error response always matches the one in the logs.

Use the returned normalized value when searching logs. See the client guide's Correlation IDs page for client examples, token error response headers, and responses without an ID.

When normalization changes a client-supplied correlation ID, the Ed-Fi API logs a CorrelationIdModified event at Information level (see Event IDs). The event reports what changed, but deliberately doesn't log the original value:

PropertyMeaning
SuppliedLength, NormalizedLengthLength of the value before and after normalization
CharactersRemovedtrue if characters were removed
Truncatedtrue if the value was cut to the maximum length
FellBackToServerIdentifiertrue if nothing usable was left, so the request's trace identifier was used instead

FellBackToServerIdentifier is the property to alert on: when it is true, the client's own identifier can't be recovered from the logs. No event is logged when the value is unchanged or when no correlation header is configured.

warning

CORRELATION_ID_MAX_LENGTH must be between 64 and 1024. With any other value, the Ed-Fi API starts but answers every request, including /health, with a 500 error until the setting is fixed and the service is restarted.

note

A client-supplied correlation ID that is truncated or has characters removed no longer matches the identifier recorded by the upstream system that generated it. To avoid this, use correlation IDs of plain printable characters within the maximum length.

Log Collection​

Both services emit the same structured request-log contract. Collector rules should target structured properties — not parse RenderedMessage.

Event IDs​

Use EventId.Id to target request lifecycle events without matching on message strings:

EventId.IdEventNameLevelMeaning
1228001HttpRequestCompletedInformation (Ed-Fi API); Information or Debug (Configuration Service, see note)Request completed — includes all 4xx responses
1228002HttpRequestFailedErrorRequest failed with a 5xx or unhandled exception
1228003CorrelationIdModifiedInformationEd-Fi API only. A client-supplied correlation ID was changed by normalization. This isn't a request event, so rules that filter on 1228001 and 1228002 don't receive it
note

The Configuration Service logs HttpRequestCompleted at Debug level for /.well-known/* paths (OIDC discovery endpoints). An Information-level collector will not receive these events. All other paths follow the Information rule above.

RequestLayer (Ed-Fi API only)​

The Ed-Fi API emits two log events per resource API request: one from the ASP.NET frontend (RequestLayer = "Frontend") and one from the core pipeline (RequestLayer = "Core"). Endpoints that are handled directly without entering the core pipeline — such as the Discovery and Health endpoints — emit only the Frontend event. For request-count dashboards and failure-rate alerts, filter to RequestLayer = "Frontend" to count only externally visible HTTP requests and avoid double-counting.

The Configuration Service does not emit RequestLayer.

Exception Field​

The Exception field is present only when the logging middleware itself observed an exception. It is omitted entirely — not set to null — when a 5xx status is produced without an observable exception (for example, when a downstream pipeline returned a 5xx status code). Collector rules must treat Exception as optional.

HTTP 413 Responses​

Oversized request body rejections (HTTP 413) are emitted as HttpRequestCompleted events, not HttpRequestFailed, because they are handled as client errors. The StatusCode property carries 413.

OpenTelemetry (OTLP) Log Export​

Both the Ed-Fi API and the Configuration Service can also send their logs to an OpenTelemetry (OTLP) endpoint, either a compatible service or an OpenTelemetry Collector that forwards them to a backend such as Splunk, Datadog, Elastic, or CloudWatch. Log export is off by default. When it is on, logs are still written to the console as well.

Each service is configured through its own OtlpLogging section:

SettingEd-Fi API variableConfiguration Service variableDescription
OtlpLogging:EnabledOTLP_LOGGING_ENABLEDCONFIGURATION_SERVICE_OTLP_LOGGING_ENABLEDtrue to export logs. Default: false
OtlpLogging:EndpointOTLP_LOGGING_ENDPOINTCONFIGURATION_SERVICE_OTLP_LOGGING_ENDPOINTAbsolute http:// or https:// URL of the OTLP endpoint, for example http://collector:4318. Required when export is on
OtlpLogging:ProtocolOTLP_LOGGING_PROTOCOLCONFIGURATION_SERVICE_OTLP_LOGGING_PROTOCOLHttpProtobuf (default) or Grpc
OtlpLogging:ServiceNameNot mappedNot mappedThe service.name attribute sent with each log. Default: EdFi.DataManagementService (Ed-Fi API) or EdFi.DmsConfigurationService (Configuration Service). Override to tell instances apart in your log backend
OtlpLogging:DeploymentEnvironmentOTLP_LOGGING_DEPLOYMENT_ENVIRONMENTCONFIGURATION_SERVICE_OTLP_LOGGING_DEPLOYMENT_ENVIRONMENTOptional name of the deployment environment, such as production
OtlpLogging:HeadersNot mappedNot mappedOptional headers sent with every export request, such as Authorization

The variable names are those used by the Docker Compose files. Without those files, set the settings directly, for example OtlpLogging__Enabled=true.

OTLP_LOGGING_ENABLED=true
OTLP_LOGGING_ENDPOINT=http://collector:4318
caution
  • Header values are often secrets. Supply them through environment variables, such as OtlpLogging__Headers__Authorization, or a secret store, never a committed configuration file.
  • The standard OTEL_EXPORTER_OTLP_* environment variables are ignored. Use the OtlpLogging settings instead.
  • If the endpoint is missing or invalid, log export isn't applied and a warning is written to the console's error output. Delivery failures never stop the service from starting or serving requests.