Secrets in Connection Strings

Ed-Fi API v8 lets a platform host keep secrets in an external secret manager, such as Azure Key Vault or AWS Systems Manager Parameter Store, instead of in configuration files, environment variables or the Configuration Service database. A secrets plugin, supplied by the host, does this in two ways:

• Secret references in connection strings. A value inside the connection string of a data store, or of one of its derivatives (such as a read replica), can name a secret, for example Password=${secret:prod/dms/ds-2026}. The Configuration Service stores the reference, never the resolved value, and asks the plugin for the value each time it reads the connection string.
• Service settings from a secret store. The plugin adds the secret store as a configuration source, so each service's own secrets, such as its encryption keys, client secrets and the Configuration Service's database connection string, come from the store.

For delivering a plugin to a container and adding it to the allowlist, see Plugins. For writing a plugin, see For Plugin Developers.

Why There Are Two Mechanisms

The two mechanisms serve different kinds of values:

• Service settings are fixed when a service starts. The deployment supplies them, and the service reads them from its configuration.
• Connection strings are data. API clients write them to the Configuration Service through /v3/dataStores and /v3/dataStoreDerivatives while the services run, and they never pass through either service's configuration.

So a configuration source can't supply a data store connection string. Adding a key such as ConnectionStrings:MyDataStore to the secret store has no effect. To keep a data store's password out of the Configuration Service database, use a secret reference.

This is a change from Ed-Fi ODS/API 7.3, where External Configuration of ODS Connection Strings supplied whole ODS connection strings from configuration sources. In Ed-Fi API v8, connection strings stay in the Configuration Service, and individual values inside them are resolved from the secret store. Adding a tenant or a data store is an API call, not a configuration change.

Secret References

Syntax

A reference has the form ${secret:<name>}. The name is one or more of the characters A–Z, a–z, 0–9, _, -, ., /, :, @ and +. Names are matched exactly and are case-sensitive, because secret store names are. In a multi-tenant deployment, the plugin is also told which tenant the connection string is being read for, so it can map the same name to a different secret for each tenant.

PostgreSQL

Host=db;Port=5432;Username=edfi;Database=edfi_datastore_2026;Password=${secret:prod/dms/ds-2026}

SQL Server

Server=db;Database=edfi_datastore_2026;User ID=edfi;Password=${secret:prod/dms/ds-2026};Encrypt=False

Submit the connection string, reference included, in the connectionString field of a POST or PUT to /v3/dataStores or /v3/dataStoreDerivatives, the same way as any other connection string. See API Client and Data Store Configuration.

• A reference can go in the value of any text keyword, not only Password. A user name, host or database name can also come from the secret store.
• Keywords the database engine reads as numbers or other types, such as Port or Connect Timeout, reject a reference when the connection string is written, because the engine can't parse it.

Edge cases

• A value can hold more than one reference. Each one is resolved separately, and a resolved value is never scanned for further references.
• Text that starts with ${ but isn't a complete reference is left as it is, so an existing password that contains ${ is unaffected. There's no escape sequence.
• Resolved values are quoted as the database engine requires, so a secret that contains ;, =, quotes or leading spaces reaches the database driver exactly as the secret store holds it.
• The returned connection string may look different. When you decrypt a connection string the Configuration Service returns, its keyword names, casing, order and quoting may differ from what you submitted.

The Ed-Fi API receives connection strings that are already resolved and never sees a reference.

When a Reference Doesn't Resolve

An unresolved reference is never passed to the Ed-Fi API as text. If no resolver plugin is installed, or the plugin can't return a value for a reference, the request fails:

• GET /v3/dataStores and GET /v3/dataStores/{id} return 500 Internal Server Error.
• GET requests to /v3/dataStoreDerivatives also return 500.
• The exception is a derivative returned as part of its data store. Its connection string is returned as null and a warning is logged, so the data store and its other derivatives still load.

A reference also fails to resolve when the plugin returns an empty value, throws an exception, or takes longer than SecretsSettings:ResolveTimeoutSeconds. The Configuration Service logs the data store, the tenant and the reference's name, never a secret value.

Settings

The SecretsSettings section of the Configuration Service controls how resolved values are cached and how long a read waits for the plugin:

Setting	Default	Description
SecretsSettings:CacheExpirationSeconds	300	How long a resolved value is reused, for each tenant and secret name, before the plugin is asked again. 0 turns caching off.
SecretsSettings:ResolveTimeoutSeconds	10	How long one read can spend waiting for the plugin, across all of the references it resolves. A request for data stores includes two reads. Must be at least 1.

The Configuration Service doesn't start if either value is out of range. In the Docker Compose files that ship with the Ed-Fi API, set them with CONFIGURATION_SERVICE_SECRETS_CACHE_EXPIRATION_SECONDS and CONFIGURATION_SERVICE_SECRETS_RESOLVE_TIMEOUT_SECONDS.

Sizing the timeout

• Keep ResolveTimeoutSeconds well under 50. The Ed-Fi API waits up to 100 seconds for the Configuration Service to return a tenant's data stores. That request includes two reads, one for the data stores and one for their derivatives, and each read can wait out the timeout.
• Leave caching on, or size the timeout for the whole read. The timeout covers every reference a read resolves. Without caching, each reference is a separate call to the plugin, so a read that returns many data stores can run out of time even when each call is fast.

See Configuration Details.

Rotating a Referenced Secret

When every refresh succeeds, a rotated secret reaches a running Ed-Fi API within the sum of two cache windows:

• SecretsSettings:CacheExpirationSeconds in the Configuration Service, which is how long it reuses a resolved value, plus
• CacheSettings:DataStoreCacheExpirationSeconds in the Ed-Fi API, which is how long it keeps the data stores it fetched from the Configuration Service.

warning

With the default settings, a rotated secret can take about 15 minutes (300 + 600 seconds) to reach the Ed-Fi API. Until then, the Ed-Fi API keeps using the old value.

That's an estimate, not a guarantee. If the Ed-Fi API can't refresh its data stores, for example because the Configuration Service or the secret store is unavailable, it logs a warning ("Failed to refresh data store cache") and keeps using the connection strings it already has, with the old value, for as long as the refresh keeps failing.

To rotate a database password without failed connections, switch to a second login, then revoke the old one. PostgreSQL and SQL Server don't let one login hold two passwords, so a second login is the only way for both credentials to work at once:

1. Create the new login at the database, and store its password in the secret store under a new secret name.
2. Update the data store's connection string through the API, with a PUT that has the new user name and a reference to the new secret name, for example Username=edfi_2027;Password=${secret:prod/dms/ds-2027}. Both values change together, and the new secret name has nothing cached, so the Configuration Service fetches it on the next read. Do the same for every derivative (through /v3/dataStoreDerivatives), and every other data store in any tenant, that uses the old login.
3. Confirm that every Ed-Fi API instance is using the new login, for example by checking at the database that no sessions still use the old login. Elapsed time alone doesn't prove this, because of the refresh failures described above.
4. Remove the old login. Removing it too early causes authentication failures in the Ed-Fi API.

Don't rotate by putting the user name in a secret reference of its own. The Configuration Service caches each secret separately, from the time it fetched it, so a user name and a password can expire at different times. A read in between can pair the new login with the old password, or the reverse, and the Ed-Fi API keeps that connection string until its own window passes.

• To rotate immediately, restart the Configuration Service, then the Ed-Fi API. Nothing pushes a rotation sooner. Restarting only the Configuration Service clears its own cache, but the Ed-Fi API keeps the data stores it already fetched until its own window passes.
• If the Ed-Fi API doesn't refresh data stores, because CacheSettings:DataStoreCacheRefreshEnabled is false or DataStoreCacheExpirationSeconds is 0 or less, it keeps the old value until it restarts.

When the new value arrives, the Ed-Fi API replaces its connection pool for that data store, just as when a connection string is edited through the API.

To check a rotation that doesn't seem to have taken effect, set SecretsSettings:CacheExpirationSeconds to 0, so the Configuration Service asks the plugin on every read.

Both cache windows are measured on the system clock, so a rotation can take longer if a server's clock was set back.

Service Settings from a Secret Store

A plugin that adds a configuration source can serve any configuration key. These are the secrets each service reads from its configuration:

Service	Key	Description
Ed-Fi API	ConfigurationServiceSettings:ClientSecret	The client secret the Ed-Fi API uses to authenticate to the Configuration Service.
Ed-Fi API	ConfigurationServiceSettings:EncryptionKey	The key the Ed-Fi API decrypts stored connection strings with.
Configuration Service	DatabaseSettings:DatabaseConnection	The Configuration Service's own database connection string. See Startup Scripts and Database Connection Strings.
Configuration Service	DatabaseSettings:EncryptionKey	The key the Configuration Service encrypts stored connection strings with.
Configuration Service	IdentitySettings:ClientSecret	The secret for IdentitySettings:ClientId. With Keycloak, the Configuration Service uses it to call the Keycloak admin API. It's required and validated for every identity provider.
Configuration Service	IdentitySettings:EncryptionKey	The key that protects the signing keys stored in the database.
Configuration Service	IdentitySettings:CertificatePassword	The production signing certificate's password.
Configuration Service	IdentitySettings:DevCertificatePassword	The development signing certificate's password.
Either service	OtlpLogging:Headers:<name>	A header sent with each OpenTelemetry log export, such as OtlpLogging:Headers:Authorization.

IdentitySettings:CertificatePassword and IdentitySettings:DevCertificatePassword aren't in the shipped appsettings.json, but the Configuration Service still reads them.

Changing one of these values requires a restart of the service that reads it. Each one is read once, when the service starts or first uses it. SecretsSettings:CacheExpirationSeconds doesn't apply to them, and neither does a configuration source that reloads on a schedule.

Removing These Keys from the Environment

Environment variables (other than those prefixed with ASPNETCORE_ or DOTNET_) and command-line arguments take priority over plugin sources. A key that the secret store serves must therefore not also be set in the environment, or the environment value is used instead. See Settings Supplied by Plugins.

With the Docker Compose files that ship with the Ed-Fi API, keep the .env variables set, and unset each key in an override file of your own with !reset null. List only the keys the secret store serves:

services:
  dms:
    environment:
      ConfigurationServiceSettings__ClientSecret: !reset null
      ConfigurationServiceSettings__EncryptionKey: !reset null
  config:
    environment:
      DatabaseSettings__DatabaseConnection: !reset null
      DatabaseSettings__EncryptionKey: !reset null
      IdentitySettings__ClientSecret: !reset null
      IdentitySettings__EncryptionKey: !reset null

Using the shipped Docker Compose files

Deleting a variable from .env doesn't work:

• In the local Compose files, which build the images from source, a deleted variable sets the key to an empty string, which still takes priority over the secret store.
• The published Compose files, which use the published images, require the two encryption key variables, so deleting either one stops Docker Compose with an error. Deleting either client secret variable sets that key to an empty string, as in the local files.
• DatabaseSettings__DatabaseConnection has a default that points to the bundled PostgreSQL container, so deleting its variable connects the Configuration Service to that database instead.

Include a service block in the override only if your Compose project defines that service. The dms block can go in the same override that sets the Ed-Fi API's Plugins__Allowed, listed in DMS_PLUGINS_COMPOSE_FILES.

A reset can be silently ignored

• The config block must be in a file listed after the Configuration Service's Compose file. The launch scripts add the files in DMS_PLUGINS_COMPOSE_FILES before that file, so a config reset there is overwritten.
• In the combined Ed-Fi API and Configuration Service stack, some Docker Compose versions don't apply a config reset at all.

Before you rely on the override, render the full Compose configuration you deploy with and confirm that each reset key is absent. The rendered output contains secret values, so don't save or share it. The operations guide in the Ed-Fi API repository has a script that does this check. See Secrets on the shipped Compose files.

Startup Scripts and Database Connection Strings

Each container's startup script can wait for the database before it starts the service. The script runs before any configuration source exists, so it only sees the environment.

• The Configuration Service waits using the host in DatabaseSettings__DatabaseConnection. That connection string can come entirely from the secret store, including on the published container image. When the variable isn't in the environment, the script skips the wait and starts the service immediately. If the database isn't ready yet, the Configuration Service exits (with the default AppSettings:DeployDatabaseOnStartup of true), so the container needs a restart policy to bring it back. The shipped Compose files use restart: unless-stopped.
• The Ed-Fi API waits for PostgreSQL using DATABASE_CONNECTION_STRING_ADMIN. This variable can't come from a secret store. The image doesn't need it: without it, the script skips the wait. The local Compose file gives it a default with superuser credentials for the bundled PostgreSQL container, and the published Compose file requires it to be set. To keep it out of the container, keep the variable set in .env and add DATABASE_CONNECTION_STRING_ADMIN: !reset null under dms in your override.

Choosing Which Plugins to Deploy

The Ed-Fi API and the Configuration Service each load plugins separately, from their own plugin directory and allowlist. What a plugin does decides where it can be allowlisted:

What the plugin does	Configuration Service	Ed-Fi API
Looks up the values of secret references, or replaces the client secret hasher	Yes	No: the Ed-Fi API won't start
Only adds the secret store as a configuration source	Yes	Yes

The difference is in how the plugin is built. Every plugin has two optional steps that the service calls at startup: one adds configuration sources, and the other registers the plugin's services.

• A plugin that only adds a configuration source implements just the first step. It registers nothing that only the Configuration Service defines, so either service can load it.
• A plugin that looks up secret references or replaces the client secret hasher also implements the second step, where it registers a secret resolver or a hasher from the EdFi.Api.Secrets package. Only the Configuration Service has those features, so the Ed-Fi API refuses to start with this plugin, even if it also adds a configuration source.

A typical deployment that uses both mechanisms therefore runs one plugin in the Configuration Service, which looks up secret references and adds a configuration source, and a second, configuration-only plugin in the Ed-Fi API. The two plugins can share the code that connects to the secret store. See Plugins for delivery, the allowlist, and how plugin-supplied settings rank against other configuration.

What This Does and Doesn't Protect

Consider each of these points before you adopt a secrets plugin.

• The plugin runs with the same permissions as the service. It sees every secret name the Configuration Service resolves, and nothing limits what it does. You control whether it runs, through each service's allowlist. See Trust.
• The plugin still needs a credential for the secret store. It can read that credential only from the existing configuration or the environment. A stored credential, such as a client secret or access key, reduces many secrets to one. A managed identity in Azure or an IAM role in AWS avoids storing one at all.
• The Configuration Service can still produce the resolved value. The value is no longer stored in the Configuration Service database or its backups. It is held in the Configuration Service's memory while cached, and the data store and derivative GET endpoints return it, encrypted with DatabaseSettings:EncryptionKey. Anyone with a token those endpoints accept, plus the encryption key, can get the current value. If your requirement is that only the secret store can produce a secret, this mechanism doesn't meet it.
• Resolved values never appear in the Configuration Service's logs. The Ed-Fi API's logs are different: when a database driver reports a failed login or an unknown database, its error message can include a resolved user name or database name, and the Ed-Fi API logs that error. The guarantee also doesn't cover the plugin itself, which is responsible for not logging what it fetches.

Write access to data stores exposes secrets

Nothing restricts which secret names a reference uses. A client that can create or update data stores or derivatives (edfi_admin_api/full_access), in any existing tenant, can store a connection string such as Host=<a server it controls>;Username=${secret:<any name>}. The Configuration Service resolves it, and the Ed-Fi API sends the value to that server when it connects, which it does at startup for every data store it loads.

Without secret references, that client could replace a stored password but never read one. With them, write access to data stores becomes access to every secret the plugin's identity can read. Grant write access to data stores accordingly, and limit the plugin's secret store identity to the secrets the Configuration Service needs. That reduces what such a client can reach, but doesn't remove the risk.

For Plugin Developers

A secrets plugin derives from EdFiApiPlugin, like every plugin, and overrides only the steps it needs:

• To add a configuration source, override ContributeConfiguration and add the secret store there. Only the EdFi.Api.Plugins package is needed.
• To look up secret references, reference the EdFi.Api.Secrets package, implement ISecretResolver, and register it in ContributeServices. A replacement client secret hasher implements IClientSecretHasher and is registered the same way.

The Ed-Fi API Secrets implementer guide in the Ed-Fi API repository covers writing a resolver, the registration rules, and worked examples of plugins for Azure Key Vault and AWS Systems Manager Parameter Store. See Building a Plugin for the rules every plugin follows, including naming, registration and packaging.