Ed-Fi API Publisher Configuration

The Ed-Fi API Publisher provides a hierarchical organization of configuration information, as documented below.

The first layer of configuration values are provided by the publisherSettings.json file, which should reside in the same folder as the Ed-Fi API Publisher's binaries. This file contains the general Options available for altering the runtime behavior. The Options values can also be supplied (overridden) using environment variables or command-line arguments, as needed.

Command-line arguments take precedence over environment variables, which in turn take precedence over the values defined in the publisherSettings.json configuration file. To use environment variables to provide configuration values, use the "Configuration Path" from the tables below, and add an EdFi:Publisher: prefix to the name of each variable. For example, to specify a named connection for the source API using an environment variable, use an environment variable name of EdFi:Publisher:Connections:Source:Name.

Options

Defines general behavior of the Ed-Fi API Publisher.

Configuration Path / Command-Line Argument	Description
Options:BearerTokenRefreshMinutes --bearerTokenRefreshMinutes	Indicates how frequently the Ed-Fi API Publisher will obtain a new bearer token from the source and target API endpoints. (Default value: 28)
Options:RetryStartingDelayMilliseconds --retryStartingDelayMilliseconds	Indicates the initial delay in milliseconds used when performing an exponential "back off" delay for retries (doubling the delay between retries after each attempt). (Default value: 100)
Options:MaxRetryAttempts --maxRetryAttempts	Indicates the number of times the Ed-Fi API Publisher will attempt to resend a request against the source or target APIs before determining that the failure is permanent. (Default value: 10)
Options:MaxDegreeOfParallelismForResourceProcessing --maxDegreeOfParallelismForResourceProcessing	Indicates the total number of resources that can be processed simultaneously. (Default value: 10)
Options:MaxDegreeOfParallelismForPostResourceItem --maxDegreeOfParallelismForPostResourceItem	Indicates the total number of threads that could be simultaneously issuing POST requests against the target API for each resource being processed. (Default value: 20)
Options:MaxDegreeOfParallelismForStreamResourcePages --maxDegreeOfParallelismForStreamResourcePages	Indicates the total number of threads that could be simultaneously issuing paged GET requests against the source API for each resource being processed. (Default value: 5)
Options:StreamingPagesWaitDurationSeconds --streamingPagesWaitDurationSeconds	Indicates the number of seconds to wait for the streaming of any of the currently streaming resources to complete before providing an update on progress using the logger. (Default value: 10)
Options:StreamingPageSize --streamingPageSize	Indicates the number of items to include in each page when streaming resources from the source API. (Default value: 75)
Options:IncludeDescriptors --includeDescriptors	Indicates whether or not to attempt to publish descriptors. (Default value: false)
Options:ErrorPublishingBatchSize --errorPublishingBatchSize	Indicates the number of items to batch in each call to the error writer. This could be used to optimize the size of a batch write depending on the operating environment (e.g. Amazon DynamoDB allows for 25 items to be written in a BatchWriteItem operation). (Default value: 25)
Options:RemediationsScriptFile --remediationsScriptFile	Indicates the file system path to a JavaScript file containing remediations for failed POST requests against the target API.
Options:UseChangeVersionPaging --useChangeVersionPaging	Indicates whether or not to use change version paging. (Default value: false)
Options:ChangeVersionPagingWindowSize --changeVersionPagingWindowSize	Indicates the change version paging window size. (Default value: 25000)
Options:EnableRateLimit --enableRateLimit	Indicates whether or not to use rate limiting. (Default value: false)
Options:RateLimitNumberExecutions --rateLimitNumberExecutions	Indicates the maximum number of executions allowed within the defined time window. (Default value: 30)
Options:RateLimitTimeSeconds --rateLimitTimeSeconds	Indicates the the time span for the rate limit in seconds. (Default value: 1)
Options:RateLimitMaxRetries --rateLimitMaxRetries	Indicates the number of times the Ed-Fi API publisher will attempt to resend a request, rejected by rate limiting, to the source or destination APIs before determining that the failure is permanent. (Default value: 10)
Options:useReversePaging --useReversePaging	Indicates whether or not to use reverse paging mode. For more information about this feature read here. (Default value: false)

API Connections

Metadata for source and targets API connections can be supplied to the publisher using values stored in a persistent configuration or through environment variables and/or command-line arguments, as documented below. It is strongly recommended that you use named connections with persistent configuration for repeated publishing operations (e.g. --sourceName=abcd --targetName=wxyz). You should not mix named connections with overrides supplied through environment variables or command-line arguments.

note:

If the Ed-Fi API Publisher is executed using explicit connection information (rather than a pre-configured named connection), the LastChangeVersionProcessed value cannot be updated automatically upon successful publishing (as there is no API connection name associated with the information). It will be the responsibility of the caller to update the value appropriately after extracting the new change version from the log output (or through some other enterprising manner). As such, for implementing a process that is intended to only publish changes from a source to a target, it is impractical to use an approach where the API connection details are provided explicitly at execution time.

To select or supply source and target connection information, the following configuration values apply:

Configuration Path	Description
Connections:Source:Name --sourceName	The name of the preconfigured connection for the source Ed-Fi ODS API.
Connections:Source:Url --sourceUrl	The URL of the source Ed-Fi ODS API. Only required if named connections are not in use.
Connections:Source:Key --sourceKey	The API key for authenticating with the source Ed-Fi ODS API. Only required if named connections are not in use.
Connections:Source:Secret --sourceSecret	The API secret for authenticating with the source Ed-Fi ODS API. Only required if named connections are not in use.
Connections:Source:Scope --sourceScope	(Optional) The EducationOrganizationId scope requested for the resulting access token. The value must be an EducationOrganizationId that is explicitly associated with the API client by the source Ed-Fi ODS API. Intended for use to allow a single API connection configuration to be used to read changes from the controlling organization's Ed-Fi ODS API, but with the operations of the Ed-Fi API Publisher authorized for a particular Education Organization.
Connections:Source:SchoolYear --sourceSchoolYear	(Optional) The SchoolYear to use with the source connection (corresponding to a year-specific ODS API deployment).
Connections:Source:Include --include	(Optional) For source API connections, the resources to publish to the target with their dependencies. The value is defined using a CSV format (comma-separated values), and should contain the partial paths to the resources (e.g. /ed-fi/students,/custom/busRoutes). For convenience when working with Ed-Fi standard resources, only the name is required (e.g. students,studentSchoolAssociations). The Ed-Fi API Publisher will also evaluate and automatically include all dependencies of the requested resources (using the dependency metadata exposed by the target API). This will ensure (barring misconfigured authorization metadata or data policies) that data can be successfully published to the target API.
Connections:Source:IncludeOnly --includeOnly	(Optional) For source API connections, the resources to publish to the target without their dependencies. The value is defined using the same format as with --include (see above). NOTE: Use caution when publishing without automatically including all dependencies.
Connections:Source:Exclude --exclude	(Optional) For source API connections, the resources (and their dependents) to NOT publish to the target. The value is defined using the same format as with --include (see above). The Ed-Fi API Publisher will also evaluate and automatically exclude all dependent resources of the excluded resources (using the dependency metadata exposed by the target API). This will ensure (barring misconfigured authorization metadata or data policies) that data can be successfully published to the target API.
Connections:Source:ExcludeOnly --excludeOnly	(Optional) For source API connections, the specific resources to skip publishing to the target (dependent resources will still be published). The value is defined using the same format as with --include (see above). NOTE: Use caution when publishing without automatically including all dependencies.
Connections:Source:IgnoreIsolation --ignoreIsolation	(Optional) A boolean value (true/false) indicating whether the source Ed-Fi ODS API data should be published without using snapshot isolation. This argument must also be provided and set to true if the source API does not support an isolated context through the use of the snapshots resource (i.e. before Ed-Fi ODS v5.2). (NOTE: This is not a flag -- the value true or false must be provided (i.e. --ignoreIsolation=true).)
Connections:Source:ProfileName --sourceProfileName	(Optional) For source API connections, the name of the API Profile to read and write data from the API. You should use similar profiles between target and source to avoid data loss.
Connections:Source:LastChangeVersionProcessed --lastChangeVersionProcessed	(Optional) Indicates the last change version successfully published from the source API, and thus the change version after which the current publishing operation should start. This value explicitly overrides any change version value obtained from a named connection.
Connections:Target:Name --targetName	The name of the pre-configured connection for the target Ed-Fi ODS API.
Connections:Target:Url --targetUrl	The URL of the target Ed-Fi ODS API. Only required if named connections are not in use.
Connections:Target:Key --targetKey	The API key for authenticating with the target Ed-Fi ODS API. Only required if named connections are not in use.
Connections:Target:Secret --targetSecret	The API secret for authenticating with the target Ed-Fi ODS API. Only required if named connections are not in use.
Connections:Target:Scope --targetScope	(Optional) The EducationOrganizationId scope requested for the resulting access token. The value must be an EducationOrganizationId that is explicitly associated with the API client by the target Ed-Fi ODS API. Intended for use to allow a single API connection configuration to be used to publish changes to the controlling organization's Ed-Fi ODS API, but with the operations of the Ed-Fi API Publisher authorized for a particular Education Organization.
Connections:Target:SchoolYear --targetSchoolYear	(Optional) The SchoolYear to use with the target connection (corresponding to a year-specific ODS API deployment).
Connections:Target:ProfileName --targetProfileName	(Optional) For target API connections, the name of the API Profile to read and write data from the API. You should use similar profiles between target and source to avoid data loss.
Connections:Target:TreatForbiddenPostAsWarning --treatForbiddenPostAsWarning	(Optional) A boolean value (true/false) indicating whether 403 Forbidden responses from POST requests against the connection (as a target) should be treated as a warning, rather than a failure. NOTE: This option can be used in scenarios where the target API may not grant the Ed-Fi API Publisher full CRUD permissions to all the dependencies of the specified resources to be written. In such a scenario, the dependent data must already exist in the target ODS or the resulting 409 Conflict responses will cause publishing failure.

Considerations in relation to key changes and deletes

Ed-Fi API Publisher will only process key changes and deletions if specific Change Window is defined. To do so use the --lastChangeVersionProcessed value and set the --useChangeVersionPaging flag to true. Another option, if you want to keep the --useChangeVersionPaging false is defining a name for the source and target, using the --sourceName and --targetName values. More information about all these values below.

Authorization Failure Handling

Defines metadata (as an array of JSON objects) about which resources could experience 403 Forbidden responses caused by data dependencies needed for successful authorization, and which other resources should be processed before retrying the original request. For example, while an API client may be able to create a Student, they won't be able to update the Student until that Student is enrolled in a School through the StudentSchoolAssociation. By defining the authorization-related dependency of the update operation on the StudentSchoolAssociation, the Ed-Fi API Publisher can know to retry the failed POST request after the association has been established.

note:

This part of the configuration can only be defined in the publisherSettings.json file.

JSON Path	Description
/authorizationFailureHandling[*]	Defines metadata for a single resource which could experience 403 Forbidden responses.
/authorizationFailureHandling[*]/path	The partial path for the resource (e.g. /ed-fi/students) for which additional 403 Forbidden processing should be performed.
/authorizationFailureHandling[*]/updatePrerequisitePaths	An array of partial paths for the resource(s) (e.g. /ed-fi/studentSchoolAssociations) that should be processed before attempting to retry the original request which resulted in an authorization failure.

The default configuration, which will probably suffice for all current Ed-Fi ODS API deployments, is as follows:

{
  "options":
  {
    ...
  },
  "authorizationFailureHandling": [
    {
      "path": "/ed-fi/students",
      "updatePrerequisitePaths": ["/ed-fi/studentSchoolAssociations"]
    },
    {
      "path": "/ed-fi/staffs",
      "updatePrerequisitePaths": [
        "/ed-fi/staffEducationOrganizationEmploymentAssociations",
        "/ed-fi/staffEducationOrganizationAssignmentAssociations"
      ]
    },
    {
      "path": "/ed-fi/parents",
      "updatePrerequisitePaths": ["/ed-fi/studentParentAssociations"]
    }
  ]
}