Skip to main content
Ed-Fi.orgDocsCommunityAcademy

📢 Ed-Fi API Publisher v1.4: Now Available

· 5 min read
Steven Arnold
Manager, Technical Programs

The Ed-Fi Alliance is pleased to announce the release of Ed-Fi API Publisher v1.4. This release adds support for Ed-Fi API v8, reads large sources faster with cursor paging, lets a failed run resume where it stopped, and reports through its exit code whether a run lost documents. It also moves the API Publisher to .NET 10.

Highlights​

Ed-Fi API v8 Support​

The API Publisher can now publish from and to Ed-Fi API v8, in any combination with the Ed-Fi ODS/API:

  • from an ODS/API to Ed-Fi API v8
  • from Ed-Fi API v8 to an ODS/API
  • between two Ed-Fi API v8 instances

Support is validated against Ed-Fi API v8.0. The API Publisher now reads each API's Discovery document to find where it serves its resources and its token endpoint, so a connection needs only the API's base URL, key and secret. New optional connection settings (authUrl, dataManagementUrlSegment, changeQueriesUrlSegment) cover APIs that sit behind a proxy or declare an address their callers cannot reach.

Considerations for API Hosts lists the claim sets, API client setup and prerequisites for an Ed-Fi API v8 target.

Cursor Paging​

When the source is an Ed-Fi ODS/API 7.3+ or Ed-Fi API 8.1+, the API Publisher reads each resource with partitioned cursor paging instead of offset paging. Each resource is split into partitions that are read in parallel, and a page deep in a resource is no slower to read than the first one, which shortens publishing for large resources. Other sources, including Ed-Fi API v8.0, are read with offset paging as before.

Cursor paging is on by default. Use --disableCursorPaging=true to return to offset paging.

Resumable Runs​

A run that fails partway through can now be continued with --resumeLastRun=true instead of starting again, for sources read with cursor paging. The API Publisher keeps its progress in a run state file, written to the working directory unless --runStatePath is set.

Exit Codes That Report Lost Documents​

Each run now ends with a summary table and a specific exit code:

CodeMeaning
0Success
1Documents rejected beyond the tolerance
2Run incomplete
3Authentication failure
4Invalid configuration

The toleratedItemErrorCount option sets how many rejected documents a run may have and still succeed.

Reliability and Memory​

  • 429 Too Many Requests responses from the source are retried, honoring Retry-After, and concurrent source requests can be capped with maxConcurrentSourceRequests.
  • Bearer token refresh is retried before the token expires, and a 401 response triggers a new token and a replay of the request.
  • Bounded processing buffers and streamed page reads lower memory growth on long runs.
  • A new processDeletesAndKeyChangesOnFullPublish option applies deletes and key changes on a full publish.

Before You Upgrade​

  • .NET 10 required. Install the .NET 10 runtime before upgrading a deployment that uses the portable EdFi.ApiPublisher package. The Windows package includes the runtime, and the Docker image is built on .NET 10.
  • Exit codes changed. A run that lost documents no longer exits with 0, and failures no longer share a single code of -1 (255 on Linux and macOS). Update any automation that tests for -1 or 255. Best-effort behavior is available with --toleratedItemErrorCount=-1.
  • Ed-Fi API v8 behind a rewriting proxy. The API Publisher now asks Ed-Fi API v8 for data directly, so a proxy that only routes /data/v3/* answers 404. Remove the rewrite, or set --sourceDataManagementUrlSegment=data/v3 or --targetDataManagementUrlSegment=data/v3. An ODS/API is unaffected.
  • Token endpoint taken from Discovery. A connection without authUrl now requests its token from the oauth URL the API declares. If the API declares an address its callers cannot reach, set authUrl on the connection.

The release notes list the other behavior changes, including the new defaults for buffering and 429 retries.

Known Issues​

Ed-Fi API v8.0 rejects requests above its default rate limit, so turn on --enableRateLimit when publishing into it. It does not provide snapshot isolation as a source, so publish from it while it is not being changed. Writes that time out on every retry are not yet reported as failed. See Known Issues Details for these and the few data values that differ between the two APIs.

Try It Out​

Download v1.4.0 from the GitHub release, then see What's New and API Publisher Configuration for the new options.

What's Next​

tip

You can track upcoming work in the Roadmap Boards on GitHub.

We'd love to hear how v1.4 works for you. Report bugs or request features through the Ed-Fi Community Hub.