Header sync from a CDN

This page shows how to give a new install its block headers from a CDN instead of from peers, and how to report the sync’s progress and outcome.

A wallet needs the block header chain to check merkle proofs (SPV). Over P2P, a fresh install fetches it 2,000 headers per round trip, which on testnet takes tens of minutes. With cdnBaseUrl set, libspiffy first downloads the headers as pre-built 80-byte binary chunks, validates them and bulk-imports them. P2P then fetches only the blocks the CDN does not have yet.

Turn it on

Pass cdnBaseUrl to initialize(), with the two optional callbacks:

// Sketch: storage, ARC and secure-storage arguments omitted.
import 'package:libspiffy/libspiffy.dart';

final system = LibSpiffyActorSystem();

await system.initialize(
  networkType: 'test',
  enableP2P: true,
  cdnBaseUrl: 'https://headers.example.com',
  onHeaderSyncProgress: (int current, int total, CdnSyncPhase phase) {
    print('headers: ${phase.name} $current/$total');
  },
  onHeaderSyncResult: (CdnSyncResult result) {
    if (!result.success) {
      print('CDN not used (${result.error}); peers supply the headers');
    }
  },
);

libspiffy builds the URLs from the base URL and the network’s canonical name (NetworkName.canonical): mainnet, testnet or regtest.

https://headers.example.com/testnet/manifest.json
https://headers.example.com/testnet/headers_0000001_0050000.bin
https://headers.example.com/testnet/headers_0050001_0100000.bin
...

The base URL must be https. initialize() does not accept a plain http CDN: the sync fails with an ArgumentError before any request, onHeaderSyncResult reports it, and the headers come from peers.

libspiffy 5.0.0 has no default CDN. If you leave cdnBaseUrl out, onHeaderSyncResult is still called once, with success: false and the error No header CDN configured.

What happens at startup

initialize() runs the CDN sync alongside the rest of its startup (lib/src/actors/libspiffy_actor_system.dart):

It opens storage and the header chain, then starts the CDN sync in the background.
It registers event types, starts the projections and spawns the actors.
It completes system.ready. From here the coordinator takes commands.
It waits for the CDN sync to finish.
If enableP2P is true, it starts P2P. Header sync over P2P continues from the tip of the stored chain. The startHeight parameter does not change that: it is only the height this node reports to peers in its version handshake.
initialize() returns.

On a first install, step 4 can take minutes. Creating or importing a wallet needs no header chain, so don’t wait for initialize() to return before you send commands. Wait for ready:

// Sketch: start libspiffy without blocking the caller on the header sync.
final started = system.initialize(cdnBaseUrl: cdnUrl, networkType: 'test');
await system.ready;      // actors take commands
// ... create wallets, show the UI ...
await started;           // CDN sync done, P2P started

A payment whose block header has not arrived yet waits for it in the SPV actor.

system.isReady reads the same state synchronously.

Retries and fallback

Each chunk download has a 30-second timeout and up to 3 attempts. If a pass still fails, the actor system starts another pass on a fresh HTTP client, up to three passes in all. Each pass resumes from the tip of the stored chain, so it does not download again what it already imported. After the third failed pass, onHeaderSyncResult reports the error and P2P fetches the remaining headers.

A failed CDN sync never fails initialize().

Chunk cache

Chunks are downloaded one at a time: download, validate, import, then the next. If you pass dataDirectory to initialize(), libspiffy uses it as the chunk cache directory, so a chunk downloaded before the app was killed is not downloaded again. Without dataDirectory, chunks are held in memory only. When you pass your own isar, libspiffy uses dataDirectory for nothing else.

Progress and result

onHeaderSyncProgress has the type CdnSyncProgressCallback: void Function(int current, int total, CdnSyncPhase phase). total is the manifest’s total header count, and current counts headers already in storage, so a resumed sync does not restart at zero.

`CdnSyncPhase`	Reported when
`fetchingManifest`	Before `manifest.json` is fetched (`current` and `total` are 0).
`downloadingChunks`	Before each chunk is downloaded or read from the cache.
`validatingChunks`	Before each chunk is validated.
`importingHeaders`	Before each chunk is written to storage.
`complete`	The pass imported every chunk it needed.
`fallbackToP2P`	The pass failed. `total` is 0. Another pass may follow.

When storage already holds every header the manifest lists, the pass reports only fetchingManifest and returns success with 0 headers imported.

onHeaderSyncResult gets one CdnSyncResult for the whole startup sync:

Field	Type	Meaning
`success`	`bool`	The last pass imported everything it needed.
`headersImported`	`int`	Headers imported by the last pass.
`finalHeight`	`int`	Height of the stored chain’s tip afterwards.
`elapsed`	`Duration`	Time the last pass took.
`error`	`String?`	Why the sync ended without success.

After startup, system.headerChain.bestHeight is the local tip and system.networkHeight is the highest height connected peers reported (0 with no peer connected). Compare the two to show “synced”.

What libspiffy checks

The CDN is treated as untrusted transport. Before it writes anything from a chunk, libspiffy checks:

Integrity: the chunk’s SHA-256 matches the manifest.
Anchoring: on an empty store, the first header is the network’s genesis block, built into the code (NetworkParams). Otherwise the first new header links to the stored tip.
Continuity: each header’s prevBlock is the hash of the one before it.
Proof of work: each header’s hash meets its own target, and that target is no easier than the network’s powLimit.
Checkpoints: block hashes match the manifest’s checkpoints. These come from the same server as the headers, so a match proves nothing alone; a mismatch rejects the chunk.

A CDN can’t make the wallet accept headers that peers would not.

Configuration reference

initialize() builds a CdnHeaderSyncConfig itself, with the default values below. You can’t pass one to initialize(). The class matters when you run CdnHeaderSyncService yourself (lib/src/spv/cdn_header_sync_config.dart).

Parameter	Default	Meaning
`baseUrl`	required	CDN base URL.
`network`	required	`testnet`, `mainnet` or `regtest`. Also selects genesis and `powLimit`.
`downloadTimeout`	30 s	Timeout per chunk download.
`maxRetries`	3	Attempts per chunk.
`validateProofOfWork`	`true`	Check each header’s proof of work. Turn off only for synthetic test data.
`verifyCheckpoints`	`true`	Compare against the manifest’s checkpoints.
`allowInsecureHttp`	`false`	Allow an `http` base URL, for a local test CDN.
`onProgress`	`null`	Progress callback.
`cacheDirectory`	`null`	Directory for the chunk cache.

libspiffy 5.0.0 removes concurrentDownloads, which nothing read: chunks are validated in sequence. Drop it from any CdnHeaderSyncConfig you build.

Build the CDN files

libspiffy’s repository has two tools in tool/. Both write the same layout: manifest.json plus headers_SSSSSSS_EEEEEEE.bin chunks of 50,000 headers by default. Chunks start at height 1, since the genesis block is built into the library.

From WhatsOnChain’s published header files:

dart run tool/fetch_woc_headers_to_cdn.dart \
  --network mainnet \
  --output /path/to/cdn/mainnet

It also takes --cache (default <output>/.woc-cache), --chunk-size and --reorg-margin (default 100). It checks genesis, linkage and proof of work before it writes, and leaves out the newest --reorg-margin blocks so a tip that is later reorganised away never reaches the CDN.

From a block_headers.json backup file:

dart run tool/export_headers_to_cdn.dart \
  --source /path/to/block_headers.json \
  --output /path/to/cdn/testnet \
  --network testnet \
  --chunk-size 50000

This one writes a checkpoint every 100,000 blocks into the manifest.

Upload the output directory to any static file server under <base>/<network>/. Each .bin file is raw 80-byte headers back to back. A CDN that is a few days behind is fine: peers supply the rest. Rebuild and upload to keep the P2P catch-up short.

Regtest

For a regtest chain (such as localnet-teranode) you usually leave cdnBaseUrl unset and let peers supply the short chain. If your app picks its settings from the network, pass null for cdnBaseUrl whenever NetworkName.isRegtest(networkType) is true.