IBD Optimizations
Making Bitcoin Core sync faster — from incremental wins to architectural breakthroughs
~16%
COMBINED SPEEDUP
5.28×
SWIFTSYNC
12h 7m
CORE 30 FULL SYNC
TWO PARALLEL EFFORTS
Incremental engineering: l0rinc's umbrella PR
Architectural breakthrough: SwiftSync skips writing intermediate UTXO states entirely, achieving a 5× speedup on default settings.
#32043 groups dozens of targeted optimizations — LevelDB tuning, serialization, parallelism — delivering ~16% faster IBD.Architectural breakthrough: SwiftSync skips writing intermediate UTXO states entirely, achieving a 5× speedup on default settings.
The Big Wins
SWIFTSYNC — PR #34004
Designed by Ruben Somsen, implemented by rustaceanrob. A pre-generated "hints file" tells the node which UTXOs will be spent before a target height. Those coins are never written to disk. Result: 41 hours → ~7.8 hours on default settings. Opens the door to parallel block validation during IBD.
PARALLEL INPUT FETCHING — PR #31132
By andrewtoth. During
ConnectBlock, input prevouts are fetched sequentially — cache misses hit disk. This PR parallelizes fetching via a lock-free MPSC queue with work stealing. Results vary dramatically by storage: 3× on cloud/network storage, 52% on ARM SBCs (RockPro64: 67h→32h), 46% on direct-attached SSD. Especially impactful for low-powered hardware and high-latency storage.LEVELDB FILE SIZE
PR
#30039: Increased from 2 MiB to 32 MiB. ~30% IBD speedup by reducing compaction cycles. Shipped in Bitcoin Core 29.0.BULK SERIALIZATION
PR
#31551: Buffered block/undo serialization for block I/O. Microbenchmarks showed reads ~30% faster and writes ~168% faster on macOS, with ~7% faster IBD overall in GCC full-sync tests.SwiftSync: How It Works
THE INSIGHT
During IBD, most UTXOs are created and spent before the sync target. Writing them to disk and then deleting them is wasted work. What if we just… didn't?
THE MECHANISM
1. A trusted snapshot generates a "hints file" (~hundreds of MB) listing which outputs survive to the target height
2. During sync, coins that won't survive are never written to the UTXO database
3. Only the net UTXO set is built — no intermediate states, no deletions
4. At the target height, the resulting UTXO set hash is verified against a known commitment
2. During sync, coins that won't survive are never written to the UTXO database
3. Only the net UTXO set is built — no intermediate states, no deletions
4. At the target height, the resulting UTXO set hash is verified against a known commitment
TRUST MODEL
A malicious hints file cannot corrupt the UTXO set — it can only waste computing time. Every created-but-not-stored output goes into a MuHash cryptographic accumulator; every spent output is removed. At the target height, the accumulator must be empty, proving the hints were correct. Currently restricted to pruned nodes only.
FUTURE POTENTIAL
Because entries are never removed from the accumulator during sync, blocks can theoretically be validated in parallel. This removes single-threaded CPU and disk I/O as bottlenecks, leaving only bandwidth and raw compute. Can also accelerate AssumeUTXO's background validation. Combined with the incremental I/O fixes, the path to sub-hour full validation is becoming visible.
Benchmarks & Resources
| OPTIMIZATION | SPEEDUP |
|---|---|
| SwiftSync (hints file) | 5.28× (41h → ~7.8h) |
| l0rinc combined (5 GiB cache) | ~16% (8.59h → 7.25h) |
| LevelDB file size (PR #30039) | ~30% IBD |
| Parallel input fetch (PR #31132) | ~30% reindex |
| Bulk serialization (PR #31551) | ~6% IBD |
| LevelDB batch size (PR #31645) | Flush: 240s → 150s |
KEY CONTRIBUTORS
l0rinc — IBD tracking PR, serialization/cache/DB optimizations • Ruben Somsen — SwiftSync design • rustaceanrob — SwiftSync implementation • andrewtoth — parallel input fetching • willcl-ark — Benchcoin infrastructure • TheCharlatan — libbitcoinkernel