haskoin-store 1.16.0 → 1.16.1
raw patch · 8 files changed
+173/−33 lines, 8 filesdep ~haskoin-coredep ~haskoin-nodePVP: major bump suggested
API removals or changes: PVP suggests a major version bump
Dependency ranges changed: haskoin-core, haskoin-node
API changes (from Hackage documentation)
+ Haskoin.Store.BlockStore.Sync: [gap] :: Sync p -> !Maybe NominalDiffTime
+ Haskoin.Store.BlockStore.Sync: stallFactor :: NominalDiffTime
+ Haskoin.Store.BlockStore.Sync: stallFloor :: NominalDiffTime
- Haskoin.Store.BlockStore.Sync: Sync :: ![BlockHash] -> !Maybe (Asked p) -> !UTCTime -> !Int -> ![Asked p] -> !Maybe (Probe p) -> Sync p
+ Haskoin.Store.BlockStore.Sync: Sync :: ![BlockHash] -> !Maybe (Asked p) -> !UTCTime -> !Int -> !Maybe NominalDiffTime -> ![Asked p] -> !Maybe (Probe p) -> Sync p
Files
- CHANGELOG.md +20/−0
- README.md +7/−4
- app/Main.hs +7/−1
- haskoin-store.cabal +13/−13
- src/Haskoin/Store/BlockStore.hs +5/−3
- src/Haskoin/Store/BlockStore/Sync.hs +50/−6
- src/Haskoin/Store/Manager.hs +2/−2
- test/Haskoin/Store/BlockStore/SyncSpec.hs +69/−4
CHANGELOG.md view
@@ -4,6 +4,26 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html) +## [1.16.1] - 2026-11-18++### Changed+- **The stall follows the syncing peer's pace.** A peer is passed over when+ it has sent nothing for four of its recent gaps between blocks, at least+ two seconds, instead of always `--block-stall` seconds, which is now the+ longest stall. A store that imports hundreds of small blocks a second+ waited ten seconds on a peer that had gone quiet; it now waits two. The+ pure rules are in `Haskoin.Store.BlockStore.Sync` (`patience`,+ `stallFloor`, `stallFactor`, and the new `gap` of `Sync`).++### Fixed+- **Header sync no longer stalls block sync.** haskoin-node waited for the+ chain to import each batch of 2,000 headers before it passed on any other+ peer message, and on Bitcoin Cash a batch took 20 seconds. Blocks and pongs+ waited behind it, and the store passed over peers that were delivering.+ haskoin-node 2.6.1 no longer waits, and haskoin-core 2.0.1 connects Bitcoin+ Cash headers about 60 times as fast. Both are unreleased and pinned by+ commit.+ ## [1.16.0] - 2026-11-18 This release has one design (`docs/one-fleet.md`): Redis is required, every
README.md view
@@ -320,7 +320,7 @@ | `-p`, `--peer` | `PEER` | none | Connect to `HOST[:PORT]`; repeat for more, space-separated in `PEER` | | `--max-peers` | `MAX_PEERS` | `20` | Most peers to connect to | | `--peer-timeout` | `PEER_TIMEOUT` | `30` | Seconds before a peer that is silent, or that owes blocks, is dropped |-| `--block-stall` | `BLOCK_STALL` | `10` | Seconds without a block from the syncing peer before another peer is asked for it too |+| `--block-stall` | `BLOCK_STALL` | `10` | Longest wait, in seconds, for a block from the syncing peer before another peer is asked too; a quick peer gets four of its recent gaps between blocks, at least 2 s | | `--block-ping` | `BLOCK_PING` | `5` | Seconds that a peer has to answer a ping to be asked for blocks, unless none answers in time | | `--max-peer-life` | `MAX_PEER_LIFE` | `172800` | Seconds after which a peer is dropped, so connections rotate | | `--block-window` | `BLOCK_WINDOW` | `64` | Blocks to request from the syncing peer at a time |@@ -597,9 +597,12 @@ longer than `--block-ping` seconds is asked only if none answers in time. This costs each new block one round trip to the quickest peer. - **A peer that does not deliver is passed over, not waited for.** If the- peer that was asked sends nothing for `--block-stall` seconds, another- is found the same way and asked for the same blocks, and whichever- delivers first is taken. The stall doubles each time, up to+ peer that was asked sends nothing for four of its recent gaps between+ blocks, at least two seconds and at most `--block-stall` seconds,+ another is found the same way and asked for the same blocks, and+ whichever delivers first is taken. A store that imports hundreds of+ blocks a second so notices a quiet peer in two seconds, while a peer+ that takes seconds over each large block is not passed over for it. The stall doubles each time, up to `--peer-timeout`, so that a slow link still gets its blocks; a peer that still owes blocks after `--peer-timeout` is disconnected. - **Reorganisations** are handled by reverting blocks until the best block
app/Main.hs view
@@ -569,7 +569,13 @@ option auto $ metavar "SECONDS" <> long "block-stall"- <> helpEnv "BLOCK_STALL" "Ask another peer for blocks that the syncing peer has not delivered for this long"+ <> helpEnv+ "BLOCK_STALL"+ ( unwords+ [ "Longest wait for a block from the syncing peer before another is asked too;",+ "shorter for a quick peer"+ ]+ ) <> showDefault <> value c.blockStall blockPing <-
haskoin-store.cabal view
@@ -5,7 +5,7 @@ -- see: https://github.com/sol/hpack name: haskoin-store-version: 1.16.0+version: 1.16.1 synopsis: Storage and index for Bitcoin and Bitcoin Cash description: An indexer for Bitcoin and Bitcoin Cash. It downloads blocks and mempool transactions from the peer-to-peer network, indexes them in RocksDB, and@@ -120,8 +120,8 @@ , file-embed >=0.0.16.0 && <0.1 , hashable >=1.5.1.0 && <1.6 , hashtables >=1.4.2 && <1.5- , haskoin-core >=2.0.0 && <2.1- , haskoin-node >=2.6.0 && <2.7+ , haskoin-core >=2.0.1 && <2.1+ , haskoin-node >=2.6.1 && <2.7 , haskoin-store-data >=3.0.0 && <3.1 , hedis >=0.16.4 && <0.17 , http-types >=0.12.6 && <0.13@@ -187,8 +187,8 @@ , file-embed >=0.0.16.0 && <0.1 , hashable >=1.5.1.0 && <1.6 , hashtables >=1.4.2 && <1.5- , haskoin-core >=2.0.0 && <2.1- , haskoin-node >=2.6.0 && <2.7+ , haskoin-core >=2.0.1 && <2.1+ , haskoin-node >=2.6.1 && <2.7 , haskoin-store , haskoin-store-data >=3.0.0 && <3.1 , hedis >=0.16.4 && <0.17@@ -253,8 +253,8 @@ , filepath >=1.5.5.0 && <1.6 , hashable >=1.5.1.0 && <1.6 , hashtables >=1.4.2 && <1.5- , haskoin-core >=2.0.0 && <2.1- , haskoin-node >=2.6.0 && <2.7+ , haskoin-core >=2.0.1 && <2.1+ , haskoin-node >=2.6.1 && <2.7 , haskoin-store , haskoin-store-data >=3.0.0 && <3.1 , hedis >=0.16.4 && <0.17@@ -351,8 +351,8 @@ , filepath >=1.5.5.0 && <1.6 , hashable >=1.5.1.0 && <1.6 , hashtables >=1.4.2 && <1.5- , haskoin-core >=2.0.0 && <2.1- , haskoin-node >=2.6.0 && <2.7+ , haskoin-core >=2.0.1 && <2.1+ , haskoin-node >=2.6.1 && <2.7 , haskoin-store , haskoin-store-data >=3.0.0 && <3.1 , hedis >=0.16.4 && <0.17@@ -423,8 +423,8 @@ , filepath >=1.5.5.0 && <1.6 , hashable >=1.5.1.0 && <1.6 , hashtables >=1.4.2 && <1.5- , haskoin-core >=2.0.0 && <2.1- , haskoin-node >=2.6.0 && <2.7+ , haskoin-core >=2.0.1 && <2.1+ , haskoin-node >=2.6.1 && <2.7 , haskoin-store , haskoin-store-data >=3.0.0 && <3.1 , hedis >=0.16.4 && <0.17@@ -489,8 +489,8 @@ , file-embed >=0.0.16.0 && <0.1 , hashable >=1.5.1.0 && <1.6 , hashtables >=1.4.2 && <1.5- , haskoin-core >=2.0.0 && <2.1- , haskoin-node >=2.6.0 && <2.7+ , haskoin-core >=2.0.1 && <2.1+ , haskoin-node >=2.6.1 && <2.7 , haskoin-store , haskoin-store-data >=3.0.0 && <3.1 , hedis >=0.16.4 && <0.17
src/Haskoin/Store/BlockStore.hs view
@@ -22,7 +22,8 @@ -- answer is the one, or a late one if none answers within 'blockPing'. Each -- block is imported as soon as it arrives. If the best block of the database is no longer in the main chain of -- headers, blocks are reverted until it is. A syncing peer that delivers--- nothing for 'blockStall' is passed over: another is found the same way and+-- nothing for a few of its recent gaps between blocks, at least two seconds+-- and at most 'blockStall', is passed over: another is found the same way and -- asked for the same blocks, and whichever delivers first is taken. A peer that sends a -- block that it was not asked for, cannot find a block, or still owes blocks -- after 'peerTimeout' is disconnected. The state of the download, and every rule@@ -464,8 +465,9 @@ mempoolTimeout :: !Int, -- | disconnect syncing peer if inactive for this long peerTimeout :: !NominalDiffTime,- -- | ask another peer for the blocks that the syncing peer has not- -- delivered for this long ("Haskoin.Store.BlockStore.Sync")+ -- | the longest that the syncing peer may deliver nothing before another+ -- peer is asked for its blocks; a quick peer is given less, a few of its+ -- recent gaps between blocks ("Haskoin.Store.BlockStore.Sync") blockStall :: !NominalDiffTime, -- | a peer is asked for blocks if it answers a ping within this long, -- or if none does ("Haskoin.Store.BlockStore.Sync")
src/Haskoin/Store/BlockStore/Sync.hs view
@@ -48,6 +48,19 @@ -- no reason to disconnect anyone. Only a peer that has still not delivered -- when the long timeout runs out is disconnected, as before. --+-- = How long a stall is+--+-- The stall follows the syncing peer's own pace ('patience'): four times+-- the recent gap between the blocks that it delivered ('gap'), but at least+-- 'stallFloor' and at most the stall that the store was given. A store that+-- imports hundreds of small blocks a second sees a peer that has gone quiet+-- in two seconds, not ten; one that takes seconds over each large block is+-- given longer. A fixed stall was the alternative: short enough to catch a+-- quiet peer quickly, it passes over every peer on a slow link, and long+-- enough for a slow link, it leaves a fast store idle for most of the time.+-- Until a block is imported, there is no pace to go by, and the stall is+-- the one given.+-- -- The stall doubles each time a peer is passed over, up to the long -- timeout, and goes back to its start when a block is imported. A store on -- a link that cannot bring a block within the first stall therefore still@@ -102,6 +115,8 @@ wantsMore, needsMoreBlocks, patience,+ stallFloor,+ stallFactor, stalled, timedOut, expired,@@ -125,6 +140,9 @@ time :: !UTCTime, -- | syncing peers passed over since a block was last imported passes :: !Int,+ -- | the recent gap between imported blocks, a moving average, if a+ -- block was imported: the pace that 'patience' follows+ gap :: !(Maybe NominalDiffTime), -- | peers that were passed over and still owe blocks passed :: ![Asked p], -- | the peers that were pinged to find the next syncing peer, if one@@ -164,7 +182,8 @@ -- | No syncing peer, and nothing wanted. idle :: UTCTime -> Sync p-idle now = Sync {wanted = [], current = Nothing, time = now, passes = 0, passed = [], probe = Nothing}+idle now =+ Sync {wanted = [], current = Nothing, time = now, passes = 0, gap = Nothing, passed = [], probe = Nothing} -- | The syncing peer, if any. syncingPeer :: Sync p -> Maybe p@@ -271,9 +290,15 @@ verdict = if h `elem` s.wanted then Requested else Late -- | A block was imported: it is no longer wanted, and the clock and the--- stall start again.+-- stall start again. The time since the clock last started goes into the+-- recent gap between blocks, with a weight of a quarter, so that a few+-- slow blocks lengthen the stall and a burst of quick ones does not make it+-- jumpy. imported :: UTCTime -> BlockHash -> Sync p -> Sync p-imported now h s = s {wanted = delete h s.wanted, time = now, passes = 0}+imported now h s = s {wanted = delete h s.wanted, time = now, passes = 0, gap = Just gap'}+ where+ sample = max 0 (now `diffUTCTime` s.time)+ gap' = maybe sample (\g -> (3 * g + sample) / 4) s.gap -- | Blocks were requested from the syncing peer. Those already wanted are -- not added again. Nothing changes without a syncing peer.@@ -349,11 +374,27 @@ needsMoreBlocks window n = n < max 1 (window `div` 4) -- | How long the syncing peer has to deliver a block before another is--- asked: the stall, doubled for each peer passed over since a block was+-- asked: 'stallFactor' times the recent gap between blocks, at least+-- 'stallFloor' and at most the stall given, or the stall given if no block+-- was imported yet; doubled for each peer passed over since a block was -- last imported, and never longer than the timeout. patience :: NominalDiffTime -> NominalDiffTime -> Sync p -> NominalDiffTime-patience stall timeout s = min timeout (stall * 2 ^ min 30 s.passes)+patience stall timeout s = min timeout (base * 2 ^ min 30 s.passes)+ where+ base = maybe stall (\g -> min stall (max stallFloor (stallFactor * g))) s.gap +-- | The shortest stall, however quick the peer was: long enough for a round+-- trip across the world and the next block of a quick peer, short enough+-- that a store importing hundreds of blocks a second idles only briefly.+stallFloor :: NominalDiffTime+stallFloor = 2++-- | How many of its recent gaps between blocks a peer may be silent before+-- it is passed over. Blocks come in bursts, so the gap varies; a factor of+-- four lets a peer take a pause well beyond its usual one.+stallFactor :: NominalDiffTime+stallFactor = 4+ -- | The peers to ask all the same, best first, if none of those pinged -- answered within the wait and none can any more: there is no other -- option. Those that answered late come first, in the order that they@@ -370,7 +411,10 @@ late g = isJust g.answered || now `diffUTCTime` g.at > wait -- | The syncing peer, if blocks are wanted and none was imported for--- longer than its 'patience': time to ask another peer, with 'pass'.+-- longer than its 'patience': time to ask another peer, with 'pass'. The+-- block store asks this both when it starts a search and when a pinged peer+-- answers, and gets the same answer both times unless a block came in+-- between. stalled :: NominalDiffTime -> NominalDiffTime -> UTCTime -> Sync p -> Maybe p stalled stall timeout now s | null s.wanted = Nothing
src/Haskoin/Store/Manager.hs view
@@ -182,8 +182,8 @@ mempoolTimeout :: !Int, -- | disconnect peer if message not received for this many seconds peerTimeout :: !NominalDiffTime,- -- | ask another peer for the blocks that the syncing peer has not- -- delivered for this many seconds+ -- | the longest that the syncing peer may deliver nothing before another+ -- peer is asked for its blocks; a quick peer is given less blockStall :: !NominalDiffTime, -- | a peer is asked for blocks if it answers a ping within this long, -- or if none does
test/Haskoin/Store/BlockStore/SyncSpec.hs view
@@ -168,9 +168,17 @@ | (now, s) <- run es ] - prop "is patient for the stall at least, and for the timeout at most" $ \es ->- and [patience stall timeout s >= stall && patience stall timeout s <= timeout | (_, s) <- run es]+ prop "is patient for the floor at least, and for the timeout at most" $ \es ->+ and [patience stall timeout s >= stallFloor && patience stall timeout s <= timeout | (_, s) <- run es] + prop "is patient for the stall given until a block is imported, then for at most that" $ \es ->+ and+ [ if isNothing s.gap then base == stall else base <= stall+ | (_, s) <- run es,+ s.passes == 0,+ let base = patience stall timeout s+ ]+ prop "times out only a syncing peer when nothing was imported for longer than the timeout" $ \es -> and [ timedOut timeout now s == if now `diffUTCTime` s.time > timeout then syncingPeer s else Nothing@@ -232,10 +240,11 @@ -- peer 1 was asked at 11 seconds, and has 20 from then stalled stall timeout (at 31) s1 `shouldBe` Nothing stalled stall timeout (at 32) s1 `shouldBe` Just 1- it "goes back to the first stall when a block is imported" $ do+ it "goes back to a stall of its own pace when a block is imported" $ do let (_, s1) = pass (at 11) 1 asked s2 = imported (at 12) (hash 1) (snd (accept 1 (hash 1) s1))- patience stall timeout s2 `shouldBe` 10+ -- one second to deliver: four seconds of patience, no doubling+ patience stall timeout s2 `shouldBe` 4 it "is disconnected if it still owes its block after the timeout" $ do let (_, s) = pass (at 11) 1 asked -- silent since the start, not since it was passed over@@ -254,6 +263,62 @@ passedPeers s3 `shouldBe` [] -- the block is not wanted any more: the request went with peer 1 fst (accept 0 (hash 1) s3) `shouldBe` Late++ describe "a stall that follows the peer's pace" $ do+ let ms = (/ 1000) . fromIntegral :: Int -> NominalDiffTime+ later = flip addUTCTime t0+ started = fromMaybe (idle t0) (start t0 (0 :: Int) (idle t0))+ -- peer 0 delivers blocks 1 to 99 of 100, one every 10 ms+ quick =+ foldl+ (\s' b -> imported (later (ms (10 * b))) (hash b) (snd (accept 0 (hash b) s')))+ (request (map hash [1 .. 100]) started)+ [1 .. 99]+ it "passes over a quick peer that went quiet after the floor, not the stall" $ do+ patience stall timeout quick `shouldBe` stallFloor+ stalled stall timeout (later (ms 990 + stallFloor)) quick `shouldBe` Nothing+ stalled stall timeout (later (ms 1000 + stallFloor)) quick `shouldBe` Just 0+ it "gives a slow peer several of its gaps, up to the stall" $ do+ let slow g =+ foldl+ (\s' b -> imported (later (g * fromIntegral b)) (hash b) (snd (accept 0 (hash b) s')))+ (request (map hash [1 .. 10]) started)+ [1 .. 9 :: Int]+ patience stall timeout (slow 1) `shouldBe` 4+ patience stall timeout (slow 5) `shouldBe` stall+ it "is not made jumpy by one quick block after slow ones" $ do+ let s = imported (later 3.1) (hash 3) (snd (accept 0 (hash 3) slow2))+ slow2 =+ imported (later 3) (hash 2) . snd . accept 0 (hash 2) $+ imported (later 1.5) (hash 1) (snd (accept 0 (hash 1) (request (map hash [1 .. 4]) started)))+ patience stall timeout s `shouldSatisfy` (> stallFloor)+ it "starts the clock of the next peer when it passes over a quiet one" $ do+ let (_, s) = pass (later (ms 1000 + stallFloor + 1)) 1 quick+ -- the next peer gets twice the floor, from when it was asked+ patience stall timeout s `shouldBe` 2 * stallFloor+ stalled stall timeout (later (ms 1000 + 3 * stallFloor)) s `shouldBe` Nothing++ describe "a peer that answers a ping while the syncing peer is stalled" $ do+ let wait = 5 :: NominalDiffTime+ asked = request [hash 1] (fromMaybe (idle t0) (start t0 (0 :: Int) (idle t0)))+ -- the tick at 11 seconds finds peer 0 stalled and pings 1 and 2+ (Just (n, pinged), searching) = ping (at 11) 3 [1, 2] asked+ it "is chosen at once, and the stalled peer is passed over for it" $ do+ pinged `shouldBe` [1, 2]+ stalled stall timeout (at 11) asked `shouldBe` Just 0+ let (chosen, s) = pong wait (at 11) 2 n searching+ chosen `shouldBe` True+ -- the block store passes over when the peer is still stalled+ stalled stall timeout (at 11) s `shouldBe` Just 0+ let (told, s') = pass (at 11) 2 s+ told `shouldBe` Just (0, [hash 1])+ syncingPeer s' `shouldBe` Just 2+ probing s' `shouldBe` False+ overdue wait (at 17) s' `shouldBe` []+ it "is not chosen if the syncing peer delivered in the meantime" $ do+ let (_, s) = pong wait (at 11) 2 n searching+ s' = imported (at 11) (hash 1) (snd (accept 0 (hash 1) s))+ stalled stall timeout (at 11) s' `shouldBe` Nothing describe "Looking for a peer to ask" $ do let wait = 5 :: NominalDiffTime