packages feed

haskoin-store 1.16.0 → 1.16.1

raw patch · 8 files changed

+173/−33 lines, 8 filesdep ~haskoin-coredep ~haskoin-nodePVP: major bump suggested

API removals or changes: PVP suggests a major version bump

Dependency ranges changed: haskoin-core, haskoin-node

API changes (from Hackage documentation)

+ Haskoin.Store.BlockStore.Sync: [gap] :: Sync p -> !Maybe NominalDiffTime
+ Haskoin.Store.BlockStore.Sync: stallFactor :: NominalDiffTime
+ Haskoin.Store.BlockStore.Sync: stallFloor :: NominalDiffTime
- Haskoin.Store.BlockStore.Sync: Sync :: ![BlockHash] -> !Maybe (Asked p) -> !UTCTime -> !Int -> ![Asked p] -> !Maybe (Probe p) -> Sync p
+ Haskoin.Store.BlockStore.Sync: Sync :: ![BlockHash] -> !Maybe (Asked p) -> !UTCTime -> !Int -> !Maybe NominalDiffTime -> ![Asked p] -> !Maybe (Probe p) -> Sync p

Files

CHANGELOG.md view
@@ -4,6 +4,26 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html) +## [1.16.1] - 2026-11-18++### Changed+- **The stall follows the syncing peer's pace.** A peer is passed over when+  it has sent nothing for four of its recent gaps between blocks, at least+  two seconds, instead of always `--block-stall` seconds, which is now the+  longest stall. A store that imports hundreds of small blocks a second+  waited ten seconds on a peer that had gone quiet; it now waits two. The+  pure rules are in `Haskoin.Store.BlockStore.Sync` (`patience`,+  `stallFloor`, `stallFactor`, and the new `gap` of `Sync`).++### Fixed+- **Header sync no longer stalls block sync.** haskoin-node waited for the+  chain to import each batch of 2,000 headers before it passed on any other+  peer message, and on Bitcoin Cash a batch took 20 seconds. Blocks and pongs+  waited behind it, and the store passed over peers that were delivering.+  haskoin-node 2.6.1 no longer waits, and haskoin-core 2.0.1 connects Bitcoin+  Cash headers about 60 times as fast. Both are unreleased and pinned by+  commit.+ ## [1.16.0] - 2026-11-18  This release has one design (`docs/one-fleet.md`): Redis is required, every
README.md view
@@ -320,7 +320,7 @@ | `-p`, `--peer` | `PEER` | none | Connect to `HOST[:PORT]`; repeat for more, space-separated in `PEER` | | `--max-peers` | `MAX_PEERS` | `20` | Most peers to connect to | | `--peer-timeout` | `PEER_TIMEOUT` | `30` | Seconds before a peer that is silent, or that owes blocks, is dropped |-| `--block-stall` | `BLOCK_STALL` | `10` | Seconds without a block from the syncing peer before another peer is asked for it too |+| `--block-stall` | `BLOCK_STALL` | `10` | Longest wait, in seconds, for a block from the syncing peer before another peer is asked too; a quick peer gets four of its recent gaps between blocks, at least 2 s | | `--block-ping` | `BLOCK_PING` | `5` | Seconds that a peer has to answer a ping to be asked for blocks, unless none answers in time | | `--max-peer-life` | `MAX_PEER_LIFE` | `172800` | Seconds after which a peer is dropped, so connections rotate | | `--block-window` | `BLOCK_WINDOW` | `64` | Blocks to request from the syncing peer at a time |@@ -597,9 +597,12 @@   longer than `--block-ping` seconds is asked only if none answers in   time. This costs each new block one round trip to the quickest peer. - **A peer that does not deliver is passed over, not waited for.** If the-  peer that was asked sends nothing for `--block-stall` seconds, another-  is found the same way and asked for the same blocks, and whichever-  delivers first is taken. The stall doubles each time, up to+  peer that was asked sends nothing for four of its recent gaps between+  blocks, at least two seconds and at most `--block-stall` seconds,+  another is found the same way and asked for the same blocks, and+  whichever delivers first is taken. A store that imports hundreds of+  blocks a second so notices a quiet peer in two seconds, while a peer+  that takes seconds over each large block is not passed over for it. The stall doubles each time, up to   `--peer-timeout`, so that a slow link still gets its blocks; a peer that   still owes blocks after `--peer-timeout` is disconnected. - **Reorganisations** are handled by reverting blocks until the best block
app/Main.hs view
@@ -569,7 +569,13 @@     option auto $       metavar "SECONDS"         <> long "block-stall"-        <> helpEnv "BLOCK_STALL" "Ask another peer for blocks that the syncing peer has not delivered for this long"+        <> helpEnv+          "BLOCK_STALL"+          ( unwords+              [ "Longest wait for a block from the syncing peer before another is asked too;",+                "shorter for a quick peer"+              ]+          )         <> showDefault         <> value c.blockStall   blockPing <-
haskoin-store.cabal view
@@ -5,7 +5,7 @@ -- see: https://github.com/sol/hpack  name:           haskoin-store-version:        1.16.0+version:        1.16.1 synopsis:       Storage and index for Bitcoin and Bitcoin Cash description:    An indexer for Bitcoin and Bitcoin Cash. It downloads blocks and mempool                 transactions from the peer-to-peer network, indexes them in RocksDB, and@@ -120,8 +120,8 @@     , file-embed >=0.0.16.0 && <0.1     , hashable >=1.5.1.0 && <1.6     , hashtables >=1.4.2 && <1.5-    , haskoin-core >=2.0.0 && <2.1-    , haskoin-node >=2.6.0 && <2.7+    , haskoin-core >=2.0.1 && <2.1+    , haskoin-node >=2.6.1 && <2.7     , haskoin-store-data >=3.0.0 && <3.1     , hedis >=0.16.4 && <0.17     , http-types >=0.12.6 && <0.13@@ -187,8 +187,8 @@     , file-embed >=0.0.16.0 && <0.1     , hashable >=1.5.1.0 && <1.6     , hashtables >=1.4.2 && <1.5-    , haskoin-core >=2.0.0 && <2.1-    , haskoin-node >=2.6.0 && <2.7+    , haskoin-core >=2.0.1 && <2.1+    , haskoin-node >=2.6.1 && <2.7     , haskoin-store     , haskoin-store-data >=3.0.0 && <3.1     , hedis >=0.16.4 && <0.17@@ -253,8 +253,8 @@     , filepath >=1.5.5.0 && <1.6     , hashable >=1.5.1.0 && <1.6     , hashtables >=1.4.2 && <1.5-    , haskoin-core >=2.0.0 && <2.1-    , haskoin-node >=2.6.0 && <2.7+    , haskoin-core >=2.0.1 && <2.1+    , haskoin-node >=2.6.1 && <2.7     , haskoin-store     , haskoin-store-data >=3.0.0 && <3.1     , hedis >=0.16.4 && <0.17@@ -351,8 +351,8 @@     , filepath >=1.5.5.0 && <1.6     , hashable >=1.5.1.0 && <1.6     , hashtables >=1.4.2 && <1.5-    , haskoin-core >=2.0.0 && <2.1-    , haskoin-node >=2.6.0 && <2.7+    , haskoin-core >=2.0.1 && <2.1+    , haskoin-node >=2.6.1 && <2.7     , haskoin-store     , haskoin-store-data >=3.0.0 && <3.1     , hedis >=0.16.4 && <0.17@@ -423,8 +423,8 @@     , filepath >=1.5.5.0 && <1.6     , hashable >=1.5.1.0 && <1.6     , hashtables >=1.4.2 && <1.5-    , haskoin-core >=2.0.0 && <2.1-    , haskoin-node >=2.6.0 && <2.7+    , haskoin-core >=2.0.1 && <2.1+    , haskoin-node >=2.6.1 && <2.7     , haskoin-store     , haskoin-store-data >=3.0.0 && <3.1     , hedis >=0.16.4 && <0.17@@ -489,8 +489,8 @@     , file-embed >=0.0.16.0 && <0.1     , hashable >=1.5.1.0 && <1.6     , hashtables >=1.4.2 && <1.5-    , haskoin-core >=2.0.0 && <2.1-    , haskoin-node >=2.6.0 && <2.7+    , haskoin-core >=2.0.1 && <2.1+    , haskoin-node >=2.6.1 && <2.7     , haskoin-store     , haskoin-store-data >=3.0.0 && <3.1     , hedis >=0.16.4 && <0.17
src/Haskoin/Store/BlockStore.hs view
@@ -22,7 +22,8 @@ -- answer is the one, or a late one if none answers within 'blockPing'. Each -- block is imported as soon as it arrives. If the best block of the database is no longer in the main chain of -- headers, blocks are reverted until it is. A syncing peer that delivers--- nothing for 'blockStall' is passed over: another is found the same way and+-- nothing for a few of its recent gaps between blocks, at least two seconds+-- and at most 'blockStall', is passed over: another is found the same way and -- asked for the same blocks, and whichever delivers first is taken. A peer that sends a -- block that it was not asked for, cannot find a block, or still owes blocks -- after 'peerTimeout' is disconnected. The state of the download, and every rule@@ -464,8 +465,9 @@     mempoolTimeout :: !Int,     -- | disconnect syncing peer if inactive for this long     peerTimeout :: !NominalDiffTime,-    -- | ask another peer for the blocks that the syncing peer has not-    -- delivered for this long ("Haskoin.Store.BlockStore.Sync")+    -- | the longest that the syncing peer may deliver nothing before another+    -- peer is asked for its blocks; a quick peer is given less, a few of its+    -- recent gaps between blocks ("Haskoin.Store.BlockStore.Sync")     blockStall :: !NominalDiffTime,     -- | a peer is asked for blocks if it answers a ping within this long,     -- or if none does ("Haskoin.Store.BlockStore.Sync")
src/Haskoin/Store/BlockStore/Sync.hs view
@@ -48,6 +48,19 @@ -- no reason to disconnect anyone. Only a peer that has still not delivered -- when the long timeout runs out is disconnected, as before. --+-- = How long a stall is+--+-- The stall follows the syncing peer's own pace ('patience'): four times+-- the recent gap between the blocks that it delivered ('gap'), but at least+-- 'stallFloor' and at most the stall that the store was given. A store that+-- imports hundreds of small blocks a second sees a peer that has gone quiet+-- in two seconds, not ten; one that takes seconds over each large block is+-- given longer. A fixed stall was the alternative: short enough to catch a+-- quiet peer quickly, it passes over every peer on a slow link, and long+-- enough for a slow link, it leaves a fast store idle for most of the time.+-- Until a block is imported, there is no pace to go by, and the stall is+-- the one given.+-- -- The stall doubles each time a peer is passed over, up to the long -- timeout, and goes back to its start when a block is imported. A store on -- a link that cannot bring a block within the first stall therefore still@@ -102,6 +115,8 @@     wantsMore,     needsMoreBlocks,     patience,+    stallFloor,+    stallFactor,     stalled,     timedOut,     expired,@@ -125,6 +140,9 @@     time :: !UTCTime,     -- | syncing peers passed over since a block was last imported     passes :: !Int,+    -- | the recent gap between imported blocks, a moving average, if a+    -- block was imported: the pace that 'patience' follows+    gap :: !(Maybe NominalDiffTime),     -- | peers that were passed over and still owe blocks     passed :: ![Asked p],     -- | the peers that were pinged to find the next syncing peer, if one@@ -164,7 +182,8 @@  -- | No syncing peer, and nothing wanted. idle :: UTCTime -> Sync p-idle now = Sync {wanted = [], current = Nothing, time = now, passes = 0, passed = [], probe = Nothing}+idle now =+  Sync {wanted = [], current = Nothing, time = now, passes = 0, gap = Nothing, passed = [], probe = Nothing}  -- | The syncing peer, if any. syncingPeer :: Sync p -> Maybe p@@ -271,9 +290,15 @@     verdict = if h `elem` s.wanted then Requested else Late  -- | A block was imported: it is no longer wanted, and the clock and the--- stall start again.+-- stall start again. The time since the clock last started goes into the+-- recent gap between blocks, with a weight of a quarter, so that a few+-- slow blocks lengthen the stall and a burst of quick ones does not make it+-- jumpy. imported :: UTCTime -> BlockHash -> Sync p -> Sync p-imported now h s = s {wanted = delete h s.wanted, time = now, passes = 0}+imported now h s = s {wanted = delete h s.wanted, time = now, passes = 0, gap = Just gap'}+  where+    sample = max 0 (now `diffUTCTime` s.time)+    gap' = maybe sample (\g -> (3 * g + sample) / 4) s.gap  -- | Blocks were requested from the syncing peer. Those already wanted are -- not added again. Nothing changes without a syncing peer.@@ -349,11 +374,27 @@ needsMoreBlocks window n = n < max 1 (window `div` 4)  -- | How long the syncing peer has to deliver a block before another is--- asked: the stall, doubled for each peer passed over since a block was+-- asked: 'stallFactor' times the recent gap between blocks, at least+-- 'stallFloor' and at most the stall given, or the stall given if no block+-- was imported yet; doubled for each peer passed over since a block was -- last imported, and never longer than the timeout. patience :: NominalDiffTime -> NominalDiffTime -> Sync p -> NominalDiffTime-patience stall timeout s = min timeout (stall * 2 ^ min 30 s.passes)+patience stall timeout s = min timeout (base * 2 ^ min 30 s.passes)+  where+    base = maybe stall (\g -> min stall (max stallFloor (stallFactor * g))) s.gap +-- | The shortest stall, however quick the peer was: long enough for a round+-- trip across the world and the next block of a quick peer, short enough+-- that a store importing hundreds of blocks a second idles only briefly.+stallFloor :: NominalDiffTime+stallFloor = 2++-- | How many of its recent gaps between blocks a peer may be silent before+-- it is passed over. Blocks come in bursts, so the gap varies; a factor of+-- four lets a peer take a pause well beyond its usual one.+stallFactor :: NominalDiffTime+stallFactor = 4+ -- | The peers to ask all the same, best first, if none of those pinged -- answered within the wait and none can any more: there is no other -- option. Those that answered late come first, in the order that they@@ -370,7 +411,10 @@     late g = isJust g.answered || now `diffUTCTime` g.at > wait  -- | The syncing peer, if blocks are wanted and none was imported for--- longer than its 'patience': time to ask another peer, with 'pass'.+-- longer than its 'patience': time to ask another peer, with 'pass'. The+-- block store asks this both when it starts a search and when a pinged peer+-- answers, and gets the same answer both times unless a block came in+-- between. stalled :: NominalDiffTime -> NominalDiffTime -> UTCTime -> Sync p -> Maybe p stalled stall timeout now s   | null s.wanted = Nothing
src/Haskoin/Store/Manager.hs view
@@ -182,8 +182,8 @@     mempoolTimeout :: !Int,     -- | disconnect peer if message not received for this many seconds     peerTimeout :: !NominalDiffTime,-    -- | ask another peer for the blocks that the syncing peer has not-    -- delivered for this many seconds+    -- | the longest that the syncing peer may deliver nothing before another+    -- peer is asked for its blocks; a quick peer is given less     blockStall :: !NominalDiffTime,     -- | a peer is asked for blocks if it answers a ping within this long,     -- or if none does
test/Haskoin/Store/BlockStore/SyncSpec.hs view
@@ -168,9 +168,17 @@       | (now, s) <- run es       ] -  prop "is patient for the stall at least, and for the timeout at most" $ \es ->-    and [patience stall timeout s >= stall && patience stall timeout s <= timeout | (_, s) <- run es]+  prop "is patient for the floor at least, and for the timeout at most" $ \es ->+    and [patience stall timeout s >= stallFloor && patience stall timeout s <= timeout | (_, s) <- run es] +  prop "is patient for the stall given until a block is imported, then for at most that" $ \es ->+    and+      [ if isNothing s.gap then base == stall else base <= stall+      | (_, s) <- run es,+        s.passes == 0,+        let base = patience stall timeout s+      ]+   prop "times out only a syncing peer when nothing was imported for longer than the timeout" $ \es ->     and       [ timedOut timeout now s == if now `diffUTCTime` s.time > timeout then syncingPeer s else Nothing@@ -232,10 +240,11 @@       -- peer 1 was asked at 11 seconds, and has 20 from then       stalled stall timeout (at 31) s1 `shouldBe` Nothing       stalled stall timeout (at 32) s1 `shouldBe` Just 1-    it "goes back to the first stall when a block is imported" $ do+    it "goes back to a stall of its own pace when a block is imported" $ do       let (_, s1) = pass (at 11) 1 asked           s2 = imported (at 12) (hash 1) (snd (accept 1 (hash 1) s1))-      patience stall timeout s2 `shouldBe` 10+      -- one second to deliver: four seconds of patience, no doubling+      patience stall timeout s2 `shouldBe` 4     it "is disconnected if it still owes its block after the timeout" $ do       let (_, s) = pass (at 11) 1 asked       -- silent since the start, not since it was passed over@@ -254,6 +263,62 @@       passedPeers s3 `shouldBe` []       -- the block is not wanted any more: the request went with peer 1       fst (accept 0 (hash 1) s3) `shouldBe` Late++  describe "a stall that follows the peer's pace" $ do+    let ms = (/ 1000) . fromIntegral :: Int -> NominalDiffTime+        later = flip addUTCTime t0+        started = fromMaybe (idle t0) (start t0 (0 :: Int) (idle t0))+        -- peer 0 delivers blocks 1 to 99 of 100, one every 10 ms+        quick =+          foldl+            (\s' b -> imported (later (ms (10 * b))) (hash b) (snd (accept 0 (hash b) s')))+            (request (map hash [1 .. 100]) started)+            [1 .. 99]+    it "passes over a quick peer that went quiet after the floor, not the stall" $ do+      patience stall timeout quick `shouldBe` stallFloor+      stalled stall timeout (later (ms 990 + stallFloor)) quick `shouldBe` Nothing+      stalled stall timeout (later (ms 1000 + stallFloor)) quick `shouldBe` Just 0+    it "gives a slow peer several of its gaps, up to the stall" $ do+      let slow g =+            foldl+              (\s' b -> imported (later (g * fromIntegral b)) (hash b) (snd (accept 0 (hash b) s')))+              (request (map hash [1 .. 10]) started)+              [1 .. 9 :: Int]+      patience stall timeout (slow 1) `shouldBe` 4+      patience stall timeout (slow 5) `shouldBe` stall+    it "is not made jumpy by one quick block after slow ones" $ do+      let s = imported (later 3.1) (hash 3) (snd (accept 0 (hash 3) slow2))+          slow2 =+            imported (later 3) (hash 2) . snd . accept 0 (hash 2) $+              imported (later 1.5) (hash 1) (snd (accept 0 (hash 1) (request (map hash [1 .. 4]) started)))+      patience stall timeout s `shouldSatisfy` (> stallFloor)+    it "starts the clock of the next peer when it passes over a quiet one" $ do+      let (_, s) = pass (later (ms 1000 + stallFloor + 1)) 1 quick+      -- the next peer gets twice the floor, from when it was asked+      patience stall timeout s `shouldBe` 2 * stallFloor+      stalled stall timeout (later (ms 1000 + 3 * stallFloor)) s `shouldBe` Nothing++  describe "a peer that answers a ping while the syncing peer is stalled" $ do+    let wait = 5 :: NominalDiffTime+        asked = request [hash 1] (fromMaybe (idle t0) (start t0 (0 :: Int) (idle t0)))+        -- the tick at 11 seconds finds peer 0 stalled and pings 1 and 2+        (Just (n, pinged), searching) = ping (at 11) 3 [1, 2] asked+    it "is chosen at once, and the stalled peer is passed over for it" $ do+      pinged `shouldBe` [1, 2]+      stalled stall timeout (at 11) asked `shouldBe` Just 0+      let (chosen, s) = pong wait (at 11) 2 n searching+      chosen `shouldBe` True+      -- the block store passes over when the peer is still stalled+      stalled stall timeout (at 11) s `shouldBe` Just 0+      let (told, s') = pass (at 11) 2 s+      told `shouldBe` Just (0, [hash 1])+      syncingPeer s' `shouldBe` Just 2+      probing s' `shouldBe` False+      overdue wait (at 17) s' `shouldBe` []+    it "is not chosen if the syncing peer delivered in the meantime" $ do+      let (_, s) = pong wait (at 11) 2 n searching+          s' = imported (at 11) (hash 1) (snd (accept 0 (hash 1) s))+      stalled stall timeout (at 11) s' `shouldBe` Nothing    describe "Looking for a peer to ask" $ do     let wait = 5 :: NominalDiffTime