haskoin-store 1.12.0 → 1.12.1

raw patch · 8 files changed

+148/−35 lines, 8 filesPVP: major bump suggested

API removals or changes: PVP suggests a major version bump

API changes (from Hackage documentation)

Files

• CHANGELOG.md +18/−0
• README.md +3/−1
• haskoin-store.cabal +1/−1
• src/Haskoin/Store/Web.hs +11/−3
• src/Haskoin/Store/Web/Publish.hs +41/−10
• swagger.yaml +7/−3
• test/Haskoin/Store/IntegrationSpec.hs +51/−14
• test/Haskoin/Store/Web/PublishSpec.hs +16/−3

@@ -4,6 +4,24 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html) +## [1.12.1] - 2026-11-13++### Fixed+- **A transaction that the store has unconfirmed is sent again** when it+  is posted again. The store answered 200 at once and sent nothing, as+  for a confirmed one. Its mempool is not held to its node's, so it could+  list a payment that the nodes forgot, after a reorganisation or when a+  node lost its mempool, and the wallet had no way to get it back to them:+  on Bitcoin Cash none at all until the store dropped it after 14 days.+  Now it goes through the same checks, is given to a node over RPC or on+  the courier, and is answered 200 when the node took it, or with the+  node's reason. A confirmed transaction is still answered at once.++### Changed+- In the library, the field `has` of `Haskoin.Store.Web.Publish.Publishing`+  is now `holds`, which says whether the store has the transaction in a+  block, in its mempool or not at all (`Held`).+ ## [1.12.0] - 2026-11-13  Long lists of transactions and raw blocks are sent while they are read,

@@ -233,7 +233,9 @@  A transaction that cannot be valid, or that a node refuses with a reason, is answered 400 with the reason. A 500 means that the store could not find out:-post the same transaction again, which sends nothing twice.+post the same transaction again, which is safe. A transaction that the store+lists as unconfirmed is given to a node again when it is posted, in case the+network forgot it. [Publishing](#publishing) has the rest.  ### 6. Stay up to date without polling

@@ -5,7 +5,7 @@ -- see: https://github.com/sol/hpack  name:           haskoin-store-version:        1.12.0+version:        1.12.1 synopsis:       Storage and index for Bitcoin and Bitcoin Cash description:    An indexer for Bitcoin and Bitcoin Cash. It downloads blocks and mempool                 transactions from the peer-to-peer network, indexes them in RocksDB, and

@@ -1530,8 +1530,10 @@  -- | Handler for @POST \/transactions@: hand the transaction to a full node, -- and answer with its hash once a node has announced it back--- ("Haskoin.Store.Web.Publish"). What the store or a node refuses is 400--- with the reason; no node, no answer and no announcement are 500.+-- ("Haskoin.Store.Web.Publish"). One that the store has unconfirmed is+-- handed to a node again, and answered once the node took it. What the+-- store or a node refuses is 400 with the reason; no node, no answer and+-- no announcement are 500. handlePostTx :: WebState -> PostTx -> IO TxId handlePostTx st (PostTx tx) =   withMetrics st (.txPost) $@@ -1637,7 +1639,7 @@       outbox = st.outbox,       rules = chainRules st,       direct = maybe (\_ -> return Publish.Unavailable) (sendNode st) st.config.nodeRpc,-      has = fmap (maybe False (not . (.deleted))) . st.query.getTxData,+      holds = fmap heldOf . st.query.getTxData,       coin = Publish.coinOf st.query,       present = \h -> L.withView own.db $ \v ->         maybe False (not . (.deleted)) <$> (L.layeredRead v).getTxData h,@@ -1658,6 +1660,12 @@     }   where     own = st.config.store+    heldOf = \case+      Just td+        | td.deleted -> Publish.Absent+        | confirmed td.block -> Publish.Confirmed+        | otherwise -> Publish.Pending+      Nothing -> Publish.Absent  -- GET Mempool / Events -- 

@@ -32,7 +32,7 @@ -- 1. __Check__ what the store can decide from its own index ('judge'): the --    coins, the amounts, the lock time, the rules of relay that do not --    change, and the signature of every coin locked to one key. A---    transaction that the store has already is answered at once.+--    transaction that the store has in a block is answered at once. -- -- 2. __Send over RPC__, where the store was given the RPC of a full node --    ('Publishing'.@direct@). The node checks everything, says why it@@ -58,6 +58,17 @@ --    bytes posted again join the first request, or get its answer, and are --    not sent again. --+-- A transaction that the store has in its mempool, unconfirmed, is+-- checked and sent again in the same way, and answered from what the node+-- did with it: there is no echo to wait for, as the store has it already.+-- The store does not hold its mempool to its node's, so it may list a+-- transaction that the nodes forgot: after a reorganisation, or when a+-- node lost its mempool. Posting it again is how a wallet gets it back to+-- them. Answering at once, as for a confirmed one, left the wallet with no+-- way out but a fee bump, which Bitcoin Cash does not have, until the+-- store dropped the transaction weeks later. It costs a node a+-- transaction it may have already, which it ignores.+-- -- = Why it is this way -- -- * __A connection of its own.__ A node never announces a transaction to@@ -110,6 +121,7 @@     Answer (..),     Verdict (..),     Direct (..),+    Held (..),     publish,      -- * What the store can check@@ -241,6 +253,17 @@     Unverified !Int   deriving (Eq, Show) +-- | Whether the store has a transaction, and where.+data Held+  = -- | the store does not have it+    Absent+  | -- | in its mempool: the network may have forgotten it, so it is sent+    -- again+    Pending+  | -- | in a block: there is nothing to send+    Confirmed+  deriving (Eq, Show)+ -- | What came of giving a transaction to a full node over its RPC. data Direct   = -- | the node took it into its mempool, or had it@@ -557,9 +580,9 @@     -- | give the transaction to a full node over its RPC, where the store     -- was given the RPC of one     direct :: !(Tx -> IO Direct),-    -- | whether the store has the transaction, from the queries of the-    -- request-    has :: !(TxHash -> IO Bool),+    -- | whether the store has the transaction, and where, from the+    -- queries of the request+    holds :: !(TxHash -> IO Held),     -- | the coin of an output, from the queries of the request ('coinOf')     coin :: !(OutPoint -> IO Coin),     -- | whether the store has the transaction now, read anew: the queries@@ -578,11 +601,10 @@  -- | Publish a transaction, and say what came of it. See the module header. publish :: Publishing -> Tx -> IO Answer-publish p tx = do-  known <- p.has txid-  if known-    then return (Verdict Accepted)-    else+publish p tx =+  p.holds txid >>= \case+    Confirmed -> return (Verdict Accepted)+    held ->       (judge <$> p.rules <*> pure tx <*> mapM (p.coin . (.outpoint)) tx.inputs) >>= \case         Left r -> return (Invalid r)         Right unverified -> do@@ -594,7 +616,10 @@             Known Silent -> Verdict <$> listening (return Silent)             Known v -> return (Verdict v)             Fresh -> do-              v <- listening (send unverified) `onException` done Unreachable+              let go+                    | held == Pending = again unverified+                    | otherwise = listening (send unverified)+              v <- go `onException` done Unreachable               done v               say v               return (Verdict v)@@ -628,6 +653,12 @@                       -- an event may have been missed, never a transaction                       Nothing -> (\has -> if has then Accepted else Silent) <$> p.present txid               v -> return v+    -- The store has it in its mempool, so no echo will come: what the node+    -- took is accepted.+    again unverified =+      send unverified >>= \case+        Silent -> return Accepted+        v -> return v     echo sub =       receive sub >>= \case         StoreMempoolNew h | h == txid -> return ()

@@ -11,7 +11,7 @@   - url: './'  info:-  version: &version 1.12.0+  version: &version 1.12.1   title: Haskoin   contact:     name: JP Rupp@@ -1972,8 +1972,12 @@         this server. That takes milliseconds when the node whitelists the server,         and seconds otherwise. -        A transaction that this server has already is answered 200 and not sent.-        Posting the same transaction again is always safe: it is sent once.+        A transaction that this server has in a block is answered 200 and not+        sent. One that it has unconfirmed is checked and handed to a full node+        again, and answered 200 when the node took it: the nodes may have+        forgotten a transaction that this server lists, after a reorganisation+        or when a node lost its mempool, and posting it again gets it back to+        them. Posting the same transaction again is always safe.       operationId: postTransaction       requestBody:         description: >-

@@ -213,6 +213,7 @@   it "serves HTTPS, and nothing else, when given a certificate" tlsTest   it "tells each client of the transactions it chose, and takes up where it left" eventsTest   it "drops a transaction double-spent in a reorganisation" reorgTest+  it "sends again a payment that the nodes forgot and the store lists" resendTest   it "applies each change once with two stores sharing Redis" onceTest   it "does not replace what it does not know" unknownTest   it "leaves the keys of older caches alone" oldKeysTest@@ -1290,8 +1291,36 @@ reorgTest :: Env -> IO () reorgTest env = do   let x = newXPub env "reorg" DeriveNormal-      k = newXPub env "coinbase" DeriveNormal   ref <- newLedger+  (coinbaseSpend, cbValue) <- youngCoinbase env (newXPub env "coinbase" DeriveNormal)+  txA <- coinbaseSpend (at env x [0, 0]) (cbValue - 100000)+  record env x ref txA+  [blockX] <- mine env 1+  checkXPub env x ref+  header <- rpc env.node "getblockheader" [String blockX]+  parent <- maybe (fail "No parent") return (textField "previousblockhash" header)+  void $ rpc env.node "invalidateblock" [String parent]+  pool <- rawMempool env.node+  (txA `elem` pool) `shouldBe` False+  -- mined to the address of the invalidated parent, in the same second and+  -- without transactions, the block would be the parent again, which the+  -- node refuses as known to be invalid+  other <- getNewAddress env.node+  void $ generateTo env.node 1 other+  txB <- coinbaseSpend (at env x [0, 1]) (cbValue - 200000)+  void $ mine env 2+  forget ref txA+  record env x ref txB+  e <- expect <$> readIORef ref+  e.txids `shouldBe` [txB]+  checkXPub env x ref++-- | A coinbase that may be spent in the next block and not before, and a+-- way to spend it to an address. A spend in that block is dropped by the+-- node when the parent of the block is invalidated, as the coinbase is+-- too young again.+youngCoinbase :: Env -> XPub -> IO (Text -> Word64 -> IO Text, Word64)+youngCoinbase env k = do   void $ mine env 1   [cbBlock] <- generateTo env.node 1 (at env k [0, 0])   block <- rpc env.node "getblock" [String cbBlock]@@ -1317,27 +1346,35 @@           ]   -- 99 blocks on top: the coinbase may be spent in the next block   void $ mine env 99+  return (coinbaseSpend, cbValue)++-- | A payment that the nodes forgot while the store still lists it is sent+-- again when it is posted again. The payment spends a coinbase in the+-- first block where it may; invalidating the parent of that block makes+-- the coinbase too young, and the node drops the payment. The store keeps+-- the longer chain until the new one is longer, and then has the payment+-- unconfirmed, as the coinbase may be spent again.+resendTest :: Env -> IO ()+resendTest env = do+  let x = newXPub env "resend" DeriveNormal+  (coinbaseSpend, cbValue) <- youngCoinbase env (newXPub env "resend-coinbase" DeriveNormal)   txA <- coinbaseSpend (at env x [0, 0]) (cbValue - 100000)-  record env x ref txA   [blockX] <- mine env 1-  checkXPub env x ref   header <- rpc env.node "getblockheader" [String blockX]   parent <- maybe (fail "No parent") return (textField "previousblockhash" header)   void $ rpc env.node "invalidateblock" [String parent]-  pool <- rawMempool env.node-  (txA `elem` pool) `shouldBe` False-  -- mined to the address of the invalidated parent, in the same second and-  -- without transactions, the block would be the parent again, which the-  -- node refuses as known to be invalid+  -- not to the address of the invalidated parent, as in 'reorgTest'   other <- getNewAddress env.node   void $ generateTo env.node 1 other-  txB <- coinbaseSpend (at env x [0, 1]) (cbValue - 200000)   void $ mine env 2-  forget ref txA-  record env x ref txB-  e <- expect <$> readIORef ref-  e.txids `shouldBe` [txB]-  checkXPub env x ref+  (txA `elem`) <$> rawMempool env.node `shouldReturn` False+  (code, v) <- getJSON env.plain ("/transaction/" <> cs txA)+  code `shouldBe` 200+  (field "block" v >>= field "mempool") `shouldSatisfy` isJust+  (_, raw) <- getJSON env.plain ("/transaction/" <> cs txA <> "/raw")+  tx <- maybe (fail "No raw transaction") (either fail return . decodeTxBody env.net . cs) (textField "result" raw)+  publishTx env tx >>= \(c, r) -> (c, textField "txid" r) `shouldBe` (200, Just txA)+  (txA `elem`) <$> rawMempool env.node `shouldReturn` True  ------------------------------------------------------------------------------- -- Cache

@@ -79,7 +79,7 @@               outbox = outbox,               rules = return (rulesOf),               direct = \_ -> return Unavailable,-              has = \h -> elem h <$> readTVarIO here,+              holds = \h -> (\hs -> if elem h hs then Pending else Absent) <$> readTVarIO here,               coin = \_ -> return (held 1000),               present = \h -> elem h <$> readTVarIO here,               events = events,@@ -172,12 +172,25 @@         echo p.events         publish p tx `shouldReturn` Verdict Accepted         sent `shouldReturn` 1-    it "sends nothing for what the store has, or cannot be valid" $+    it "sends nothing for what the store has in a block, or cannot be valid" $       desk ctx (\_ _ -> return Took) $ \p sent -> do-        publish p {has = \_ -> return True} tx `shouldReturn` Verdict Accepted+        publish p {holds = \_ -> return Confirmed} tx `shouldReturn` Verdict Accepted         publish p {coin = \_ -> return Missing} tx           `shouldReturn` Invalid (UnknownOutput (output 1))         sent `shouldReturn` 0+    it "sends again what the store has unconfirmed, each time, and waits for no echo" $+      desk ctx (\_ _ -> return Took) $ \p sent -> do+        let pending = p {holds = \_ -> return Pending}+        replicateM_ 2 $ publish pending tx `shouldReturn` Verdict Accepted+        sent `shouldReturn` 2+        -- checked first, like any other: a coinbase too young is not sent+        publish pending {coin = \_ -> return (Coin {output = TxOut 1000 "", mined = Just 950, coinbase = True})} tx+          `shouldReturn` Invalid (Immature (output 1))+        sent `shouldReturn` 2+    it "says why a node refuses what the store has unconfirmed" $+      desk ctx (\_ _ -> return (Rejected "missing inputs")) $ \p sent -> do+        publish p {holds = \_ -> return Pending} tx `shouldReturn` Verdict (Refused "missing inputs")+        sent `shouldReturn` 1     it "gives the reason of a node, again without asking it again" $       desk ctx (\_ _ -> return (Rejected "too cheap")) $ \p sent -> do         replicateM_ 2 $