haskoin-store 1.8.1 → 1.9.0

raw patch · 76 files changed

+37368/−8170 lines, 76 filesdep +case-insensitivedep +file-embeddep +processdep −scottydep ~QuickCheckdep ~aesondep ~aeson-prettysetup-changedPVP ok

version bump matches the API change (PVP)

Dependencies added: case-insensitive, file-embed, process, wai-extra, warp-tls

Dependencies removed: scotty

Dependency ranges changed: QuickCheck, aeson, aeson-pretty, base, base16, base64, bytes, bytestring, cereal, conduit, containers, data-default, deepseq, fast-logger, filepath, hashable, hashtables, haskoin-core, haskoin-node, haskoin-store-data, hedis, hspec, http-types, lens, monad-control, mtl, network, nqe, optparse-applicative, random, rocksdb-haskell-jprupp, rocksdb-query, statsd-rupp, stm, string-conversions, text, time, transformers, unliftio, unordered-containers, vault, vector, wai, wai-websockets, warp, websockets, wreq

API changes (from Hackage documentation)

Files

• CHANGELOG.md +2828/−0
• README.md +936/−0
• Setup.hs +0/−2
• app/Main.hs +147/−98
• bench/Main.hs +20/−20
• haskoin-store.cabal +418/−184
• integration/Haskoin/Store/Integration/Image.hs +196/−0
• integration/Haskoin/Store/Integration/Node.hs +237/−0
• integration/Haskoin/Store/Integration/Podman.hs +190/−0
• integration/Haskoin/Store/Integration/Valkey.hs +57/−0
• src/Haskoin/Store.hs +42/−25
• src/Haskoin/Store/BlockStore.hs +1217/−1498
• src/Haskoin/Store/BlockStore/Pending.hs +166/−0
• src/Haskoin/Store/BlockStore/Sync.hs +178/−0
• src/Haskoin/Store/Cache.hs +1534/−1761
• src/Haskoin/Store/Cache/Protocol.hs +122/−0
• src/Haskoin/Store/Cache/lua/bodies.lua +25/−0
• src/Haskoin/Store/Cache/lua/commit.lua +92/−0
• src/Haskoin/Store/Cache/lua/delete.lua +18/−0
• src/Haskoin/Store/Cache/lua/evict.lua +23/−0
• src/Haskoin/Store/Cache/lua/page.lua +39/−0
• src/Haskoin/Store/Cache/lua/spends.lua +20/−0
• src/Haskoin/Store/Cache/lua/unlock.lua +5/−0
• src/Haskoin/Store/Common.hs +266/−217
• src/Haskoin/Store/Database/Mempool.hs +505/−0
• src/Haskoin/Store/Database/Migrate.hs +678/−0
• src/Haskoin/Store/Database/Query.hs +696/−0
• src/Haskoin/Store/Database/Query/Layered.hs +807/−0
• src/Haskoin/Store/Database/Query/Pooled.hs +518/−0
• src/Haskoin/Store/Database/Reader.hs +173/−504
• src/Haskoin/Store/Database/Types.hs +412/−55
• src/Haskoin/Store/Database/Views.hs +136/−0
• src/Haskoin/Store/Database/Writer.hs +384/−153
• src/Haskoin/Store/Fleet.hs +201/−0
• src/Haskoin/Store/Fleet/Member.hs +1078/−0
• src/Haskoin/Store/Fleet/Pool.hs +320/−0
• src/Haskoin/Store/Fleet/Redis.hs +400/−0
• src/Haskoin/Store/Fleet/XPub.hs +642/−0
• src/Haskoin/Store/Fleet/lua/move.lua +74/−0
• src/Haskoin/Store/Fleet/lua/pool.lua +477/−0
• src/Haskoin/Store/Fleet/lua/read.lua +172/−0
• src/Haskoin/Store/Fleet/lua/report.lua +54/−0
• src/Haskoin/Store/Fleet/lua/xpub.lua +401/−0
• src/Haskoin/Store/Logic.hs +313/−275
• src/Haskoin/Store/Logic/Layered.hs +792/−0
• src/Haskoin/Store/Manager.hs +211/−154
• src/Haskoin/Store/Web.hs +3566/−3076
• src/Haskoin/Store/Web/Docs.hs +88/−0
• src/Haskoin/Store/Web/Events.hs +380/−0
• src/Haskoin/Store/Web/Router.hs +231/−0
• stress/Main.hs +913/−0
• stress/wrk.lua +32/−0
• swagger.yaml +8042/−0
• test/Haskoin/Store/BlockStore/PendingSpec.hs +94/−0
• test/Haskoin/Store/BlockStore/SyncSpec.hs +129/−0
• test/Haskoin/Store/BlockStoreSpec.hs +0/−16
• test/Haskoin/Store/Cache/ProtocolSpec.hs +158/−0
• test/Haskoin/Store/CacheSpec.hs +30/−8
• test/Haskoin/Store/CommonSpec.hs +154/−7
• test/Haskoin/Store/Database/MigrateSpec.hs +490/−0
• test/Haskoin/Store/Database/Query/LayeredSpec.hs +173/−0
• test/Haskoin/Store/Database/ViewsSpec.hs +67/−0
• test/Haskoin/Store/Fleet/MemberSpec.hs +312/−0
• test/Haskoin/Store/Fleet/PoolSpec.hs +182/−0
• test/Haskoin/Store/Fleet/RedisSpec.hs +423/−0
• test/Haskoin/Store/Fleet/XPubSpec.hs +77/−0
• test/Haskoin/Store/FleetSpec.hs +199/−0
• test/Haskoin/Store/Histories.hs +783/−0
• test/Haskoin/Store/IntegrationSpec.hs +1958/−0
• test/Haskoin/Store/Logic/LayeredSpec.hs +264/−0
• test/Haskoin/Store/LogicSpec.hs +95/−79
• test/Haskoin/Store/TestBlocks.hs +0/−4
• test/Haskoin/Store/Web/EventsSpec.hs +191/−0
• test/Haskoin/Store/Web/RouterSpec.hs +75/−0
• test/Haskoin/Store/WebSpec.hs +20/−5
• test/Haskoin/StoreSpec.hs +22/−29

@@ -0,0 +1,2828 @@+# Changelog+All notable changes to this project will be documented in this file.++The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)+and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html)++## [Unreleased]++## [1.9.0] - 2026-11-10++### Added++- `--fleet` (`FLEET`): servers that share a Redis answer as one. Every+  request is answered from the newest block that all live servers have and+  from one mempool that they share in Redis, so that the same request gets+  the same answer whichever server a load balancer picks, also while a+  block or a transaction is on its way through them.+  - Each server keeps its database as it was at each of its last six+    blocks, reports them under a lease of ten seconds, and offers its+    unconfirmed transactions to the shared pool. The fleet tip moves in one+    step, together with what the blocks in between did to the pool.+  - A server that does not have the block that the fleet answers from says+    503 with `behind-fleet`. One whose health check fails stops reporting,+    and stops counting when its lease runs out.+  - Without Redis a server answers from its own database and `/health` says+    203.+  - `Haskoin.Store.Manager` runs the member (`Store.fleet`), the block+    store takes a view of the database right after each block+    (`BlockStoreConfig.onBest`), and the web server answers through+    `Haskoin.Store.Database.Query.Pooled`.+  - An integration test runs two servers as a fleet against a real node and+    Redis and compares their answers with a payment in the pool and after+    the block that confirms it.+  - `docs/layered-database/FleetCost.hs` times a store alone against the+    same store in a fleet, with 40,000 transactions in the pool: a fleet+    adds about one round trip to Redis to a request. A request is answered+    from the tip that the member last read, which its first read of the+    pool checks; a transaction's body comes with what spends its outputs;+    and a first page is as large as the page asked for.++- A cache of xpubs for a fleet (`Haskoin.Store.Fleet.XPub`), under the+  options of the cache of a server alone: `--redis-min-addrs`,+  `--redis-min-txs` and `--redis-max-xpubs`. Until now `--fleet` answered+  every request about an xpub from the database, with a read of the shared+  pool for each address that it touches.+  - An entry has the chain's part of a wallet at one block, which is the+    same for the whole cache, and the part of the shared pool, which the+    script of the pool keeps as it changes the pool. Transactions and+    outputs of both are in one sorted set each, in the order of the API.+  - The cache answers only while its block is the fleet tip. The server+    that moves the tip brings the cache along, with what the new blocks+    did at the addresses that entries have. If it does not, or after a+    reorganisation, the cache is emptied.+  - The cache is at a move of the fleet tip, which the pool counts, and+    not just at a block: the tip can leave a block and come back to it.+  - Every read of balances runs the scan of the xpub over the entry and+    misses unless it reads exactly the addresses of the entry, so a+    payment in the pool to an unused address cannot leave the gap wrong.+  - Entries are built in the background by the server that missed, and+    extended when an xpub gains addresses.+  - A request about a wallet of 464 addresses and 19,499 transactions+    takes 1.3 to 2.2 milliseconds through it, where a fleet took 5 to 38;+    for a wallet with 400 addresses in the pool, 3 to 4 where it took 5 to+    126 (`docs/layered-database/fleet-cost.txt`).+  - The test of the member compares every answer about a wallet through+    the cache with the uncached one after each step of random histories,+    and the two servers of the integration test do the same over HTTP.+  - StatsD metrics under `fleet.cache.*`: hits, misses, builds and their+    time, builds given up, evictions, and the times that the cache was+    emptied. The log says why the cache was emptied each time, and with+    `--debug` has a line for each entry built and each block that the+    cache was brought to.+- Built with `rocksdb-haskell-jprupp` 2.7.1, and every scan by address+  uses an iterator that RocksDB stops where the address ends, so that a+  scan cannot walk over deleted keys that follow it. The migration reads+  the transactions with an iterator that is asked for the order of them+  all.+- Built with `haskoin-node` 2.5.0, which has the fixes of its security+  review: among them, a block from a peer is checked against its header+  before the store sees it, so a peer can no longer pass off transactions+  of its choice under a real header. It also allows the new binding. On+  regtest its discovery no longer finds a node on this machine: name the+  node with `--peer`.+- **`/events` agrees across a fleet.** The servers of a fleet send the+  events of the block and the mempool that they share, read from a stream+  in Redis that the pool's script writes as it makes each change, and no+  longer those of their own nodes. A block is told when the fleet has+  moved to it, with every block that it moved through.+- **`/events` takes up where a client left.** `?after=` with the hash of+  the last event heard starts the stream after it, over HTTP and over the+  WebSocket. The server keeps its last 10,000 events in one log that all+  clients read (`Haskoin.Store.Web.Events`), in place of a queue for each.+  It answers 404 when it does not have the event. The servers of a fleet+  keep the same events, so a client may come back to another one.+- **A WebSocket client chooses the transactions it hears about**: by+  address, by extended public key with a type of address, or with a Bloom+  filter as in BIP 37, with the commands of `EventCommand` in+  `haskoin-store-data` 1.11. Blocks are always sent, and a client that+  sends no command hears everything, as before. A key is watched up to its+  gap, which moves on with its payments.+- **The choice can be in the URL of `/events`**, so that it holds from+  the first event: `quiet=true` for blocks alone, `addresses`, `xpub` with+  `derive` (one type or several), and `filter` with `functions`, `tweak`+  and `flags`. The HTTP stream takes them too, also under `POST` with a+  form. `{"type":"none"}` goes back to blocks alone.+- **The WebSocket speaks binary too**: binary frames when the handshake+  asks for binary or the client sends a binary frame, for events and+  commands alike.+- **HTTPS**, optional and off by default: `--tls-cert` and `--tls-key`+  (`TLS_CERT`, `TLS_KEY`) make the server speak HTTPS on its port.+- **The reference of the API at the root of the server.** `/` is a page+  that shows `swagger.yaml` as it was when the server was built, and can+  try requests against it; `/swagger.yaml` is the file.+- **`HEAD` and `OPTIONS`** are answered: the headers of what `GET` gives,+  and what a browser may send from another site.+- **Answers are compressed** with gzip for clients that accept it, but for+  the stream of events.+- **Lists in a body.** The endpoints that take a list of addresses,+  blocks, heights or transactions accept `POST` with the parameters in a+  form, so that a long list need not fit in a URL.+- **`Retry-After`** on every 503.+- **Errors in binary.** A request that asks for the binary format gets+  its errors in that format too, the 503 of a server behind its fleet+  included. They were JSON whatever was asked. This needs+  `haskoin-store-data` 1.10, whose client reads both.+- **The `Accept` header is read as a list with weights.** Binary is sent+  when `application/octet-stream` is listed with a weight above 0 that is+  no less than that of `application/json`. The header used to be compared+  with that one string, so a list or a weight silently gave JSON. Every+  answer has `Vary: Accept`.+- **The raw lists start with a count**, in binary: `/transactions/raw`,+  `/block/{block}/transactions/raw` and `/block/height/{height}/raw` were+  the one kind of list without it. A binary client of those three+  endpoints has to change with the server.+- `haskoin-store --help` names the environment variable of every option.+- `swagger.yaml` documents the binary format: how to ask for it, the+  encoding of each type that the API returns, and what differs from JSON.+- Gauges `fleet.cache.xpubs` and `fleet.cache.addresses`: how many xpubs+  the cache of a fleet holds, and how many addresses between them.++### Changed++- A WebSocket client of `/events` that sends something the server cannot+  read as a command is disconnected, with code 1008 and the reason. What+  clients sent used to be ignored.+- The HTTP stream of `/events` no longer holds a view of the database for+  as long as its client listens.++- Since version 2.6.4 the binding reads the key and the value of a row of+  an iterator without the cost of a safe foreign call.+  Every list and every count takes between a half and a third of the time:+  the first 100 transactions of a busy address 221 microseconds instead of+  575, and the number of transactions of a wallet with 19,499 of them 34+  milliseconds instead of 78.+- **Bloom filters are on by default**, as was always meant: until the+  binding was fixed, `--bloom` built none. `--no-bloom`, or `BLOOM=false`,+  turns them off; `--bloom` is still accepted. An existing database gets+  filters, and the memory that they take in the block cache, as its table+  files are written again.+- **Transactions that leave the mempool unconfirmed are no longer kept.**+  The API already answered 404 for them, but their bodies stayed in the+  database, marked as deleted, for ever, and each server had its own.+  - A server remembers the last hundred thousand that it deleted, in+    memory, so as not to fetch them again from its peers. The fleet's rule+    for conflicting transactions and the cache of a server alone, which+    both asked the database whether a transaction had been deleted, ask+    that memory instead.+  - The cache of a server alone takes a transaction that Redis has as+    unconfirmed for more than ten minutes, and that the database does not+    have, for deleted and not for one that it has not seen yet.+  - Bodies that version 20 marked as deleted stay in a migrated database+    and are ignored. A page that starts at a deleted transaction is empty.+- Removed reports that no longer describe the store: five intermediate+  stress runs, the measurement of a mempool mirrored in memory, and the+  superseded runs in `docs/layered-database/query-cost.txt`.+  `docs/layered-database.md` is rewritten as a description of what+  exists, a quarter of its length.+- **Database version 22: the record of a confirmed transaction is written+  once**, by the block that confirms it, and is the same on every server+  that has the block, so that what the chain's column families hold+  follows from the chain alone.+  - **Its time is the time of its block.** A server that had seen a+    transaction in its mempool used to keep the time of that when a block+    confirmed it: two servers then gave two values of `time` for the same+    confirmed transaction, which the servers of a fleet must not do. A+    transaction that returns to the mempool when its block leaves the main+    chain has the time of that block there.+  - **The spenders of its outputs are under keys of their own**, in the+    `spender` column family, sixteen outputs to a key (`SpenderKey`).+    They were in the record, which each block that spent one of its+    outputs read and wrote whole again. A spend now reads and writes a+    value of a few dozen bytes and leaves the record alone.+  - Reading a confirmed transaction in full takes a second key: 18+    microseconds where it took 11.5, and 0.8 milliseconds for a hundred+    where it took 0.4. Lists by address and by xpub cost the same, and so+    does writing, on the small records of the measurements.+  - A database of version 21 is migrated the first time the store opens+    it (`migrateRecords`). **It reads every transaction of the database+    and writes nearly every one again**, since nothing lists those that+    need it and most have a spender in them: many hours for a database of+    Bitcoin, and room for a second copy of its transactions until RocksDB+    has compacted the first away. The log says how far it is, and it can+    be interrupted and started again.+  - The tests that run the store through random histories now choose them+    from a database of version 20 that goes through the same+    (`withShadow`). Given a layered database, the chooser found next to+    nothing in the mempool, and the blocks of the tests of the fleet+    confirmed almost none of it.+- **Database version 21: the mempool has an area of its own.** Unconfirmed+  transactions are no longer written into the indices of the chain. They+  are kept apart and applied on top when a query is answered+  (`docs/layered-database.md`), so the chain's data changes only when a+  block is imported or reverted.+  - A database of version 20 is migrated the first time the store opens it,+    and the log says how many mempool transactions were moved. **Older+    stores refuse the database afterwards**: copy a production database+    before upgrading.+  - The block store imports with the rules of `Haskoin.Store.Logic.Layered`.+    A block that spends an output of a transaction that is only in the+    mempool is refused as an orphan; no valid block does that.+  - Each request of the web server reads one snapshot of the database for+    everything it answers, so the parts of a response agree with each+    other. `routerScoped` in `Haskoin.Store.Web.Router` is what gives a+    request its snapshot.+  - `--wipe-mempool` deletes the keys of the mempool area, in less time+    than it took to delete each transaction by the rules, and it no longer+    keeps the wiped transactions as deleted ones.+  - `StoreRead` has a new query, `getAddressesTxCount`.+  - Programs that embed the store and read its database with+    `Haskoin.Store.Database.Query` have to move to+    `Haskoin.Store.Database.Query.Layered`: the old queries do not see the+    mempool of a database of version 21.+- Xpub addresses are derived up to an exact gap: every address up to the last+  used one of each chain, then exactly `--xpub-gap` unused ones, and always+  the first `--xpub-gap-init` addresses. Addresses used to be read 32 at a+  time with the gap checked between batches, so the store looked between 33+  and 64 addresses past the last used one, and started counting again at index+  20. This is why api.haskoin.com saw a payment 34 addresses past the last+  used one, which wallets with a gap of 20 or 32 do not see. Payments that far+  away are no longer seen; raise `--xpub-gap` to see them.+- New Redis cache of xpubs (`Haskoin.Store.Cache`), replacing the old one:+  - Whenever a transaction touches a cached xpub, or a reorganisation changes+    one of its transactions, the store reads the whole xpub again from its+    database and replaces the entry in one Lua script, so cached answers are+    always answers of the database.+  - Everything a response needs is in Redis, transaction bodies included,+    shared between xpubs, with the spenders of their outputs kept up to date.+    A store that has not imported a transaction yet still serves it.+  - Several stores can share one Redis. Each change of a cached xpub is+    claimed and applied once, by the first store to see it; stores that catch+    up find their changes done and do no work. A store never replaces an entry+    that knows more than its database (a higher block, or a transaction it has+    not seen), so stores do not undo each other's work, and every entry is+    written together with the bodies it needs, so a store that is behind can+    serve it in full.+  - The rules for claims and for replacing entries are pure decisions+    (`Haskoin.Store.Cache.Protocol`). A model test runs several stores+    catching up with a chain in random interleavings against a model of+    Redis, and checks that they end with the entry of the tip, that no+    store replaces an entry with one that knows less, that each change is+    written once, and that stores catching up after the first write nothing.+  - A reload writes only the differences between the xpub in the database+    and its entry in Redis, read in chunks first, instead of rewriting the+    whole entry; its script checks that the entry did not change since.+    Rewriting an xpub of 15,000 transactions held Redis for up to 77 ms,+    during which it answered no other client. A store reloads up to four+    xpubs at once.+  - A page of the transactions or unspent outputs of a cached xpub is one+    round trip to Redis: a Lua script finds where it starts, from the+    position of the start transaction in the xpub itself, and reads it.+    It took four or five, and read the whole body of the start+    transaction to find its position.+  - A store marks a cached xpub as used at most every ten seconds, for+    eviction, instead of on every read, which made every read a write.+  - `--redis-min-txs` (`REDIS_MIN_TXS`, default 1000) also caches xpubs+    with at least that many transactions in their addresses, however few+    addresses they have.+  - Keys start with `hs2:<network>:`, so the cache can share a Redis with the+    cache of older versions and with stores of other networks.+  - `--redis-max-xpubs` (`REDIS_MAX_XPUBS`, default 10000) replaces+    `--redis-max-keys`: the least recently used xpubs past this many are+    evicted. `--redis-sync-interval` is removed.+  - StatsD metrics: `cache.hits`, `cache.misses`, `cache.reloads`,+    `cache.stale`, `cache.evictions` and `cache.reload` (milliseconds).+- `n_tx` of `/blockchain/multiaddr` counts every transaction of the shown+  addresses and xpubs, not only those on the page, and `n_tx_filtered` those+  that pass the filter. `n_tx` of `/blockchain/rawaddr` for an xpub is its+  number of transactions, not the size of the page.+- The web server runs directly on WAI and Warp, without Scotty. Routes,+  parameters and passing a request to the next route are in+  `Haskoin.Store.Web.Router`, a small router over WAI; a handler is a plain+  function from the state of the server and the request to a response in+  `IO`, and stops early by throwing an `Except`, which becomes the JSON+  error. Every response carries `Access-Control-Allow-Origin: *`, error+  responses included. The endpoints, their parameters and their responses+  are as before.+- No monad transformers, except conduits, and no functions polymorphic over+  `MonadIO` or `MonadUnliftIO`: everything runs in plain `IO`. Database queries are+  functions that take the `DatabaseReader` (`Haskoin.Store.Database.Query`);+  imports take a `Writer` handle (`Haskoin.Store.Database.Writer`,+  `Haskoin.Store.Logic`); the block store and the manager pass their state+  explicitly. The classes `StoreReadBase` and `StoreReadExtra` are replaced+  by `StoreRead`, a record of the same queries as functions in `IO`, which+  `databaseRead` gives for a database; `getTransaction` and the other+  helpers of `Haskoin.Store.Common` take it as their first argument. The state+  of the web server, which every handler takes as an argument instead of+  reading it from a `ReaderT`, holds the queries to answer with, the cache+  in front of those about xpubs;+  `nocache=true` gives a handler a state whose queries read only the+  database. `withStore`, `runWeb`, `runWebSocket` and the `blockStore*`+  functions are in `IO`. `WriterT`, `ImportM`, the `StoreWrite` class,+  `BlockT`, `CacheT`, `DatabaseReaderT` and `withDB` were removed earlier+  in this release. This changes the library API.+- The package uses the GHC2024 edition, with `DuplicateRecordFields`,+  `NoFieldSelectors`, `OverloadedRecordDot` and `OverloadedStrings` on for+  every module, and explicit deriving strategies.++- haskoin-store-data moves to its own repository,+  [jprupp/haskoin-store-data](https://github.com/jprupp/haskoin-store-data),+  with its history and its own releases, and haskoin-store moves to the root+  of this one; `store/` and `data/` are gone. haskoin-store depends on+  `haskoin-store-data ^>= 1.9`. The package has a full description, PVP+  upper bounds on every dependency, and no `Setup.hs`.+- Legacy addresses are called `legacy` in the `derive` parameter, the default.+  `normal`, the name they had, is still accepted, so old URLs keep working;+  it said nothing about the kind of address. Error messages use the new+  name. Built with haskoin-store-data at commit `b304acd2`.+- The executable gives memory back to the system as soon as its heap no+  longer needs it (`--disable-delayed-os-memory-return` in its default RTS+  options). By default GHC marks such memory free for the kernel to take when+  it runs short, and Linux counts it as resident until then, so the heap of a+  busy store seemed to grow without end, by the size of each peak, with little+  live data. The stress test records the resident GHC heap and how much of it+  is lazily freed. The heap profile and the runs that found this are in+  `stress/results/2026-11-09-memory`, and `scripts/heap-profile-summary.py`+  summarises a heap profile.+- The cache brings an entry up to date from what changed, instead of reading+  the whole xpub from RocksDB and Redis on every change, which the stress test+  showed was almost all the time of a reload, growing to seconds at 100,000+  transactions per xpub. A reload reads the changed transactions, the outputs+  they create and spend at addresses of the xpub, and the balances of its+  addresses, which are compared with Redis's to catch a change the entry+  missed. The whole xpub is still read when it is first cached, when the+  entry is more than a block behind, when an address with transactions enters+  or leaves the gap, when a balance does not match, and every ten minutes.+  Entries keep an index of their unspent outputs by outpoint+  (`x:<id>:uop`), in format version 3; entries of version 2 are upgraded by+  their next reload. New metric `cache.full_reloads`.+- A reload whose balances show changes it was not told of first looks for+  them, instead of reading the whole xpub: the transactions that the database+  has for the addresses whose balances moved, in the mempool or in blocks+  after the one the entry was written at, and the mempool transactions of the+  entry that have since been confirmed or dropped, up to 1,000. Under load the+  database is often ahead of the events being applied, which made about one+  reload in ten of the stress test a full one.+- Bodies that Redis already has, such as those of the transactions that a+  block confirms, are rewritten before the commit of a reload, 200 to a+  script, instead of in it. The commit of a block that confirms a thousand+  transactions of an xpub had about 8,000 arguments and held Redis for 10 to+  30 ms, the only commands of the stress test that took that long.+- The scheduled full reload of a cached xpub comes every five to ten+  minutes, an interval that depends on the xpub, instead of every ten for+  all. Xpubs cached together were read whole together each time: in the full+  stress test, four xpubs of tens of thousands of transactions raised the+  heap from under 100 MiB to 785 MiB for a few seconds every ten minutes.+- `docs/cache-consistency.md` analyses how a cached xpub can differ from the+  database without any check noticing, including spenders of cached+  transactions that no reload repairs, and what would prevent it.+- `docs/layered-database.md`: a design for keeping the mempool out of the+  chain's indices, in an area of its own that is applied on top when a query+  is answered, with the measurements behind it and what it means for the+  cache. The store still runs on the layout of version 20; this adds the+  parts that could be proved ahead of the switch:+  - `Haskoin.Store.Database.Types` has the keys of the mempool area+    (`MemSpenderKey`, `MemAddrKey`, `MemDeletedKey`) and a marker for+    migrations that work in several writes (`MigrationKey`).+  - `Haskoin.Store.Database.Mempool` has the keys that one mempool+    transaction contributes, and what the mempool does to the balance and+    the unspent outputs of an address.+  - `Haskoin.Store.Database.Migrate` moves a database of version 20 to the+    layered layout without a resync: it takes every mempool transaction out+    of the chain's indices and writes it to the mempool area, in atomic+    batches that can be interrupted. Nothing calls it yet.+  - `withLayeredReader` opens a database with the extra column family,+    `mempool-addr`, that the layout needs.+  - Tests build databases with the store's own rules from random histories+    of payments, blocks, replacements, deletions and reorganisations,+    migrate them, and check that every query answers as before, that the+    chain's column families are those of a database that never saw a+    mempool, and that emptying the mempool area leaves the chain's answers.+- `Haskoin.Store.Logic.Layered`: the rules that import and revert blocks+  and add, replace and delete mempool transactions in the layered layout of+  `docs/layered-database.md`, with the functions, results and errors of+  `Haskoin.Store.Logic`. Blocks are written to the chain's column families+  alone, and the mempool to its own area. The store does not use them yet.+  - `Haskoin.Store.Database.Writer` can write the mempool area in the same+    atomic batch as the chain; batches of today's rules write nothing there.+  - Tests give the same random history to both sets of rules, on Bitcoin and+    on Bitcoin Cash: payments, replacements, deletions, blocks in and out of+    order, reorganisations, and transactions and blocks that must be+    refused. Every operation must have the same result and move the same+    transactions in and out of the mempool, every query must answer the same+    after it, and at the end the database of today's rules, once migrated,+    must equal that of the new rules key for key.+  - One difference is on purpose: a block that spends an output of an+    unconfirmed transaction is refused by the new rules, as that output is+    not in the chain. Today it is imported.+- `Haskoin.Store.Database.Query.Layered`: every query of the store for the+  layered layout, with the names and arguments of+  `Haskoin.Store.Database.Query`, and `layeredRead`, which gives them as the+  record of queries that the web server answers with. The store does not use+  them yet.+  - A query takes a `View`, a RocksDB snapshot that only `withView` makes,+    so that the chain and the mempool are read as they were at one moment.+  - Pages are cut from the mempool's part of a list followed by the chain's+    part, by today's rules for starts, offsets and limits, for one address,+    for several, and for extended public keys.+  - Mempool spenders are looked up only for the outputs of a transaction+    that no block spends, in one multi-get for a whole page of transactions.+  - `MemDeletedKeyP` matches deleted transactions by the start of their+    hash, for the short transaction numbers of the Blockchain.info API.+  - Tests ask today's queries and the new ones for pages with random+    limits, offsets and starts, on databases migrated from today's layout,+    on databases built by the rules of the layered layout, and on databases+    upgraded in the middle of a history. They also check that an open view+    goes on answering from its moment while blocks and transactions are+    written.+- A summary for each address that the mempool touches, in the layered+  layout (`MemSummary`, under `MemAddrSumKey`): the number of its mempool+  transactions, and the amounts and numbers of the outputs that the mempool+  pays to it, spends, and both pays and spends. The measurements showed the+  queries of that layout to be too slow for an address with much in the+  mempool, and for any address right after a block, because they scanned+  everything that the mempool has for an address to answer anything.+  - A balance is now two reads by key, and an address with nothing in the+    mempool is found by a key that is not there.+  - Lists of transactions and of unspent outputs read the mempool's entries+    only for addresses that have some, and only as far as the page goes.+  - The rules keep the summary with every transaction that enters or leaves+    the mempool, and the migration writes all of them from the entries that+    it moved. The tests work every summary out again after each operation.+  - Measured again with `QueryCost.hs`: the balance of an address with+    6,216 entries in the mempool takes 11 microseconds instead of 35+    milliseconds, a page of its transactions what it takes in version 20,+    the balances of a wallet of 464 busy addresses 1.9 milliseconds instead+    of 110, and nothing gets slower right after a block. Keeping the+    summaries costs about 110 microseconds for each transaction that passes+    through the mempool. `docs/layered-database.md` has the tables.+- `docs/layered-database/QueryCost.hs` writes one synthetic history into a+  database of each layout and times the rules and the queries of both,+  checking that the answers agree. `docs/layered-database.md` has the+  results and what follows from them: writing costs the same when syncing+  and 14% more for a transaction that passes through the mempool; the new+  queries cost 10 to 15 microseconds more for a request when the mempool+  has nothing for what is asked; and they are too slow, as built, for+  addresses that the mempool touches a lot and for any address right after+  a block. The document proposes a summary for each address in the mempool+  area, read by key, and two changes to the RocksDB binding.+- Documentation: a page that starts at a height has no unconfirmed+  transactions, whatever the height. The comment on `AtBlock` said that a+  height above the best block includes them, and the API reference could be+  read that way. The behaviour is unchanged.+- The stress report breaks the commands that blocked Redis down by script,+  and commits by full and incremental reloads, with their numbers of+  arguments.+- `Haskoin.Store.Fleet`: the rules by which stores that share a Redis agree+  on one block to answer from, the fleet tip, as pure decisions with a model+  test. It is the first part of the design for a fleet that answers as one,+  in `docs/layered-database.md`. Nothing uses it yet.+  `Haskoin.Store.Fleet.Redis` carries the rules out against Redis, with a+  Lua script for each step that has to be one step, and a test that takes+  random sequences of steps through a real Redis and through the rules and+  compares them after each. `Haskoin.Store.Fleet.Pool` has the rules of the+  mempool that such stores share, as of the fleet tip: what may be offered,+  which of two conflicting transactions stays, what leaves and why, and how+  the pool follows the tip forwards and across a reorganisation. A model+  test keeps it valid on top of the tip through random offers, blocks and+  reversals. One more Lua script carries these rules out, moving the tip and+  changing the pool in one step, and is compared with them step by step in a+  real Redis like the others; the comparison found a case that the rules had+  left open, a transaction that comes back from an abandoned block and+  spends what the pool already spends, which now wins. The pool also keeps,+  for each address, what its transactions do there: the transactions and+  the outputs paid to it in the order of a list, the outputs of it that are+  spent, and their sums, so that a balance is one read and a list is read+  by the page, as in a store's own mempool area.+- `Haskoin.Store.Database.Query.Pooled`: the queries of the store with a+  mempool that is not its own on top of the chain, for the shared pool of a+  fleet. `read.lua` and `poolRead` read that pool from Redis, each read+  with a check that the fleet tip has not moved. The tests hold these+  queries to those of the layered layout with the store's own mempool as+  the pool. `fleetAdvance` can empty the pool for a store that is alone and+  does not know the way from the old tip.+- `Haskoin.Store.Fleet.Member`: what a store does as a member of a fleet.+  It keeps a view at each recent block, reports them, moves the fleet tip+  with what the blocks in between did to the pool, offers its mempool, and+  answers from the view at the tip with the pool on top. A test with a real+  Redis takes a store that is alone in its fleet through random histories+  and compares every answer with the store's own after each step. It found+  that a body kept in the pool went stale when a parent was confirmed or+  came back, because a transaction inherits from its unconfirmed parents+  whether it can be replaced by fee; the pool now keeps that apart from the+  body and corrects it as transactions come and go.+- `Haskoin.Store.Database.Views` keeps the database as it was at each recent+  block, a RocksDB snapshot for each, so that a store can answer from the+  block that its fleet agrees on and not only from its newest. Views that+  are no longer needed are let go, and one that a request is reading is+  released when the request is done. Nothing uses it yet.+- A store whose Redis does not answer goes on answering from its database,+  instead of failing every request that the cache would have served. A+  request waits at most two seconds for Redis; after that the store leaves+  Redis alone until a health check finds it again. Meanwhile `/health`+  answers 203, Non-Authoritative Information, with the header+  `Haskoin-Degraded: redis`, when every other check passes: a status in the+  200s, so that load balancers keep the store, and not 200, so that+  monitoring notices. The body of `/health` has a new check, `redis`, for a+  store that has a cache; it does not count towards `ok`. It comes with+  haskoin-store-data 1.9.0, which the store now takes from Hackage.+- Breaking: transactions that were deleted are no longer returned. A+  transaction that was replaced, dropped from the mempool, or was the+  coinbase of a block that left the main chain used to answer by id with+  `"deleted": true`; clients that did not look at that field took it for a+  pending payment. It now gives 404, is left out of lists of transactions+  fetched by id, and a page cannot start at it. The `deleted` field stays in+  responses and is always false. The bodies stay in the database.+- Breaking: parameters are checked. A request with a parameter that its+  endpoint does not accept, such as a misspelt `limt=10` or `pretty` on the+  Blockchain.info endpoints, which never honoured it, gives 400+  `Unknown parameter: …`; and a value that cannot be read, such as+  `notx=yes`, `limit=-1`, `filter=4`, `format=xml` or an unknown currency,+  gives 400 `Invalid value for parameter …`. Both used to be ignored, and+  the client got an answer to a question it did not ask. Requests are+  checked before anything is done. The Blockchain.info endpoints accept+  `format=json`, which their clients often send. Each route of the router+  declares the parameters it accepts, and reading an undeclared one is an+  error of the program, so the two cannot drift apart.+- An xpub asked about with the wrong `derive` gives 400, with a message that+  names the right kind, instead of an empty wallet: when the xpub has no+  transactions with the kind asked for, the store checks the first+  `--xpub-gap-init` addresses of the other kinds. This applies to the `/xpub`+  endpoints and to `/blockchain/rawaddr`. A wallet unused with every kind+  still gets an empty answer, and wallets in use cost nothing more.+- `swagger.yaml` explains the API, not only lists it: an introduction to its+  conventions (amounts, ordering, paging, formats, errors, unconfirmed+  transactions, reorganisations, xpubs, live events, Blockchain.info+  compatibility) and the reasons for them, and a description of every+  endpoint, parameter and field. It documents `/dbstats`, the WebSocket at+  `/events` and the `time` field of the health check, which were missing,+  and corrects the health check examples and the version.+- The README tutorial pages with `height`, the parameter the server reads;+  it said `start`, which the server ignores.+- Every line of code is shorter than 120 columns, `swagger.yaml` included,+  whose long hexadecimal and xpub examples are split into equal pieces that+  parse to the same values. `scripts/format` formats `integration/` and+  `stress/` too.+- Scripts live in files of their own: the Lua scripts of the cache in+  `src/Haskoin/Store/Cache/lua/`, embedded at compile time with+  `file-embed`; the script of wrk in `stress/wrk.lua`, one for every+  endpoint, given its paths on the command line; and the shell steps of CI+  in `scripts/check-local-deps.sh` and `scripts/install-system-libraries.sh`.+  The scripts do the same as before.++### Fixed++- `--bloom` builds Bloom filters. It built none when the store was built+  against RocksDB 6.24 or later, as the Docker image is, because+  `rocksdb-haskell-jprupp` passed the size of the filter to RocksDB with the+  wrong type. Nothing failed; a lookup of a key that is not there just went+  on reading data blocks to find that out. The store is now built with+  `rocksdb-haskell-jprupp` 2.6.3, which has the fix. A store that already+  runs with `--bloom` starts building filters when it is upgraded, for the+  table files written from then on: about 1.25 bytes for each key, held in+  the block cache. Size that on a full database, or turn the option off to+  keep things as they were.+- `--bloom` no longer asks for filters on the two indices by address, which+  are scanned and not read by key. With a filter, a scan of an address that+  has nothing steps over every key deleted since the last flush: 810+  microseconds after 5,000 mempool transactions were deleted, against 13+  without filters. No store was affected, because no filter was ever built;+  with the fixed binding they would have been. The option still builds+  filters for transactions, unspent outputs, blocks, balances and the+  mempool.+- `/blockchain/multiaddr` no longer reads every transaction of the wallet to+  count them. `n_tx` of one xpub is its count (a `ZCARD` when it is cached);+  for several xpubs and addresses, the database counts the distinct+  transactions of all their addresses from the keys of its index. The+  transactions of a page are read in pages of about the size asked for, and+  their bodies a page at a time, instead of one by one; `/blockchain/rawaddr`+  does the same. In the stress test, with xpubs of about 15,000+  transactions, the median time of a multiaddr request went from 3.2 seconds+  to 36 milliseconds, and it no longer grows with the wallet.+- Containers left by an integration test run that was killed are removed+  by the next run, as intended: listing them used a template that Podman+  refuses, so they were never found.+- `--peer HOST[:PORT]` works as documented. It was read as a Haskell+  string, so only `--peer '"HOST:PORT"'`, with the quotes, was accepted.+- The syncing peer is disconnected when it sends a block that was not+  requested, as the documentation of the block store said. The block was+  imported if the node knew its header. The state of the block download is+  now a pure state machine (`Haskoin.Store.BlockStore.Sync`) with property+  tests, which also found that a block listed twice in one request was+  outstanding twice.+- The block store no longer starts a thread for every transaction it+  requests, to forget the request twenty seconds later. Requested and+  received mempool transactions are one pure value+  (`Haskoin.Store.BlockStore.Pending`), with property tests, and old+  requests are forgotten on the clock tick of the block store.+- Build against haskoin-node 2.4.0, from the head of its+  `fix-header-sync-stall` branch. The block store reserves its syncing peer+  as an owner of its own (`reservePeer`, `releasePeer`), so no other part of+  the node can release it; header searches run with `chainRunHeaders`. The+  store stopped following the chain+  when a peer announced new blocks with an inventory message instead of+  headers, which peers do when many blocks are found at once; it stayed at+  the old height until the peer reconnected.+- Mempool transactions that entered the mempool in the same second could be+  listed twice in the transactions of an xpub or a multiaddr wallet, and+  their order differed from page to page. Every merge now orders them as the+  database and the cache store them (`txRefKey`, `unspentKey`).+- `nocache=true` on `/xpub/:xpub/transactions` and `/xpub/:xpub/unspent` no+  longer reads, or fills, the cache.+- `confirmations` of `/blockchain/unspent` no longer wraps around for an+  output in a block newer than the chain of the store.++### Added++- `scripts/stress-cache.sh`, a stress test of the Redis cache: a regtest+  chain, Valkey and wrk in Podman, a haskoin-store server, hundreds of+  thousands of transactions between a few wallets, and wrk on their xpubs+  all the while, with a report (`scripts/stress-report.py`) of reloads,+  lag, resources, the commands that blocked Redis (from its slow log) and+  latencies, and a check of the cached answers at the end. The Podman+  helpers of the integration tests are an internal library (`integration`)+  that the stress program shares, and the test image has wrk in a layer of+  its own.+- `stress/results/2026-11-09/`: the results of a full default stress run,+  kept for comparison with later runs.+- `runWebSocket` serves the API on a socket that already listens. The tests+  bind port 0 and pass the socket, so their web servers get free ports with+  no window in which another program could take them; the containers publish+  their ports on random ports of the host.+- Integration tests against regtest chains of Bitcoin Core and Bitcoin Cash+  Node, with Valkey, in Podman containers built from Ubuntu 26.04 and+  release tarballs checked against pinned hashes, one cached layer per+  download, so nothing is downloaded twice. They are skipped when Podman+  cannot run containers or `HASKOIN_STORE_NO_PODMAN` is set.++## [1.8.1] - 2026-11-07++### Changed++- Documentation: the README explains the purpose and design of the store, with+  installation steps, a full configuration reference, monitoring and a+  tutorial, and the Haddock of every module and function says why it works the+  way it does.+- Build against nqe 0.6.9, rocksdb-haskell-jprupp 2.6.1, rocksdb-query 0.6.1,+  statsd-rupp 0.5.0.2, secp256k1-haskell 2.0.1, haskoin-node 2.3.1 and+  Stackage nightly-2026-11-07.++## [1.8.0] - 2026-11-07++### Changed++- Index the mempool by the time each transaction entered it. `/mempool`,+  `/blockchain/unconfirmed-transactions` and the health check read only the+  page they return, newest first, instead of loading and sorting the whole+  mempool on every request. Data version 20: a version 18 or 19 database is+  migrated when it is opened, with no resync.+- The block store no longer reads the whole mempool to publish mempool+  events, to set the mempool gauge or to find expired transactions. Each+  import reports which transactions it added to the mempool or removed from+  it; pruning reads only the oldest entries of the index. Before, every+  block, every batch of new transactions and every clock tick (several per+  second) read the whole mempool at least once.+- Count the transactions of an xpub without building the list of all of them+  when the cache is not used.+- Keep the derived addresses of the 512 to 1024 most recently queried xpubs+  in memory. Every xpub query derived each address again before reading a+  balance. Addresses depend only on the xpub, so the cache never goes stale.+- Page the unspent outputs of an xpub by merging its addresses in order of+  block, stopping once the page is full, instead of decoding up to offset+  plus limit outputs of every address and sorting them all. The pages are+  the same.+- Count the transactions of an xpub by merging the raw keys of its+  addresses, without decoding keys or reading values.+- Open the iterators of a query over many addresses in one bracket instead+  of one bracket each. Every call into RocksDB walked the stack past one+  frame per address.+- Double the page, up to 1024 rows, when streaming a source by offset, as+  `/blockchain/unspent` does. The rows read and skipped grew with the square+  of the rows returned.+- Check that an xpub is cached in the same Redis round trip as the read of+  its balances, transaction count, or a page of its transactions or unspent+  outputs that does not start at a transaction.+- Serialize a transaction at most twice to compute its id, size and weight,+  and stop looking up its witnesses with a quadratic list index.+- Build on haskoin-core 1.6.1, which encodes CashAddr and Base58 addresses+  faster: about 10% less CPU per request on endpoints that return many+  addresses.+- The executable uses every core of the host by default (`-N` instead of+  `-N4`). Set `+RTS -N<n> -RTS` on the command line to choose a number.+- Read the balances of many addresses, and lists of transactions, with one+  RocksDB multi-get instead of one read per key. `StoreReadBase` has a new+  method, `getManyTxData`, which defaults to reading the transactions one by+  one; `getTransactions` reads many transactions with it.+- Give RocksDB a block cache of 1 GiB by default, shared by all column+  families, and keep their index and filter blocks in it at high priority,+  with those of level 0 pinned. Before, each column family had RocksDB's+  default cache of 32 MiB, and index and filter blocks stayed in memory+  outside any cache for every open file. Set the size with `--block-cache-mb`+  or `BLOCK_CACHE_MB`; 0 restores the RocksDB defaults. `StoreConfig` and+  `withDatabaseReader` take the size in bytes. Requires rocksdb-haskell-jprupp+  2.6.0.++### Added++- `store/bench`: a benchmark server that syncs a synthetic regtest chain and+  serves the web API on it, and times the encoding of a response with the+  argument `micro`. `scripts/bench-web.sh` runs wrk against it and reports+  the CPU time of the server per request.++## [1.7.0] - 2026-11-06++### Changed++- Build on haskoin-core 1.6.0 and haskoin-node 2.3.0, the releases with the+  fixes of the November 2026 security review. Their changelogs list the+  changes to message parsing, header validation and peer handling that the+  store now runs with.+- An output has an address only when the network defines one for its script,+  as in haskoin-core 1.6.0. A SegWit-shaped output has no address on Bitcoin+  Cash. A 256-bit script hash output (`OP_HASH256`, a push of 32 bytes,+  `OP_EQUAL`) has none on Bitcoin. Such an output is no longer indexed under+  an address, and its `address` is `null`. Entries that an older version+  wrote under such an address stay in the database. No request can name them,+  since the address has no text form on the network.+- `POST /transactions` takes a body that holds one transaction and nothing+  else. Bytes after the transaction are refused, as Bitcoin Core does for a+  raw transaction. They used to be ignored.+- `POST /transactions` takes the transaction in the format that peers of the+  network read. A network without SegWit, such as Bitcoin Cash, has the+  format without witness only.+- `GET /peers` gives the `version` of a peer as a signed number, as+  haskoin-core 1.6.0 reads it. A peer that sends version ffffffff has version+  -1, where it used to appear as 4294967295.+- Breaking, in `haskoin-store-data`: `toTransaction` takes the `Network` as+  its first argument, and the `version` field of `PeerInfo` is an `Int32`.+  The binary form of `PeerInfo` is the same bytes as before.+- From haskoin-node 2.3.0: a peer is disconnected when it sends more than+  50000 entries in an `inv` or `getdata` message. On Bitcoin Cash a message+  may be at most 2 MiB, unless it carries block content. Transactions from+  Bitcoin Cash peers are read in the format without witness only.++### Fixed++- A response no longer fails on a transaction with an output that has no+  address on the network. Since haskoin-core 1.5.0 the JSON encoder of such+  an address threw, which cut the response off. One such output is in Bitcoin+  block 211997 (transaction+  `a4bfa8ab6435ae5f25dae9d89e4eb67dfa94283ca751f393c1ddc5a837bbc31b`,+  output 0).++## [1.6.5] - 2026-11-04++### Fixed++- Build on haskoin-core 1.5.1, which evaluates the ASERT difficulty rule at the+  parent block. With 1.5.0 the store rejected a valid block on Bitcoin Cash+  mainnet at height 975899.++## [1.6.4] - 2026-11-04++### Added++- Support Bitcoin testnet4 (`NET=btctest4`), from haskoin-core 1.4.0.+- Option `--block-window` (`BLOCK_WINDOW`) to set how many blocks to request+  from the syncing peer at a time.+- Option `--write-buffer-mb` (`WRITE_BUFFER_MB`) to cap RocksDB memtables+  across all column families, 256 MiB by default (0 = unlimited). Without a+  cap the nine column families can keep up to 1.1 GiB of memtables.+- Option `--max-wal-mb` (`MAX_WAL_MB`), 1024 MiB by default (0 = RocksDB+  default), to bound the RocksDB write-ahead log: once it is larger, the+  column families holding its oldest file are flushed so the file can be+  deleted.+- `/dbstats` starts with the memory RocksDB uses outside the GHC heap:+  memtables and table readers for each column family, and the block cache.+- The mempool is kept in its own column family, one key per transaction,+  instead of as a single list value that every import batch read, loaded+  into a hash table and wrote back whole. Data version 19: a version 18+  database is migrated when it is opened, with no resync.++### Changed++- Replace monad-logger with the fast-logger `Logger` from haskoin-node 2.0.0.+- Request 64 blocks at a time from the syncing peer instead of 500, and ask+  for more once fewer than 16 remain, to bound memory use while catching up.+- haskoin-store-data builds against secp256k1-haskell 2.0.0, which hides the+  raw `PubKey` constructor: `XPubSpec` reads its stored key with `pubKey`,+  which checks it.+- Build on haskoin-core 1.5.0, haskoin-node 2.1.2 and secp256k1-haskell 2.0.0.+  `genesisBlock` no longer takes a context.+- A pay-to-public-key output has no address any more, as in haskoin-core+  1.5.0. Its coins cannot be spent with the input script of the P2PKH address+  of the same key, so they are no longer counted under that address.+- The block store no longer reads the whole mempool four extra times on every+  internal ping: it compares mempool snapshots only when there are pending+  transactions to import or expired ones to delete.+- Run with four capabilities (`-N4`) instead of one per core (`-N`), so that+  memory overhead does not grow with the size of the host. Override with+  `+RTS -N<n> -RTS`.+- Docker image: pin the base image to Debian trixie (RocksDB 9.10) by digest+  instead of `debian:latest`, and run with `MALLOC_ARENA_MAX=2` to limit+  malloc fragmentation from RocksDB.+- Each `/events` client, websocket or HTTP stream, gets a queue of at most+  10,000 events and is disconnected when it fills up, instead of the queue+  growing for as long as the client stays connected. Store events that+  `/events` does not show are no longer queued for its clients.+- A request waiting for another to load an xpub into the cache retries for+  about 30 seconds, with at most 5 seconds between attempts, and then serves+  the xpub from the database. It used to retry forever with a delay that+  doubled each time.++- Continuous integration on GitHub Actions: build, run both test suites,+  and check that the heap stays flat while syncing 6000 regtest blocks.++### Fixed++- Support full-RBF: accept replacements of transactions that do not signal BIP125.+- Correct various space leaks.+- `Unspent` implements `coinInputKind`, new in haskoin-core 1.5.0, from its+  output script, so coin selection with it no longer fails at run time.++### Added++- Support for testnet4 on both networks.++## [1.6.3] - 2026-08-31++### Added++- Allow disabling web server to sync block chain only.++## [1.6.2] - 2026-08-13++### Fixed++- Fix upstream regressions in haskoin-node.++## [1.6.1] - 2026-08-13++### Fixed++- Support latest version of haskoin-node.++## [1.6.0] - 2026-08-12++### Fixed++- Support for Bitcoin Cash 256-bit scripts.+- Address parsing now allows only correct lengths.++## [1.5.16] - 2025-05-08++### Changed++- Move to personal repository.++### Removed++- Disable `NO_SLOW` as an option.++### Added++- Allow to set `NO_XPUB` to disable extended public keys.++## [1.5.14] - 2024-12-04++### Changed++- Move `/dbstats` and `/mempool` to slow endpoints section.++### Added++- Cull old transactions from mempool automatically.++## [1.5.13] - 2024-10-25++### Changed++- Dummy release: update upstream LTS Haskell dependency in `stack.yaml` file.++## [1.5.12] - 2024-07-22++### Added++- Metric to track time to import a new block.+- Parameter `no_compact` allowed for backwards compatiblity.++## [1.5.11] - 2024-05-24++### Fixed++- Parameter `noCompact` corrected for `multiaddr` endpoint.++## [1.5.10] - 2024-05-24++### Fixed++- Prune obsolete transactions from outgoing peer requests.+- Remove unnecessary module imports.++## [1.5.9] - 2024-05-23++### Fixed++- Only sync mempool once for each peer.++## [1.5.8] - 2024-05-23++### Fixed++- Dummy release to bump upstream dependency on `haskoin-node`.++## [1.5.7] - 2024-05-23++### Fixed++- Dummy release to bump upstream dependency on `statsd-rupp`.++## [1.5.6] - 2024-05-22++### Changed++- Allow to toggle RocksDB Bloom filters off.++### Fixed++- Correct command-line option behaviour for some rarely-used parametres.++## [1.5.5] - 2024-05-19++### Changed++- Lock Redis before checking if the database needs to be pruned, rather than after.+- Lock xpub in Redis to avoid computing derivations multiple times.++### Fixed++- Rename redis-min-gap to redis-min-addrs to match environment config variable.++## [1.5.4] - 2024-05-07++### Fixed++- Fix Redis bug where old keys weren't pruned quickly enough.++### Changed++- Remove Nomad-specific environment variable parsing for statsd.++## [1.5.2] - 2024-04-24++### Changed++- Use default GHC garbage collector.++## [1.5.1] - 2024-04-23++### Changed++- Compile with GHC 9.8.2.+- Enable optimisations.++## [1.5.0] - 2024-04-06++### Fixed++- Use an optimised topological sort for importing transactions in bulk.++## [1.4.0] - 2024-03-27++### Changed++- Use default optimisation settings.+- Use default RocksDB parameters.++## [1.3.0] - 2024-03-14++### Changed++- Bump upstream dependencies.++## [1.2.5] - 2023-01-31++### Fixed++- Test suite now working again.++## [1.2.4] - 2023-01-31++### Changed++- Bump upstream dependencies.++## [1.2.3] - 2023-12-06++### Changed++- Transactions received via POST are just forwarded to every peer without delay or validation.+- Bump dependency on NQE to resolve async issue.++## [1.2.2] - 2023-10-21++### Fixed++- Fix comparison function for unspent outputs.++### Changed++- Avoid deprecated `param` function from Scotty.++## [1.2.1] - 2023-10-04++### Fixed++- Correct web exception output format (JSON).+- Correct block path parsing order.++## [1.2.0] - 2023-10-03++### Changed++- Upgrade `scotty` dependency and fix incompatibilities.++## [1.1.0] - 2023-09-15++### Added++- Add cache height stats.+- Add database iterator stats.++### Changed++- Detect xpub gap more strictly.+- Use statsd-rupp instead of ekg-stats.+- All stat reporting keys changed.++### Removed++- Remove unsafeInterleaveIO from Web.hs.+- Remove unnecessary WAI middlewares.+- Remove server and client error reporting per endpoint.++## [1.0.2] - 2023-08-25++### Changed++- Code refactoring and simplification.+- Improve logging output.++### Removed++- Do not impose time limits on REST API responses.++### Fixed++- Remove loop in Blockchain.info unspents function.+- Correct documentation for /blockchain/rawblock endpoint.+- Fix no match in record selector address for coinbases.++## [1.0.1] - 2023-08-03++### Changed++- Add compatibility with latest nightly LTS Haskell.++## [1.0.0] - 2023-07-28++### Changed++- Compatibility with latest haskoin-core and haskoin-node packages.+- Use DuplicateRecordFields and OverloadedRecordDot language extensions.+- Modernize package in a backwards-incompatible manner.+- Refactor package extensively.++## [0.65.10] - 2023-05-12++### Changed++- Revert to older LTS Haskell to attempt to resolve memory leak.++### Removed++- Remove unnecessary lock release.++## [0.65.9] - 2023-05-09++### Changed++- Be more aggressive caching individual transactions.+- Introduce a periodic cache mempool sync task.++## [0.65.8] - 2023-05-08++### Changed++- Perform health check in separate thread.++## [0.65.7] - 2023-05-05++### Changed++- Do not sync mempool to Redis if node is not up-to-date.++## [0.65.6] - 2023-05-04++### Changed++- Improve caching ingestion performance.++## [0.65.5] - 2022-09-19++### Added++- Fee field added to Blockchain.info-style export-history endpoint.++### Changed++- Bump version due to Git mistake.++## [0.65.3] - 2022-04-13++### Added++- IPv6 peer support.++## [0.65.2] - 2022-04-13++### Fixed++- Correct lazy bytestring deserialization algorithm for web client.++## [0.65.1] - 2022-04-13++### Changed++- Simplify function in Cache module.+- Bump depenedencies.++## [0.65.0] - 2022-02-03++### Changed++- Spenders now embedded into tx objects in database for performance.+- Use faster IORef instead of slow TVar when ingesting.+- Use mutable hash tables when ingesting.+- Allow to disable unscalable API endpoints.++## [0.64.19] - 2022-02-01++### Added++- Allow to set query timeouts.++## [0.64.18] - 2022-01-21++### Fixed++- Off-by-one error when importing change addresses to cache.++## [0.64.17] - 2022-01-21++### Fixed++- Add change addresses to cache when necessary.++## [0.64.16] - 2022-01-10++### Fixed++- Correct fee calculation for blocks that do not claim all rewards.++## [0.64.15] - 2022-01-07++### Fixed++- Correct scale of cache pruning code.++## [0.64.14] - 2022-01-07++### Fixed++- Always prune to 80% of max cache size.++## [0.64.13] - 2022-01-07++### Fixed++- Avoid doing a hissy fit on transient cache issue.++## [0.64.12] - 2022-01-07++### Fixed++- Prune a little more cache to allow for new keys.++## [0.64.11] - 2022-01-06++### Fixed++- Prune cache more often.++## [0.64.10] - 2021-12-30++### Changed++- Return empty addresses in xpub balances call.++## [0.64.9] - 2021-12-29++### Fixed++- Avoid notifying same transactions in mempool.++## [0.64.8] - 2021-12-26++### Changed++- Use better Fourmolu format.+- Be more precise when counting retrieved items.++### Fixed++- Avoid transactions duplicated in xpub data.+- Refactor some functions to make them more robust.++## [0.64.7] - 2021-12-26++### Fixed++- Optimisations for various Blockchain endpoints.+- More accurate item counts for various web endpoints.++## [0.64.6] - 2021-12-25++### Fixed++- Record cache index time just once per xpub.++## [0.64.5] - 2021-12-25++### Changed++- Add suffix cached to cache entries to avoid conflicts with data entries.++## [0.64.4] - 2021-12-25++### Fixed+- Bring dots back and avoid stats hacks.++## [0.64.3] - 2021-12-25++### Fixed++- Fix typo.++## [0.64.2] - 2021-12-25++### Changed++- Replace dots for underscores in metrics.++## [0.64.1] - 2021-12-25++### Changed++- Automatic formatting.+- Improve cache debug logging.++### Fixed++- Add item count to 'all' query metrics.++## [0.64.0] - 2021-12-25++### Changed++- Improve metrics.++### Fixed++- Get lock before importing transactions to cache.++## 0.63.0+### Fixed+- Make cache updates lightweight.++## 0.62.1+### Fixed+- Fix idempotent transaction elimination bug.++## 0.62.0+### Added+- Configurable Blockchain price URLs.+- Reuse HTTP connections with wreq sessions.++## 0.61.1+### Changed+- Upstream improvements on Base58 performance.++## 0.61.0+### Fixed+- Fixed recursion bug in Blockchain format conversion function.++## 0.60.0+### Fixed+- Correct notifications subsystem.++## 0.59.0+### Fixed+- Correct internal dependency.++## 0.58.0+### Fixed+- Correct output freeing algorithm.++## 0.57.0+### Changed+- Report errors freeing outputs.++## 0.56.0+### Changed+- Report all deleted mempool transactions.+- Crash on certain data corruption.++## 0.55.0+### Changed+- Always have a previous output object in Blockchain inputs.++## 0.54.0+### Added+- WebSocket support.+- Testnet4 support.++## 0.53.11+### Changed+- Dummy version increase to signal upstream update of Haskoin Core.++## 0.53.10+### Fixed+- Correct test that was reversed in previous version.++## 0.53.9+### Removed+- Mempool is no longer synced by default to fix public Bitcoin Core regression.++## 0.53.8+### Changed+- Put derivations stat inside database.++## 0.53.7+### Added+- Added counters for database retrievals.+- Added counter for xpub derivations.++### Changed+- Removed some buggy or unnecessary stats.++## 0.53.6+### Changed+- Improve web server statistics.++## 0.53.5+### Addded+- Endpoint to delete an xpub from cache.++## 0.53.4+### Added+- Debugging when reverting block.++### Fixed+- Negative (headers - blocks) diff shown incorrectly.+- Infinite loop when removing a block's coinbase transaction.++## 0.53.3+### Added+- Blockchain.info endpoint: `q/pubkeyhash/:addr`.+- Blockchain.info endpoint: `q/getsentbyaddress/:addr`.+- Blockchain.info endpoint: `q/pubkeyaddr/:addr`.+- Configurable request body limit size.++## 0.53.2+### Fixed+- Return output in satoshi for some endpoints.++## 0.53.1+### Fixed+- Return 404 in case of bad address.++## 0.53.0+### Added+- Blockchain.info endpoint: `q/addresstohash/:addr`.+- Blockchain.info endpoint: `q/addrpubkey/:pubkey`.+- Blockchain.info endpoint: `q/hashpubkey/:pubkey`.+- Blockchain.info endpoint: `q/getblockcount`.+- Blockchain.info endpoint: `q/latesthash`.+- Blockchain.info endpoint: `q/bcperblock`.+- Blockchain.info endpoint: `q/txtotalbtcoutput/:txid`.+- Blockchain.info endpoint: `q/txtotalbtcinput/:txid`.+- Blockchain.info endpoint: `q/txfee/:txid`.+- Blockchain.info endpoint: `q/txresult/:txhash/:addr`.+- Blockchain.info endpoint: `q/getreceivedbyaddress/:addr`.+- Blockchain.info endpoint: `q/addressbalance/:addr`.+- Blockchain.info endpoint: `q/addressfirstseen/:addr`.+- Support xpubs in `blockchain/rawaddr` endpoint.++### Fixed+- Various latest blocks endpoint fixes.+- Allow empty parameters in Blockchain.info POST requests.++## 0.52.13+### Fixed+- Upstream fixes for unknown inv types.++## 0.52.12+### Fixed+- Fix limits for legacy endpoints.++## 0.52.11+### Added+- Alias for Blockchain.info raw address endpoint.++### Fixed+- Fix unknown message types without payload upstream.+- Randomise peer timeouts upstream.++## 0.52.10+### Fixed+- Fix binary search algorithm for blocks upstream.++## 0.52.9+### Added+- Add Blockchain.info latest block endpoint.+- Add Blockchain.info unconfirmed-transactions endpoint.+- Add Blockchain.info blocks for one day endpoint.++## 0.52.8+### Fixed+- Fix special case for zero limit.++## 0.52.7+### Fixed+- Fix cache bug when too many transactions in mempool.++## 0.52.6+### Changed+- Use rationals for Blockchain.info export-history endpoint arithmetics.++## 0.52.5+### Added+- Support for Blockchain.info export-history endpoint.++### Fixed+- Use strict bytestring decoder for HTTP API client.++## 0.52.4+### Fixed+- Use Base58 addresses by default in Blockchain.info balance querios.++## 0.52.3+### Fixed+- Errors should also contain CORS headers.++## 0.52.2+### Fixed+- Blockchain.info raw address limit should use `limit` parametre and not `n`.++## 0.52.1+### Changed+- Stream multiple transactions when requested.++### Fixed+- Fix missing unspent outputs.++## 0.52.0+### Changed+- Remove unnecessary performance optimisations.++### Fixed+- Reduce balance retrievals for xpubs.+- Hack UTXO ordering to avoid unwanted behaviours.+- Do not merge conduits when there are too many.++## 0.51.0+### Fixed+- Data type for RequestTooLarge was missing.++## 0.50.3+### Added+- Log HTTP request bodies.++### Removed+- Remove web request timeout.++### Fixed+- Do not ignore unknown start tx anchors in conduits.+- Use fast transaction counting in cache.++### Changed+- Use WAI middlewares to manage stats.++## 0.50.2+### Fixed+- Do not use lazy parsers for incoming data that may throw exceptions.++## 0.50.1+### Fixed+- Do not allow incoming POST requests of unlimited body size.++## 0.50.0+### Added+- Limit number of simultaneous xpubs being cached.++## 0.49.0+### Changed+- Improve conduit merging algorithm.++## 0.48.0+### Added+- Timeouts and token bucket for web requests.+- Configurable price fetch interval.++## 0.47.6+### Fixed+- Fix web.all stat.++## 0.47.5+### Changed+- Simplify stats.++## 0.47.4+### Added+- Implement Blockchain.info block height.+- Implement Blockchain.info raw address.+- Implement Blockchain.info balances.++### Fixed+- Make Blockchain.info unspent endpoint more efficient.+- Make Blockchain.info xpubs endpoint more efficient.++### Changed+- Use compact JSON serialization by default everywhere.++## 0.47.3+### Fixed+- Fix serialization bug with unspents.++## 0.47.2+### Fixed+- Fix serialization bug for health check.++## 0.47.1+### Fixed+- Fix serialization bugs for web data types.++## 0.47.0+### Added+- Support for legacy block endpoints.++### Changed+- Serialization now uses bytes library and does interleaved IO.++## 0.46.6+### Added+- Allow comma-separated values in multiaddr.++## 0.46.5+### Changed+- Throw error when offset exceeded instead of truncating it.++## 0.46.4+### Fixed+- Do not compute balances separately for loose addresses when part of an xpub.++### Changed+- Do not fetch entire transaction list for loose addresses.+- Transaction count in wallet object is just a synonym for filtered tx count.+- Use conduits to prevent over-fetching transactions from database.++## 0.46.3+### Fixed+- Make hex value encoding compatible with Java BigInteger decoder.++## 0.46.2+### Changed+- Allow negative confirmations for compatibility with old API.++## 0.46.1+### Added+- Filters for transactions in multiaddr endpoint.++## 0.46.0+### Fixed+- Release to fix unintentional upload.++## 0.45.0+### Added+- Using a tx_index that results in a txid conflict returns a 409 - Conflict.++## 0.44.0+### Changed+- Numeric txid now 53 bits long and doesn't return transactions when hashes collide.++## 0.43.1+### Fixed+- Do not overcount statistics.++## 0.43.0+### Changed+- Transaction index now simply encodes txid as a long number (sorry JS users).++## 0.42.4+### Changed+- Optimise encoding of JSON data for Blockchain.info API.++## 0.42.3+### Changed+- Narrow down the balance for transactions to onlyShow set.++## 0.42.2+### Changed+- Make unspent algorithm more efficient.++## 0.42.1+### Added+- Support for Blockchain.info API unspent endpoint.++### Fixed+- Fix RBF transactions on Blockchain.info API.++## 0.42.0+### Removed+- No more pruning of old mempool transactions.++## 0.41.4+### Fixed+- Correct error computing wallet balances.++## 0.41.3+### Changed+- Use whole active set when computing results for multiaddr.+- Compute correct count of transactions for wallet.++## 0.41.2+### Changed+- Caching mempool is not much slower when pruning, so reversing that change.++## 0.41.1+### Changed+- Only prune mempool every ten minutes or so.++## 0.41.0+### Removed+- No more cache eviction endpoint.++## 0.40.22+### Added+- More metrics for cache.++## 0.40.21+### Added+- Support HTTP GET for multiaddr endpoint.++## 0.40.20+### Added+- Prefix 'app.' in statsd when using Nomad variables.++## 0.40.19+### Added+- Statsd support and pleny of statistics.++### Removed+- Removed use of sync lock for importing xpubs in cache due to volume.++## 0.40.18+### Added+- New options to control cache lock retry count and delay.++### Changed+- Cache xpubs asyncrhonously.+- Mempool syncs fully every refresh cycle.++## 0.40.17+### Changed+- Delete more than 1000 xpubs from Redis if needed.+- Attempt to get lock ten times in order to index a new xpub.++## 0.40.16+### Fixed+- Fix best block not set.++## 0.40.15+### Fixed+- Do not update cache best block if it is higher.++## 0.40.14+### Fixed+- Always respond to cache pings to avoid getting stuck.++## 0.40.13+### Added+- Be more careful about importing blocks to the cache.++### Removed+- Reduce excessive cache logging.++## 0.40.12+### Fixed+- Correct bad arithmetics in cache cooldown.++## 0.40.11+### Added+- Debug log cache cooldown.++### Fixed+- Unlimited retry lock acquisition now works.+- Make locking tighter++## 0.40.10+### Changed+- Reduce the number of things that cache do when pinged.++## 0.40.9+### Fixed+- Avoid filling the cache actor mailbox with pings.++## 0.40.8+### Fixed+- Correct a typo in cache code.++## 0.40.7+### Changed+- Improve cache mempool sync algorithm.++## 0.40.6+### Changed+- Make cache cooldown a number.++## 0.40.5+### Added+- Log every non-successful query.++## 0.40.4+### Fixed+- Use Redis TTL for cache cooldown.++## 0.40.3+### Fixed+- Mempool no longer gets saved in cache when node is not in sync.++## 0.40.2+### Changed+- Change source of price data to https://blockchain.info/ticker.++## 0.40.1+### Removed+- Do not expose Paths_haskoin_store module explicitly.++## 0.40.0+### Added+- Blockchain.info API compatibility for multiaddr and rawtx endpoints.++### Changed+- Segwit fields are no longer omitted on incompatible networks.++## 0.39.0+### Changed+- Upgrade Haskoin Core to support higher version on witness addresses.++## 0.38.4+### Fixed+- Fix block search by timestamp.++## 0.38.3+### Changed+- Disable low latency garbage collector by default.++## 0.38.2+### Changed+- Use GHC 8.10 and its low latency garbage collector.++## 0.38.1+### Changed+- Optimise algorithm for fork safety check.++## 0.38.0+### Added+- Support for Bitcoin Cash hard fork.++## 0.37.5+### Added+- Configurable cache refresh time.++## 0.37.4+### Fixed+- Now request mempool when connecting even if already in sync.++### Changed+- Add a delay to mempool sync to allow for block headers to catch up.+- Attempt to sync mempool whenever syncing is done or a new peer connects.++## 0.37.3+### Changed+- Correct logic error when deleting superseded transactions.++## 0.37.2+### Changed+- Simplify multiple algorithms to hopefully solve corruption bug.+- Allow to disable mempool transaction indexing via command line.++## 0.37.1+### Changed+- Better locking around caching operations.+- Have a global cooldown on cache mempool updates.++### Fixed+- Fix address unspent retrieval using correct column family.++## 0.37.0+### Added+- Add support for column families and prefixes breaking database backwards compatibility.++## 0.36.2+### Fixed+- Fix another place where mempool sorting was wrong.++## 0.36.1+### Fixed+- Fix wrong sorting of mempool transactions in database.++## 0.36.0+### Changed+- Do not use STM transactions because it is impossible to reason about preloading.+- Make importing blocks after mempool is populated faster.++## 0.35.2+### Changed+- Reduce frequency of cache mempool updates.++## 0.35.1+### Changed+- Remove cache lock retries.+- Do not use lock when caching a new xpub.++## 0.35.0+### Changed+- Use new RocksDB Haskell bindings.+- Make headers - blocks diff configurable in health check.+- Simplify locking in cache.++## 0.34.8+### Changed+- Minor refactoring.+- Only send one mempool message ever.++## 0.34.7+### Changed+- Fix block heights endpoint URL.++## 0.34.6+### Removed+- Remove asynchronous database retrieval permanently due to instability.++## 0.34.5+### Added+- Preload memory before freeing outputs.++## 0.34.4+### Added+- Asynchronous preloading only for blocks.++## 0.34.3+### Removed+- Disable asynchronous preloading to prevent segmentation fault error.++## 0.34.2+### Removed+- No more upper limit for mempool call.++### Fixed+- Health check uses minimum between last block and mempool tx.++## 0.34.1+### Added+- Pending mempool transactions health check.++### Removed+- No more upper limit for mempool call.++## 0.34.0+### Changed+- New format for health check.++## 0.33.1+### Changed+- Upload with updated Cabal files.++## 0.33.0+### Fixed+- Fix outputs showing up as both spent and unspent.++### Changed+- Simplify data structures used to hold uncommitted changes.+- Split StoreRead class in two.+- Improve pre-loading algorithms.++## 0.32.3+### Fixed+- Fix command line argument mappings to env vars.++## 0.32.2+### Fixed+- Do not complain when deleting non-existent unconfirmed txs.++## 0.32.1+### Fixed+- Fix corruption bug when deleting transactions during some reorgs.+- Fix tests on data package.+- Rename upstream address string functions.++## 0.32.0+### Changed+- Concurrently load dependencies when importing transaction data.+- Import each transaction using STM.+- Rework logging.+- Refactor all block chain importing algorithms.+- Use peer locking when importing data.+- Change license to MIT.++### Added+- Make max peer count configurable.++### Fixed+- Prevent premature peer timeouts when importing blocks.++## 0.31.0+### Added+- Add Haskell web client code to data library.+- Use type-safe definitions for web paths and parameters.+- Support JSON pretty encoding and enable by default for some endpoints.+- Allow environment variables for configuration.++### Changed+- Change command-line arguments to better fit GNU style.+- Refactor code to manipulate address balances.+- Improve logging.+- Rate-limit mempool importing.+- Do not add xpubs to cache if node not in sync.+- Do not import transactions to cache if no xpubs in cache.+- Refactor transaction importing logic extremely.+- Refactor memory database and writer.++### Fixed+- Fix BTC double-count transaction bug.+- Add missing xpub derive parameter to xpub evict documentation.++## 0.30.1+### Changed+- Limit mempool API endpoint list.++## 0.30.0+### Changed+- Raw JSON values are now wrapped in an object with a result property.+- Better logging when importing transactions into mempool.++### Fixed+- Transaction limit by block height works now.++## 0.29.3+### Changed+- Revert concurrent retrieval changes from 0.29.1 and 0.29.0 to solve stability+  issues.+- Bitcoin Cash (BCH) replaces BTC as default network in CLI.+- Peer discovery enabled automatically if no peers specified in CLI.+- Use separate Stack configuration file for data package.++## 0.29.2+### Changed+- Increase timeouts for manager and chain health checks.++## 0.29.1+### Added+- Perform xpub operations concurrently.++## 0.29.0+### Changed+- BlockTx renamed to TxRef.+- Remove unnecessary monad transformers to simplify import code.++### Added+- Retrieve balances and unspent outputs concurrently when importing.+- Coin instance for Unspent in data package.++## 0.28.0+### Changed+- Change the way that limits, start, and offset values are handled.++### Fixed+- Fix incorrect limit responses.++## 0.27.0+### Added+- Complete support for offset.++### Changed+- Improve monad stack for cache.++## 0.26.6+### Added+- Support for offset in all endpoints where it makes sense.++## 0.26.5+### Fixed+- Sort after using the nub function or use Data.List.nub in exceptional cases.++## 0.26.4+### Changed+- Don't do certain validity checks on confirmed transactions.++### Fixed+- Do not crash when syncing the blockchain from scratch on a populated cache.++## 0.26.3+### Changed+- Replace nub function everywhere with faster version written using hash sets.++### Fixed+- Fix shared cache locking.++## 0.26.2+### Changed+- Simplify cache block import code.+- Improve cache import performance.++## 0.26.1+### Fixed+- Do not call zadd or zrem in Redis with incorrect argument count.++### Changed+- Do not use slow Transaction type in Logic.+- Improve logging while importing blocks.+- Better transaction import algorithm.++## 0.26.0+### Changed+- Add information to some binary types for feature parity with JSON.++## 0.25.4+### Changed+- Hide segwit data on non-segwit networks.++### Fixed+- Fix work which now always serialises as a number literal.++## 0.25.3+### Changed+- Fix version display on health check.++## 0.25.2+### Changed+- Fix wrong version of Haskoin Store Data dependency.++## 0.25.1+### Changed+- Improve code organisation.+- Split data definitions and serialisation into its own package.++## 0.24.0+### Changed+- Use mocking to simulate peers instead of connecting to the network while testing.+- Depend on latest version of Haskoin Core for faster JSON serialisation.+- Clean up confusing JSON encoding/decoding codebase.++## 0.23.24+### Changed+- Implement faster JSON encoding using toEncoding from the Aeson library.++## 0.23.23+### Changed+- Do not read full mempool so often from cache code.++## 0.23.22+### Changed+- Depend on latest Haskoin Node that fixes debugging regression.++## 0.23.21+### Changed+- Reduce non-debug log verbosity.++## 0.23.20+### Changed+- Better Haskoin Node logging.++## 0.23.19+### Changed+- Use set peer timeout instead of constant in block store. ++## 0.23.18+### Changed+- Allow setting peer timeouts in command line.++### Fixed+- Bump Haskoin Node dependency to fix another premature timeout condition.++## 0.23.17+### Fixed+- Touch syncing peer whenever we process one of its blocks to avoid premature timeout.++## 0.23.16+### Changed+- Better algorithm to avoid importing transaction multiple times in cache.+- Depend on latest Haskoin Node.++## 0.23.15+### Changed+- Do not storm the cache with mempool transactions.++## 0.23.13+### Fixed+- Do not ignore deleted incoming transactions.++## 0.23.12+### Fixed+- Do not do RBF checks when replacing a mempool transaction with a confirmed one.++## 0.23.11+### Fixed+- Streamline mempool transaction importing.++## 0.23.10+### Fixed+- Fix transactions not recorded in cache mempool set.+- Fix transactions being downloaded multiple times.++### Removed+- Do not store orphan transactions in database.++### Changed+- Use sets for incoming transactions instead of lists.+- Do not do anything to the cache if there are no xpubs in it.++## 0.23.9+### Fixed+- Wiping mempool fixed.++## 0.23.8+### Added+- Chunk transactions to be deleted from mempool.++## 0.23.7+### Added+- Mempool improvements.++## 0.23.6+### Added+- Ability to wipe mempool at start.+- Improvements to mempool processing code.++## 0.23.5+### Changed+- Tighten the locking loop to avoid slow cache building.++## 0.23.5+### Fixed+- Wrong error in cache when acquiring lock.++## 0.23.4+### Added+- Add extra debug logging for cache code.++### Fixed+- Fix a bug with xpub growing algorithm.++### Changed+- Use locks instead of transactions to update cache.++## 0.23.3+### Fixed+- Reduce contention when many instances of Haskoin Store share a cache.++## 0.23.2+### Added+- Allow retrieving xpub data without using cache.++## 0.23.1+### Added+- Allow xpub eviction from cache via API.+- Clarify cache address addition code.++### Fixed+- Balances were incorrectly computed in cache when new transactions arrived.++## 0.23.0+### Added+- Support for Redis transactions.+- Use a smaller initial gap for empty xpubs.++### Changed+- Remove custom JSON encoding class.+- Refactor and code simplification.++## 0.22.5+### Fixed+- Cache was being completely pruned.++## 0.22.4+### Fixed+- Cache now prunes correctly.++## 0.22.3+### Fixed+- Bug was making cache get stuck when pruning.++## 0.22.2+### Changed+- More efficient algorithms for caching and cache misses.+- Better debug logging of cache hits and misses.++## 0.22.1+### Added+- More debug logging for cache hits.++### Fixed+- Bug using maximum against empty list.++## 0.22.0+### Changed+- Extreme code refactoring.+- Move all code to Haskoin and drop Network from modules.++### Added+- Use Redis pipelining when importing multiple transactions into cache.+- Implement configurable LRU for Redis cache.+- Import xpubs directly into cache from web worker thread when a key is requested.++### Removed+- Only expose a few modules to external API.++## 0.21.7+### Changed+- Improve build configuration.++### Fixed+- Use minimum used addresses instead of minimum index for xpub cache decision.++## 0.21.6+### Fixed+- Fix missing xpub unspent outputs when using cache.++## 0.21.5+### Added+- Only store xpubs in cache if they have more than a threshold addresses used.++## 0.21.4+### Fixed+- Fix shared cache case where head is set beyond header chain by another node.++## 0.21.3+### Fixed+- Fix bug where best head was not being registered in cache.+- Fix best head in cache being decoded incorrectly.++## 0.21.2+### Added+- Complete support for Redis xpub cache.++## 0.21.1+### Fixed+- Latest version of secp256k1-haskell works with Debian 9.++## 0.21.0+### Fixed+- Fix output of web API calls when issuing limits with offsets.++### Changed+- Massive refactoring of entire codebase.++### Added+- Work in progress Redis caching for extended public keys.++## 0.20.2+### Changed+- Filter xpub address balances on web API to show only addresses that have been used.++### Removed+- Remove paths and addresses from xpub summary.++## 0.20.1+### Changed+- Refactor code greatly.+- Depend on new Haskoin Store package to avoid missing tx broadcasts.+- Merge StoreStream class into StoreRead.+- Move former streaming functions to use lists instead of conduits.+- Remove excessively verbose debugging.++## 0.20.0+### Added+- Set minimum version bounds for some dependencies.+- Now compatible with GHC 8.8.+- Extended key caching system using Redis.++### Changed+- Massively refactored codebase.+- Less verbose debug logging.++### Removed+- Removed conduits for faster queries.++## 0.19.6+### Added+- Use a parallel strategy to compute key derivations.++### Fixed+- Do not fail health check upon transaction timeout while syncing.++### Changed+- Do not use conduits for xpub balance streams.+- Multiple minor refactorings.++## 0.19.5+### Changed+- Minor refactor to block import code.++### Fixed+- Minor fix to transaction timeout check.++## 0.19.4+### Fixed+- Clarify and correct health check algorithm.++## 0.19.3+### Changed+- Add address transactions to cache.+- Improve multi-address transaction retrieval algorithms.++## 0.19.2+### Removed+- Cache-Control header turned out to be unnecessary.++### Fixed+- Fix some minor errors in web module.++## 0.19.1+### Added+- Set Cache-Control header to no-cache.++## 0.19.0+### Changed+- Store mempool in single key/value pair.++## 0.18.11+### Changed+- Do not stream mempool.++## 0.18.10+### Removed+- Disable timeout checks for testnets.++## 0.18.9+### Added+- Endpoint to locate a block by unix timestamp.++### Removed+- No more persistence for peers due to dependency on newest haskoin-node.++## 0.18.8+### Added+- Transaction and block timeouts for health check.+- Raw blocks.++## 0.18.7+### Fixed+- Missing tranasctions on xpub listings.++## 0.18.6+### Removed+- Follow Stack advise removing `-O2` GHC option.++## 0.18.5+### Added+- Compatibility with SegWit on extended public key endpoints.++### Changed+- Fix syncing peer not reset after timeout.+- Use simpler monad for streaming data.++## 0.18.4+### Changed+- Bump Haskoin Node to fix peers not stored and excessively verbose logging.++## 0.18.3+### Changed+- Configurable HTTP request logging. Disabled by default.++## 0.18.2+### Changed+- Fix for memory leak.++## 0.18.1+### Removed+- Remove transaction count from xpub summary object.++## 0.18.0+### Changed+- Simplified limits and start point handling on web API.+- Made transaction streaming algorithm faster for xpub transactions.+- Extended public key summary output contains all addresses that have received transactions.++### Added+- Fine-grained control for maximum limits via command line options.+- Transaction hash as starting point.+- Block hash as starting point.+- Timestamp as starting point.+- Configurable xpub gap limit.+- Transaction count added to xpub summary.+- UTXO count added to xpub summary.++### Removed+- Mempool endpoint now has no limits or offsets and always returns full list.+- Extended public key summary output no longer includes any transactions.+- Offsets not allowed for transaction lists involving multiple addresses or xpubs.+- Confusing block position parameter no longer part of web API.++## 0.17.2+### Changed+- Stream address balances and unspent outputs.+- Add configurable max value for limit and offset which defaults at 10000.++## 0.17.1+### Changed+- When posting a transaction to the network, timeout is now five seconds.+- Improve error message when transaction post timeout reached.+- Remove obsolete `not found` error for transaction post.+- Endpoints for retrieving blocks now do streaming for better performance.+- Improve Swagger API documentation.++## 0.17.0+### Added+- Endpoints for retrieving block transactions.+- Endpoint for retrieving set of latest blocks.++### Changed+- Use standardized JSON and binary serialization schemes for raw transaction endpoints.++## 0.16.6+### Changed+- Now logging info messages too by default.+- Consolidated web logging in middleware.++### Removed+- UUIDs for web requests.++## 0.16.5+### Removed+- Remove concurrency from xpub balance requests to prevent RocksDB segfaults.++## 0.16.4+### Removed+- Remove concurrency from requests using iterators to prevent RocksDB from segfaulting.++## 0.16.3+### Added+- Debugging information for web API.++## 0.16.2+### Changed+- Debugging disabled by default (use `--debug` to enable).++## 0.16.1+### Added+- Cache mempool transactions.+- Improve initial syncing performance.++## 0.16.0+### Added+- Orphan transaction support.+- Full address balance cache in RocksDB.+- Full unspent output cache in RocksDB.++### Changed+- Significantly refactor code.+- Move web stuff to its own module.+- Change types related to databases.+- Make xpub balance, transaction and unspent queries fetch data in parallel.++## 0.15.2+### Added+- Internal data types to support orphan transactions.++### Changed+- Do not spam block actor with pings.+- Fix balance/unspent cache not reverting when importing fails.+- Fix transaction sorting algorithm not including transaction position information.+- Fix conflicting mempool transaction preventing block from importing.++## 0.15.1+### Changed+- Fix duplicate coinbase transaction id bug.++## 0.15.0+### Removed+- Removed `PreciseUnixTime` data type.++### Changed+- Use 64 bits for Unix time representation.+- Data model now uses simplified Unix time representation.++## 0.14.9+### Added+- Last external/change index information to xpub summary object.++## 0.14.8+### Added+- Endpoint for xpub summaries.+- Endpoints for full transactions.+- Ability to query by offset/limit.+- More API documentation.++## 0.14.7+### Added+- Debug information for `publishTx`.++### Changed+- Transaction publishing no longer requests mempool.+- Fixed serialization for `TxId` freezing the entire program by infinitely recursing.++## 0.14.6+### Changed+- Enable full threading again as it was deemed not responsible for the freezing behaviour.++## 0.14.5+### Changed+- Enable threading but leave it at a single thread to be able to open more than 1024 files.++## 0.14.4+### Changed+- Target LTS Haskell 13.20 and disable threading in new attempt to fix freezing bug.++## 0.14.3+### Changed+- Remove `-O2`.++## 0.14.2+### Changed+- Target LTS Haskell 12.26 to attempt to fix freezing bug.++## 0.14.1+### Added+- Extra debugging around code that freezes.++### Changed+- Bump dependency on `haskoin-node`.++## 0.14.0+### Removed+- Dump slow protobuf serialization.++### Added+- Add custom serialization.+- Extra debug logging.++### Changed+- Bump `haskoin-core` and `haskoin-node`.++## 0.13.1+### Changed+- Bump `haskoin-node` in `stack.yaml`.+- Do not send empty `getdata` messages.++## 0.13.0+### Added+- Primitive content negotiation for web exceptions.+- Protobuf support for errors.+- Protobuf support for tranasction ids.++### Changed+- Protobuf format changed in non-backwards-compatible manner for transaction ids.++## 0.12.0+### Added+- Support for binary serialization using Protocol Buffers.+- New endpoints for binary raw transactions (not hex-encoded).++### Changed+- Services field now a hex string instead of a number to avoid overflowing signed 64-bit integer.+- Flatten list of block data objects when responding to request for multiple block heights.+- Errors now reported in plain text without container JSON object.+- Transaction broadcasts are responded to with transaction id in plaintext hex (no JSON).+- Remove database snapshots to improve performance.++## 0.11.2+### Changed+- Fix duplicate mempool transaction announcements in event stream.++## 0.11.1+### Removed+- Removed latest block time check.++## 0.11.0+### Changed+- Improve post transactions endpoint.++## 0.10.1+### Changed+- Fix bug where transaction lists from multiple addresses would sort incorrectly.+- Address gap reduced to 20.++## 0.10.0+### Removed+- Remove addresses from transaction lists.+- No longer use container objects for xpub transactions.++## 0.9.3+### Added+- Permissive CORS headers to allow queries from any domain.+- Improved documentation using real-world examples from the BCH testnet.++## 0.9.2+### Added+- HTTP JSON API switch to turn off transaction list when retrieving blocks.++## 0.9.1+### Added+- Total block fees.+- Total block outputs.+- Block subsidy.++### Changed+- Do not consider the blocks less one block away from headers as out of sync in health check.+- Health check now returns HTTP 503 when not OK or out of sync.++## 0.9.0+### Added+- Version to health check output.+- Block weight for segwit.+- Transaction weight for segwit.++## 0.8.1+### Added+- Health check endpoint.++## 0.8.0+### Added+- Limits and skips.+- Add timestamps to transactions.+- Add transaction count to address balance object.+- Add Merkle root to block data.+- Total funds received by an address now shows up in balance.+- Balances for any address that ever had funds show up in xpub endpoints.++### Changed+- Data model update.+- Performance improvement for xpub calls.+- Transactions are returned in reverse mempool/block order (highest or most recent first).+- Balance objects do not get deleted from database ever.++## 0.6.9+### Changed+- Reduce number of coinbase checks to 10,000 ancestors.++## 0.6.8+### Changed+- Further optimize coinbase after height checks.++## 0.6.7+### Changed+- Impose restrictions on recursion for coinbase after height checks.++## 0.6.6+### Added+- Check whether a transaction can be traced back to a coinbase after certain height.++## 0.6.5+### Changed+- Delete transactions in reverse topological order when reverting a block.++## 0.6.4+### Changed+- Do not fail silently when importing orphan transactions into the mempool.++## 0.6.3+### Changed+- Dummy release to bump haskoin-node in stack.yaml.++## 0.6.2+### Changed+- Correct bug where coinbase transactions weren't properly flagged.++## 0.6.1+### Changed+- Compatibility with Bitcoin Cash hard fork.+- Various bug fixes.++## 0.6.0+### Added+- Address balance cache in memory.++### Changed+- Simplify data model further.+- Fix bug importing outputs with UTXO cache.+- Unspent balances cannot be negative.++## 0.5.0+### Added+- Add UTXO cache in memory.+- Get transactions with witness data in segwit networks.++### Changed+- Paths for derivations in xpubs is a list and no longer a string.+- Various bug fixes.++## 0.4.2+### Removed+- Remove extended public key itself from output of relevant endpoints to save bandwidth.++## 0.4.1+### Changed+- Fix bug when deleting coinbase transactions.+- Extended public key API support.++## 0.4.0+### Changed+- Generate events for mempool transactions.+- Respond with entire block data when querying blocks by height.++## 0.3.1+### Changed+- Do not import transactions to mempool while not synced.+- Only sync mempool against a single peer.+- Allow duplicate transactions to fix re-introduced sync bug.++## 0.3.0+### Added+- Update dependencies.+- Keep orphan blocks and deleted transactions in database.+- Add a `mainchain` field for block data and a `deleted` field for transactions.+- Stream records for performance.+- Show witness data for transaction inputs in SegWit network.+- Support RBF in SegWit network.++### Changed+- Refactor all data access code away from actor.+- Refactor import logic away from actor.+- Abstract data access using typeclasses.+- Implement data access using clean layered architecture.+- Make most of import logic code pure.+- Database now in `db` as opposed to `blocks` directory.+- Use latest `haskoin-node`.++### Removed+- Remove some data from peer information output.+- Remove full transaction from address transaction data.+- Remove limits from address transaction data.+- Remove block data from previous output.+- Remove spender from JSON response when output not spent.+- Remove block hash from block reference.++## 0.2.3+### Removed+- Do not send transaction notifications if a block is being imported.++## 0.2.2+### Added+- Peer information endpoint.++### Changed+- Update `haskoin-node`.++## 0.2.1+### Changed+- Fix tests++## 0.2.0+### Added+- Documentation everywhere.+- Ability to retrieve address transactions.++### Changed+- New versions of NQE and Haskoin Node upstream.+- Improve and simplify API.+- Multi-element endpoints return arrays of arrays.+- Database snapshots for all queries are now mandatory.++### Removed+- Retrieving unspent and spent outputs for an address.+- Redundant API endpoints for multiple elements.++## 0.1.3+### Changed+- Fix a bug with transaction notifications.+- Improve handling orphan transactions.++## 0.1.2+### Changed+- Specify dependencies better.++## 0.1.1+### Changed+- Dependency `secp256k1` is now `secp256k1-haskell`.++## 0.1.0+### Added+- New `CHANGELOG.md` file.+- Bitcoin (BTC) and Bitcoin Cash (BCH) compatibility.+- RocksDB database.+- Mempool support.+- HTTP streaming for events.+- CashAddr support.+- Bech32 support.+- Rate limits.++### Changed+- Split out of former `haskoin` repository.+- Use hpack and `package.yaml`.++### Removed+- Removed Stylish Haskell configuration file.+- Remvoed `haskoin-core` and `haskoin-wallet` packages from this repository.

@@ -0,0 +1,936 @@+# Haskoin Store++[![CI](https://github.com/jprupp/haskoin-store/actions/workflows/ci.yml/badge.svg)](https://github.com/jprupp/haskoin-store/actions/workflows/ci.yml)+[![Hackage haskoin-store](https://img.shields.io/hackage/v/haskoin-store.svg?label=haskoin-store)](https://hackage.haskell.org/package/haskoin-store)+[![Hackage haskoin-store-data](https://img.shields.io/hackage/v/haskoin-store-data.svg?label=haskoin-store-data)](https://hackage.haskell.org/package/haskoin-store-data)+[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/jprupp/haskoin-store/blob/master/haskoin-store/LICENSE)++A Bitcoin (BTC) and Bitcoin Cash (BCH) block chain indexer with a REST API,+written in Haskell. It connects to the peer-to-peer network directly, so it+does not need a full node next to it, and keeps its indices in RocksDB.++- [API reference](https://github.com/jprupp/haskoin-store/blob/master/swagger.yaml)+- [Library documentation](https://hackage.haskell.org/package/haskoin-store)+  and [client documentation](https://hackage.haskell.org/package/haskoin-store-data)+- [Changelog](https://github.com/jprupp/haskoin-store/blob/master/CHANGELOG.md)+- [Report a bug](https://github.com/jprupp/haskoin-store/issues)++## Contents++- [Why Haskoin Store](#why-haskoin-store)+- [Design principles](#design-principles)+- [Features](#features)+- [Quick start](#quick-start)+- [Building from source](#building-from-source)+- [Running](#running)+- [Tutorial: following a wallet](#tutorial-following-a-wallet)+- [Configuration](#configuration)+- [API](#api)+- [How it works](#how-it-works)+- [Monitoring](#monitoring)+- [Using it from Haskell](#using-it-from-haskell)+- [Packages](#packages)+- [Development](#development)+- [Related projects](#related-projects)+- [License](#license)++## Why Haskoin Store++A Bitcoin node can tell you what is in block 800000, but not what the balance+of an address is, which transactions belong to a wallet, or which outputs it+can spend. Wallets, block explorers, payment processors and accounting tools+all need exactly those answers, for many users at once, in milliseconds.++The usual way to get them is to run a full node and an indexer next to it, and+operate both. Haskoin Store is one program instead. It connects to the+peer-to-peer network itself, downloads blocks and unconfirmed transactions,+keeps indices by address, transaction and extended public key in RocksDB, and+serves them over a REST API. It has been used to serve public Bitcoin and+Bitcoin Cash APIs such as api.haskoin.com.++## Design principles++- **Index for the questions people ask.** Every index is laid out so that an+  API query is one seek and a short scan in RocksDB, newest first, which keeps+  response times flat as the chain grows.+- **Never half-write.** Each block, and each batch of mempool transactions, is+  one atomic database write. A crash leaves the store at the previous block,+  with no repair step on restart.+- **Unconfirmed transactions matter.** The mempool is indexed and persisted like+  the chain, ordered before confirmed history, and handles replace-by-fee on+  Bitcoin. A wallet sees a payment the moment the network does.+- **Extended public keys are first class.** Wallets are mostly xpubs, so the+  store derives their addresses up to an exact gap, remembers the derivations,+  and can keep large wallets ready in a Redis shared by several stores.+- **Bounded resources.** API limits stop one request from monopolising the+  server, slow event clients are dropped, and RocksDB memory has a budget.+- **Operable.** A health endpoint that tells a load balancer when an instance is+  behind, StatsD metrics for everything, configuration through flags or+  environment variables, and a Docker image.+- **Follow the work, not a node.** The store trusts the chain with the most+  proof of work, as reported by [haskoin-node](https://github.com/jprupp/haskoin-node),+  and checks what it needs to keep its indices consistent. It does not+  re-validate scripts; run it against the network you trust, or next to your+  own node with `--peer`.++## Features++| Area            | What you get                                                                                |+| --------------- | ------------------------------------------------------------------------------------------- |+| Networks        | Bitcoin and Bitcoin Cash: mainnet, testnets and regtest                                     |+| Addresses       | Balances, transactions and unspent outputs (UTXO) of any address, alone or many at once      |+| Xpubs           | Balances, transactions and UTXO of extended public keys, for legacy, nested SegWit and native SegWit addresses, with an optional Redis cache for large wallets |+| Blocks and txs  | Lookup by hash, height or time; raw serialized forms; transactions of a block               |+| Mempool         | Persistent, indexed by arrival time, with replace-by-fee on BTC and pruning of old entries   |+| Formats         | JSON, pretty JSON, or a compact binary encoding                                              |+| Live events     | New blocks and transactions as a JSON stream or over a WebSocket                            |+| Compatibility   | Endpoints in the style of Blockchain.info under `/blockchain/`                               |+| Operations      | Health check, StatsD metrics, RocksDB memory limits, Docker image                           |++### What is not included++- No wallet: it never holds keys or signs anything.+- No full validation of scripts and consensus rules. It follows the chain with+  the most work and indexes what peers send.+- No address or transaction search by anything but its hash; no indices on+  scripts without an address.++## Quick start++With [Docker](https://www.docker.com/):++```sh+git clone https://github.com/jprupp/haskoin-store.git+cd haskoin-store+docker build -t haskoin-store .+docker run -d --name haskoin-store -p 3000:3000 \+  -e NET=btc -e DISCOVER=true \+  -v haskoin-store-data:/home/haskoin/.haskoin-store \+  haskoin-store+curl 'http://localhost:3000/block/best?pretty=true'+```++The first sync downloads and indexes the whole chain, which takes a long+time and a lot of disk. The API answers while it syncs; `/health` reports+whether the node is in sync.++## Building from source++You need [Stack](https://docs.haskellstack.org/) and three C libraries:+[zlib](https://zlib.net/),+[libsecp256k1](https://github.com/bitcoin-core/secp256k1) and+[RocksDB](https://rocksdb.org/). Stack installs the right GHC by itself.++**Debian or Ubuntu**++```sh+sudo apt install git pkg-config zlib1g-dev libsecp256k1-dev librocksdb-dev haskell-stack+```++**Fedora**++```sh+sudo dnf install git pkgconf zlib-devel libsecp256k1-devel rocksdb-devel stack+```++Then build and install the `haskoin-store` binary to `~/.local/bin`:++```sh+git clone https://github.com/jprupp/haskoin-store.git+cd haskoin-store+stack build --copy-bins+~/.local/bin/haskoin-store --help+```++**Any distribution, with [Nix](https://nixos.org/)**: Stack fetches the C+libraries listed in `stack.yaml` from Nix instead of the system.++```sh+stack --nix build --copy-bins+```++## Running++```sh+haskoin-store --net btc --discover+```++Give it either `--discover` or at least one `--peer`; with neither it turns+discovery on by itself. Data goes in a directory named after the network inside+`--dir`, so one data directory can hold several networks.++For a big public deployment, also set `MALLOC_ARENA_MAX=2` (the Docker image+does) to keep glibc from fragmenting RocksDB's memory.++## Tutorial: following a wallet++This walks through what a wallet back end typically does with the API, on a+store that is already running on `localhost:3000`.++### 1. Wait for the store to be ready++```sh+curl -s -o /dev/null -w '%{http_code}\n' localhost:3000/health+```++`200` means the index is caught up with the network. While it syncs you get+`503`, and the body says which check fails.++### 2. Get the balance of a wallet++Most wallets are an extended public key. Ask for its summary, and say which+kind of addresses it derives (`legacy`, the default, `compat` for nested+SegWit, `segwit` for native SegWit):++```sh+curl 'localhost:3000/xpub/xpub6C...?derive=segwit&pretty=true'+```++The answer has confirmed and unconfirmed balances, and the index of the last+used receiving and change address, so the wallet knows where to derive the+next address.++If you get `derive` wrong, the store says so: a wallet with no transactions+of the kind you asked for, but some of another kind, gets a 400 whose message+names the right one, instead of an empty wallet that looks like a zero+balance.++### 3. List its transactions, page by page++```sh+curl 'localhost:3000/xpub/xpub6C.../transactions?derive=segwit&limit=50'+```++Results are newest first, unconfirmed ones on top. To get the next page, start+from the last transaction you received instead of using an offset:++```sh+curl 'localhost:3000/xpub/xpub6C.../transactions?derive=segwit&limit=50&offset=1&height=<last txid>'+```++`offset=1` skips the transaction you already have. Paging by starting point+stays fast however deep you go, and does not shift when new transactions+arrive.++### 4. Find spendable outputs++```sh+curl 'localhost:3000/xpub/xpub6C.../unspent?derive=segwit'+```++Each output comes with its derivation path, so the wallet knows which key signs+it.++### 5. Broadcast a payment++```sh+curl -X POST --data-binary @tx.hex localhost:3000/transactions+```++The body is the signed transaction in hex. The store sends it to its peers and+returns the transaction id. It appears in the mempool, and in the wallet's+history, as soon as a peer relays it back.++### 6. Stay up to date without polling++```sh+curl -N localhost:3000/events+```++Each line is a new block or a mempool change. A wallet back end keeps this open+and refreshes the affected wallets when an event arrives, instead of asking+every few seconds.++After a lost connection, come back with the hash of the last event you+heard, and the stream starts with what came after it:++```sh+curl -N "localhost:3000/events?after=9a5c…"+```++The server keeps its last 10,000 events. It answers 404 when it no longer+has the one you name: read your wallets again, and listen without `after`.++Over a WebSocket at the same path, a client can ask for only the+transactions it cares about. It sends one or more of these, as text:++```json+{"type":"watch","addresses":["bc1q…","1A1z…"]}+{"type":"watch","xpub":"xpub6C…","derive":"segwit"}+{"type":"filterload","filter":"b50f","functions":11,"tweak":0,"flags":0}+```++From then on it hears of a transaction only when it touches one of the+addresses, an address of the key, or matches the Bloom filter (BIP 37, as+SPV wallets use). Blocks are always sent. A key is followed as a wallet+uses it: the gap of unused addresses moves on as payments come. For a key+with several types of address, send one message for each type.++A client that connects and then sends its messages gets every transaction+until the first message is read. To start with a choice, put it in the URL,+of the WebSocket or of the HTTP stream:++```sh+curl -N "localhost:3000/events?quiet=true"                  # blocks only+curl -N "localhost:3000/events?addresses=bc1q…,1A1z…"+curl -N "localhost:3000/events?xpub=xpub6C…&derive=legacy,segwit"+```++`quiet=true` sends blocks alone until the client watches something; a+WebSocket client with a long list connects with it and sends the list in+messages. The [API reference](swagger.yaml) has every message and+parameter, and the binary form of each message.++### 7. Make big wallets fast++Start the store with `--redis` and a Redis (or Valkey) server. Xpubs with at+least `--redis-min-addrs` used addresses, or at least `--redis-min-txs`+transactions in their addresses, are then cached after their first+query and answered from Redis, transactions in full included. Several stores+can share one Redis: they keep each other's cache up to date, and a store that+has not imported a transaction yet still serves it from the cache. Add+`nocache=true` to a query to read the database instead.++Servers behind a load balancer should also be started with `--fleet`+([A fleet of servers](#a-fleet-of-servers)): they then answer every request+alike, and share a cache of xpubs that is built for that.++## Configuration++Every option has an environment variable that sets its default, which is handy+in containers. Booleans in variables accept `true`/`false`, `yes`/`no`,+`on`/`off` and `1`/`0`. `haskoin-store --help` shows the current values.++### Network and peers++| Option | Environment | Default | Meaning |+| --- | --- | --- | --- |+| `-n`, `--net` | `NET` | `bch` | Network: `btc`, `bch`, or a test network (see `--help`) |+| `-d`, `--dir` | `DIR` | `~/.haskoin-store` | Data directory |+| `--discover` | `DISCOVER` | off | Discover peers through DNS seeds and relayed addresses; on regtest it finds none, so name a `--peer` |+| `-p`, `--peer` | `PEER` | none | Connect to `HOST[:PORT]`; repeat for more, space-separated in `PEER` |+| `--max-peers` | `MAX_PEERS` | `20` | Most peers to connect to |+| `--peer-timeout` | `PEER_TIMEOUT` | `120` | Seconds before an unresponsive peer is dropped |+| `--max-peer-life` | `MAX_PEER_LIFE` | `172800` | Seconds after which a peer is dropped, so connections rotate |+| `--block-window` | `BLOCK_WINDOW` | `64` | Blocks to request from the syncing peer at a time |++### Mempool++| Option | Environment | Default | Meaning |+| --- | --- | --- | --- |+| `--no-mempool` | `NO_MEMPOOL` | off | Do not index unconfirmed transactions |+| `--sync-mempool` | `SYNC_MEMPOOL` | off | Ask each peer for its whole mempool once in sync |+| `--wipe-mempool` | `WIPE_MEMPOOL` | off | Empty the mempool area of the database at start |+| `--mempool-timeout` | `MEMPOOL_TIMEOUT` | `14` | Days after which unconfirmed transactions are deleted (0 = never) |++### Web server and limits++| Option | Environment | Default | Meaning |+| --- | --- | --- | --- |+| `--host` / `--port` | `HOST` / `PORT` | `*` / `3000` | Address the API listens on |+| `--noweb` | `NO_WEB` | off | Index only, no API |+| `--tls-cert` / `--tls-key` | `TLS_CERT` / `TLS_KEY` | none | Serve HTTPS with this certificate chain and private key, both files in PEM |+| `--no-xpub` | `NO_XPUB` | off | Disable the xpub endpoints |+| `--no-blockchain-info` | `NO_BLOCKCHAIN_INFO` | off | Disable the `/blockchain/` endpoints and price lookups |+| `--max-item-count` | `MAX_ITEM_COUNT` | `200000` | Most items in a list (0 = no limit) |+| `--max-full-item-count` | `MAX_FULL_ITEM_COUNT` | `5000` | Most full transactions in a list (0 = no limit) |+| `--max-offset` | `MAX_OFFSET` | `50000` | Largest `offset` (0 = no limit) |+| `--def-item-count` | `DEF_ITEM_COUNT` | `100` | Items in a list when no `limit` is given |+| `--xpub-gap` | `XPUB_GAP` | `32` | Unused addresses kept after the last used one of each chain of an xpub; payments further away are not seen |+| `--xpub-gap-init` | `XPUB_GAP_INIT` | `20` | Addresses looked at for an xpub with no transactions; an xpub with none used among them counts as unused |+| `--ticker-url` | `TICKER_URL` | Blockchain.info ticker | Source of current prices |+| `--price-history-url` | `PRICE_HISTORY_URL` | Blockchain.info price index | Source of historical prices |+| `--ticker-refresh` | `TICKER_REFRESH` | `90000000` | Microseconds between price updates |++### Redis cache++| Option | Environment | Default | Meaning |+| --- | --- | --- | --- |+| `--redis` | `REDIS` | off | Cache large xpubs in Redis |+| `--redis-url` | `REDIS_URL` | local Redis | Redis URL, such as `redis://host:6379` |+| `--redis-min-addrs` | `REDIS_MIN_ADDRS` | `100` | Cache xpubs with at least this many used addresses |+| `--redis-min-txs` | `REDIS_MIN_TXS` | `1000` | Or with at least this many transactions in their addresses |+| `--redis-max-xpubs` | `REDIS_MAX_XPUBS` | `10000` | Evict the least recently used xpubs past this many |++Every key of the cache starts with `hs2:<network>:`, so a Redis can be shared+by stores of several networks, and by stores of older versions, whose cache+uses other keys. `--redis-max-keys` and `--redis-sync-interval` of older+versions are gone.++### A fleet of servers++| Option | Environment | Default | Meaning |+| --- | --- | --- | --- |+| `--fleet` | `FLEET` | off | Answer as one of the servers that share the Redis of `--redis` |++Servers behind a load balancer import the same blocks and transactions at+slightly different times, so the same request can get different answers+depending on which of them it reaches. Start every one of them with+`--redis`, the same `--redis-url` and `--fleet`, and they all answer from+the same block and the same mempool:++- **The block** is the newest that every live server has. Each server+  imports blocks when it gets them and keeps its database as it was at each+  of its last six, and a request is answered from the block that the fleet+  agreed on. The fleet moves to a new block in one step, when the last+  server has it.+- **The mempool** is one pool in Redis, which every server offers its+  unconfirmed transactions to and all answer from. A transaction has one+  time of arrival there, so lists come in the same order everywhere.+- **A server that falls behind** answers 503 with `behind-fleet` rather+  than something that the others would not say, and stops counting after+  ten seconds without a report, so that it holds nobody up. A server whose+  health check fails stops reporting.+- **`/events` is the same stream on every server.** What changes the+  shared mempool or moves the fleet to a block writes an event to Redis+  in the same step, and every server sends those events, in that order,+  instead of what its own node saw. A block is told when the whole fleet+  can answer about it. Each server keeps the same last events, so a+  client that reconnects with `after=` may land on any of them.+- **Without Redis** each server answers from its own database, and+  `/health` says 203, as it does for the cache.+- **Large xpubs are cached for the whole fleet**, under the same+  `--redis-min-addrs`, `--redis-min-txs` and `--redis-max-xpubs` as the+  cache of a server alone, which should be the same on every server. An+  entry holds the chain's part of a wallet at the block that the fleet+  answers from, and the pool's part is kept by the pool itself, so a+  cached answer is the one that the database gives, on every server. The+  server that moves the fleet to a new block brings the cache there; in+  the moment before it has, and for a wallet that just gained or lost an+  address, requests are answered from the database. A reorganisation+  empties the cache. With it a request about a wallet of 464 addresses+  and 19,499 transactions takes 1.3 to 2.2 milliseconds, where it took 5+  to 38.++The rules, and the tests that hold Redis to them, are described in "A fleet+that answers as one" in `docs/layered-database.md`.++### Health check++| Option | Environment | Default | Meaning |+| --- | --- | --- | --- |+| `--max-lagging-blocks` | `MAX_LAGGING_BLOCKS` | `3` | Most blocks the index may lag behind the headers |+| `--max-pending-txs` | `MAX_PENDING_TXS` | `10000` | Most transactions waiting to be imported |+| `--min-peers` | `MIN_PEERS` | `1` | Fewest connected peers |+| `--block-timeout` | `BLOCK_TIMEOUT` | `14400` | Most seconds since the last block |+| `--tx-timeout` | `TX_TIMEOUT` | `1800` | Most seconds since the last transaction |+| `--health-check-interval` | `HEALTH_CHECK_INTERVAL` | `30` | Seconds between checks |++### Metrics and logging++| Option | Environment | Default | Meaning |+| --- | --- | --- | --- |+| `--statsd` | `STATSD` | off | Send metrics to StatsD |+| `--statsd-host` / `--statsd-port` | `STATSD_HOST` / `STATSD_PORT` | `localhost` / `8125` | StatsD server |+| `--statsd-prefix` | `STATSD_PREFIX` | `haskoin_store` | Prefix of every metric |+| `--statsd-no-events` | `STATSD_NO_EVENTS` | off | Do not send individual samples |+| `--statsd-aggregates` | `STATSD_AGGREGATES` | off | Also send aggregated statistics every second |+| `--debug` | `DEBUG` | off | Verbose logging |++### RocksDB++| Option | Environment | Default | Meaning |+| --- | --- | --- | --- |+| `--no-bloom` | `BLOOM` | on | No Bloom filters on what is read by key; `BLOOM=false` is the same |+| `--max-open-files` | `MAX_OPEN_FILES` | `2000` | Open files; bounds index and filter memory (0 = unlimited) |+| `--write-buffer-mb` | `WRITE_BUFFER_MB` | `256` | Memtables across all column families (0 = unlimited) |+| `--max-wal-mb` | `MAX_WAL_MB` | `1024` | Write-ahead log size that forces flushes (0 = RocksDB default) |+| `--block-cache-mb` | `BLOCK_CACHE_MB` | `1024` | Shared block cache, holding index and filter blocks too (0 = RocksDB default) |++Memory used by RocksDB is roughly the block cache plus the memtables, on top of+the Haskell heap. `/dbstats` shows where it goes.++Bloom filters are on by default. They let RocksDB skip the table files that+cannot have a key, which is most of what a lookup of a transaction, an+output or a balance would otherwise read.++- **What has them:** the data that is read by key: transactions, spenders,+  unspent outputs, blocks, balances and the mempool. The two indices by+  address, which are scanned, do not.+- **What they cost:** about 1.25 bytes for each key, held in the block+  cache. On a chain with billions of keys that is gigabytes: size+  `--block-cache-mb` for it, or turn them off with `--no-bloom`.+- **An existing database** gets filters for the table files that are+  written from now on, so their memory comes gradually, as RocksDB+  compacts. The migration to database version 22 rewrites nearly every+  transaction, and with it most of the filters arrive at once.+- **`--bloom`** was the way to turn them on, and is still accepted. Up to+  version 1.8.1 it built no filters at all on RocksDB 6.24 or later,+  because of a wrong type in the binding, fixed in+  `rocksdb-haskell-jprupp` 2.6.3.++## API++Full reference: [`swagger.yaml`](swagger.yaml), also served at+[api.haskoin.com](https://api.haskoin.com/). It explains the conventions of+the API (amounts, ordering, paging, formats, errors, unconfirmed transactions+and reorganisations) and the reasons for them, as well as every endpoint.++```sh+curl localhost:3000/address/bc1q…/balance+curl localhost:3000/xpub/xpub6C…/balances+curl localhost:3000/transaction/<txid>+curl -N localhost:3000/events              # stream of new blocks and txs+```++- The root of the server, `/`, is the reference of its API as a page that+  can try requests, and `/swagger.yaml` the file behind it.+- Add `?pretty=true` for indented JSON.+- Send `Accept: application/octet-stream` for the binary format: see+  [Binary format](#binary-format) below.+- Lists are paged with `limit` and `offset`, newest first. `height` begins+  a page at a transaction hash, a block hash, a block height or a Unix time+  instead.+- Parameters are checked: a parameter that an endpoint does not accept, or+  a value it cannot read, gives 400 with a message that names it, instead of+  being ignored. The Blockchain.info endpoints accept `format=json`, which+  clients written for them often send.+- Errors come back with an HTTP status, in the format that was asked for:+  400 for bad input, 404 for unknown things, 409 when a short transaction+  number is ambiguous, 413 for requests that ask for too much.++### HTTP and HTTPS++- **`HEAD` and `OPTIONS`** are answered, the second with what a browser+  may send from another site.+- **Answers are compressed** for a client that sends+  `Accept-Encoding: gzip`, but for the stream of events.+- **Long lists go in a body.** The endpoints that take a list of+  addresses, blocks, heights or transactions also take it under `POST`,+  as a form (`application/x-www-form-urlencoded`), so that it does not+  have to fit in a URL. `/transactions` does not: `POST` there publishes+  a transaction.+- **A 503 says when to try again**, in `Retry-After`.+- **HTTPS is optional and off by default.** With `--tls-cert` and+  `--tls-key` the server speaks HTTPS on its port, and nothing else. A+  public service usually has a proxy in front that holds the certificate;+  this is for a server by itself. The files are read at start, so a+  renewed certificate takes a restart.++### Binary format++Every endpoint but those under `/blockchain/` and `/dbstats` also answers+in a compact binary format, which is smaller and faster to decode than+JSON. The Haskell client in `haskoin-store-data` speaks it, and+`swagger.yaml` has a section, "Binary format", that is enough to write a+client in another language: the primitives, the layout of every type, and+what each endpoint returns.++- **Ask for it** with `Accept: application/octet-stream`. In a list of+  types it wins when its `q` weight is above 0 and no less than that of+  `application/json`; a wildcard or no header gives JSON. The answer has+  `Content-Type: application/octet-stream`, and every answer has+  `Vary: Accept`.+- **Errors come in binary too**: one byte that says which error, and a+  message or a list of transactions for those that have one. Up to+  version 1.8.1 they were JSON whatever was asked, so a client that also+  talks to older servers reads the `Content-Type` of an error.+- **Lists start with their length**, the raw lists of blocks and+  transactions included, which up to version 1.8.1 had none.+- **`/events` is the exception**: a stream of events of 33 bytes each,+  with no count, for as long as the connection is open.+- **Over the WebSocket** binary is binary frames, for the events and for+  the commands that choose transactions. The server answers in the kind+  of frame that it was last sent.+- Everything is big-endian, except what is copied from the peer-to-peer+  protocol, such as a whole transaction or block.+++### Endpoints at a glance++| Area | Endpoints |+| --- | --- |+| Blocks | `/block/best`, `/block/latest`, `/block/:hash`, `/blocks`, `/block/height/:h`, `/block/heights`, `/block/time/:t`, `/block/mtp/:t`, each with a `/raw` variant where it makes sense |+| Transactions | `/transaction/:txid`, `/transactions`, `/transactions/block/:hash`, `/transaction/:txid/after/:height`, `POST /transactions`, `/raw` variants |+| Mempool | `/mempool` |+| Addresses | `/address/:addr/{transactions,transactions/full,balance,unspent}` and the multi-address forms `/address/{transactions,transactions/full,balances,unspent}?addresses=…` |+| Xpubs | `/xpub/:xpub`, `/xpub/:xpub/{transactions,transactions/full,balances,unspent}`, `DELETE /xpub/:xpub` (drop from cache); `derive=legacy\|compat\|segwit` picks the address type (`normal`, the old name of `legacy`, still works) |+| Network | `/peers`, `/health` (503 when unhealthy, 203 without its Redis), `/dbstats` |+| Events | `/events`, as JSON lines over HTTP or as a WebSocket; `after=` resumes, and a WebSocket client can filter by address, xpub or Bloom filter |+| Blockchain.info | `/blockchain/multiaddr`, `/blockchain/balance`, `/blockchain/rawaddr/:addr`, `/blockchain/unspent`, `/blockchain/rawtx/:txid`, `/blockchain/rawblock/:hash`, `/blockchain/latestblock`, `/blockchain/unconfirmed-transactions`, `/blockchain/export-history`, `/blockchain/q/…` |++Turn off the xpub endpoints with `--no-xpub` and the Blockchain.info ones+with `--no-blockchain-info`.++## How it works++```+peers ──► haskoin-node ──► block store ──► RocksDB ──► web server ──► clients+                │               │             ▲+                └─ headers      └─ events ──► cache ◄──► Redis ◄──► other stores+```++- **haskoin-node** keeps a pool of peers and syncs block headers.+- The **block store** (`Haskoin.Store.BlockStore`) downloads blocks from one+  peer at a time and mempool transactions from all of them, and imports them+  with the rules in `Haskoin.Store.Logic.Layered`. Each import is one atomic+  RocksDB batch, so a crash never leaves a half-written block.+- Reorganisations are handled by reverting blocks until the best block is+  back on the main chain of headers.+- The **cache** (`Haskoin.Store.Cache`) keeps xpubs with many used addresses+  or many transactions in Redis: their balances, transactions, unspent+  outputs and the bodies of their transactions. It never computes an entry+  from what it guesses changed: when a block, a mempool transaction or a+  reorganisation touches a cached xpub, the store reads from RocksDB, with+  the same queries that answer uncached requests, the transactions that+  changed and the outputs they create and spend, and writes the differences+  in one Lua script, together with every transaction body the entry needs+  that Redis does not have yet (bodies it has, such as those of the+  transactions a block confirms, are rewritten just before, in small+  scripts, so that a large block does not hold Redis for long).+  The time of a reload no longer grows with the size of the xpub. The store+  reads the whole xpub instead when it first caches it, and whenever it+  cannot be sure the entry has every earlier change: when the entry is more+  than a block behind, when an address with transactions enters or leaves+  the gap, when the balance of an address that no change touched differs+  from Redis's and its newest transactions do not explain why, and on a+  schedule, every five to ten minutes. The interval depends on the xpub, so+  that xpubs cached together are not all read whole at the same moment,+  which would hold them all in memory at once. Under load the database is+  often a little ahead of the events being applied, so a reload first adds+  the transactions that the database has for such an address since the+  block the entry was written at, and the mempool transactions of the entry+  that have since been confirmed or dropped. The script+  writes nothing if the entry changed since it was read, so the entry is+  always what the database says; and Redis, which answers one client at a+  time, is held for the size of the change, not of the xpub. A page of a+  cached xpub is one round trip to Redis. The Lua scripts are in+  `src/Haskoin/Store/Cache/lua/`, one file each, embedded in the library+  when it is compiled. Stores sharing a Redis claim each change, so it+  is applied once, by the first store to see it; a store never replaces an+  entry that knows more than its database (a higher block, or a transaction+  it has not seen), and a store that is catching up does no cache work.+  Without Redis, xpub queries read RocksDB directly. The ways an entry can+  still come to differ from the database unnoticed, such as a missed spend of+  an output of a cached transaction, and what would rule them out, are in+  `docs/cache-consistency.md`.+- A **fleet** (`Haskoin.Store.Fleet.Member`, with `--fleet`) has a cache of+  xpubs of another kind (`Haskoin.Store.Fleet.XPub`), because its servers+  do not answer from their own databases but from one block and one shared+  mempool. An entry has two layers: what the chain has for the addresses of+  the xpub at the block that the fleet answers from, and what the shared+  pool has for them, which the script of the pool writes as it changes the+  pool. The first follows from the chain, so any server writes the same,+  and nothing has to be claimed or repaired: the server that moves the+  fleet to a new block applies what the block did, and a cache that was+  not brought along is emptied. `docs/layered-database.md` has the design,+  the tests and the measurements.+- The **web server** (`Haskoin.Store.Web`) answers from RocksDB, and from the+  cache for cached xpubs and for transaction bodies that Redis has.++Xpub addresses are derived up to an exact gap: every address up to the last+used one of each chain, then exactly `--xpub-gap` unused ones, and always the+first `--xpub-gap-init` addresses. A payment further away is not seen, as a+wallet following BIP 44 with the same gap would not see it; raise the gap to+see it.++An xpub does not record which kind of address its wallet derives, so a+request names it with `derive`. When an xpub has no transactions with the+kind asked for, the store reads the first `--xpub-gap-init` addresses of the+other kinds, and if one of them is used it refuses the request with 400 and+names that kind: an empty answer would look like a wallet with no money, a+mistake that is easy to miss. The extra reads happen only for wallets that+look empty, and not at all on networks without SegWit, which have one kind of+address.++The database layout is described in `Haskoin.Store.Database.Types`.++Unconfirmed transactions are kept in an area of their own, and applied on+top of the chain when a query is answered, so that nothing the mempool does+changes the chain's data. This is version 21 of the database, the layered+layout of `docs/layered-database.md`, which has the reasons, the+measurements and the evidence. Up to version 20 they were written into the+same indices as the block chain. A database of version 20 is migrated the+first time this version of the store opens it: its mempool is moved to the+new area, in batches that can be interrupted, and the log says how many+transactions were moved. **There is no way back**: an older store refuses+the database afterwards, so take a copy of a production database first.++Version 22 makes the record of a confirmed transaction something that is+written once, by the block that confirms it, and the same on every server+that has that block.++- **Its time is the time of its block.** Until then a server that had seen+  a transaction in its mempool kept the time of that when a block+  confirmed it, so two servers gave two times for one confirmed+  transaction, and a server that synced later gave a third.+- **The spenders of its outputs are under keys of their own**, sixteen+  outputs to a key. Until then they were in the record, which was read and+  written whole again each time a block spent one of its outputs. Reading+  a confirmed transaction in full now takes a second key: 18 microseconds+  where it took 11.5, measured, and 0.8 milliseconds for a hundred where+  it took 0.4. Lists by address and by xpub do not read records and cost+  the same.++A transaction that leaves the mempool unconfirmed, because it was replaced,+double-spent by a block or too old, is no longer kept: the store forgets+it, and the API answers 404 for it, as it already did. Until version 22+its body stayed in the database, marked as deleted, for ever. The bodies+that are there in a migrated database are left where they are and+ignored, since finding them would mean reading every transaction.++A database of version 21, or of version 20 once its mempool is moved, is+brought to this the first time this version of the store opens it.+**The store reads every transaction that it has, and writes nearly every+one again, before it starts**: nothing lists the records that need it, and+most have a spender in them. The log says how far it is every million+transactions. A database of Bitcoin has more than a billion, so plan for+many hours, and for room on the disk for a second copy of the transactions+until RocksDB has compacted the first away. It can be interrupted, and+starts again from the first transaction, skipping what is done. As before,+**there is no way back**: take a copy first.+The parts:++- the migration from version 20 (`Haskoin.Store.Database.Migrate`);+- the keys that a mempool transaction writes, the summary that is kept for+  each address they touch, and what the mempool does to the balance and the+  unspent outputs of an address (`Haskoin.Store.Database.Mempool`);+- the rules that import blocks and transactions in that layout+  (`Haskoin.Store.Logic.Layered`), which the tests hold to the results of+  the rules of version 20, operation by operation and key by key;+- the queries over both areas (`Haskoin.Store.Database.Query.Layered`), each+  on a snapshot of the database, one for each request of the web server,+  which the tests hold to the answers of the queries of version 20, page by+  page.++The rules and queries of version 20 (`Haskoin.Store.Logic`,+`Haskoin.Store.Database.Query`) stay in the library for those tests, and+nothing in the store calls them.++## Monitoring++- **`/health`** returns the result of the latest check as JSON, with status 200+  when every check passes and 503 otherwise. The checks are: index lag behind+  the headers, age of the last block and of the last transaction, transactions+  waiting to be imported, and connected peers. Point your load balancer at it.+- **Without Redis** a store that has a cache goes on answering, from its own+  database, and `/health` says 203 instead of 200, with the header+  `Haskoin-Degraded: redis` and `"redis": {"ok": false}` in its JSON. A request waits at most two seconds for Redis+  before the store gives up on it, and the store tries again at every health+  check. 203 is in the range that load balancers take for healthy, so the+  store stays in rotation; alert on anything but 200 to hear of it.+- **StatsD**: with `--statsd`, the store reports the timing of every kind of+  request (`web.*`), database reads (`db.*`), cache hits, misses, reloads and+  full reloads (`cache.*`),+  and block store gauges such as block and header heights and mempool size+  (`store.*`). A server of a fleet reports what it does with the fleet's+  cache of xpubs under `fleet.cache.*`:++  | Metric | Kind | Meaning |+  | --- | --- | --- |+  | `fleet.cache.hits` | counter | Requests about an xpub whose balances the cache gave |+  | `fleet.cache.misses` | counter | Requests about an xpub that the database answered |+  | `fleet.cache.builds` | counter | Entries that this server built, or added addresses to |+  | `fleet.cache.build` | timing | Milliseconds to build an entry |+  | `fleet.cache.abandoned` | counter | Builds given up because the fleet or the entry moved on |+  | `fleet.cache.evictions` | counter | Entries deleted for being the least recently used |+  | `fleet.cache.emptied` | counter | Times that this server emptied the cache |+  | `fleet.cache.xpubs` | gauge | Xpubs in the cache, whichever server cached them |+  | `fleet.cache.addresses` | gauge | Addresses that those xpubs have between them |++  Misses include small xpubs, which are never cached, so the share of hits+  says how much of the load the cache takes, not how well it holds what it+  has. `fleet.cache.emptied` should move only at a reorganisation or when a+  server stops at the wrong moment: the log of the server says why each+  time, at the level of information. With `--debug` it also has a line for+  each entry built and each block that the cache was brought to.+- **`/dbstats`** returns RocksDB memory use per column family and RocksDB's own+  statistics, as text.+- **In a fleet** (`--fleet`), any request can be answered 503 with+  `{"error": "behind-fleet"}` by a server that does not have the block that+  the fleet answers from. Have the load balancer try another server, and+  alert if one keeps saying it: its `/health` will say why.+- **Memory.** The resident memory of the store is its GHC heap plus RocksDB:+  the block cache and memtables fill up to `--block-cache-mb` and+  `--write-buffer-mb`, then stay there. The executable is built with+  `--disable-delayed-os-memory-return`, so memory that the heap no longer+  needs goes back to the system at once. Without it, Linux keeps counting+  returned memory as resident (`LazyFree` in `/proc/<pid>/smaps`) until it+  needs it, and the heap looks as if it never shrinks after a peak, such as+  the full reload of a large cached xpub. Pass `+RTS+  --disable-delayed-os-memory-return -RTS` yourself if you build the store+  some other way. To tell such memory from a leak, see+  `stress/results/2026-11-09-memory/README.md`: how to read `smaps`, and a+  heap profile of a busy store, with live data flat at about 10 MiB.+- **Logs** go to standard error, one line per message, tagged with the+  component, such as `[Info#BlockStore] Best block: ...`.++## Using it from Haskell++Query a running server with `haskoin-store-data`:++```haskell+import Control.Monad.Except (runExceptT)+import Data.Default (def)+import Haskoin (btc, textToAddr, withContext)+import Haskoin.Store.WebClient++main :: IO ()+main = withContext $ \ctx -> do+  let cfg = def {net = btc} -- https://api.haskoin.com/btc/...+      Just addr = textToAddr btc "bc1q…"+  r <- runExceptT $ apiCall ctx cfg (GetAddrBalance addr)+  print r+```++The client appends the network name to `host`, as in the layout of+api.haskoin.com. To reach your own server, put it behind a proxy that maps+`/btc/` (or the name of your network) to it.++Or embed the indexer in your own program with `haskoin-store`: `withStore`+starts the node, the block store and the cache from a `StoreConfig`, and the+functions of `Haskoin.Store.Database.Query.Layered` query its database in+`IO`, on a view of it taken with `withView`, such as+`withView store.db (\v -> getBalance v addr)`.+Subscribe to `Store.pub` to receive `StoreEvent`s.++Every module has Haddock documentation; build it with `stack haddock`.++## Packages++Haskoin Store is two Haskell packages, each in its own repository:++- **`haskoin-store`**, this repository: the indexer, the web server and the+  executable.+- **[`haskoin-store-data`](https://github.com/jprupp/haskoin-store-data)**:+  the data types, their JSON and binary encodings, and a client for the API+  (`Haskoin.Store.WebClient`). Use this one to talk to a Haskoin Store server+  from Haskell. It is small to depend on, as it has no database or node.++They were released together from this repository up to 1.8.1. A change to+the wire format is made in haskoin-store-data, released, and then picked up+here; `swagger.yaml` stays here and describes the server.++## Development++```sh+stack build --test --no-run-tests+stack test                     # includes a 6000-block heap-growth test+scripts/format                 # ormolu, the code style used here+scripts/bench-web.sh           # load-test the API with wrk+scripts/stress-cache.sh        # stress the Redis cache, with a report+scripts/check-local-deps.sh    # what CI checks: no local paths in stack.yaml+scripts/install-system-libraries.sh  # the C libraries, on Ubuntu or Debian+```++`stack.yaml` must pin released versions of every dependency. Pointing+`extra-deps` at a sibling checkout such as `../haskoin-core` is fine for+local testing, but CI rejects it.++The tests use [hspec](https://hspec.github.io/). The unit tests need no network+access. The integration tests (`Haskoin.Store.IntegrationSpec`) run regtest+chains of Bitcoin Core and Bitcoin Cash Node, and Valkey, in+[Podman](https://podman.io/) containers, and check the store against them: the+exact gap, spends and change, mempool order, multiaddr counts, reorganisations,+and that the cache answers exactly like the database on every xpub endpoint.++- They are skipped if Podman cannot run containers, or if+  `HASKOIN_STORE_NO_PODMAN` is set.+- The first run builds the image `localhost/haskoin-store-test`, which needs+  network access: Ubuntu 26.04 pinned by digest, Valkey from a fixed snapshot of+  the Ubuntu archive, and the release tarballs of both nodes, checked against+  SHA-256 hashes pinned in `test/Haskoin/Store/Integration/Image.hs`. No+  third-party image is used.+- Containers are removed when the tests end, data lives in tmpfs mounts and+  under `/dev/shm`, and every container has a memory limit.+- Set `HASKOIN_TEST_LOG=1` to print the log of the store under test.++The test suite runs with at most four cores and a 4 GiB heap, so that a+runaway test fails instead of exhausting the machine.++`package.yaml` is the source of the package description. Change it rather+than `haskoin-store.cabal`, which+[hpack](https://github.com/sol/hpack) generates.++### Stress test of the cache++`scripts/stress-cache.sh [OUT_DIR] [-- OPTIONS]` measures the cache while it+works hard. It runs a regtest chain of Bitcoin Core, Valkey and wrk in Podman+containers from the test image, and the `haskoin-store` binary with the+cache on. A few wallets, whose keys the stress program (`stress/Main.hs`)+holds, make hundreds of thousands of transactions between them, signed by+the program and sent to the node in batches, with a block after each batch.+Every transaction changes two cached xpubs, so the cache reloads them over+and over, while wrk asks the server about the xpubs on five endpoints+(summary, balances, transactions, unspent outputs and multiaddr) the whole+time. At the end it compares the cached answers with those of the database,+and `scripts/stress-report.py` writes `OUT_DIR/report.md`: how fast the+transactions went in, how many reloads the cache did, how many of them read+the whole xpub, and how long they took as the xpubs grew, how far the server+fell behind, its CPU and memory, the+memory of Valkey, the commands that blocked Redis for 10 ms or more (from+its slow log, by script, with the number of arguments of each, which grows+with the size of a commit), and the latencies of each endpoint over time.++- The defaults are 200,000 transactions between 4 wallets of 200 receiving+  and 200 change addresses, 2,000 per block; pass `-- --help` for the+  options, and `-- --txs 20000` for a run of a few minutes. A default run+  takes about half an hour and up to about 7 GiB of memory.+- The raw measurements stay in `OUT_DIR` as JSON lines (`rounds.jsonl`,+  `samples.jsonl`, `wrk.jsonl`), with `summary.json` and the log of the+  server.+- No container is left behind, even if the script is interrupted or+  killed: the program removes its containers when it ends, and the script+  removes any that carry its label and stops the server.+- Results of past runs are kept in `stress/results/<date>/` for comparison:+  the report, the measurements (without the server log), and a README with+  the commits, the machine and what the run found. A new run worth keeping+  goes there too.+- To look for a leak, run the server with a heap profile by type, which needs+  no profiling build, and summarise it: the live heap over time and the types+  of closure that grew most.++  ```sh+  scripts/stress-cache.sh /tmp/run -- --txs 60000 \+    --server-rts '-N4 -M2g -T -hT -i10 -po/tmp/heap'+  scripts/heap-profile-summary.py /tmp/heap.hp+  ```++### Measurements for the layered database++`docs/layered-database/` has the programs behind the numbers in+`docs/layered-database.md`, with their output. They are not part of the+package; each says in its header how to build and run it.++- `MigrateBench.hs` times the migration to the layered layout against+  wiping the same mempool.+- `QueryCost.hs` writes the same synthetic history into a database of each+  layout, with each set of rules, and times both: importing blocks and+  mempool transactions, and every kind of query, for addresses and wallets+  that have nothing, a little and much in the mempool. It checks that both+  databases give the same answers.+- `FleetCost.hs` asks one store the same questions alone and as a member+  of a fleet, with its mempool in a pool in Redis, checks that the answers+  are the same, and times both. It asks about wallets a third way too,+  through the fleet's cache of xpubs.++### Contributing++Pull requests are welcome.++- Format the code with `scripts/format` (Ormolu), and keep every line of+  code shorter than 120 columns.+- Put scripts in files of their own rather than embedding them: Lua for+  Redis under `src/Haskoin/Store/Cache/lua/`, shell under `scripts/`, called+  from CI rather than written into the workflow.+- Add tests for the behaviour you change.+- Describe the change under a new version in `CHANGELOG.md`, which follows+  [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).+- Update `swagger.yaml` when you change the API.++## Related projects++- [haskoin-core](https://github.com/jprupp/haskoin-core) has the Bitcoin data+  types and protocol.+- [haskoin-node](https://github.com/jprupp/haskoin-node) connects to peers and+  syncs headers.+- [nqe](https://github.com/jprupp/nqe),+  [rocksdb-haskell-jprupp](https://github.com/jprupp/rocksdb-haskell),+  [rocksdb-query](https://github.com/jprupp/rocksdb-query) and+  [statsd-rupp](https://github.com/jprupp/statsd-rupp) provide concurrency,+  storage and metrics.++## License++Haskoin Store is released under the+[MIT License](https://github.com/jprupp/haskoin-store/blob/master/haskoin-store/LICENSE).

@@ -1,2 +0,0 @@-import Distribution.Simple-main = defaultMain

@@ -1,20 +1,10 @@ {-# LANGUAGE ApplicativeDo #-} {-# LANGUAGE CPP #-}-{-# LANGUAGE DuplicateRecordFields #-}-{-# LANGUAGE FlexibleContexts #-}-{-# LANGUAGE ImportQualifiedPost #-}-{-# LANGUAGE LambdaCase #-}-{-# LANGUAGE OverloadedRecordDot #-}-{-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE RecordWildCards #-}-{-# LANGUAGE TupleSections #-}-{-# LANGUAGE NoFieldSelectors #-}  module Main (main) where  import Control.Monad (when)-import Control.Monad.Cont (ContT (ContT), runContT)-import Control.Monad.Trans (lift) import Data.Char (toLower) import Data.Default (Default (..)) import Data.List (intercalate)@@ -34,7 +24,6 @@     withLogger,     withSource,   )-import Haskoin.Store.Common (loopForever) import Haskoin.Store   ( Store (..),     StoreConfig (..),@@ -43,9 +32,11 @@     runWeb,     withStore,   )+import Haskoin.Store.Common (loopForever) import NQE (receive, withSubscription) import Options.Applicative-  ( Parser,+  ( Mod,+    Parser,     auto,     execParser,     flag,@@ -53,6 +44,7 @@     header,     help,     helper,+    hidden,     info,     long,     many,@@ -79,7 +71,6 @@     withStats,   ) import Text.Read (readMaybe)-import UnliftIO (MonadIO) import UnliftIO.Directory   ( createDirectoryIfMissing,     getAppUserDataDirectory,@@ -112,10 +103,11 @@     minPeers :: !Int,     webLimits :: !WebLimits,     redis :: !Bool,+    fleet :: !Bool,     redisURL :: !String,     redisMinAddrs :: !Int,-    redisMaxKeys :: !Integer,-    redisSyncInterval :: !Int,+    redisMinTxs :: !Int,+    redisMaxXPubs :: !Int,     noMempool :: !Bool,     wipeMempool :: !Bool,     syncMempool :: !Bool,@@ -136,6 +128,8 @@     noBlockchainInfo :: !Bool,     noXPub :: !Bool,     healthCheckInterval :: !Int,+    tlsCert :: !String,+    tlsKey :: !String,     bloom :: !Bool,     noWeb :: !Bool,     maxOpenFiles :: !Int,@@ -145,13 +139,13 @@   }  -- | Read an environment variable with a parser, or return the default if it is unset or invalid.-env :: (MonadIO m) => String -> a -> (String -> Maybe a) -> m a+env :: String -> a -> (String -> Maybe a) -> IO a env e d p = do   ms <- lookupEnv e   return $ fromMaybe d $ p =<< ms  -- | Default configuration, taken from environment variables where set.-defConfig :: (MonadIO m) => m Config+defConfig :: IO Config defConfig = do   dir <-     getDir@@ -177,14 +171,16 @@     getWebLimits   redis <-     env "REDIS" False parseBool+  fleet <-+    env "FLEET" False parseBool   redisURL <-     env "REDIS_URL" "" pure   redisMinAddrs <-     env "REDIS_MIN_ADDRS" 100 readMaybe-  redisMaxKeys <--    env "REDIS_MAX_KEYS" 100000000 readMaybe-  redisSyncInterval <--    env "REDIS_SYNC_INTERVAL" 30 readMaybe+  redisMinTxs <-+    env "REDIS_MIN_TXS" 1000 readMaybe+  redisMaxXPubs <-+    env "REDIS_MAX_XPUBS" 10000 readMaybe   noMempool <-     env "NO_MEMPOOL" False parseBool   wipeMempool <-@@ -226,9 +222,13 @@   healthCheckInterval <-     env "HEALTH_CHECK_INTERVAL" 30 readMaybe   bloom <--    env "BLOOM" False parseBool+    env "BLOOM" True parseBool   noWeb <-     env "NO_WEB" False parseBool+  tlsCert <-+    env "TLS_CERT" "" pure+  tlsKey <-+    env "TLS_KEY" "" pure   maxOpenFiles <-     env "MAX_OPEN_FILES" 2000 readMaybe   writeBufferMB <-@@ -283,6 +283,12 @@   "0" -> Just False   _ -> Nothing +-- | The help of an option, with the environment variable that sets its+-- default: every option has one, and someone who runs the store in a+-- container reads the help to find its name.+helpEnv :: String -> String -> Mod f a+helpEnv name text = help (text <> " [env: " <> name <> "]")+ -- | Command line parser, with defaults from the given configuration. config :: Config -> Parser Config config c = do@@ -291,21 +297,21 @@       metavar "DIRECTORY"         <> long "dir"         <> short 'd'-        <> help "Data directory"+        <> helpEnv "DIR" "Data directory"         <> showDefault         <> value c.dir   host <-     strOption $       metavar "ADDRESS"         <> long "host"-        <> help "Network address to bind"+        <> helpEnv "HOST" "Network address to bind"         <> showDefault         <> value c.host   port <-     option auto $       metavar "PORT"         <> long "port"-        <> help "REST API listening port"+        <> helpEnv "PORT" "REST API listening port"         <> showDefault         <> value c.port   net <-@@ -313,21 +319,21 @@       metavar netNames         <> long "net"         <> short 'n'-        <> help "Network to connect to"+        <> helpEnv "NET" "Network to connect to"         <> showDefault         <> value c.net   discover <-     flag c.discover True $       long "discover"-        <> help "Peer discovery"+        <> helpEnv "DISCOVER" "Peer discovery"   peers <-     fmap (mappend c.peers) $       many $-        option auto $+        strOption $           metavar "HOSTNAME[:PORT]"             <> long "peer"             <> short 'p'-            <> help "Network peer (as many as required)"+            <> helpEnv "PEER" "Network peer (as many as required)"   version <-     switch $       long "version"@@ -336,270 +342,308 @@   debug <-     flag c.debug True $       long "debug"-        <> help "Show debug messages"+        <> helpEnv "DEBUG" "Show debug messages"   maxPendingTxs <-     option auto $       metavar "COUNT"         <> long "max-pending-txs"-        <> help "Maximum pending txs to fail health check"+        <> helpEnv "MAX_PENDING_TXS" "Maximum pending txs to fail health check"         <> showDefault         <> value c.maxPendingTxs   maxLaggingBlocks <-     option auto $       metavar "COUNT"         <> long "max-lagging-blocks"-        <> help "Maximum number of unindexed blocks"+        <> helpEnv "MAX_LAGGING_BLOCKS" "Maximum number of unindexed blocks"         <> showDefault         <> value c.maxLaggingBlocks   minPeers <-     option auto $       metavar "COUNT"         <> long "min-peers"-        <> help "Minimum number of connected peers for health check"+        <> helpEnv "MIN_PEERS" "Minimum number of connected peers for health check"         <> showDefault         <> value c.minPeers   noWeb <-     flag c.noWeb True $       long "noweb"-        <> help "Disable web server"+        <> helpEnv "NO_WEB" "Disable web server"   webLimits <- do     maxItemCount <-       option auto $         metavar "COUNT"           <> long "max-item-count"-          <> help "Hard limit for simple listings (0 = inf)"+          <> helpEnv "MAX_ITEM_COUNT" "Hard limit for simple listings (0 = inf)"           <> showDefault           <> value c.webLimits.maxItemCount     maxFullItemCount <-       option auto $         metavar "COUNT"           <> long "max-full-item-count"-          <> help "Hard limit for full listings (0 = inf)"+          <> helpEnv "MAX_FULL_ITEM_COUNT" "Hard limit for full listings (0 = inf)"           <> showDefault           <> value c.webLimits.maxFullItemCount     maxOffset <-       option auto $         metavar "OFFSET"           <> long "max-offset"-          <> help "Hard limit for offsets (0 = inf)"+          <> helpEnv "MAX_OFFSET" "Hard limit for offsets (0 = inf)"           <> showDefault           <> value c.webLimits.maxOffset     defItemCount <-       option auto $         metavar "COUNT"           <> long "def-item-count"-          <> help "Soft default limit (0 = inf)"+          <> helpEnv "DEF_ITEM_COUNT" "Soft default limit (0 = inf)"           <> showDefault           <> value c.webLimits.defItemCount     xpubGap <-       option auto $         metavar "GAP"           <> long "xpub-gap"-          <> help "Max gap for xpub queries"+          <> helpEnv "XPUB_GAP" "Max gap for xpub queries"           <> showDefault           <> value c.webLimits.xpubGap     xpubGapInit <-       option auto $         metavar "GAP"           <> long "xpub-gap-init"-          <> help "Max gap for empty xpubs"+          <> helpEnv "XPUB_GAP_INIT" "Max gap for empty xpubs"           <> showDefault           <> value c.webLimits.xpubGapInit     blockTimeout <-       option auto $         metavar "SECONDS"           <> long "block-timeout"-          <> help "Block lag health timeout"+          <> helpEnv "BLOCK_TIMEOUT" "Block lag health timeout"           <> showDefault           <> value c.webLimits.blockTimeout     txTimeout <-       option auto $         metavar "SECONDS"           <> long "tx-timeout"-          <> help "Last tx received health timeout"+          <> helpEnv "TX_TIMEOUT" "Last tx received health timeout"           <> showDefault           <> value c.webLimits.txTimeout     return WebLimits {..}   redis <-     flag c.redis True $       long "redis"-        <> help "Redis cache for xpub data"+        <> helpEnv "REDIS" "Redis cache for xpub data"+  fleet <-+    flag c.fleet True $+      long "fleet"+        <> helpEnv+          "FLEET"+          ( unwords+              [ "Answer as one of the servers that share the Redis of --redis:",+                "all from the same block and the same mempool"+              ]+          )   redisURL <-     strOption $       metavar "URL"         <> long "redis-url"-        <> help "URL for Redis cache"+        <> helpEnv "REDIS_URL" "URL for Redis cache"         <> value c.redisURL   redisMinAddrs <-     option auto $-      metavar "GAP"+      metavar "COUNT"         <> long "redis-min-addrs"-        <> help "Minimum xpub address count to cache in Redis"+        <> helpEnv "REDIS_MIN_ADDRS" "Cache xpubs with at least this many used addresses"         <> showDefault         <> value c.redisMinAddrs-  redisMaxKeys <-+  redisMinTxs <-     option auto $       metavar "COUNT"-        <> long "redis-max-keys"-        <> help "Maximum number of keys in Redis"+        <> long "redis-min-txs"+        <> helpEnv "REDIS_MIN_TXS" "Cache xpubs with at least this many transactions in their addresses"         <> showDefault-        <> value c.redisMaxKeys-  redisSyncInterval <-+        <> value c.redisMinTxs+  redisMaxXPubs <-     option auto $-      metavar "SECONDS"-        <> long "redis-sync-interval"-        <> help "Sync mempool to Redis interval"+      metavar "COUNT"+        <> long "redis-max-xpubs"+        <> helpEnv "REDIS_MAX_XPUBS" "Evict the least recently used xpubs from Redis past this many"         <> showDefault-        <> value c.redisSyncInterval+        <> value c.redisMaxXPubs   noMempool <-     flag c.noMempool True $       long "no-mempool"-        <> help "Do not index mempool transactions"+        <> helpEnv "NO_MEMPOOL" "Do not index mempool transactions"   wipeMempool <-     flag c.wipeMempool True $       long "wipe-mempool"-        <> help "Wipe indexed mempool at start"+        <> helpEnv "WIPE_MEMPOOL" "Wipe indexed mempool at start"   syncMempool <-     flag c.syncMempool True $       long "sync-mempool"-        <> help "Attempt to download peer mempools"+        <> helpEnv "SYNC_MEMPOOL" "Attempt to download peer mempools"   blockWindow <-     option auto $       metavar "COUNT"         <> long "block-window"-        <> help "Blocks to request from the syncing peer at a time"+        <> helpEnv "BLOCK_WINDOW" "Blocks to request from the syncing peer at a time"         <> showDefault         <> value c.blockWindow   mempoolTimeout <-     option auto $       metavar "DAYS"         <> long "mempool-timeout"-        <> help "Cull old mempool transactions"+        <> helpEnv "MEMPOOL_TIMEOUT" "Cull old mempool transactions"         <> showDefault         <> value c.mempoolTimeout   peerTimeout <-     option auto $       metavar "SECONDS"         <> long "peer-timeout"-        <> help "Unresponsive peer timeout"+        <> helpEnv "PEER_TIMEOUT" "Unresponsive peer timeout"         <> showDefault         <> value c.peerTimeout   maxPeerLife <-     option auto $       metavar "SECONDS"         <> long "max-peer-life"-        <> help "Maximum peer connection time"+        <> helpEnv "MAX_PEER_LIFE" "Maximum peer connection time"         <> showDefault         <> value c.maxPeerLife   maxPeers <-     option auto $       metavar "COUNT"         <> long "max-peers"-        <> help "Do not connect to more than this many peers"+        <> helpEnv "MAX_PEERS" "Do not connect to more than this many peers"         <> showDefault         <> value c.maxPeers   statsd <-     flag c.statsd True $       long "statsd"-        <> help "Enable statsd metrics"+        <> helpEnv "STATSD" "Enable statsd metrics"   statsdHost <-     strOption $       metavar "HOSTNAME"         <> long "statsd-host"-        <> help "Host to send statsd metrics"+        <> helpEnv "STATSD_HOST" "Host to send statsd metrics"         <> showDefault         <> value c.statsdHost   statsdPort <-     option auto $       metavar "PORT"         <> long "statsd-port"-        <> help "Port to send statsd metrics"+        <> helpEnv "STATSD_PORT" "Port to send statsd metrics"         <> showDefault         <> value c.statsdPort   statsdPrefix <-     strOption $       metavar "PREFIX"         <> long "statsd-prefix"-        <> help "Prefix for statsd metrics"+        <> helpEnv "STATSD_PREFIX" "Prefix for statsd metrics"         <> showDefault         <> value c.statsdPrefix   statsdNoEvents <-     flag c.statsdNoEvents True $       long "statsd-no-events"-        <> help "Do not report individual events to statsd"+        <> helpEnv "STATSD_NO_EVENTS" "Do not report individual events to statsd"   statsdAggregates <-     flag c.statsdAggregates True $       long "statsd-aggregates"-        <> help "Send statistical aggregate reports to statsd every second"+        <> helpEnv "STATSD_AGGREGATES" "Send statistical aggregate reports to statsd every second"   tickerRefresh <-     option auto $       metavar "MICROSECONDS"         <> long "ticker-refresh"-        <> help "How often to retrieve price information"+        <> helpEnv "TICKER_REFRESH" "How often to retrieve price information"         <> showDefault         <> value c.tickerRefresh   tickerURL <-     strOption $       metavar "URL"         <> long "ticker-url"-        <> help "Blockchain.info price ticker URL"+        <> helpEnv "TICKER_URL" "Blockchain.info price ticker URL"         <> showDefault         <> value c.tickerURL   priceHistoryURL <-     strOption $       metavar "URL"         <> long "price-history-url"-        <> help "Blockchain.info price history URL"+        <> helpEnv "PRICE_HISTORY_URL" "Blockchain.info price history URL"         <> showDefault         <> value c.priceHistoryURL   noBlockchainInfo <-     flag c.noBlockchainInfo True $       long "no-blockchain-info"-        <> help "Disable Blockchain.info-style API endpoints"+        <> helpEnv "NO_BLOCKCHAIN_INFO" "Disable Blockchain.info-style API endpoints"   noXPub <-     flag c.noXPub True $       long "no-xpub"-        <> help "Disable extended public key support"+        <> helpEnv "NO_XPUB" "Disable extended public key support"   healthCheckInterval <-     option auto $       metavar "SECONDS"         <> long "health-check-interval"-        <> help "Background check update interval"+        <> helpEnv "HEALTH_CHECK_INTERVAL" "Background check update interval"         <> showDefault         <> value c.healthCheckInterval-  bloom <--    flag c.bloom True $-      long "bloom"-        <> help "RocksDB Bloom filters"+  tlsCert <-+    strOption $+      metavar "FILE"+        <> long "tls-cert"+        <> helpEnv "TLS_CERT" "Serve HTTPS with this certificate chain, in PEM; needs --tls-key"+        <> value c.tlsCert+  tlsKey <-+    strOption $+      metavar "FILE"+        <> long "tls-key"+        <> helpEnv "TLS_KEY" "Private key of the certificate of --tls-cert, in PEM"+        <> value c.tlsKey+  noBloom <-+    switch $+      long "no-bloom"+        <> helpEnv "BLOOM=false" "No RocksDB Bloom filters on the data that is read by key, which are on by default"+  -- it was the way to turn them on, and commands that have it must still+  -- start+  _ <- switch (long "bloom" <> hidden)   maxOpenFiles <-     option auto $       metavar "COUNT"         <> long "max-open-files"-        <> help "RocksDB max_open_files; bounds index/filter memory (0 = unlimited)"+        <> helpEnv "MAX_OPEN_FILES" "RocksDB max_open_files; bounds index/filter memory (0 = unlimited)"         <> showDefault         <> value c.maxOpenFiles   writeBufferMB <-     option auto $       metavar "MB"         <> long "write-buffer-mb"-        <> help "RocksDB memtable budget across all column families in MiB (0 = unlimited)"+        <> helpEnv "WRITE_BUFFER_MB" "RocksDB memtable budget across all column families in MiB (0 = unlimited)"         <> showDefault         <> value c.writeBufferMB   maxWalMB <-     option auto $       metavar "MB"         <> long "max-wal-mb"-        <> help "RocksDB write-ahead log size in MiB that forces flushes so old log files can be deleted (0 = RocksDB default)"+        <> helpEnv+          "MAX_WAL_MB"+          ( unwords+              [ "RocksDB write-ahead log size in MiB that forces flushes so old",+                "log files can be deleted (0 = RocksDB default)"+              ]+          )         <> showDefault         <> value c.maxWalMB   blockCacheMB <-     option auto $       metavar "MB"         <> long "block-cache-mb"-        <> help "RocksDB block cache in MiB, shared by all column families, holding index and filter blocks too (0 = RocksDB default of 32 MiB per column family)"+        <> helpEnv+          "BLOCK_CACHE_MB"+          ( unwords+              [ "RocksDB block cache in MiB, shared by all column families,",+                "holding index and filter blocks too (0 = RocksDB default of",+                "32 MiB per column family)"+              ]+          )         <> showDefault         <> value c.blockCacheMB-  pure Config {..}+  pure Config {bloom = c.bloom && not noBloom, ..}  -- | Network by name. networkReader :: String -> Either String Network@@ -632,19 +676,18 @@ run :: Config -> IO () run cfg =   withContext $ \ctx ->-    withLogger level $ \logger -> flip runContT return $ do+    withLogger level $ \logger -> do       net <- either error return $ networkReader cfg.net-      stats <- ContT $ with_stats logger net-      let dir = cfg.dir </> net.name-      logInfo (withSource "haskoin-store" logger) $-        "Creating working directory (if not found): " <> cs dir-      createDirectoryIfMissing True dir-      store <- ContT $ run_store logger net ctx stats dir-      if cfg.noWeb-        then lift . withSubscription store.pub $ \sub ->-          loopForever (receive sub)-        else do-          lift $ run_web logger stats store+      with_stats logger net $ \stats -> do+        let dir = cfg.dir </> net.name+        logInfo (withSource "haskoin-store" logger) $+          "Creating working directory (if not found): " <> cs dir+        createDirectoryIfMissing True dir+        run_store logger net ctx stats dir $ \store ->+          if cfg.noWeb+            then withSubscription store.pub $ \sub ->+              loopForever (receive sub)+            else run_web logger stats store   where     run_store logger net ctx stats dir =       withStore@@ -656,10 +699,12 @@             net = net,             ctx = ctx,             redis = if cfg.redis then Just cfg.redisURL else Nothing,+            fleet = cfg.fleet,             gap = cfg.webLimits.xpubGap,             initGap = cfg.webLimits.xpubGapInit,             redisMinAddrs = cfg.redisMinAddrs,-            redisMaxKeys = cfg.redisMaxKeys,+            redisMinTxs = cfg.redisMinTxs,+            redisMaxXPubs = cfg.redisMaxXPubs,             wipeMempool = cfg.wipeMempool,             noMempool = cfg.noMempool,             syncMempool = cfg.syncMempool,@@ -669,7 +714,6 @@             maxPeerLife = fromIntegral cfg.maxPeerLife,             connect = withConnection,             stats = stats,-            redisSyncInterval = cfg.redisSyncInterval,             bloom = cfg.bloom,             maxOpenFiles = if cfg.maxOpenFiles > 0 then Just cfg.maxOpenFiles else Nothing,             writeBufferSize =@@ -705,6 +749,11 @@             noXPub = cfg.noXPub,             noBlockchainInfo = cfg.noBlockchainInfo,             healthCheckInterval = cfg.healthCheckInterval,+            tls = case (cfg.tlsCert, cfg.tlsKey) of+              ("", "") -> Nothing+              ("", _) -> error "--tls-key needs --tls-cert"+              (_, "") -> error "--tls-cert needs --tls-key"+              (certificate, key) -> Just (certificate, key),             logger = withSource "Web" logger           }     with_stats logger net go

@@ -1,9 +1,3 @@-{-# LANGUAGE DuplicateRecordFields #-}-{-# LANGUAGE ImportQualifiedPost #-}-{-# LANGUAGE LambdaCase #-}-{-# LANGUAGE OverloadedRecordDot #-}-{-# LANGUAGE OverloadedStrings #-}- -- | Load-test server for the web API. -- -- Syncs a synthetic regtest chain from an in-process mock peer into a fresh@@ -20,29 +14,28 @@ module Main (main) where  import Conduit-import Control.Monad (forever, unless)-import Control.Monad.Reader (runReaderT)-import Data.ByteString (ByteString)-import Data.ByteString qualified as B-import Data.HashMap.Strict qualified as HashMap-import Data.List (find) import Control.DeepSeq (NFData, force)+import Control.Monad (forever, unless) import Data.Aeson (toEncoding) import Data.Aeson.Encoding (encodingToLazyByteString)+import Data.ByteString (ByteString)+import Data.ByteString qualified as B import Data.Default (def)+import Data.HashMap.Strict qualified as HashMap import Data.IntMap.Strict qualified as IntMap-import Data.Vector qualified as V-import GHC.Clock (getMonotonicTimeNSec)-import Haskoin.Store.Data+import Data.List (find) import Data.Maybe (fromMaybe, listToMaybe, mapMaybe) import Data.Serialize (decode, encode, runGet, runPut) import Data.Text qualified as T import Data.Text.IO qualified as T import Data.Time.Clock.POSIX (getPOSIXTime)+import Data.Vector qualified as V import Data.Word (Word32)+import GHC.Clock (getMonotonicTimeNSec) import Haskoin import Haskoin.Node import Haskoin.Store+import Haskoin.Store.Database.Query qualified as Q import NQE import Network.Socket (SockAddr (..)) import System.Environment (getArgs)@@ -72,7 +65,7 @@         T.putStrLn $ "hot address: " <> addrText net (hotAddr ctx)         T.putStrLn $ "xpub: " <> xPubExport net ctx (benchXPub ctx)         T.putStrLn $ "xpub address: " <> addrText net (xpubAddr ctx 0 0)-        bh <- runReaderT getBestBlock store.db+        bh <- Q.getBestBlock store.db         T.putStrLn $ "best block: " <> maybe "" blockHashToHex bh         putStrLn $ "serving on port " <> show port         hFlush stdout@@ -83,7 +76,10 @@  waitHeight :: Store -> Inbox StoreEvent -> Int -> IO () waitHeight store sub n = do-  h <- maybe 0 (.height) <$> runReaderT (getBestBlock >>= maybe (return Nothing) getBlock) store.db+  h <-+    Q.getBestBlock store.db >>= \case+      Nothing -> return 0+      Just bh -> maybe 0 (.height) <$> Q.getBlock store.db bh   unless (fromIntegral h >= n) $ do     _ <- receiveMatch sub $ \case       StoreBestBlock b -> Just b@@ -104,9 +100,11 @@       gap = 32,       initGap = 20,       redisMinAddrs = 20,-      redisMaxKeys = 100 * 1000 * 1000,+      redisMinTxs = 1000,+      redisMaxXPubs = 10000,       noMempool = False,       wipeMempool = False,+      fleet = False,       syncMempool = False,       blockWindow = 64,       mempoolTimeout = 14,@@ -114,7 +112,6 @@       maxPeerLife = 48 * 3600,       connect = mockPeer blocks requests net ad,       stats = Nothing,-      redisSyncInterval = 30,       bloom = True,       maxOpenFiles = Nothing,       writeBufferSize = Nothing,@@ -147,6 +144,7 @@       noXPub = False,       noBlockchainInfo = False,       healthCheckInterval = 30,+      tls = Nothing,       logger = quiet     } @@ -161,7 +159,9 @@ xpubAddr ctx m i = fst (deriveAddr ctx (pubSubKey ctx (benchXPub ctx) m) i)  hotAddr :: Ctx -> Address-hotAddr ctx = pubKeyAddr ctx (derivePublicKey ctx (wrapSecKey True (fromMaybe (error "key") (secKey (B.replicate 32 0x42)))))+hotAddr ctx = pubKeyAddr ctx (derivePublicKey ctx (wrapSecKey True key))+  where+    key = fromMaybe (error "key") (secKey (B.replicate 32 0x42))  scriptOf :: Ctx -> Address -> ByteString scriptOf ctx = fromMaybe (error "script") . addressToScriptBS ctx

@@ -1,204 +1,362 @@-cabal-version: 1.12+cabal-version: 2.0  -- This file has been generated from package.yaml by hpack version 0.39.6. -- -- see: https://github.com/sol/hpack  name:           haskoin-store-version:        1.8.1+version:        1.9.0 synopsis:       Storage and index for Bitcoin and Bitcoin Cash-description:    Please see the README on GitHub at <https://github.com/jprupp/haskoin-store#readme>+description:    An indexer for Bitcoin and Bitcoin Cash. It downloads blocks and mempool+                transactions from the peer-to-peer network, indexes them in RocksDB, and+                serves a REST API with balances, transactions and unspent outputs of+                addresses and extended public keys, a Blockchain.info compatible API, and+                events over WebSockets. An optional Redis cache, shared by several stores,+                keeps large wallets fast. See the README for installation and+                configuration. category:       Bitcoin, Finance, Network-homepage:       http://github.com/jprupp/haskoin-store#readme-bug-reports:    http://github.com/jprupp/haskoin-store/issues+homepage:       https://github.com/jprupp/haskoin-store#readme+bug-reports:    https://github.com/jprupp/haskoin-store/issues author:         JP Rupp maintainer:     jprupp@protonmail.ch+copyright:      (c) Haskoin Developers license:        MIT license-file:   LICENSE build-type:     Simple+tested-with:+    GHC == 9.12.4+extra-source-files:+    src/Haskoin/Store/Cache/lua/bodies.lua+    src/Haskoin/Store/Cache/lua/commit.lua+    src/Haskoin/Store/Cache/lua/delete.lua+    src/Haskoin/Store/Cache/lua/evict.lua+    src/Haskoin/Store/Cache/lua/page.lua+    src/Haskoin/Store/Cache/lua/spends.lua+    src/Haskoin/Store/Cache/lua/unlock.lua+    src/Haskoin/Store/Fleet/lua/move.lua+    src/Haskoin/Store/Fleet/lua/pool.lua+    src/Haskoin/Store/Fleet/lua/read.lua+    src/Haskoin/Store/Fleet/lua/report.lua+    src/Haskoin/Store/Fleet/lua/xpub.lua+    stress/wrk.lua+    swagger.yaml+extra-doc-files:+    README.md+    CHANGELOG.md  source-repository head   type: git-  location: https://github.com/jprupp/haskoin-store.git+  location: https://github.com/jprupp/haskoin-store  library   exposed-modules:       Haskoin.Store       Haskoin.Store.BlockStore+      Haskoin.Store.BlockStore.Pending+      Haskoin.Store.BlockStore.Sync       Haskoin.Store.Cache+      Haskoin.Store.Cache.Protocol       Haskoin.Store.Common+      Haskoin.Store.Database.Mempool+      Haskoin.Store.Database.Migrate+      Haskoin.Store.Database.Query+      Haskoin.Store.Database.Query.Layered+      Haskoin.Store.Database.Query.Pooled       Haskoin.Store.Database.Reader       Haskoin.Store.Database.Types+      Haskoin.Store.Database.Views       Haskoin.Store.Database.Writer+      Haskoin.Store.Fleet+      Haskoin.Store.Fleet.Member+      Haskoin.Store.Fleet.Pool+      Haskoin.Store.Fleet.Redis+      Haskoin.Store.Fleet.XPub       Haskoin.Store.Logic+      Haskoin.Store.Logic.Layered       Haskoin.Store.Manager       Haskoin.Store.Web+      Haskoin.Store.Web.Docs+      Haskoin.Store.Web.Events+      Haskoin.Store.Web.Router   other-modules:       Paths_haskoin_store+  autogen-modules:+      Paths_haskoin_store   hs-source-dirs:       src+  default-extensions:+      DuplicateRecordFields+      NoFieldSelectors+      OverloadedRecordDot+      OverloadedStrings   ghc-options: -Wall -Wcompat -Widentities -Wincomplete-record-updates -Wincomplete-uni-patterns -Wmissing-export-lists -Wmissing-home-modules -Wpartial-fields -Wredundant-constraints   build-depends:-      aeson >=1.4.7.1-    , aeson-pretty >=0.8.8-    , base >=4.9 && <5-    , base16 >=2.0-    , bytes >=0.17-    , bytestring >=0.10.10.0-    , cereal >=0.5.8.1-    , conduit >=1.3.2-    , containers >=0.6.2.1-    , data-default >=0.7.1.1-    , deepseq >=1.4.4.0-    , hashable >=1.3.0.0-    , hashtables >=1.2.4.2-    , haskoin-core >=1.6.1-    , haskoin-node >=2.3.0-    , haskoin-store-data ==1.8.1-    , hedis >=0.16.1-    , http-types >=0.12.3-    , lens >=4.18.1-    , monad-control >=1.0.2.3-    , mtl >=2.2.2-    , network >=3.1.1.1-    , nqe >=0.6.5-    , random >=1.1-    , rocksdb-haskell-jprupp >=2.6.0-    , rocksdb-query >=0.6.0-    , scotty >=0.22-    , statsd-rupp >=0.4-    , stm >=2.5.0.0-    , string-conversions >=0.4.0.1-    , text >=1.2.3.0-    , time >=1.9.3-    , transformers >=0.5.6.2-    , unliftio >=0.2.12.1-    , unordered-containers >=0.2.10.0-    , vault >=0.3.1.5-    , wai >=3.2.2.1-    , wai-websockets >=3.0.1.2-    , warp >=3.3.10-    , websockets >=0.13-    , wreq >=0.5.3.2-  default-language: Haskell2010+      aeson >=2.2.5.1 && <2.3+    , aeson-pretty >=0.8.11 && <0.9+    , base >=4.21.2.0 && <4.22+    , base16 ==2.0.*+    , bytes >=0.17.5 && <0.18+    , bytestring >=0.12.2.0 && <0.13+    , case-insensitive >=1.2.1.0 && <1.3+    , cereal >=0.5.8.3 && <0.6+    , conduit >=1.3.6.1 && <1.4+    , containers ==0.7.*+    , data-default >=0.8.0.2 && <0.9+    , deepseq >=1.5.1.0 && <1.6+    , file-embed >=0.0.16.0 && <0.1+    , hashable >=1.5.1.0 && <1.6+    , hashtables >=1.4.2 && <1.5+    , haskoin-core >=1.6.1 && <1.7+    , haskoin-node ==2.5.*+    , haskoin-store-data ==1.11.*+    , hedis >=0.16.4 && <0.17+    , http-types >=0.12.6 && <0.13+    , lens >=5.3.6 && <5.4+    , monad-control >=1.0.3.1 && <1.1+    , mtl >=2.3.2 && <2.4+    , network >=3.2.12.0 && <3.3+    , nqe >=0.6.9 && <0.7+    , random >=1.3.1 && <1.4+    , rocksdb-haskell-jprupp >=2.7.1 && <2.8+    , rocksdb-query >=0.6.1 && <0.7+    , statsd-rupp >=0.5.0.2 && <0.6+    , stm >=2.5.3.1 && <2.6+    , string-conversions >=0.4.0.1 && <0.5+    , text >=2.1.4 && <2.2+    , time ==1.14.*+    , transformers >=0.6.3.0 && <0.7+    , unliftio >=0.2.25.1 && <0.3+    , unordered-containers >=0.2.21 && <0.3+    , vault >=0.3.2.0 && <0.4+    , wai >=3.2.5 && <3.3+    , wai-extra >=3.1.18 && <3.2+    , wai-websockets >=3.0.1.2 && <3.1+    , warp >=3.4.17 && <3.5+    , warp-tls >=3.4.14 && <3.5+    , websockets >=0.13.0.0 && <0.14+    , wreq >=0.5.4.5 && <0.6+  default-language: GHC2024 +library integration+  exposed-modules:+      Haskoin.Store.Integration.Image+      Haskoin.Store.Integration.Node+      Haskoin.Store.Integration.Podman+      Haskoin.Store.Integration.Valkey+  other-modules:+      Paths_haskoin_store+  autogen-modules:+      Paths_haskoin_store+  hs-source-dirs:+      integration+  default-extensions:+      DuplicateRecordFields+      NoFieldSelectors+      OverloadedRecordDot+      OverloadedStrings+  ghc-options: -Wall -Wcompat -Widentities -Wincomplete-record-updates -Wincomplete-uni-patterns -Wmissing-export-lists -Wmissing-home-modules -Wpartial-fields -Wredundant-constraints+  build-depends:+      aeson >=2.2.5.1 && <2.3+    , aeson-pretty >=0.8.11 && <0.9+    , base >=4.21.2.0 && <4.22+    , base16 ==2.0.*+    , bytes >=0.17.5 && <0.18+    , bytestring >=0.12.2.0 && <0.13+    , case-insensitive >=1.2.1.0 && <1.3+    , cereal >=0.5.8.3 && <0.6+    , conduit >=1.3.6.1 && <1.4+    , containers ==0.7.*+    , data-default >=0.8.0.2 && <0.9+    , deepseq >=1.5.1.0 && <1.6+    , file-embed >=0.0.16.0 && <0.1+    , hashable >=1.5.1.0 && <1.6+    , hashtables >=1.4.2 && <1.5+    , haskoin-core >=1.6.1 && <1.7+    , haskoin-node ==2.5.*+    , haskoin-store-data ==1.11.*+    , hedis >=0.16.4 && <0.17+    , http-types >=0.12.6 && <0.13+    , lens >=5.3.6 && <5.4+    , monad-control >=1.0.3.1 && <1.1+    , mtl >=2.3.2 && <2.4+    , network >=3.2.12.0 && <3.3+    , nqe >=0.6.9 && <0.7+    , process >=1.6.26.1 && <1.7+    , random >=1.3.1 && <1.4+    , rocksdb-haskell-jprupp >=2.7.1 && <2.8+    , rocksdb-query >=0.6.1 && <0.7+    , statsd-rupp >=0.5.0.2 && <0.6+    , stm >=2.5.3.1 && <2.6+    , string-conversions >=0.4.0.1 && <0.5+    , text >=2.1.4 && <2.2+    , time ==1.14.*+    , transformers >=0.6.3.0 && <0.7+    , unliftio >=0.2.25.1 && <0.3+    , unordered-containers >=0.2.21 && <0.3+    , vault >=0.3.2.0 && <0.4+    , wai >=3.2.5 && <3.3+    , wai-extra >=3.1.18 && <3.2+    , wai-websockets >=3.0.1.2 && <3.1+    , warp >=3.4.17 && <3.5+    , warp-tls >=3.4.14 && <3.5+    , websockets >=0.13.0.0 && <0.14+    , wreq >=0.5.4.5 && <0.6+  default-language: GHC2024+ executable haskoin-store   main-is: Main.hs   other-modules:       Paths_haskoin_store+  autogen-modules:+      Paths_haskoin_store   hs-source-dirs:       app-  ghc-options: -Wall -Wcompat -Widentities -Wincomplete-record-updates -Wincomplete-uni-patterns -Wmissing-export-lists -Wmissing-home-modules -Wpartial-fields -Wredundant-constraints -threaded -rtsopts -with-rtsopts=-N+  default-extensions:+      DuplicateRecordFields+      NoFieldSelectors+      OverloadedRecordDot+      OverloadedStrings+  ghc-options: -Wall -Wcompat -Widentities -Wincomplete-record-updates -Wincomplete-uni-patterns -Wmissing-export-lists -Wmissing-home-modules -Wpartial-fields -Wredundant-constraints -threaded -rtsopts "-with-rtsopts=-N --disable-delayed-os-memory-return"   build-depends:-      aeson >=1.4.7.1-    , aeson-pretty >=0.8.8-    , base >=4.9 && <5-    , base16 >=2.0-    , bytes >=0.17-    , bytestring >=0.10.10.0-    , cereal >=0.5.8.1-    , conduit >=1.3.2-    , containers >=0.6.2.1-    , data-default >=0.7.1.1-    , deepseq >=1.4.4.0-    , filepath-    , hashable >=1.3.0.0-    , hashtables >=1.2.4.2-    , haskoin-core >=1.6.1-    , haskoin-node >=2.3.0+      aeson >=2.2.5.1 && <2.3+    , aeson-pretty >=0.8.11 && <0.9+    , base >=4.21.2.0 && <4.22+    , base16 ==2.0.*+    , bytes >=0.17.5 && <0.18+    , bytestring >=0.12.2.0 && <0.13+    , case-insensitive >=1.2.1.0 && <1.3+    , cereal >=0.5.8.3 && <0.6+    , conduit >=1.3.6.1 && <1.4+    , containers ==0.7.*+    , data-default >=0.8.0.2 && <0.9+    , deepseq >=1.5.1.0 && <1.6+    , file-embed >=0.0.16.0 && <0.1+    , filepath >=1.5.5.0 && <1.6+    , hashable >=1.5.1.0 && <1.6+    , hashtables >=1.4.2 && <1.5+    , haskoin-core >=1.6.1 && <1.7+    , haskoin-node ==2.5.*     , haskoin-store-    , haskoin-store-data ==1.8.1-    , hedis >=0.16.1-    , http-types >=0.12.3-    , lens >=4.18.1-    , monad-control >=1.0.2.3-    , mtl >=2.2.2-    , network >=3.1.1.1-    , nqe >=0.6.5-    , optparse-applicative-    , random >=1.1-    , rocksdb-haskell-jprupp >=2.6.0-    , rocksdb-query >=0.6.0-    , scotty >=0.22-    , statsd-rupp >=0.4-    , stm >=2.5.0.0-    , string-conversions >=0.4.0.1-    , text >=1.2.3.0-    , time >=1.9.3-    , transformers >=0.5.6.2-    , unliftio >=0.2.12.1-    , unordered-containers >=0.2.10.0-    , vault >=0.3.1.5-    , wai >=3.2.2.1-    , wai-websockets >=3.0.1.2-    , warp >=3.3.10-    , websockets >=0.13-    , wreq >=0.5.3.2-  default-language: Haskell2010+    , haskoin-store-data ==1.11.*+    , hedis >=0.16.4 && <0.17+    , http-types >=0.12.6 && <0.13+    , lens >=5.3.6 && <5.4+    , monad-control >=1.0.3.1 && <1.1+    , mtl >=2.3.2 && <2.4+    , network >=3.2.12.0 && <3.3+    , nqe >=0.6.9 && <0.7+    , optparse-applicative >=0.19.0.0 && <0.20+    , random >=1.3.1 && <1.4+    , rocksdb-haskell-jprupp >=2.7.1 && <2.8+    , rocksdb-query >=0.6.1 && <0.7+    , statsd-rupp >=0.5.0.2 && <0.6+    , stm >=2.5.3.1 && <2.6+    , string-conversions >=0.4.0.1 && <0.5+    , text >=2.1.4 && <2.2+    , time ==1.14.*+    , transformers >=0.6.3.0 && <0.7+    , unliftio >=0.2.25.1 && <0.3+    , unordered-containers >=0.2.21 && <0.3+    , vault >=0.3.2.0 && <0.4+    , wai >=3.2.5 && <3.3+    , wai-extra >=3.1.18 && <3.2+    , wai-websockets >=3.0.1.2 && <3.1+    , warp >=3.4.17 && <3.5+    , warp-tls >=3.4.14 && <3.5+    , websockets >=0.13.0.0 && <0.14+    , wreq >=0.5.4.5 && <0.6+  default-language: GHC2024  test-suite spec   type: exitcode-stdio-1.0   main-is: Spec.hs   other-modules:-      Haskoin.Store.BlockStoreSpec+      Haskoin.Store.BlockStore.PendingSpec+      Haskoin.Store.BlockStore.SyncSpec+      Haskoin.Store.Cache.ProtocolSpec       Haskoin.Store.CacheSpec       Haskoin.Store.CommonSpec+      Haskoin.Store.Database.MigrateSpec+      Haskoin.Store.Database.Query.LayeredSpec+      Haskoin.Store.Database.ViewsSpec+      Haskoin.Store.Fleet.MemberSpec+      Haskoin.Store.Fleet.PoolSpec+      Haskoin.Store.Fleet.RedisSpec+      Haskoin.Store.Fleet.XPubSpec+      Haskoin.Store.FleetSpec+      Haskoin.Store.Histories+      Haskoin.Store.IntegrationSpec+      Haskoin.Store.Logic.LayeredSpec       Haskoin.Store.LogicSpec       Haskoin.Store.TestBlocks+      Haskoin.Store.Web.EventsSpec+      Haskoin.Store.Web.RouterSpec       Haskoin.Store.WebSpec       Haskoin.StoreSpec       Paths_haskoin_store+  autogen-modules:+      Paths_haskoin_store   hs-source-dirs:       test-  ghc-options: -Wall -Wcompat -Widentities -Wincomplete-record-updates -Wincomplete-uni-patterns -Wmissing-export-lists -Wmissing-home-modules -Wpartial-fields -Wredundant-constraints -threaded -rtsopts "-with-rtsopts=-N -T"+  default-extensions:+      DuplicateRecordFields+      NoFieldSelectors+      OverloadedRecordDot+      OverloadedStrings+  ghc-options: -Wall -Wcompat -Widentities -Wincomplete-record-updates -Wincomplete-uni-patterns -Wmissing-export-lists -Wmissing-home-modules -Wpartial-fields -Wredundant-constraints -threaded -rtsopts "-with-rtsopts=-N4 -T -M4g"   build-depends:-      QuickCheck >=2.13.2-    , aeson >=1.4.7.1-    , aeson-pretty >=0.8.8-    , base >=4.9 && <5-    , base16 >=2.0-    , base64 >=0.4.1-    , bytes >=0.17-    , bytestring >=0.10.10.0-    , cereal >=0.5.8.1-    , conduit >=1.3.2-    , containers >=0.6.2.1-    , data-default >=0.7.1.1-    , deepseq >=1.4.4.0-    , fast-logger >=3.0-    , hashable >=1.3.0.0-    , hashtables >=1.2.4.2-    , haskoin-core >=1.6.1-    , haskoin-node >=2.3.0+      QuickCheck >=2.17.1.0 && <2.18+    , aeson >=2.2.5.1 && <2.3+    , aeson-pretty >=0.8.11 && <0.9+    , base >=4.21.2.0 && <4.22+    , base16 ==2.0.*+    , base64 >=1.0.1 && <1.1+    , bytes >=0.17.5 && <0.18+    , bytestring >=0.12.2.0 && <0.13+    , case-insensitive >=1.2.1.0 && <1.3+    , cereal >=0.5.8.3 && <0.6+    , conduit >=1.3.6.1 && <1.4+    , containers ==0.7.*+    , data-default >=0.8.0.2 && <0.9+    , deepseq >=1.5.1.0 && <1.6+    , fast-logger >=3.2.9 && <3.3+    , file-embed >=0.0.16.0 && <0.1+    , hashable >=1.5.1.0 && <1.6+    , hashtables >=1.4.2 && <1.5+    , haskoin-core >=1.6.1 && <1.7+    , haskoin-node ==2.5.*     , haskoin-store-    , haskoin-store-data ==1.8.1-    , hedis >=0.16.1-    , hspec >=2.7.1-    , http-types >=0.12.3-    , lens >=4.18.1-    , monad-control >=1.0.2.3-    , mtl >=2.2.2-    , network >=3.1.1.1-    , nqe >=0.6.5-    , random >=1.1-    , rocksdb-haskell-jprupp >=2.6.0-    , rocksdb-query >=0.6.0-    , scotty >=0.22-    , statsd-rupp >=0.4-    , stm >=2.5.0.0-    , string-conversions >=0.4.0.1-    , text >=1.2.3.0-    , time >=1.9.3-    , transformers >=0.5.6.2-    , unliftio >=0.2.12.1-    , unordered-containers >=0.2.10.0-    , vault >=0.3.1.5-    , wai >=3.2.2.1-    , wai-websockets >=3.0.1.2-    , warp >=3.3.10-    , websockets >=0.13-    , wreq >=0.5.3.2-  default-language: Haskell2010+    , haskoin-store-data ==1.11.*+    , hedis >=0.16.4 && <0.17+    , hspec >=2.11.18 && <2.12+    , http-types >=0.12.6 && <0.13+    , integration+    , lens >=5.3.6 && <5.4+    , monad-control >=1.0.3.1 && <1.1+    , mtl >=2.3.2 && <2.4+    , network >=3.2.12.0 && <3.3+    , nqe >=0.6.9 && <0.7+    , process >=1.6.26.1 && <1.7+    , random >=1.3.1 && <1.4+    , rocksdb-haskell-jprupp >=2.7.1 && <2.8+    , rocksdb-query >=0.6.1 && <0.7+    , statsd-rupp >=0.5.0.2 && <0.6+    , stm >=2.5.3.1 && <2.6+    , string-conversions >=0.4.0.1 && <0.5+    , text >=2.1.4 && <2.2+    , time ==1.14.*+    , transformers >=0.6.3.0 && <0.7+    , unliftio >=0.2.25.1 && <0.3+    , unordered-containers >=0.2.21 && <0.3+    , vault >=0.3.2.0 && <0.4+    , wai >=3.2.5 && <3.3+    , wai-extra >=3.1.18 && <3.2+    , wai-websockets >=3.0.1.2 && <3.1+    , warp >=3.4.17 && <3.5+    , warp-tls >=3.4.14 && <3.5+    , websockets >=0.13.0.0 && <0.14+    , wreq >=0.5.4.5 && <0.6+  default-language: GHC2024   build-tool-depends: hspec-discover:hspec-discover  benchmark haskoin-store-bench@@ -206,51 +364,127 @@   main-is: Main.hs   other-modules:       Paths_haskoin_store+  autogen-modules:+      Paths_haskoin_store   hs-source-dirs:       bench+  default-extensions:+      DuplicateRecordFields+      NoFieldSelectors+      OverloadedRecordDot+      OverloadedStrings   ghc-options: -Wall -Wcompat -Widentities -Wincomplete-record-updates -Wincomplete-uni-patterns -Wmissing-export-lists -Wmissing-home-modules -Wpartial-fields -Wredundant-constraints -threaded -rtsopts -with-rtsopts=-N   build-depends:-      aeson >=1.4.7.1-    , aeson-pretty >=0.8.8-    , base >=4.9 && <5-    , base16 >=2.0-    , bytes >=0.17-    , bytestring >=0.10.10.0-    , cereal >=0.5.8.1-    , conduit >=1.3.2-    , containers >=0.6.2.1-    , data-default >=0.7.1.1-    , deepseq >=1.4.4.0-    , hashable >=1.3.0.0-    , hashtables >=1.2.4.2-    , haskoin-core >=1.6.1-    , haskoin-node >=2.3.0+      aeson >=2.2.5.1 && <2.3+    , aeson-pretty >=0.8.11 && <0.9+    , base >=4.21.2.0 && <4.22+    , base16 ==2.0.*+    , bytes >=0.17.5 && <0.18+    , bytestring >=0.12.2.0 && <0.13+    , case-insensitive >=1.2.1.0 && <1.3+    , cereal >=0.5.8.3 && <0.6+    , conduit >=1.3.6.1 && <1.4+    , containers ==0.7.*+    , data-default >=0.8.0.2 && <0.9+    , deepseq >=1.5.1.0 && <1.6+    , file-embed >=0.0.16.0 && <0.1+    , hashable >=1.5.1.0 && <1.6+    , hashtables >=1.4.2 && <1.5+    , haskoin-core >=1.6.1 && <1.7+    , haskoin-node ==2.5.*     , haskoin-store-    , haskoin-store-data ==1.8.1-    , hedis >=0.16.1-    , http-types >=0.12.3-    , lens >=4.18.1-    , monad-control >=1.0.2.3-    , mtl >=2.2.2-    , network >=3.1.1.1-    , nqe >=0.6.5-    , random >=1.1-    , rocksdb-haskell-jprupp >=2.6.0-    , rocksdb-query >=0.6.0-    , scotty >=0.22-    , statsd-rupp >=0.4-    , stm >=2.5.0.0-    , string-conversions >=0.4.0.1-    , text >=1.2.3.0-    , time >=1.9.3-    , transformers >=0.5.6.2-    , unliftio >=0.2.12.1-    , unordered-containers >=0.2.10.0-    , vault >=0.3.1.5-    , vector-    , wai >=3.2.2.1-    , wai-websockets >=3.0.1.2-    , warp >=3.3.10-    , websockets >=0.13-    , wreq >=0.5.3.2-  default-language: Haskell2010+    , haskoin-store-data ==1.11.*+    , hedis >=0.16.4 && <0.17+    , http-types >=0.12.6 && <0.13+    , lens >=5.3.6 && <5.4+    , monad-control >=1.0.3.1 && <1.1+    , mtl >=2.3.2 && <2.4+    , network >=3.2.12.0 && <3.3+    , nqe >=0.6.9 && <0.7+    , random >=1.3.1 && <1.4+    , rocksdb-haskell-jprupp >=2.7.1 && <2.8+    , rocksdb-query >=0.6.1 && <0.7+    , statsd-rupp >=0.5.0.2 && <0.6+    , stm >=2.5.3.1 && <2.6+    , string-conversions >=0.4.0.1 && <0.5+    , text >=2.1.4 && <2.2+    , time ==1.14.*+    , transformers >=0.6.3.0 && <0.7+    , unliftio >=0.2.25.1 && <0.3+    , unordered-containers >=0.2.21 && <0.3+    , vault >=0.3.2.0 && <0.4+    , vector >=0.13.2.0 && <0.14+    , wai >=3.2.5 && <3.3+    , wai-extra >=3.1.18 && <3.2+    , wai-websockets >=3.0.1.2 && <3.1+    , warp >=3.4.17 && <3.5+    , warp-tls >=3.4.14 && <3.5+    , websockets >=0.13.0.0 && <0.14+    , wreq >=0.5.4.5 && <0.6+  default-language: GHC2024++benchmark haskoin-store-stress+  type: exitcode-stdio-1.0+  main-is: Main.hs+  other-modules:+      Paths_haskoin_store+  autogen-modules:+      Paths_haskoin_store+  hs-source-dirs:+      stress+  default-extensions:+      DuplicateRecordFields+      NoFieldSelectors+      OverloadedRecordDot+      OverloadedStrings+  ghc-options: -Wall -Wcompat -Widentities -Wincomplete-record-updates -Wincomplete-uni-patterns -Wmissing-export-lists -Wmissing-home-modules -Wpartial-fields -Wredundant-constraints -threaded -rtsopts -with-rtsopts=-N2+  build-depends:+      aeson >=2.2.5.1 && <2.3+    , aeson-pretty >=0.8.11 && <0.9+    , base >=4.21.2.0 && <4.22+    , base16 ==2.0.*+    , bytes >=0.17.5 && <0.18+    , bytestring >=0.12.2.0 && <0.13+    , case-insensitive >=1.2.1.0 && <1.3+    , cereal >=0.5.8.3 && <0.6+    , conduit >=1.3.6.1 && <1.4+    , containers ==0.7.*+    , data-default >=0.8.0.2 && <0.9+    , deepseq >=1.5.1.0 && <1.6+    , file-embed >=0.0.16.0 && <0.1+    , hashable >=1.5.1.0 && <1.6+    , hashtables >=1.4.2 && <1.5+    , haskoin-core >=1.6.1 && <1.7+    , haskoin-node ==2.5.*+    , haskoin-store+    , haskoin-store-data ==1.11.*+    , hedis >=0.16.4 && <0.17+    , http-types >=0.12.6 && <0.13+    , integration+    , lens >=5.3.6 && <5.4+    , monad-control >=1.0.3.1 && <1.1+    , mtl >=2.3.2 && <2.4+    , network >=3.2.12.0 && <3.3+    , nqe >=0.6.9 && <0.7+    , optparse-applicative >=0.19.0.0 && <0.20+    , process >=1.6.26.1 && <1.7+    , random >=1.3.1 && <1.4+    , rocksdb-haskell-jprupp >=2.7.1 && <2.8+    , rocksdb-query >=0.6.1 && <0.7+    , statsd-rupp >=0.5.0.2 && <0.6+    , stm >=2.5.3.1 && <2.6+    , string-conversions >=0.4.0.1 && <0.5+    , text >=2.1.4 && <2.2+    , time ==1.14.*+    , transformers >=0.6.3.0 && <0.7+    , unliftio >=0.2.25.1 && <0.3+    , unordered-containers >=0.2.21 && <0.3+    , vault >=0.3.2.0 && <0.4+    , wai >=3.2.5 && <3.3+    , wai-extra >=3.1.18 && <3.2+    , wai-websockets >=3.0.1.2 && <3.1+    , warp >=3.4.17 && <3.5+    , warp-tls >=3.4.14 && <3.5+    , websockets >=0.13.0.0 && <0.14+    , wreq >=0.5.4.5 && <0.6+  default-language: GHC2024

@@ -0,0 +1,196 @@+-- |+-- Module      : Haskoin.Store.Integration.Image+-- Copyright   : (c) Haskoin Developers+-- License     : MIT+-- Maintainer  : jprupp@protonmail.ch+-- Stability   : experimental+-- Portability : POSIX+--+-- The container image of the integration tests, built from pinned and+-- verified sources only.+--+-- The tests use no third-party images. The image starts from the official+-- Ubuntu 26.04 image, pinned by the digest of its multi-architecture index,+-- installs Valkey (the Redis-compatible server Ubuntu ships) and wrk from the+-- Ubuntu archive at a fixed snapshot date (apt checks the signatures of the+-- archive; the certificates needed to reach the snapshot service over HTTPS+-- come from the regular archive first), and downloads Bitcoin Core and+-- Bitcoin Cash Node from their official release locations, checking each+-- tarball against a SHA-256 hash pinned here. A tarball that does not match+-- stops the build.+--+-- The hashes were taken from the signed checksum files of each release and+-- checked against the release signing keys of each project when they were+-- pinned:+--+-- * Bitcoin Core 32.1: @SHA256SUMS@ from bitcoincore.org, with good+--   signatures from eight of the builders whose keys are in the+--   @bitcoin-core/guix.sigs@ repository;+-- * Bitcoin Cash Node 29.2.0: @SHA256SUMS.29.2.0.asc.*@ from the+--   @bitcoin-cash-node/announcements@ repository, with good signatures from+--   two maintainers whose keys are in @contrib/gitian-signing/pubkeys.txt@.+--+-- To move to new releases, change the versions and hashes below after+-- checking them the same way. The tag of the image is a hash of the whole+-- recipe, so a change builds a new image and the old one is no longer used.+module Haskoin.Store.Integration.Image+  ( testImage,+    coreBin,+    bchnBin,+    valkeyBin,+    wrkBin,+  )+where++import Data.ByteString.Char8 qualified as C8+import Data.List (intercalate)+import Data.Serialize (encode)+import Data.Text qualified as T+import Haskoin (encodeHex, sha256)+import System.Exit (ExitCode (..))+import System.Process (readProcessWithExitCode)+import UnliftIO.Temporary (withSystemTempDirectory)++-- | Ubuntu 26.04, by the digest of its multi-architecture index.+base :: String+base = "docker.io/library/ubuntu@sha256:6a06111c3b0a8600feadb270aec771dc239bf407b25625f93f44ab00e11f4d3c"++-- | Date of the snapshot of the Ubuntu archive to install packages from, so+-- that the same versions are installed every time.+snapshot :: String+snapshot = "20261101T000000Z"++-- | Bitcoin Core release and the hashes of its Linux tarballs for x86_64 and+-- aarch64.+coreVersion, coreAmd64, coreArm64 :: String+coreVersion = "32.1"+coreAmd64 = "61545c17c3120fb7412f896d248b11993af04ec8817aad3649c55b2afb044963"+coreArm64 = "7ed37c7373eb6aa3e3f72842596a1afeba1b223717033cade9842b25773e1922"++-- | Bitcoin Cash Node release and the hashes of its Linux tarballs for x86_64+-- and aarch64.+bchnVersion, bchnAmd64, bchnArm64 :: String+bchnVersion = "29.2.0"+bchnAmd64 = "23bbe9ff749136a9b832a2a59023ce76a5391322c1ea2f4f24b5b91f035081bc"+bchnArm64 = "27e177e5a07dc78c2c0d18798ea2e3925111955509ecdc476e828e7ce8f46383"++-- | Daemon of Bitcoin Core in the image.+coreBin :: String+coreBin = "/opt/bitcoin-core/bin/bitcoind"++-- | Daemon of Bitcoin Cash Node in the image.+bchnBin :: String+bchnBin = "/opt/bitcoin-cash-node/bin/bitcoind"++-- | Valkey server in the image.+valkeyBin :: String+valkeyBin = "/usr/bin/valkey-server"++-- | The HTTP load generator wrk in the image, for the stress test of the+-- cache.+wrkBin :: String+wrkBin = "/usr/bin/wrk"++-- | Recipe of the image.+--+-- Every download is its own layer, ordered from the one that changes least+-- often: the packages of the snapshot, then Bitcoin Core, then Bitcoin Cash+-- Node, then wrk for the stress test, which came last. Podman keeps the+-- layers it builds and reuses each one whose instruction, and those before+-- it, did not change, so moving one node to a new release downloads only+-- that release. Nothing in the image depends on the time it is built.+--+-- The recipe stays here rather than in a Containerfile of its own: it is+-- made from the versions and hashes defined in this module, and the tag of+-- the image is a hash of it, so the tests and the stress program, which may+-- run from any directory, need no file to find. It is written one+-- instruction and one shell command per line.+containerfile :: String+containerfile =+  unlines $+    [ "FROM " <> base,+      "ARG TARGETARCH",+      -- the snapshot archive is only served over HTTPS, so first get the+      -- certificates to reach it from the regular archive (over HTTP, with+      -- the packages checked by the signatures of the archive), then install+      -- everything, the certificates included, from the snapshot+      "RUN apt-get update --error-on=any \\",+      " && apt-get install -y --no-install-recommends ca-certificates \\",+      " && apt-get update --error-on=any --snapshot " <> snapshot <> " \\",+      " && apt-get install -y --no-install-recommends --snapshot "+        <> snapshot+        <> " ca-certificates curl valkey-server \\",+      " && rm -rf /var/lib/apt/lists/*"+    ]+      <> release+        "/opt/bitcoin-core"+        ( "https://bitcoincore.org/bin/bitcoin-core-"+            <> coreVersion+            <> "/bitcoin-"+            <> coreVersion+            <> "-$a-linux-gnu.tar.gz"+        )+        coreAmd64+        coreArm64+      <> release+        "/opt/bitcoin-cash-node"+        ( "https://github.com/bitcoin-cash-node/bitcoin-cash-node/releases/download/v"+            <> bchnVersion+            <> "/bitcoin-cash-node-"+            <> bchnVersion+            <> "-$a-linux-gnu.tar.gz"+        )+        bchnAmd64+        bchnArm64+      <> [ "RUN apt-get update --error-on=any --snapshot " <> snapshot <> " \\",+           " && apt-get install -y --no-install-recommends --snapshot " <> snapshot <> " wrk \\",+           " && rm -rf /var/lib/apt/lists/*"+         ]+  where+    -- download a release for the architecture of the image, check its hash,+    -- and unpack it; @$a@ in the URL stands for the architecture+    release dir url amd64 arm64 =+      [ "RUN set -eu; \\",+        " case \"$TARGETARCH\" in \\",+        "  amd64) a=x86_64; h=" <> amd64 <> ";; \\",+        "  arm64) a=aarch64; h=" <> arm64 <> ";; \\",+        "  *) echo \"unsupported architecture $TARGETARCH\"; exit 1;; \\",+        " esac; \\",+        " curl -fsSL -o /tmp/release.tar.gz \"" <> url <> "\"; \\",+        " echo \"$h  /tmp/release.tar.gz\" | sha256sum -c -; \\",+        " mkdir -p " <> dir <> "; \\",+        " tar -xzf /tmp/release.tar.gz -C " <> dir <> " --strip-components=1; \\",+        " rm /tmp/release.tar.gz"+      ]++-- | Tag of the image: a hash of its recipe.+tag :: String+tag =+  "localhost/haskoin-store-test:"+    <> take 16 (T.unpack (encodeHex (encode (sha256 (C8.pack containerfile)))))++-- | Build the image unless it exists already, and return its tag. The first+-- build downloads the base image, the packages and the two releases, and may+-- take a few minutes; later builds of a changed recipe reuse the layers that+-- did not change (see the recipe above). Images and layers are never removed by+-- the tests. The build may use 1 GiB of memory.+testImage :: IO String+testImage = do+  (exists, _, _) <- readProcessWithExitCode "podman" ["image", "exists", tag] ""+  case exists of+    ExitSuccess -> return tag+    _ -> withSystemTempDirectory "haskoin-store-image" $ \ctx -> do+      (code, out, err) <-+        readProcessWithExitCode+          "podman"+          ["build", "--layers", "--memory", "1g", "--tag", tag, "--file", "-", ctx]+          containerfile+      case code of+        ExitSuccess -> return tag+        _ ->+          fail $+            intercalate+              "\n"+              ["Could not build the test image " <> tag <> ":", lastLines out, lastLines err]+  where+    lastLines = unlines . reverse . take 30 . reverse . lines

@@ -0,0 +1,237 @@+-- |+-- Module      : Haskoin.Store.Integration.Node+-- Copyright   : (c) Haskoin Developers+-- License     : MIT+-- Maintainer  : jprupp@protonmail.ch+-- Stability   : experimental+-- Portability : POSIX+--+-- Regtest nodes for the integration tests: Bitcoin Core for Bitcoin and+-- Bitcoin Cash Node for Bitcoin Cash, each in a container, driven through+-- JSON-RPC.+--+-- A regtest node gives the tests a chain they fully control: they mine blocks+-- on demand, send transactions from the wallet of the node, and sign+-- transactions that spend from the keys of the test xpubs. The store under+-- test connects to the node over the peer-to-peer protocol, as it would to+-- the real networks.+--+-- Only RPC calls that both nodes have are used. Bitcoin Cash Node has no+-- @generateblock@, so the reorganisation test is built on coinbase maturity+-- instead (see "Haskoin.Store.IntegrationSpec").+module Haskoin.Store.Integration.Node+  ( NodeKind (..),+    Node (..),+    nodeContainer,+    nodeNetwork,+    RpcError (..),+    rpc,+    waitRpc,+    ensureWallet,+    field,+    textField,+    btc,+    sats,+    getNewAddress,+    generateTo,+    sendTo,+    sendMany,+    getRawTx,+    bestBlockHash,+    rawMempool,+  )+where++import Control.Lens ((&), (?~), (^.))+import Control.Monad (void)+import Data.Aeson (Value (..), eitherDecode, object, (.=))+import Data.Aeson.Key qualified as Key+import Data.Aeson.KeyMap qualified as KeyMap+import Data.Foldable (toList)+import Data.Text (Text)+import Data.Text qualified as T+import Data.Word (Word64)+import Haskoin (Network, bchRegTest, btcRegTest)+import Haskoin.Store.Integration.Image (bchnBin, coreBin)+import Haskoin.Store.Integration.Podman (ContainerSpec (..))+import Network.Wreq qualified as Wreq+import UnliftIO (Exception, throwIO, tryAny)+import UnliftIO.Concurrent (threadDelay)++-- | Which node software.+data NodeKind = BitcoinCore | BitcoinCashNode+  deriving (Eq, Show)++-- | A node and how to reach it.+data Node = Node+  { kind :: !NodeKind,+    -- | URL of the JSON-RPC interface+    url :: !String,+    -- | port of the peer-to-peer interface on 127.0.0.1+    p2p :: !Int+  }++-- | Network constants of the regtest chain of a node.+nodeNetwork :: NodeKind -> Network+nodeNetwork BitcoinCore = btcRegTest+nodeNetwork BitcoinCashNode = bchRegTest++-- | Container of a regtest node, from the test image+-- ("Haskoin.Store.Integration.Image"): its chain is in a tmpfs of 512 MiB,+-- it may use 1 GiB of memory, and its caches and mempool are kept small.+--+-- Every peer is whitelisted, so that the store may ask for the mempool and+-- gets transactions relayed without delay.+nodeContainer :: NodeKind -> String -> ContainerSpec+nodeContainer kind image =+  ContainerSpec+    { name = case kind of+        BitcoinCore -> "bitcoin-core"+        BitcoinCashNode -> "bitcoin-cash-node",+      image = image,+      options =+        [ "--memory",+          "1g",+          "--tmpfs",+          "/chain:rw,size=512m,mode=1777",+          "--entrypoint",+          case kind of+            BitcoinCore -> coreBin+            BitcoinCashNode -> bchnBin+        ],+      command =+        [ "-regtest",+          "-datadir=/chain",+          "-printtoconsole",+          "-server",+          "-rpcuser=test",+          "-rpcpassword=test",+          "-rpcbind=0.0.0.0",+          "-rpcallowip=0.0.0.0/0",+          "-bind=0.0.0.0",+          "-listen=1",+          "-whitelist=0.0.0.0/0",+          "-txindex=1",+          "-fallbackfee=0.0002",+          "-dbcache=16",+          "-maxmempool=16"+        ]+          <> case kind of+            -- releases of Bitcoin Cash Node stop serving RPC some months+            -- after they come out unless told not to+            BitcoinCashNode -> ["-expire=0"]+            BitcoinCore -> [],+      ports = [18443, 18444]+    }++-- | Error returned by a node.+data RpcError = RpcError !Text !Value+  deriving (Show)++instance Exception RpcError++-- | Call a method of the node and return its result.+rpc :: Node -> Text -> [Value] -> IO Value+rpc node method params = do+  let opts =+        Wreq.defaults+          & Wreq.auth ?~ Wreq.basicAuth "test" "test"+          & Wreq.checkResponse ?~ (\_ _ -> return ())+      body =+        object+          [ "jsonrpc" .= ("1.0" :: Text),+            "id" .= ("test" :: Text),+            "method" .= method,+            "params" .= params+          ]+  r <- Wreq.postWith opts node.url body+  case eitherDecode (r ^. Wreq.responseBody) of+    Left e -> fail ("Bad answer to " <> T.unpack method <> ": " <> e)+    Right v -> case (field "error" v, field "result" v) of+      (Just Null, Just x) -> return x+      (Nothing, Just x) -> return x+      (Just e, _) -> throwIO (RpcError method e)+      _ -> fail ("No result in answer to " <> T.unpack method)++-- | Wait until the node answers RPC calls, for up to 90 seconds.+waitRpc :: Node -> IO ()+waitRpc node = go (360 :: Int)+  where+    go n =+      tryAny (rpc node "getblockchaininfo" []) >>= \case+        Right _ -> return ()+        Left e+          | n <= 0 -> fail ("The node did not start: " <> show e)+          | otherwise -> threadDelay 250000 >> go (n - 1)++-- | Make sure the node has exactly one wallet loaded, so that wallet calls+-- need no wallet name. Bitcoin Cash Node loads a default wallet; Bitcoin Core+-- has none until one is created.+ensureWallet :: Node -> IO ()+ensureWallet node =+  rpc node "listwallets" [] >>= \case+    Array ws | not (null ws) -> return ()+    _ -> void $ rpc node "createwallet" [String "test"]++-- | Field of a JSON object.+field :: Text -> Value -> Maybe Value+field k (Object o) = KeyMap.lookup (Key.fromText k) o+field _ _ = Nothing++-- | Text field of a JSON object.+textField :: Text -> Value -> Maybe Text+textField k v = case field k v of+  Just (String t) -> Just t+  _ -> Nothing++-- | Amount in satoshi as a JSON number of coins.+btc :: Word64 -> Value+btc s = Number (fromIntegral s / 100000000)++-- | JSON number of coins in satoshi.+sats :: Value -> Word64+sats (Number n) = round (n * 100000000)+sats v = error ("Not an amount: " <> show v)++asText :: Value -> IO Text+asText (String t) = return t+asText v = fail ("Expected text, got " <> show v)++-- | New address of the wallet of the node.+getNewAddress :: Node -> IO Text+getNewAddress node = rpc node "getnewaddress" [] >>= asText++-- | Mine blocks paying to an address. Returns their hashes.+generateTo :: Node -> Int -> Text -> IO [Text]+generateTo node n addr =+  rpc node "generatetoaddress" [Number (fromIntegral n), String addr] >>= \case+    Array hs -> mapM asText (toList hs)+    v -> fail ("Unexpected answer to generatetoaddress: " <> show v)++-- | Send from the wallet of the node to an address. Returns the transaction+-- hash.+sendTo :: Node -> Text -> Word64 -> IO Text+sendTo node addr amount =+  rpc node "sendtoaddress" [String addr, btc amount] >>= asText++-- | Send from the wallet of the node to several addresses in one+-- transaction. Returns the transaction hash.+sendMany :: Node -> [(Text, Word64)] -> IO Text+sendMany node outs =+  rpc node "sendmany" [String "", object [Key.fromText a .= btc v | (a, v) <- outs]]+    >>= asText++-- | Decoded transaction.+getRawTx :: Node -> Text -> IO Value+getRawTx node txid = rpc node "getrawtransaction" [String txid, Bool True]++-- | Hash of the best block.+bestBlockHash :: Node -> IO Text+bestBlockHash node = rpc node "getbestblockhash" [] >>= asText++-- | Hashes of the transactions in the mempool.+rawMempool :: Node -> IO [Text]+rawMempool node =+  rpc node "getrawmempool" [] >>= \case+    Array ts -> mapM asText (toList ts)+    v -> fail ("Unexpected answer to getrawmempool: " <> show v)

@@ -0,0 +1,190 @@+-- |+-- Module      : Haskoin.Store.Integration.Podman+-- Copyright   : (c) Haskoin Developers+-- License     : MIT+-- Maintainer  : jprupp@protonmail.ch+-- Stability   : experimental+-- Portability : POSIX+--+-- Containers for the integration tests, run with Podman.+--+-- The integration tests need real nodes and a real Redis, which are easiest+-- to get, at pinned versions and without installing anything, as container+-- images. Podman runs them without a daemon and without root, so a developer+-- or a CI runner can use it as it is.+--+-- Everything is kept off the disk and cleaned up:+--+-- * containers run with @--rm@ and are removed with @podman rm --force@ when+--   the test that started them ends, even if it fails;+-- * the volumes that images declare are mounted as tmpfs+--   (@--image-volume tmpfs@), and data directories are tmpfs mounts of+--   limited size;+-- * every container has a memory limit;+-- * containers carry a label with the process ID of the test run, and+--   containers left by a test run that was killed (whose process is gone) are+--   removed before new ones start.+--+-- Ports are published on random ports of the loopback interface, so several+-- test runs can share a machine.+module Haskoin.Store.Integration.Podman+  ( podmanUsable,+    ContainerSpec (..),+    Container (..),+    withContainer,+    hostPort,+    withRamDir,+  )+where++import Control.Monad (forM_, unless, void)+import Data.Aeson qualified as A+import Data.Aeson.Key qualified as Key+import Data.Aeson.KeyMap qualified as KeyMap+import Data.ByteString.Char8 qualified as C8+import Data.ByteString.Lazy qualified as BL+import Data.Char (isDigit, isSpace)+import Data.List (dropWhileEnd)+import Data.Text qualified as T+import System.Environment (lookupEnv)+import System.Exit (ExitCode (..))+import System.IO (hPutStrLn, stderr)+import System.Process (getCurrentPid, readProcessWithExitCode)+import UnliftIO (bracket, onException, timeout, tryAny)+import UnliftIO.Directory (doesDirectoryExist, doesPathExist, getTemporaryDirectory)+import UnliftIO.Temporary (withTempDirectory)++-- | Whether Podman can run containers here, or why not. Set+-- @HASKOIN_STORE_NO_PODMAN@ to a value other than @0@ to skip the tests that+-- need it.+podmanUsable :: IO (Either String ())+podmanUsable = do+  skip <- lookupEnv "HASKOIN_STORE_NO_PODMAN"+  case skip of+    Just v | v `notElem` ["", "0"] -> return (Left "HASKOIN_STORE_NO_PODMAN is set")+    _ -> do+      r <-+        tryAny . timeout (60 * 1000 * 1000) $+          readProcessWithExitCode "podman" ["info", "--format", "{{.Host.Arch}}"] ""+      -- containers left by a test run that was killed are removed before+      -- anything else, even if this run fails before starting any+      case r of+        Right (Just (ExitSuccess, _, _)) -> removeStale+        _ -> return ()+      return $ case r of+        Left e -> Left ("Podman is not available: " <> show e)+        Right Nothing -> Left "podman info did not answer within a minute"+        Right (Just (ExitSuccess, _, _)) -> Right ()+        Right (Just (_, _, err)) -> Left ("Podman cannot run containers: " <> trim err)++-- | What to run.+data ContainerSpec = ContainerSpec+  { -- | short name, part of the container name+    name :: !String,+    -- | image, with a pinned tag+    image :: !String,+    -- | options of @podman run@, such as memory limits and tmpfs mounts+    options :: ![String],+    -- | command and arguments, after the image+    command :: ![String],+    -- | TCP ports to publish on random ports of 127.0.0.1+    ports :: ![Int]+  }++-- | A running container.+data Container = Container+  { -- | container ID+    cid :: !String,+    -- | container name+    name :: !String+  }++-- | Label on every container started by the tests. Its value is the process+-- ID of the test run.+testLabel :: String+testLabel = "haskoin-store-test"++-- | Run a container while an action runs, and remove it afterwards. If the+-- action throws, the last lines of the log of the container are printed to+-- help find out why.+withContainer :: ContainerSpec -> (Container -> IO a) -> IO a+withContainer spec act = do+  removeStale+  pid <- show <$> getCurrentPid+  let cname = testLabel <> "-" <> pid <> "-" <> spec.name+      args =+        ["run", "--detach", "--rm", "--name", cname]+          <> ["--label", testLabel <> "=" <> pid, "--image-volume", "tmpfs"]+          <> spec.options+          <> concatMap (\p -> ["--publish", "127.0.0.1::" <> show p]) spec.ports+          <> [spec.image]+          <> spec.command+  bracket (start cname args) stop $ \c -> act c `onException` logs c+  where+    start cname args = do+      (code, out, err) <- readProcessWithExitCode "podman" args ""+      case code of+        ExitSuccess -> return Container {cid = trim out, name = cname}+        _ -> do+          -- a container may exist even though the command failed+          void $ readProcessWithExitCode "podman" ["rm", "--force", cname] ""+          fail ("podman run " <> spec.image <> " failed: " <> trim err)+    stop :: Container -> IO ()+    stop c =+      void $ readProcessWithExitCode "podman" ["rm", "--force", "--time", "2", c.cid] ""+    logs :: Container -> IO ()+    logs c = do+      (_, out, err) <- readProcessWithExitCode "podman" ["logs", "--tail", "40", c.cid] ""+      hPutStrLn stderr $ "--- last lines of the log of " <> c.name <> " ---\n" <> out <> err++-- | Remove the containers of test runs whose process is gone. The list+-- comes as JSON: the Go templates of @podman ps@ cannot index the labels in+-- every version of Podman.+removeStale :: IO ()+removeStale = do+  (code, out, _) <-+    readProcessWithExitCode+      "podman"+      ["ps", "--all", "--filter", "label=" <> testLabel, "--format", "json"]+      ""+  case (code, A.decode (BL.fromStrict (C8.pack out))) of+    (ExitSuccess, Just cs) ->+      forM_ (cs :: [A.Value]) $ \c -> case staleOf c of+        Just (cid, pid) | all isDigit pid -> do+          alive <- doesPathExist ("/proc/" <> pid)+          unless alive . void $ readProcessWithExitCode "podman" ["rm", "--force", cid] ""+        _ -> return ()+    _ -> return ()+  where+    staleOf c = do+      A.Object o <- Just c+      A.String cid <- KeyMap.lookup "Id" o+      A.Object labels <- KeyMap.lookup "Labels" o+      A.String pid <- KeyMap.lookup (Key.fromString testLabel) labels+      return (T.unpack cid, T.unpack pid)++-- | Port of the host where a port of a container is published.+hostPort :: Container -> Int -> IO Int+hostPort c p = do+  (code, out, err) <- readProcessWithExitCode "podman" ["port", c.cid, show p <> "/tcp"] ""+  case (code, lines out) of+    (ExitSuccess, l : _)+      | [(n, "")] <- reads (reverse (takeWhile (/= ':') (reverse (trim l)))) ->+          return n+    _ -> fail ("Cannot find the published port " <> show p <> " of " <> c.name <> ": " <> trim err)++trim :: String -> String+trim = dropWhileEnd isSpace . dropWhile isSpace++-- | A temporary directory in memory: under @/dev/shm@ if there is one, else+-- under @$XDG_RUNTIME_DIR@, else the usual temporary directory.+withRamDir :: (FilePath -> IO a) -> IO a+withRamDir f = do+  shm <- doesDirectoryExist "/dev/shm"+  xdg <- lookupEnv "XDG_RUNTIME_DIR"+  tmp <- getTemporaryDirectory+  let base+        | shm = "/dev/shm"+        | Just x <- xdg = x+        | otherwise = tmp+  withTempDirectory base "haskoin-store-test" f

@@ -0,0 +1,57 @@+-- |+-- Module      : Haskoin.Store.Integration.Valkey+-- Copyright   : (c) Haskoin Developers+-- License     : MIT+-- Maintainer  : jprupp@protonmail.ch+-- Stability   : experimental+-- Portability : POSIX+--+-- A Redis server for the integration tests and the stress test: Valkey, the+-- Redis-compatible server that Ubuntu ships, from the test image.+module Haskoin.Store.Integration.Valkey+  ( redisContainer,+    waitRedis,+  )+where++import Database.Redis qualified as Redis+import Haskoin.Store.Integration.Image (valkeyBin)+import Haskoin.Store.Integration.Podman (ContainerSpec (..))+import UnliftIO (bracket, tryAny)+import UnliftIO.Concurrent (threadDelay)++-- | Valkey, the Redis-compatible server of Ubuntu, from the test image, listening on every interface of the container+-- (only its published port is reachable, on 127.0.0.1), with nothing saved+-- to disk and at most 128 MiB of data.+redisContainer :: String -> ContainerSpec+redisContainer image =+  ContainerSpec+    { name = "redis",+      image = image,+      options = ["--memory", "256m", "--entrypoint", valkeyBin],+      command =+        [ "--bind",+          "0.0.0.0",+          "--protected-mode",+          "no",+          "--save",+          "",+          "--appendonly",+          "no",+          "--maxmemory",+          "128mb"+        ],+      ports = [6379]+    }++-- | Wait until Redis answers, for up to 30 seconds.+waitRedis :: String -> IO ()+waitRedis url = go (120 :: Int)+  where+    info = either error id (Redis.parseConnectInfo url)+    go n =+      tryAny (bracket (Redis.checkedConnect info) Redis.disconnect (const (return ()))) >>= \case+        Right () -> return ()+        Left e+          | n <= 0 -> fail ("Redis did not start: " <> show e)+          | otherwise -> threadDelay 250000 >> go (n - 1)

@@ -1,5 +1,3 @@-{-# LANGUAGE DuplicateRecordFields #-}- -- | -- Module      : Haskoin.Store -- Copyright   : (c) Haskoin Developers@@ -14,33 +12,49 @@ -- <https://hackage.haskell.org/package/haskoin-node haskoin-node>, downloads -- blocks and mempool transactions, and indexes them in RocksDB. The indices -- answer questions about blocks, transactions, address balances, unspent--- outputs and extended public keys. An optional Redis cache speeds up queries--- about extended public keys with many addresses. A REST API built on these--- indices is in "Haskoin.Store.Web".+-- outputs and extended public keys. A REST API built on these indices is in+-- "Haskoin.Store.Web". (The Redis cache of extended public keys is being+-- rewritten; until it is back, every query reads the database.) -- -- = Running a store ----- 'withStore' starts everything from a 'StoreConfig': the database, the node,--- the block store that imports data, and the cache if configured. The 'Store'--- it returns has a 'DatabaseReader' and, if the cache is on, a 'CacheConfig'.--- Run queries with 'withDB', or inside 'withCache' to use the cache.+-- 'withStore' starts everything from a 'StoreConfig': the database, the node+-- and the block store that imports data. The 'Store' it returns has a+-- 'DatabaseReader'; query it with the functions of+-- "Haskoin.Store.Database.Query", which take the reader and run in 'IO'. -- -- @ -- 'withStore' cfg $ \\store ->---   'withSubscription' store.pub $ \\events -> do---     best <- 'withDB' store.db 'getBestBlock'+--   withSubscription store.pub $ \\events -> do+--     best <- Haskoin.Store.Database.Query.getBestBlock store.db --     ... -- @ -- -- = Modules -- -- * "Haskoin.Store.Data": types of the data returned by the store and the API.--- * "Haskoin.Store.Common": the 'StoreReadBase' and 'StoreReadExtra' query---   interfaces.--- * "Haskoin.Store.Database.Reader": queries against RocksDB.+-- * "Haskoin.Store.Common": helpers for paging, ordering and deriving+--   addresses, and 'StoreRead', the queries of a store as a record of+--   functions in 'IO'.+-- * "Haskoin.Store.Database.Reader": opening the RocksDB database.+-- * "Haskoin.Store.Database.Query": queries against RocksDB, as plain+--   functions.+-- * "Haskoin.Store.Database.Mempool", "Haskoin.Store.Database.Migrate",+--   "Haskoin.Store.Logic.Layered" and+--   "Haskoin.Store.Database.Query.Layered": a layout of the database in+--   which the mempool is kept apart from the chain, the migration to it, the+--   rules that import blocks and transactions in it, and its queries. The+--   store does not use them yet; see @docs\/layered-database.md@.+-- * "Haskoin.Store.Fleet": how stores that share a Redis agree on the block+--   that they all answer from, as pure decisions, and+--   "Haskoin.Store.Fleet.Redis", which carries them out.+--   "Haskoin.Store.Fleet.Pool" has the rules of the mempool that they share.+--   Nothing uses them yet.+-- * "Haskoin.Store.Database.Views": the database as it was at each recent+--   block, for a store that answers from the block its fleet agrees on.+--   Nothing uses it yet. -- * "Haskoin.Store.BlockStore": the process that imports blocks and --   transactions.--- * "Haskoin.Store.Cache": the Redis cache for extended public keys. -- * "Haskoin.Store.Web": the REST API server. -- = Purpose --@@ -64,9 +78,11 @@ --   query is a seek and a short scan, newest first; see --   "Haskoin.Store.Database.Types". ----- * __One query interface, several back ends.__ The database, the import buffer---   and the Redis cache all implement 'StoreReadBase' and 'StoreReadExtra', so---   the same query code runs against any of them.+-- * __Plain functions in 'IO'.__ Queries take the database reader, or a+--   'StoreRead' record of them, which the web server builds with the Redis+--   cache in front of the queries about xpubs. Nothing runs in a stack of+--   monad transformers, except the conduits that merge sorted streams. The+--   web server runs on WAI with a router of its own. -- -- * __The mempool is a first-class citizen.__ Unconfirmed transactions are indexed --   like confirmed ones, ordered before them, and kept across restarts, because@@ -84,16 +100,15 @@     module Haskoin.Store.Data,      -- * Cache+    Cache,     CacheConfig (..),-    CacheT,-    CacheError (..),-    withCache,-    connectRedis,-    isInCache,+    cacheIsCached,+    cacheDelete,+    cacheEvict,      -- * Store Reader-    StoreReadBase (..),-    StoreReadExtra (..),+    StoreRead (..),+    databaseRead,     Limits (..),     Start (..), @@ -110,6 +125,8 @@ import Haskoin.Store.Cache import Haskoin.Store.Common import Haskoin.Store.Data+-- the instances of the store classes for the database reader+import Haskoin.Store.Database.Query (databaseRead) import Haskoin.Store.Database.Reader import Haskoin.Store.Database.Types import Haskoin.Store.Manager

@@ -1,1502 +1,1221 @@ {-# LANGUAGE DeriveAnyClass #-}-{-# LANGUAGE DuplicateRecordFields #-}-{-# LANGUAGE FlexibleContexts #-}-{-# LANGUAGE FlexibleInstances #-}-{-# LANGUAGE ImportQualifiedPost #-}-{-# LANGUAGE LambdaCase #-}-{-# LANGUAGE NamedFieldPuns #-}-{-# LANGUAGE OverloadedRecordDot #-}-{-# LANGUAGE OverloadedStrings #-}-{-# LANGUAGE RecordWildCards #-}-{-# LANGUAGE TupleSections #-}-{-# LANGUAGE NoFieldSelectors #-}-{-# OPTIONS_GHC -Wno-ambiguous-fields #-}---- |--- Module      : Haskoin.Store.BlockStore--- Copyright   : (c) Haskoin Developers--- License     : MIT--- Maintainer  : jprupp@protonmail.ch--- Stability   : experimental--- Portability : POSIX------ The block store process downloads blocks and transactions from peers and--- imports them into the database.------ = Blocks------ When the best header of the node is ahead of the best block in the database,--- the block store picks one online peer as its syncing peer and asks it for--- blocks in batches of 'blockWindow'. Each block is imported as soon as it--- arrives. If the best block of the database is no longer in the main chain of--- headers, blocks are reverted until it is. A syncing peer that sends a block--- that was not requested, cannot find a block, or stays silent for longer than--- 'peerTimeout' is disconnected.------ = Mempool------ Once the database is in sync, the block store asks each peer for its mempool--- (if 'syncMempool' is set) and requests every transaction that peers--- announce. Received transactions are queued and imported in batches on every--- clock tick, in dependency order. Transactions that spend unknown outputs are--- kept for up to an hour, waiting for their parents. Mempool transactions older--- than 'mempoolTimeout' days are deleted.------ = Feeding the block store------ The block store does not listen to the node by itself.--- "Haskoin.Store.Manager" forwards node events to it with the--- @blockStore*@ functions of this module.--- = Why it works this way------ * __Blocks come from one peer at a time.__ Blocks must be imported in order,---   so downloading from several peers at once would only buffer blocks that cannot---   be imported yet. One syncing peer, kept busy with a sliding window of---   requests, uses the network well without unbounded buffering.------ * __The window refills early.__ The next batch is requested when fewer than a---   quarter of the window are still outstanding, so the peer never sits idle---   while the store imports, and the backlog never grows much beyond one window.------ * __Mempool sync waits for the tip.__ Unconfirmed transactions only make sense---   against the current chain, so they are requested once the database is in---   sync.------ * __Orphans get a grace period.__ A transaction can arrive before its parent.---   It is held for up to an hour, waiting for the parent, instead of being---   dropped and requested again.------ * __Imports are batched.__ Mempool transactions are imported up to 2000 at a---   time on each clock tick, which amortises the cost of a database write over---   many transactions during bursts.-module Haskoin.Store.BlockStore-  ( -- * Block Store-    BlockStore,-    BlockStoreConfig (..),-    withBlockStore,-    blockStorePeerConnect,-    blockStorePeerConnectSTM,-    blockStorePeerDisconnect,-    blockStorePeerDisconnectSTM,-    blockStoreHead,-    blockStoreHeadSTM,-    blockStoreBlock,-    blockStoreBlockSTM,-    blockStoreNotFound,-    blockStoreNotFoundSTM,-    blockStoreTx,-    blockStoreTxSTM,-    blockStoreTxHash,-    blockStoreTxHashSTM,-    blockStorePendingTxs,-    blockStorePendingTxsSTM,-    needsMoreBlocks,-  )-where--import Control.Monad-  ( forM,-    forM_,-    mzero,-    unless,-    void,-    when,-  )-import Control.Monad.Reader-  ( MonadReader,-    ReaderT (..),-    ask,-    asks,-  )-import Control.Monad.Trans (lift)-import Control.Monad.Trans.Maybe (MaybeT (MaybeT), runMaybeT)-import Data.Bool (bool)-import Data.ByteString qualified as B-import Data.HashMap.Strict (HashMap)-import Data.HashMap.Strict qualified as HashMap-import Data.HashSet (HashSet)-import Data.HashSet qualified as HashSet-import Data.List (delete)-import Data.Maybe-  ( catMaybes,-    fromJust,-    fromMaybe,-    isJust,-    mapMaybe,-  )-import Data.Serialize (encode)-import Data.String (fromString)-import Data.String.Conversions (cs)-import Data.Text (Text)-import Data.Text qualified as T-import Data.Time.Clock-  ( NominalDiffTime,-    UTCTime,-    diffUTCTime,-    getCurrentTime,-  )-import Data.Time.Clock.POSIX-  ( getPOSIXTime,-    posixSecondsToUTCTime,-    utcTimeToPOSIXSeconds,-  )-import Data.Time.Format-  ( defaultTimeLocale,-    formatTime,-  )-import Haskoin-  ( Block (..),-    BlockHash (..),-    BlockHeader (..),-    BlockHeight,-    BlockNode (..),-    Ctx,-    GetData (..),-    InvType (..),-    InvVector (..),-    Message (..),-    Network (..),-    OutPoint (..),-    Tx (..),-    TxHash (..),-    TxIn (..),-    blockHashToHex,-    headerHash,-    txHash,-    txHashToHex,-  )-import Haskoin.Node-  ( Chain,-    LogLevel (..),-    Logger,-    OnlinePeer (..),-    Peer (..),-    PeerMgr,-    chainBlockMain,-    chainGetAncestor,-    chainGetBest,-    chainGetBlock,-    chainGetParents,-    getPeers,-    killPeer,-    logAt,-    sendMessage,-    setBusy,-    setFree,-  )-import Haskoin.Store.Common-import Haskoin.Store.Data-import Haskoin.Store.Database.Reader-import Haskoin.Store.Database.Writer-import Haskoin.Store.Logic-  ( ImportException (Orphan),-    ImportM,-    deleteUnconfirmedTx,-    importBlock,-    initBest,-    newMempoolTx,-    revertBlock,-  )-import NQE-  ( InChan,-    Listen,-    Mailbox,-    Publisher,-    inboxToMailbox,-    newInbox,-    publish,-    query,-    receive,-    send,-    sendSTM,-  )-import System.Metrics.StatsD-import System.Random (randomRIO)-import UnliftIO-  ( Exception,-    MonadIO,-    MonadUnliftIO,-    STM,-    TVar,-    async,-    atomically,-    catch,-    liftIO,-    link,-    modifyTVar,-    modifyTVar',-    newTVarIO,-    readTVar,-    readTVarIO,-    throwIO,-    try,-    withAsync,-    writeTVar,-  )-import UnliftIO.Concurrent (threadDelay)---- | Messages handled by the block store process.-data BlockStoreMessage-  = -- | the node has a new best header-    BlockNewBest !BlockNode-  | -- | a peer connected-    BlockPeerConnect !Peer-  | -- | a peer disconnected-    BlockPeerDisconnect !Peer-  | -- | a peer sent a block-    BlockReceived !Peer !Block-  | -- | a peer could not find blocks-    BlockNotFound !Peer ![BlockHash]-  | -- | a peer sent a transaction-    TxRefReceived !Peer !Tx-  | -- | a peer announced transactions-    TxRefAvailable !Peer ![TxHash]-  | -- | clock tick for housekeeping; reply when done-    BlockPing !(Listen ())---- | Fatal errors of the block store.-data BlockException-  = -- | the header of a block in the database is not known to the node-    BlockNotInChain !BlockHash-  | -- | the database has no best block-    Uninitialized-  | -- | the database is inconsistent-    CorruptDatabase-  | -- | the node has no header at this height below this block-    AncestorNotInChain !BlockHeight !BlockHash-  | -- | importing mempool transactions failed-    MempoolImportFailed-  deriving (Show, Eq, Ord, Exception)---- | State of the sync against the syncing peer.-data Syncing = Syncing-  { -- | peer that blocks are downloaded from-    peer :: !Peer,-    -- | when it last delivered a block, or when syncing started-    time :: !UTCTime,-    -- | blocks requested and not yet imported, in chain order-    blocks :: ![BlockHash]-  }---- | Transaction received from a peer and waiting to be imported.-data PendingTx = PendingTx-  { -- | when it was received-    time :: !UTCTime,-    -- | the transaction-    tx :: !Tx,-    -- | parent transactions that are missing; empty if it can be imported-    deps :: !(HashSet TxHash)-  }-  deriving (Show, Eq, Ord)---- | Block store process state.-data BlockStore = BlockStore-  { -- | mailbox of the process-    mailbox :: !(Mailbox BlockStoreMessage),-    -- | configuration-    config :: !BlockStoreConfig,-    -- | syncing peer, if any-    peer :: !(TVar (Maybe Syncing)),-    -- | transactions received and waiting to be imported-    txs :: !(TVar (HashMap TxHash PendingTx)),-    -- | transactions requested from peers in the last twenty seconds-    requested :: !(TVar (HashSet TxHash)),-    -- | StatsD metrics, if enabled-    metrics :: !(Maybe StoreMetrics),-    -- | labels of the peers whose mempool was already requested-    syncedMempool :: !(TVar (HashSet Text))-  }---- | StatsD metrics of the block store.-data StoreMetrics = StoreMetrics-  { -- | height of the best block (@store.blocks@)-    blocks :: !StatGauge,-    -- | height of the best header (@store.headers@)-    headers :: !StatGauge,-    -- | transactions waiting to be imported (@store.queued_txs@)-    queuedTxs :: !StatGauge,-    -- | online peers (@store.peers@)-    peers :: !StatGauge,-    -- | transactions in the mempool (@store.mempool@)-    mempool :: !StatGauge,-    -- | milliseconds to import a block (@store.block@)-    block :: !StatTiming-  }---- | Register the block store metrics, if StatsD is configured, with their current values.-newStoreMetrics :: (MonadIO m) => BlockStoreConfig -> m (Maybe StoreMetrics)-newStoreMetrics cfg =-  forM cfg.stats $ \s -> liftIO $ do-    m <- withDB cfg.db getMempool-    b <--      fmap (maybe 0 (.height)) $-        withDB cfg.db $-          runMaybeT $-            MaybeT getBestBlock >>= MaybeT . getBlock-    h <- chainGetBest cfg.chain-    p <- getPeers cfg.peerMgr-    blocks <- g s "blocks" (fromIntegral b)-    headers <- g s "headers" (fromIntegral h.height)-    queuedTxs <- g s "queued_txs" 0-    peers <- g s "peers" (length p)-    mempool <- g s "mempool" (length m)-    block <- t s "block" 0-    return StoreMetrics {..}-  where-    g s x = newStatGauge s ("store." <> x)-    t s x = newStatTiming s ("store." <> x)---- | Update the gauge of the best block height.-setStoreHeight :: (MonadIO m) => BlockT m ()-setStoreHeight = void $ runMaybeT $ do-  m <- MaybeT (asks (.metrics))-  h <- MaybeT getBestBlock-  b <- MaybeT (getBlock h)-  setGauge m.blocks (fromIntegral b.height)---- | Update the gauge of the best header height.-setHeadersHeight :: (MonadIO m) => BlockT m ()-setHeadersHeight = void $ runMaybeT $ do-  m <- MaybeT (asks (.metrics))-  n <- liftIO . chainGetBest =<< asks (.config.chain)-  setGauge m.headers (fromIntegral n.height)---- | Update the gauge of transactions waiting to be imported.-setPendingTxs :: (MonadIO m) => BlockT m ()-setPendingTxs = void $ runMaybeT $ do-  m <- MaybeT (asks (.metrics))-  p <- readTVarIO =<< asks (.txs)-  setGauge m.queuedTxs (HashMap.size p)---- | Update the gauge of online peers.-setPeersConnected :: (MonadIO m) => BlockT m ()-setPeersConnected = void $ runMaybeT $ do-  m <- MaybeT (asks (.metrics))-  p <- liftIO . getPeers =<< asks (.config.peerMgr)-  setGauge m.peers (length p)---- | Record how long a successful block import took.-withBlockTiming :: (MonadIO m) => BlockT m (Either l r) -> BlockT m (Either l r)-withBlockTiming go = do-  asks (.metrics) >>= \case-    Just m -> do-      start <- round . (* 1000) <$> liftIO getPOSIXTime-      go >>= \case-        Right x -> do-          end <- round . (* 1000) <$> liftIO getPOSIXTime-          addTiming m.block (end - start)-          return (Right x)-        Left y -> return (Left y)-    Nothing -> go---- | Configuration for a block store.-data BlockStoreConfig = BlockStoreConfig-  { -- | secp256k1 context-    ctx :: !Ctx,-    -- | peer manager from running node-    peerMgr :: !PeerMgr,-    -- | chain from a running node-    chain :: !Chain,-    -- | listener for store events-    pub :: !(Publisher StoreEvent),-    -- | RocksDB database handle-    db :: !DatabaseReader,-    -- | network constants-    net :: !Network,-    -- | do not index new mempool transactions-    noMempool :: !Bool,-    -- | wipe mempool at start-    wipeMempool :: !Bool,-    -- | sync mempool from peers-    syncMempool :: !Bool,-    -- | blocks to request from the syncing peer in one batch-    blockWindow :: !Int,-    -- | delete transactions older than this number of days from mempool-    mempoolTimeout :: !Int,-    -- | disconnect syncing peer if inactive for this long-    peerTimeout :: !NominalDiffTime,-    -- | StatsD client, if metrics are enabled-    stats :: !(Maybe Stats),-    -- | log messages here-    logger :: !Logger-  }---- | Monad transformer of the block store process.-type BlockT m = ReaderT BlockStore m---- | Log a message with the logger of the block store.-logDebug, logInfo, logWarn, logError ::-  (MonadIO m, MonadReader BlockStore m) => Text -> m ()-logDebug = blockLog LevelDebug-logInfo = blockLog LevelInfo-logWarn = blockLog LevelWarn-logError = blockLog LevelError---- | Log a message at a level with the logger of the block store.-blockLog :: (MonadIO m, MonadReader BlockStore m) => LogLevel -> Text -> m ()-blockLog l msg = asks (.config.logger) >>= \lg -> logAt l lg msg---- | Run an import against the database, returning the import error if it fails.-runImport ::-  (MonadIO m) =>-  Network ->-  Ctx ->-  ImportM a ->-  BlockT m (Either ImportException a)-runImport net ctx f = fmap fst <$> runImportMempool net ctx f---- | Like 'runImport', but also return how the import changed the mempool.--- The mempool gauge is kept up to date from those changes instead of--- counting the whole mempool on every clock tick.-runImportMempool ::-  (MonadIO m) =>-  Network ->-  Ctx ->-  ImportM a ->-  BlockT m (Either ImportException (a, MempoolChanges))-runImportMempool net ctx f = do-  r <- ReaderT $ \r ->-    liftIO . try $ runWriterMempool net ctx r.config.db r.config.logger f-  forM_ r $ \(_, c) -> asks (.metrics) >>= mapM_ (\m -> do-    incrementGauge m.mempool (length c.added)-    decrementGauge m.mempool (length c.removed))-  return r---- | Run an import and publish the transactions that it added to the--- mempool or removed from it, then the new best block if there is one.--- Transactions of the block are not published.-runImportNotify ::-  (MonadIO m) =>-  Maybe Block ->-  ImportM a ->-  BlockT m (Either ImportException a)-runImportNotify block f = do-  net <- getNetwork-  ctx <- getCtx-  runImportMempool net ctx f >>= \case-    Left e -> return (Left e)-    Right (x, c) -> do-      l <- asks (.config.pub)-      forM_ (filter notInBlock c.removed) $ \h ->-        publish (StoreMempoolDelete h) l-      forM_ (filter notInBlock c.added) $ \h ->-        publish (StoreMempoolNew h) l-      forM_ block $ \b ->-        publish (StoreBestBlock (headerHash b.header)) l-      return (Right x)-  where-    inBlock = maybe HashSet.empty (HashSet.fromList . map txHash . (.txs)) block-    notInBlock = not . (`HashSet.member` inBlock)---- | Run a database query from the block store.-runRocksDB :: ReaderT DatabaseReader m a -> BlockT m a-runRocksDB f =-  ReaderT $ runReaderT f . (.config.db)--instance (MonadIO m) => StoreReadBase (BlockT m) where-  getCtx =-    asks (.config.ctx)-  getNetwork =-    runRocksDB getNetwork-  getBestBlock =-    runRocksDB getBestBlock-  getBlocksAtHeight =-    runRocksDB . getBlocksAtHeight-  getBlock =-    runRocksDB . getBlock-  getTxData =-    runRocksDB . getTxData-  getManyTxData =-    runRocksDB . getManyTxData-  getSpender =-    runRocksDB . getSpender-  getUnspent =-    runRocksDB . getUnspent-  getBalance =-    runRocksDB . getBalance-  getMempool =-    runRocksDB getMempool-  getMempoolPage =-    runRocksDB . getMempoolPage--instance (MonadUnliftIO m) => StoreReadExtra (BlockT m) where-  getMaxGap =-    runRocksDB getMaxGap-  getInitialGap =-    runRocksDB getInitialGap-  getAddressesTxs as =-    runRocksDB . getAddressesTxs as-  getAddressesUnspents as =-    runRocksDB . getAddressesUnspents as-  getAddressUnspents a =-    runRocksDB . getAddressUnspents a-  getAddressTxs a =-    runRocksDB . getAddressTxs a-  getNumTxData =-    runRocksDB . getNumTxData-  getBalances =-    runRocksDB . getBalances-  xPubBals =-    runRocksDB . xPubBals-  xPubUnspents x l =-    runRocksDB . xPubUnspents x l-  xPubTxs x l =-    runRocksDB . xPubTxs x l-  xPubTxCount x =-    runRocksDB . xPubTxCount x---- | Run block store process.-withBlockStore ::-  (MonadUnliftIO m) =>-  BlockStoreConfig ->-  (BlockStore -> m a) ->-  m a-withBlockStore cfg action = do-  pb <- newTVarIO Nothing-  ts <- newTVarIO HashMap.empty-  rq <- newTVarIO HashSet.empty-  sy <- newTVarIO HashSet.empty-  inbox <- newInbox-  metrics <- newStoreMetrics cfg-  let r =-        BlockStore-          { mailbox = inboxToMailbox inbox,-            config = cfg,-            peer = pb,-            txs = ts,-            requested = rq,-            metrics = metrics,-            syncedMempool = sy-          }-  withAsync (runReaderT (go inbox) r) $ \a -> do-    link a-    action r-  where-    go inbox = do-      ini-      wipe-      run inbox-    del txs = do-      logAt LevelInfo cfg.logger $-        "Deleting " <> cs (show (length txs)) <> " transactions"-      forM_ txs $ \(_, th) -> deleteUnconfirmedTx False th-    wipe_it txs = do-      net <- getNetwork-      ctx <- getCtx-      let (txs1, txs2) = splitAt 1000 txs-      unless (null txs1) $-        runImport net ctx (del txs1) >>= \case-          Left e -> do-            logError $-              "Could not wipe mempool: " <> cs (show e)-            throwIO e-          Right () -> wipe_it txs2-    wipe-      | cfg.wipeMempool =-          getMempool >>= wipe_it-      | otherwise =-          return ()-    ini = do-      net <- getNetwork-      ctx <- getCtx-      runImport net ctx initBest >>= \case-        Left e -> do-          logError $-            "Could not initialize: " <> cs (show e)-          throwIO e-        Right () -> return ()-    run inbox =-      withAsync (pingMe (inboxToMailbox inbox)) $ \a ->-        link a >> runBlockStoreLoop inbox---- | Handle messages forever.-runBlockStoreLoop ::-  (InChan mbox, MonadUnliftIO m) =>-  mbox BlockStoreMessage ->-  ReaderT BlockStore m b-runBlockStoreLoop inbox =-  loopForever $ do-    logDebug "Waiting for new event..."-    msg <- receive inbox-    ReaderT $ runReaderT $ processBlockStoreMessage msg---- | Whether the best block in the database is the best header of the node. If--- so, release the syncing peer.-isInSync :: (MonadIO m) => BlockT m Bool-isInSync =-  getBestBlock >>= \case-    Nothing -> do-      logError "Block database uninitialized"-      throwIO Uninitialized-    Just bb -> do-      cb <- asks (.config.chain) >>= liftIO . chainGetBest-      if headerHash cb.header == bb-        then clearSyncingState >> return True-        else return False---- | Run an action only if the database is in sync.-guardSync :: (MonadIO m) => BlockT m () -> BlockT m ()-guardSync f = isInSync >>= \s -> when s f---- | Run an action only if the mempool is indexed.-guardMempool :: (Monad m) => BlockT m () -> BlockT m ()-guardMempool f = asks (.config.noMempool) >>= \n -> unless n f---- | Ask a peer for its mempool, once per peer, if in sync and mempool sync is enabled.-syncMempool :: (MonadUnliftIO m) => Peer -> BlockT m ()-syncMempool p =-  guardMempool . guardSync $ do-    s <- asks (.config.syncMempool)-    m <- asks (.syncedMempool)-    y <- not . HashSet.member p.label <$> readTVarIO m-    when (s && y) $ do-      logDebug $-        "Requesting mempool from peer: " <> p.label-      atomically . modifyTVar m $ HashSet.insert p.label-      liftIO $ MMempool `sendMessage` p---- | Import a block from the syncing peer, then ask for more blocks or, if now in--- sync, sync the mempool. Disconnects the peer if it is not the syncing peer, if--- the block is unknown to the node, or if the block cannot be imported.-processBlock ::-  (MonadUnliftIO m) =>-  Peer ->-  Block ->-  BlockT m ()-processBlock peer block = void . runMaybeT $ do-  checkPeer peer >>= \case-    True -> return ()-    False -> do-      logError $-        "Non-syncing peer "-          <> peer.label-          <> " sent block "-          <> blockHashToHex blockhash-      killPeer peer-      mzero-  node <--    getBlockNode blockhash >>= \case-      Just b -> return b-      Nothing -> do-        logError $-          "Peer "-            <> peer.label-            <> " sent unknown block "-            <> blockHashToHex blockhash-        killPeer peer-        mzero-  logDebug $-    "Processing block "-      <> blockText node Nothing-      <> " from peer "-      <> peer.label-  lift $ do-    ret <- withBlockTiming $ runImportNotify (Just block) (importBlock block node)-    case ret of-      Left e -> failure e-      Right _ -> success node-  where-    header = block.header-    blockhash = headerHash header-    hexhash = blockHashToHex blockhash-    success node = do-      logInfo $-        "Best block: " <> blockText node (Just block)-      removeSyncingBlock $ headerHash node.header-      touchPeer-      isInSync >>= \case-        False -> syncMe-        True -> do-          updateOrphans-          syncMempool peer-    failure e = do-      logError $-        "Error importing block "-          <> hexhash-          <> " from peer: "-          <> peer.label-          <> ": "-          <> cs (show e)-      killPeer peer---- | Set the blocks requested from the syncing peer.-setSyncingBlocks ::-  (MonadReader BlockStore m, MonadIO m) =>-  [BlockHash] ->-  m ()-setSyncingBlocks hs =-  asks (.peer) >>= \box ->-    atomically $-      modifyTVar box $ \case-        Nothing -> Nothing-        Just x -> Just (x :: Syncing) {blocks = hs}---- | Blocks requested from the syncing peer and not yet imported.-getSyncingBlocks :: (MonadReader BlockStore m, MonadIO m) => m [BlockHash]-getSyncingBlocks =-  asks (.peer) >>= readTVarIO >>= \case-    Nothing -> return []-    Just x -> return x.blocks---- | Add blocks to those requested from the syncing peer.-addSyncingBlocks ::-  (MonadReader BlockStore m, MonadIO m) =>-  [BlockHash] ->-  m ()-addSyncingBlocks hs =-  asks (.peer) >>= \box ->-    atomically $-      modifyTVar box $ \case-        Nothing -> Nothing-        Just x -> Just (x :: Syncing) {blocks = x.blocks <> hs}---- | Remove a block from those requested from the syncing peer.-removeSyncingBlock ::-  (MonadReader BlockStore m, MonadIO m) =>-  BlockHash ->-  m ()-removeSyncingBlock h = do-  box <- asks (.peer)-  atomically $-    modifyTVar box $ \case-      Nothing -> Nothing-      Just x -> Just (x :: Syncing) {blocks = delete h x.blocks}---- | Whether a peer is the syncing peer.-checkPeer :: (MonadIO m, MonadReader BlockStore m) => Peer -> m Bool-checkPeer p = fmap (fromMaybe False) <$> runMaybeT $ do-  Syncing {peer} <- MaybeT getSyncingState-  return $ peer == p---- | Header of a block from the node.-getBlockNode ::-  (MonadIO m, MonadReader BlockStore m) =>-  BlockHash ->-  m (Maybe BlockNode)-getBlockNode blockhash = do-  ch <- asks (.config.chain)-  liftIO $ chainGetBlock ch blockhash---- | Disconnect a peer that could not find requested blocks.-processNoBlocks ::-  (MonadIO m) =>-  Peer ->-  [BlockHash] ->-  BlockT m ()-processNoBlocks p hs = do-  forM_ (zip [(1 :: Int) ..] hs) $ \(i, h) ->-    logError $-      "Block "-        <> cs (show i)-        <> "/"-        <> cs (show (length hs))-        <> " "-        <> blockHashToHex h-        <> " not found by peer: "-        <> p.label-  killPeer p---- | Queue a transaction received from a peer for import.-processTx :: (MonadIO m) => Peer -> Tx -> BlockT m ()-processTx p tx = guardMempool $ do-  t <- liftIO getCurrentTime-  logDebug $-    "Received tx "-      <> txHashToHex (txHash tx)-      <> " by peer: "-      <> p.label-  addPendingTx $ PendingTx t tx HashSet.empty---- | Drop queued transactions received more than an hour ago.-pruneOrphans :: (MonadIO m) => BlockT m ()-pruneOrphans = guardMempool $ do-  ts <- asks (.txs)-  now <- liftIO getCurrentTime-  -- keep pending transactions received within the last hour, drop older ones-  atomically . modifyTVar' ts . HashMap.filter $ \p ->-    now `diffUTCTime` p.time <= 3600---- | Queue a transaction for import.-addPendingTx :: (MonadIO m) => PendingTx -> BlockT m ()-addPendingTx p = do-  ts <- asks (.txs)-  rq <- asks (.requested)-  atomically $ do-    modifyTVar ts $ HashMap.insert th p-    modifyTVar rq $ HashSet.delete th-  setPendingTxs-  where-    th = txHash p.tx---- | Remember for twenty seconds that a transaction was requested, so that it is not requested again.-addRequestedTx :: (MonadIO m) => TxHash -> BlockT m ()-addRequestedTx th = do-  qbox <- asks (.requested)-  atomically $ modifyTVar qbox $ HashSet.insert th-  liftIO . void . async $ do-    threadDelay 20000000-    atomically $ modifyTVar qbox $ HashSet.delete th---- | Whether a transaction is queued or was requested recently.-isPending :: (MonadIO m) => TxHash -> BlockT m Bool-isPending th = do-  tbox <- asks (.txs)-  qbox <- asks (.requested)-  atomically $ do-    ts <- readTVar tbox-    rs <- readTVar qbox-    return $-      th `HashMap.member` ts-        || th `HashSet.member` rs---- | Take up to this many queued transactions that have no missing parents, in--- dependency order, out of the queue.-pendingTxs :: (MonadIO m) => Int -> BlockT m [PendingTx]-pendingTxs i = do-  selected <--    asks (.txs) >>= \box -> atomically $ do-      pending <- readTVar box-      let (selected, rest) = select pending-      writeTVar box rest-      return selected-  setPendingTxs-  return selected-  where-    select pend =-      let eligible = HashMap.filter (null . (.deps)) pend-          orphans = HashMap.difference pend eligible-          selected = take i $ sortit eligible-          remaining = HashMap.filter (`notElem` selected) eligible-       in (selected, remaining <> orphans)-    sortit m =-      let sorted = sortTxs $ map (.tx) $ HashMap.elems m-          txids = map (txHash . snd) sorted-       in mapMaybe (`HashMap.lookup` m) txids---- | Mark a transaction as no longer missing for the queued transactions that spend it.-fulfillOrphans :: (MonadIO m) => BlockStore -> TxHash -> m ()-fulfillOrphans block_read th =-  atomically $ modifyTVar box (HashMap.map fulfill)-  where-    box = block_read.txs-    fulfill p = p {deps = HashSet.delete th p.deps}---- | Drop queued transactions that are already in the database, and recompute the--- missing parents of the others.-updateOrphans ::-  ( StoreReadBase m,-    MonadIO m,-    MonadReader BlockStore m-  ) =>-  m ()-updateOrphans = do-  box <- asks (.txs)-  pending <- readTVarIO box-  let orphans = HashMap.filter (not . null . (.deps)) pending-  updated <- forM orphans $ \p -> do-    let tx = p.tx-    exists (txHash tx) >>= \case-      True -> return Nothing-      False -> Just <$> fill_deps p-  let pruned = HashMap.map fromJust $ HashMap.filter isJust updated-  atomically $ writeTVar box pruned-  where-    exists th =-      getTxData th >>= \case-        Nothing -> return False-        Just TxData {deleted = True} -> return False-        Just TxData {deleted = False} -> return True-    prev_utxos tx = catMaybes <$> mapM (getUnspent . (.outpoint)) tx.inputs-    fulfill p unspent =-      let unspent_hash = unspent.outpoint.hash-          new_deps = HashSet.delete unspent_hash p.deps-       in p {deps = new_deps}-    fill_deps p = do-      let tx = p.tx-      unspents <- prev_utxos tx-      return $ foldl fulfill p unspents---- | Queue a transaction that spends unknown outputs, recording which parents are missing.-newOrphanTx ::-  (MonadIO m) =>-  BlockStore ->-  UTCTime ->-  Tx ->-  WriterT m ()-newOrphanTx block_read time tx = do-  logAt LevelDebug block_read.config.logger $-    "Import tx "-      <> txHashToHex (txHash tx)-      <> ": Orphan"-  let box = block_read.txs-  unspents <- catMaybes <$> mapM getUnspent prevs-  let unspent_set = HashSet.fromList (map (.outpoint) unspents)-      missing_set = HashSet.difference prev_set unspent_set-      missing_txs = HashSet.map (.hash) missing_set-  atomically . modifyTVar box $-    HashMap.insert-      (txHash tx)-      PendingTx-        { time = time,-          tx = tx,-          deps = missing_txs-        }-  where-    prev_set = HashSet.fromList prevs-    prevs = map (.outpoint) tx.inputs---- | Import a transaction into the mempool. Returns 'True' if imported. A--- transaction that spends unknown outputs is queued as an orphan; other errors--- are ignored.-importMempoolTx :: BlockStore -> UTCTime -> Tx -> ImportM Bool-importMempoolTx block_read time tx =-  catch new_mempool_tx handle_error-  where-    tx_hash = txHash tx-    handle_error :: ImportException -> ImportM Bool-    handle_error Orphan = do-      newOrphanTx block_read time tx-      return False-    handle_error _ = return False-    seconds = floor (utcTimeToPOSIXSeconds time)-    new_mempool_tx = do-      t <- newMempoolTx tx seconds-      logAt LevelInfo block_read.config.logger $-        bool "Already have" "Imported" t-          <> " tx "-          <> txHashToHex (txHash tx)-      when t $ fulfillOrphans block_read tx_hash-      return t---- | Import up to 2000 queued transactions.-processMempool :: (MonadIO m) => BlockT m ()-processMempool = guardMempool $ do-  txs <- pendingTxs 2000-  block_read <- ask-  unless (null txs) $ import_txs block_read txs-  where-    run_import block_read p =-      importMempoolTx block_read p.time p.tx-    import_txs block_read txs =-      runImportNotify Nothing (mapM (run_import block_read) txs) >>= \case-        Left e -> report_error e-        Right _ -> return ()-    report_error e = do-      logError $-        "Error processing mempool: " <> cs (show e)-      throwIO e---- | Delete up to 1000 mempool transactions older than the configured number of days.-pruneMempool :: (MonadIO m) => BlockT m ()-pruneMempool =-  guardMempool $ do-    days <- asks (.config.mempoolTimeout)-    when (days > 0) $ do-      time <- (floor . utcTimeToPOSIXSeconds) <$> liftIO getCurrentTime-      let thresh = time - (fromIntegral days * 24 * 60 * 60)-      -- runs on every clock tick: read only the expired entries, oldest-      -- first, from the index by time-      txs <- map snd <$> runRocksDB (getMempoolBefore thresh 1000)-      unless (null txs) $ do-        logInfo $-          "Deleting " <> cs (show (length txs)) <> " old transactions from mempool"-        runImportNotify Nothing (mapM_ (deleteUnconfirmedTx False) txs) >>= \case-          Left e ->-            logError $-              "Error pruning mempool: " <> cs (show e)-          Right () -> return ()---- | Request from a peer the announced transactions that are neither queued,--- recently requested nor in the database. Only while in sync.-processTxs ::-  (MonadIO m) =>-  Peer ->-  [TxHash] ->-  BlockT m ()-processTxs p hs = guardMempool . guardSync $ do-  logDebug $-    "Received inventory with "-      <> cs (show (length hs))-      <> " transactions from peer: "-      <> p.label-  xs <- catMaybes <$> zip_counter process_tx-  unless (null xs) $ go xs-  where-    len = length hs-    zip_counter = forM (zip [(1 :: Int) ..] hs) . uncurry-    process_tx i h =-      isPending h >>= \case-        True -> do-          logDebug $-            "Tx "-              <> cs (show i)-              <> "/"-              <> cs (show len)-              <> " "-              <> txHashToHex h-              <> ": "-              <> "Already Pending"-          return Nothing-        False ->-          getTxData h >>= \case-            Just _ -> do-              logDebug $-                "Tx "-                  <> cs (show i)-                  <> "/"-                  <> cs (show len)-                  <> " "-                  <> txHashToHex h-                  <> ": "-                  <> "Already Imported"-              return Nothing-            Nothing -> do-              logDebug $-                "Tx "-                  <> cs (show i)-                  <> "/"-                  <> cs (show len)-                  <> " "-                  <> txHashToHex h-                  <> ": "-                  <> "Requesting"-              return (Just h)-    go xs = do-      mapM_ addRequestedTx xs-      net <- asks (.config.net)-      let inv = if net.segWit then InvWitnessTx else InvTx-          vec = map (InvVector inv . (.get)) xs-          msg = MGetData (GetData vec)-      liftIO $ msg `sendMessage` p---- | Record that the syncing peer just delivered something.-touchPeer ::-  ( MonadIO m,-    MonadReader BlockStore m-  ) =>-  m ()-touchPeer =-  getSyncingState >>= \case-    Nothing -> return ()-    Just _ -> do-      box <- asks (.peer)-      now <- liftIO getCurrentTime-      atomically $-        modifyTVar box $-          fmap $-            \x -> (x :: Syncing) {time = now}---- | Disconnect the syncing peer if it has been silent for too long.-checkTime :: (MonadIO m) => BlockT m ()-checkTime =-  asks (.peer) >>= readTVarIO >>= \case-    Nothing -> return ()-    Just-      Syncing-        { time = t,-          peer = p-        } -> do-        now <- liftIO getCurrentTime-        peer_time_out <- asks (.config.peerTimeout)-        when (now `diffUTCTime` t > peer_time_out) $ do-          logError $-            "Syncing peer timeout: " <> p.label-          killPeer p---- | Revert blocks until the best block in the database is in the main chain of headers.-revertToMainChain :: (MonadIO m) => BlockT m ()-revertToMainChain = do-  h <- headerHash . (.header) <$> getBest-  ch <- asks (.config.chain)-  net <- getNetwork-  ctx <- getCtx-  liftIO (chainBlockMain ch h) >>= \x -> unless x $ do-    logWarn $-      "Reverting best block: "-        <> blockHashToHex h-    runImport net ctx (revertBlock h) >>= \case-      Left e -> do-        logError $-          "Could not revert block "-            <> blockHashToHex h-            <> ": "-            <> cs (show e)-        throwIO e-      Right () -> setSyncingBlocks []-    revertToMainChain---- | Header of the best block in the database.-getBest :: (MonadIO m) => BlockT m BlockNode-getBest = do-  bb <--    getBestBlock >>= \case-      Just b -> return b-      Nothing -> do-        logError "No best block set"-        throwIO Uninitialized-  ch <- asks (.config.chain)-  liftIO (chainGetBlock ch bb) >>= \case-    Just x -> return x-    Nothing -> do-      logError $-        "Header not found for best block: "-          <> blockHashToHex bb-      throwIO (BlockNotInChain bb)---- | Header of the last block requested from the syncing peer, or of the best block if none.-getSyncBest :: (MonadIO m) => BlockT m BlockNode-getSyncBest = do-  bb <--    getSyncingBlocks >>= \case-      [] ->-        getBestBlock >>= \case-          Just b -> return b-          Nothing -> do-            logError "No best block set"-            throwIO Uninitialized-      hs -> return $ last hs-  ch <- asks (.config.chain)-  liftIO (chainGetBlock ch bb) >>= \case-    Just x -> return x-    Nothing -> do-      logError $-        "Header not found for block: "-          <> blockHashToHex bb-      throwIO (BlockNotInChain bb)---- | The syncing peer, when it is time to ask it for the next batch of blocks.-shouldSync :: (MonadIO m) => BlockT m (Maybe Peer)-shouldSync =-  isInSync >>= \case-    True -> return Nothing-    False ->-      getSyncingState >>= \case-        Nothing -> return Nothing-        Just Syncing {peer = p, blocks = bs} -> do-          window <- blockWindow-          return $-            if needsMoreBlocks window (length bs)-              then Just p-              else Nothing---- | Blocks to request from the syncing peer in one batch. Never below one,--- or the store would never ask for any block.-blockWindow :: (Monad m) => BlockT m Int-blockWindow = asks (max 1 . (.config.blockWindow))---- | Whether to ask for the next batch, given the window size and how many--- requested blocks are still waiting to be imported. Asking once fewer than--- a quarter of the window remain keeps the peer busy, while the queue of--- downloaded blocks never grows much beyond one window. The threshold is at--- least one, so that a window smaller than four still refills once empty.-needsMoreBlocks :: Int -> Int -> Bool-needsMoreBlocks window outstanding =-  outstanding < max 1 (window `div` 4)---- | Ask the syncing peer for the next batch of blocks, if the window has room.-syncMe :: (MonadIO m) => BlockT m ()-syncMe = do-  revertToMainChain-  shouldSync >>= \case-    Nothing -> return ()-    Just p -> do-      bb <- getSyncBest-      bh <- getbh-      when (bb /= bh) $ do-        bns <- sel bb bh-        iv <- getiv bns-        logDebug $-          "Requesting "-            <> fromString (show (length iv))-            <> " blocks from peer: "-            <> p.label-        addSyncingBlocks $ map (headerHash . (.header)) bns-        liftIO $ MGetData (GetData iv) `sendMessage` p-  where-    getiv bns = do-      w <- asks (.config.net.segWit)-      let i = if w then InvWitnessBlock else InvBlock-          f = InvVector i . (.get) . headerHash . (.header)-      return $ map f bns-    getbh =-      liftIO . chainGetBest =<< asks (.config.chain)-    -- The blocks right after the last requested one, up to one window or-    -- the chain tip, whichever comes first.-    sel bb bh = do-      window <- blockWindow-      top <- findAncestor (min (bb.height + fromIntegral window) bh.height) bh-      ch <- asks (.config.chain)-      ps <- liftIO $ chainGetParents ch (bb.height + 1) top-      return $ ps <> [top]---- | Ancestor of a header at a height. Throws 'AncestorNotInChain' if missing.-findAncestor ::-  (MonadIO m, MonadReader BlockStore m) =>-  BlockHeight ->-  BlockNode ->-  m BlockNode-findAncestor height target = do-  ch <- asks (.config.chain)-  liftIO (chainGetAncestor ch height target) >>= \case-    Just ancestor -> return ancestor-    Nothing -> do-      let h = headerHash target.header-      logError $-        "Could not find header for ancestor of block "-          <> blockHashToHex h-          <> " at height "-          <> cs (show target.height)-      throwIO $ AncestorNotInChain height h---- | Forget a peer: release it if it was the syncing peer, and forget that its mempool was requested.-finishPeer ::-  (MonadIO m, MonadReader BlockStore m) =>-  Peer ->-  m ()-finishPeer p = do-  box <- asks (.peer)-  readTVarIO box >>= \case-    Just Syncing {peer = p'} | p == p' -> reset_it box-    _ -> return ()-  y <- asks (.syncedMempool)-  atomically $ modifyTVar y $ HashSet.delete p.label-  where-    reset_it box = do-      atomically $ writeTVar box Nothing-      logDebug $ "Releasing peer: " <> p.label-      liftIO $ setFree p---- | Make a peer the syncing peer if there is none and the peer is not busy.-trySetPeer :: (MonadIO m) => Peer -> BlockT m Bool-trySetPeer p =-  getSyncingState >>= \case-    Just _ -> return False-    Nothing -> set_it-  where-    set_it =-      liftIO (setBusy p) >>= \case-        False -> return False-        True -> do-          logDebug $-            "Locked peer: " <> p.label-          box <- asks (.peer)-          now <- liftIO getCurrentTime-          atomically . writeTVar box $-            Just-              Syncing-                { peer = p,-                  time = now,-                  blocks = []-                }-          return True---- | If not in sync and there is no syncing peer, pick the fastest free peer and start syncing.-trySyncing :: (MonadIO m) => BlockT m ()-trySyncing =-  isInSync >>= \case-    True -> return ()-    False ->-      getSyncingState >>= \case-        Just _ -> return ()-        Nothing -> online_peer-  where-    recurse [] = return ()-    recurse (p : ps) =-      trySetPeer p >>= \case-        False -> recurse ps-        True -> syncMe-    online_peer = do-      ops <- liftIO . getPeers =<< asks (.config.peerMgr)-      let ps = map (.mailbox) ops-      recurse ps---- | Start syncing against a newly connected peer if needed, or request its mempool if in sync.-trySyncingPeer :: (MonadUnliftIO m) => Peer -> BlockT m ()-trySyncingPeer p =-  isInSync >>= \case-    True -> syncMempool p-    False ->-      trySetPeer p >>= \case-        False -> return ()-        True -> syncMe---- | State of the sync against the syncing peer, if any.-getSyncingState ::-  (MonadIO m, MonadReader BlockStore m) => m (Maybe Syncing)-getSyncingState =-  readTVarIO =<< asks (.peer)---- | Release the syncing peer, if any.-clearSyncingState ::-  (MonadIO m, MonadReader BlockStore m) => m ()-clearSyncingState =-  asks (.peer) >>= readTVarIO >>= \case-    Nothing -> return ()-    Just Syncing {peer = p} -> finishPeer p---- | Handle a message sent to the block store.-processBlockStoreMessage ::-  (MonadUnliftIO m) =>-  BlockStoreMessage ->-  BlockT m ()-processBlockStoreMessage (BlockNewBest node) = do-  logDebug $-    "New best block mined at height "-      <> cs (show node.height)-      <> ": "-      <> blockHashToHex (headerHash node.header)-  trySyncing-processBlockStoreMessage (BlockPeerConnect p) = do-  logDebug $-    "New peer connected: " <> p.label-  trySyncingPeer p-processBlockStoreMessage (BlockPeerDisconnect p) = do-  logDebug $-    "Peer disconnected: " <> p.label-  finishPeer p-processBlockStoreMessage (BlockReceived p b) = do-  logDebug $-    "Received block: "-      <> blockHashToHex (headerHash b.header)-  processBlock p b-processBlockStoreMessage (BlockNotFound p bs) = do-  logDebug $-    "Blocks not found by peer "-      <> p.label-      <> ": "-      <> T.unwords (map blockHashToHex bs)-  processNoBlocks p bs-processBlockStoreMessage (TxRefReceived p tx) = do-  logDebug $-    "Transaction received from peer "-      <> p.label-      <> ": "-      <> txHashToHex (txHash tx)-  processTx p tx-processBlockStoreMessage (TxRefAvailable p ts) = do-  logDebug $-    "Transactions available from peer "-      <> p.label-      <> ": "-      <> T.unwords (map txHashToHex ts)-  processTxs p ts-processBlockStoreMessage (BlockPing r) = do-  logDebug "Internal clock event"-  setStoreHeight-  setHeadersHeight-  setPendingTxs-  setPeersConnected-  trySyncing-  processMempool-  pruneOrphans-  pruneMempool-  checkTime-  atomically (r ())---- | Send a clock tick to the block store every 0.1 to 1 second, waiting for each to be handled.-pingMe :: (MonadIO m) => Mailbox BlockStoreMessage -> m ()-pingMe mbox =-  loopForever $ do-    BlockPing `query` mbox-    delay <- liftIO $ randomRIO (100 * 1000, 1000 * 1000)-    threadDelay delay---- | Tell the block store that a peer connected.-blockStorePeerConnect :: (MonadIO m) => Peer -> BlockStore -> m ()-blockStorePeerConnect peer store =-  BlockPeerConnect peer `send` store.mailbox---- | Tell the block store that a peer disconnected.-blockStorePeerDisconnect ::-  (MonadIO m) => Peer -> BlockStore -> m ()-blockStorePeerDisconnect peer store =-  BlockPeerDisconnect peer `send` store.mailbox---- | Tell the block store that the node has a new best header.-blockStoreHead ::-  (MonadIO m) => BlockNode -> BlockStore -> m ()-blockStoreHead node store =-  BlockNewBest node `send` store.mailbox---- | Give the block store a block received from a peer.-blockStoreBlock ::-  (MonadIO m) => Peer -> Block -> BlockStore -> m ()-blockStoreBlock peer block store =-  BlockReceived peer block `send` store.mailbox---- | Tell the block store that a peer could not find blocks.-blockStoreNotFound ::-  (MonadIO m) => Peer -> [BlockHash] -> BlockStore -> m ()-blockStoreNotFound peer blocks store =-  BlockNotFound peer blocks `send` store.mailbox---- | Give the block store a transaction received from a peer.-blockStoreTx ::-  (MonadIO m) => Peer -> Tx -> BlockStore -> m ()-blockStoreTx peer tx store =-  TxRefReceived peer tx `send` store.mailbox---- | Tell the block store that a peer announced transactions.-blockStoreTxHash ::-  (MonadIO m) => Peer -> [TxHash] -> BlockStore -> m ()-blockStoreTxHash peer txhashes store =-  TxRefAvailable peer txhashes `send` store.mailbox---- | 'STM' version of 'blockStorePeerConnect'.-blockStorePeerConnectSTM ::-  Peer -> BlockStore -> STM ()-blockStorePeerConnectSTM peer store =-  BlockPeerConnect peer `sendSTM` store.mailbox---- | 'STM' version of 'blockStorePeerDisconnect'.-blockStorePeerDisconnectSTM ::-  Peer -> BlockStore -> STM ()-blockStorePeerDisconnectSTM peer store =-  BlockPeerDisconnect peer `sendSTM` store.mailbox---- | 'STM' version of 'blockStoreHead'.-blockStoreHeadSTM ::-  BlockNode -> BlockStore -> STM ()-blockStoreHeadSTM node store =-  BlockNewBest node `sendSTM` store.mailbox---- | 'STM' version of 'blockStoreBlock'.-blockStoreBlockSTM ::-  Peer -> Block -> BlockStore -> STM ()-blockStoreBlockSTM peer block store =-  BlockReceived peer block `sendSTM` store.mailbox---- | 'STM' version of 'blockStoreNotFound'.-blockStoreNotFoundSTM ::-  Peer -> [BlockHash] -> BlockStore -> STM ()-blockStoreNotFoundSTM peer blocks store =-  BlockNotFound peer blocks `sendSTM` store.mailbox---- | 'STM' version of 'blockStoreTx'.-blockStoreTxSTM ::-  Peer -> Tx -> BlockStore -> STM ()-blockStoreTxSTM peer tx store =-  TxRefReceived peer tx `sendSTM` store.mailbox---- | 'STM' version of 'blockStoreTxHash'.-blockStoreTxHashSTM ::-  Peer -> [TxHash] -> BlockStore -> STM ()-blockStoreTxHashSTM peer txhashes store =-  TxRefAvailable peer txhashes `sendSTM` store.mailbox---- | Number of transactions queued or requested by the block store.-blockStorePendingTxs ::-  (MonadIO m) => BlockStore -> m Int-blockStorePendingTxs =-  atomically . blockStorePendingTxsSTM---- | 'STM' version of 'blockStorePendingTxs'.-blockStorePendingTxsSTM ::-  BlockStore -> STM Int-blockStorePendingTxsSTM BlockStore {..} = do-  x <- HashMap.keysSet <$> readTVar txs-  y <- readTVar requested-  return $ HashSet.size $ x `HashSet.union` y+{-# LANGUAGE RecordWildCards #-}+{-# OPTIONS_GHC -Wno-ambiguous-fields #-}++-- |+-- Module      : Haskoin.Store.BlockStore+-- Copyright   : (c) Haskoin Developers+-- License     : MIT+-- Maintainer  : jprupp@protonmail.ch+-- Stability   : experimental+-- Portability : POSIX+--+-- The block store process downloads blocks and transactions from peers and+-- imports them into the database.+--+-- = Blocks+--+-- When the best header of the node is ahead of the best block in the database,+-- the block store picks one online peer as its syncing peer and asks it for+-- blocks in batches of 'blockWindow'. Each block is imported as soon as it+-- arrives. If the best block of the database is no longer in the main chain of+-- headers, blocks are reverted until it is. A syncing peer that sends a block+-- that was not requested, cannot find a block, or stays silent for longer than+-- 'peerTimeout' is disconnected. The state of the download, and every rule+-- about it, is the pure state machine of "Haskoin.Store.BlockStore.Sync";+-- this module carries out what its transitions decide.+--+-- = Mempool+--+-- Once the database is in sync, the block store asks each peer for its mempool+-- (if 'syncMempool' is set) and requests every transaction that peers+-- announce. Received transactions are queued and imported in batches on every+-- clock tick, in dependency order. Transactions that spend unknown outputs are+-- kept for up to an hour, waiting for their parents. Mempool transactions older+-- than 'mempoolTimeout' days are deleted. The transactions on their way in,+-- requested or received, are the pure value of+-- "Haskoin.Store.BlockStore.Pending".+--+-- = Feeding the block store+--+-- The block store does not listen to the node by itself.+-- "Haskoin.Store.Manager" forwards node events to it with the+-- @blockStore*@ functions of this module.+-- = Why it works this way+--+-- * __Blocks come from one peer at a time.__ Blocks must be imported in order,+--   so downloading from several peers at once would only buffer blocks that cannot+--   be imported yet. One syncing peer, kept busy with a sliding window of+--   requests, uses the network well without unbounded buffering.+--+-- * __The window refills early.__ The next batch is requested when fewer than a+--   quarter of the window are still outstanding, so the peer never sits idle+--   while the store imports, and the backlog never grows much beyond one window.+--+-- * __Mempool sync waits for the tip.__ Unconfirmed transactions only make sense+--   against the current chain, so they are requested once the database is in+--   sync.+--+-- * __Orphans get a grace period.__ A transaction can arrive before its parent.+--   It is held for up to an hour, waiting for the parent, instead of being+--   dropped and requested again.+--+-- * __Imports are batched.__ Mempool transactions are imported up to 2000 at a+--   time on each clock tick, which amortises the cost of a database write over+--   many transactions during bursts.+module Haskoin.Store.BlockStore+  ( -- * Block Store+    BlockStore,+    BlockStoreConfig (..),+    withBlockStore,+    blockStorePeerConnect,+    blockStorePeerConnectSTM,+    blockStorePeerDisconnect,+    blockStorePeerDisconnectSTM,+    blockStoreHead,+    blockStoreHeadSTM,+    blockStoreBlock,+    blockStoreBlockSTM,+    blockStoreNotFound,+    blockStoreNotFoundSTM,+    blockStoreTx,+    blockStoreTxSTM,+    blockStoreTxHash,+    blockStoreTxHashSTM,+    blockStorePendingTxs,+    blockStorePendingTxsSTM,+    needsMoreBlocks,+  )+where++import Control.Monad+  ( forM,+    forM_,+    unless,+    void,+    when,+    (>=>),+  )+import Data.Bool (bool)+import Data.ByteString qualified as B+import Data.HashMap.Strict qualified as HashMap+import Data.HashSet (HashSet)+import Data.HashSet qualified as HashSet+import Data.Maybe+  ( catMaybes,+    isJust,+  )+import Data.Serialize (encode)+import Data.String (fromString)+import Data.String.Conversions (cs)+import Data.Text (Text)+import Data.Text qualified as T+import Data.Time.Clock+  ( NominalDiffTime,+    UTCTime,+    getCurrentTime,+  )+import Data.Time.Clock.POSIX+  ( getPOSIXTime,+    posixSecondsToUTCTime,+    utcTimeToPOSIXSeconds,+  )+import Data.Time.Format+  ( defaultTimeLocale,+    formatTime,+  )+import Haskoin+  ( Block (..),+    BlockHash (..),+    BlockHeader (..),+    BlockHeight,+    BlockNode (..),+    Ctx,+    GetData (..),+    InvType (..),+    InvVector (..),+    Message (..),+    Network (..),+    OutPoint (..),+    Tx (..),+    TxHash (..),+    TxIn (..),+    blockHashToHex,+    headerHash,+    txHash,+    txHashToHex,+  )+import Haskoin.Node+  ( Chain,+    LogLevel (..),+    Logger,+    OnlinePeer (..),+    Owner,+    Peer (..),+    PeerMgr,+    chainBlockMain,+    chainGetAncestor,+    chainGetBest,+    chainGetBlock,+    chainGetParents,+    getPeers,+    killPeer,+    logAt,+    newOwner,+    releasePeer,+    reservePeer,+    sendMessage,+  )+import Haskoin.Store.BlockStore.Pending (Pending, PendingTx (..))+import Haskoin.Store.BlockStore.Pending qualified as Pending+import Haskoin.Store.BlockStore.Sync+  ( Accept (..),+    Sync (..),+    needsMoreBlocks,+  )+import Haskoin.Store.BlockStore.Sync qualified as Sync+import Haskoin.Store.Common (StoreEvent (..), loopForever)+import Haskoin.Store.Data+import Haskoin.Store.Database.Mempool (wipeMempoolArea)+import Haskoin.Store.Database.Query (getMempoolBefore)+import Haskoin.Store.Database.Query qualified as Q+import Haskoin.Store.Database.Query.Layered qualified as L+import Haskoin.Store.Database.Reader+import Haskoin.Store.Database.Writer (MempoolChanges (..), Writer, runWriterMempool)+import Haskoin.Store.Database.Writer qualified as W+import Haskoin.Store.Logic (ImportException (..))+import Haskoin.Store.Logic.Layered+  ( deleteUnconfirmedTx,+    importBlock,+    initBest,+    newMempoolTx,+    revertBlock,+  )+import NQE+  ( InChan,+    Listen,+    Mailbox,+    Publisher,+    inboxToMailbox,+    newInbox,+    publish,+    query,+    receive,+    send,+    sendSTM,+  )+import System.Metrics.StatsD+import System.Random (randomRIO)+import UnliftIO+  ( Exception,+    STM,+    TVar,+    atomically,+    catch,+    link,+    modifyTVar,+    modifyTVar',+    newTVarIO,+    readTVar,+    readTVarIO,+    stateTVar,+    throwIO,+    try,+    withAsync,+  )+import UnliftIO.Concurrent (threadDelay)++-- | Messages handled by the block store process.+data BlockStoreMessage+  = -- | the node has a new best header+    BlockNewBest !BlockNode+  | -- | a peer connected+    BlockPeerConnect !Peer+  | -- | a peer disconnected+    BlockPeerDisconnect !Peer+  | -- | a peer sent a block+    BlockReceived !Peer !Block+  | -- | a peer could not find blocks+    BlockNotFound !Peer ![BlockHash]+  | -- | a peer sent a transaction+    TxRefReceived !Peer !Tx+  | -- | a peer announced transactions+    TxRefAvailable !Peer ![TxHash]+  | -- | clock tick for housekeeping; reply when done+    BlockPing !(Listen ())++-- | Fatal errors of the block store.+data BlockException+  = -- | the header of a block in the database is not known to the node+    BlockNotInChain !BlockHash+  | -- | the database has no best block+    Uninitialized+  | -- | the database is inconsistent+    CorruptDatabase+  | -- | the node has no header at this height below this block+    AncestorNotInChain !BlockHeight !BlockHash+  | -- | importing mempool transactions failed+    MempoolImportFailed+  deriving (Show, Eq, Ord, Exception)++-- | Block store process state.+data BlockStore = BlockStore+  { -- | mailbox of the process+    mailbox :: !(Mailbox BlockStoreMessage),+    -- | configuration+    config :: !BlockStoreConfig,+    -- | state of the block download ("Haskoin.Store.BlockStore.Sync")+    sync :: !(TVar (Sync Peer)),+    -- | transactions received and waiting to be imported+    -- ("Haskoin.Store.BlockStore.Pending")+    pending :: !(TVar Pending),+    -- | StatsD metrics, if enabled+    metrics :: !(Maybe StoreMetrics),+    -- | labels of the peers whose mempool was already requested+    syncedMempool :: !(TVar (HashSet Text)),+    -- | who reserves the syncing peer, so that no one else releases it+    owner :: !Owner,+    -- | transactions that left the mempool unconfirmed since the store+    -- started, as many as 'mostForgotten'. The database keeps nothing of+    -- them, and peers go on announcing them: without this the store would+    -- fetch each one again, only to refuse it again.+    forgotten :: !(TVar (HashSet TxHash))+  }++-- | How many deleted transactions the store remembers not to fetch again.+-- Past that many it forgets them all and starts over, which costs a few+-- fetches that come to nothing.+mostForgotten :: Int+mostForgotten = 100000++-- | StatsD metrics of the block store.+data StoreMetrics = StoreMetrics+  { -- | height of the best block (@store.blocks@)+    blocks :: !StatGauge,+    -- | height of the best header (@store.headers@)+    headers :: !StatGauge,+    -- | transactions waiting to be imported (@store.queued_txs@)+    queuedTxs :: !StatGauge,+    -- | online peers (@store.peers@)+    peers :: !StatGauge,+    -- | transactions in the mempool (@store.mempool@)+    mempool :: !StatGauge,+    -- | milliseconds to import a block (@store.block@)+    block :: !StatTiming+  }++-- | Register the block store metrics, if StatsD is configured, with their current values.+newStoreMetrics :: BlockStoreConfig -> IO (Maybe StoreMetrics)+newStoreMetrics cfg =+  forM cfg.stats $ \s -> do+    m <- Q.getMempool cfg.db+    b <-+      Q.getBestBlock cfg.db >>= \case+        Nothing -> return 0+        Just h -> maybe 0 (.height) <$> Q.getBlock cfg.db h+    h <- chainGetBest cfg.chain+    p <- getPeers cfg.peerMgr+    blocks <- g s "blocks" (fromIntegral b)+    headers <- g s "headers" (fromIntegral h.height)+    queuedTxs <- g s "queued_txs" 0+    peers <- g s "peers" (length p)+    mempool <- g s "mempool" (length m)+    block <- t s "block" 0+    return StoreMetrics {..}+  where+    g s x = newStatGauge s ("store." <> x)+    t s x = newStatTiming s ("store." <> x)++-- | Update the gauge of the best block height.+setStoreHeight :: BlockStore -> IO ()+setStoreHeight self =+  forM_ self.metrics $ \m ->+    getBestBlock self >>= mapM_ (getBlock self >=> mapM_ (setGauge m.blocks . fromIntegral . (.height)))++-- | Update the gauge of the best header height.+setHeadersHeight :: BlockStore -> IO ()+setHeadersHeight self =+  forM_ self.metrics $ \m -> do+    n <- chainGetBest self.config.chain+    setGauge m.headers (fromIntegral n.height)++-- | Update the gauge of transactions waiting to be imported.+setPendingTxs :: BlockStore -> IO ()+setPendingTxs self =+  forM_ self.metrics $ \m -> do+    p <- readTVarIO self.pending+    setGauge m.queuedTxs (Pending.queued p)++-- | Update the gauge of online peers.+setPeersConnected :: BlockStore -> IO ()+setPeersConnected self =+  forM_ self.metrics $ \m -> do+    p <- getPeers self.config.peerMgr+    setGauge m.peers (length p)++-- | Record how long a successful block import took.+withBlockTiming :: BlockStore -> IO (Either l r) -> IO (Either l r)+withBlockTiming self go = do+  case self.metrics of+    Just m -> do+      start <- round . (* 1000) <$> getPOSIXTime+      go >>= \case+        Right x -> do+          end <- round . (* 1000) <$> getPOSIXTime+          addTiming m.block (end - start)+          return (Right x)+        Left y -> return (Left y)+    Nothing -> go++-- | Configuration for a block store.+data BlockStoreConfig = BlockStoreConfig+  { -- | secp256k1 context+    ctx :: !Ctx,+    -- | peer manager from running node+    peerMgr :: !PeerMgr,+    -- | chain from a running node+    chain :: !Chain,+    -- | listener for store events+    pub :: !(Publisher StoreEvent),+    -- | RocksDB database handle+    db :: !DatabaseReader,+    -- | network constants+    net :: !Network,+    -- | do not index new mempool transactions+    noMempool :: !Bool,+    -- | wipe mempool at start+    wipeMempool :: !Bool,+    -- | what to do with the best block right after it changes, and when the+    -- store starts, before anything else is written to the database: a+    -- member of a fleet takes a view of the database for the block here+    -- ('Haskoin.Store.Fleet.Member.noteBest')+    onBest :: !(BlockHash -> IO ()),+    -- | sync mempool from peers+    syncMempool :: !Bool,+    -- | blocks to request from the syncing peer in one batch+    blockWindow :: !Int,+    -- | delete transactions older than this number of days from mempool+    mempoolTimeout :: !Int,+    -- | disconnect syncing peer if inactive for this long+    peerTimeout :: !NominalDiffTime,+    -- | StatsD client, if metrics are enabled+    stats :: !(Maybe Stats),+    -- | log messages here+    logger :: !Logger+  }++-- | Log a message with the logger of the block store.+logDebug, logInfo, logWarn, logError :: BlockStore -> Text -> IO ()+logDebug self = logAt LevelDebug self.config.logger+logInfo self = logAt LevelInfo self.config.logger+logWarn self = logAt LevelWarn self.config.logger+logError self = logAt LevelError self.config.logger++-- | Run an import against the database, returning the import error if it fails.+runImport :: BlockStore -> Network -> Ctx -> (Writer -> IO a) -> IO (Either ImportException a)+runImport self net ctx f = fmap fst <$> runImportMempool self net ctx f++-- | Like 'runImport', but also return how the import changed the mempool.+-- The mempool gauge is kept up to date from those changes instead of+-- counting the whole mempool on every clock tick.+runImportMempool :: BlockStore -> Network -> Ctx -> (Writer -> IO a) -> IO (Either ImportException (a, MempoolChanges))+runImportMempool self net ctx f = do+  r <- try $ runWriterMempool net ctx self.config.db self.config.logger f+  forM_ r $ \(_, c) ->+    forM_+      self.metrics+      ( \m -> do+          incrementGauge m.mempool (length c.added)+          decrementGauge m.mempool (length c.removed)+      )+  return r++-- | Run an import and publish the transactions that it added to the+-- mempool or removed from it, then the new best block if there is one.+-- Transactions of the block are not published.+runImportNotify :: BlockStore -> Maybe Block -> (Writer -> IO a) -> IO (Either ImportException a)+runImportNotify self block f = do+  net <- getNetwork self+  ctx <- getCtx self+  runImportMempool self net ctx f >>= \case+    Left e -> return (Left e)+    Right (x, c) -> do+      let l = self.config.pub+      let deleted = filter notInBlock c.removed+      atomically . modifyTVar' self.forgotten $ \seen ->+        let more = foldr HashSet.insert seen deleted+         in if HashSet.size more > mostForgotten then HashSet.fromList deleted else more+      -- one that came back is not forgotten any more+      atomically (modifyTVar' self.forgotten (\seen -> foldr HashSet.delete seen c.added))+      forM_ deleted $ \h ->+        publish (StoreMempoolDelete h) l+      forM_ (filter notInBlock c.added) $ \h ->+        publish (StoreMempoolNew h) l+      forM_ block $ \b -> do+        -- before the event, so that whoever hears of the block finds what+        -- was done for it+        self.config.onBest (headerHash b.header)+        publish (StoreBestBlock (headerHash b.header)) l+      return (Right x)+  where+    inBlock = maybe HashSet.empty (HashSet.fromList . map txHash . (.txs)) block+    notInBlock = not . (`HashSet.member` inBlock)++-- | Run a database query from the block store.+-- | Best block in the database.+getBestBlock :: BlockStore -> IO (Maybe BlockHash)+getBestBlock self = Q.getBestBlock self.config.db++-- | Data of a block in the database.+getBlock :: BlockStore -> BlockHash -> IO (Maybe BlockData)+getBlock self = Q.getBlock self.config.db++-- | Data of a transaction in the database: confirmed, in the mempool, or+-- deleted from it and marked so.+getTxData :: BlockStore -> TxHash -> IO (Maybe TxData)+getTxData self h = L.withView self.config.db (`L.getTxData` h)++-- | Unspent output in the database, of the chain or of the mempool.+getUnspent :: BlockStore -> OutPoint -> IO (Maybe Unspent)+getUnspent self p = L.withView self.config.db (`L.getUnspent` p)++-- | Network of the block store.+getNetwork :: BlockStore -> IO Network+getNetwork self = return self.config.net++-- | Secp256k1 context of the block store.+getCtx :: BlockStore -> IO Ctx+getCtx self = return self.config.ctx++-- | Run block store process.+withBlockStore ::+  BlockStoreConfig ->+  (BlockStore -> IO a) ->+  IO a+withBlockStore cfg action = do+  pb <- newTVarIO Idle+  ts <- newTVarIO Pending.empty+  sy <- newTVarIO HashSet.empty+  inbox <- newInbox+  metrics <- newStoreMetrics cfg+  owner <- (newOwner "block store")+  forgotten <- newTVarIO HashSet.empty+  let r =+        BlockStore+          { mailbox = inboxToMailbox inbox,+            config = cfg,+            sync = pb,+            pending = ts,+            metrics = metrics,+            syncedMempool = sy,+            owner = owner,+            forgotten = forgotten+          }+  withAsync ((go r inbox)) $ \a -> do+    link a+    action r+  where+    go self inbox = do+      ini+      wipe+      run+      where+        -- the mempool is an area of its own, so wiping it is deleting the+        -- keys of that area: nothing of the chain has to be put right+        wipe = when cfg.wipeMempool $ do+          keys <- wipeMempoolArea cfg.db+          forM_ self.metrics $ \m -> setGauge m.mempool 0+          logAt LevelInfo cfg.logger $+            "Wiped the mempool: " <> cs (show keys) <> " keys deleted"+        ini = do+          net <- getNetwork self+          ctx <- getCtx self+          runImport self net ctx initBest >>= \case+            Left e -> do+              logError self $+                "Could not initialize: " <> cs (show e)+              throwIO e+            Right () -> getBestBlock self >>= mapM_ cfg.onBest+        run =+          withAsync (pingMe (inboxToMailbox inbox)) $ \a ->+            link a >> runBlockStoreLoop self inbox++-- | Handle messages forever.+runBlockStoreLoop :: (InChan mbox) => BlockStore -> mbox BlockStoreMessage -> IO b+runBlockStoreLoop self inbox =+  loopForever $ do+    logDebug self "Waiting for new event..."+    msg <- receive inbox+    processBlockStoreMessage self msg++-- | Whether the best block in the database is the best header of the node. If+-- so, release the syncing peer.+isInSync :: BlockStore -> IO Bool+isInSync self =+  getBestBlock self >>= \case+    Nothing -> do+      logError self "Block database uninitialized"+      throwIO Uninitialized+    Just bb -> do+      cb <- chainGetBest self.config.chain+      if headerHash cb.header == bb+        then clearSyncingState self >> return True+        else return False++-- | Run an action only if the database is in sync.+guardSync :: BlockStore -> IO () -> IO ()+guardSync self f = isInSync self >>= \s -> when s f++-- | Run an action only if the mempool is indexed.+guardMempool :: BlockStore -> IO () -> IO ()+guardMempool self f = unless self.config.noMempool f++-- | Ask a peer for its mempool, once per peer, if in sync and mempool sync is enabled.+syncMempool :: BlockStore -> Peer -> IO ()+syncMempool self p =+  guardMempool self . guardSync self $ do+    let s = self.config.syncMempool+    let m = self.syncedMempool+    y <- not . HashSet.member p.label <$> readTVarIO m+    when (s && y) $ do+      logDebug self $+        "Requesting mempool from peer: " <> p.label+      atomically . modifyTVar m $ HashSet.insert p.label+      MMempool `sendMessage` p++-- | Import a block from the syncing peer, then ask for more blocks or, if now in+-- sync, sync the mempool. Disconnects the peer if it is not the syncing peer, if+-- the block is unknown to the node, or if the block cannot be imported.+processBlock :: BlockStore -> Peer -> Block -> IO ()+processBlock self peer block =+  Sync.accept peer blockhash <$> readTVarIO self.sync >>= \case+    NotSyncingPeer -> refuse "Non-syncing peer"+    NotRequested -> refuse "Syncing peer"+    Requested ->+      getBlockNode self blockhash >>= \case+        Nothing -> do+          logError self $+            "Peer "+              <> peer.label+              <> " sent unknown block "+              <> blockHashToHex blockhash+          killPeer peer+        Just node -> do+          logDebug self $+            "Processing block "+              <> blockText node Nothing+              <> " from peer "+              <> peer.label+          ret <- withBlockTiming self $ runImportNotify self (Just block) (\wr -> importBlock wr block node)+          case ret of+            Left e -> failure e+            Right _ -> success node+  where+    header = block.header+    blockhash = headerHash header+    hexhash = blockHashToHex blockhash+    refuse who = do+      logError self $+        who+          <> " "+          <> peer.label+          <> " sent a block that was not requested: "+          <> hexhash+      killPeer peer+    success node = do+      logInfo self $+        "Best block: " <> blockText node (Just block)+      now <- getCurrentTime+      transition self $ \st -> ((), Sync.imported now blockhash st)+      isInSync self >>= \case+        False -> syncMe self+        True -> do+          updateOrphans self+          syncMempool self peer+    failure e = do+      logError self $+        "Error importing block "+          <> hexhash+          <> " from peer: "+          <> peer.label+          <> ": "+          <> cs (show e)+      killPeer peer++-- | Change the state of the block download with a transition of+-- "Haskoin.Store.BlockStore.Sync", atomically, and give its result.+transition :: BlockStore -> (Sync Peer -> (a, Sync Peer)) -> IO a+transition self = atomically . stateTVar self.sync++-- | Header of a block from the node.+getBlockNode :: BlockStore -> BlockHash -> IO (Maybe BlockNode)+getBlockNode self blockhash = do+  let ch = self.config.chain+  chainGetBlock ch blockhash++-- | Disconnect a peer that could not find requested blocks.+processNoBlocks :: BlockStore -> Peer -> [BlockHash] -> IO ()+processNoBlocks self p hs = do+  forM_ (zip [(1 :: Int) ..] hs) $ \(i, h) ->+    logError self $+      "Block "+        <> cs (show i)+        <> "/"+        <> cs (show (length hs))+        <> " "+        <> blockHashToHex h+        <> " not found by peer: "+        <> p.label+  killPeer p++-- | Queue a transaction received from a peer for import.+processTx :: BlockStore -> Peer -> Tx -> IO ()+processTx self p tx = guardMempool self $ do+  t <- getCurrentTime+  logDebug self $+    "Received tx "+      <> txHashToHex (txHash tx)+      <> " by peer: "+      <> p.label+  changePending self (Pending.received t tx HashSet.empty)++-- | Forget old requests, and drop transactions that waited for their parents+-- for too long.+pruneOrphans :: BlockStore -> IO ()+pruneOrphans self = guardMempool self $ do+  now <- getCurrentTime+  changePending self (Pending.prune now)++-- | Change the mempool transactions on their way into the database.+changePending :: BlockStore -> (Pending -> Pending) -> IO ()+changePending self f = do+  atomically $ modifyTVar' self.pending f+  setPendingTxs self++-- | Whether a transaction is queued or was requested recently.+isPending :: BlockStore -> TxHash -> IO Bool+isPending self th = Pending.isPending th <$> readTVarIO self.pending++-- | Take up to this many queued transactions that have no missing parents, in+-- dependency order, out of the queue.+pendingTxs :: BlockStore -> Int -> IO [PendingTx]+pendingTxs self i = do+  selected <- atomically $ stateTVar self.pending (Pending.takeReady i)+  setPendingTxs self+  return selected++-- | Drop queued transactions that are already in the database, and recompute the+-- missing parents of the others.+updateOrphans :: BlockStore -> IO ()+updateOrphans self = do+  orphans <- Pending.waiting <$> readTVarIO self.pending+  updates <- forM orphans $ \p -> do+    let h = txHash p.tx+    exists h >>= \case+      True -> return (h, Nothing)+      False -> do+        found <- catMaybes <$> mapM (getUnspent self . (.outpoint)) p.tx.inputs+        let have = HashSet.fromList (map (.outpoint.hash) found)+        return (h, Just (HashSet.difference p.deps have))+  changePending self (Pending.refresh (HashMap.fromList updates))+  where+    exists th =+      getTxData self th >>= \case+        Nothing -> return False+        Just TxData {deleted = True} -> return False+        Just TxData {deleted = False} -> return True++-- | Whether the store has a transaction, or deleted it not long ago: in+-- both cases there is nothing to fetch.+known :: BlockStore -> TxHash -> IO (Maybe ())+known self h =+  HashSet.member h <$> readTVarIO self.forgotten >>= \case+    True -> return (Just ())+    False -> (() <$) <$> getTxData self h++-- | Queue a transaction that spends unknown outputs, recording which parents are missing.+newOrphanTx ::+  BlockStore ->+  UTCTime ->+  Tx ->+  Writer ->+  IO ()+newOrphanTx self time tx wr = do+  logAt LevelDebug self.config.logger $+    "Import tx "+      <> txHashToHex (txHash tx)+      <> ": Orphan"+  unspents <- catMaybes <$> mapM (W.getUnspent wr) prevs+  let have = HashSet.fromList (map (.outpoint) unspents)+      missing = HashSet.map (.hash) (HashSet.difference (HashSet.fromList prevs) have)+  atomically . modifyTVar' self.pending $ Pending.received time tx missing+  where+    prevs = map (.outpoint) tx.inputs++-- | Import a transaction into the mempool. Returns 'True' if imported. A+-- transaction that spends unknown outputs is queued as an orphan; other errors+-- are ignored.+importMempoolTx :: BlockStore -> UTCTime -> Tx -> Writer -> IO Bool+importMempoolTx block_read time tx wr =+  catch new_mempool_tx handle_error+  where+    tx_hash = txHash tx+    handle_error :: ImportException -> IO Bool+    handle_error Orphan = do+      newOrphanTx block_read time tx wr+      return False+    handle_error _ = return False+    seconds = floor (utcTimeToPOSIXSeconds time)+    new_mempool_tx = do+      t <- newMempoolTx wr tx seconds+      logAt LevelInfo block_read.config.logger $+        bool "Already have" "Imported" t+          <> " tx "+          <> txHashToHex (txHash tx)+      when t . atomically . modifyTVar' block_read.pending $ Pending.imported tx_hash+      return t++-- | Import up to 2000 queued transactions.+processMempool :: BlockStore -> IO ()+processMempool self = guardMempool self $ do+  txs <- pendingTxs self 2000+  let block_read = self+  unless (null txs) $ import_txs block_read txs+  where+    run_import block_read wr p =+      importMempoolTx block_read p.time p.tx wr+    import_txs block_read txs =+      runImportNotify self Nothing (\wr -> mapM (run_import block_read wr) txs) >>= \case+        Left e -> report_error e+        Right _ -> return ()+    report_error e = do+      logError self $+        "Error processing mempool: " <> cs (show e)+      throwIO e++-- | Delete up to 1000 mempool transactions older than the configured number of days.+pruneMempool :: BlockStore -> IO ()+pruneMempool self =+  guardMempool self $ do+    let days = self.config.mempoolTimeout+    when (days > 0) $ do+      time <- (floor . utcTimeToPOSIXSeconds) <$> getCurrentTime+      let thresh = time - (fromIntegral days * 24 * 60 * 60)+      -- runs on every clock tick: read only the expired entries, oldest+      -- first, from the index by time+      let db = self.config.db+      txs <- map snd <$> (getMempoolBefore db thresh 1000)+      unless (null txs) $ do+        logInfo self $+          "Deleting " <> cs (show (length txs)) <> " old transactions from mempool"+        runImportNotify self Nothing (\wr -> mapM_ (deleteUnconfirmedTx wr False) txs) >>= \case+          Left e ->+            logError self $+              "Error pruning mempool: " <> cs (show e)+          Right () -> return ()++-- | Request from a peer the announced transactions that are neither queued,+-- recently requested nor in the database. Only while in sync.+processTxs :: BlockStore -> Peer -> [TxHash] -> IO ()+processTxs self p hs = guardMempool self . guardSync self $ do+  logDebug self $+    "Received inventory with "+      <> cs (show (length hs))+      <> " transactions from peer: "+      <> p.label+  xs <- catMaybes <$> zip_counter process_tx+  unless (null xs) $ go xs+  where+    len = length hs+    zip_counter = forM (zip [(1 :: Int) ..] hs) . uncurry+    process_tx i h =+      isPending self h >>= \case+        True -> do+          logDebug self $+            "Tx "+              <> cs (show i)+              <> "/"+              <> cs (show len)+              <> " "+              <> txHashToHex h+              <> ": "+              <> "Already Pending"+          return Nothing+        False ->+          known self h >>= \case+            Just _ -> do+              logDebug self $+                "Tx "+                  <> cs (show i)+                  <> "/"+                  <> cs (show len)+                  <> " "+                  <> txHashToHex h+                  <> ": "+                  <> "Already Imported"+              return Nothing+            Nothing -> do+              logDebug self $+                "Tx "+                  <> cs (show i)+                  <> "/"+                  <> cs (show len)+                  <> " "+                  <> txHashToHex h+                  <> ": "+                  <> "Requesting"+              return (Just h)+    go xs = do+      now <- getCurrentTime+      changePending self (Pending.requested now xs)+      let net = self.config.net+      let inv = if net.segWit then InvWitnessTx else InvTx+          vec = map (InvVector inv . (.get)) xs+          msg = MGetData (GetData vec)+      msg `sendMessage` p++-- | Disconnect the syncing peer if it has been silent for too long.+checkTime :: BlockStore -> IO ()+checkTime self = do+  now <- getCurrentTime+  Sync.timedOut self.config.peerTimeout now <$> readTVarIO self.sync >>= \case+    Nothing -> return ()+    Just p -> do+      logError self $ "Syncing peer timeout: " <> p.label+      killPeer p++-- | Revert blocks until the best block in the database is in the main chain of headers.+revertToMainChain :: BlockStore -> IO ()+revertToMainChain self = do+  h <- headerHash . (.header) <$> getBest self+  let ch = self.config.chain+  net <- getNetwork self+  ctx <- getCtx self+  (chainBlockMain ch h) >>= \x -> unless x $ do+    logWarn self $+      "Reverting best block: "+        <> blockHashToHex h+    runImport self net ctx (`revertBlock` h) >>= \case+      Left e -> do+        logError self $+          "Could not revert block "+            <> blockHashToHex h+            <> ": "+            <> cs (show e)+        throwIO e+      Right () -> do+        getBestBlock self >>= mapM_ self.config.onBest+        transition self $ \st -> ((), Sync.forgetRequests st)+    revertToMainChain self++-- | Header of the best block in the database.+getBest :: BlockStore -> IO BlockNode+getBest self = do+  bb <-+    getBestBlock self >>= \case+      Just b -> return b+      Nothing -> do+        logError self "No best block set"+        throwIO Uninitialized+  let ch = self.config.chain+  (chainGetBlock ch bb) >>= \case+    Just x -> return x+    Nothing -> do+      logError self $+        "Header not found for best block: "+          <> blockHashToHex bb+      throwIO (BlockNotInChain bb)++-- | Header of the last block requested from the syncing peer, or of the best block if none.+getSyncBest :: BlockStore -> IO BlockNode+getSyncBest self = do+  bb <-+    Sync.lastRequested <$> readTVarIO self.sync >>= \case+      Just h -> return h+      Nothing ->+        getBestBlock self >>= \case+          Just b -> return b+          Nothing -> do+            logError self "No best block set"+            throwIO Uninitialized+  chainGetBlock self.config.chain bb >>= \case+    Just x -> return x+    Nothing -> do+      logError self $+        "Header not found for block: "+          <> blockHashToHex bb+      throwIO (BlockNotInChain bb)++-- | The syncing peer, when it is time to ask it for the next batch of blocks.+shouldSync :: BlockStore -> IO (Maybe Peer)+shouldSync self =+  isInSync self >>= \case+    True -> return Nothing+    False -> Sync.wantsMore (blockWindow self) <$> readTVarIO self.sync++-- | Blocks to request from the syncing peer in one batch. Never below one,+-- or the store would never ask for any block.+blockWindow :: BlockStore -> Int+blockWindow self = max 1 self.config.blockWindow++-- | Ask the syncing peer for the next batch of blocks, if the window has room.+syncMe :: BlockStore -> IO ()+syncMe self = do+  revertToMainChain self+  shouldSync self >>= \case+    Nothing -> return ()+    Just p -> do+      bb <- getSyncBest self+      bh <- chainGetBest self.config.chain+      when (bb /= bh) $ do+        bns <- sel bb bh+        logDebug self $+          "Requesting "+            <> fromString (show (length bns))+            <> " blocks from peer: "+            <> p.label+        transition self $ \st -> ((), Sync.request (map (headerHash . (.header)) bns) st)+        MGetData (GetData (map inv bns)) `sendMessage` p+  where+    inv = InvVector kind . (.get) . headerHash . (.header)+    kind = if self.config.net.segWit then InvWitnessBlock else InvBlock+    -- The blocks right after the last requested one, up to one window or+    -- the chain tip, whichever comes first.+    sel bb bh = do+      top <- findAncestor self (min (bb.height + fromIntegral (blockWindow self)) bh.height) bh+      ps <- chainGetParents self.config.chain (bb.height + 1) top+      return $ ps <> [top]++-- | Ancestor of a header at a height. Throws 'AncestorNotInChain' if missing.+findAncestor :: BlockStore -> BlockHeight -> BlockNode -> IO BlockNode+findAncestor self height target = do+  let ch = self.config.chain+  (chainGetAncestor ch height target) >>= \case+    Just ancestor -> return ancestor+    Nothing -> do+      let h = headerHash target.header+      logError self $+        "Could not find header for ancestor of block "+          <> blockHashToHex h+          <> " at height "+          <> cs (show target.height)+      throwIO $ AncestorNotInChain height h++-- | Forget a peer: release it if it was the syncing peer, and forget that its mempool was requested.+finishPeer :: BlockStore -> Peer -> IO ()+finishPeer self p = do+  released <- transition self (Sync.release p)+  when released $ do+    logDebug self $ "Releasing peer: " <> p.label+    void (releasePeer self.owner p)+  atomically $ modifyTVar self.syncedMempool $ HashSet.delete p.label++-- | Make a peer the syncing peer if there is none and the peer is not busy.+trySetPeer :: BlockStore -> Peer -> IO Bool+trySetPeer self p =+  isJust . Sync.syncingPeer <$> readTVarIO self.sync >>= \case+    True -> return False+    False ->+      reservePeer self.owner p >>= \case+        False -> return False+        True -> do+          now <- getCurrentTime+          started <- transition self $ \st -> case Sync.start now p st of+            Just st' -> (True, st')+            Nothing -> (False, st)+          if started+            then logDebug self ("Locked peer: " <> p.label)+            else void (releasePeer self.owner p)+          return started++-- | If not in sync and there is no syncing peer, pick the first free peer and start syncing.+trySyncing :: BlockStore -> IO ()+trySyncing self =+  isInSync self >>= \case+    True -> return ()+    False ->+      isJust . Sync.syncingPeer <$> readTVarIO self.sync >>= \case+        True -> return ()+        False -> getPeers self.config.peerMgr >>= recurse . map (.mailbox)+  where+    recurse [] = return ()+    recurse (p : ps) =+      trySetPeer self p >>= \case+        False -> recurse ps+        True -> syncMe self++-- | Start syncing against a newly connected peer if needed, or request its mempool if in sync.+trySyncingPeer :: BlockStore -> Peer -> IO ()+trySyncingPeer self p =+  isInSync self >>= \case+    True -> syncMempool self p+    False ->+      trySetPeer self p >>= \case+        False -> return ()+        True -> syncMe self++-- | Release the syncing peer, if any.+clearSyncingState :: BlockStore -> IO ()+clearSyncingState self =+  Sync.syncingPeer <$> readTVarIO self.sync >>= mapM_ (finishPeer self)++-- | Handle a message sent to the block store.+processBlockStoreMessage :: BlockStore -> BlockStoreMessage -> IO ()+processBlockStoreMessage self (BlockNewBest node) = do+  logDebug self $+    "New best block mined at height "+      <> cs (show node.height)+      <> ": "+      <> blockHashToHex (headerHash node.header)+  trySyncing self+processBlockStoreMessage self (BlockPeerConnect p) = do+  logDebug self $+    "New peer connected: " <> p.label+  trySyncingPeer self p+processBlockStoreMessage self (BlockPeerDisconnect p) = do+  logDebug self $+    "Peer disconnected: " <> p.label+  finishPeer self p+processBlockStoreMessage self (BlockReceived p b) = do+  logDebug self $+    "Received block: "+      <> blockHashToHex (headerHash b.header)+  processBlock self p b+processBlockStoreMessage self (BlockNotFound p bs) = do+  logDebug self $+    "Blocks not found by peer "+      <> p.label+      <> ": "+      <> T.unwords (map blockHashToHex bs)+  processNoBlocks self p bs+processBlockStoreMessage self (TxRefReceived p tx) = do+  logDebug self $+    "Transaction received from peer "+      <> p.label+      <> ": "+      <> txHashToHex (txHash tx)+  processTx self p tx+processBlockStoreMessage self (TxRefAvailable p ts) = do+  logDebug self $+    "Transactions available from peer "+      <> p.label+      <> ": "+      <> T.unwords (map txHashToHex ts)+  processTxs self p ts+processBlockStoreMessage self (BlockPing r) = do+  logDebug self "Internal clock event"+  setStoreHeight self+  setHeadersHeight self+  setPendingTxs self+  setPeersConnected self+  trySyncing self+  processMempool self+  pruneOrphans self+  pruneMempool self+  checkTime self+  atomically (r ())++-- | Send a clock tick to the block store every 0.1 to 1 second, waiting for each to be handled.+pingMe :: Mailbox BlockStoreMessage -> IO ()+pingMe mbox =+  loopForever $ do+    BlockPing `query` mbox+    delay <- randomRIO (100 * 1000, 1000 * 1000)+    threadDelay delay++-- | Tell the block store that a peer connected.+blockStorePeerConnect :: Peer -> BlockStore -> IO ()+blockStorePeerConnect peer store =+  BlockPeerConnect peer `send` store.mailbox++-- | Tell the block store that a peer disconnected.+blockStorePeerDisconnect ::+  Peer -> BlockStore -> IO ()+blockStorePeerDisconnect peer store =+  BlockPeerDisconnect peer `send` store.mailbox++-- | Tell the block store that the node has a new best header.+blockStoreHead ::+  BlockNode -> BlockStore -> IO ()+blockStoreHead node store =+  BlockNewBest node `send` store.mailbox++-- | Give the block store a block received from a peer.+blockStoreBlock ::+  Peer -> Block -> BlockStore -> IO ()+blockStoreBlock peer block store =+  BlockReceived peer block `send` store.mailbox++-- | Tell the block store that a peer could not find blocks.+blockStoreNotFound ::+  Peer -> [BlockHash] -> BlockStore -> IO ()+blockStoreNotFound peer blocks store =+  BlockNotFound peer blocks `send` store.mailbox++-- | Give the block store a transaction received from a peer.+blockStoreTx ::+  Peer -> Tx -> BlockStore -> IO ()+blockStoreTx peer tx store =+  TxRefReceived peer tx `send` store.mailbox++-- | Tell the block store that a peer announced transactions.+blockStoreTxHash ::+  Peer -> [TxHash] -> BlockStore -> IO ()+blockStoreTxHash peer txhashes store =+  TxRefAvailable peer txhashes `send` store.mailbox++-- | 'STM' version of 'blockStorePeerConnect'.+blockStorePeerConnectSTM ::+  Peer -> BlockStore -> STM ()+blockStorePeerConnectSTM peer store =+  BlockPeerConnect peer `sendSTM` store.mailbox++-- | 'STM' version of 'blockStorePeerDisconnect'.+blockStorePeerDisconnectSTM ::+  Peer -> BlockStore -> STM ()+blockStorePeerDisconnectSTM peer store =+  BlockPeerDisconnect peer `sendSTM` store.mailbox++-- | 'STM' version of 'blockStoreHead'.+blockStoreHeadSTM ::+  BlockNode -> BlockStore -> STM ()+blockStoreHeadSTM node store =+  BlockNewBest node `sendSTM` store.mailbox++-- | 'STM' version of 'blockStoreBlock'.+blockStoreBlockSTM ::+  Peer -> Block -> BlockStore -> STM ()+blockStoreBlockSTM peer block store =+  BlockReceived peer block `sendSTM` store.mailbox++-- | 'STM' version of 'blockStoreNotFound'.+blockStoreNotFoundSTM ::+  Peer -> [BlockHash] -> BlockStore -> STM ()+blockStoreNotFoundSTM peer blocks store =+  BlockNotFound peer blocks `sendSTM` store.mailbox++-- | 'STM' version of 'blockStoreTx'.+blockStoreTxSTM ::+  Peer -> Tx -> BlockStore -> STM ()+blockStoreTxSTM peer tx store =+  TxRefReceived peer tx `sendSTM` store.mailbox++-- | 'STM' version of 'blockStoreTxHash'.+blockStoreTxHashSTM ::+  Peer -> [TxHash] -> BlockStore -> STM ()+blockStoreTxHashSTM peer txhashes store =+  TxRefAvailable peer txhashes `sendSTM` store.mailbox++-- | Number of transactions queued or requested by the block store.+blockStorePendingTxs ::+  BlockStore -> IO Int+blockStorePendingTxs =+  atomically . blockStorePendingTxsSTM++-- | 'STM' version of 'blockStorePendingTxs'.+blockStorePendingTxsSTM ::+  BlockStore -> STM Int+blockStorePendingTxsSTM store = Pending.size <$> readTVar store.pending  -- | Describe a block for logs: height, time, hash and, if the block is given, size. blockText :: BlockNode -> Maybe Block -> Text