haskoin-store 1.9.1 → 1.10.0
raw patch · 7 files changed
+189/−678 lines, 7 filesdep ~haskoin-store-dataPVP ok
version bump matches the API change (PVP)
Dependency ranges changed: haskoin-store-data
API changes (from Hackage documentation)
- Haskoin.Store.Web: acceptsBinary :: Text -> Bool
Files
- CHANGELOG.md +30/−0
- README.md +5/−37
- haskoin-store.cabal +7/−7
- src/Haskoin/Store/Web.hs +54/−128
- swagger.yaml +8/−453
- test/Haskoin/Store/IntegrationSpec.hs +82/−32
- test/Haskoin/Store/WebSpec.hs +3/−21
CHANGELOG.md view
@@ -6,6 +6,36 @@ ## [Unreleased] +## [1.10.0] - 2026-11-12++The API speaks JSON only. Its binary format is gone, from the server and+from the documentation.++### Removed++- **The binary format.** `Accept: application/octet-stream` no longer+ gives binary: every answer is JSON, errors and `/events` included. The+ "Binary format" sections of `README.md` and `swagger.yaml` are gone.+ Clients built with haskoin-store-data before 2.0.0 ask for binary and+ do not work with this server; version 2.0.0 of its client speaks JSON.+- **Binary frames over the WebSocket** of `/events`. Events are sent in+ text frames, and a binary frame from the client closes the connection+ as a command that cannot be read.+- **Raw bytes in `POST /transactions`.** The body is the transaction in+ hex.++### Changed++- Answers have `Vary: Accept-Encoding`, without `Accept`, which no longer+ changes what is answered.+- Depends on haskoin-store-data 2.0.0.++### Added++- `docs/fleet-mempool.md`: how the shared mempool of a fleet works, why+ it holds, and its weak points. A report in the repository, not in the+ package.+ ## [1.9.1] - 2026-11-11 ### Changed
README.md view
@@ -82,7 +82,7 @@ | Xpubs | Balances, transactions and UTXO of extended public keys, for legacy, nested SegWit and native SegWit addresses, with an optional Redis cache for large wallets | | Blocks and txs | Lookup by hash, height or time; raw serialized forms; transactions of a block | | Mempool | Persistent, indexed by arrival time, with replace-by-fee on BTC and pruning of old entries |-| Formats | JSON, pretty JSON, or a compact binary encoding |+| Formats | JSON, or pretty JSON | | Live events | New blocks and transactions as a JSON stream or over a WebSocket | | Compatibility | Endpoints in the style of Blockchain.info under `/blockchain/` | | Operations | Health check, StatsD metrics, RocksDB memory limits, Docker image |@@ -279,7 +279,7 @@ `quiet=true` sends blocks alone until the client watches something; a WebSocket client with a long list connects with it and sends the list in messages. The [API reference](swagger.yaml) has every message and-parameter, and the binary form of each message.+parameter. ### 7. Make big wallets fast @@ -482,9 +482,7 @@ - The root of the server, `/`, is the reference of its API as a page that can try requests, and `/swagger.yaml` the file behind it.-- Add `?pretty=true` for indented JSON.-- Send `Accept: application/octet-stream` for the binary format: see- [Binary format](#binary-format) below.+- Answers are JSON. Add `?pretty=true` for indented JSON. - Lists are paged with `limit` and `offset`, newest first. `height` begins a page at a transaction hash, a block hash, a block height or a Unix time instead.@@ -492,8 +490,7 @@ a value it cannot read, gives 400 with a message that names it, instead of being ignored. The Blockchain.info endpoints accept `format=json`, which clients written for them often send.-- Errors come back with an HTTP status, in the format that was asked for:- 400 for bad input, 404 for unknown things, 409 when a short transaction+- Errors come back as JSON with an HTTP status: 400 for bad input, 404 for unknown things, 409 when a short transaction number is ambiguous, 413 for requests that ask for too much. ### HTTP and HTTPS@@ -514,35 +511,6 @@ this is for a server by itself. The files are read at start, so a renewed certificate takes a restart. -### Binary format--Every endpoint but those under `/blockchain/` and `/dbstats` also answers-in a compact binary format, which is smaller and faster to decode than-JSON. The Haskell client in `haskoin-store-data` speaks it, and-`swagger.yaml` has a section, "Binary format", that is enough to write a-client in another language: the primitives, the layout of every type, and-what each endpoint returns.--- **Ask for it** with `Accept: application/octet-stream`. In a list of- types it wins when its `q` weight is above 0 and no less than that of- `application/json`; a wildcard or no header gives JSON. The answer has- `Content-Type: application/octet-stream`, and every answer has- `Vary: Accept`.-- **Errors come in binary too**: one byte that says which error, and a- message or a list of transactions for those that have one. Up to- version 1.8.1 they were JSON whatever was asked, so a client that also- talks to older servers reads the `Content-Type` of an error.-- **Lists start with their length**, the raw lists of blocks and- transactions included, which up to version 1.8.1 had none.-- **`/events` is the exception**: a stream of events of 33 bytes each,- with no count, for as long as the connection is open.-- **Over the WebSocket** binary is binary frames, for the events and for- the commands that choose transactions. The server answers in the kind- of frame that it was last sent.-- Everything is big-endian, except what is copied from the peer-to-peer- protocol, such as a whole transaction or block.-- ### Endpoints at a glance | Area | Endpoints |@@ -878,7 +846,7 @@ - **`haskoin-store`**, this repository: the indexer, the web server and the executable. - **[`haskoin-store-data`](https://github.com/jprupp/haskoin-store-data)**:- the data types, their JSON and binary encodings, and a client for the API+ the data types, their JSON, and a client for the API (`Haskoin.Store.WebClient`). Use this one to talk to a Haskoin Store server from Haskell. It is small to depend on, as it has no database or node.
haskoin-store.cabal view
@@ -5,7 +5,7 @@ -- see: https://github.com/sol/hpack name: haskoin-store-version: 1.9.1+version: 1.10.0 synopsis: Storage and index for Bitcoin and Bitcoin Cash description: An indexer for Bitcoin and Bitcoin Cash. It downloads blocks and mempool transactions from the peer-to-peer network, indexes them in RocksDB, and@@ -109,7 +109,7 @@ , hashtables >=1.4.2 && <1.5 , haskoin-core >=1.6.1 && <1.7 , haskoin-node >=2.5.1 && <2.6- , haskoin-store-data ==1.11.*+ , haskoin-store-data ==2.0.* , hedis >=0.16.4 && <0.17 , http-types >=0.12.6 && <0.13 , lens >=5.3.6 && <5.4@@ -174,7 +174,7 @@ , hashtables >=1.4.2 && <1.5 , haskoin-core >=1.6.1 && <1.7 , haskoin-node >=2.5.1 && <2.6- , haskoin-store-data ==1.11.*+ , haskoin-store-data ==2.0.* , hedis >=0.16.4 && <0.17 , http-types >=0.12.6 && <0.13 , lens >=5.3.6 && <5.4@@ -238,7 +238,7 @@ , haskoin-core >=1.6.1 && <1.7 , haskoin-node >=2.5.1 && <2.6 , haskoin-store- , haskoin-store-data ==1.11.*+ , haskoin-store-data ==2.0.* , hedis >=0.16.4 && <0.17 , http-types >=0.12.6 && <0.13 , lens >=5.3.6 && <5.4@@ -328,7 +328,7 @@ , haskoin-core >=1.6.1 && <1.7 , haskoin-node >=2.5.1 && <2.6 , haskoin-store- , haskoin-store-data ==1.11.*+ , haskoin-store-data ==2.0.* , hedis >=0.16.4 && <0.17 , hspec >=2.11.18 && <2.12 , http-types >=0.12.6 && <0.13@@ -395,7 +395,7 @@ , haskoin-core >=1.6.1 && <1.7 , haskoin-node >=2.5.1 && <2.6 , haskoin-store- , haskoin-store-data ==1.11.*+ , haskoin-store-data ==2.0.* , hedis >=0.16.4 && <0.17 , http-types >=0.12.6 && <0.13 , lens >=5.3.6 && <5.4@@ -459,7 +459,7 @@ , haskoin-core >=1.6.1 && <1.7 , haskoin-node >=2.5.1 && <2.6 , haskoin-store- , haskoin-store-data ==1.11.*+ , haskoin-store-data ==2.0.* , hedis >=0.16.4 && <0.17 , http-types >=0.12.6 && <0.13 , integration
src/Haskoin/Store/Web.hs view
@@ -15,18 +15,17 @@ -- -- 'runWeb' serves the endpoints described in "Haskoin.Store.WebCommon" and a -- set of endpoints compatible with the Blockchain.info API under--- @\/blockchain\/@. Responses are JSON by default, pretty-printed with--- @?pretty=true@, or binary if the request has the header--- @Accept: application\/octet-stream@. New blocks and mempool transactions are--- streamed at @\/events@, as lines of JSON or over a WebSocket.+-- @\/blockchain\/@. Responses are JSON, pretty-printed with @?pretty=true@.+-- New blocks and mempool transactions are streamed at @\/events@, as lines+-- of JSON or over a WebSocket. -- -- Limits on the size of responses are set with 'WebLimits'. Errors are -- returned as JSON with an HTTP status that depends on the 'Except'. -- = Design of the API ----- * __Two encodings of the same data.__ JSON is easy to use from anywhere. The--- binary encoding is several times smaller and faster to produce and parse, and--- is what the Haskell client uses.+-- * __One encoding.__ The API speaks JSON, which any language reads, and+-- which compression makes small. A second format would have to be kept+-- in step with the first for every change. -- -- * __Limits protect the server.__ A single request for every transaction of a -- busy address could take minutes and gigabytes. 'WebLimits' caps list sizes@@ -73,7 +72,6 @@ -- * Request parsing decodeTxBody,- acceptsBinary, -- * Logging logIt,@@ -149,7 +147,6 @@ fromJust, fromMaybe, isJust,- listToMaybe, mapMaybe, maybeToList, )@@ -219,7 +216,6 @@ import Haskoin.Store.Web.Docs (swaggerPage, swaggerYaml) import Haskoin.Store.Web.Events import Haskoin.Store.Web.Router hiding (header)-import Haskoin.Store.Web.Router qualified as Router import Haskoin.Store.WebCommon import Haskoin.Transaction import Haskoin.Util@@ -276,7 +272,6 @@ pendingRequest, rejectRequestWith, requestPath,- sendBinaryData, sendTextData, ) import Network.WebSockets qualified as WebSockets@@ -610,9 +605,7 @@ -- | Encoding of a response. data SerialAs- = -- | binary, for @Accept: application\/octet-stream@- SerialAsBinary- | -- | compact JSON+ = -- | compact JSON SerialAsJSON | -- | indented JSON, for @?pretty=true@ SerialAsPrettyJSON@@ -731,9 +724,9 @@ | pathInfo request == ["events"] = routerScoped routes ($ routes) notFound request | otherwise = routerScoped routes scoped notFound request routes = routesOf st- notFound req NoRoute = errorResponse st req ThingNotFound- notFound req (Unknown ks) =- errorResponse st req . UserError $+ notFound _ NoRoute = errorResponse st ThingNotFound+ notFound _ (Unknown ks) =+ errorResponse st . UserError $ "Unknown parameter: " <> T.unpack (T.intercalate ", " ks) saidWell <- newIORef Nothing saidRedis <- newIORef True@@ -840,24 +833,18 @@ | otherwise = inner req respond allowed = "GET, HEAD, POST, DELETE, OPTIONS" -- answers are compressed for a client that says it reads gzip: JSON- -- shrinks several times over, and the binary format, which is mostly- -- hashes, by less. The events are a stream that has to leave as it is- -- written, and are left alone.+ -- shrinks several times over. The events are a stream that has to+ -- leave as it is written, and are left alone. compressed inner req | pathInfo req == ["events"] = inner req- | otherwise = gzip gzipSettings inner req- gzipSettings =- Gzip.defaultGzipSettings- { Gzip.gzipCheckMime = \m -> Gzip.defaultCheckMime m || "application/octet-stream" `C.isPrefixOf` m- }- -- any origin may ask, and what is answered depends on the format that- -- was asked for, which a cache between the client and the server has- -- to know: it must not give the JSON of one client to another that- -- asked for binary. The same goes for whether it is compressed, and- -- the two are said in one header.+ | otherwise = gzip Gzip.defaultGzipSettings inner req+ -- any origin may ask, and whether the answer is compressed depends on+ -- what the client said it reads, which a cache between the client and+ -- the server has to know: it must not give gzip to a client that+ -- cannot read it. anyOrigin = modifyResponse (mapResponseHeaders varied) varied headers =- [("Access-Control-Allow-Origin", "*"), ("Vary", "Accept, Accept-Encoding")]+ [("Access-Control-Allow-Origin", "*"), ("Vary", "Accept-Encoding")] <> filter ((/= "Vary") . fst) headers -- | Answer the errors of the handler of a route: an 'Except' with its@@ -869,37 +856,28 @@ answer req e | Just Next <- fromException e = throwIO Next | Just TipMoved <- fromException e = throwIO TipMoved- | Just x <- fromException e = errorResponse st req x+ | Just x <- fromException e = errorResponse st x | otherwise = do logError st.config.logger $ fmtReq req.wai <> ": " <> cs (show e)- errorResponse st req ServerError+ errorResponse st ServerError -- | What a store that is a member of a fleet answers when it has no view of -- the block that the fleet answers from: 503, so that a load balancer sends -- the request to another store. Any answer of its own could differ from -- those of the other stores, which is what a fleet is there to prevent. behindResponse :: WebState -> Req -> IO Response-behindResponse st req = errorResponse st req BehindFleet+behindResponse st _ = errorResponse st BehindFleet --- | The response to an error, counted in the metrics, in the format that--- the request asked for.------ Errors were JSON whatever was asked, because 'Except' had no binary--- encoding. A client that speaks binary had to carry a JSON decoder for--- the one answer that it least expects. From haskoin-store-data 1.10 an--- error has a binary encoding like everything else.-errorResponse :: WebState -> Req -> Except -> IO Response-errorResponse st req e = do+-- | The response to an error, in JSON, counted in the metrics.+errorResponse :: WebState -> Except -> IO Response+errorResponse st e = do forM_ st.metrics $ \m -> mapM_ (`incrementCounter` 1) $ if | statusIsClientError (errStatus e) -> Just m.clientErrors | statusIsServerError (errStatus e) -> Just m.serverErrors | otherwise -> Nothing- return $- if wantsBinary req- then responseLBS (errStatus e) ((hContentType, "application/octet-stream") : again) (runPutL (serialize e))- else responseLBS (errStatus e) ((hContentType, "application/json") : again) (A.encode e)+ return $ responseLBS (errStatus e) ((hContentType, "application/json") : again) (A.encode e) where -- a server behind its fleet catches up within a block or is taken out -- of rotation: a client or a load balancer that reads this header@@ -1165,8 +1143,8 @@ pathCommon st ps parser action (const status200) encJson encValue False -- | The route of an API resource: parse the request, run the handler, and--- encode the response as JSON or binary, as the client asked. It accepts the--- parameters given, and @pretty@.+-- encode the response as JSON. It accepts the parameters given, and+-- @pretty@. pathCommon :: (ApiResource a b) => WebState ->@@ -1183,7 +1161,7 @@ proto <- serialAs req pretty res <- action =<< parser req return $- responseLBS (status res) [(hContentType, serialType proto)] (protoSerial proto encJson encValue res)+ responseLBS (status res) [(hContentType, "application/json")] (protoSerial proto encJson encValue res) where toProxy :: (Req -> IO a) -> Proxy a toProxy = const Proxy@@ -1201,64 +1179,23 @@ return $ responseLBS status200 [(hContentType, "text/plain; charset=utf-8")] (TL.encodeUtf8 t) --- | Encode a response in binary, JSON or pretty JSON.+-- | Encode a response in JSON or pretty JSON. protoSerial ::- (Serial a) => SerialAs -> (a -> Encoding) -> (a -> Value) -> a -> L.ByteString-protoSerial SerialAsBinary _ _ = runPutL . serialize protoSerial SerialAsJSON f _ = encodingToLazyByteString . f protoSerial SerialAsPrettyJSON _ g = encodePretty' defConfig {confTrailingNewline = True} . g --- | The content type of an encoding.-serialType :: SerialAs -> C.ByteString-serialType SerialAsBinary = "application/octet-stream"-serialType _ = "application/json"---- | Whether a request asks for the binary encoding: its @Accept@ header--- names @application\/octet-stream@, and gives it no less weight than it--- gives JSON.------ The header is a list of media types, each with a weight from 0 to 1,--- written @;q=@, that is 1 when left out. Binary is what is sent when--- @application\/octet-stream@ is in the list with a weight above 0 that is--- at least that of @application\/json@, if that is there too. A wildcard--- such as @*\/*@ asks for nothing in particular and gets JSON, the--- default, as does a request without the header. The header used to be--- compared with the one string, so that a client that sent a list, or a--- weight, got JSON without being told.-wantsBinary :: Req -> Bool-wantsBinary req = maybe False acceptsBinary (Router.header req "accept")---- | 'wantsBinary', given the value of the @Accept@ header.-acceptsBinary :: Text -> Bool-acceptsBinary accept =- case weight "application/octet-stream" of- Just q | q > 0 -> maybe True (q >=) (weight "application/json")- _ -> False- where- ranges = map range (T.splitOn "," accept)- weight name = lookup name ranges- range :: Text -> (Text, Double)- range part = case map T.strip (T.splitOn ";" part) of- [] -> ("", 0)- name : params -> (T.toLower name, fromMaybe 1 (listToMaybe (mapMaybe quality params)))- quality setting = case T.splitOn "=" setting of- [k, v] | T.toLower (T.strip k) == "q" -> readMaybe (T.unpack (T.strip v)) <|> Just 0- _ -> Nothing---- | The encoding that a request asks for: binary if it 'wantsBinary', else--- JSON, pretty if the @pretty@ parameter or the default says so.+-- | The encoding that a request asks for: JSON, pretty if the @pretty@+-- parameter or the default says so. serialAs :: Req -> Bool -> IO SerialAs-serialAs req pretty- | wantsBinary req = return SerialAsBinary- | otherwise = do- p <- paramOr req readBool "pretty" pretty- return $ if p then SerialAsPrettyJSON else SerialAsJSON+serialAs req pretty = do+ p <- paramOr req readBool "pretty" pretty+ return $ if p then SerialAsPrettyJSON else SerialAsJSON -- GET Block / GET Blocks -- @@ -1605,10 +1542,8 @@ -- log of events ("Haskoin.Store.Web.Events"). -- -- The client may send the commands of t'EventCommand' to choose the--- transactions that it hears about. Each side speaks JSON in text frames--- or binary in binary frames, and the server speaks as it was last spoken--- to; before the client has said anything, as the @Accept@ header of the--- handshake asks, like any request. A command that the server cannot read+-- transactions that it hears about. Each side speaks JSON in text frames;+-- a binary frame is a command that cannot be read. A command that the server cannot read -- or cannot do closes the connection with the reason, as a node does with -- an SPV client: there are no answers to wait for. --@@ -1631,8 +1566,6 @@ let request = pendingRequest pending (path, query) = C.break (== '?') (requestPath request) asked = parseQuery query- accept = lookup "Accept" (WebSockets.requestHeaders request)- binary = maybe False (acceptsBinary . T.decodeUtf8Lenient) accept if path /= "/events" then turnAway pending ThingNotFound else@@ -1640,42 +1573,37 @@ Left e -> turnAway pending e Right (start, watch) -> do conn <- acceptRequest pending- format <- newTVarIO binary -- websockets only sees pongs while reading, which the -- thread that hears the commands does withPingPong pingPong conn $ \c ->- race_ (hear c watch format) (tell c watch format start) `catch` gone+ race_ (hear c watch) (tell c watch start) `catch` gone -- a client that leaves is not an error to report gone :: WebSockets.ConnectionException -> IO () gone _ = return () -- the commands of the client, until one cannot be done- hear c watch format = do+ hear c watch = do message <- WebSockets.receiveDataMessage c- let (binary, read') = case message of- WebSockets.Text bytes _ -> (False, A.decode bytes >>= parseMaybe (unmarshalValue (net, ctx)))- WebSockets.Binary bytes -> (True, either (const Nothing) Just (unmarshal (net, ctx) (L.toStrict bytes)))- atomically (writeTVar format binary)+ let read' = case message of+ WebSockets.Text bytes _ -> A.decode bytes >>= parseMaybe (unmarshalValue (net, ctx))+ WebSockets.Binary _ -> Nothing case read' of Nothing -> refuse c "Could not read the command" Just cmd -> try (command ctx gap s.query.xPubBals watch cmd) >>= \case- Right (Right ()) -> hear c watch format+ Right (Right ()) -> hear c watch Right (Left why) -> refuse c why Left (_ :: Except) -> refuse c "Could not do the command" -- 1008: the message is against what the server allows refuse c why = WebSockets.sendCloseCode c 1008 (T.pack why) -- the events from a place in the log on, those that the client chose- tell c watch format from =+ tell c watch from = atomically (readEvents s.eventLog from) >>= \case Nothing -> WebSockets.sendClose c ("Event queue full" :: Text) Just entries -> do forM_ entries $ \e -> do wanted <- atomically (stateTVar watch (relevant ctx gap e.seen))- when wanted $- readTVarIO format >>= \case- True -> sendBinaryData c (runPutL (serialize e.event))- False -> sendTextData c (A.encode e.event)- tell c watch format (from + fromIntegral (length entries))+ when wanted $ sendTextData c (A.encode e.event)+ tell c watch (from + fromIntegral (length entries)) turnAway pending e = rejectRequestWith pending@@ -1687,15 +1615,15 @@ } -- | Handler for @GET \/events@: stream of new blocks and of transactions--- that enter or leave the mempool, one per line in JSON or one after the--- other in binary, until the client falls off the start of the log.+-- that enter or leave the mempool, one per line in JSON, until the client+-- falls off the start of the log. -- @after@ takes up from an event ('eventsStart'). handleEvents :: WebState -> Req -> IO Response handleEvents st req = do proto <- serialAs req False let asked name = lookup name req.params (start, watch) <- eventsOpen st asked >>= either (raise st) return- return . responseStream status200 [(hContentType, serialType proto)] $ \write flush' ->+ return . responseStream status200 [(hContentType, "application/json")] $ \write flush' -> let send from = do flush' -- ends the response when the client falls too far behind@@ -1711,7 +1639,6 @@ serial proto e = lazyByteString $ protoSerial proto toEncoding toJSON e <> newLine proto- newLine SerialAsBinary = mempty newLine SerialAsJSON = "\n" newLine SerialAsPrettyJSON = mempty own = st.config.store@@ -3273,7 +3200,7 @@ | h.redis == Just False = (status203, [("Haskoin-Degraded", "redis")]) | otherwise = (status200, []) return $- responseLBS status ((hContentType, serialType proto) : extra) (protoSerial proto toEncoding toJSON h)+ responseLBS status ((hContentType, "application/json") : extra) (protoSerial proto toEncoding toJSON h) where proxy = Proxy :: Proxy GetHealth @@ -3434,7 +3361,7 @@ Right tx -> return tx -- | Read a transaction from a request body. The body is the transaction in--- hex or as raw bytes. White space in the hex is ignored.+-- hex. White space around and inside it is ignored. -- -- The body must hold one transaction and nothing else. Bytes after the -- transaction are refused, as Bitcoin Core does for a raw transaction.@@ -3444,17 +3371,16 @@ -- witness only. Its nodes would not read the SegWit format, so the store -- does not take it either. decodeTxBody :: Network -> C.ByteString -> Either String Tx-decodeTxBody net body = fromHex <> fromBytes body+decodeTxBody net body =+ case parseBase16 (C.filter (not . isSpace) body) of+ Right b16 -> fromBytes (decodeBase16 b16)+ Left e -> Left (cs e) where fromBytes = runGetS $ do tx <- getNetworkTx net done <- isEmpty unless done $ fail "Trailing bytes after transaction" return tx- fromHex =- case parseBase16 (C.filter (not . isSpace) body) of- Right b16 -> fromBytes (decodeBase16 b16)- Left e -> Left (cs e) -- | Offset in the @offset@ parameter. Responds with an error if above the maximum. parseOffset :: WebState -> Req -> IO OffsetParam
swagger.yaml view
@@ -91,17 +91,11 @@ ## Formats - - **JSON** by default, without spaces, to keep responses small.+ - **JSON**, without spaces, to keep responses small. - **Pretty JSON** with `pretty=true`, indented for people to read.- - **Binary** with the header `Accept: application/octet-stream`: the compact- serialization that the Haskell client in- [haskoin-store-data](https://github.com/jprupp/haskoin-store-data)- decodes. It is smaller and faster to decode than JSON. "Binary format"- below gives every byte of it. Endpoints that end in `/raw` return data serialized as in the peer-to-peer- protocol, in hexadecimal inside JSON (`{"result": "0100…"}`), or as bytes- with the binary format.+ protocol, in hexadecimal inside JSON (`{"result": "0100…"}`). ## Parameters are checked @@ -141,9 +135,6 @@ | 503 | | From `/health`: the server is not ready. | | 503 | `behind-fleet` | The server is one of a fleet and does not have the block that the fleet answers from; another server of the fleet does. | - A request that asks for the binary format gets its errors in binary too:- see "Binary format".- `/health` also answers 203 for a server that works but cannot reach its Redis; see there. @@ -267,287 +258,14 @@ `application/x-www-form-urlencoded`. A URL has room for some hundred addresses; a body has no such limit. `/transactions` is the exception: `POST` there publishes a transaction.- - **`Vary: Accept, Accept-Encoding`** is on every answer, so that a cache- keeps the formats apart.+ - **`Vary: Accept-Encoding`** is on every answer, so that a cache keeps+ compressed and plain answers apart. - **`Retry-After`** is on a 503: one second from a server that is behind its fleet, five from a health check that fails. - **The reference itself** is at the root of every server: `/` is this page, for the version that the server runs, and `/swagger.yaml` the file behind it. - ## Binary format-- This section is enough to write a client in any language.-- ### Asking for it-- - Send the header `Accept: application/octet-stream`. In a list of types,- binary is sent when that type has a `q` weight above 0 that is no less- than the weight of `application/json`, if that is listed too. A- wildcard such as `*/*`, or no header, gives JSON. There is no query- parameter for it, and `pretty` has no effect on it.- - Every answer has `Vary: Accept`, so that a cache keeps the two formats- apart.- - A binary answer has `Content-Type: application/octet-stream`. The body is- one value, with nothing before or after it.- - **An error comes in the format asked for**, like any other answer. In- binary it is an `Except`, described below. Servers up to version 1.8.1- sent every error as JSON: read the `Content-Type` of an error to tell.- - `/health` answers 503 and 203 with a normal body, in the format asked for.- - The only endpoint that reads a body is `POST /transactions`: one- transaction as in the peer-to-peer protocol, as raw bytes or as hex. The- server tells them apart by content, not by `Content-Type`.- - The Blockchain.info endpoints and `/dbstats` have no binary form.- - Over the WebSocket at `/events`, binary is binary frames: the server- sends them when the handshake asked for binary, and from the first binary- frame that the client sends.-- ### Primitives-- Everything is big-endian, except inside the peer-to-peer structures (`Tx`,- `BlockHeader`, `OutPoint`, `Block`), which are byte for byte what nodes send.-- | Name | Encoding |- | --- | --- |- | `u8`, `u32`, `u64` | Unsigned integer of 1, 4 or 8 bytes, big-endian. |- | `i32`, `i64` | Two's complement, big-endian. |- | `u32le`, `u64le` | Little-endian; only in peer-to-peer structures. |- | `bool` | One byte: `00` false, `01` true. |- | `bytes` | `u64` length, then that many bytes. |- | `text` | `bytes` that hold UTF-8. |- | `list T` | `u64` count, then that many `T`. |- | `opt T` | `00` for none; `01` then `T`. |- | `map T` | `list` of pairs `u64` key, `T`; keys ascending, each once. |- | `hash256` | 32 bytes, in the reverse order of the hex in JSON. |- | `hash160` | 20 bytes, in the order of the hex. |- | `varint` | Bitcoin's CompactSize, shortest form; only in peer-to-peer structures. |- | `integer` | See below. |-- There are no variable-length integers outside the peer-to-peer structures:- every length and count is a `u64`. A type with several forms starts with a- tag byte, given in its table.-- `integer` holds a number of any size. `00` then `i32` when it fits 32 bits.- Otherwise `01`, a sign byte (`01`, or `ff` for negative), a `u64` length,- and the absolute value in that many bytes, least significant first, with no- zero byte at the end.-- ### Peer-to-peer structures-- | Type | Layout |- | --- | --- |- | `OutPoint` | `hash256` transaction id, `u32le` output index. 36 bytes. |- | `BlockHeader` | `u32le` version, `hash256` previous, `hash256` Merkle root, `u32le` time, bits, nonce. |- | `Tx` | `u32le` version, `varint` count, inputs, `varint` count, outputs, `u32le` lock time. |- | `Tx` input | `OutPoint`, `varint` length, script, `u32le` sequence. |- | `Tx` output | `u64le` value, `varint` length, script. |- | `Block` | `BlockHeader`, `varint` count, that many `Tx`. |-- A `Tx` with a witness has the SegWit form: `00 01` after the version, and- before the lock time one stack per input, each a `varint` count of items,- each a `varint` length and bytes. A `Tx` in which no input has a witness is- written in the form without it, on every network.-- ### Address-- An address is a tag and a hash. It does not carry its network, so it decodes- without knowing it. Writing it as text needs the network.-- | Tag | Then | Kind |- | --- | --- | --- |- | `00` | `hash160` | Pay to public key hash. |- | `01` | `hash160` | Pay to script hash. |- | `02` | `hash160` | Pay to witness public key hash. |- | `03` | `hash256` | Pay to witness script hash. |- | `04` | `u8` version, `bytes` program (40 at most) | Other SegWit address, such as Taproot. |- | `05` | `hash256` | Pay to script hash of 32 bytes (BCH). |- | `06` | `u8` type, `bytes` hash (64 at most) | Other CashAddr address. |-- The `hash256` of tags `03` and `05` is in the order of the script, not- reversed.-- ### BlockRef and TxRef-- `BlockRef` says where a transaction is. Its numbers are **inverted**: each- is written as the largest value of its width minus the number, so that the- bytes sort newest first. Subtract again to read them.-- | Tag | Then | Meaning |- | --- | --- | --- |- | `00` | `u64`: 2^64 - 1 - time | In the mempool; time first seen. |- | `01` | `u32`: 2^32 - 1 - height, `u32`: 2^32 - 1 - position | In a block, at that position in it. |-- `TxRef` is a `BlockRef`, then the `hash256` transaction id.-- ### Transaction-- | Field | Encoding | Note |- | --- | --- | --- |- | block | `BlockRef` | |- | version | `u32` | |- | locktime | `u32` | |- | inputs | `list Input` | |- | outputs | `list Output` | |- | deleted | `bool` | |- | rbf | `bool` | |- | time | `u64` | Not inverted. |- | txid | `hash256` | |- | size | `u32` | With the witness. |- | weight | `u32` | Present on every network. |- | fee | `u64` | 0 for a coinbase. |-- `Input`, by tag:-- | Field | Encoding | `00` coinbase | `01` other |- | --- | --- | --- | --- |- | outpoint | `OutPoint` | yes | yes |- | sequence | `u32` | yes | yes |- | sigscript | `bytes` | yes | yes |- | pkscript | `bytes` | | yes |- | value | `u64` | | yes |- | witness | `list bytes` | yes | yes |- | address | `opt Address` | | yes |-- The witness list is present on every network, and empty where there is none.-- `Output`: `u64` value, `bytes` script, `opt Spender`, `opt Address`.-- `Spender`: `hash256` id of the spending transaction, `u32` input index.-- `TxData` is the record of a transaction in the database. No endpoint- returns it; it is here for programs that read the database.-- | Field | Encoding | Note |- | --- | --- | --- |- | block | `BlockRef` | |- | tx | `Tx` | |- | prevs | `map Prev` | By input index. `Prev`: `bytes` script, `u64` value. |- | deleted | `bool` | |- | rbf | `bool` | |- | time | `u64` | |- | spenders | `map Spender` | By output index. |-- ### Blocks, balances and unspent outputs-- `BlockData`. The hash is not sent: it is the double SHA-256 of the header.-- | Field | Encoding | Note |- | --- | --- | --- |- | height | `u32` | |- | mainchain | `bool` | |- | work | `integer` | |- | header | `BlockHeader` | |- | size | `u32` | |- | weight | `u32` | Present on every network. |- | tx | `list hash256` | |- | outputs | `u64` | |- | fee | `u64` | |- | subsidy | `u64` | |-- | Type | Layout |- | --- | --- |- | `Balance` | `Address`, then `u64` confirmed, unconfirmed, utxo, txs, received. |- | `Unspent` | `BlockRef`, `OutPoint`, `u64` value, `bytes` script, `opt Address`. |- | `XPubBal` | `list u32` path, `Balance`. |- | `XPubUnspent` | `list u32` path, `Unspent`. |- | `XPubSummary` | `u64` confirmed, unconfirmed, received, utxo; `u32` external, change. |- | `TxId` | `hash256`. |- | `Event` | Tag `00` new best block or `01` mempool transaction, then `hash256`. 33 bytes. |- | `PeerInfo` | `bytes` user agent, `text` address, `i32` version, `u64` services, `bool` relay. |-- ### Commands of the event WebSocket-- What a client sends in a binary frame over the WebSocket at `/events`, one- command to a frame: a tag, then what it carries.-- | Tag | Then | Command |- | --- | --- | --- |- | `00` | `list Address` | Watch these addresses too. |- | `01` | `list Address` | Stop watching these addresses. |- | `02` | `XPub`, `u8` derivation | Watch this extended public key too. |- | `03` | `XPub`, `u8` derivation | Stop watching this key. |- | `04` | `Filter` | Set the Bloom filter, in place of the one before. |- | `05` | `bytes`, 520 at most | Add an element to the Bloom filter. |- | `06` | | Remove the Bloom filter. |- | `07` | | Forget all of it, and hear of every transaction. |- | `08` | | Forget all of it, and hear of blocks alone. |-- `XPub` is the 78 bytes of the key as BIP 32 serializes it, which is what- the Base58Check of its text decodes to: `u32` version of the network, `u8`- depth, 4 bytes fingerprint of the parent, `u32` index, 32 bytes chain code,- 33 bytes compressed public key. The derivation is `00` legacy, `01` compat,- `02` segwit.-- `Filter` is the `filterload` message of the peer-to-peer protocol (BIP 37):- `varint` length and that many bytes of filter, 36,000 at most, `u32le`- number of hash functions, 50 at most, `u32le` tweak, `u8` flags.-- ### Health-- Each part starts with its own `ok`, which repeats what its numbers say.-- | Field | Encoding | Note |- | --- | --- | --- |- | ok | `bool` | The whole check. |- | blocks | `bool` ok, `u32` headers, `u32` blocks, `i32` max | |- | last-block | `bool` ok, `i64` age, `i64` max | |- | last-tx | `bool` ok, `i64` age, `i64` max | |- | pending-txs | `bool` ok, `i64` count, `i64` max | |- | peers | `bool` ok, `i64` count, `i64` min | |- | net | `text` | |- | version | `text` | |- | time | `u64` | A number, not the text of the JSON. |- | redis | `u8` | `00` no cache, `01` not answering, `02` answering. |-- Older servers end the body before `redis`; read that as no cache.-- ### What each endpoint returns-- Lists are `list T`. The events are the one exception: they have no count,- because they have no end.-- An error, with any status that is not a success, is an `Except`: one byte- that says which, and for some a body.-- | Byte | Error | Body |- | --- | --- | --- |- | `00` | `not-found-or-invalid-arg` | |- | `01` | `server-error` | |- | `02` | `bad-request` | |- | `03` | `user-error` | The message: `bytes`, UTF-8. |- | `04` | `string-error` | The message: `bytes`, UTF-8. |- | `05` | `multiple-tx-index` | `list hash256`: the transactions that match. |- | `06` | `server-timeout` | |- | `07` | `request-too-large` | |- | `08` | `behind-fleet` | |-- | Endpoints | Body |- | --- | --- |- | `/block/{block}`, `/block/best`, `/block/time/…`, `/block/mtp/…` | `BlockData` |- | `/blocks`, `/block/latest`, `/block/height/…`, `/block/heights` | `list BlockData` |- | `/transaction/{txid}` | `Transaction` |- | `/transactions`, `/transactions/block/…`, `…/transactions/full` | `list Transaction` |- | `/address/…/transactions`, `/xpub/…/transactions` | `list TxRef` |- | `/address/{address}/balance` | `Balance` |- | `/address/balances` | `list Balance` |- | `/address/…/unspent` | `list Unspent` |- | `/xpub/{xpub}` | `XPubSummary` |- | `/xpub/{xpub}/balances` | `list XPubBal` |- | `/xpub/{xpub}/unspent` | `list XPubUnspent` |- | `DELETE /xpub/{xpub}` | `bool` |- | `/transaction/{txid}/after/{height}` | `opt bool` |- | `POST /transactions` | `TxId` |- | `/mempool` | `list hash256` |- | `/peers` | `list PeerInfo` |- | `/health` | Health, as above. |- | `…/raw` of one block or transaction | `Block` or `Tx`, bare. |- | `…/raw` of several | `list Block` or `list Tx`. Servers up to 1.8.1 sent them with no count. |- | `/events` | An endless stream of `Event`, **no count** and nothing between them. |-- The wrappers add no bytes: the `result` object of the JSON- (`GenericResult`, `RawResult`, `RawResultList` in the Haskell client) is- its content alone, and `SerialList` is `list T`.- tags: - name: Address description: |@@ -648,10 +366,6 @@ count: 20 min: 1 time: '2021-06-14 10:21:43 UTC'- application/octet-stream:- schema:- type: string- format: binary '503': description: unavailable content:@@ -727,10 +441,6 @@ block: height: 688707 position: 786- application/octet-stream:- schema:- type: string- format: binary '404': description: Invalid address. content:@@ -842,10 +552,6 @@ time: 1621533785 rbf: false weight: 900- application/octet-stream:- schema:- type: string- format: binary '404': description: Invalid address. content:@@ -897,10 +603,6 @@ block: height: 688707 position: 786- application/octet-stream:- schema:- type: string- format: binary '404': description: Invalid address. content:@@ -1085,10 +787,6 @@ time: 1621533785 rbf: false weight: 900- application/octet-stream:- schema:- type: string- format: binary '404': description: Invalid address. content:@@ -1125,10 +823,6 @@ utxo: 0 txs: 2 received: 2275183814- application/octet-stream:- schema:- type: string- format: binary '404': description: Invalid address. content:@@ -1170,10 +864,6 @@ utxo: 0 txs: 2 received: 2259981628- application/octet-stream:- schema:- type: string- format: binary '404': description: Invalid Address. content:@@ -1242,10 +932,6 @@ index: 0 pkscript: 76a9143fd3cf57704a93e9782a97a4fb5a3bf7fb1a604788ac value: 24597765- application/octet-stream:- schema:- type: string- format: binary '404': description: Invalid address. content:@@ -1318,10 +1004,6 @@ index: 0 pkscript: 76a91468150eecfe9abfbe5660c4953bdc8a4c127f91b488ac value: 52128096- application/octet-stream:- schema:- type: string- format: binary '404': description: Invalid address. content:@@ -1408,10 +1090,6 @@ fees: 87081 outputs: 115319020328 work: 440514608165117083892886234- application/octet-stream:- schema:- type: string- format: binary '/block/best/raw': get:@@ -1445,10 +1123,6 @@ 0120ffffffff0100f2052a0100000043410408ab2f56361f83064e4ce51acc29\ 1fb57c2cbcdb1d6562f6278c43a1406b548fd6cefc11bcc29eb620d5861cb9ed\ 69dc39f2422f54b06a8af4f78c8276cfdc6bac00000000"- application/octet-stream:- schema:- type: string- format: binary '/block/latest': get:@@ -1521,10 +1195,6 @@ fees: 9075725 outputs: 58906363954005 work: 440511085324684718115244800- application/octet-stream:- schema:- type: string- format: binary '/block/{block}': get:@@ -1568,10 +1238,6 @@ fees: 0 outputs: 5301000000 work: 451709610344319134- application/octet-stream:- schema:- type: string- format: binary '404': description: Block not found. content:@@ -1635,10 +1301,6 @@ 185c81e2aa2c41ab175407c09484ce9694b44953fcb751206564a9c24dd094d4\ 2fdbfdd5aad3e063ce6af4cfaaea4ea14fbbffffffff0140420f000000000019\ 76a91439aa3d569e06a1d7926dc4be1193c99bf2eb9ee088ac00000000"- application/octet-stream:- schema:- type: string- format: binary '404': description: Block not found. content:@@ -1702,10 +1364,6 @@ fees: 0 outputs: 0 work: 363124009197915- application/octet-stream:- schema:- type: string- format: binary '/block/height/{height}': get:@@ -1747,10 +1405,6 @@ fees: 0 outputs: 0 work: 863301599433- application/octet-stream:- schema:- type: string- format: binary '404': description: Block not found. content:@@ -1787,10 +1441,6 @@ 0138ffffffff0100f2052a010000004341045e071dedd1ed03721c6e9bba28fc\ 276795421a378637fb41090192bb9f208630dcbac5862a3baeb9df3ca6e4e256\ b7fd2404824c20198ca1b004ee2197866433ac00000000"- application/octet-stream:- schema:- type: string- format: binary '404': description: Block not found. content:@@ -1840,10 +1490,6 @@ fees: 0 outputs: 0 work: 863301599433- application/octet-stream:- schema:- type: string- format: binary '404': description: Block not found. content:@@ -1885,10 +1531,6 @@ 6f722062616e6b73ffffffff0100f2052a01000000434104678afdb0fe554827\ 1967f1a67130b7105cd6a828e03909a67962e0ea1f61deb649f6bc3f4cef38c4\ f35504e51ec112de5c384df7ba0b8d578a4c702b6bf11d5fac00000000"- application/octet-stream:- schema:- type: string- format: binary '404': description: Block not found. content:@@ -1938,10 +1580,6 @@ fees: 0 outputs: 0 work: 889071796431- application/octet-stream:- schema:- type: string- format: binary '404': description: Block not found. content:@@ -1981,10 +1619,6 @@ 010effffffff0100f2052a01000000434104bc184a82d824a8ff59576e8b6843\ 767407e4f35edb5ad725460764a30ddf91479fd0eb1b0c2b4371a06b3fdae345\ 3262fcf3358afafe3dbebccf2fdefcd0853cac00000000"- application/octet-stream:- schema:- type: string- format: binary '404': description: Block not found. content:@@ -2062,10 +1696,6 @@ fees: 0 outputs: 0 work: 1292804882733- application/octet-stream:- schema:- type: string- format: binary '404': description: Block not found. content:@@ -2135,10 +1765,6 @@ time: 1621593351 rbf: false weight: 876- application/octet-stream:- schema:- type: string- format: binary '404': description: Transaction not found. content:@@ -2174,10 +1800,6 @@ 4a8d8182b6de761f6af3b8c6ba8f0000000002ce040000000000001976a914e7\ f58a9d4dfa6fdf481f03ff9e696c1c1eadd36688acb4168900000000001976a9\ 14dd1eee80cebba451dc26f6423881b8836fe86a8388ac00000000"- application/octet-stream:- schema:- type: string- format: binary '404': description: Transaction not found. content:@@ -2222,10 +1844,6 @@ type: boolean nullable: true example: true- application/octet-stream:- schema:- type: string- format: binary '404': description: Invalid argument. content:@@ -2332,10 +1950,6 @@ time: 1621593351 rbf: false weight: 876- application/octet-stream:- schema:- type: string- format: binary '404': description: Invalid argument. content:@@ -2350,8 +1964,8 @@ description: | Send a signed transaction to the network, and get its id back. - The body is the transaction in hexadecimal (white space is ignored) or as- raw bytes. It must be in the format that nodes of the network read: on BCH,+ The body is the transaction in hexadecimal (white space is ignored). It+ must be in the format that nodes of the network read: on BCH, which has no SegWit, the format without witness data. **A 200 means the transaction was sent, not that the network accepted it.**@@ -2386,11 +2000,6 @@ c8926b65f64e312dd821b243a37c000000000211180000000000001976a91434\ 04e444809abd438d85eb18e8d07c796e364afe88ac399b7c00000000001976a9\ 1486a7f1a2047ed6719fa1ad5484f15c37c104887c88ac00000000"- application/octet-stream:- schema:- type: string- format: binary- description: Bitcoin protocol transaction. responses: '200': description: Transaction accepted by network peer.@@ -2400,10 +2009,6 @@ $ref: '#/components/schemas/TxId' example: txid: 0b7cb07d848959fbd02ad6e2ed21fac888b8e8011c6b8c5362d05c703066ae67- application/octet-stream:- schema:- type: string- format: binary '400': description: Transaction rejected. content:@@ -2592,10 +2197,6 @@ time: 1293623863 rbf: false weight: 900- application/octet-stream:- schema:- type: string- format: binary '404': description: Block not found. content:@@ -2659,10 +2260,6 @@ 4dd094d42fdbfdd5aad3e063ce6af4cfaaea4ea14fbbffffffff0140420f0000\ 0000001976a91439aa3d569e06a1d7926dc4be1193c99bf2eb9ee088ac000000\ 00"- application/octet-stream:- schema:- type: string- format: binary '404': description: Block not found. content:@@ -2708,10 +2305,6 @@ 4a8d8182b6de761f6af3b8c6ba8f0000000002cf0400000000000017a914d1b0\ b139593754efcaab5ad20d13dd3be82e5bfe875e1c8900000000001976a914dd\ 1eee80cebba451dc26f6423881b8836fe86a8388ac00000000"- application/octet-stream:- schema:- type: string- format: binary '404': description: Invalid argument. content:@@ -2748,10 +2341,6 @@ - a60e3b0775e08877d0a8edac874090034f2fa55a15fb339b98b18abb5c8cbdf3 - c9c50b1f75b4a69319c51ffae3a47df7de99b02089e696b711244fd4591eb51e - 11acbc09285f99d2cc11b707b61952258d554dcffe9a91ab0f6a22d1b06ac618- application/octet-stream:- schema:- type: string- format: binary '/events': get:@@ -2861,10 +2450,8 @@ message each, and `after` works the same. The server pings every 30 seconds and drops clients that do not answer within 60. - Messages are JSON in text frames or binary in binary frames ("Binary- format" above). The server answers in the kind of frame that the- client last sent; before the client has sent any, in binary if the- `Accept` header of the handshake asked for it, else in JSON.+ Messages are JSON in text frames. A binary frame from the client is a+ message that the server cannot read, and closes the connection. ## Choosing transactions @@ -2934,10 +2521,6 @@ example: type: tx id: 3e225aa5d41b0f7560ee641f86b00728adc626102b348f9d319e0fdfd848a0db- application/octet-stream:- schema:- type: string- format: binary '400': description: A parameter could not be read, or names too much to watch. content:@@ -2988,10 +2571,6 @@ version: 70015 services: "0000000000000425" relay: true- application/octet-stream:- schema:- type: string- format: binary '/xpub/{xpub}': get:@@ -3028,10 +2607,6 @@ indices: change: 27 external: 73- application/octet-stream:- schema:- type: string- format: binary '400': description: | The wallet has no transactions with this `derive`, but has some with@@ -3083,10 +2658,6 @@ type: boolean nullable: false example: true- application/octet-stream:- schema:- type: string- format: binary '404': description: Invalid argument. content:@@ -3329,10 +2900,6 @@ rbf: false weight: 1624 - application/octet-stream:- schema:- type: string- format: binary '400': description: | The wallet has no transactions with this `derive`, but has some with@@ -3394,10 +2961,6 @@ block: height: 678151 position: 657- application/octet-stream:- schema:- type: string- format: binary '400': description: | The wallet has no transactions with this `derive`, but has some with@@ -3475,10 +3038,6 @@ utxo: 0 txs: 2 received: 23000- application/octet-stream:- schema:- type: string- format: binary '400': description: | The wallet has no transactions with this `derive`, but has some with@@ -3563,10 +3122,6 @@ path: - 1 - 26- application/octet-stream:- schema:- type: string- format: binary '400': description: | The wallet has no transactions with this `derive`, but has some with
test/Haskoin/Store/IntegrationSpec.hs view
@@ -64,7 +64,7 @@ ( Address (..), BloomFlags (..), Ctx,- Network,+ Network (..), OutPoint (..), TxHash (..), XPrvKey,@@ -75,15 +75,16 @@ bloomInsert, deriveXPubKey, encodeHex,+ headerHash, hexToBlockHash, hexToTxHash, makeXPrvKey,- marshal, marshalJSON, prepareContext, prvSubKey, pubSubKey, sha256,+ txHash, txHashToHex, xPrvWif, xPubAddr,@@ -117,11 +118,17 @@ import Haskoin.Store.Common (StoreEvent (..)) import Haskoin.Store.Data ( Balance (..),+ BlockData (..), BlockRef (..), Event (..), EventCommand (..), Except (..),+ HealthCheck (..),+ BlockHealth (..),+ RawResult (..),+ SerialList (..), Spender (..),+ Transaction (..), TxData (..), XPubBal (..), )@@ -132,6 +139,8 @@ import Haskoin.Store.Integration.Node import Haskoin.Store.Integration.Podman import Haskoin.Store.Integration.Valkey+import Haskoin.Store.WebClient qualified as Client+import Haskoin.Store.WebCommon (ApiResource, NoCache (..), NoTx (..)) import NQE (receive, withSubscription) import Network.Socket qualified as Socket import Network.WebSockets qualified as WS@@ -176,7 +185,8 @@ it "catches up with changes that the database has before their events" catchUpTest it "keeps the spenders of cached transactions up to date" spenderTest it "evicts the least recently used xpubs" evictTest- it "answers an error in the format that was asked for" errorFormatTest+ it "answers an error in JSON, whatever was asked for" errorFormatTest+ it "answers the Haskell client of haskoin-store-data" clientTest it "speaks HTTP as browsers, proxies and long requests need it" httpTest it "serves HTTPS, and nothing else, when given a certificate" tlsTest it "tells each client of the transactions it chose, and takes up where it left" eventsTest@@ -1559,9 +1569,9 @@ (== 200) . fst <$> getJSON env.cached "/health" checkXPub env x ref --- | An error is answered in binary to a request that asks for binary, and--- in JSON otherwise, and every answer says that it depends on what was--- asked for.+-- | An error is answered in JSON, whatever the request asks for, and+-- every answer says that it depends on whether the client reads gzip, and+-- on nothing else. errorFormatTest :: Env -> IO () errorFormatTest env = do let unknown = "/transaction/" <> replicate 64 'a'@@ -1577,17 +1587,53 @@ r ^. Wreq.responseHeader "Vary", r ^. Wreq.responseBody )- (code, kind, vary, bytes) <- ask (Just "application/octet-stream")- (code, kind, vary) `shouldBe` (404, "application/octet-stream", "Accept, Accept-Encoding")- -- the error, as haskoin-store-data encodes it- decode (BL.toStrict bytes) `shouldBe` Right ThingNotFound- -- in a list with a weight, which used to give JSON- (_, listed, _, _) <- ask (Just "text/plain;q=0.1, application/octet-stream;q=0.9")- listed `shouldBe` "application/octet-stream"- (code', kind', vary', json) <- ask Nothing- (code', kind', vary') `shouldBe` (404, "application/json", "Accept, Accept-Encoding")- (body json ! "error") `shouldBe` String "not-found-or-invalid-arg"+ forM_ [Just "application/octet-stream", Nothing] $ \accept -> do+ (code, kind, vary, json) <- ask accept+ (code, kind, vary) `shouldBe` (404, "application/json", "Accept-Encoding")+ (body json ! "error") `shouldBe` String "not-found-or-invalid-arg" +-- | The client of haskoin-store-data reads what the server answers: values+-- in the JSON of the network, plain JSON, raw transactions, errors, and the+-- lines of the events.+clientTest :: Env -> IO ()+clientTest env = do+ let x = newXPub env "client" DeriveNormal+ -- the client puts the name of the network in the path, as+ -- api.haskoin.com has it, and the server of the tests has none+ conf = Client.ApiConfig {net = env.net {name = ""}, host = env.plain}+ call :: (ApiResource a b) => a -> IO b+ call r = Client.apiCall env.ctx conf r >>= either (fail . show) return+ heard <- newIORef []+ listener <- async . Client.apiEvents conf $ \e -> modifyIORef' heard (e :) >> return True+ -- the client is listening once it hears of a block+ void (mine env 1)+ waitFor "the client to listen" (not . null <$> readIORef heard)+ ref <- newLedger+ paid <- pay env x ref [([0, 0], 100000)]+ txid <- maybe (fail "Not a hash") return (hexToTxHash paid)+ -- the node relays the payment in its own time: a block before it would+ -- confirm it without it ever entering the mempool of the store+ waitFor "the client to hear of the payment" ((EventTx txid `elem`) <$> readIORef heard)+ [mined] <- mine env 1+ block <- call (Client.GetBlockBest (NoTx True))+ blockHashToHex (headerHash block.header) `shouldBe` mined+ tx <- call (Client.GetTx txid)+ tx.txid `shouldBe` txid+ raw <- call (Client.GetTxRaw txid)+ txHash raw.get `shouldBe` txid+ balances <- call (Client.GetAddrsBalance [addrOf env x [0, 0]])+ map (.confirmed) balances.get `shouldBe` [100000]+ keys <- call (Client.GetXPubBalances x.spec.key DeriveNormal (NoCache False))+ sum (map (.balance.confirmed) keys.get) `shouldBe` 100000+ health <- call Client.GetHealth+ health.blocks.blocks `shouldSatisfy` (> 0)+ -- an error comes back as the error that it is+ unknown <- Client.apiCall env.ctx conf (Client.GetTx (TxHash (sha256 ("nothing" :: C8.ByteString))))+ either Just (const Nothing) unknown `shouldBe` Just ThingNotFound+ hash <- maybe (fail "Not a hash") return (hexToBlockHash mined)+ waitFor "the client to hear of the block" ((EventBlock hash `elem`) <$> readIORef heard)+ cancel listener+ -- | A web server on the store of the tests, given a certificate that is -- made for the occasion: it answers over HTTPS, and does not answer a -- request that is not encrypted. The requests are made with curl, told@@ -1651,11 +1697,12 @@ -- | The events of the server over a WebSocket. A client that says nothing -- hears of every transaction; one that names an extended public key, an--- address or a Bloom filter hears of the transactions that touch them, in--- JSON or in binary as it spoke, and all hear of the block. The key is+-- address or a Bloom filter hears of the transactions that touch them, and+-- all hear of the block. The key is -- followed past the gap it had when it was named. A client that comes -- back with the hash of an event hears what came after it, and one that--- sends what the server cannot read is disconnected.+-- sends what the server cannot read, a binary frame among it, is+-- disconnected. eventsTest :: Env -> IO () eventsTest env = do let x = newXPub env "events" DeriveNormal@@ -1663,14 +1710,13 @@ port = read (reverse (takeWhile (/= ':') (reverse env.plain))) :: Int client path = WS.runClient "127.0.0.1" port path json = BL.toStrict . marshalJSON (env.net, env.ctx)- binary = marshal (env.net, env.ctx) -- every event that a client hears after saying something, in order hearing path say heard = client path $ \c -> do- mapM_ (either (WS.sendTextData c) (WS.sendBinaryData c)) (say :: [Either C8.ByteString C8.ByteString])+ mapM_ (WS.sendTextData c) (say :: [C8.ByteString]) forever $ WS.receiveDataMessage c >>= \case WS.Text bytes _ -> modifyIORef' heard (either error id (eitherDecode bytes) :)- WS.Binary bytes -> modifyIORef' heard (either error id (decode (BL.toStrict bytes)) :)+ WS.Binary _ -> error "A binary frame from the server" eventId = \case EventTx h -> txHashToHex h EventBlock h -> blockHashToHex h@@ -1683,9 +1729,9 @@ mapM async [ hearing "/events" [] all',- hearing "/events" [Left (json (EventWatchXPub x.spec))] key,- hearing "/events" [Right (binary (EventWatch [addrOf env y [0, 0]]))] address,- hearing "/events" [Right (binary (EventFilterLoad bloom))] filtered,+ hearing "/events" [json (EventWatchXPub x.spec)] key,+ hearing "/events" [json (EventWatch [addrOf env y [0, 0]])] address,+ hearing "/events" [json (EventFilterLoad bloom)] filtered, -- chosen in the URL, before the first event: blocks alone, a key -- under a type of address, an address hearing "/events?quiet=true" [] quiet,@@ -1745,13 +1791,17 @@ fst <$> getJSON env.plain "/events?after=nonsense" `shouldReturn` 400 refused <- tryAny (client ("/events?after=" <> replicate 64 '0') (\_ -> return ())) either (const True) (const False) refused `shouldBe` True- -- a command that cannot be read ends the connection, with the reason- closed <- try . client "/events" $ \c -> do- WS.sendTextData c ("{\"type\":\"nonsense\"}" :: C8.ByteString)- void (WS.receiveDataMessage c)- case closed of- Left (WS.CloseRequest code why) -> (code, why) `shouldBe` (1008, "Could not read the command")- wrong -> expectationFailure ("Not closed: " <> show wrong)+ -- a command that cannot be read ends the connection, with the reason,+ -- and so does a binary frame+ let sent message = do+ closed <- try . client "/events" $ \c -> do+ WS.sendDataMessage c message+ void (WS.receiveDataMessage c)+ case closed of+ Left (WS.CloseRequest code why) -> (code, why) `shouldBe` (1008, "Could not read the command")+ wrong -> expectationFailure ("Not closed: " <> show wrong)+ sent (WS.Text "{\"type\":\"nonsense\"}" Nothing)+ sent (WS.Binary "\x06") -- | The first lines of a stream of events, which has no end: the request -- is given up once they are there.
test/Haskoin/Store/WebSpec.hs view
@@ -16,7 +16,7 @@ MaxHealth (..), TimeHealth (..), )-import Haskoin.Store.Web (acceptsBinary, ailments, decodeTxBody, logIt)+import Haskoin.Store.Web (ailments, decodeTxBody, logIt) import Network.HTTP.Types (status200, status404, status500, status503) import Network.Wai (Request (..), defaultRequest, responseLBS) import Network.Wai.Internal (ResponseReceived (..))@@ -109,24 +109,6 @@ alone = behind {peers = CountHealth {count = 0, min = 1}} :: HealthCheck ailments behind `shouldBe` "5 blocks behind the headers" ailments alone `shouldBe` "5 blocks behind the headers, 0 peers"- describe "acceptsBinary" $ do- it "takes the binary type by itself" $- acceptsBinary "application/octet-stream" `shouldBe` True- it "takes it in a list, in any case and with spaces" $ do- acceptsBinary "text/html, Application/Octet-Stream" `shouldBe` True- acceptsBinary "application/octet-stream , */*" `shouldBe` True- it "takes it with a weight above zero" $- acceptsBinary "application/octet-stream;q=0.5" `shouldBe` True- it "gives JSON to a wildcard, to nothing and to JSON" $ do- acceptsBinary "*/*" `shouldBe` False- acceptsBinary "" `shouldBe` False- acceptsBinary "application/json" `shouldBe` False- it "gives JSON when binary is refused with a weight of zero" $- acceptsBinary "application/octet-stream;q=0, application/json" `shouldBe` False- it "follows the weights when both are named" $ do- acceptsBinary "application/json;q=0.9, application/octet-stream" `shouldBe` True- acceptsBinary "application/json, application/octet-stream;q=0.9" `shouldBe` False- acceptsBinary "application/json;q=0.5, application/octet-stream; q=0.5" `shouldBe` True describe "decodeTxBody" $ do -- The id is the double SHA-256 of the bytes of plainTxHex, reversed. It -- was computed outside this library. The witness is not part of the id,@@ -135,9 +117,9 @@ idOf = fmap (txHashToHex . txHash) it "reads a transaction given as hex" $ idOf (decodeTxBody btc plainTxHex) `shouldBe` Right txid- it "reads a transaction given as raw bytes" $ do+ it "refuses a transaction given as raw bytes" $ do let bytes = fromJust (decodeHex (decodeUtf8 plainTxHex))- idOf (decodeTxBody btc bytes) `shouldBe` Right txid+ decodeTxBody btc bytes `shouldSatisfy` isLeft it "ignores white space around the hex" $ idOf (decodeTxBody btc (" " <> plainTxHex <> "\r\n")) `shouldBe` Right txid it "refuses bytes after the transaction" $