haskoin-store 1.16.2 → 1.16.3
raw patch · 6 files changed
+287/−32 lines, 6 filesdep +unixPVP ok
version bump matches the API change (PVP)
Dependencies added: unix
API changes (from Hackage documentation)
+ Haskoin.Store.Web.Listen: parseMode :: String -> Maybe FileMode
+ Haskoin.Store.Web.Listen: systemdSocket :: IO (Maybe Socket)
+ Haskoin.Store.Web.Listen: withUnixSocket :: FilePath -> FileMode -> (Socket -> IO a) -> IO a
Files
- CHANGELOG.md +10/−0
- README.md +44/−1
- app/Main.hs +62/−26
- haskoin-store.cabal +9/−1
- src/Haskoin/Store/Web.hs +10/−4
- src/Haskoin/Store/Web/Listen.hs +152/−0
CHANGELOG.md view
@@ -4,6 +4,16 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html) +## [1.16.3] - 2026-11-18++### Added+- **The API can listen on a Unix socket.** `--socket PATH` (`SOCKET`) makes+ one, with the mode of `--socket-mode` (`SOCKET_MODE`, `0660` by default),+ removes a stale one at start and refuses a path in use. A socket passed+ by systemd socket activation (`LISTEN_FDS`) wins over both and over+ `--host` and `--port`. The new `Haskoin.Store.Web.Listen` has both. The+ access log shows `unix` for a client on such a socket.+ ## [1.16.2] - 2026-11-18 ### Changed
README.md view
@@ -336,6 +336,8 @@ | Option | Environment | Default | Meaning | | --- | --- | --- | --- | | `--host` / `--port` | `HOST` / `PORT` | `*` / `3000` | Address the API listens on |+| `--socket` | `SOCKET` | none | Listen on a Unix socket at this path instead of `--host` and `--port` ([Unix sockets](#unix-sockets)) |+| `--socket-mode` | `SOCKET_MODE` | `0660` | Permissions of that socket, in octal | | `--noweb` | `NO_WEB` | off | Index only, no API | | `--tls-cert` / `--tls-key` | `TLS_CERT` / `TLS_KEY` | none | Serve HTTPS with this certificate chain and private key, both files in PEM | | `--no-blockchain-info` | `NO_BLOCKCHAIN_INFO` | off | Disable the `/blockchain/` endpoints and price lookups |@@ -566,6 +568,45 @@ this is for a server by itself. The files are read at start, so a renewed certificate takes a restart. +### Unix sockets++A store with its proxy on the same machine needs no TCP port. On a Unix+socket, who may connect is set by the file's owner and mode, and nothing+on the network can reach the store by mistake. There are two ways to get+one:++- **`--socket PATH`**: the store makes the socket, with the mode of+ `--socket-mode` (`0660` by default: owner and group), and removes it when+ it stops. A socket left by a store that did not stop cleanly is removed+ at start if nothing answers on it; a store already listening there, or a+ file that is not a socket, stops the new one instead. The path may be at+ most 107 bytes long.+- **Socket activation by systemd**: a `.socket` unit makes the socket and+ owns its path and permissions, and the store takes it from systemd+ (`LISTEN_FDS`). Clients that connect while the store restarts wait+ instead of failing. A socket from systemd wins over `--socket`, which+ wins over `--host` and `--port`, so adding the unit needs no change to+ the store's options. For a user service:++ ```ini+ # ~/.config/systemd/user/haskoin-store-bch.socket+ [Socket]+ ListenStream=%t/haskoin-store/bch.sock+ SocketMode=0660++ [Install]+ WantedBy=sockets.target+ ```++ A `.socket` unit starts the `.service` of the same name.++Test it with `curl --unix-socket /run/user/1000/haskoin-store/bch.sock+http://localhost/health`. nginx reaches it with+`server unix:/run/user/1000/haskoin-store/bch.sock;` in an `upstream`. A+client on a Unix socket has no address, so the access log shows `unix` for+it unless the proxy sends `X-Forwarded-For`. Nothing else in the store+looks at the client's address, and WebSocket events work the same.+ ### Endpoints at a glance | Area | Endpoints |@@ -884,6 +925,7 @@ | `Haskoin.Store.Fleet.Member`, `Haskoin.Store.Fleet.XPub` | A fleet, and its cache | | `Haskoin.Store.Web`, `Haskoin.Store.Web.Events` | The web server, and `/events` | | `Haskoin.Store.Web.Pieces` | Answers that are sent while they are read |+| `Haskoin.Store.Web.Listen` | A Unix socket to listen on, made by the store or passed by systemd | | `Haskoin.Store.Web.Admission` | The limit on long requests answered at once, and the line in front of it (CoDel) | | `Haskoin.Store.Web.Congestion` | The delay of GHC's scheduler, and the controller that moves the limit with it | | `Haskoin.Store.Runtime` | GHC's garbage collector, as StatsD metrics |@@ -1171,7 +1213,8 @@ - **Every request**, with `--access-log`, which is off by default: `203.0.113.7 GET /block/best HTTP/1.1: 200 OK, 0.4 ms`. A public server usually has a proxy in front that keeps this log better. The- address is the first of `X-Forwarded-For` when a proxy sent one.+ address is the first of `X-Forwarded-For` when a proxy sent one, and+ `unix` for a client on a [Unix socket](#unix-sockets) without one. An outage of Redis is one warning when it starts, `Could not … Going on without`, and one line when it is over, `Can … again`. The attempts in
app/Main.hs view
@@ -34,13 +34,16 @@ XPubConfig (..), defaultXPubConfig, runWeb,+ runWebSocket, withStore, ) import Haskoin.Store.Common (loopForever) import Haskoin.Store.Database.Stalls (defaultBackgroundJobs) import Haskoin.Store.Runtime (runtimeMetrics)+import Haskoin.Store.Web.Listen (parseMode, systemdSocket, withUnixSocket) import Haskoin.Store.XPub.Indexer (Settings (..)) import NQE (receive, withSubscription)+import Numeric (showOct) import Options.Applicative ( Mod, Parser,@@ -55,11 +58,13 @@ info, long, many,+ maybeReader, metavar, option, progDesc, short, showDefault,+ showDefaultWith, strOption, switch, value,@@ -79,6 +84,7 @@ defStatConfig, withStats, )+import System.Posix.Types (FileMode) import Text.Read (readMaybe) -- | Version of the package, set at build time.@@ -97,6 +103,8 @@ { dir :: !FilePath, host :: !String, port :: !Int,+ socketPath :: !FilePath,+ socketMode :: !FileMode, net :: !String, discover :: !Bool, peers :: ![String],@@ -158,6 +166,10 @@ env "HOST" "*" pure port <- env "PORT" 3000 readMaybe+ socketPath <-+ env "SOCKET" "" pure+ socketMode <-+ env "SOCKET_MODE" 0o660 parseMode net <- env "NET" "bch" pure discover <-@@ -343,6 +355,19 @@ <> helpEnv "PORT" "REST API listening port" <> showDefault <> value c.port+ socketPath <-+ strOption $+ metavar "PATH"+ <> long "socket"+ <> helpEnv "SOCKET" "Listen on a Unix socket at this path instead of host and port"+ <> value c.socketPath+ socketMode <-+ option (maybeReader parseMode) $+ metavar "MODE"+ <> long "socket-mode"+ <> helpEnv "SOCKET_MODE" "Permissions of the Unix socket, in octal"+ <> showDefaultWith (\m -> '0' : showOct m "")+ <> value c.socketMode net <- strOption $ metavar netNames@@ -853,32 +878,43 @@ else Nothing, logger = logger }- run_web logger stats store =- runWeb- WebConfig- { host = cfg.host,- port = cfg.port,- store = store,- limits = cfg.webLimits,- maxPendingTxs = cfg.maxPendingTxs,- maxLaggingBlocks = cfg.maxLaggingBlocks,- minPeers = cfg.minPeers,- version = haskoinStoreVersion,- stats = stats,- tickerRefresh = cfg.tickerRefresh,- tickerURL = cfg.tickerURL,- priceHistoryURL = cfg.priceHistoryURL,- noBlockchainInfo = cfg.noBlockchainInfo,- healthCheckInterval = cfg.healthCheckInterval,- accessLog = cfg.accessLog,- nodeRpc = if null cfg.nodeRpc then Nothing else Just cfg.nodeRpc,- tls = case (cfg.tlsCert, cfg.tlsKey) of- ("", "") -> Nothing- ("", _) -> error "--tls-key needs --tls-cert"- (_, "") -> error "--tls-cert needs --tls-key"- (certificate, key) -> Just (certificate, key),- logger = withSource "Web" logger- }+ run_web logger stats store = do+ activated <- systemdSocket+ case activated of+ Just sock -> do+ logInfo logger "Web server on the socket that systemd passed"+ runWebSocket conf sock+ Nothing+ | null cfg.socketPath -> runWeb conf+ | otherwise -> withUnixSocket cfg.socketPath cfg.socketMode $ \sock -> do+ logInfo logger $ "Web server on the Unix socket " <> T.pack cfg.socketPath+ runWebSocket conf sock+ where+ conf =+ WebConfig+ { host = cfg.host,+ port = cfg.port,+ store = store,+ limits = cfg.webLimits,+ maxPendingTxs = cfg.maxPendingTxs,+ maxLaggingBlocks = cfg.maxLaggingBlocks,+ minPeers = cfg.minPeers,+ version = haskoinStoreVersion,+ stats = stats,+ tickerRefresh = cfg.tickerRefresh,+ tickerURL = cfg.tickerURL,+ priceHistoryURL = cfg.priceHistoryURL,+ noBlockchainInfo = cfg.noBlockchainInfo,+ healthCheckInterval = cfg.healthCheckInterval,+ accessLog = cfg.accessLog,+ nodeRpc = if null cfg.nodeRpc then Nothing else Just cfg.nodeRpc,+ tls = case (cfg.tlsCert, cfg.tlsKey) of+ ("", "") -> Nothing+ ("", _) -> error "--tls-key needs --tls-cert"+ (_, "") -> error "--tls-cert needs --tls-key"+ (certificate, key) -> Just (certificate, key),+ logger = withSource "Web" logger+ } with_stats logger net go | cfg.statsd = do logInfo (withSource "Main" logger) $
haskoin-store.cabal view
@@ -5,7 +5,7 @@ -- see: https://github.com/sol/hpack name: haskoin-store-version: 1.16.2+version: 1.16.3 synopsis: Storage and index for Bitcoin and Bitcoin Cash description: An indexer for Bitcoin and Bitcoin Cash. It downloads blocks and mempool transactions from the peer-to-peer network, indexes them in RocksDB, and@@ -80,6 +80,7 @@ Haskoin.Store.Web.Congestion Haskoin.Store.Web.Docs Haskoin.Store.Web.Events+ Haskoin.Store.Web.Listen Haskoin.Store.Web.Pieces Haskoin.Store.Web.Publish Haskoin.Store.Web.Router@@ -118,6 +119,7 @@ , containers ==0.7.* , data-default >=0.8.0.2 && <0.9 , deepseq >=1.5.1.0 && <1.6+ , directory >=1.3.10.1 && <1.4 , file-embed >=0.0.16.0 && <0.1 , hashable >=1.5.1.0 && <1.6 , hashtables >=1.4.2 && <1.5@@ -141,6 +143,7 @@ , time ==1.14.* , time-manager >=0.4.2 && <0.5 , transformers >=0.6.3.0 && <0.7+ , unix >=2.8.8.0 && <2.9 , unordered-containers >=0.2.21 && <0.3 , vault >=0.3.2.0 && <0.4 , wai >=3.2.5 && <3.3@@ -211,6 +214,7 @@ , time ==1.14.* , time-manager >=0.4.2 && <0.5 , transformers >=0.6.3.0 && <0.7+ , unix >=2.8.8.0 && <2.9 , unordered-containers >=0.2.21 && <0.3 , vault >=0.3.2.0 && <0.4 , wai >=3.2.5 && <3.3@@ -276,6 +280,7 @@ , time ==1.14.* , time-manager >=0.4.2 && <0.5 , transformers >=0.6.3.0 && <0.7+ , unix >=2.8.8.0 && <2.9 , unordered-containers >=0.2.21 && <0.3 , vault >=0.3.2.0 && <0.4 , wai >=3.2.5 && <3.3@@ -377,6 +382,7 @@ , time ==1.14.* , time-manager >=0.4.2 && <0.5 , transformers >=0.6.3.0 && <0.7+ , unix >=2.8.8.0 && <2.9 , unordered-containers >=0.2.21 && <0.3 , vault >=0.3.2.0 && <0.4 , wai >=3.2.5 && <3.3@@ -447,6 +453,7 @@ , time ==1.14.* , time-manager >=0.4.2 && <0.5 , transformers >=0.6.3.0 && <0.7+ , unix >=2.8.8.0 && <2.9 , unordered-containers >=0.2.21 && <0.3 , vault >=0.3.2.0 && <0.4 , vector >=0.13.2.0 && <0.14@@ -515,6 +522,7 @@ , time ==1.14.* , time-manager >=0.4.2 && <0.5 , transformers >=0.6.3.0 && <0.7+ , unix >=2.8.8.0 && <2.9 , unordered-containers >=0.2.21 && <0.3 , vault >=0.3.2.0 && <0.4 , wai >=3.2.5 && <3.3
src/Haskoin/Store/Web.hs view
@@ -322,7 +322,7 @@ statusIsClientError, statusIsServerError, )-import Network.Socket (Socket)+import Network.Socket (SockAddr (SockAddrUnix), Socket) import Network.Wai ( Application, Middleware,@@ -752,7 +752,9 @@ -- | Like 'runWeb', on a socket that already listens; the host and port of the -- configuration are ignored. Binding port 0 and passing the socket gives a -- free port with no window in which another program can take it, which--- finding a free port first and binding it later has.+-- finding a free port first and binding it later has. The socket may also+-- be a Unix socket, made by the store or passed by systemd+-- ("Haskoin.Store.Web.Listen"). runWebSocket :: WebConfig -> Socket -> IO () runWebSocket config sock = serveWeb config $ case config.tls of Nothing -> runSettingsSocket defaultSettings sock@@ -4307,7 +4309,8 @@ -- It is off by default: a public server has a proxy in front that keeps -- such a log, and knows the address of the client, where the server sees -- that of the proxy unless the proxy passes it on in @X-Forwarded-For@.--- It is for a server that is run without one.+-- It is for a server that is run without one. A client on a Unix socket+-- ("Haskoin.Store.Web.Listen") has no address and shows as @unix@. logIt :: Logger -> Bool -> Word32 -> Middleware logIt logger access slow app req respond = do start <- getPOSIXTime@@ -4324,7 +4327,10 @@ millis = cs (showFFloat (Just 1) (realToFrac took * 1000 :: Double) "") client = case lookup "X-Forwarded-For" (requestHeaders req) of Just given -> T.strip (T.takeWhile (/= ',') (T.decodeUtf8Lenient given))- Nothing -> cs (show (remoteHost req))+ Nothing -> case remoteHost req of+ -- a client on a Unix socket has no address+ SockAddrUnix _ -> "unix"+ a -> cs (show a) if | statusIsServerError s && s /= status503 -> logError logger msg | access -> logInfo logger (client <> " " <> msg <> ", " <> millis <> " ms")
+ src/Haskoin/Store/Web/Listen.hs view
@@ -0,0 +1,152 @@+{-# LANGUAGE LambdaCase #-}++-- |+-- Module : Haskoin.Store.Web.Listen+-- Copyright : (c) Haskoin Developers+-- License : MIT+-- Maintainer : jprupp@protonmail.ch+-- Stability : experimental+-- Portability : POSIX+--+-- Where the web server listens when it is not a TCP port: a Unix socket,+-- made here or handed over by systemd.+--+-- A store behind a proxy on the same machine needs no TCP port at all. A+-- Unix socket is a file, so who may connect is set by its owner and mode,+-- not by firewall rules or the address it is bound to, and nothing on the+-- network can reach it by mistake. It also skips the TCP stack, which a+-- proxy on the same machine gains nothing from.+--+-- There are two ways to get one, and the store takes either:+--+-- * __Socket activation__ ('systemdSocket'). A systemd @.socket@ unit+-- creates the socket, owns its path and permissions, and passes it to the+-- service as file descriptor 3, with @LISTEN_PID@ and @LISTEN_FDS@ in its+-- environment. This is the better way under systemd: the socket exists+-- before the store starts, so clients that connect during a restart wait+-- instead of failing, and the unit file says who may use it.+--+-- * __A path__ ('withUnixSocket'). The store creates the socket itself, for+-- machines without systemd, containers and tests.+--+-- A socket from systemd wins over a path, and a path over a TCP port, so a+-- unit file that adds a @.socket@ unit needs no change to the store's own+-- options.+--+-- Over a Unix socket a client has no address: the access log shows @unix@+-- for it, unless a proxy names the client in @X-Forwarded-For@. Nothing+-- else in the store looks at the client's address; the congestion control+-- of "Haskoin.Store.Web.Admission" goes by how busy the cores are, not by+-- who asks.+module Haskoin.Store.Web.Listen+ ( systemdSocket,+ withUnixSocket,+ parseMode,+ )+where++import Control.Exception (bracket, bracketOnError, throwIO, try)+import Control.Monad (unless, when)+import Foreign.C.String (peekCAString)+import Network.Socket+ ( Family (AF_UNIX),+ SockAddr (SockAddrUnix),+ Socket,+ SocketType (Stream),+ bind,+ close,+ connect,+ defaultProtocol,+ listen,+ maxListenQueue,+ mkSocket,+ setNonBlockIfNeeded,+ socket,+ )+import Numeric (readOct)+import System.Directory (removeFile)+import System.Environment (lookupEnv, unsetEnv)+import System.IO.Error (isDoesNotExistError)+import System.Posix.Files (getFileStatus, isSocket, setFileMode)+import System.Posix.Internals (withFilePath)+import System.Posix.Process (getProcessID)+import System.Posix.Types (FileMode)+import Text.Read (readMaybe)++-- | The first socket that systemd passed to this process, if it passed any.+--+-- systemd sets @LISTEN_PID@ to the process that the sockets are for, and+-- @LISTEN_FDS@ to how many there are, starting at file descriptor 3. The+-- variables are checked against this process, so that a child that+-- inherited them does not take the sockets, and are then removed, so that+-- no child of this one does. Only the first socket is used; a unit with+-- several @ListenStream@ lines should list one.+systemdSocket :: IO (Maybe Socket)+systemdSocket = do+ pid <- lookupEnv "LISTEN_PID"+ fds <- lookupEnv "LISTEN_FDS"+ me <- getProcessID+ case (readMaybe =<< pid, readMaybe =<< fds) of+ (Just p, Just n)+ | p == toInteger me && n >= (1 :: Int) -> do+ mapM_ unsetEnv ["LISTEN_PID", "LISTEN_FDS", "LISTEN_FDNAMES"]+ -- systemd leaves the descriptor blocking; GHC's I/O manager needs+ -- it not to be, or an accept would block a whole capability+ setNonBlockIfNeeded 3+ Just <$> mkSocket 3+ _ -> return Nothing++-- | Run an action with a Unix socket that listens at a path, and remove the+-- path when the action ends.+--+-- A socket file left at the path by a store that did not stop cleanly is+-- removed first, but only if nothing answers on it: a second store started+-- by mistake on the same path fails instead of taking the socket from the+-- first. A file at the path that is not a socket is never removed.+--+-- The mode is set after the socket is bound and before it listens, so there+-- is no moment in which a client could connect with the wrong permissions.+--+-- The path may be at most 107 bytes long, a limit of the system; a relative+-- path is taken from the working directory, and is one way around it.+withUnixSocket :: FilePath -> FileMode -> (Socket -> IO a) -> IO a+withUnixSocket path mode act = do+ -- the kernel keeps the path in 108 bytes, ending in a zero; the length is+ -- that of the path's bytes in the encoding of the file system+ pathBytes <- withFilePath path (fmap length . peekCAString)+ when (pathBytes > 107) . throwIO . userError $+ "The path of the Unix socket is longer than 107 bytes, which the system allows: "+ <> path+ removeStale path+ bracket open (\s -> close s >> removeFile path) act+ where+ open = bracketOnError (socket AF_UNIX Stream defaultProtocol) close $ \s -> do+ bind s (SockAddrUnix path)+ setFileMode path mode+ listen s maxListenQueue+ return s++-- | Remove a socket file that nothing listens on.+removeStale :: FilePath -> IO ()+removeStale path =+ try (getFileStatus path) >>= \case+ Left e+ | isDoesNotExistError e -> return ()+ | otherwise -> throwIO e+ Right st -> do+ unless (isSocket st) $+ throwIO . userError $+ path <> " exists and is not a socket"+ live <- bracket (socket AF_UNIX Stream defaultProtocol) close $ \s ->+ either (const False) (const True)+ <$> (try (connect s (SockAddrUnix path)) :: IO (Either IOError ()))+ when live $+ throwIO . userError $+ path <> " is in use by another server"+ removeFile path++-- | Read a file mode written in octal, such as @0660@ or @660@.+parseMode :: String -> Maybe FileMode+parseMode s = case readOct s of+ [(m, "")] | m <= 0o7777 -> Just m+ _ -> Nothing