monad-logger-logstash (empty) → 0.1.0.0

raw patch · 5 files changed

+405/−0 lines, 5 filesdep +aesondep +basedep +logstash

Dependencies added: aeson, base, logstash, monad-logger, retry, stm, stm-chans, text, transformers, unliftio

Files

• ChangeLog.md +5/−0
• LICENSE +21/−0
• README.md +175/−0
• monad-logger-logstash.cabal +49/−0
• src/Control/Monad/Logger/Logstash.hs +155/−0

@@ -0,0 +1,5 @@+# Changelog for logstash++## v0.1++- First release with support for connecting to `tcp` (incl. TLS) Logstash inputs and the `json_lines` codec.

@@ -0,0 +1,21 @@+MIT License++Copyright (c) 2020 Michael B. Gale++Permission is hereby granted, free of charge, to any person obtaining a copy+of this software and associated documentation files (the "Software"), to deal+in the Software without restriction, including without limitation the rights+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell+copies of the Software, and to permit persons to whom the Software is+furnished to do so, subject to the following conditions:++The above copyright notice and this permission notice shall be included in all+copies or substantial portions of the Software.++THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE+SOFTWARE.

@@ -0,0 +1,175 @@+# Haskell client library for Logstash++![MIT](https://img.shields.io/github/license/mbg/logstash)+![CI](https://github.com/mbg/logstash/workflows/Build/badge.svg?branch=master)+![stackage-nightly](https://github.com/mbg/logstash/workflows/stackage-nightly/badge.svg)++This library implements a client for Logstash in Haskell. The following features are currently supported:++- Connections to Logstash via TCP or TLS (`tcp` input type).+- Support for the `json_lines` codec out of the box and custom codecs can be implemented (arbitrary `ByteString` data can be sent across the connections).+- This library can either be used without any logging framework or as a backend for [`monad-logger`](http://hackage.haskell.org/package/monad-logger/).+- Log messages can either be written synchronously or asynchronously.++For example, to connect to a Logstash server via TCP at `127.0.0.1:5000` (configuration given by `def`) and send a JSON document synchronously with a timeout of 1s and the default retry policy from [`Control.Retry`](https://hackage.haskell.org/package/retry/docs/Control-Retry.html):++```haskell+data Doc = Doc String++instance ToJSON Doc where+    toJSON (Doc msg) = object [ "message" .= msg ]++main :: IO ()+main = runLogstashConn (logstashTcp def) retryPolicyDefault 1000000 $+    stashJsonLine (Doc "Hello World")+```++Only the `tcp` input type (with or without TLS) is currently supported. For example, without TLS, the Logstash input configuration should roughly be:++```conf+input {+    tcp {+        port => 5000+        codec => "json_lines"+    }+}+```++With TLS, the expected Logstash configuration should roughly be:++```conf+input {+    tcp {+        port => 5000+        ssl_cert => "/usr/share/logstash/tls/cert.pem"+        ssl_key => "/usr/share/logstash/tls/key.pem"+        ssl_key_passphrase => "foobar"+        ssl_enable => true +        ssl_verify => false+        codec => "json_lines"+    }+}+```++## Configuring connections++Connections to Logstash are represented by the `LogstashConnection` type. To connect to Logstash via `tcp` use the `Logstash.TCP` module which exports four principal functions. Note that none of these functions establish any connections when they are called - instead, they allow `runLogstashConn` and `runLogstashPool` to establish connections/reuse them as needed:++- `logstashTcp` which, given a hostname and a port, will produce an `Acquire` that can be used with `runLogstashConn`.+- `logstashTcpPool` which, given a hostname and a port, will produce a `Pool` that can be used with `runLogstashPool`.+- `logstashTls` which, given a hostname, a port, and TLS client parameters, will produce an `Acquire` that can be used with `runLogstashConn`.+- `logstashTlsPool` which, given a hostname, a port, and TLS client parameters, will produce a `Pool` that can be used with `runLogstashPool`.++For `logstashTls` and `logstashTlsPool`, TLS `ClientParams` are required. It is worth noting that the `defaultParamsClient` function in the `tls` package does **not** set any supported ciphers and does **not** load the system trust store by default. For relatively sane defaults, it is worth using `newDefaultClientParams` from [`network-simple-tls`](http://hackage.haskell.org/package/network-simple-tls/) instead. For example:++```haskell+main :: IO ()+main = do +    params <- newDefaultClientParams ("127.0.0.1", "")++    runLogstashConn (logstashTls def params) retryPolicyDefault 1000000 $ +        stashJsonLine myDocument+```++## Logging things++The `Logstash` module exports functions for synchronous and asynchronous logging. Synchronous logging is acceptable for applications or parts of applications that are largely single-threaded where blocking on writes to Logstash is not an issue. For multi-threaded applications, such as web applications or services, you may wish to write log messages to Logstash asynchronously instead. In the latter model, log messages are added to a bounded queue which is processed asynchronously by worker threads. ++### Synchronously++The logging functions exported by the `Logstash` module are backend-independent can be invoked synchronously with `runLogstash`, which is overloaded to work with either `Acquire LogstashConnection` or `LogstashPool` (`Pool LogstashConnection`) values and maps to one of the two implementations described below. In either case, you must supply a retry policy and a timeout (in microseconds). The retry policy determines whether performing the logging action should be re-attempted if an exception occurs. The order of operations is:++1. The retry policy is applied.+2. A connection is established using the provided Logstash context.+3. The timeout is applied.+4. The Logstash computation is executed.++If the computation is successful, each step will only be executed once. If an exception is raised by the computation or the timeout, the connection to the Logstash server is terminated and the exception propagated to the retry policy. If the retry policy determines that the computation should be re-attempted, steps 2-4 will happen again. The timeout applies to every attempt individually and should be chosen appropriately in conjunction with the retry policy in mind.++Depending on whether the Logstash context is a `Acquire LogstashConnection` value or a `LogstashPool` (`Pool LogstashConnection`) value, the `runLogstash` functions maps to one of:++- `runLogstashConn` for `Acquire LogstashConnection` (e.g. the result of `logstashTcp` or `logstashTls`).+- `runLogstashPool` for `Pool LogstashConnection` (e.g. the result of `logstashTcpPool` or `logstashTlsPool`). If a connection is available in the pool, that connection will be used. If no connection is available but there is an empty space in the pool, a new connection will be established. If neither is true, this function blocks until a connection is available. The computation that is provided as the second argument is then run with the connection. In the event of an exception, the connection is not returned to the pool.++#### Stashing things by hand++The following functions allow sending data synchronously via the Logstash connection:++- `stash` is a general-purpose function for sending `ByteString` data to the server. No further processing is performed on the data.+- `stashJsonLine` is for use with the `json_line` codec. The argument is encoded as JSON and a `\n` character is appended, which is then sent to the server. ++Any exception raised by the above `stash`ing functions will likely be due to a bad connection. The `runLogstash` functions apply the retry policy before establishing a connection, so in the event that an exception is raised, a new connection will be established for the next attempt.++### Asynchronously++The `withLogstashQueue` function is used for asynchronous logging. When called, it sets up a bounded queue that is then used to communicate log messages to worker threads which dispatch them to Logstash. A minimal example with default settings is shown below:++```haskell+data Doc = Doc String++instance ToJSON Doc where+    toJSON (Doc msg) = object [ "message" .= msg ]++main :: IO ()+main = do+    let ctx = logstashTcp def+    let cfg = defaultLogstashQueueCfg ctx+    +    withLogstashQueue cfg (const stashJsonLine) [] $ \queue -> do+        atomically $ writeTBMQueue queue (Doc "Hello World")+```++The `[]` given to `withLogstashQueue` allows installing exception handlers that are used to handle the case where a log message has exhausted the retry policy. This can e.g. be used to fall back to the standard output for logging as a last resort and to stop the worker thread from getting terminated by an exception that may be recoverable.++The queue is automatically closed when the inner computation returns. The worker threads will continue running until the queue is empty and then terminate. `withLogstashQueue` will not return until all worker threads have returned.++### Usage with `monad-logger`++The `monad-logger-logstash` package provides convenience functions and types for working with [`monad-logger`](http://hackage.haskell.org/package/monad-logger/). ++#### Synchronous logging++The following example demonstrates how to use the `runLogstashLoggerT` function with a TCP connection to Logstash, the default retry policy from [`Control.Retry`](https://hackage.haskell.org/package/retry/docs/Control-Retry.html), a 1s timeout for each attempt, and the `json_lines` codec:++```haskell+main :: IO ()+main = do +    let ctx = logstashTcp def+    runLogstashLoggerT ctx retryPolicyDefault 1000000 (const stashJsonLine) $ +        logInfoN "Hello World"+```++Each call to a logging function such as `logInfoN` in the example will result in the log message being written to Logstash synchronously. ++#### Asynchronous logging++The `withLogstashLoggerT` function is the analogue of `withLogstashQueue` for `monad-logger`. It performs the same setup as `withLogstashQueue`, but automatically adds all log messages from logging functions to the queue. A minimal example with default settings is:++```haskell+main :: IO ()+main = do +    let ctx = logstashTcp def+    withLogstashLoggerT (defaultLogstashQueueCfg ctx) (const stashJsonLine) [] $ +        logInfoN "Hello World"+```++### Usage with `katip`++The `katip-logstash` package provides convenience functions and types for working with [`katip`](http://hackage.haskell.org/package/katip/). ++#### Asynchronous logging++The `withLogstashScribe` function is the analogue of `withLogstashQueue` for `katip`. It performs the same setup as `withLogstashQueue`, but provides a `Scribe` instead of the raw queue. A minimal example with default settings is (adapted from the `katip` documentation):++```haskell+main :: IO ()+main = do +    let ctx = logstashTcp def+    withLogstashScribe (defaultLogstashQueueCfg ctx) (const $ pure True) (itemJson V3) (const stashJsonLine) [] $ \logstashScribe -> do+        let makeLogEnv = registerScribe "logstash" logstashScribe defaultScribeSettings =<< initLogEnv "MyApp" "production"+        bracket makeLogEnv closeScribes $ \le -> do+            let initialContext = ()+            let initialNamespace = "main"+            runKatipContextT le initialContext initialNamespace $ do+                $(logTM) InfoS "Hello World"+```

@@ -0,0 +1,49 @@+cabal-version: 1.12++-- This file has been generated from package.yaml by hpack version 0.33.0.+--+-- see: https://github.com/sol/hpack+--+-- hash: bb6eb2cc0b9c0d5e471d1f67b2ea8090faedc57f0ec5d4876da0475551b756f4++name:           monad-logger-logstash+version:        0.1.0.0+synopsis:       Logstash backend for monad-logger.+description:    Please see the README on GitHub at <https://github.com/mbg/logstash#readme>+category:       System+homepage:       https://github.com/mbg/logstash#readme+bug-reports:    https://github.com/mbg/logstash/issues+author:         Michael B. Gale+maintainer:     m.gale@warwick.ac.uk+copyright:      Copyright (c) 2020 Michael B. Gale+license:        MIT+license-file:   LICENSE+build-type:     Simple+extra-source-files:+    README.md+    ChangeLog.md++source-repository head+  type: git+  location: https://github.com/mbg/logstash++library+  exposed-modules:+      Control.Monad.Logger.Logstash+  other-modules:+      Paths_monad_logger_logstash+  hs-source-dirs:+      src+  default-extensions: RecordWildCards OverloadedStrings+  build-depends:+      aeson+    , base >=4.7 && <5+    , logstash+    , monad-logger+    , retry+    , stm+    , stm-chans+    , text+    , transformers+    , unliftio+  default-language: Haskell2010

@@ -0,0 +1,155 @@+--------------------------------------------------------------------------------+-- Logstash backend for monad-logger                                          --+--------------------------------------------------------------------------------+-- This source code is licensed under the MIT license found in the LICENSE    --+-- file in the root directory of this source tree.                            --+--------------------------------------------------------------------------------++-- | This module implements `runLogstashLoggerT` which can be+-- used to write log messages that arise in a `LoggingT` computation to a+-- given `LogstashContext`. The following example demonstrates how to use the +-- `runLogstashLoggerT` function with a TCP connection to Logstash, the+-- default retry policy from `Control.Retry`, a 1s timeout for each attempt,+-- and the @json_lines@ codec:+-- +-- > main :: IO ()+-- > main = do +-- >    let ctx = logstashTcp def+-- >    runLogstashLoggerT ctx retryPolicyDefault 1000000 (const stashJsonLine) $ +-- >         logInfoN "Hello World"+--+-- Assuming a suitable Logstash server that can receive this message,+-- something like the following JSON document should be indexed (see the +-- documentation for `Control.Monad.Logger` for information about how to include+-- more information in log messages):+--+-- > { +-- >    "@version":"1",+-- >    "message":"Hello World",+-- >    "log.origin.file.line":0,+-- >    "log.origin.file.module":"<unknown>",+-- >    "log.origin.file.package":"<unknown>",+-- >    "log.origin.file.start.column":0,+-- >    "log.origin.file.start.line":0,+-- >    "log.origin.file.end.column":0,+-- >    "log.origin.file.end.line":0,+-- >    "log.origin.file.name":"<unknown>",+-- >    "log.logger":"",+-- >    "log.level":"info"+-- > }+--+-- If an error or a timeout occurs while writing to the Logstash connection,+-- the retry policy determines whether and when sending the message is+-- attempted again. If all attempts fail, the most recent exception is thrown+-- to the caller.+module Control.Monad.Logger.Logstash (+    runLogstashLoggerT,+    stashJsonLine,++    withLogstashLoggerT,++    -- * Re-exports+    LogstashContext(..)+) where ++--------------------------------------------------------------------------------++import Control.Concurrent+import Control.Concurrent.STM+import Control.Concurrent.STM.TBMQueue+import Control.Exception (Handler)+import Control.Monad+import Control.Monad.Logger+import Control.Monad.Trans.Reader+import Control.Retry++import Data.Aeson+import Data.Maybe+import Data.Text (Text)+import Data.Text.Encoding (decodeUtf8)++import UnliftIO (MonadUnliftIO)++import Logstash hiding (stashJsonLine)+import qualified Logstash as L (stashJsonLine)++--------------------------------------------------------------------------------++-- | `runLogstashLoggerT` @context retryPolicy time codec logger@ runs a +-- `LoggingT` computation which writes all log entries to the Logstash +-- @context@ using the given @codec@. The @retryPolicy@ determines whether +-- and how the handler should deal with failures that arise. Each attempt+-- that is made by the @retryPolicy@ will have a timeout of @time@+-- microseconds applied to it. +runLogstashLoggerT +    :: LogstashContext ctx +    => ctx +    -> RetryPolicyM IO+    -> Integer+    -> ( RetryStatus -> +         (Loc, LogSource, LogLevel, LogStr) -> +         ReaderT LogstashConnection IO ()+       )+    -> LoggingT m a +    -> m a+runLogstashLoggerT ctx policy time codec log = runLoggingT log $ +    \logLoc logSource logLevel logStr -> runLogstash ctx policy time $ +    \s -> codec s (logLoc, logSource, logLevel, logStr)++-- | `withLogstashLoggerT` @cfg codec exceptionHandlers logger@ is like+-- `withLogstashQueue` except for `LoggingT` computations so that log messages+-- are automatically added to the queue.+withLogstashLoggerT+    :: (LogstashContext ctx, MonadUnliftIO m)+    => LogstashQueueCfg ctx+    -> ( RetryStatus -> +         (Loc, LogSource, LogLevel, LogStr) ->+         ReaderT LogstashConnection IO ()+       )+    -> [(Loc, LogSource, LogLevel, LogStr) -> Handler ()]+    -> LoggingT m a+    -> m a+withLogstashLoggerT cfg dispatch hs log = withLogstashQueue cfg dispatch hs $ +    \queue -> runLoggingT log $+    \logLoc logSource logLevel logStr -> atomically $ +        writeTBMQueue queue (logLoc, logSource, logLevel, logStr)++--------------------------------------------------------------------------------++-- | `stashJsonLine` @entry@ serialises @entry@ as JSON using reasonable+-- defaults for Elasticsearch based on +-- https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html+-- and sends the result to Logstash using the @json_lines@ codec.+stashJsonLine :: (Loc, LogSource, LogLevel, LogStr) +              -> ReaderT LogstashConnection IO ()+stashJsonLine (logEntryLoc, logEntrySource, logEntryLevel, logEntryMessage) = +    L.stashJsonLine $ object +    [ "message" .= decodeUtf8 (fromLogStr logEntryMessage)+    , "log" .= object +        [ "logger" .= logEntrySource+        , "level" .= jsonLogLevel logEntryLevel+        , "origin" .= object +            [ "file" .= object +                [ "name" .= loc_filename logEntryLoc +                , "line" .= fst (loc_start logEntryLoc)+                -- the following fields are not part of the ECS+                , "package" .= loc_package logEntryLoc+                , "module" .= loc_module logEntryLoc+                , "start" .= jsonCharPos (loc_start logEntryLoc)+                , "end" .= jsonCharPos (loc_end logEntryLoc)+                ] +            ]+        ]+    ]+    where jsonLogLevel :: LogLevel -> Text+          jsonLogLevel LevelDebug = "debug"+          jsonLogLevel LevelInfo = "info"+          jsonLogLevel LevelWarn = "warn"+          jsonLogLevel LevelError = "error"+          jsonLogLevel (LevelOther x) = x++          jsonCharPos :: (Int, Int) -> Value+          jsonCharPos (line, column) =+              object [ "line" .= line, "column" .= column ]++--------------------------------------------------------------------------------