katip-logstash (empty) → 0.1.0.0

raw patch · 5 files changed

+304/−0 lines, 5 filesdep +aesondep +basedep +katip

Dependencies added: aeson, base, katip, logstash, retry, stm, stm-chans, text, transformers, unliftio

Files

• ChangeLog.md +5/−0
• LICENSE +21/−0
• README.md +175/−0
• katip-logstash.cabal +49/−0
• src/Katip/Scribes/Logstash.hs +54/−0

@@ -0,0 +1,5 @@+# Changelog for logstash++## v0.1++- First release with support for connecting to `tcp` (incl. TLS) Logstash inputs and the `json_lines` codec.

@@ -0,0 +1,21 @@+MIT License++Copyright (c) 2020 Michael B. Gale++Permission is hereby granted, free of charge, to any person obtaining a copy+of this software and associated documentation files (the "Software"), to deal+in the Software without restriction, including without limitation the rights+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell+copies of the Software, and to permit persons to whom the Software is+furnished to do so, subject to the following conditions:++The above copyright notice and this permission notice shall be included in all+copies or substantial portions of the Software.++THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE+SOFTWARE.

@@ -0,0 +1,175 @@+# Haskell client library for Logstash++![MIT](https://img.shields.io/github/license/mbg/logstash)+![CI](https://github.com/mbg/logstash/workflows/Build/badge.svg?branch=master)+![stackage-nightly](https://github.com/mbg/logstash/workflows/stackage-nightly/badge.svg)++This library implements a client for Logstash in Haskell. The following features are currently supported:++- Connections to Logstash via TCP or TLS (`tcp` input type).+- Support for the `json_lines` codec out of the box and custom codecs can be implemented (arbitrary `ByteString` data can be sent across the connections).+- This library can either be used without any logging framework or as a backend for [`monad-logger`](http://hackage.haskell.org/package/monad-logger/).+- Log messages can either be written synchronously or asynchronously.++For example, to connect to a Logstash server via TCP at `127.0.0.1:5000` (configuration given by `def`) and send a JSON document synchronously with a timeout of 1s and the default retry policy from [`Control.Retry`](https://hackage.haskell.org/package/retry/docs/Control-Retry.html):++```haskell+data Doc = Doc String++instance ToJSON Doc where+    toJSON (Doc msg) = object [ "message" .= msg ]++main :: IO ()+main = runLogstashConn (logstashTcp def) retryPolicyDefault 1000000 $+    stashJsonLine (Doc "Hello World")+```++Only the `tcp` input type (with or without TLS) is currently supported. For example, without TLS, the Logstash input configuration should roughly be:++```conf+input {+    tcp {+        port => 5000+        codec => "json_lines"+    }+}+```++With TLS, the expected Logstash configuration should roughly be:++```conf+input {+    tcp {+        port => 5000+        ssl_cert => "/usr/share/logstash/tls/cert.pem"+        ssl_key => "/usr/share/logstash/tls/key.pem"+        ssl_key_passphrase => "foobar"+        ssl_enable => true +        ssl_verify => false+        codec => "json_lines"+    }+}+```++## Configuring connections++Connections to Logstash are represented by the `LogstashConnection` type. To connect to Logstash via `tcp` use the `Logstash.TCP` module which exports four principal functions. Note that none of these functions establish any connections when they are called - instead, they allow `runLogstashConn` and `runLogstashPool` to establish connections/reuse them as needed:++- `logstashTcp` which, given a hostname and a port, will produce an `Acquire` that can be used with `runLogstashConn`.+- `logstashTcpPool` which, given a hostname and a port, will produce a `Pool` that can be used with `runLogstashPool`.+- `logstashTls` which, given a hostname, a port, and TLS client parameters, will produce an `Acquire` that can be used with `runLogstashConn`.+- `logstashTlsPool` which, given a hostname, a port, and TLS client parameters, will produce a `Pool` that can be used with `runLogstashPool`.++For `logstashTls` and `logstashTlsPool`, TLS `ClientParams` are required. It is worth noting that the `defaultParamsClient` function in the `tls` package does **not** set any supported ciphers and does **not** load the system trust store by default. For relatively sane defaults, it is worth using `newDefaultClientParams` from [`network-simple-tls`](http://hackage.haskell.org/package/network-simple-tls/) instead. For example:++```haskell+main :: IO ()+main = do +    params <- newDefaultClientParams ("127.0.0.1", "")++    runLogstashConn (logstashTls def params) retryPolicyDefault 1000000 $ +        stashJsonLine myDocument+```++## Logging things++The `Logstash` module exports functions for synchronous and asynchronous logging. Synchronous logging is acceptable for applications or parts of applications that are largely single-threaded where blocking on writes to Logstash is not an issue. For multi-threaded applications, such as web applications or services, you may wish to write log messages to Logstash asynchronously instead. In the latter model, log messages are added to a bounded queue which is processed asynchronously by worker threads. ++### Synchronously++The logging functions exported by the `Logstash` module are backend-independent can be invoked synchronously with `runLogstash`, which is overloaded to work with either `Acquire LogstashConnection` or `LogstashPool` (`Pool LogstashConnection`) values and maps to one of the two implementations described below. In either case, you must supply a retry policy and a timeout (in microseconds). The retry policy determines whether performing the logging action should be re-attempted if an exception occurs. The order of operations is:++1. The retry policy is applied.+2. A connection is established using the provided Logstash context.+3. The timeout is applied.+4. The Logstash computation is executed.++If the computation is successful, each step will only be executed once. If an exception is raised by the computation or the timeout, the connection to the Logstash server is terminated and the exception propagated to the retry policy. If the retry policy determines that the computation should be re-attempted, steps 2-4 will happen again. The timeout applies to every attempt individually and should be chosen appropriately in conjunction with the retry policy in mind.++Depending on whether the Logstash context is a `Acquire LogstashConnection` value or a `LogstashPool` (`Pool LogstashConnection`) value, the `runLogstash` functions maps to one of:++- `runLogstashConn` for `Acquire LogstashConnection` (e.g. the result of `logstashTcp` or `logstashTls`).+- `runLogstashPool` for `Pool LogstashConnection` (e.g. the result of `logstashTcpPool` or `logstashTlsPool`). If a connection is available in the pool, that connection will be used. If no connection is available but there is an empty space in the pool, a new connection will be established. If neither is true, this function blocks until a connection is available. The computation that is provided as the second argument is then run with the connection. In the event of an exception, the connection is not returned to the pool.++#### Stashing things by hand++The following functions allow sending data synchronously via the Logstash connection:++- `stash` is a general-purpose function for sending `ByteString` data to the server. No further processing is performed on the data.+- `stashJsonLine` is for use with the `json_line` codec. The argument is encoded as JSON and a `\n` character is appended, which is then sent to the server. ++Any exception raised by the above `stash`ing functions will likely be due to a bad connection. The `runLogstash` functions apply the retry policy before establishing a connection, so in the event that an exception is raised, a new connection will be established for the next attempt.++### Asynchronously++The `withLogstashQueue` function is used for asynchronous logging. When called, it sets up a bounded queue that is then used to communicate log messages to worker threads which dispatch them to Logstash. A minimal example with default settings is shown below:++```haskell+data Doc = Doc String++instance ToJSON Doc where+    toJSON (Doc msg) = object [ "message" .= msg ]++main :: IO ()+main = do+    let ctx = logstashTcp def+    let cfg = defaultLogstashQueueCfg ctx+    +    withLogstashQueue cfg (const stashJsonLine) [] $ \queue -> do+        atomically $ writeTBMQueue queue (Doc "Hello World")+```++The `[]` given to `withLogstashQueue` allows installing exception handlers that are used to handle the case where a log message has exhausted the retry policy. This can e.g. be used to fall back to the standard output for logging as a last resort and to stop the worker thread from getting terminated by an exception that may be recoverable.++The queue is automatically closed when the inner computation returns. The worker threads will continue running until the queue is empty and then terminate. `withLogstashQueue` will not return until all worker threads have returned.++### Usage with `monad-logger`++The `monad-logger-logstash` package provides convenience functions and types for working with [`monad-logger`](http://hackage.haskell.org/package/monad-logger/). ++#### Synchronous logging++The following example demonstrates how to use the `runLogstashLoggerT` function with a TCP connection to Logstash, the default retry policy from [`Control.Retry`](https://hackage.haskell.org/package/retry/docs/Control-Retry.html), a 1s timeout for each attempt, and the `json_lines` codec:++```haskell+main :: IO ()+main = do +    let ctx = logstashTcp def+    runLogstashLoggerT ctx retryPolicyDefault 1000000 (const stashJsonLine) $ +        logInfoN "Hello World"+```++Each call to a logging function such as `logInfoN` in the example will result in the log message being written to Logstash synchronously. ++#### Asynchronous logging++The `withLogstashLoggerT` function is the analogue of `withLogstashQueue` for `monad-logger`. It performs the same setup as `withLogstashQueue`, but automatically adds all log messages from logging functions to the queue. A minimal example with default settings is:++```haskell+main :: IO ()+main = do +    let ctx = logstashTcp def+    withLogstashLoggerT (defaultLogstashQueueCfg ctx) (const stashJsonLine) [] $ +        logInfoN "Hello World"+```++### Usage with `katip`++The `katip-logstash` package provides convenience functions and types for working with [`katip`](http://hackage.haskell.org/package/katip/). ++#### Asynchronous logging++The `withLogstashScribe` function is the analogue of `withLogstashQueue` for `katip`. It performs the same setup as `withLogstashQueue`, but provides a `Scribe` instead of the raw queue. A minimal example with default settings is (adapted from the `katip` documentation):++```haskell+main :: IO ()+main = do +    let ctx = logstashTcp def+    withLogstashScribe (defaultLogstashQueueCfg ctx) (const $ pure True) (itemJson V3) (const stashJsonLine) [] $ \logstashScribe -> do+        let makeLogEnv = registerScribe "logstash" logstashScribe defaultScribeSettings =<< initLogEnv "MyApp" "production"+        bracket makeLogEnv closeScribes $ \le -> do+            let initialContext = ()+            let initialNamespace = "main"+            runKatipContextT le initialContext initialNamespace $ do+                $(logTM) InfoS "Hello World"+```

@@ -0,0 +1,49 @@+cabal-version: 1.12++-- This file has been generated from package.yaml by hpack version 0.33.0.+--+-- see: https://github.com/sol/hpack+--+-- hash: 76f4ba122c69b7c3a30388131525e5bde7fc8f73d66194b90cc0b6de83216703++name:           katip-logstash+version:        0.1.0.0+synopsis:       Logstash backend for katip.+description:    Please see the README on GitHub at <https://github.com/mbg/logstash#readme>+category:       Logging+homepage:       https://github.com/mbg/logstash#readme+bug-reports:    https://github.com/mbg/logstash/issues+author:         Michael B. Gale+maintainer:     m.gale@warwick.ac.uk+copyright:      Copyright (c) 2020 Michael B. Gale+license:        MIT+license-file:   LICENSE+build-type:     Simple+extra-source-files:+    README.md+    ChangeLog.md++source-repository head+  type: git+  location: https://github.com/mbg/logstash++library+  exposed-modules:+      Katip.Scribes.Logstash+  other-modules:+      Paths_katip_logstash+  hs-source-dirs:+      src+  default-extensions: RecordWildCards OverloadedStrings+  build-depends:+      aeson+    , base >=4.7 && <5+    , katip+    , logstash+    , retry+    , stm+    , stm-chans+    , text+    , transformers+    , unliftio+  default-language: Haskell2010

@@ -0,0 +1,54 @@+--------------------------------------------------------------------------------+-- Logstash backend for katip                                                 --+--------------------------------------------------------------------------------+-- This source code is licensed under the MIT license found in the LICENSE    --+-- file in the root directory of this source tree.                            --+--------------------------------------------------------------------------------++{-# LANGUAGE RankNTypes #-}++-- | This module implements convenience functions for using the `Logstash`+-- module with the katip logging framework.+module Katip.Scribes.Logstash (+    withLogstashScribe+) where ++--------------------------------------------------------------------------------++import Control.Concurrent.STM+import Control.Concurrent.STM.TBMQueue+import Control.Exception+import Control.Monad.Trans.Reader+import Control.Retry++import UnliftIO (MonadUnliftIO)+import Katip+import Logstash++--------------------------------------------------------------------------------++-- | `withLogstashScribe` @cfg permitFunc toItem codec handlers cont@ is like+-- `withLogstashQueue` except that a `Scribe` is passed to @cont@ instead of+-- the raw queue. The @toItem@ function is required to convert `Item` values+-- to whatever input @codec@ expects. A reasonable default might be `toObject`+-- or a wrapper around the `LogItem` type class. +withLogstashScribe +    :: (LogstashContext ctx, MonadUnliftIO m)+    => LogstashQueueCfg ctx -- ^ The configuration for the Logstash client.+    -> PermitFunc -- ^ A function which determines which entries to include.+    -> (forall a. LogItem a => Item a -> item) +    -> (RetryStatus -> item -> ReaderT LogstashConnection IO ())+    -> [item -> Handler ()] -- ^ Exception handlers.+    -> (Scribe -> m r) -- ^ A continuation which the `Scribe` is given to.+    -> m r+withLogstashScribe cfg permit f codec hs cont = +    withLogstashQueue cfg codec hs $ \queue -> cont $ Scribe{+        liPush = atomically . writeTBMQueue queue . f,+        -- we set the finaliser to close the queue, even though this handled+        -- by `withLogstashQueue` already since it is possible to close a+        -- given `Scribe` long before returning from `cont`.+        scribeFinalizer = atomically $ closeTBMQueue queue,+        scribePermitItem = permit+    }++--------------------------------------------------------------------------------