hotel-california (empty) → 0.0.1.0

raw patch · 11 files changed

+627/−0 lines, 11 filesdep +basedep +bytestringdep +hotel-californiasetup-changed

Dependencies added: base, bytestring, hotel-california, hs-opentelemetry-api, hs-opentelemetry-exporter-otlp, hs-opentelemetry-propagator-w3c, hs-opentelemetry-sdk, hs-opentelemetry-utils-exceptions, hs-opentelemetry-vendor-honeycomb, http-types, optparse-applicative, text, time, typed-process, unliftio

Files

• CHANGELOG.md +15/−0
• LICENSE +30/−0
• README.md +124/−0
• Setup.hs +2/−0
• app/Main.hs +54/−0
• hotel-california.cabal +214/−0
• src/HotelCalifornia.hs +3/−0
• src/HotelCalifornia/Exec.hs +55/−0
• src/HotelCalifornia/Tracing.hs +79/−0
• src/HotelCalifornia/Tracing/TraceParent.hs +49/−0
• test/Spec.hs +2/−0

@@ -0,0 +1,15 @@+# Changelog for `hotel-california`++All notable changes to this project will be documented in this file.++The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),+and this project adheres to the+[Haskell Package Versioning Policy](https://pvp.haskell.org/).++## Unreleased++## 0.0.1.0 - 2023-09-12++- Initial Release+- Introduce `exec` functionality+

@@ -0,0 +1,30 @@+Copyright Author name here (c) 2023++All rights reserved.++Redistribution and use in source and binary forms, with or without+modification, are permitted provided that the following conditions are met:++    * Redistributions of source code must retain the above copyright+      notice, this list of conditions and the following disclaimer.++    * Redistributions in binary form must reproduce the above+      copyright notice, this list of conditions and the following+      disclaimer in the documentation and/or other materials provided+      with the distribution.++    * Neither the name of Author name here nor the names of other+      contributors may be used to endorse or promote products derived+      from this software without specific prior written permission.++THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

@@ -0,0 +1,124 @@+# hotel-california++`hotel-california` is inspired by [`Trace`](https://github.com/Pondidum/Trace/) and [`otel-cli`](https://github.com/equinix-labs/otel-cli), a pair of utilities for tracing shell scripts.+We needed something like this in order to instrument local builds, so we could understand how much time people are spending waiting on builds, tests, etc.++# Usage++The binary name is `hotel`.+Currently, the only supported command is `exec`++```sh+$ hotel exec --help+Usage: hotel exec [-s|--span-name SPAN_NAME] SCRIPT [SCRIPT...]++  Execute the given command with tracing enabled++Available options:+  -h,--help                Show this help text+  -s,--span-name SPAN_NAME The name of the span that the program reports. By+                           default, this is the script you pass in.+  SCRIPT                   The command to run, along with any arguments. Best to+                           use -- before providing the script, otherwise it may+                           pass arguments to `hotel` instead of to your script+```++Currently, the program only looks in environment variables for configuration.++- `OTEL_EXPORTER_OTLP_ENDPOINT` (with a default to `localhost:4317`)+- `OTEL_SERVICE_NAME`+- `OTEL_EXPORTER_OTLP_HEADERS`+- `OTEL_RESOURCE_ATTRIBUTES`+- and probably others, see+  [`hs-opentelemetry-sdk`](https://hackage.haskell.org/package/hs-opentelemetry-sdk)+  for more information++# Background/FAQ++## Lol what's up with the name++Well `otel-cli` is a great name for the tool.+But this is a Haskell implementation, so I need an `h` in there somewhere.+`hotel-cli` sounds good.+But wait... `cli` ... what else could I do with that?+Ahah!++Sorry++## Difference from `Trace`++The `Trace` tool has the following basic workflow:++```sh+$ TRACE_PARENT=$(trace start "build")+$ make build+$ trace finish+```++When you call `trace start "trace-name"`, it generates a `TraceId` and `SpanId` and records that to a file in the temporary directory.+The filename carries the trace ID and span ID.+The file contains the name of the trace, the start time, and any other metadata you provide.++When you call `trace finish`, the tool looks for the `TRACE_PARENT` environment variable.+It then looks in the `$TMP/traces/state` directory for a file that matches the `TRACE_PARENT`.+It loads the file, creates a `Span` with the timestamp given in the file, and then calls `span.End`.+The tool then makes a network request to report this data.++### The Problems++#### Performance++The tool allows you to start groups and run commands that will make individual spans, allowing you to understand the overall trace.+`trace group start` is similar to `trace start` - it creates a new Span ID, attaches it to the parent span, and writes that to the temporary directory.+`trace task` does something a bit more idiomatic - it creates a `Span`, runs the command you provide, and then does `Span.end`.+`trace group finish` is similar to `trace finish` - it loads the parent span information for the group, creates a `Span`, and calls `span.End` with the timestamps loaded from the group.++The toy example I did is here:++```sh+$ TRACE_PARENT=$(trace start "why"); \+  GROUP=$(trace group start "why-1"); \+  trace task "$GROUP" -- go build; \+  trace group finish "$GROUP"; \+  trace finish+```++According to Honeycomb, this spends 170ms doing `go build`, and then incurs another 650ms to complete the entire process - the `why-1` group takes roughly 370ms extra, and then the final `trace finish` call adds another 300ms.++This performance hit may not be substantial for the apparent intention of the tool - instrumenting CI builds - but it is going to be a problem for *my* intention with the tool - instrumenting local developer workflows.+300ms is significant, but not terrible if incurred once.+However, incurring that for each step we want to record?+That's a problem.++The out-of-the-box solution is to use an OpenTelemetry collector that is local to the machine, and can report the spans periodically, in the background.+This is an extra deployment step, so it'd be nice to avoid that, if possible.++#### Nesting++The API requires you to work in this manner:++```sh+$ TRACE_PARENT=$(trace start "my-trace")+$ OUTER_GROUP=$(trace group start "neat")+$ INNER_GROUP=$(trace group start "neat" "$OUTER_GROUP")+$ trace task "$INNER_GROUP" -- make build+$ trace group finish "$INNER_GROUP"+$ trace group finish "$OUTER_GROUP"+$ trace finish+```++So any time you do `trace start`, you create an identifier for a parent span.+But any time you do `trace group finish`, you look up the relevant parent span+ID and then actually *create* a *child span*.++This makes it difficult to create a span, and just "know" if you're in a root or not.+You would need this in order to provide a composable interface: shell scripts calling other shell scripts which can all record spans.++## Difference from `otel-cli`++Well, `otel-cli` solves most of the above problems.+The main entry point is `otel-cli exec`, which runs a command for you, and reports a span for it.+You can nest `otel-cli exec` calls arbitrarily, which works nicely.+However, it too had some issues, with the most challenging being a bug around signals.+I simply couldn't figure out the behavior around signals in Golang, and all available internet advice wasn't exactly helpful.+I decided then to spike out this tool.

@@ -0,0 +1,2 @@+import Distribution.Simple+main = defaultMain

@@ -0,0 +1,54 @@+module Main where++import Data.Version (showVersion)+import Paths_hotel_california (version)+import HotelCalifornia.Exec+import HotelCalifornia.Tracing (withGlobalTracing)+import Options.Applicative++data Command = Command+    { commandGlobalOptions :: GlobalOptions+    , commandSubCommand :: SubCommand+    }++data GlobalOptions = GlobalOptions++data SubCommand+    = Exec ExecArgs++optionsParser :: ParserInfo Command+optionsParser =+    info' parser' "Welcome to `hotel-california`"+  where+  -- thanks danidiaz for the blog post+    info' :: Parser a -> String -> ParserInfo a+    info' p desc = info+        (helper <*> p)+        (fullDesc <> progDesc desc)++    parser' :: Parser Command+    parser' =+        Command+            <$> generalOptionsParser+            <*> subCommandParser+            <**> simpleVersioner (showVersion version)++    generalOptionsParser =+        pure GlobalOptions++    subCommandParser :: Parser SubCommand+    subCommandParser =+        subparser $ foldMap command'+                    [ ("exec", "Execute the given command with tracing enabled", Exec <$> parseExecArgs)+                    ]++    command' (cmdName,desc,parser) =+        command cmdName (info' parser desc)++main :: IO ()+main = do+    withGlobalTracing do+        Command {..} <- execParser optionsParser+        case commandSubCommand of+            Exec execArgs ->+                runExecArgs execArgs

@@ -0,0 +1,214 @@+cabal-version: 1.12++-- This file has been generated from package.yaml by hpack version 0.35.2.+--+-- see: https://github.com/sol/hpack++name:           hotel-california+version:        0.0.1.0+description:    Please see the README on GitHub at <https://github.com/parsonsmatt/hotel-california#readme>+homepage:       https://github.com/parsonsmatt/hotel-california#readme+bug-reports:    https://github.com/parsonsmatt/hotel-california/issues+author:         Matt Parsons+maintainer:     parsonsmatt@gmail.com+copyright:      2023 Matt Parsons+license:        BSD3+license-file:   LICENSE+build-type:     Simple+extra-source-files:+    README.md+    CHANGELOG.md++source-repository head+  type: git+  location: https://github.com/parsonsmatt/hotel-california++library+  exposed-modules:+      HotelCalifornia+      HotelCalifornia.Exec+      HotelCalifornia.Tracing+      HotelCalifornia.Tracing.TraceParent+  other-modules:+      Paths_hotel_california+  hs-source-dirs:+      src+  default-extensions:+      BlockArguments+      DataKinds+      DefaultSignatures+      DeriveFunctor+      DeriveGeneric+      DeriveLift+      DerivingStrategies+      DerivingVia+      FlexibleContexts+      FlexibleInstances+      GADTs+      GeneralizedNewtypeDeriving+      ImportQualifiedPost+      InstanceSigs+      LambdaCase+      MultiParamTypeClasses+      MultiWayIf+      NamedFieldPuns+      NegativeLiterals+      NumericUnderscores+      OverloadedLabels+      OverloadedStrings+      PartialTypeSignatures+      PatternSynonyms+      RankNTypes+      RecordWildCards+      ScopedTypeVariables+      StandaloneDeriving+      TypeApplications+      TypeFamilies+      UndecidableInstances+      ViewPatterns+      OverloadedRecordDot+      TypeOperators+      StrictData+      ApplicativeDo+  ghc-options: -Wall -Wcompat -Widentities -Wincomplete-record-updates -Wincomplete-uni-patterns -Wmissing-home-modules -Wpartial-fields -Wredundant-constraints -threaded+  build-depends:+      base >=4.7 && <5+    , bytestring+    , hs-opentelemetry-api >=0.1.0.0+    , hs-opentelemetry-exporter-otlp+    , hs-opentelemetry-propagator-w3c+    , hs-opentelemetry-sdk >=0.0.3.6+    , hs-opentelemetry-utils-exceptions+    , hs-opentelemetry-vendor-honeycomb+    , http-types+    , optparse-applicative+    , text+    , time+    , typed-process+    , unliftio+  default-language: Haskell2010++executable hotel+  main-is: Main.hs+  other-modules:+      Paths_hotel_california+  hs-source-dirs:+      app+  default-extensions:+      BlockArguments+      DataKinds+      DefaultSignatures+      DeriveFunctor+      DeriveGeneric+      DeriveLift+      DerivingStrategies+      DerivingVia+      FlexibleContexts+      FlexibleInstances+      GADTs+      GeneralizedNewtypeDeriving+      ImportQualifiedPost+      InstanceSigs+      LambdaCase+      MultiParamTypeClasses+      MultiWayIf+      NamedFieldPuns+      NegativeLiterals+      NumericUnderscores+      OverloadedLabels+      OverloadedStrings+      PartialTypeSignatures+      PatternSynonyms+      RankNTypes+      RecordWildCards+      ScopedTypeVariables+      StandaloneDeriving+      TypeApplications+      TypeFamilies+      UndecidableInstances+      ViewPatterns+      OverloadedRecordDot+      TypeOperators+      StrictData+      ApplicativeDo+  ghc-options: -Wall -Wcompat -Widentities -Wincomplete-record-updates -Wincomplete-uni-patterns -Wmissing-home-modules -Wpartial-fields -Wredundant-constraints -threaded -threaded -rtsopts -with-rtsopts=-N+  build-depends:+      base >=4.7 && <5+    , bytestring+    , hotel-california+    , hs-opentelemetry-api >=0.1.0.0+    , hs-opentelemetry-exporter-otlp+    , hs-opentelemetry-propagator-w3c+    , hs-opentelemetry-sdk >=0.0.3.6+    , hs-opentelemetry-utils-exceptions+    , hs-opentelemetry-vendor-honeycomb+    , http-types+    , optparse-applicative+    , text+    , time+    , typed-process+    , unliftio+  default-language: Haskell2010++test-suite hotel-california-test+  type: exitcode-stdio-1.0+  main-is: Spec.hs+  other-modules:+      Paths_hotel_california+  hs-source-dirs:+      test+  default-extensions:+      BlockArguments+      DataKinds+      DefaultSignatures+      DeriveFunctor+      DeriveGeneric+      DeriveLift+      DerivingStrategies+      DerivingVia+      FlexibleContexts+      FlexibleInstances+      GADTs+      GeneralizedNewtypeDeriving+      ImportQualifiedPost+      InstanceSigs+      LambdaCase+      MultiParamTypeClasses+      MultiWayIf+      NamedFieldPuns+      NegativeLiterals+      NumericUnderscores+      OverloadedLabels+      OverloadedStrings+      PartialTypeSignatures+      PatternSynonyms+      RankNTypes+      RecordWildCards+      ScopedTypeVariables+      StandaloneDeriving+      TypeApplications+      TypeFamilies+      UndecidableInstances+      ViewPatterns+      OverloadedRecordDot+      TypeOperators+      StrictData+      ApplicativeDo+  ghc-options: -Wall -Wcompat -Widentities -Wincomplete-record-updates -Wincomplete-uni-patterns -Wmissing-home-modules -Wpartial-fields -Wredundant-constraints -threaded -threaded -rtsopts -with-rtsopts=-N+  build-depends:+      base >=4.7 && <5+    , bytestring+    , hotel-california+    , hs-opentelemetry-api >=0.1.0.0+    , hs-opentelemetry-exporter-otlp+    , hs-opentelemetry-propagator-w3c+    , hs-opentelemetry-sdk >=0.0.3.6+    , hs-opentelemetry-utils-exceptions+    , hs-opentelemetry-vendor-honeycomb+    , http-types+    , optparse-applicative+    , text+    , time+    , typed-process+    , unliftio+  default-language: Haskell2010

@@ -0,0 +1,3 @@+-- | The 'HotelCalifornia' library is intended to provide an easy entry+-- point into OpenTelemetry tracing.+module HotelCalifornia where

@@ -0,0 +1,55 @@+-- | This module defines the code for actually executing a command with tracing+-- enabled.+module HotelCalifornia.Exec where++import Data.List.NonEmpty (NonEmpty(..))+import qualified Data.List.NonEmpty as NEL+import Data.Maybe (fromMaybe)+import Data.Text (Text)+import qualified Data.Text as Text+import HotelCalifornia.Tracing+import HotelCalifornia.Tracing.TraceParent+import System.Environment (getEnvironment)+import Options.Applicative+import System.Exit+import System.Process.Typed++data ExecArgs = ExecArgs+    { execArgsScript :: NonEmpty String+    , execArgsSpanName :: Maybe Text+    }++parseExecArgs :: Parser ExecArgs+parseExecArgs = do+    execArgsSpanName <- optional do+        option str $ mconcat+            [ metavar "SPAN_NAME"+            , long "span-name"+            , short 's'+            , help "The name of the span that the program reports. By default, this is the script you pass in."+            ]+    execArgsScript1 <- argument str (metavar "SCRIPT" <> help "The command to run, along with any arguments. Best to use -- before providing the script, otherwise it may pass arguments to `hotel` instead of to your script")+    execArgsScriptRest <- many $ argument str (metavar "SCRIPT...")+    pure ExecArgs+        { execArgsScript = execArgsScript1 :| execArgsScriptRest+        , ..+        }++runExecArgs :: ExecArgs -> IO ()+runExecArgs ExecArgs {..} = do+    let script =+            unwords $ NEL.toList execArgsScript+        spanName =+            fromMaybe (Text.pack script) execArgsSpanName++    inSpan' spanName \span_ -> do+        newEnv <- spanContextToEnvironment span_+        fullEnv <- mappend newEnv <$> getEnvironment++        let processConfig = shell $ unwords $ NEL.toList execArgsScript+        exitCode <- runProcess $ setEnv fullEnv $ processConfig+        case exitCode of+            ExitSuccess ->+                pure ()+            _ ->+                exitWith exitCode

@@ -0,0 +1,79 @@+module HotelCalifornia.Tracing+    ( module HotelCalifornia.Tracing+    , defaultSpanArguments+    ) where++import Control.Monad+import qualified Data.ByteString.Char8 as BS8+import Data.Text (Text)+import Data.Time+import HotelCalifornia.Tracing.TraceParent+import OpenTelemetry.Context as Context hiding (lookup)+import OpenTelemetry.Context.ThreadLocal (attachContext)+import OpenTelemetry.Trace hiding+       ( SpanKind(..)+       , SpanStatus(..)+       , addAttribute+       , addAttributes+       , createSpan+       , inSpan+       , inSpan'+       , inSpan''+       )+import qualified OpenTelemetry.Trace as Trace+import qualified OpenTelemetry.Vendor.Honeycomb as Honeycomb+import UnliftIO++-- | Initialize the global tracing provider for the application and run an action+--   (that action is generally the entry point of the application), cleaning+--   up the provider afterwards.+--+--   This also sets up an empty context (creating a new trace ID).+withGlobalTracing :: MonadUnliftIO m => m a -> m a+withGlobalTracing act = do+    void $ attachContext Context.empty+    liftIO setParentSpanFromEnvironment+    bracket initializeTracing shutdownTracerProvider $ \_ -> do+        -- note: this is not in a span since we don't have a root span yet so it+        -- would not wind up in the trace in a helpful way anyway+        void $+          Honeycomb.getOrInitializeHoneycombTargetInContext initializationTimeout+            `catch` \(e :: SomeException) -> do+              -- we are too early in initialization to be able to use a normal logger,+              -- but this needs to get out somehow.+              --+              -- honeycomb links are not load-bearing, so we let them just not come+              -- up if the API fails.+              liftIO . BS8.hPutStrLn stderr $ "error setting up Honeycomb trace links: " <> (BS8.pack $ displayException e)+              pure Nothing++        act+  where+    initializationTimeout = secondsToNominalDiffTime 3++initializeTracing :: MonadUnliftIO m => m TracerProvider+initializeTracing = do+  (processors, tracerOptions') <- liftIO getTracerProviderInitializationOptions+  provider <- createTracerProvider processors tracerOptions'+  setGlobalTracerProvider provider+  pure provider++globalTracer :: MonadIO m => m Tracer+globalTracer = getGlobalTracerProvider >>= \tp -> pure $ makeTracer tp "hotel-california" tracerOptions++inSpan' :: (MonadUnliftIO m) => Text -> (Span -> m a) -> m a+inSpan' spanName =+    inSpanWith' spanName defaultSpanArguments++inSpanWith :: (MonadUnliftIO m) => Text -> SpanArguments -> m a -> m a+inSpanWith spanName args action =+    inSpanWith' spanName args \_ -> action++inSpanWith' :: (MonadUnliftIO m) => Text -> SpanArguments -> (Span -> m a) -> m a+inSpanWith' spanName args action = do+    tr <- globalTracer+    Trace.inSpan'' tr spanName args action++inSpan :: (MonadUnliftIO m) => Text -> m a -> m a+inSpan spanName =+    inSpanWith spanName defaultSpanArguments

@@ -0,0 +1,49 @@+-- | This module defines the type of 'TraceParent' which can be parsed+module HotelCalifornia.Tracing.TraceParent where++import Data.Foldable (for_)+import qualified Data.ByteString as BS+import qualified Data.ByteString.Char8 as BS8+import qualified Data.Text.Encoding as TE+import qualified Data.Text as Text+import OpenTelemetry.Propagator.W3CTraceContext+import OpenTelemetry.Trace.Core (SpanContext, isRemote, wrapSpanContext, Span)+import System.Environment+import OpenTelemetry.Context.ThreadLocal+import qualified OpenTelemetry.Context as Ctxt++-- | This function looks up the @TRACEPARENT@ and @TRACECONTEXT@ environment+-- variables and returns a @'Maybe' 'SpanContext'@ constructed from them.+spanContextFromEnvironment :: IO (Maybe SpanContext)+spanContextFromEnvironment = do+    mtraceParent <- lookupEnvBS "TRACEPARENT"+    mtraceContext <- lookupEnvBS "TRACESTATE"+    pure $ decodeSpanContext mtraceParent mtraceContext+  where+    lookupEnvBS :: String -> IO (Maybe BS.ByteString)+    lookupEnvBS str = fmap (TE.encodeUtf8 . Text.pack) <$> lookupEnv str++-- | This function takes the given 'Span' and converts it into a list of+-- environment variables consisting of:+--+-- @+-- [ ( "TRACEPARENT", traceParent)+-- , ( "TRACESTATE", traceState)+-- ]+-- @+spanContextToEnvironment :: Span -> IO [(String, String)]+spanContextToEnvironment spanContext = do+    (traceParent, traceState) <- encodeSpanContext spanContext+    pure+        [ ("TRACEPARENT", BS8.unpack traceParent)+        , ("TRACESTATE", BS8.unpack traceState)+        ]+++-- | This function should be called after you've initialized and attached the+-- thread local 'Context'.+setParentSpanFromEnvironment :: IO ()+setParentSpanFromEnvironment = do+    mspanContext <- spanContextFromEnvironment+    for_ mspanContext \spanContext -> do+        adjustContext $ Ctxt.insertSpan (wrapSpanContext (spanContext {isRemote = True}))

@@ -0,0 +1,2 @@+main :: IO ()+main = putStrLn "Test suite not yet implemented"