aws-kinesis-client 0.4.0.1 → 0.4.0.2
raw patch · 7 files changed
+721/−267 lines, 7 filesdep +aws-configuration-toolsdep +configuration-toolsdep ~aws-kinesis-clientdep ~lens-actionbuild-type:Customsetup-changed
Dependencies added: aws-configuration-tools, configuration-tools
Dependency ranges changed: aws-kinesis-client, lens-action
Files
- CHANGELOG.md +14/−0
- Setup.hs +406/−1
- aws-kinesis-client.cabal +7/−5
- cli/CLI.hs +32/−41
- cli/CLI/Config.hs +244/−0
- cli/CLI/Options.hs +0/−220
- example_config.yml +18/−0
CHANGELOG.md view
@@ -1,3 +1,17 @@+### v0.4.0.2++This release contains (only) breaking changes to the Consumer CLI.++We have switched to using the+[configuration-tools](http://hackage.haskell.org/package/configuration-tools)+package for passing options to the CLI. This makes configuration much more+robust, and adds support for using a YAML-formatted configuration file, as well+as granularly overriding options in a cascading manner (through multiple+configuration files, or by passing command line arguments).++See `example_config.yml` for an example configuration file. Run `kinesis-cli+--help` to see supported command-line arguments.+ ### v0.4.0.1 This release fixes a bug in the Consumer where a saved state could get
Setup.hs view
@@ -1,2 +1,407 @@+-- ------------------------------------------------------ --+-- Copyright © 2014 AlephCloud Systems, Inc.+-- ------------------------------------------------------ --++{-# LANGUAGE CPP #-}+{-# LANGUAGE OverloadedStrings #-}+{-# LANGUAGE ScopedTypeVariables #-}++{-# OPTIONS_HADDOCK show-extensions #-}++-- | This module contains a @Setup.hs@ script that hooks into the cabal build+-- process at the end of the configuration phase and generates a module with+-- package information for each component of the cabal package.+--+-- The modules are created in the /autogen/ build directory where also the+-- @Path_@ module is created by cabal's simple build setup. This is usually the+-- directory @.\/dist\/build\/autogen@.+--+-- For a library component the module is named just @PkgInfo@. For all other+-- components the module is named @PkgInfo_COMPONENT_NAME@ where+-- @COMPONENT_NAME@ is the name of the component with @-@ characters replaced by+-- @_@.+--+-- For instance, if a cabal package contains a library and an executable that is+-- called /my-app/, the following modules are created: @PkgInfo@ and+-- @PkgInfo_my_app@.+--+--+-- = Usage as Setup Script+--+-- There are three ways how this module can be used:+--+-- 1. Copy the code of this module into a file called @Setup.hs@ in the root+-- directory of your package.+--+-- 2. If the /configuration-tools/ package is already installed in the system+-- where the build is done, following code can be used as @Setup.hs@ script:+--+-- > module Main (main) where+-- >+-- > import Configuration.Utils.Setup+--+-- 3. For usage within a more complex @Setup.hs@ script you shall import this+-- module qualified and use the 'mkPkgInfoModules' function. For example:+--+-- > module Main (main) where+-- >+-- > import qualified Configuration.Utils.Setup as ConfTools+-- >+-- > main :: IO ()+-- > main = defaultMainWithHooks (ConfTools.mkPkgInfoModules simpleUserHooks)+-- >+--+-- With all methods the field @Build-Type@ in the package description (cabal) file+-- must be set to @Custom@:+--+-- > Build-Type: Custom+--+--+-- = Integration With "Configuration.Utils"+--+-- You can integrate the information provided by the @PkgInfo@ modules with the+-- command line interface of an application by importing the respective module+-- for the component and using the+-- 'Configuration.Utils.runWithPkgInfoConfiguration' function from the module+-- "Configuration.Utils" as show in the following example:+--+-- > {-# LANGUAGE OverloadedStrings #-}+-- > {-# LANGUAGE FlexibleInstances #-}+-- >+-- > module Main+-- > ( main+-- > ) where+-- >+-- > import Configuration.Utils+-- > import PkgInfo+-- >+-- > instance FromJSON (() -> ()) where parseJSON _ = pure id+-- >+-- > mainInfo :: ProgramInfo ()+-- > mainInfo = programInfo "Hello World" (pure id) ()+-- >+-- > main :: IO ()+-- > main = runWithPkgInfoConfiguration mainInfo pkgInfo . const $ putStrLn "hello world"+--+-- With that the resulting application supports the following additional command+-- line options:+--+-- [@--version@, @-v@]+-- prints the version of the application and exits.+--+-- [@--info@, @-i@]+-- prints a short info message for the application and exits.+--+-- [@--long-info@]+-- print a detailed info message for the application and exits.+-- Beside component name, package name, version, revision, and copyright+-- the message also contain information about the compiler that+-- was used for the build, the build architecture, build flags,+-- the author, the license type, and a list of all direct and+-- indirect dependencies along with their licenses and copyrights.+--+-- [@--license@]+-- prints the text of the lincense of the application and exits.+--+module Main+( main+, mkPkgInfoModules+) where++import Distribution.PackageDescription import Distribution.Simple-main = defaultMain+import Distribution.Simple.Setup+import qualified Distribution.InstalledPackageInfo as I+import Distribution.Simple.LocalBuildInfo+import Distribution.Simple.BuildPaths+import Distribution.Simple.PackageIndex+import Distribution.Text+import System.Process++import Control.Applicative+import Control.Monad++import qualified Data.ByteString as B+import Data.ByteString.Char8 (pack)+import Data.Char (isSpace)+import Data.List (intercalate)+import Data.Monoid++import Prelude hiding (readFile, writeFile)++import System.Directory (doesFileExist, doesDirectoryExist, createDirectoryIfMissing)+import System.Exit (ExitCode(ExitSuccess))++#ifndef MIN_VERSION_Cabal+#define NO_CABAL_MACROS 1+#define MIN_VERSION_Cabal(a,b,c) 0+#endif++#ifdef NO_CABAL_MACROS+import Data.Maybe+import Data.Typeable+#endif++-- | Include this function when your setup doesn't contain any+-- extra functionality.+--+main :: IO ()+main = defaultMainWithHooks (mkPkgInfoModules simpleUserHooks)++-- | Modifies the given record of hooks by adding functionality that+-- creates a package info module for each component of the cabal package.+--+-- This function is intended for usage in more complex @Setup.hs@ scripts.+-- If your setup doesn't contain any other function you can just import+-- the 'main' function from this module.+--+-- The modules are created in the /autogen/ build directory where also the+-- @Path_@ module is created by cabal's simple build setup. This is usually the+-- directory @.\/dist\/build\/autogen@.+--+-- For a library component the module is named just @PkgInfo@. For all other+-- components the module is named @PkgInfo_COMPONENT_NAME@ where+-- @COMPONENT_NAME@ is the name of the component with @-@ characters replaced by+-- @_@.+--+mkPkgInfoModules+ :: UserHooks+ -> UserHooks+mkPkgInfoModules hooks = hooks+ { postConf = mkPkgInfoModulesPostConf (postConf hooks)+ }++mkPkgInfoModulesPostConf+ :: (Args -> ConfigFlags -> PackageDescription -> LocalBuildInfo -> IO ())+ -> Args+ -> ConfigFlags+ -> PackageDescription+ -> LocalBuildInfo+ -> IO ()+mkPkgInfoModulesPostConf hook args flags pkgDesc bInfo = do+ mkModules+ hook args flags pkgDesc bInfo+ where+ mkModules = mapM_ (f . \(a,_,_) -> a) $ componentsConfigs bInfo+ f cname = case cname of+ CLibName -> updatePkgInfoModule Nothing pkgDesc bInfo+ CExeName s -> updatePkgInfoModule (Just s) pkgDesc bInfo+ CTestName s -> updatePkgInfoModule (Just s) pkgDesc bInfo+ CBenchName s -> updatePkgInfoModule (Just s) pkgDesc bInfo++pkgInfoModuleName :: Maybe String -> String+pkgInfoModuleName Nothing = "PkgInfo"+pkgInfoModuleName (Just cn) = "PkgInfo_" ++ map tr cn+ where+ tr '-' = '_'+ tr c = c++pkgInfoFileName :: Maybe String -> LocalBuildInfo -> FilePath+pkgInfoFileName cn bInfo = autogenModulesDir bInfo ++ "/" ++ pkgInfoModuleName cn ++ ".hs"++trim :: String -> String+trim = f . f+ where f = reverse . dropWhile isSpace++getVCS :: IO (Maybe RepoType)+getVCS =+ doesDirectoryExist ".hg" >>= \x0 -> if x0+ then return (Just Mercurial)+ else doesDirectoryExist ".git" >>= \x1 -> return $ if x1+ then Just Git+ else Nothing++flagNameStr :: FlagName -> String+flagNameStr (FlagName s) = s++pkgInfoModule :: Maybe String -> PackageDescription -> LocalBuildInfo -> IO B.ByteString+pkgInfoModule cName pkgDesc bInfo = do+ (tag, revision, branch) <- getVCS >>= \x -> case x of+ Just Mercurial -> hgInfo+ Just Git -> gitInfo+ _ -> noVcsInfo++ let vcsBranch = if branch == "default" || branch == "master" then "" else branch+ vcsVersion = intercalate "-" . filter (/= "") $ [tag, revision, vcsBranch]+ flags = map (flagNameStr . fst) . filter snd . configConfigurationsFlags . configFlags $ bInfo++ licenseString <- licenseFilesText pkgDesc++ return $ B.intercalate "\n"+ [ "{-# LANGUAGE OverloadedStrings #-}"+ , "{-# LANGUAGE RankNTypes #-}"+ , ""+ , "module " <> (pack . pkgInfoModuleName) cName <> " where"+ , ""+ , " import Data.String (IsString)"+ , " import Data.Monoid"+ , ""+ , " name :: IsString a => Maybe a"+ , " name = " <> maybe "Nothing" (\x -> "Just \"" <> pack x <> "\"") cName+ , ""+ , " tag :: IsString a => a"+ , " tag = \"" <> pack tag <> "\""+ , ""+ , " revision :: IsString a => a"+ , " revision = \"" <> pack revision <> "\""+ , ""+ , " branch :: IsString a => a"+ , " branch = \"" <> pack branch <> "\""+ , ""+ , " branch' :: IsString a => a"+ , " branch' = \"" <> pack vcsBranch <> "\""+ , ""+ , " vcsVersion :: IsString a => a"+ , " vcsVersion = \"" <> pack vcsVersion <> "\""+ , ""+ , " compiler :: IsString a => a"+ , " compiler = \"" <> (pack . display . compilerId . compiler) bInfo <> "\""+ , ""+ , " flags :: IsString a => [a]"+ , " flags = " <> (pack . show) flags+ , ""+ , " optimisation :: IsString a => a"+ , " optimisation = \"" <> (displayOptimisationLevel . withOptimization) bInfo <> "\""+ , ""+ , " arch :: IsString a => a"+ , " arch = \"" <> (pack . display . hostPlatform) bInfo <> "\""+ , ""+ , " license :: IsString a => a"+ , " license = \"" <> (pack . display . license) pkgDesc <> "\""+ , ""+ , " licenseText :: IsString a => a"+ , " licenseText = " <> (pack . show) licenseString+ , ""+ , " copyright :: IsString a => a"+ , " copyright = \"" <> (pack . copyright) pkgDesc <> "\""+ , ""+ , " author :: IsString a => a"+ , " author = \"" <> (pack . author) pkgDesc <> "\""+ , ""+ , " homepage :: IsString a => a"+ , " homepage = \"" <> (pack . homepage) pkgDesc <> "\""+ , ""+ , " package :: IsString a => a"+ , " package = \"" <> (pack . display . package) pkgDesc <> "\""+ , ""+ , " packageName :: IsString a => a"+ , " packageName = \"" <> (pack . display . packageName) pkgDesc <> "\""+ , ""+ , " packageVersion :: IsString a => a"+ , " packageVersion = \"" <> (pack . display . packageVersion) pkgDesc <> "\""+ , ""+ , " dependencies :: IsString a => [a]"+ , " dependencies = " <> (pack . show . map (display . packageId) . allPackages . installedPkgs) bInfo+ , ""+ , " dependenciesWithLicenses :: IsString a => [a]"+ , " dependenciesWithLicenses = " <> (pack . show . map pkgIdWithLicense . allPackages . installedPkgs) bInfo+ , ""+ , " versionString :: (Monoid a, IsString a) => a"+ , " versionString = case name of"+ , " Nothing -> package <> \" (revision \" <> vcsVersion <> \")\""+ , " Just n -> n <> \"-\" <> packageVersion <> \" (package \" <> package <> \" revision \" <> vcsVersion <> \")\""+ , ""+ , " info :: (Monoid a, IsString a) => a"+ , " info = versionString <> \"\\n\" <> copyright"+ , ""+ , " longInfo :: (Monoid a, IsString a) => a"+ , " longInfo = info <> \"\\n\\n\""+ , " <> \"Author: \" <> author <> \"\\n\""+ , " <> \"License: \" <> license <> \"\\n\""+ , " <> \"Homepage: \" <> homepage <> \"\\n\""+ , " <> \"Build with: \" <> compiler <> \" (\" <> arch <> \")\" <> \"\\n\""+ , " <> \"Build flags: \" <> mconcat (map (\\x -> \" \" <> x) flags) <> \"\\n\""+ , " <> \"Optimisation: \" <> optimisation <> \"\\n\\n\""+ , " <> \"Dependencies:\\n\" <> mconcat (map (\\x -> \" \" <> x <> \"\\n\") dependenciesWithLicenses)"+ , ""+ , " pkgInfo :: (Monoid a, IsString a) => (a, a, a, a)"+ , " pkgInfo ="+ , " ( info"+ , " , longInfo"+ , " , versionString"+ , " , licenseText"+ , " )"+ , ""+ ]+ where+ displayOptimisationLevel NoOptimisation = "none"+ displayOptimisationLevel NormalOptimisation = "normal"+ displayOptimisationLevel MaximumOptimisation = "maximum"++updatePkgInfoModule :: Maybe String -> PackageDescription -> LocalBuildInfo -> IO ()+updatePkgInfoModule cName pkgDesc bInfo = do+ createDirectoryIfMissing True $ autogenModulesDir bInfo+ newFile <- pkgInfoModule cName pkgDesc bInfo+ let update = B.writeFile fileName newFile+ doesFileExist fileName >>= \x -> if x+ then do+ oldRevisionFile <- B.readFile fileName+ when (oldRevisionFile /= newFile) update+ else+ update+ where+ fileName = pkgInfoFileName cName bInfo++licenseFilesText :: PackageDescription -> IO B.ByteString+licenseFilesText pkgDesc =+ B.intercalate "\n------------------------------------------------------------\n" <$> mapM fileText+#ifdef NO_CABAL_MACROS+ (getLicenseFiles pkgDesc)+#elif MIN_VERSION_Cabal(1,20,0)+ (licenseFiles pkgDesc)+#else+ [licenseFile pkgDesc]+#endif+ where+ fileText file = doesFileExist file >>= \x -> if x+ then B.readFile file+ else return ""++#ifdef NO_CABAL_MACROS+-- The name and the type of the @licenseFile :: String@ field of 'PackageDescription' changed+-- in Cabal version 1.20 to @licenseFiles :: [String]@. Both versions are used with GHC-7.8.+-- When compiling @Setup.hs@ the @MIN_VERSION_...@ macros are not available. This function is an+-- ugly hack to do conditional compilation without CPP.+--+-- It will break if the number of record fields changes. If that's the case we will probably+-- have to either+--+-- * Use typeable to pattern match on the number of constructor arguments for 'PackageDescription',+-- * use TH hacks ala @$( if versionBranch cabalVersion < [...] ... )@, and/or+-- * make guesses about the cabal version based on the @__GLASGOW_HASKELL__@ macro.+--+getLicenseFiles :: PackageDescription -> [FilePath]+getLicenseFiles (PackageDescription _ _ l _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) =+ fromMaybe (error "unsupport Cabal library version") $ cast l <|> (return <$> cast l)+#endif++hgInfo :: IO (String, String, String)+hgInfo = do+ tag <- trim <$> readProcess "hg" ["id", "-r", "max(ancestors(\".\") and tag())", "-t"] ""+ rev <- trim <$> readProcess "hg" ["id", "-i"] ""+ branch <- trim <$> readProcess "hg" ["id", "-b"] ""+ return (tag, rev, branch)++gitInfo :: IO (String, String, String)+gitInfo = do+ tag <- do+ (exitCode, out, _err) <- readProcessWithExitCode "git" ["describe", "--exact-match", "--tags", "--abbrev=0"] ""+ case exitCode of+ ExitSuccess -> return $ trim out+ _ -> return ""+ rev <- trim <$> readProcess "git" ["rev-parse", "--short", "HEAD"] ""+ branch <- trim <$> readProcess "git" ["rev-parse", "--abbrev-ref", "HEAD"] ""+ return (tag, rev, branch)++noVcsInfo :: IO (String, String, String)+noVcsInfo = return ("", "", "")++pkgIdWithLicense :: I.InstalledPackageInfo -> String+pkgIdWithLicense a = (display . packageId) a+ ++ " ["+ ++ (display . I.license) a+ ++ (if cr /= "" then ", " ++ cr else "")+ ++ "]"+ where+ cr = (unwords . words . I.copyright) a+
aws-kinesis-client.cabal view
@@ -1,5 +1,5 @@ name: aws-kinesis-client-version: 0.4.0.1+version: 0.4.0.2 synopsis: A producer & consumer client library for AWS Kinesis description: This package provides a Producer client for bulk-writing messages to a@@ -13,9 +13,9 @@ maintainer: jon@jonmsterling.com copyright: Copyright (c) 2013-2015 PivotCloud, Inc. category: Cloud-build-type: Simple+build-type: Custom cabal-version: >=1.10-extra-source-files: CHANGELOG.md+extra-source-files: CHANGELOG.md, example_config.yml source-repository head type: git@@ -77,15 +77,17 @@ default-language: Haskell2010 hs-source-dirs: cli main-is: CLI.hs- other-modules: CLI.Options+ other-modules: CLI.Config build-depends: base >=4.7 && <5.0, base-unicode-symbols, aeson >=0.8, aws >=0.10.5, aws-general >=0.2.1, aws-kinesis >=0.1.4,- aws-kinesis-client >=0.4.0.1,+ aws-kinesis-client >=0.4.0.2,+ aws-configuration-tools >=0.1.0.0, conduit >=1.2.3.1,+ configuration-tools >=0.2.12, http-conduit >=2.1.5, kan-extensions >=4.2, lens >=4.7,
cli/CLI.hs view
@@ -25,7 +25,6 @@ {-# LANGUAGE MultiParamTypeClasses #-} {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE RecordWildCards #-}-{-# LANGUAGE TemplateHaskell #-} {-# LANGUAGE TupleSections #-} {-# LANGUAGE UndecidableInstances #-} {-# LANGUAGE UnicodeSyntax #-}@@ -46,8 +45,11 @@ import Aws.Kinesis.Client.Common import Aws.Kinesis.Client.Consumer -import CLI.Options+import CLI.Config +import Configuration.Utils+import Configuration.Utils.Aws.Credentials+ import Control.Concurrent.Lifted import Control.Concurrent.Async.Lifted import Control.Exception.Lifted@@ -55,6 +57,7 @@ import Control.Monad import Control.Monad.Trans import Control.Monad.Trans.Control+import Control.Monad.Trans.Except import Control.Monad.Codensity import Control.Monad.Reader.Class import Control.Monad.Trans.Reader (ReaderT(..))@@ -69,12 +72,13 @@ import Data.Traversable import Data.Typeable -import Options.Applicative import qualified Network.HTTP.Conduit as HC import Prelude.Unicode import System.Exit import System.IO.Error +import PkgInfo_kinesis_cli+ data CLIError = MissingCredentials | NoInstanceMetadataCredentials@@ -83,38 +87,18 @@ instance Exception CLIError type MonadCLI m- = ( MonadReader CLIOptions m+ = ( MonadReader Config m , MonadIO m , MonadBaseControl IO m ) -fetchCredentials- ∷ MonadCLI m- ⇒ m Credentials-fetchCredentials = do- view clioUseInstanceMetadata ≫= \case- True →- loadCredentialsFromInstanceMetadata- ≫= maybe (throwIO NoInstanceMetadataCredentials) return- False →- view clioAccessKeys ≫= \case- Just (CredentialsFromAccessKeys aks) →- makeCredentials- (aks ^. akAccessKeyId)- (aks ^. akSecretAccessKey)- Just (CredentialsFromFile path) →- loadCredentialsFromFile path credentialsDefaultKey- ≫= maybe (throwIO MissingCredentials) return- Nothing →- throwIO MissingCredentials- managedKinesisKit ∷ MonadCLI m ⇒ Codensity m KinesisKit managedKinesisKit = do- CLIOptions{..} ← ask+ Config{..} ← ask manager ← managedHttpManager- credentials ← lift fetchCredentials+ credentials ← lift $ runExceptT (credentialsFromConfig _configCredentialConfig) ≫= either throwIO return return KinesisKit { _kkManager = manager , _kkConfiguration = Configuration@@ -122,7 +106,7 @@ , credentials = credentials , logger = defaultLog Warning }- , _kkKinesisConfiguration = defaultKinesisConfiguration _clioRegion+ , _kkKinesisConfiguration = defaultKinesisConfiguration _configRegion } cliConsumerKit@@ -130,9 +114,9 @@ ⇒ KinesisKit → m ConsumerKit cliConsumerKit kinesisKit = do- CLIOptions{..} ← ask+ Config{..} ← ask savedStreamState ←- case _clioStateIn of+ case _configStateIn of Just path → liftIO $ do code ← catch (Just <$> BL8.readFile path) $ \case exn@@ -142,16 +126,16 @@ A.eitherDecode <$> code Nothing → return Nothing return $- makeConsumerKit kinesisKit _clioStreamName+ makeConsumerKit kinesisKit _configStreamName & ckBatchSize .~ 100- & ckIteratorType .~ _clioIteratorType+ & ckIteratorType .~ _configIteratorType & ckSavedStreamState .~ savedStreamState app ∷ MonadCLI m ⇒ Codensity m ExitCode app = do- CLIOptions{..} ← ask+ Config{..} ← ask consumer ← managedKinesisKit@@ -162,18 +146,18 @@ source = consumerSource consumer step n r = succ n <$ liftIO (B8.putStrLn $ recordData r) countingSink = CL.foldM step (1 ∷ Int)- sink = case _clioLimit of+ sink = case _configLimit of Just limit → CL.isolate limit =$ countingSink Nothing → countingSink runConsumer = do n ← catch (source $$ sink) $ \SomeException{} → return 0- return $ maybe True (n ≥) _clioLimit+ return $ maybe True (n ≥) _configLimit -- If a timeout is set, then we will race the timeout against the consumer. result ← lift $- case _clioTimeout of+ case _configTimeout of Just seconds → race (threadDelay $ seconds * 1000000) runConsumer Nothing → Right <$> runConsumer @@ -183,12 +167,12 @@ -- if we timed out: if there a requested limit, then we failed to get -- it (and this is a failure); if there was no requested limit, then we -- consider this a success.- Left () → isn't _Just _clioLimit+ Left () → isn't _Just _configLimit -- if we did not timeout, then it is a success Right b → b - lift ∘ when successful ∘ void ∘ for _clioStateOut $ \outPath → do+ lift ∘ when successful ∘ void ∘ for _configStateOut $ \outPath → do state ← consumerStreamState consumer liftIO ∘ BL8.writeFile outPath $ A.encode state @@ -197,12 +181,19 @@ then ExitSuccess else ExitFailure 1 +mainInfo ∷ ProgramInfo Config+mainInfo =+ programInfoValidate+ "Kinesis CLI"+ pConfig+ defaultConfig+ validateConfig+ main ∷ IO () main = do- exitCode ←- liftIO (execParser parserInfo)- ≫= runReaderT (lowerCodensity app)- exitWith exitCode+ runWithPkgInfoConfiguration mainInfo pkgInfo $+ runReaderT (lowerCodensity app)+ >=> exitWith managedHttpManager ∷ ( MonadIO m
+ cli/CLI/Config.hs view
@@ -0,0 +1,244 @@+-- Copyright (c) 2013-2015 PivotCloud, Inc.+--+-- CLI.Config+--+-- Please feel free to contact us at licensing@pivotmail.com with any+-- contributions, additions, or other feedback; we would love to hear from+-- you.+--+-- Licensed under the Apache License, Version 2.0 (the "License"); you may+-- not use this file except in compliance with the License. You may obtain a+-- copy of the License at http://www.apache.org/licenses/LICENSE-2.0+--+-- Unless required by applicable law or agreed to in writing, software+-- distributed under the License is distributed on an "AS IS" BASIS, WITHOUT+-- WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the+-- License for the specific language governing permissions and limitations+-- under the License.++{-# LANGUAGE FlexibleContexts #-}+{-# LANGUAGE FlexibleInstances #-}+{-# LANGUAGE LambdaCase #-}+{-# LANGUAGE OverloadedStrings #-}+{-# LANGUAGE RecordWildCards #-}+{-# LANGUAGE UnicodeSyntax #-}++-- |+-- Module: CLI.Config+-- Copyright: Copyright © 2013-2015 PivotCloud, Inc.+-- License: Apache-2.0+-- Maintainer: Jon Sterling <jsterling@alephcloud.com>+-- Stability: experimental+--+module CLI.Config+( -- * Configuration Type+ Config(..)+, defaultConfig+ -- ** Lenses+, configStreamName+, configRegion+, configLimit+, configTimeout+, configIteratorType+, configCredentialConfig+, configStateIn+, configStateOut+ -- ** Parser+, pConfig+ -- ** Validation+, validateConfig+) where++import Aws.General+import Aws.Kinesis+import Configuration.Utils hiding (Lens')+import Configuration.Utils.Aws.Credentials+import Control.Lens hiding ((.=))++import Control.Monad+import Control.Monad.Error.Class+import Control.Monad.Unicode+import Data.Monoid.Unicode+import qualified Data.Text as T+import qualified Data.Text.Lens as T+import Options.Applicative+import Options.Applicative.Types+import Prelude.Unicode++data Config+ = Config+ { _configStreamName ∷ !StreamName+ , _configRegion ∷ !Region+ , _configLimit ∷ !(Maybe Int)+ , _configTimeout ∷ !(Maybe Int)+ , _configIteratorType ∷ !ShardIteratorType+ , _configCredentialConfig ∷ !CredentialConfig+ , _configStateIn ∷ !(Maybe FilePath)+ , _configStateOut ∷ !(Maybe FilePath)+ } deriving Show++invalidStreamName ∷ StreamName+invalidStreamName = "NOT_A_VALID_STREAM_NAME"++defaultConfig ∷ Config+defaultConfig = Config+ { _configStreamName = invalidStreamName+ , _configRegion = UsWest2+ , _configLimit = Nothing+ , _configTimeout = Nothing+ , _configIteratorType = TrimHorizon+ , _configCredentialConfig = defaultCredentialConfig+ , _configStateIn = Nothing+ , _configStateOut = Nothing+ }++configStreamName ∷ Lens' Config StreamName+configStreamName = lens _configStreamName $ \x y → x { _configStreamName = y }++configRegion ∷ Lens' Config Region+configRegion = lens _configRegion $ \x y → x { _configRegion = y }++configLimit ∷ Lens' Config (Maybe Int)+configLimit = lens _configLimit $ \x y → x { _configLimit = y }++configTimeout ∷ Lens' Config (Maybe Int)+configTimeout = lens _configTimeout $ \x y → x { _configTimeout = y }++configIteratorType ∷ Lens' Config ShardIteratorType+configIteratorType = lens _configIteratorType $ \x y → x { _configIteratorType = y }++configCredentialConfig ∷ Lens' Config CredentialConfig+configCredentialConfig = lens _configCredentialConfig $ \x y → x { _configCredentialConfig = y }++configStateIn ∷ Lens' Config (Maybe FilePath)+configStateIn = lens _configStateIn $ \x y → x { _configStateIn = y }++configStateOut∷ Lens' Config (Maybe FilePath)+configStateOut = lens _configStateOut $ \x y → x { _configStateOut = y }++-- makeLenses ''Config++instance FromJSON (Config → Config) where+ parseJSON =+ withObject "Config" $ \o → id+ <$< configStreamName ..: "stream_name" % o+ <*< setProperty configRegion "region" (withText "Region" $ either fail return ∘ fromText) o+ <*< configLimit ..: "limit" % o+ <*< configTimeout ..: "timeout" % o+ <*< configIteratorType ..: "iterator_type" % o+ <*< configCredentialConfig %.: "aws_credentials" % o+ <*< configStateIn ..: "restore_state" % o+ <*< configStateOut ..: "save_state" % o++instance ToJSON Config where+ toJSON Config{..} = object+ [ "stream_name" .= _configStreamName+ , "region" .= String (toText _configRegion)+ , "limit" .= _configLimit+ , "timeout" .= _configTimeout+ , "iterator_type" .= _configIteratorType+ , "aws_credentials" .= _configCredentialConfig+ , "restore_state" .= _configStateIn+ , "save_state" .= _configStateOut+ ]+++eitherTextReader+ ∷ (T.IsText i, T.IsText e)+ ⇒ (i → Either e a)+ → ReadM a+eitherTextReader f =+ eitherReader $+ (_Left %~ view T.unpacked) ∘ f ∘ view T.packed++streamNameParser ∷ Parser StreamName+streamNameParser =+ option (eitherTextReader streamName) $+ long "stream-name"+ ⊕ short 's'+ ⊕ metavar "SN"+ ⊕ help "Fetch from the Kinesis stream named `SN`"++limitParser ∷ Parser Int+limitParser =+ option auto $+ long "limit"+ ⊕ short 'l'+ ⊕ metavar "L"+ ⊕ help "Fetch `L` records. If a limit is provided, then the run will only be considered successful if it results in the CLI fetching `L` records; otherwise, a run is always considered successful."++timeoutParser ∷ Parser Int+timeoutParser =+ option auto $+ long "timeout"+ ⊕ short 't'+ ⊕ metavar "T"+ ⊕ help "Terminate the consumer after `T` seconds. Even if a limit has been provided, the consumer will terminate after at most `T` seconds."++regionParser ∷ Parser Region+regionParser =+ option regionReader $+ long "region"+ ⊕ value UsWest2+ ⊕ help "Choose an AWS Kinesis region (default: us-west-2)"++regionReader ∷ ReadM Region+regionReader = do+ fromText ∘ T.pack <$> readerAsk ≫=+ either readerError return++iteratorTypeReader ∷ ReadM ShardIteratorType+iteratorTypeReader = do+ readIteratorType ∘ T.pack <$> readerAsk ≫=+ either readerError return++ where+ readIteratorType = \case+ "LATEST" → Right Latest+ "Latest" → Right Latest+ "TRIM_HORIZON" → Right TrimHorizon+ "TrimHorizon" → Right TrimHorizon+ it → Left $ "Unsupported shard iterator type: " ⊕ T.unpack it++iteratorTypeParser ∷ Parser ShardIteratorType+iteratorTypeParser =+ option iteratorTypeReader $+ long "iterator-type"+ ⊕ short 'i'+ ⊕ metavar "IT"+ ⊕ help "Iterator type (LATEST|TRIM_HORIZON)"+ ⊕ value TrimHorizon+ ⊕ showDefault++stateOutParser ∷ Parser FilePath+stateOutParser =+ strOption $+ long "save-state"+ ⊕ help "Write the last known state of each shard to a file; this will only occur when the run has completed in a \"successful\" state."+ ⊕ metavar "FILE"++stateInParser ∷ Parser FilePath+stateInParser =+ strOption $+ long "restore-state"+ ⊕ help "Read a saved stream state from a file. For any shards whose state is restored, the 'AFTER_SEQUENCE_NUMBER' iterator type will be used; other shards will use the iterator type you have specified. Some shards may have been merged or closed between when the state was saved and restored; at this point, no effort has been made to do anything here beyond the obvious (shards are identified by their shard-id)."+ ⊕ metavar "FILE"++pConfig ∷ MParser Config+pConfig = id+ <$< configStreamName .:: streamNameParser+ <*< configRegion .:: regionParser+ <*< configLimit .:: optional limitParser+ <*< configTimeout .:: optional timeoutParser+ <*< configIteratorType .:: iteratorTypeParser+ <*< configCredentialConfig %:: pCredentialConfig "aws-"+ <*< configStateIn .:: optional stateInParser+ <*< configStateOut .:: optional stateOutParser++validateConfig ∷ ConfigValidation Config λ+validateConfig Config{..} = do+ when (_configStreamName == invalidStreamName) $+ throwError "stream name not configured"++ validateCredentialConfig _configCredentialConfig+
− cli/CLI/Options.hs
@@ -1,220 +0,0 @@--- Copyright (c) 2013-2015 PivotCloud, Inc.------ Aws.Kinesis.Client.Consumer------ Please feel free to contact us at licensing@pivotmail.com with any--- contributions, additions, or other feedback; we would love to hear from--- you.------ Licensed under the Apache License, Version 2.0 (the "License"); you may--- not use this file except in compliance with the License. You may obtain a--- copy of the License at http://www.apache.org/licenses/LICENSE-2.0------ Unless required by applicable law or agreed to in writing, software--- distributed under the License is distributed on an "AS IS" BASIS, WITHOUT--- WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the--- License for the specific language governing permissions and limitations--- under the License.--{-# LANGUAGE OverloadedStrings #-}-{-# LANGUAGE RecordWildCards #-}-{-# LANGUAGE TemplateHaskell #-}-{-# LANGUAGE UnicodeSyntax #-}---- |--- Module: CLI.Options--- Copyright: Copyright © 2013-2015 PivotCloud, Inc.--- License: Apache-2.0--- Maintainer: Jon Sterling <jsterling@alephcloud.com>--- Stability: experimental----module CLI.Options-( CLIOptions(..)-, clioStreamName-, clioLimit-, clioTimeout-, clioIteratorType-, clioAccessKeys-, clioStateIn-, clioStateOut-, clioUseInstanceMetadata-, clioRegion-, CredentialsInput(..)-, AccessKeys(..)-, akAccessKeyId-, akSecretAccessKey-, optionsParser-, parserInfo-) where--import Aws.General-import Aws.Kinesis-import qualified Data.ByteString.Char8 as B8-import Control.Applicative.Unicode-import Control.Lens-import Control.Monad.Unicode-import Data.Monoid.Unicode-import qualified Data.Text as T-import qualified Data.Text.Lens as T-import Options.Applicative-import Options.Applicative.Types-import Prelude.Unicode--data AccessKeys- = AccessKeys- { _akAccessKeyId ∷ !B8.ByteString- , _akSecretAccessKey ∷ !B8.ByteString- } deriving Show--data CredentialsInput- = CredentialsFromAccessKeys AccessKeys- | CredentialsFromFile FilePath- deriving Show--data CLIOptions- = CLIOptions- { _clioStreamName ∷ !StreamName- , _clioRegion ∷ !Region- , _clioLimit ∷ !(Maybe Int)- , _clioTimeout ∷ !(Maybe Int)- , _clioIteratorType ∷ !ShardIteratorType- , _clioAccessKeys ∷ !(Maybe CredentialsInput)- , _clioUseInstanceMetadata ∷ !Bool- , _clioStateIn ∷ !(Maybe FilePath)- , _clioStateOut ∷ !(Maybe FilePath)- } deriving Show--makeLenses ''AccessKeys-makeLenses ''CLIOptions--eitherTextReader- ∷ ( T.IsText i- , T.IsText e- )- ⇒ (i → Either e a)- → ReadM a-eitherTextReader f =- eitherReader $- (_Left %~ view T.unpacked) ∘ f ∘ view T.packed--accessKeyIdParser ∷ Parser B8.ByteString-accessKeyIdParser =- fmap B8.pack ∘ strOption $- long "access-key-id"- ⊕ metavar "ID"- ⊕ help "Your AWS access key id"--secretAccessKeyParser ∷ Parser B8.ByteString-secretAccessKeyParser =- fmap B8.pack ∘ strOption $- long "secret-access-key"- ⊕ metavar "SK"- ⊕ help "Your AWS secret access key"--accessKeysParser ∷ Parser AccessKeys-accessKeysParser =- pure AccessKeys- ⊛ accessKeyIdParser- ⊛ secretAccessKeyParser--accessKeysPathParser ∷ Parser FilePath-accessKeysPathParser =- strOption $- long "access-keys-path"- ⊕ metavar "PATH"- ⊕ help "The path to a file containing your access keys. To be formatted \"default ID SECRET\"; you may provide this instead of the command line options for credentials"--streamNameParser ∷ Parser StreamName-streamNameParser =- option (eitherTextReader streamName) $- long "stream-name"- ⊕ short 's'- ⊕ metavar "SN"- ⊕ help "Fetch from the Kinesis stream named `SN`"--limitParser ∷ Parser Int-limitParser =- option auto $- long "limit"- ⊕ short 'l'- ⊕ metavar "L"- ⊕ help "Fetch `L` records. If a limit is provided, then the run will only be considered successful if it results in the CLI fetching `L` records; otherwise, a run is always considered successful."--timeoutParser ∷ Parser Int-timeoutParser =- option auto $- long "timeout"- ⊕ short 't'- ⊕ metavar "T"- ⊕ help "Terminate the consumer after `T` seconds. Even if a limit has been provided, the consumer will terminate after at most `T` seconds."--iteratorTypeParser ∷ Parser ShardIteratorType-iteratorTypeParser =- option auto $- long "iterator-type"- ⊕ short 'i'- ⊕ metavar "IT"- ⊕ help "Iterator type (Latest|TrimHorizon)"- ⊕ value TrimHorizon- ⊕ showDefault--stateOutParser ∷ Parser FilePath-stateOutParser =- strOption $- long "save-state"- ⊕ help "Write the last known state of each shard to a file; this will only occur when the run has completed in a \"successful\" state."- ⊕ metavar "FILE"--stateInParser ∷ Parser FilePath-stateInParser =- strOption $- long "restore-state"- ⊕ help "Read a saved stream state from a file. For any shards whose state is restored, the 'AFTER_SEQUENCE_NUMBER' iterator type will be used; other shards will use the iterator type you have specified. Some shards may have been merged or closed between when the state was saved and restored; at this point, no effort has been made to do anything here beyond the obvious (shards are identified by their shard-id)."- ⊕ metavar "FILE"--credentialsInputParser ∷ Parser CredentialsInput-credentialsInputParser =- foldr (<|>) empty $- [ CredentialsFromAccessKeys <$> accessKeysParser- , CredentialsFromFile <$> accessKeysPathParser- ]--useInstanceMetadataParser ∷ Parser Bool-useInstanceMetadataParser =- switch $- long "use-instance-metadata"- ⊕ help "Read the credentials from the instance metadata"--regionReader ∷ ReadM Region-regionReader = do- fromText ∘ T.pack <$> readerAsk ≫=- either readerError return---regionParser ∷ Parser Region-regionParser =- option regionReader $- long "region"- ⊕ value UsWest2- ⊕ help "Choose an AWS Kinesis region (default: us-west-2)"--optionsParser ∷ Parser CLIOptions-optionsParser =- pure CLIOptions- ⊛ streamNameParser- ⊛ regionParser- ⊛ optional limitParser- ⊛ optional timeoutParser- ⊛ iteratorTypeParser- ⊛ optional credentialsInputParser- ⊛ useInstanceMetadataParser- ⊛ optional stateInParser- ⊛ optional stateOutParser--parserInfo ∷ ParserInfo CLIOptions-parserInfo =- info (helper ⊛ optionsParser) $- fullDesc- ⊕ progDesc "Fetch a given number of records from a Kinesis stream; unlike the standard command line utilities, this interface is suitable for use with a sharded stream. If you both specify a saved stream state to be restored and an iterator type, the latter will be used on any shards which are not contained in the saved state. Minimally, you must specify your AWS credentials, a stream name, and an optional limit & timeout."- ⊕ header "The Kinesis Consumer CLI v0.4.0.1"-
+ example_config.yml view
@@ -0,0 +1,18 @@+stream_name: mystream++aws_credentials:+ file:+ key_name: default+ file_name: /Users/you/.aws-keys++iterator_type: TRIM_HORIZON++region: us-west-2++limit: null+timeout: null++# state restoration+restore_state: null # saved_state.json+save_state: null # saved_state.json+