curl-runnings (empty) → 0.1.0

raw patch · 9 files changed

+585/−0 lines, 9 filesdep +aesondep +aeson-prettydep +basesetup-changed

Dependencies added: aeson, aeson-pretty, base, bytestring, cmdargs, curl-runnings, http-conduit, text, unordered-containers, yaml

Files

• LICENSE +21/−0
• README.md +121/−0
• Setup.hs +2/−0
• app/Main.hs +67/−0
• curl-runnings.cabal +76/−0
• src/Testing/CurlRunnings.hs +113/−0
• src/Testing/CurlRunnings/Internal.hs +12/−0
• src/Testing/CurlRunnings/Types.hs +171/−0
• test/Spec.hs +2/−0

@@ -0,0 +1,21 @@+MIT License++Copyright (c) 2018 Avi Press++Permission is hereby granted, free of charge, to any person obtaining a copy+of this software and associated documentation files (the "Software"), to deal+in the Software without restriction, including without limitation the rights+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell+copies of the Software, and to permit persons to whom the Software is+furnished to do so, subject to the following conditions:++The above copyright notice and this permission notice shall be included in all+copies or substantial portions of the Software.++THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE+SOFTWARE.

@@ -0,0 +1,121 @@+# curl-runnings++[![Build Status](https://travis-ci.org/aviaviavi/curl-runnings.svg?branch=master)](https://travis-ci.org/aviaviavi/curl-runnings)++Feel the rhythm! Feel the rhyme! Get on up, it's testing time! curl-runnings!++curl-runnings is a framework for writing declarative, curl based tests for your APIs. ++### Why?++When writing curl based smoke/integration tests for APIs using bash and `curl`+is very convenient, but quickly becomes hard to maintain. Writing matchers for+json output quickly becomes unweildy and error prone. Writing these sorts of+tests in a more traditional programming language is fine, but certainly more+time consuming to write than some simple curl requests. curl-runnings aims to+make it very simple to write tests that curl some endpoints and verify the+output looks sane.++With curl-runnings, you can write your tests just as data in a yaml or json file,+curl runnings will take care of the rest!++While yaml/json is the current way to write curl-runnings tests, this project is+being built in a way that should lend itself well to an embedded domain specific+language, which is a future goal for the project++### Installing++There are few options to install:++- clone this repo and run `stack install`+- download the releases +- (soon) - `cabal install curl-runnings`++### Writing a test specification++A test spec is a top level list of test cases, each item represents a single curl and set of assertions about the response:++```yaml+---+# the top level of the file is an array of test cases+- name: test 1 # required+  url: http://your-endpoint.com # required+  requestMethod: GET # required+  # specify the json payload we expect here, if any+  expectData: # optional+    # `tag` refers to the type of matcher we want to use+    # valid tags are `Exactly` | `Contains`+    tag: Exactly+    # check for exactly this payload+    contents:+      okay: true+      msg: ''+  expectStatus: # requried+    # `tag` refers to the type of matcher we want to use+    # valid tags are `ExactCode` (contents :: number) | `AnyCodeIn` (contents :: [number])+    tag: ExactCode+    contents: 200+- name: test 2+  url: http://your-endpoint.com/path+  requestMethod: POST+  expectStatus:+    # Any code listed in `contents` is valid+    tag: AnyCodeIn+    contents:+    - 200+    - 201+  # json data to send with the request+  requestData:+    hello: there+    num: 1+- name: test 3+  url: http://your-url.com/other/path+  requestMethod: GET+  expectData:+    # apply a list of matchers to the returned json payload+    tag: Contains+    contents:+    # valid tags are `ValueMatch` | `KeyValueMatch`+    # find the value `true` anywhere in the payload. This can be useful for matching against values +    # where you don't know the key ahead of time, or for values in a top level array.+    - tag: ValueMatch+      contents: true+    # `KeyValueMatch` looks for the key/value pair anywhere in the payload+    # here, {'okay': true} must be somewhere in the return payload+    - tag: KeyValueMatch+      matchKey: okay+      matchValue: true+  expectStatus:+    tag: ExactCode+    contents: 200+```++### Running++Once you've written a spec, simply run it with:++```bash+$ curl-runnings -f path/to/your/spec.yaml+```++If all your tests pass, curl-runnings will cleanly exit with a 0 code. A code of+1 will be returned if any tests failed.++For more info:++```bash+$ curl-runnings --help+```+++### Future work++- [x] Json specifications for tests+- [x] Yaml specifications for tests+- [ ] Embedded dsl for specifications for tests. As the specification gets more complex.+  - [ ] Spec out dsl that can compile down into a yaml/json spec+  - [ ] Implement dsl+- [ ] More specification features+  - [ ] timeouts+  - [ ] retry logic+  - [ ] ability to configure alerts

@@ -0,0 +1,2 @@+import Distribution.Simple+main = defaultMain

@@ -0,0 +1,67 @@+{-# LANGUAGE DeriveDataTypeable #-}+{-# LANGUAGE OverloadedStrings  #-}++module Main where++import           Data.Aeson+import qualified Data.ByteString.Char8           as B8S+import qualified Data.ByteString.Lazy            as B+import qualified Data.Text                       as T+import           Data.Version                    (showVersion)+import qualified Data.Yaml                       as Y+import           Paths_curl_runnings             (version)+import           System.Console.CmdArgs+import           System.Environment+import           System.Exit+import           Testing.CurlRunnings+import           Testing.CurlRunnings.Internal+import           Testing.CurlRunnings.Types+import           Text.Printf++-- | Command line flags+data CurlRunnings = CurlRunnings+  { file :: FilePath+  } deriving (Show, Data, Typeable, Eq)++-- | cmdargs object+argParser :: CurlRunnings+argParser =+  CurlRunnings {file = def &= typFile &= help "File to run"} &=+  summary ("curl-runnings " ++ showVersion version) &=+  program "curl-runnings" &=+  help "Use the --file or -f flag to specify an intput file spec to run"++-- | decode a json or yaml file into a suite object+decodeFile :: FilePath -> IO (Either String CurlSuite)+decodeFile specPath =+  case last $ T.splitOn "." (T.pack specPath) of+    "json" ->+      eitherDecode' <$> B.readFile specPath :: IO (Either String CurlSuite)+    "yaml" ->+      Y.decodeEither <$> B8S.readFile specPath :: IO (Either String CurlSuite)+    "yml" ->+      Y.decodeEither <$> B8S.readFile specPath :: IO (Either String CurlSuite)+    _ ->+      return . Left $+      printf+        "Invalid spec path %s"+        (T.pack specPath)++main :: IO ()+main = do+  mainArgs <- cmdArgs argParser+  case file mainArgs of+    "" ->+      putStrLn+        "Please specify an input file with the --file (-f) flag or use --help for more information"+    path -> do+      home <- getEnv "HOME"+      suite <- decodeFile . T.unpack $ T.replace "~" (T.pack home) (T.pack path)+      case suite of+        Right s -> do+          results <- runSuite s+          if any isFailing results+            then putStrLn (makeRed "Some tests failed") >>+                 exitWith (ExitFailure 1)+            else putStrLn $ makeGreen "All tests passed!"+        Left messgage -> putStrLn $ "Couldn't read input json or yaml file: " ++ messgage

@@ -0,0 +1,76 @@+-- This file has been generated from package.yaml by hpack version 0.20.0.+--+-- see: https://github.com/sol/hpack+--+-- hash: d8d3fa8a875ec9e27b6db4c1ad13dcc50d7354bbbc647ae07ff08a91b0f4c79b++name:           curl-runnings+version:        0.1.0+synopsis:       A framework for declaratively writing curl based API tests+description:    Please see the README on Github at <https://github.com/aviaviavi/curl-runnings#readme>+category:       Testing+homepage:       https://github.com/aviaviavi/curl-runnings#readme+bug-reports:    https://github.com/aviaviavi/curl-runnings/issues+author:         Avi Press+maintainer:     mail@avi.press+copyright:      2018 Avi Press+license:        MIT+license-file:   LICENSE+build-type:     Simple+cabal-version:  >= 1.10++extra-source-files:+    README.md++source-repository head+  type: git+  location: https://github.com/aviaviavi/curl-runnings++library+  hs-source-dirs:+      src+  build-depends:+      aeson >=1.2.4.0 && <1.3+    , aeson-pretty >=0.8.5 && <0.9+    , base >=4.7 && <5+    , bytestring >=0.10.8.2 && <0.11+    , http-conduit >=2.2.4 && <2.3+    , text >=1.2.2.2 && <1.3+    , unordered-containers >=0.2.8.0 && <0.3+  exposed-modules:+      Testing.CurlRunnings+      Testing.CurlRunnings.Types+      Testing.CurlRunnings.Internal+  other-modules:+      Paths_curl_runnings+  default-language: Haskell2010++executable curl-runnings+  main-is: Main.hs+  hs-source-dirs:+      app+  ghc-options: -threaded -rtsopts -with-rtsopts=-N+  build-depends:+      aeson >=1.2.4.0 && <1.3+    , base >=4.7 && <5+    , bytestring >=0.10.8.2 && <0.11+    , cmdargs >=0.10.20 && <0.11+    , curl-runnings+    , text >=1.2.2.2 && <1.3+    , yaml >=0.8.28 && <0.9+  other-modules:+      Paths_curl_runnings+  default-language: Haskell2010++test-suite curl-runnings-test+  type: exitcode-stdio-1.0+  main-is: Spec.hs+  hs-source-dirs:+      test+  ghc-options: -threaded -rtsopts -with-rtsopts=-N+  build-depends:+      base >=4.7 && <5+    , curl-runnings+  other-modules:+      Paths_curl_runnings+  default-language: Haskell2010

@@ -0,0 +1,113 @@+-- | curl-runnings is a framework for writing declaratively writing curl based tests for your API's.+-- Write your test specifications with yaml or json, and you're done!+--+module Testing.CurlRunnings+    (+      runCase+    , runSuite+    ) where++import           Control.Monad+import           Data.Aeson+import qualified Data.ByteString.Char8      as B8S+import qualified Data.ByteString.Lazy       as B+import qualified Data.HashMap.Strict        as H+import           Data.Maybe+import qualified Data.Text                  as T+import           Network.HTTP.Conduit+import           Network.HTTP.Simple+import           Testing.CurlRunnings.Types++-- | Run a single test case, and returns the result. IO is needed here since this method is responsible+-- for actually curling the test case endpoint and parsing the result.+runCase :: CurlCase -> IO CaseResult+runCase curlCase = do+  initReq <- parseRequest $ url curlCase+  response <- httpBS . setRequestBodyJSON (requestData curlCase) $ initReq {method = B8S.pack . show $ requestMethod curlCase}+  returnVal <-+    (return . decode . B.fromStrict $ getResponseBody response) :: IO (Maybe Value)+  let returnCode = getResponseStatusCode response+      assertionErrors =+        map fromJust $+        filter+          isJust+          [checkBody curlCase returnVal, checkCode curlCase returnCode]+  return $+    case assertionErrors of+      []       -> CasePass curlCase+      failures -> CaseFail curlCase failures++safeLast :: [a] -> Maybe a+safeLast [] = Nothing+safeLast x  = Just $ last x++printR :: Show a => a -> IO a+printR x = print x >> return x++-- | Runs the test cases in order and stop when an error is hit. Returns all the results+runSuite :: CurlSuite -> IO [CaseResult]+runSuite (CurlSuite cases) =+  foldM+    (\prevResults curlCase ->+       case safeLast prevResults of+         Just (CaseFail _ _) -> return prevResults+         Just (CasePass _) -> do+           result <- runCase curlCase >>= printR+           return $ prevResults ++ [result]+         Nothing -> do+           result <- runCase curlCase >>= printR+           return [result])+    []+    cases++-- | Check if the retrieved value fail's the case's assertion+checkBody :: CurlCase -> Maybe Value -> Maybe AssertionFailure+-- | We are looking for an exact payload match, and we have a payload to check+checkBody curlCase@(CurlCase _ _ _ _ (Just matcher@(Exactly expectedValue)) _) (Just receivedBody)+  | expectedValue /= receivedBody =+    Just $ DataFailure curlCase matcher (Just receivedBody)+  | otherwise = Nothing+-- | We are checking a list of expected subvalues, and we have a payload to check+checkBody curlCase@(CurlCase _ _ _ _ (Just matcher@(Contains expectedSubvalues)) _) (Just receivedBody)+  | jsonContainsAll receivedBody expectedSubvalues = Nothing+  | otherwise = Just $ DataFailure curlCase matcher (Just receivedBody)+-- | We expected a body but didn't get one+checkBody curlCase@(CurlCase _ _ _ _ (Just anything) _) Nothing = Just $ DataFailure curlCase anything Nothing+-- | No assertions on the body+checkBody (CurlCase _ _ _ _ Nothing _) _ = Nothing++-- | Does the json value contain all of these sub-values?+jsonContainsAll :: Value -> [JsonSubExpr] -> Bool+jsonContainsAll jsonValue =+  all (\match -> case match  of+          ValueMatch subval -> subval `elem` traverseValue jsonValue+          KeyValueMatch key subval ->+            containsKeyVal jsonValue key subval+      )++-- | Does the json value contain the given key value pair?+containsKeyVal :: Value -> T.Text -> Value -> Bool+containsKeyVal jsonValue key val = case jsonValue of+  Object o -> isJust $ H.lookup key o+  _        -> False++-- | Fully traverse the json and return a list of all the values+traverseValue :: Value -> [Value]+traverseValue val =+  case val of+    Object o     -> val : concatMap traverseValue (H.elems o)+    Array o      -> val : concatMap traverseValue o+    n@(Number _) -> [n]+    s@(String _) -> [s]+    b@(Bool _)   -> [b]+    Null         -> []++-- | Verify the returned http status code is ok, construct the right failure+-- type if needed+checkCode :: CurlCase -> Int -> Maybe AssertionFailure+checkCode curlCase@(CurlCase _ _ _ _ _ (ExactCode expectedCode)) receivedCode+  | expectedCode /= receivedCode = Just $ StatusFailure curlCase receivedCode+  | otherwise = Nothing+checkCode curlCase@(CurlCase _ _ _ _ _ (AnyCodeIn l)) receivedCode+  | receivedCode `notElem` l = Just $ StatusFailure curlCase receivedCode+  | otherwise = Nothing

@@ -0,0 +1,12 @@+-- | This module specifies any utilities used by this package. At this time,+-- consider everything in this module to be private to the curl-runnings package+module Testing.CurlRunnings.Internal+  ( makeRed+  , makeGreen+  ) where++makeGreen :: String -> String+makeGreen s = "\x1B[32m" ++ s ++ "\x1B[0m"++makeRed :: String -> String+makeRed s = "\x1B[31m" ++ s ++ "\x1B[0m"

@@ -0,0 +1,171 @@+{-# LANGUAGE DeriveGeneric #-}+{-# LANGUAGE OverloadedStrings #-}++-- | Data types for curl-runnings tests+module Testing.CurlRunnings.Types+  ( CurlSuite(..)+  , CurlCase(..)+  , HttpMethod(..)+  , JsonMatcher(..)+  , JsonSubExpr(..)+  , StatusCodeMatcher(..)+  , AssertionFailure(..)+  , CaseResult(..)+  , isPassing+  , isFailing+  ) where++import Data.Aeson+import Data.Aeson.Encode.Pretty+import qualified Data.ByteString.Lazy.Char8 as B8+import qualified Data.Text as T+import GHC.Generics+import Text.Printf+import Testing.CurlRunnings.Internal++-- | A basic enum for supported HTTP verbs+data HttpMethod+  = GET+  | POST+  | PUT+  | PATCH+  | DELETE+  deriving (Show, Generic)++instance FromJSON HttpMethod++instance ToJSON HttpMethod++-- | A predicate to apply to the json body from the response+data JsonMatcher+  -- | Performs `==`+  = Exactly Value+  -- | A list of matchers to make assertions about some subset of the response.+  | Contains [JsonSubExpr]++  deriving (Show, Generic)++instance FromJSON JsonMatcher++instance ToJSON JsonMatcher++-- | A matcher for a subvalue of a json payload+data JsonSubExpr+  -- | Assert some value anywhere in the json has a value equal to a given+  --  value. The motivation for this field is largely for checking contents of a+  --  top level array. It's also useful if you don't know the key ahead of time.+  = ValueMatch Value+  -- | Assert the key value pair can be found somewhere the json.+  | KeyValueMatch { matchKey :: T.Text+                  , matchValue :: Value }+  deriving (Show, Generic)++instance FromJSON JsonSubExpr++instance ToJSON JsonSubExpr++-- | Check the status code of a response. You can specify one or many valid codes.+data StatusCodeMatcher+  = ExactCode Int+  | AnyCodeIn [Int]+  deriving (Show, Generic)++instance FromJSON StatusCodeMatcher++instance ToJSON StatusCodeMatcher++-- | A single curl test case, the basic foundation of a curl-runnings test.+data CurlCase = CurlCase+  { name :: String -- ^ The name of the test case+  , url :: String -- ^ The target url to test+  , requestMethod :: HttpMethod -- ^ Berb to use for the request+  , requestData :: Maybe Value -- ^ Payload to send with the request, if any+  , expectData :: Maybe JsonMatcher -- ^ The assertions to make on the response payload, if any+  , expectStatus :: StatusCodeMatcher -- ^ Assertion about the status code returned by the target+  } deriving (Show, Generic)++instance FromJSON CurlCase++instance ToJSON CurlCase++-- | Represents the different type of test failures we can have. A single test case+-- | might return many assertion failures.+data AssertionFailure+  -- | The json we got back was wrong. We include this redundant field (it's+  -- included in the CurlCase field above) in order to enforce at the type+  -- level that we have to be expecting some data in order to have this type of+  -- failure.+  = DataFailure CurlCase+                JsonMatcher+                (Maybe Value)+  -- | The status code we got back was wrong+  | StatusFailure CurlCase+                  Int+  -- | Something else+  | UnexpectedFailure++instance Show AssertionFailure where+  show (StatusFailure c receivedCode) =+    case expectStatus c of+      ExactCode code ->+        printf+          "Incorrect status code from %s. Expected: %s. Actual: %s"+          (url c)+          (show code)+          (show receivedCode)+      AnyCodeIn codes ->+        printf+          "Incorrect status code from %s. Expected one of: %s. Actual: %s"+          (url c)+          (show codes)+          (show receivedCode)+  show (DataFailure curlCase expected receivedVal) =+    case expected of+      Exactly expectedVal ->+        printf+          "JSON response from %s didn't match spec. Expected: %s. Actual: %s"+          (url curlCase)+          (B8.unpack (encodePretty expectedVal))+          (B8.unpack (encodePretty receivedVal))+      (Contains expectedVals) ->+        printf+          "JSON response from %s didn't contain the matcher. Expected: %s to be each be subvalues in: %s"+          (url curlCase)+          (B8.unpack (encodePretty expectedVals))+          (B8.unpack (encodePretty receivedVal))+  show UnexpectedFailure = "Unexpected Error D:"++-- | A type representing the result of a single curl, and all associated+-- assertions+data CaseResult+  = CasePass CurlCase+  | CaseFail CurlCase+             [AssertionFailure]++instance Show CaseResult where+  show (CasePass c) = makeGreen "[PASS] " ++ name c+  show (CaseFail c failures) =+    makeRed "[FAIL] " +++    name c +++    "\n" +++    concatMap ((\s -> "\nAssertion failed: " ++ s) . (++ "\n") . show) failures++-- | A wrapper type around a set of test cases. This is the top level spec type+-- that we parse a test spec file into+newtype CurlSuite =+  CurlSuite [CurlCase]+  deriving (Show, Generic)++instance FromJSON CurlSuite++instance ToJSON CurlSuite++-- | Simple predicate that checks if the result is passing+isPassing :: CaseResult -> Bool+isPassing (CasePass _) = True+isPassing (CaseFail _ _) = False++-- | Simple predicate that checks if the result is failing+isFailing :: CaseResult -> Bool+isFailing (CasePass _) = False+isFailing (CaseFail _ _) = True

@@ -0,0 +1,2 @@+main :: IO ()+main = putStrLn "Test suite not yet implemented"