curl-runnings 0.15.0 → 0.16.0
raw patch · 7 files changed
+193/−43 lines, 7 filesdep +dhalldep +dhall-jsondep +hashablePVP ok
version bump matches the API change (PVP)
Dependencies added: dhall, dhall-json, hashable, transformers
API changes (from Hackage documentation)
Files
- README.md +31/−27
- curl-runnings.cabal +8/−3
- examples/example-spec.dhall +97/−0
- src/Testing/CurlRunnings.hs +37/−2
- src/Testing/CurlRunnings/Internal.hs +1/−1
- src/Testing/CurlRunnings/Types.hs +16/−10
- test/Spec.hs +3/−0
README.md view
@@ -6,37 +6,20 @@ _Feel the rhythm! Feel the rhyme! Get on up, it's testing time! curl-runnings!_ -curl-runnings is a framework for writing declarative, curl based tests for your-APIs. Write your tests quickly and correctly with a straight-forward-specification in yaml or json that can encode simple but powerful matchers-against responses.+A common form of black-box API testing boils down to simply making requests to+an endpoint and verifying properties of the response. curl-runnings aims to make+writing tests like this fast and easy. +curl-runnings is a framework for writing declarative tests for your APIs in a+fashion equivalent to performing `curl`s and verifying the responses. Write your+tests quickly and correctly with a straight-forward specification in+[Dhall](https://dhall-lang.org/), yaml, or json that can encode simple but+powerful matchers against responses.+ Alternatively, you can use the curl-runnings library to write your tests in Haskell (though a Haskell setup is absolutely not required to use this tool). -### Why? -This library came out of a pain-point my coworkers and I were running into-during development: Writing integration tests for our APIs was generally-annoying. They were time consuming to write especially considering how basic-they were, and we are a small startup where developer time is in short supply.-Over time, we found ourselves sometimes just writing bash scripts that would-`curl` our various endpoints and check the output with very basic matchers.-These tests were fast to write, but quickly became difficult to maintain as-complexity was added. Not only did maintenance become challenging, but the whole-system was very error prone and confidence in the tests overall was decreasing.-At the end of the day, we needed to just curl some endpoints and verify the-output looks sane, and do this quickly and correctly. This is precisely the goal-of curl-runnings.--Now you can write your tests just as data in a yaml or json file,-and curl-runnings will take care of the rest!--While yaml/json is the current way to write curl-runnings tests, this project is-being built in a way that should lend itself well to an embedded domain specific-language, which is a future goal for the project. curl-runnings specs in Dhall-is also being developed and may fulfill the same needs.- ### Installing The best way to install curl-runnings is with the [scarf](https://scarf.sh)@@ -55,9 +38,30 @@ Curl runnings tests are just data! A test spec is an object containing an array of `cases`, where each item represents a single curl and set of assertions about-the response. Write your tests specs in a yaml or json file. Note: the legacy+the response. Write your tests specs in a Dhall, yaml or json file. Note: the legacy format of a top level array of test cases is still supported, but may not be in future releases.++```dhall+let JSON = https://prelude.dhall-lang.org/JSON/package.dhall++let CurlRunnings = ./dhall/curl-runnings.dhall++in CurlRunnings.hydrateCase+ CurlRunnings.Case::{+ , expectData = Some+ ( CurlRunnings.ExpectData.Exactly+ ( JSON.object+ [ { mapKey = "okay", mapValue = JSON.bool True },+ { mapKey = "message", mapValue = JSON.string "a message" }]+ )+ )+ , expectStatus = 200+ , name = "test 1"+ , requestMethod = CurlRunnings.HttpMethod.GET+ , url = "http://your-endpoing.com/status"+ }+``` ```yaml
curl-runnings.cabal view
@@ -4,10 +4,10 @@ -- -- see: https://github.com/sol/hpack ----- hash: fa38d9aa3670cced2c7c5d811b17d2c7e74c4f05119c7090c2fedbffc4ce33b1+-- hash: aabd9fe5d4e540623836d06dc044e10c6cb87610029be5629a67355670b24c5d name: curl-runnings-version: 0.15.0+version: 0.16.0 synopsis: A framework for declaratively writing curl based API tests description: Please see the README on Github at <https://github.com/aviaviavi/curl-runnings#readme> category: Testing@@ -15,13 +15,14 @@ bug-reports: https://github.com/aviaviavi/curl-runnings/issues author: Avi Press maintainer: mail@avi.press-copyright: 2018 Avi Press+copyright: 2020 Avi Press license: MIT license-file: LICENSE build-type: Simple extra-source-files: README.md examples/auth.yaml+ examples/example-spec.dhall examples/example-spec.json examples/example-spec.yaml examples/importable.yaml@@ -52,7 +53,10 @@ , case-insensitive >=0.2.1 , clock >=0.7.2 , connection >=0.2.8+ , dhall >=1.8.2+ , dhall-json >=1.0.9 , directory >=1.3.0.2+ , hashable >=1.3.0.0 , hspec >=2.4.4 , hspec-expectations >=0.8.2 , http-client-tls >=0.3.5.3@@ -63,6 +67,7 @@ , regex-posix >=0.95.2 , text >=1.2.2.2 , time >=1.8.0.2+ , transformers >=0.5.2.0 , unordered-containers >=0.2.8.0 , vector >=0.12.0 , yaml >=0.8.28
+ examples/example-spec.dhall view
@@ -0,0 +1,97 @@+-- Your curl-runnings specs can be written in dhall, which can give you great+-- type safety and interpolation abilities.++-- The curl-runnings dhall module offers some of the types and functions that+-- can give you extra safety, or you can target the json specification directly+-- if you prefer. You can import the dhall module directly or via url:+-- https://raw.githubusercontent.com/aviaviavi/curl-runnings/master/dhall/curl-runnings.dhall+let JSON = https://prelude.dhall-lang.org/JSON/package.dhall++let CurlRunnings = ./dhall/curl-runnings.dhall++let List/map = https://prelude.dhall-lang.org/List/map++let Optional/map = https://prelude.dhall-lang.org/Optional/map++let Map = https://prelude.dhall-lang.org/Map/Type++let host = "https://tabdextension.com"++in CurlRunnings.hydrateCases+ [ CurlRunnings.Case::{+ , expectData = Some+ ( CurlRunnings.ExpectData.Exactly+ ( JSON.object+ [ { mapKey = "ping", mapValue = JSON.string "pong" } ]+ )+ )+ , expectStatus = 200+ , name = "test 1"+ , requestMethod = CurlRunnings.HttpMethod.GET+ , url = host ++ "/ping"+ }+ , CurlRunnings.Case::{+ , expectData = Some+ ( CurlRunnings.ExpectData.Contains+ [ CurlRunnings.PartialMatcher.KeyMatch "ping"+ , CurlRunnings.PartialMatcher.ValueMatch (JSON.string "pong")+ , CurlRunnings.PartialMatcher.KeyValueMatch+ { key = "ping", value = JSON.string "pong" }+ ]+ )+ , expectStatus = 200+ , name = "test 2"+ , requestMethod = CurlRunnings.HttpMethod.GET+ , url = host ++ "/ping"+ }+ , CurlRunnings.Case::{+ , expectData = Some+ ( CurlRunnings.ExpectData.NotContains+ [ CurlRunnings.PartialMatcher.KeyMatch "poing"+ , CurlRunnings.PartialMatcher.ValueMatch (JSON.string "pongg")+ , CurlRunnings.PartialMatcher.KeyValueMatch+ { key = "ping", value = JSON.string "poong" }+ ]+ )+ , expectStatus = 200+ , name = "test 3"+ , requestMethod = CurlRunnings.HttpMethod.GET+ , url = host ++ "/ping"+ }+ , CurlRunnings.Case::{+ , expectData = Some+ ( CurlRunnings.ExpectData.MixedContains+ { contains =+ [ CurlRunnings.PartialMatcher.KeyMatch "ping"+ , CurlRunnings.PartialMatcher.ValueMatch+ (JSON.string "\$<RESPONSES[-1].ping>")+ , CurlRunnings.PartialMatcher.KeyValueMatch+ { key = "ping", value = JSON.string "pong" }+ ]+ , notContains =+ [ CurlRunnings.PartialMatcher.KeyMatch "poing"+ , CurlRunnings.PartialMatcher.ValueMatch (JSON.string "pongg")+ , CurlRunnings.PartialMatcher.KeyValueMatch+ { key = "ping", value = JSON.string "poong" }+ ]+ }+ )+ , expectStatus = 200+ , name = "test 4"+ , requestMethod = CurlRunnings.HttpMethod.GET+ , url = host ++ "/ping"+ }+ , CurlRunnings.Case::{+ , expectStatus = 405+ , name = "test 5"+ , requestMethod = CurlRunnings.HttpMethod.POST+ , url = host ++ "/ping"+ , queryParameters = [ { mapKey = "test", mapValue = "asdf" } ]+ , requestData = Some+ ( CurlRunnings.RequestData.JSON+ ( JSON.object+ [ { mapKey = "key", mapValue = JSON.string "value" } ]+ )+ )+ }+ ]
src/Testing/CurlRunnings.hs view
@@ -13,8 +13,12 @@ , decodeFile ) where +import Control.Arrow import Control.Exception+import qualified Control.Exception import Control.Monad+import Control.Monad.IO.Class+import Control.Monad.Trans.Except import Data.Aeson import Data.Aeson.Types import qualified Data.ByteString.Base64 as B64@@ -28,9 +32,19 @@ import Data.Monoid import qualified Data.Text as T import qualified Data.Text.Encoding as T+import qualified Data.Text.IO as TIO+import qualified Data.Text.Lazy as TL+import qualified Data.Text.Lazy.IO as TLIO import Data.Time.Clock import qualified Data.Vector as V+import qualified Data.Vector as V+import qualified Data.Yaml as Y import qualified Data.Yaml.Include as YI+import qualified Dhall+import qualified Dhall.Import+import qualified Dhall.JSON+import qualified Dhall.Parser+import qualified Dhall.TypeCheck import Network.Connection (TLSSettings (..)) import Network.HTTP.Client.TLS (mkManagerSettings) import Network.HTTP.Conduit@@ -45,8 +59,7 @@ import Text.Printf import Text.Regex.Posix ---- | decode a json or yaml file into a suite object+-- | decode a json, yaml, or dhall file into a suite object decodeFile :: FilePath -> IO (Either String CurlSuite) decodeFile specPath = doesFileExist specPath >>= \exists ->@@ -56,9 +69,31 @@ eitherDecode' <$> B.readFile specPath :: IO (Either String CurlSuite) "yaml" -> mapLeft show <$> YI.decodeFileEither specPath "yml" -> mapLeft show <$> YI.decodeFileEither specPath+ "dhall" -> do+ runExceptT $ do+ let showErrorWithMessage :: (Show a) => String -> a -> String+ showErrorWithMessage message err = message ++ ": " ++ (show err)+ raw <- liftIO $ TIO.readFile specPath+ expr <-+ withExceptT (showErrorWithMessage "parser") . ExceptT . return $+ Dhall.Parser.exprFromText "dhall parser" (raw :: Dhall.Text)+ expr' <- liftIO $ Dhall.Import.load expr+ ExceptT $+ return $ do+ _ <-+ left (showErrorWithMessage "typeof") $+ Dhall.TypeCheck.typeOf expr'+ val <-+ left (showErrorWithMessage "to json") $+ Dhall.JSON.dhallToJSON expr'+ left (showErrorWithMessage "from json") . resultToEither $+ fromJSON val _ -> return . Left $ printf "Invalid spec path %s" specPath else return . Left $ printf "%s not found" specPath +resultToEither :: Result a -> Either String a+resultToEither (Error s) = Left s+resultToEither (Success a) = Right a noVerifyTlsManagerSettings :: ManagerSettings noVerifyTlsManagerSettings = mkManagerSettings noVerifyTlsSettings Nothing
src/Testing/CurlRunnings/Internal.hs view
@@ -71,7 +71,7 @@ type CurlRunningsUnsafeLogger a = (LogLevel -> T.Text -> a -> a) makeLogger :: LogLevel -> CurlRunningsLogger-makeLogger threshold level text = when (level <= threshold) $ P.pPrint text+makeLogger threshold level text = when (level <= threshold) $ putStrLn $ T.unpack text makeUnsafeLogger :: Show a => LogLevel -> CurlRunningsUnsafeLogger a makeUnsafeLogger threshold level text object =
src/Testing/CurlRunnings/Types.hs view
@@ -41,6 +41,7 @@ import Data.Aeson import Data.Aeson.Types import qualified Data.Char as C+import Data.Hashable (Hashable) import qualified Data.HashMap.Strict as H import Data.Maybe import Data.Monoid@@ -82,15 +83,19 @@ instance FromJSON JsonMatcher where parseJSON (Object v)- | isJust $ H.lookup "exactly" v = Exactly <$> v .: "exactly"- | isJust (H.lookup "contains" v) && isJust (H.lookup "notContains" v) = do- c <- Contains <$> v .: "contains"- n <- NotContains <$> v .: "notContains"- return $ MixedContains [c, n]- | isJust $ H.lookup "contains" v = Contains <$> v .: "contains"- | isJust $ H.lookup "notContains" v = NotContains <$> v .: "notContains"+ | justAndNotEmpty "exactly" v = Exactly <$> v .: "exactly"+ | justAndNotEmpty "contains" v && justAndNotEmpty "notContains" v = do+ c <- Contains <$> v .: "contains"+ n <- NotContains <$> v .: "notContains"+ return $ MixedContains [c, n]+ | justAndNotEmpty "contains" v = Contains <$> v .: "contains"+ | justAndNotEmpty "notContains" v = NotContains <$> v .: "notContains" parseJSON invalid = typeMismatch "JsonMatcher" invalid +justAndNotEmpty :: (Eq k, Hashable k) => k -> H.HashMap k Value -> Bool+justAndNotEmpty key obj =+ (isJust $ H.lookup key obj) && (H.lookup key obj /= Just Null)+ -- | Simple predicate to check value constructor type isContains :: JsonMatcher -> Bool isContains (Contains _) = True@@ -171,18 +176,19 @@ instance FromJSON JsonSubExpr where parseJSON (Object v)- | isJust $ H.lookup "keyValueMatch" v =+ | justAndNotEmpty "keyValueMatch" v = let toParse = fromJust $ H.lookup "keyValueMatch" v in case toParse of Object o -> KeyValueMatch <$> o .: "key" <*> o .: "value" _ -> typeMismatch "JsonSubExpr" toParse- | isJust $ H.lookup "keyMatch" v =+ | justAndNotEmpty "keyMatch" v = let toParse = fromJust $ H.lookup "keyMatch" v in case toParse of String s -> return $ KeyMatch s _ -> typeMismatch "JsonSubExpr" toParse- | isJust $ H.lookup "valueMatch" v = ValueMatch <$> v .: "valueMatch"+ | justAndNotEmpty "valueMatch" v = ValueMatch <$> v .: "valueMatch" parseJSON invalid = typeMismatch "JsonSubExpr" invalid+ instance ToJSON JsonSubExpr -- | Check the status code of a response. You can specify one or many valid codes.
test/Spec.hs view
@@ -28,6 +28,9 @@ it "should provide valid example json specs" $ testValidSpec "/examples/example-spec.json" + it "should provide valid example dhall specs" $+ testValidSpec "/examples/example-spec.dhall"+ -- note that this doesn't actually try to parse the interpolations themselves, -- but that would be useful thing to add here it "should provid a valid interpolation spec" $