hspec-golden-aeson 0.7.0.0 → 0.9.0.0
raw patch · 7 files changed
+263/−50 lines, 7 filesdep +HUnitnew-uploader
Dependencies added: HUnit
Files
- ChangeLog.md +80/−0
- README.md +99/−0
- hspec-golden-aeson.cabal +10/−5
- src/Test/Aeson/Internal/ADT/GoldenSpecs.hs +30/−20
- src/Test/Aeson/Internal/GoldenSpecs.hs +25/−12
- src/Test/Aeson/Internal/RandomSamples.hs +7/−9
- src/Test/Aeson/Internal/Utils.hs +12/−4
+ ChangeLog.md view
@@ -0,0 +1,80 @@+# Revision history for hspec-golden-aeson++## 0.9.0.0 -- 2021-03-15+* Breaking change: Objects are now serialized with sorted keys for better+ cross-platform compatibility++## 0.8.0.0 -- 2021-03-12++* Breaking change: Seed is now an `Int32` so golden files are more portable. This+ requires regenerating all golden files which have a seed that overflows+* Breaking change: Golden files are no longer generated automatically if they+ don't exist, to create them, set the `CREATE_MISSING_GOLDEN` environment variable.+ This is to prevent missing golden files from silently making golden tests+ degrade to round-trip tests+* Add a `RECREATE_MISSING_GOLDEN` environemnt variable. When present it will+ cause golden files to be re-created if they cause the test to fail. This is+ useful for updating golden files when serialization has been purposedly+ modified and to update the seed if it breaks due to overflow now that it is+ only 32bit wide.++## 0.7.0.0 -- 2018-05-17++* Breaking change: allow roundtripAndGoldenADTSpecs test to pass when random samples generated from the seed in the golden file do not produce the same Haskell samples, but yet decoding and re-encoding the golden file still produces the same bytes as in the golden file.+* Add an additional faulty file ending in `.faulty.reencoded.json` when the byte-for-byte decode/encode round-trip fails. This allows you to compare the encoding changes without the noise of the random sample change. In this case, the test will output a message indicating whether decoding the golden file produces the same Haskell values as decoding the re-encoded files. If they produce the same values, that is likely a minor encoding change, but still a change so tests fail.+* Add `RandomMismatchOption` to `Settings` so you can have the old behavior of failing tests when random samples change.++## 0.6.0.0 -- 2018-01-04++* Test encoding in `roundtripAndGoldenADTSpecs' and 'roundtripAndGoldenADTSpecsWithSettings` functions. This may break current tests because only decoding was tested previously.++## 0.5.1.0 -- 2018-01-04++* Remove 'Wredundant-constraints' flag.++## 0.5.0.0 -- 2018-01-02++* Add 'Arbitrary' requirement for 'roundtripADTSpecs', 'roundtripAndGoldenADTSpecs' and 'roundtripAndGoldenADTSpecsWithSettings' because 'Arbitrary' was a redundant constrain for 'ToADTArbitrary' in quickcheck-arbitrary-adt.++## 0.4.0.0 -- 2017-12-10++* Fix behavior for 'mkGoldenFileForType'. Intention is to create a file in a dir for each constructor, but it was only creating a file for one of the constructors of a type.++## 0.3.1.0 -- 2017-12-02++* Expose 'roundtripAndGoldenSpecsWithSettings'.++## 0.3.0.0 -- 2017-11-14++* Add mkGoldenFileForType.+* Rename internal function fromTypeable to mkTypeNameInfo.+* Move TopDir, ModuleName, TypeName, TypeNameInfo and mkTypeNameInfo into Test.Aeson.Internal.Util.++## 0.2.1.0 -- 2017-08-08++* Added the ability to run an automated test withought needing a Show, Eq, or Typeable instance.+* Cleaned up error messages, mostly involving redundant types++## 0.2.0.3 -- 2016-09-08++* Tests were breaking because Test.Types.MismatchedToAndFromSerialization was+missing from the cabal file.++## 0.2.0.2 -- 2016-09-07++* Forgot to add fixes to the test, in previous version they were not compiling.++## 0.2.0.1 -- 2016-09-01++* Fix error in 'goldenSpecsWithNote', behavior for useModuleNameAsSubDirectory was flipped around and also providing the module name surrounded by quotes.+* Add more tests.++## 0.2.0.0 -- 2016-08-23++* Make directory and golden file naming flexible.+* Include optional Settings type that allows users to set the directory name and size of tests.+* Require quickcheck-arbitrary-adt >= 0.2.0.0.++## 0.1.0.0 -- 2016-08-12++* First version.
+ README.md view
@@ -0,0 +1,99 @@+# hspec-golden-aeson++## Arbitrary++GoldenSpecs functions write golden files if they do not exist. When they do+exist they use a seed from the golden file to create an arbitrary value of a+type and check if the serialization matches the file. If it fails it means+that there has been a change in the Aeson serialization or a change in the+data type.++RoundtripSpecs make sure that a type is able to be encoded to JSON, decoded+from JSON back to the original type, and equal the same value. If it fails+then there may be an issue with the ToJSON and FromJSON instances.++## ToADTArbitrary++`ToADTArbitrary` is a type class that helps create arbitrary values for every+constructor in a type. GoldenADTSpecs and RoundtripADTSpecs function similarly+to their `Arbitrary` counterparts, but will specifically test each constructor.+This is very useful for sum types.++## Usage++```haskell+{-# LANGUAGE DeriveGeneric #-}++-- base+import GHC.Generics (Generic)+import Data.Proxy++-- aeson+import Data.Aeson (ToJSON)++-- QuickCheck+import Test.QuickCheck (Arbitrary (..), oneof)++-- quickcheck-arbitrary-adt+import Test.QuickCheck.Arbitrary.ADT (ToADTArbitrary)++-- hspec-golden-aeson+import Test.Aeson.GenericSpecs (mkGoldenFileForType)++-- product type+data Person =+ Person+ { name :: String+ , address :: String+ , age :: Int+ } deriving (Eq,Read,Show,Generic)++instance ToJSON Person++-- derive ToADTArbitrary generically+instance ToADTArbitrary Person+instance Arbitrary Person where+ arbitrary = Person <$> arbitrary <*> arbitrary <*> arbitrary++-- sum type+data OnOrOff+ = On+ | Off+ deriving (Eq,Read,Show,Generic)++instance ToJSON OnOrOff++-- derive ToADTArbitrary generically+instance ToADTArbitrary OnOrOff+instance Arbitrary OnOrOff where+ arbitrary = oneof [pure On, pure Off]+++main :: IO ()+main = do+ -- only create the golden files, do not run tests+ mkGoldenFileForType 10 (Proxy :: Proxy Person) "golden"+ mkGoldenFileForType 10 (Proxy :: Proxy OnOrOff) "golden"+ + -- for an arbitrary instance of each constructor+ -- make sure that ToJSON and FromJSON are defined such that + -- they same value is maintained+ roundtripSpecs (Proxy :: Proxy Person)+ roundtripSpecs (Proxy :: Proxy OnOrOff)++ -- the first time it create files if they do not exist+ -- if they exist then it will test serialization against the files along with the roundtrip tests+ -- files are saved in "golden" dir.+ goldenSpecs (Proxy :: Proxy Person)+ goldenSpecs (Proxy :: Proxy OnOrOff)+ + -- use the module name as a dir when saving and opening files+ goldenSpecs (defaultSettings { useModuleNameAsSubDirectory = True }) (Proxy :: Proxy Person)+ goldenSpecs (defaultSettings { useModuleNameAsSubDirectory = True }) (Proxy :: Proxy OnOrOff)+ + -- control the location of the golde files+ let topDir = "json-tests"+ goldenSpecs (defaultSettings {goldenDirectoryOption = CustomDirectoryName topDir}) (Proxy :: Proxy Person)+ goldenSpecs (defaultSettings {goldenDirectoryOption = CustomDirectoryName topDir}) (Proxy :: Proxy OnOrOff)++```
hspec-golden-aeson.cabal view
@@ -1,11 +1,13 @@--- This file has been generated from package.yaml by hpack version 0.28.2.+cabal-version: 1.12++-- This file has been generated from package.yaml by hpack version 0.33.0. -- -- see: https://github.com/sol/hpack ----- hash: 96e9eaa9508a1b5c37e6826a1052424945f28adfd1feed94092e594e5c9d96fd+-- hash: 11e24a472a71971f598d5ce65cdafa167658b0e163868ee1de71561acdc8301f name: hspec-golden-aeson-version: 0.7.0.0+version: 0.9.0.0 synopsis: Use tests to monitor changes in Aeson serialization description: Use tests to monitor changes in Aeson serialization category: Testing@@ -17,7 +19,9 @@ license: BSD3 license-file: LICENSE build-type: Simple-cabal-version: >= 1.10+extra-source-files:+ ChangeLog.md+ README.md source-repository head type: git@@ -38,7 +42,8 @@ src ghc-options: -Wall build-depends:- QuickCheck+ HUnit+ , QuickCheck , aeson , aeson-pretty , base >=4.7 && <5
src/Test/Aeson/Internal/ADT/GoldenSpecs.hs view
@@ -24,20 +24,22 @@ import Control.Monad import Data.Aeson (ToJSON, FromJSON)-import qualified Data.Aeson as A-import Data.Aeson.Encode.Pretty import Data.ByteString.Lazy (writeFile, readFile)+import Data.Int (Int32)+import Data.Maybe (isJust) import Data.Proxy import Prelude hiding (writeFile,readFile) import System.Directory+import System.Environment (lookupEnv) import System.FilePath import System.Random import Test.Aeson.Internal.RandomSamples import Test.Aeson.Internal.Utils import Test.Hspec+import Test.HUnit.Lang (HUnitFailure) import Test.QuickCheck import Test.QuickCheck.Arbitrary.ADT @@ -72,12 +74,24 @@ -- | test a single set of values from a constructor for a given type. testConstructor :: forall a. (Eq a, Show a, FromJSON a, ToJSON a, ToADTArbitrary a) => Settings -> String -> String -> ConstructorArbitraryPair a -> SpecWith ( Arg (IO ()))-testConstructor Settings{..} moduleName typeName cap = do+testConstructor Settings{..} moduleName typeName cap = it ("produces the same JSON as is found in " ++ goldenFile) $ do exists <- doesFileExist goldenFile+ let fixIfFlag err = do+ doFix <- isJust <$> lookupEnv "RECREATE_BROKEN_GOLDEN"+ if doFix+ then createGoldenFile sampleSize cap goldenFile+ else throwIO err if exists then compareWithGolden randomMismatchOption topDir mModuleName typeName cap goldenFile- else createGoldenFile sampleSize cap goldenFile+ `catches` [ Handler (\(err :: HUnitFailure) -> fixIfFlag err)+ , Handler (\(err :: AesonDecodeError) -> fixIfFlag err)+ ]+ else do+ doCreate <- isJust <$> lookupEnv "CREATE_MISSING_GOLDEN"+ if doCreate+ then createGoldenFile sampleSize cap goldenFile+ else expectationFailure $ "Missing golden file: " <> goldenFile where goldenFile = mkGoldenFilePath topDir mModuleName typeName cap topDir = case goldenDirectoryOption of@@ -97,13 +111,11 @@ newSamples <- mkRandomADTSamplesForConstructor sampleSize (Proxy :: Proxy a) (capConstructor cap) goldenSeed whenFails (writeComparisonFile newSamples) $ do goldenBytes <- readFile goldenFile- goldenSamples :: RandomSamples a <-- either (throwIO . ErrorCall) return $- A.eitherDecode' goldenBytes+ goldenSamples :: RandomSamples a <- aesonDecodeIO goldenBytes if newSamples == goldenSamples then -- random samples match; test encoding of samples (the above check only tested the decoding)- encodePretty newSamples == goldenBytes `shouldBe` True+ encodePrettySortedKeys newSamples == goldenBytes `shouldBe` True else do let -- whether to pass the test or fail due to random value mismatch@@ -118,7 +130,7 @@ "\n" ++ "WARNING: New random samples do not match those in " ++ goldenFile ++ ".\n" ++ " Testing round-trip decoding/encoding of golden file."- let reencodedGoldenSamples = encodePretty goldenSamples+ let reencodedGoldenSamples = encodePrettySortedKeys goldenSamples if reencodedGoldenSamples == goldenBytes then -- pass the test because round-trip decode/encode still gives the same bytes@@ -126,9 +138,7 @@ else do -- how significant is the serialization change? writeReencodedComparisonFile goldenSamples- testSamples :: RandomSamples a <-- either (throwIO . ErrorCall) return $- A.eitherDecode' reencodedGoldenSamples+ testSamples :: RandomSamples a <- aesonDecodeIO reencodedGoldenSamples let failureMessage = if testSamples == goldenSamples@@ -146,12 +156,12 @@ faultyReencodedFile = mkFaultyReencodedFilePath topDir mModuleName typeName cap writeComparisonFile newSamples = do- writeFile faultyFile (encodePretty newSamples)+ writeFile faultyFile (encodePrettySortedKeys newSamples) putStrLn $ "\n" ++ "INFO: Written the current encodings into " ++ faultyFile ++ "." writeReencodedComparisonFile samples = do- writeFile faultyReencodedFile (encodePretty samples)+ writeFile faultyReencodedFile (encodePrettySortedKeys samples) putStrLn $ "\n" ++ "INFO: Written the re-encodings into " ++ faultyReencodedFile ++ "."@@ -161,9 +171,9 @@ Int -> ConstructorArbitraryPair a -> FilePath -> IO () createGoldenFile sampleSize cap goldenFile = do createDirectoryIfMissing True (takeDirectory goldenFile)- rSeed <- randomIO :: IO Int+ rSeed <- randomIO :: IO Int32 rSamples <- mkRandomADTSamplesForConstructor sampleSize (Proxy :: Proxy a) (capConstructor cap) rSeed- writeFile goldenFile $ encodePretty rSamples+ writeFile goldenFile $ encodePrettySortedKeys rSamples putStrLn $ "\n" ++@@ -202,7 +212,7 @@ -- | Create a number of arbitrary instances of a particular constructor given -- a sample size and a random seed. mkRandomADTSamplesForConstructor :: forall a. (ToADTArbitrary a) =>- Int -> Proxy a -> String -> Int -> IO (RandomSamples a)+ Int -> Proxy a -> String -> Int32 -> IO (RandomSamples a) mkRandomADTSamplesForConstructor sampleSize Proxy conName rSeed = do generatedADTs <- generate gen let caps = concat $ adtCAPs <$> generatedADTs@@ -211,7 +221,7 @@ return $ RandomSamples rSeed arbs where correctedSampleSize = if sampleSize <= 0 then 1 else sampleSize- gen = setSeed rSeed $ replicateM correctedSampleSize (toADTArbitrary (Proxy :: Proxy a))+ gen = setSeed (fromIntegral rSeed) $ replicateM correctedSampleSize (toADTArbitrary (Proxy :: Proxy a)) -- | Make a Golden File for the Proxy of a type if the file does not exist. mkGoldenFileForType :: forall a. (ToJSON a, ToADTArbitrary a) => Int -> Proxy a -> FilePath -> IO ()@@ -225,7 +235,7 @@ then pure () else do createDirectoryIfMissing True (takeDirectory goldenFile)- rSeed <- randomIO :: IO Int+ rSeed <- randomIO :: IO Int32 rSamples <- mkRandomADTSamplesForConstructor sampleSize (Proxy :: Proxy a) (capConstructor constructor) rSeed- writeFile goldenFile $ encodePretty rSamples+ writeFile goldenFile $ encodePrettySortedKeys rSamples ) constructors
src/Test/Aeson/Internal/GoldenSpecs.hs view
@@ -21,20 +21,23 @@ import Control.Monad import Data.Aeson-import Data.Aeson.Encode.Pretty import Data.ByteString.Lazy hiding (putStrLn)+import Data.Int (Int32)+import Data.Maybe (isJust) import Data.Proxy import Data.Typeable import Prelude hiding (readFile, writeFile) import System.Directory+import System.Environment (lookupEnv) import System.FilePath import System.Random import Test.Aeson.Internal.RandomSamples import Test.Aeson.Internal.Utils import Test.Hspec+import Test.HUnit.Lang (HUnitFailure) import Test.QuickCheck -- | Tests to ensure that JSON encoding has not unintentionally changed. This@@ -73,9 +76,21 @@ describe ("JSON encoding of " ++ addBrackets (unTypeName typeNameTypeName) ++ note) $ it ("produces the same JSON as is found in " ++ goldenFile) $ do exists <- doesFileExist goldenFile+ let fixIfFlag err = do+ doFix <- isJust <$> lookupEnv "RECREATE_BROKEN_GOLDEN"+ if doFix+ then createGoldenfile settings proxy goldenFile+ else throwIO err if exists then compareWithGolden typeNameInfo proxy goldenFile comparisonFile- else createGoldenfile settings proxy goldenFile+ `catches` [ Handler (\(err :: HUnitFailure) -> fixIfFlag err)+ , Handler (\(err :: AesonDecodeError) -> fixIfFlag err)+ ]+ else do+ doCreate <- isJust <$> lookupEnv "CREATE_MISSING_GOLDEN"+ if doCreate+ then createGoldenfile settings proxy goldenFile+ else expectationFailure $ "Missing golden file: " <> goldenFile -- | The golden files already exist. Serialize values with the same seed from@@ -89,10 +104,8 @@ newSamples <- mkRandomSamples sampleSize proxy goldenSeed whenFails (writeComparisonFile newSamples) $ do goldenBytes <- readFile goldenFile- goldenSamples :: RandomSamples a <-- either (throwIO . ErrorCall) return $- eitherDecode' goldenBytes- if encode newSamples == encode goldenSamples+ goldenSamples :: RandomSamples a <- aesonDecodeIO goldenBytes+ if encodePrettySortedKeys newSamples == encodePrettySortedKeys goldenSamples then return () else do -- fallback to testing roundtrip decoding/encoding of golden file@@ -100,7 +113,7 @@ "\n" ++ "WARNING: Encoding new random samples do not match " ++ goldenFile ++ ".\n" ++ " Testing round-trip decoding/encoding of golden file."- if encodePretty goldenSamples == goldenBytes+ if encodePrettySortedKeys goldenSamples == goldenBytes then return () else do writeReencodedComparisonFile goldenSamples@@ -114,12 +127,12 @@ OverwriteGoldenFile -> goldenFile faultyReencodedFilePath = mkFaultyReencodedFile typeNameInfo writeComparisonFile newSamples = do- writeFile filePath (encodePretty newSamples)+ writeFile filePath (encodePrettySortedKeys newSamples) putStrLn $ "\n" ++ "INFO: Written the current encodings into " ++ filePath ++ "." writeReencodedComparisonFile samples = do- writeFile faultyReencodedFilePath (encodePretty samples)+ writeFile faultyReencodedFilePath (encodePrettySortedKeys samples) putStrLn $ "\n" ++ "INFO: Written the reencoded goldenFile into " ++ faultyReencodedFilePath ++ "."@@ -131,7 +144,7 @@ createDirectoryIfMissing True (takeDirectory goldenFile) rSeed <- randomIO rSamples <- mkRandomSamples sampleSize proxy rSeed- writeFile goldenFile (encodePretty rSamples)+ writeFile goldenFile (encodePrettySortedKeys rSamples) putStrLn $ "\n" ++@@ -170,9 +183,9 @@ -- | Create a number of arbitrary instances of a type -- a sample size and a random seed. mkRandomSamples :: forall a . Arbitrary a =>- Int -> Proxy a -> Int -> IO (RandomSamples a)+ Int -> Proxy a -> Int32 -> IO (RandomSamples a) mkRandomSamples sampleSize Proxy rSeed = RandomSamples rSeed <$> generate gen where correctedSampleSize = if sampleSize <= 0 then 1 else sampleSize gen :: Gen [a]- gen = setSeed rSeed $ replicateM correctedSampleSize (arbitrary :: Gen a)+ gen = setSeed (fromIntegral rSeed) $ replicateM correctedSampleSize (arbitrary :: Gen a)
src/Test/Aeson/Internal/RandomSamples.hs view
@@ -11,13 +11,15 @@ {-# LANGUAGE DeriveGeneric #-} {-# LANGUAGE ScopedTypeVariables #-}+{-# LANGUAGE TypeApplications #-} module Test.Aeson.Internal.RandomSamples where -import Control.Exception+import Test.Aeson.Internal.Utils (aesonDecodeIO) import Data.Aeson import Data.ByteString.Lazy (ByteString)+import Data.Int (Int32) import GHC.Generics @@ -31,7 +33,7 @@ -- try to reproduce the same samples by generating the arbitraries with a seed. data RandomSamples a = RandomSamples {- seed :: Int+ seed :: Int32 , samples :: [a] } deriving (Eq, Ord, Show, Generic) @@ -43,13 +45,9 @@ setSeed rSeed (MkGen g) = MkGen $ \ _randomSeed size -> g (mkQCGen rSeed) size -- | Reads the seed without looking at the samples.-readSeed :: ByteString -> IO Int-readSeed s = case eitherDecode s :: Either String (RandomSamples Value) of- Right rSamples -> return $ seed rSamples- Left err -> throwIO $ ErrorCall err+readSeed :: ByteString -> IO Int32+readSeed = fmap seed . aesonDecodeIO @(RandomSamples Value) -- | Read the sample size. readSampleSize :: ByteString -> IO Int-readSampleSize s = case eitherDecode s :: Either String (RandomSamples Value) of- Right rSamples -> return . length . samples $ rSamples- Left err -> throwIO $ ErrorCall err+readSampleSize = fmap (length . samples) . aesonDecodeIO @(RandomSamples Value)
src/Test/Aeson/Internal/Utils.hs view
@@ -16,6 +16,7 @@ import Control.Exception import Data.Aeson+import Data.Aeson.Encode.Pretty import Data.ByteString.Lazy (ByteString) import Data.Proxy import Data.Typeable@@ -74,8 +75,8 @@ -- roundtrip tests. shouldBeIdentity :: (Eq a, Show a, Arbitrary a) => Proxy a -> (a -> IO a) -> Property-shouldBeIdentity Proxy function =- property $ \ (a :: a) -> function a `shouldReturn` a+shouldBeIdentity Proxy func =+ property $ \ (a :: a) -> func a `shouldReturn` a -- | This function will compare one JSON encoding to a subsequent JSON encoding, thus eliminating the need for an Eq instance checkAesonEncodingEquality :: forall a . (ToJSON a, FromJSON a) => JsonShow a -> Bool@@ -89,9 +90,13 @@ aesonDecodeIO :: FromJSON a => ByteString -> IO a aesonDecodeIO bs = case eitherDecode bs of Right a -> return a- Left msg -> throwIO $ ErrorCall- ("aeson couldn't parse value: " ++ msg)+ Left msg -> throwIO $ AesonDecodeError msg +data AesonDecodeError = AesonDecodeError String+ deriving (Show, Eq)++instance Exception AesonDecodeError+ -- | Used to eliminate the need for an Eq instance newtype JsonShow a = JsonShow a @@ -153,3 +158,6 @@ case goldenDirectoryOption of GoldenDirectory -> "golden" CustomDirectoryName d -> d++encodePrettySortedKeys :: ToJSON a => a -> ByteString+encodePrettySortedKeys = encodePretty' defConfig { confCompare = compare }