diff --git a/CHANGELOG.md b/CHANGELOG.md --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,13 @@ Changes ======= +Version 1.4.2 +------------- + +* Add `consoleTestReporterWithHook` +* Suggest pattern to rerun an individual failing test +* Add `Test.Tasty.Patterns.Printer` + Version 1.4.1 ------------- @@ -21,7 +28,7 @@ --------------- The only point of this release is to introduce compatibility with GHCs back to 7.0 -(see https://github.com/feuerbach/tasty/pull/287). +(see https://github.com/UnkindPartition/tasty/pull/287). Note, however, that these changes are not merged to the master branch, and the future releases will only support the GHC/base versions from the last 5 years, @@ -120,7 +127,7 @@ hopefully the breakage won't be too big. I'm sorry about any problems caused by the change. -See for the discussion. +See for the discussion. * The field separator in patterns is changed from slash (`/`) to period (`.`), and `.` is now allowed in raw patterns. @@ -345,7 +352,7 @@ --------------- Be careful not to export the `Show (a -> b)` instance, see - + Version 0.8.1.2 --------------- @@ -449,7 +456,7 @@ Version 0.4.0.1 --------------- -Fix a bug ([#25](https://github.com/feuerbach/tasty/issues/25)) +Fix a bug ([#25](https://github.com/UnkindPartition/tasty/issues/25)) Version 0.4 ----------- diff --git a/Control/Concurrent/Async.hs b/Control/Concurrent/Async.hs --- a/Control/Concurrent/Async.hs +++ b/Control/Concurrent/Async.hs @@ -266,4 +266,4 @@ {-# INLINE rawForkIO #-} rawForkIO :: IO () -> IO ThreadId rawForkIO action = IO $ \ s -> - case (fork# action s) of (# s1, tid #) -> (# s1, ThreadId tid #) + case (fork# (unIO action) s) of (# s1, tid #) -> (# s1, ThreadId tid #) diff --git a/README.md b/README.md --- a/README.md +++ b/README.md @@ -18,7 +18,7 @@ To find out what's new, read the **[change log][]**. -[change log]: https://github.com/feuerbach/tasty/blob/master/core/CHANGELOG.md +[change log]: https://github.com/UnkindPartition/tasty/blob/master/core/CHANGELOG.md ## Example @@ -75,7 +75,7 @@ And here is the output of the above program: -![](https://raw.github.com/feuerbach/tasty/master/screenshot.png) +![](https://raw.github.com/UnkindPartition/tasty/master/screenshot.png) (Note that whether QuickCheck finds a counterexample to the third property is determined by chance.) @@ -103,7 +103,7 @@ (based on [smallcheck](https://hackage.haskell.org/package/smallcheck)) * [tasty-quickcheck](https://hackage.haskell.org/package/tasty-quickcheck) — for randomized property-based testing (based on [QuickCheck](http://hackage.haskell.org/package/QuickCheck)) -* [tasty-hedgehog](https://github.com/qfpl/tasty-hedgehog) — for randomized +* [tasty-hedgehog](https://hackage.haskell.org/package/tasty-hedgehog) — for randomized property-based testing (based on [Hedgehog](https://hackage.haskell.org/package/hedgehog)) * [tasty-hspec](https://hackage.haskell.org/package/tasty-hspec) — for [Hspec](https://hspec.github.io/) tests @@ -112,7 +112,11 @@ (based on [LeanCheck](https://hackage.haskell.org/package/leancheck)) * [tasty-program](https://hackage.haskell.org/package/tasty-program) — run external program and test whether it terminates successfully -* [tasty-wai](https://hackage.haskell.org/package/tasty-wai) - for testing [wai](https://hackage.haskell.org/wai) endpoints. +* [tasty-wai](https://hackage.haskell.org/package/tasty-wai) — + for testing [wai](https://hackage.haskell.org/wai) endpoints. +* [tasty-inspection-testing](https://hackage.haskell.org/package/tasty-inspection-testing) — + for compile-time testing of code properties + (based on [inspection-testing](http://hackage.haskell.org/package/inspection-testing)). [tasty-golden]: https://hackage.haskell.org/package/tasty-golden @@ -149,10 +153,13 @@ you * [tasty-hunit-adapter](https://hackage.haskell.org/package/tasty-hunit-adapter) converts existing HUnit test suites into tasty test suites -* [tasty-discover](https://github.com/lwm/tasty-discover) automatically discovers +* [tasty-discover](https://hackage.haskell.org/package/tasty-discover) automatically discovers your tests. -* [tasty-expected-failure](https://github.com/nomeata/tasty-expected-failure) provides +* [tasty-expected-failure](https://hackage.haskell.org/package/tasty-expected-failure) provides test markers for when you expect failures or wish to ignore tests. +* [tasty-bench](https://hackage.haskell.org/package/tasty-bench) covers performance +regression testing and extends `tasty` to a benchmark framework +similar to `criterion` and `gauge`. ## Options @@ -182,10 +189,11 @@ Usage: test [-p|--pattern PATTERN] [-t|--timeout DURATION] [-l|--list-tests] [-j|--num-threads NUMBER] [-q|--quiet] [--hide-successes] [--color never|always|auto] [--ansi-tricks ARG] - [--smallcheck-depth NUMBER] [--quickcheck-tests NUMBER] - [--quickcheck-replay SEED] [--quickcheck-show-replay] - [--quickcheck-max-size NUMBER] [--quickcheck-max-ratio NUMBER] - [--quickcheck-verbose] [--quickcheck-shrinks NUMBER] + [--smallcheck-depth NUMBER] [--smallcheck-max-count NUMBER] + [--quickcheck-tests NUMBER] [--quickcheck-replay SEED] + [--quickcheck-show-replay] [--quickcheck-max-size NUMBER] + [--quickcheck-max-ratio NUMBER] [--quickcheck-verbose] + [--quickcheck-shrinks NUMBER] Available options: -h,--help Show this help text @@ -200,13 +208,16 @@ the exit code --hide-successes Do not print tests that passed successfully --color never|always|auto - When to use colored output (default: 'auto') + When to use colored output (default: auto) --ansi-tricks ARG Enable various ANSI terminal tricks. Can be set to - 'true' (default) or 'false'. + 'true' or 'false'. (default: true) --smallcheck-depth NUMBER Depth to use for smallcheck tests + --smallcheck-max-count NUMBER + Maximum smallcheck test count --quickcheck-tests NUMBER - Number of test cases for QuickCheck to generate + Number of test cases for QuickCheck to generate. + Underscores accepted: e.g. 10_000_000 --quickcheck-replay SEED Random seed to use for replaying a previous test run (use same --quickcheck-max-size) --quickcheck-show-replay Show a replay token for replaying tests @@ -768,7 +779,7 @@ **A**: It is not recommended that you print anything to the console when using the console test reporter (which is the default one). - See [#103](https://github.com/feuerbach/tasty/issues/103) for the + See [#103](https://github.com/UnkindPartition/tasty/issues/103) for the discussion. Some ideas on how to work around this: @@ -784,7 +795,7 @@ output. A workaround is to disable ANSI tricks: pass `--ansi-tricks=false` on the command line or set `TASTY_ANSI_TRICKS=false` in the environment. - See [issue #152](https://github.com/feuerbach/tasty/issues/152). + See [issue #152](https://github.com/UnkindPartition/tasty/issues/152). ## Press @@ -799,7 +810,7 @@ * [Custom options in Tasty][custom-options-article] * [Resources in Tasty (update)](https://ro-che.info/articles/2013-12-29-tasty-resources-2) * [Announcing tasty-rerun](https://ocharles.org.uk/blog/posts/2014-01-20-announcing-tasty-rerun.html) -* [Code testing in Haskell revisited (with Tasty)](https://lambda.jstolarek.com/2014/01/code-testing-in-haskell-revisited-with-tasty/) +* [Code testing in Haskell revisited (with Tasty)](http://ics.p.lodz.pl/~stolarek/blog/2014/01/code-testing-in-haskell-revisited-with-tasty/) * [New patterns in tasty][awk-patterns-article] * [Screencast: Dynamic Test Suites in Haskell using Hspec and Tasty](https://www.youtube.com/watch?v=PGsDvgmZF7A) * [Automatically generated directories for tasty tests][tasty-directories] @@ -815,7 +826,7 @@ Maintainers ----------- -[Roman Cheplyaka](https://github.com/feuerbach) is the primary maintainer. +[Roman Cheplyaka](https://github.com/UnkindPartition) is the primary maintainer. [Oliver Charles](https://github.com/ocharles) is the backup maintainer. Please get in touch with him if the primary maintainer cannot be reached. diff --git a/Test/Tasty.hs b/Test/Tasty.hs --- a/Test/Tasty.hs +++ b/Test/Tasty.hs @@ -21,7 +21,7 @@ -- > assertBool "Oops, 7 is odd" (even 7) -- > ] -- --- Take a look at the : +-- Take a look at the : -- it contains a comprehensive list of test providers, a bigger example, -- and a lot of other information. diff --git a/Test/Tasty/Ingredients/ConsoleReporter.hs b/Test/Tasty/Ingredients/ConsoleReporter.hs --- a/Test/Tasty/Ingredients/ConsoleReporter.hs +++ b/Test/Tasty/Ingredients/ConsoleReporter.hs @@ -3,6 +3,7 @@ -- | Console reporter ingredient module Test.Tasty.Ingredients.ConsoleReporter ( consoleTestReporter + , consoleTestReporterWithHook , Quiet(..) , HideSuccesses(..) , AnsiTricks(..) @@ -25,7 +26,7 @@ , withConsoleFormat ) where -import Prelude hiding (fail) +import Prelude hiding (fail, EQ) import Control.Monad.State hiding (fail) import Control.Monad.Reader hiding (fail,reader) import Control.Concurrent.STM @@ -34,8 +35,12 @@ import Test.Tasty.Providers.ConsoleFormat import Test.Tasty.Run import Test.Tasty.Ingredients +import Test.Tasty.Ingredients.ListTests import Test.Tasty.Options import Test.Tasty.Options.Core +import Test.Tasty.Patterns +import Test.Tasty.Patterns.Printer +import Test.Tasty.Patterns.Types import Test.Tasty.Runners.Reducers import Test.Tasty.Runners.Utils import Text.Printf @@ -44,6 +49,7 @@ #ifdef VERSION_wcwidth import Data.Char.WCWidth (wcwidth) #endif +import Data.List (isInfixOf) import Data.Maybe import Data.Monoid (Any(..)) import Data.Typeable @@ -87,6 +93,16 @@ instance Semigroup TestOutput where (<>) = mappend +applyHook :: ([TestName] -> Result -> IO Result) -> TestOutput -> TestOutput +applyHook hook = go [] + where + go path (PrintTest name printName printResult) = + PrintTest name printName (printResult <=< hook (name : path)) + go path (PrintHeading name printName printBody) = + PrintHeading name printName (go (name : path) printBody) + go path (Seq a b) = Seq (go path a) (go path b) + go _ Skip = mempty + type Level = Int -- | Build the 'TestOutput' for a 'TestTree' and 'OptionSet'. The @colors@ @@ -377,13 +393,56 @@ -- | A simple console UI consoleTestReporter :: Ingredient -consoleTestReporter = - TestReporter - [ Option (Proxy :: Proxy Quiet) - , Option (Proxy :: Proxy HideSuccesses) - , Option (Proxy :: Proxy UseColor) - , Option (Proxy :: Proxy AnsiTricks) - ] $ +consoleTestReporter = TestReporter consoleTestReporterOptions $ \opts tree -> + let + TestPattern pattern = lookupOption opts + tests = testsNames opts tree + hook = (return .) . appendPatternIfTestFailed tests pattern + TestReporter _ cb = consoleTestReporterWithHook hook + in cb opts tree + +appendPatternIfTestFailed + :: [TestName] -- ^ list of (pre-intercalated) test names + -> Maybe Expr -- ^ current pattern, if any + -> [TestName] -- ^ name of current test, represented as a list of group names + -> Result -- ^ vanilla result + -> Result +appendPatternIfTestFailed [_] _ _ res = res -- if there is only one test, nothing to refine +appendPatternIfTestFailed _ _ [] res = res -- should be impossible +appendPatternIfTestFailed tests currentPattern (name : names) res = case resultOutcome res of + Success -> res + Failure{} -> res { resultDescription = resultDescription res ++ msg } + where + msg = "\nUse -p '" ++ escapeQuotes (printAwkExpr pattern) ++ "' to rerun this test only." + + escapeQuotes = concatMap $ \c -> if c == '\'' then "'\\''" else [c] + + findPattern [_] pat _ = ERE pat + findPattern _ pat [] = EQ (Field (IntLit 0)) (StringLit pat) + findPattern ts pat (n : ns) = let pat' = n ++ '.' : pat in + findPattern (filter (pat' `isInfixOf`) ts) pat' ns + + individualPattern = findPattern (filter (name `isInfixOf`) tests) name names + + pattern = maybe id And currentPattern individualPattern + +consoleTestReporterOptions :: [OptionDescription] +consoleTestReporterOptions = + [ Option (Proxy :: Proxy Quiet) + , Option (Proxy :: Proxy HideSuccesses) + , Option (Proxy :: Proxy UseColor) + , Option (Proxy :: Proxy AnsiTricks) + ] + +-- | A simple console UI with a hook to postprocess results, +-- depending on their names and external conditions +-- (e. g., its previous outcome, stored in a file). +-- Names are listed in reverse order: +-- from test's own name to a name of the outermost test group. +-- +-- @since 1.4.2 +consoleTestReporterWithHook :: ([TestName] -> Result -> IO Result) -> Ingredient +consoleTestReporterWithHook hook = TestReporter consoleTestReporterOptions $ \opts tree -> Just $ \smap -> do let @@ -413,7 +472,7 @@ ?colors = useColor whenColor isTermColor let - toutput = buildTestOutput opts tree + toutput = applyHook hook $ buildTestOutput opts tree case () of { _ | hideSuccesses && isTerm && ansiTricks -> @@ -474,9 +533,9 @@ -- These tricks sometimes fail, however—in particular, when the test names -- happen to be longer than the width of the terminal window. See -- --- * +-- * -- --- * +-- * -- -- When that happens, this option can be used to disable the tricks. In -- that case, the test name will be printed only once the test fails. diff --git a/Test/Tasty/Parallel.hs b/Test/Tasty/Parallel.hs --- a/Test/Tasty/Parallel.hs +++ b/Test/Tasty/Parallel.hs @@ -47,7 +47,7 @@ -- Don't let the main thread be garbage-collected -- Otherwise we may get a "thread blocked indefinitely in an STM -- transaction" exception when a child thread is blocked and GC'd. - -- (See e.g. https://github.com/feuerbach/tasty/issues/15) + -- (See e.g. https://github.com/UnkindPartition/tasty/issues/15) -- FIXME is this still needed? _ <- newStablePtr callingThread diff --git a/Test/Tasty/Patterns/Eval.hs b/Test/Tasty/Patterns/Eval.hs --- a/Test/Tasty/Patterns/Eval.hs +++ b/Test/Tasty/Patterns/Eval.hs @@ -6,7 +6,7 @@ import Control.Monad.Error.Class (throwError) -- see #201 import qualified Data.Sequence as Seq import Data.Foldable -import Data.List +import Data.List hiding (length) import Data.Maybe import Data.Char import Test.Tasty.Patterns.Types diff --git a/Test/Tasty/Patterns/Parser.hs b/Test/Tasty/Patterns/Parser.hs --- a/Test/Tasty/Patterns/Parser.hs +++ b/Test/Tasty/Patterns/Parser.hs @@ -29,7 +29,7 @@ deriving (Functor, Applicative, Alternative, Monad, MonadPlus) data ParseResult a = Success a | Invalid | Ambiguous [a] - deriving Show + deriving (Eq, Show) token :: Token a -> Parser a token a = Parser (a <* skipSpaces) diff --git a/Test/Tasty/Patterns/Printer.hs b/Test/Tasty/Patterns/Printer.hs new file mode 100644 --- /dev/null +++ b/Test/Tasty/Patterns/Printer.hs @@ -0,0 +1,65 @@ +{-# LANGUAGE LambdaCase #-} + +module Test.Tasty.Patterns.Printer + ( printAwkExpr + ) + where + +import Prelude hiding (LT, GT, EQ) +import Test.Tasty.Patterns.Types + +-- | @since 1.4.2 +printAwkExpr :: Expr -> String +printAwkExpr e = go 0 e "" + +go :: Int -> Expr -> ShowS +go p = \case + NF -> showString "NF" + IntLit n -> showsPrec p n + StringLit xs -> showChar '"' . showString (escapeString xs) . showChar '"' + ERE xs -> showChar '/' . showString (escapeERE xs) . showChar '/' + + Field x -> showParen (p >= 9) $ showChar '$' . go 9 x + + -- Cf. comment for Test.Tasty.Patterns.Parser.expr2 to understand + -- why we put showParens when precedence is 6 not 8. + Neg x -> showParen (p >= 6) $ showChar '-' . go 8 x + Not x -> showParen (p >= 8) $ showChar '!' . go 8 x + + Add x y -> showParen (p >= 7) $ go 7 x . showChar '+' . go 7 y + Sub x y -> showParen (p >= 7) $ go 7 x . showChar '-' . go 7 y + + Concat x y -> showParen (p >= 6) $ go 6 x . showChar ' ' . go 6 y + + LT x y -> showParen (p >= 5) $ go 5 x . showChar '<' . go 5 y + LE x y -> showParen (p >= 5) $ go 5 x . showString "<=" . go 5 y + GT x y -> showParen (p >= 5) $ go 5 x . showChar '>' . go 5 y + GE x y -> showParen (p >= 5) $ go 5 x . showString ">=" . go 5 y + EQ x y -> showParen (p >= 5) $ go 5 x . showString "==" . go 5 y + NE x y -> showParen (p >= 5) $ go 5 x . showString "!=" . go 5 y + + Match x y -> showParen (p >= 4) $ go 4 x . showChar '~' . go 4 (ERE y) + NoMatch x y -> showParen (p >= 4) $ go 4 x . showString "!~" . go 4 (ERE y) + + And x y -> showParen (p >= 2) $ go 2 x . showString "&&" . go 2 y + + Or x y -> showParen (p >= 1) $ go 1 x . showString "||" . go 1 y + + If c t f -> showParen (p >= 0) $ go 0 c . showChar '?' . go 0 t . showChar ':' . go 0 f + + ToUpperFn x -> showString "toupper(" . go 0 x . showChar ')' + ToLowerFn x -> showString "tolower(" . go 0 x . showChar ')' + + LengthFn Nothing -> showString "length()" + LengthFn (Just x) -> showString "length(" . go 0 x . showChar ')' + + SubstrFn x y Nothing -> showString "substr(" . go 0 x . showChar ',' . go 0 y . showChar ')' + SubstrFn x y (Just z) -> showString "substr(" . go 0 x . showChar ',' . go 0 y . showChar ',' . go 0 z . showChar ')' + + MatchFn x y -> showString "match(" . go 0 x . showChar ',' . go 0 (ERE y) . showChar ')' + +escapeString :: String -> String +escapeString = concatMap $ \c -> (if c `elem` "\\\"" then ('\\' :) else id) [c] + +escapeERE :: String -> String +escapeERE = concatMap $ \c -> (if c `elem` "\\/" then ('\\' :) else id) [c] diff --git a/Test/Tasty/Patterns/Types.hs b/Test/Tasty/Patterns/Types.hs --- a/Test/Tasty/Patterns/Types.hs +++ b/Test/Tasty/Patterns/Types.hs @@ -1,5 +1,9 @@ +{-# LANGUAGE DeriveGeneric #-} + module Test.Tasty.Patterns.Types where +import GHC.Generics + data Expr = IntLit !Int | NF -- ^ number of fields @@ -27,4 +31,4 @@ | LengthFn (Maybe Expr) | MatchFn Expr String | SubstrFn Expr Expr (Maybe Expr) - deriving (Show, Eq) + deriving (Show, Eq, Generic) diff --git a/Test/Tasty/Run.hs b/Test/Tasty/Run.hs --- a/Test/Tasty/Run.hs +++ b/Test/Tasty/Run.hs @@ -200,7 +200,7 @@ -- I'm not sure whether we should get rid of this altogether. For most -- providers this is either difficult to implement or doesn't make -- sense at all. - -- See also https://github.com/feuerbach/tasty/issues/33 + -- See also https://github.com/UnkindPartition/tasty/issues/33 yieldProgress _ = return () type InitFinPair = (Seq.Seq Initializer, Seq.Seq Finalizer) diff --git a/Test/Tasty/Runners/Utils.hs b/Test/Tasty/Runners/Utils.hs --- a/Test/Tasty/Runners/Utils.hs +++ b/Test/Tasty/Runners/Utils.hs @@ -38,7 +38,7 @@ -- This function should be used to display messages generated by the test -- suite (such as test result descriptions). -- --- See e.g. +-- See e.g. formatMessage :: String -> IO String formatMessage = go 3 where diff --git a/tasty.cabal b/tasty.cabal --- a/tasty.cabal +++ b/tasty.cabal @@ -2,7 +2,7 @@ -- see http://haskell.org/cabal/users-guide/ name: tasty -version: 1.4.1 +version: 1.4.2 synopsis: Modern and extensible testing framework description: Tasty is a modern testing framework for Haskell. It lets you combine your unit tests, golden @@ -12,8 +12,8 @@ license-file: LICENSE author: Roman Cheplyaka maintainer: Roman Cheplyaka -homepage: https://github.com/feuerbach/tasty -bug-reports: https://github.com/feuerbach/tasty/issues +homepage: https://github.com/UnkindPartition/tasty +bug-reports: https://github.com/UnkindPartition/tasty/issues -- copyright: category: Testing build-type: Simple @@ -22,7 +22,7 @@ Source-repository head type: git - location: git://github.com/feuerbach/tasty.git + location: git://github.com/UnkindPartition/tasty.git subdir: core flag clock @@ -49,6 +49,7 @@ -- for testing only Test.Tasty.Patterns.Types Test.Tasty.Patterns.Parser + Test.Tasty.Patterns.Printer Test.Tasty.Patterns.Eval other-modules: Control.Concurrent.Async