diff --git a/LICENSE b/LICENSE new file mode 100644 --- /dev/null +++ b/LICENSE @@ -0,0 +1,20 @@ +Copyright (c) 2015 Joachim Breitner + +Permission is hereby granted, free of charge, to any person obtaining +a copy of this software and associated documentation files (the +"Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be included +in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, +TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE +SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. diff --git a/README.md b/README.md new file mode 100644 --- /dev/null +++ b/README.md @@ -0,0 +1,47 @@ +tasty-expected-failure +====================== + +What is this? +------------- + +With the function `expectFail` in the provided module +`ExpectedFailure`, you can mark that you expect test cases to fail, +and not to pass. + +This can for example be used for test-driven development: Create the tests, +mark them with `expectFail`, and you can still push +to the main branch, without your continuous integration branch failing. + +Once someone implements the feature or fixes the bug (maybe unknowingly), the +test suite will tell him so, due to the now unexpectedly passing test, and he +can remove the `expectFail` marker. + +The module also provides `ignoreTest` to avoid +running a test. Both funtions are implemented via the more general +`warpTest`, which is also provided. + +Why is this not provided by tasty? +---------------------------------- + +`` + +The author of the tasty library prefers to provide a minimal experience in the +tasty library, instead of a batteries-included approach, and chose not to +include these 39 lines of code in tasty. See the +[issue](https://github.com/feuerbach/tasty/issues/114) for the discussion. + +Instead I wrote 37 lines of cabal file, a similar number of lines of README, +created a git repository, created a travis file, run travis to figure out on +what versions it builds (something that would have happened automatically with +a pull request for tasty), upload to hackge, add to stackage. + +Furthermore, there is little discoverability: If it were part of the tasty API, +users would stumble over it. Now they likely won’t. And if they do, they have +to worry about whether it is still in sync with `tasty`, they have to add it to +their build-depends, they have to import yet another module. Distribution +packagers will have yet another package where they have to create the +packaging, check the copyright, and run autobuilders for. + +Sigh. + +`` diff --git a/Setup.hs b/Setup.hs new file mode 100644 --- /dev/null +++ b/Setup.hs @@ -0,0 +1,2 @@ +import Distribution.Simple +main = defaultMain diff --git a/Test/Tasty/ExpectedFailure.hs b/Test/Tasty/ExpectedFailure.hs new file mode 100644 --- /dev/null +++ b/Test/Tasty/ExpectedFailure.hs @@ -0,0 +1,69 @@ +{-# LANGUAGE DeriveDataTypeable, ScopedTypeVariables #-} +module Test.Tasty.ExpectedFailure (expectFail, ignoreTest, wrapTest) where + +import Test.Tasty.Options +import Test.Tasty.Runners +import Test.Tasty.Providers +import Data.Typeable +import Data.Tagged + +data WrappedTest t = WrappedTest (IO Result -> IO Result) t + deriving Typeable + +instance forall t. IsTest t => IsTest (WrappedTest t) where + run opts (WrappedTest wrap t) prog = wrap (run opts t prog) + testOptions = retag (testOptions :: Tagged t [OptionDescription]) + +-- | 'wrapTest' allows you to modify the behavoiur of the tests, e.g. by +-- modifying the result or not running the test at all. It is used to implement +-- 'expectFail' and 'ignoreTest'. +wrapTest :: (IO Result -> IO Result) -> TestTree -> TestTree +wrapTest wrap = go + where + go (SingleTest n t) = SingleTest n (WrappedTest wrap t) + go (TestGroup name tests) = TestGroup name (map go tests) + go (PlusTestOptions plus tree) = PlusTestOptions plus (go tree) + go (WithResource spec gentree) = WithResource spec (go . gentree) + go (AskOptions f) = AskOptions (go . f) + + +-- | Marks all tests in the give test as expected failures: The tests will +-- still be run, but if they succeed, it is reported as a test suite failure, +-- and conversely a the failure of the test is ignored. +-- +-- Any output of a failing test is still printed. +-- +-- This is useful if, in a test driven development, tests are written and +-- commited to the master branch before their implementation: It allows the +-- tests to fail (as expected) without making the whole test suite fail. +-- +-- Similarly, regressions and bugs can be documented in the test suite this +-- way, until a fix is commited, and if a fix is applied (intentionally or +-- accidentially), the test suite will remind you to remove the 'expectFail' +-- marker. +expectFail :: TestTree -> TestTree +expectFail = wrapTest (fmap change) + where + change r + | resultSuccessful r + = r { resultOutcome = Failure TestFailed + , resultDescription = resultDescription r `append` "(unexpected success)" + , resultShortDescription = "PASS (unexpected)" + } + | otherwise + = r { resultOutcome = Success + , resultDescription = resultDescription r `append` "(expected failure)" + , resultShortDescription = "FAIL (expected)" + } + "" `append` s = s + t `append` s | last t == '\n' = t ++ s ++ "\n" + | otherwise = t ++ "\n" ++ s + + +-- | Prevents the tests from running and reports them as succeeding. This maybe +-- be desireable as an alternative comment the tests out, as they are still +-- typechecked, and the test report lists them, as a reminder that there are +-- ignored test. +ignoreTest :: TestTree -> TestTree +ignoreTest = wrapTest $ const $ return $ + (testPassed "") { resultShortDescription = "IGNORED" } diff --git a/tasty-expected-failure.cabal b/tasty-expected-failure.cabal new file mode 100644 --- /dev/null +++ b/tasty-expected-failure.cabal @@ -0,0 +1,43 @@ +name: tasty-expected-failure +version: 0.11 +synopsis: Mark tasty tests as failure expected +description: + With the function 'Test.Tasty.ExpectedFailure.expectFail' in the provided module + "Test.Tasty.ExpectedFailure", you can mark that you expect test cases to fail, + and not to pass. + . + This can for example be used for test-driven development: Create the tests, + mark them with 'Test.Tasty.ExpectedFailure.expectFail', and you can still push + to the main branch, without your continuous integration branch failing. + . + Once someone implements the feature or fixes the bug (maybe unknowingly), the + test suite will tell him so, due to the now unexpectedly passing test, and he + can remove the 'Test.Tasty.ExpectedFailure.expectFail' marker. + . + The module also provides 'Test.Tasty.ExpectedFailure.ignoreTest' to avoid + running a test. Both funtions are implemented via the more general + 'Test.Tasty.ExpectedFailure.warpTest', which is also provided. +homepage: http://github.com/nomeata/tasty-expected-failure +license: MIT +license-file: LICENSE +author: Joachim Breitner +maintainer: mail@joachim-breitner.de +copyright: 2015 Joachim Breitner +category: Testing +build-type: Simple +extra-source-files: README.md +cabal-version: >=1.10 +tested-with: GHC == 7.4.2, GHC == 7.6.3, GHC == 7.8.4, GHC == 7.11.* + +library + exposed-modules: + Test.Tasty.ExpectedFailure + build-depends: + base >= 4.5 && <4.9, + tagged >= 0.7 && < 0.8, + tasty >= 0.11 + default-language: Haskell2010 + +source-repository head + type: git + location: git://github.com/nomeata/tasty-expected-failure