diff --git a/README.textile b/README.textile --- a/README.textile +++ b/README.textile @@ -14,19 +14,12 @@ -If you want to build the example, to check it's all working: - -
-runghc Setup.lhs configure -fexample
-runghc Setup.lhs build
-dist/build/test-framework-example/test-framework-example
-
- +If you want to build the example, in order to check it's all working you will need to "download it from GitHub":http://github.com/batterseapower/test-framework/tree/master/example/Test/Framework/Example.lhs. Once you have it follow the instructions in the file itself as to how to build it. h2. Description -A test framework, with built-in support for "HUnit":http://hackage.haskell.org/cgi-bin/hackage-scripts/package/HUnit and "QuickCheck":http://hackage.haskell.org/cgi-bin/hackage-scripts/package/QuickCheck tests. The main benefit of using this framework is that you get a nice console based test runner with the following features: +An extensible test framework, with support for "HUnit":http://hackage.haskell.org/cgi-bin/hackage-scripts/package/HUnit and "QuickCheck":http://hackage.haskell.org/cgi-bin/hackage-scripts/package/QuickCheck tests through the "test-framework-hunit":http://hackage.haskell.org/cgi-bin/hackage-scripts/package/test-framework-hunit and "test-framework-quickcheck":http://hackage.haskell.org/cgi-bin/hackage-scripts/package/test-framework-quickcheck packages. The main benefit of using this framework is that you get a nice console based test runner with the following features: * Run tests in parallel but report results in a deterministic order (to aid diff-based analysis of test output) * Progress reporting for individual QuickCheck properties and for the whole suite being run @@ -39,7 +32,7 @@ h2. Example -An example testsuite is provided in the package, which you can build by supplying @-fexample@ to the Cabal @configure@ step as described above. You can also view the most recent version online at "GitHub":http://github.com/batterseapower/test-framework/tree/master/Test/Framework/Example.lhs. +You can view the most recent version of the testsuite online at "GitHub":http://github.com/batterseapower/test-framework/tree/master/example/Test/Framework/Example.lhs, as described in the "Installing" section. The example is not currently available via Hackage. There are two essential components to getting running with the test framework: setting up the tests to be run, and making the program run the tests in the provided console test runner. diff --git a/Test/Framework/Example.lhs b/Test/Framework/Example.lhs deleted file mode 100644 --- a/Test/Framework/Example.lhs +++ /dev/null @@ -1,110 +0,0 @@ -== RUNNING == - -ghc -package test-framework -threaded Example.lhs -o Example -./Example --maximum-generated-tests=5000 +RTS -N2 - - -== ATTRIBUTION == - -Tthe example properties come from the parallel QuickCheck driver (pqc), -see http://code.haskell.org/~dons/code/pqc/. The BSD license is repeated -below, per the licensing conditions of pqc. - -== LICENSING == - -Copyright Don Stewart 2006. - -All rights reserved. - -Redistribution and use in source and binary forms, with or without -modification, are permitted provided that the following conditions are -met: - - * Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - - * Redistributions in binary form must reproduce the above - copyright notice, this list of conditions and the following - disclaimer in the documentation and/or other materials provided - with the distribution. - - * Neither the name of Don Stewart nor the names of other - contributors may be used to endorse or promote products derived - from this software without specific prior written permission. - -THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS -"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT -LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR -A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT -OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, -SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT -LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, -DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY -THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE -OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - -\begin{code} - -import Test.Framework -import Test.Framework.Providers.HUnit -import Test.Framework.Providers.QuickCheck - -import Test.QuickCheck -import Test.HUnit - -import Data.List - - -main = defaultMain tests - -tests = [ - testGroup "Sorting Group 1" [ - testProperty "sort1" prop_sort1, - testProperty "sort2" prop_sort2, - testProperty "sort3" prop_sort3 - ], - testGroup "Sorting Group 2" [ - testProperty "sort4" prop_sort4, - testProperty "sort5" prop_sort5, - testProperty "sort6" prop_sort6, - testCase "sort7" test_sort7, - testCase "sort8" test_sort8 - ] - ] - - -prop_sort1 xs = sort xs == sortBy compare xs - where types = (xs :: [Int]) - -prop_sort2 xs = - (not (null xs)) ==> - (head (sort xs) == minimum xs) - where types = (xs :: [Int]) - -prop_sort3 xs = (not (null xs)) ==> - last (sort xs) == maximum xs - where types = (xs :: [Int]) - -prop_sort4 xs ys = - (not (null xs)) ==> - (not (null ys)) ==> - (head (sort (xs ++ ys)) == min (minimum xs) (minimum ys)) - where types = (xs :: [Int], ys :: [Int]) - -prop_sort5 xs ys = - (not (null xs)) ==> - (not (null ys)) ==> - (head (sort (xs ++ ys)) == max (maximum xs) (maximum ys)) - where types = (xs :: [Int], ys :: [Int]) - -prop_sort6 xs ys = - (not (null xs)) ==> - (not (null ys)) ==> - (last (sort (xs ++ ys)) == max (maximum xs) (maximum ys)) - where types = (xs :: [Int], ys :: [Int]) - -test_sort7 = sort [8, 7, 2, 5, 4, 9, 6, 1, 0, 3] @?= [0..9] - -test_sort8 = error "This test deliberately contains a user error" -\end{code} diff --git a/Test/Framework/Providers/HUnit.hs b/Test/Framework/Providers/HUnit.hs deleted file mode 100644 --- a/Test/Framework/Providers/HUnit.hs +++ /dev/null @@ -1,56 +0,0 @@ -module Test.Framework.Providers.HUnit ( - testCase - ) where - -import Test.Framework.Providers.API - -import Test.HUnit.Lang - - --- | Create a 'Test' for a HUnit 'Assertion' -testCase :: TestName -> Assertion -> Test -testCase name = Test name . TestCase - - -instance TestResultlike TestCaseRunning TestCaseResult where - testSucceeded = testCaseSucceeded - -data TestCaseRunning = TestCaseRunning - -instance Show TestCaseRunning where - show TestCaseRunning = "Running" - -data TestCaseResult = TestCasePassed - | TestCaseFailed String - | TestCaseError String - -instance Show TestCaseResult where - show result = case result of - TestCasePassed -> "OK" - TestCaseFailed message -> "Failed: " ++ message - TestCaseError message -> "ERROR: " ++ message - -testCaseSucceeded :: TestCaseResult -> Bool -testCaseSucceeded TestCasePassed = True -testCaseSucceeded _ = False - - -newtype TestCase = TestCase Assertion - -instance Testlike TestCaseRunning TestCaseResult TestCase where - runTest topts (TestCase assertion) = runTestCase topts assertion - testTypeName _ = "Test Cases" - -runTestCase :: CompleteTestOptions -> Assertion -> IO (TestCaseRunning :~> TestCaseResult, IO ()) -runTestCase topts assertion = runImprovingIO $ do - yieldImprovement TestCaseRunning - mb_result <- maybeTimeoutImprovingIO (unK $ topt_timeout topts) $ liftIO (myPerformTestCase assertion) - return (mb_result `orElse` TestCaseError "Timed out") - -myPerformTestCase :: Assertion -> IO TestCaseResult -myPerformTestCase assertion = do - result <- performTestCase assertion - return $ case result of - Nothing -> TestCasePassed - Just (True, message) -> TestCaseFailed message - Just (False, message) -> TestCaseError message diff --git a/Test/Framework/Providers/QuickCheck.hs b/Test/Framework/Providers/QuickCheck.hs deleted file mode 100644 --- a/Test/Framework/Providers/QuickCheck.hs +++ /dev/null @@ -1,96 +0,0 @@ -module Test.Framework.Providers.QuickCheck ( - testProperty - ) where - -import Test.Framework.Providers.API - -import Test.QuickCheck hiding (Property) - -import Data.List - -import System.Random - - --- | Create a 'Test' for a QuickCheck 'Testable' property -testProperty :: Testable a => TestName -> a -> Test -testProperty name = Test name . Property - - -instance TestResultlike PropertyTestCount PropertyResult where - testSucceeded = propertySucceeded - --- | Used to document numbers which we expect to be intermediate test counts from running properties -type PropertyTestCount = Int - --- | The failure information from the run of a property -data PropertyResult = PropertyResult { - pr_status :: PropertyStatus, - pr_used_seed :: Int, - pr_tests_run :: Maybe PropertyTestCount -- Due to technical limitations, it's currently not possible to find out the number of - -- tests previously run if the test times out, hence we need a Maybe here for that case. - } - -data PropertyStatus = PropertyOK -- ^ The property is true as far as we could check it - | PropertyArgumentsExhausted -- ^ The property may be true, but we ran out of arguments to try it out on - | PropertyFalsifiable [String] -- ^ The property was not true. The list of strings are the arguments inducing failure. - | PropertyTimedOut -- ^ The property timed out during execution - -instance Show PropertyResult where - show (PropertyResult { pr_status = status, pr_used_seed = used_seed, pr_tests_run = mb_tests_run }) - = case status of - PropertyOK -> "OK, passed " ++ tests_run_str ++ " tests" - PropertyArgumentsExhausted -> "Arguments exhausted after " ++ tests_run_str ++ " tests" - PropertyFalsifiable test_args -> "Falsifiable with seed " ++ show used_seed ++ ", after " ++ tests_run_str ++ " tests:\n" ++ unlinesConcise test_args - PropertyTimedOut -> "Timed out after " ++ tests_run_str ++ " tests" - where - tests_run_str = fmap show mb_tests_run `orElse` "an unknown number of" - -propertySucceeded :: PropertyResult -> Bool -propertySucceeded result = propertyStatusIsSuccess (pr_status result) - -propertyStatusIsSuccess :: PropertyStatus -> Bool -propertyStatusIsSuccess PropertyOK = True -propertyStatusIsSuccess PropertyArgumentsExhausted = True -propertyStatusIsSuccess _ = False - - -data Property = forall a. Testable a => Property a - -instance Testlike PropertyTestCount PropertyResult Property where - runTest topts (Property testable) = runProperty topts testable - testTypeName _ = "Properties" - -runProperty :: Testable a => CompleteTestOptions -> a -> IO (PropertyTestCount :~> PropertyResult, IO ()) -runProperty topts testable = do - (gen, seed) <- newSeededStdGen (unK $ topt_seed topts) - runImprovingIO $ do - mb_result <- maybeTimeoutImprovingIO (unK (topt_timeout topts)) $ myCheck topts gen testable - return $ toPropertyResult seed $ case mb_result of - Nothing -> (PropertyTimedOut, Nothing) - Just (status, tests_run) -> (status, Just tests_run) - where - toPropertyResult seed (status, mb_tests_run) = PropertyResult { - pr_status = status, - pr_used_seed = seed, - pr_tests_run = mb_tests_run - } - -myCheck :: (Testable a) => CompleteTestOptions -> StdGen -> a -> ImprovingIO PropertyTestCount f (PropertyStatus, PropertyTestCount) -myCheck topts rnd a = myTests topts (evaluate a) rnd 0 0 [] - -myTests :: CompleteTestOptions -> Gen Result -> StdGen -> PropertyTestCount -> PropertyTestCount -> [[String]] -> ImprovingIO PropertyTestCount f (PropertyStatus, PropertyTestCount) -myTests topts gen rnd0 ntest nfail stamps - | ntest == unK (topt_maximum_generated_tests topts) = do return (PropertyOK, ntest) - | nfail == unK (topt_maximum_unsuitable_generated_tests topts) = do return (PropertyArgumentsExhausted, ntest) - | otherwise = do - yieldImprovement ntest - case ok result of - Nothing -> - myTests topts gen rnd1 ntest (nfail + 1) stamps - Just True -> - myTests topts gen rnd1 (ntest + 1) nfail (stamp result:stamps) - Just False -> - return (PropertyFalsifiable (arguments result), ntest) - where - result = generate (configSize defaultConfig ntest) rnd2 gen - (rnd1, rnd2) = split rnd0 diff --git a/test-framework.cabal b/test-framework.cabal --- a/test-framework.cabal +++ b/test-framework.cabal @@ -1,11 +1,11 @@ Name: test-framework -Version: 0.1.2 +Version: 0.2.0 Cabal-Version: >= 1.2 Category: Testing Synopsis: Framework for running and organising tests, with HUnit and QuickCheck support -Description: Allows QuickCheck properties and HUnit test cases to be assembled into test groups, run in parallel (but reported - in deterministic order, to aid diff interpretation) and filtered and controlled by command line options. - All of this comes with colored test output, progress reporting and test statistics output. +Description: Allows tests such as QuickCheck properties and HUnit test cases to be assembled into test groups, run in + parallel (but reported in deterministic order, to aid diff interpretation) and filtered and controlled by + command line options. All of this comes with colored test output, progress reporting and test statistics output. License: BSD3 License-File: LICENSE Extra-Source-Files: README.textile @@ -22,19 +22,7 @@ Description: Build the tests Default: False -Flag Example - Description: Build the example testsuite - Default: False -Flag HUnit - Description: Enable HUnit integration: disabling this is not recommended, as it changes the package API - Default: True - -Flag QuickCheck - Description: Enable QuickCheck integration: disabling this is not recommended, as it changes the package API - Default: True - - Library Exposed-Modules: Test.Framework Test.Framework.Options @@ -57,14 +45,6 @@ Test.Framework.Runners.TimedConsumption Test.Framework.Utilities - if flag(quickcheck) - Build-Depends: QuickCheck >= 1.1 - Exposed-Modules: Test.Framework.Providers.QuickCheck - - if flag(hunit) - Build-Depends: HUnit >= 1.2 - Exposed-Modules: Test.Framework.Providers.HUnit - Build-Depends: ansi-terminal >= 0.4.0, ansi-wl-pprint >= 0.4.0, regex-posix >= 0.72 if flag(splitBase) Build-Depends: base >= 3, random >= 1.0, containers >= 0.1 @@ -89,7 +69,7 @@ Executable test-framework-tests Main-Is: Test/Framework/Tests.hs - Build-Depends: QuickCheck >= 1.1, HUnit >= 1.2, ansi-terminal >= 0.4.0, ansi-wl-pprint >= 0.4.0, regex-posix >= 0.72 + Build-Depends: HUnit >= 1.2, ansi-terminal >= 0.4.0, ansi-wl-pprint >= 0.4.0, regex-posix >= 0.72 if flag(splitBase) Build-Depends: base >= 3, random >= 1.0, containers >= 0.1 else @@ -111,31 +91,4 @@ Cpp-Options: -DCOMPILER_GHC if !flag(tests) - Buildable: False - -Executable test-framework-example - Main-Is: Test/Framework/Example.lhs - - Build-Depends: QuickCheck >= 1.1, HUnit >= 1.2, ansi-terminal >= 0.4.0, ansi-wl-pprint >= 0.4.0, regex-posix >= 0.72 - if flag(splitBase) - Build-Depends: base >= 3, random >= 1.0, containers >= 0.1 - else - Build-Depends: base < 3 - - Extensions: CPP - PatternGuards - ExistentialQuantification - RecursiveDo - FlexibleInstances - TypeSynonymInstances - TypeOperators - FunctionalDependencies - MultiParamTypeClasses - - Ghc-Options: -threaded - - if impl(ghc) - Cpp-Options: -DCOMPILER_GHC - - if !flag(example) Buildable: False