testrunner (empty) → 0.9
raw patch · 11 files changed
+1086/−0 lines, 11 filesdep +HUnitdep +QuickCheckdep +basesetup-changed
Dependencies added: HUnit, QuickCheck, base, random, regex-compat, stm
Files
- LICENSE +339/−0
- README +28/−0
- Setup.hs +3/−0
- Test/Runner.hs +26/−0
- Test/Runner/Backends.hs +90/−0
- Test/Runner/Driver.hs +132/−0
- Test/Runner/Frontend.hs +136/−0
- Test/Runner/Test.hs +83/−0
- homepage.html +52/−0
- testrunner.cabal +37/−0
- using-testrunner.html +160/−0
+ LICENSE view
@@ -0,0 +1,339 @@+ GNU GENERAL PUBLIC LICENSE+ Version 2, June 1991++ Copyright (C) 1989, 1991 Free Software Foundation, Inc.,+ 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA+ Everyone is permitted to copy and distribute verbatim copies+ of this license document, but changing it is not allowed.++ Preamble++ The licenses for most software are designed to take away your+freedom to share and change it. By contrast, the GNU General Public+License is intended to guarantee your freedom to share and change free+software--to make sure the software is free for all its users. This+General Public License applies to most of the Free Software+Foundation's software and to any other program whose authors commit to+using it. (Some other Free Software Foundation software is covered by+the GNU Lesser General Public License instead.) You can apply it to+your programs, too.++ When we speak of free software, we are referring to freedom, not+price. Our General Public Licenses are designed to make sure that you+have the freedom to distribute copies of free software (and charge for+this service if you wish), that you receive source code or can get it+if you want it, that you can change the software or use pieces of it+in new free programs; and that you know you can do these things.++ To protect your rights, we need to make restrictions that forbid+anyone to deny you these rights or to ask you to surrender the rights.+These restrictions translate to certain responsibilities for you if you+distribute copies of the software, or if you modify it.++ For example, if you distribute copies of such a program, whether+gratis or for a fee, you must give the recipients all the rights that+you have. You must make sure that they, too, receive or can get the+source code. And you must show them these terms so they know their+rights.++ We protect your rights with two steps: (1) copyright the software, and+(2) offer you this license which gives you legal permission to copy,+distribute and/or modify the software.++ Also, for each author's protection and ours, we want to make certain+that everyone understands that there is no warranty for this free+software. If the software is modified by someone else and passed on, we+want its recipients to know that what they have is not the original, so+that any problems introduced by others will not reflect on the original+authors' reputations.++ Finally, any free program is threatened constantly by software+patents. We wish to avoid the danger that redistributors of a free+program will individually obtain patent licenses, in effect making the+program proprietary. To prevent this, we have made it clear that any+patent must be licensed for everyone's free use or not licensed at all.++ The precise terms and conditions for copying, distribution and+modification follow.++ GNU GENERAL PUBLIC LICENSE+ TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION++ 0. This License applies to any program or other work which contains+a notice placed by the copyright holder saying it may be distributed+under the terms of this General Public License. The "Program", below,+refers to any such program or work, and a "work based on the Program"+means either the Program or any derivative work under copyright law:+that is to say, a work containing the Program or a portion of it,+either verbatim or with modifications and/or translated into another+language. (Hereinafter, translation is included without limitation in+the term "modification".) Each licensee is addressed as "you".++Activities other than copying, distribution and modification are not+covered by this License; they are outside its scope. The act of+running the Program is not restricted, and the output from the Program+is covered only if its contents constitute a work based on the+Program (independent of having been made by running the Program).+Whether that is true depends on what the Program does.++ 1. You may copy and distribute verbatim copies of the Program's+source code as you receive it, in any medium, provided that you+conspicuously and appropriately publish on each copy an appropriate+copyright notice and disclaimer of warranty; keep intact all the+notices that refer to this License and to the absence of any warranty;+and give any other recipients of the Program a copy of this License+along with the Program.++You may charge a fee for the physical act of transferring a copy, and+you may at your option offer warranty protection in exchange for a fee.++ 2. You may modify your copy or copies of the Program or any portion+of it, thus forming a work based on the Program, and copy and+distribute such modifications or work under the terms of Section 1+above, provided that you also meet all of these conditions:++ a) You must cause the modified files to carry prominent notices+ stating that you changed the files and the date of any change.++ b) You must cause any work that you distribute or publish, that in+ whole or in part contains or is derived from the Program or any+ part thereof, to be licensed as a whole at no charge to all third+ parties under the terms of this License.++ c) If the modified program normally reads commands interactively+ when run, you must cause it, when started running for such+ interactive use in the most ordinary way, to print or display an+ announcement including an appropriate copyright notice and a+ notice that there is no warranty (or else, saying that you provide+ a warranty) and that users may redistribute the program under+ these conditions, and telling the user how to view a copy of this+ License. (Exception: if the Program itself is interactive but+ does not normally print such an announcement, your work based on+ the Program is not required to print an announcement.)++These requirements apply to the modified work as a whole. If+identifiable sections of that work are not derived from the Program,+and can be reasonably considered independent and separate works in+themselves, then this License, and its terms, do not apply to those+sections when you distribute them as separate works. But when you+distribute the same sections as part of a whole which is a work based+on the Program, the distribution of the whole must be on the terms of+this License, whose permissions for other licensees extend to the+entire whole, and thus to each and every part regardless of who wrote it.++Thus, it is not the intent of this section to claim rights or contest+your rights to work written entirely by you; rather, the intent is to+exercise the right to control the distribution of derivative or+collective works based on the Program.++In addition, mere aggregation of another work not based on the Program+with the Program (or with a work based on the Program) on a volume of+a storage or distribution medium does not bring the other work under+the scope of this License.++ 3. You may copy and distribute the Program (or a work based on it,+under Section 2) in object code or executable form under the terms of+Sections 1 and 2 above provided that you also do one of the following:++ a) Accompany it with the complete corresponding machine-readable+ source code, which must be distributed under the terms of Sections+ 1 and 2 above on a medium customarily used for software interchange; or,++ b) Accompany it with a written offer, valid for at least three+ years, to give any third party, for a charge no more than your+ cost of physically performing source distribution, a complete+ machine-readable copy of the corresponding source code, to be+ distributed under the terms of Sections 1 and 2 above on a medium+ customarily used for software interchange; or,++ c) Accompany it with the information you received as to the offer+ to distribute corresponding source code. (This alternative is+ allowed only for noncommercial distribution and only if you+ received the program in object code or executable form with such+ an offer, in accord with Subsection b above.)++The source code for a work means the preferred form of the work for+making modifications to it. For an executable work, complete source+code means all the source code for all modules it contains, plus any+associated interface definition files, plus the scripts used to+control compilation and installation of the executable. However, as a+special exception, the source code distributed need not include+anything that is normally distributed (in either source or binary+form) with the major components (compiler, kernel, and so on) of the+operating system on which the executable runs, unless that component+itself accompanies the executable.++If distribution of executable or object code is made by offering+access to copy from a designated place, then offering equivalent+access to copy the source code from the same place counts as+distribution of the source code, even though third parties are not+compelled to copy the source along with the object code.++ 4. You may not copy, modify, sublicense, or distribute the Program+except as expressly provided under this License. Any attempt+otherwise to copy, modify, sublicense or distribute the Program is+void, and will automatically terminate your rights under this License.+However, parties who have received copies, or rights, from you under+this License will not have their licenses terminated so long as such+parties remain in full compliance.++ 5. You are not required to accept this License, since you have not+signed it. However, nothing else grants you permission to modify or+distribute the Program or its derivative works. These actions are+prohibited by law if you do not accept this License. Therefore, by+modifying or distributing the Program (or any work based on the+Program), you indicate your acceptance of this License to do so, and+all its terms and conditions for copying, distributing or modifying+the Program or works based on it.++ 6. Each time you redistribute the Program (or any work based on the+Program), the recipient automatically receives a license from the+original licensor to copy, distribute or modify the Program subject to+these terms and conditions. You may not impose any further+restrictions on the recipients' exercise of the rights granted herein.+You are not responsible for enforcing compliance by third parties to+this License.++ 7. If, as a consequence of a court judgment or allegation of patent+infringement or for any other reason (not limited to patent issues),+conditions are imposed on you (whether by court order, agreement or+otherwise) that contradict the conditions of this License, they do not+excuse you from the conditions of this License. If you cannot+distribute so as to satisfy simultaneously your obligations under this+License and any other pertinent obligations, then as a consequence you+may not distribute the Program at all. For example, if a patent+license would not permit royalty-free redistribution of the Program by+all those who receive copies directly or indirectly through you, then+the only way you could satisfy both it and this License would be to+refrain entirely from distribution of the Program.++If any portion of this section is held invalid or unenforceable under+any particular circumstance, the balance of the section is intended to+apply and the section as a whole is intended to apply in other+circumstances.++It is not the purpose of this section to induce you to infringe any+patents or other property right claims or to contest validity of any+such claims; this section has the sole purpose of protecting the+integrity of the free software distribution system, which is+implemented by public license practices. Many people have made+generous contributions to the wide range of software distributed+through that system in reliance on consistent application of that+system; it is up to the author/donor to decide if he or she is willing+to distribute software through any other system and a licensee cannot+impose that choice.++This section is intended to make thoroughly clear what is believed to+be a consequence of the rest of this License.++ 8. If the distribution and/or use of the Program is restricted in+certain countries either by patents or by copyrighted interfaces, the+original copyright holder who places the Program under this License+may add an explicit geographical distribution limitation excluding+those countries, so that distribution is permitted only in or among+countries not thus excluded. In such case, this License incorporates+the limitation as if written in the body of this License.++ 9. The Free Software Foundation may publish revised and/or new versions+of the General Public License from time to time. Such new versions will+be similar in spirit to the present version, but may differ in detail to+address new problems or concerns.++Each version is given a distinguishing version number. If the Program+specifies a version number of this License which applies to it and "any+later version", you have the option of following the terms and conditions+either of that version or of any later version published by the Free+Software Foundation. If the Program does not specify a version number of+this License, you may choose any version ever published by the Free Software+Foundation.++ 10. If you wish to incorporate parts of the Program into other free+programs whose distribution conditions are different, write to the author+to ask for permission. For software which is copyrighted by the Free+Software Foundation, write to the Free Software Foundation; we sometimes+make exceptions for this. Our decision will be guided by the two goals+of preserving the free status of all derivatives of our free software and+of promoting the sharing and reuse of software generally.++ NO WARRANTY++ 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY+FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN+OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES+PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED+OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF+MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS+TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE+PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,+REPAIR OR CORRECTION.++ 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR+REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,+INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING+OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED+TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY+YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER+PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE+POSSIBILITY OF SUCH DAMAGES.++ END OF TERMS AND CONDITIONS++ How to Apply These Terms to Your New Programs++ If you develop a new program, and you want it to be of the greatest+possible use to the public, the best way to achieve this is to make it+free software which everyone can redistribute and change under these terms.++ To do so, attach the following notices to the program. It is safest+to attach them to the start of each source file to most effectively+convey the exclusion of warranty; and each file should have at least+the "copyright" line and a pointer to where the full notice is found.++ <one line to give the program's name and a brief idea of what it does.>+ Copyright (C) <year> <name of author>++ This program is free software; you can redistribute it and/or modify+ it under the terms of the GNU General Public License as published by+ the Free Software Foundation; either version 2 of the License, or+ (at your option) any later version.++ This program is distributed in the hope that it will be useful,+ but WITHOUT ANY WARRANTY; without even the implied warranty of+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the+ GNU General Public License for more details.++ You should have received a copy of the GNU General Public License along+ with this program; if not, write to the Free Software Foundation, Inc.,+ 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.++Also add information on how to contact you by electronic and paper mail.++If the program is interactive, make it output a short notice like this+when it starts in an interactive mode:++ Gnomovision version 69, Copyright (C) year name of author+ Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.+ This is free software, and you are welcome to redistribute it+ under certain conditions; type `show c' for details.++The hypothetical commands `show w' and `show c' should show the appropriate+parts of the General Public License. Of course, the commands you use may+be called something other than `show w' and `show c'; they could even be+mouse-clicks or menu items--whatever suits your program.++You should also get your employer (if you work as a programmer) or your+school, if any, to sign a "copyright disclaimer" for the program, if+necessary. Here is a sample; alter the names:++ Yoyodyne, Inc., hereby disclaims all copyright interest in the program+ `Gnomovision' (which makes passes at compilers) written by James Hacker.++ <signature of Ty Coon>, 1 April 1989+ Ty Coon, President of Vice++This General Public License does not permit incorporating your program into+proprietary programs. If your program is a subroutine library, you may+consider it more useful to permit linking proprietary applications with the+library. If this is what you want to do, use the GNU Lesser General+Public License instead of this License.
+ README view
@@ -0,0 +1,28 @@+= testrunner =++== Description ==++testrunner is a framework to run unit tests. It has the+following distinguishing features:++* It can run unit tests in parallel.++* It can run QuickCheck and HUnit tests as well as simple+ boolean expressions.++* It comes with a ready-made main function for your unit test+ executable.++* This main function recognizes command-line arguments to select+ tests by name and replay QuickCheck tests.++== How to install ==++Configure the package with 'runghc Setup.hs configure --user'+Build it with 'runghc Setup.hs build'+Then copy the built library with 'runghc Setup.hs install'++== How to use ==++See the file using-testrunner.html+
+ Setup.hs view
@@ -0,0 +1,3 @@+import Distribution.Simple+main = defaultMain+
+ Test/Runner.hs view
@@ -0,0 +1,26 @@+-- Copyright (C) 2009 Reinier Lamers+--+-- This program is free software; you can redistribute it and/or modify+-- it under the terms of the GNU General Public License as published by+-- the Free Software Foundation; either version 2, or (at your option)+-- any later version.+--+-- This program is distributed in the hope that it will be useful,+-- but WITHOUT ANY WARRANTY; without even the implied warranty of+-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the+-- GNU General Public License for more details.+--+-- You should have received a copy of the GNU General Public License+-- along with this program; see the file COPYING. If not, write to+-- the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor,+-- Boston, MA 02110-1301, USA.++module Test.Runner ( module Test.Runner.Frontend,+ module Test.Runner.Driver,+ module Test.Runner.Backends+ ) where++import Test.Runner.Frontend+import Test.Runner.Driver+import Test.Runner.Backends+
+ Test/Runner/Backends.hs view
@@ -0,0 +1,90 @@+-- Copyright (C) 2009 Reinier Lamers+--+-- This program is free software; you can redistribute it and/or modify+-- it under the terms of the GNU General Public License as published by+-- the Free Software Foundation; either version 2, or (at your option)+-- any later version.+--+-- This program is distributed in the hope that it will be useful,+-- but WITHOUT ANY WARRANTY; without even the implied warranty of+-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the+-- GNU General Public License for more details.+--+-- You should have received a copy of the GNU General Public License+-- along with this program; see the file COPYING. If not, write to+-- the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor,+-- Boston, MA 02110-1301, USA.+{-# LANGUAGE GADTs, FlexibleInstances #-}++-- | Test.Runner.Backends contains the types and functions that make it possible+-- to run tests constructed with different test packages with the same driver+-- framework from Test.Runner.Driver.+module Test.Runner.Backends ( TestRunnerTest(..), RunnableTest(..),+ RunWithQuickCheck(..), runWithQuickCheck+ ) where++import Data.List ( intersperse )+import qualified Test.QuickCheck as QC ( Testable(..), quickCheckWithResult,+ Result(..),+ Args, stdArgs )+import Test.HUnit ( Test, PutText(..), runTestText, errors, failures )++-- | The class of all types that testrunner can treat as a test. The method+-- 'run' should return @Nothing@ if the test succeeds, or @Just s@ if the test+-- fails, where @s@ is a human-readable description of the failure.+class RunnableTest a where+ run :: a -> IO (Maybe String)+ -- | 'runWithArgs' runs the test with specified QuickCheck arguments. For+ -- all non-QuickCheck tests, this defaults to just @run@.+ runWithArgs :: QC.Args -> a -> IO (Maybe String)+ runWithArgs = const run++-- | Any expression that returns @True@ upon success and @False@ upon failure+-- can be treated as a test by testrunner.+instance RunnableTest Bool where+ run e = return $ if e then Nothing else Just "Boolean test failed"++-- | Any @IO@ action that returns @True@ upon success and @False@ upon failure+-- can be treated as a test by testrunner.+instance RunnableTest (IO Bool) where+ run a = a >>= run++-- | 'RunWithQuickCheck' turns a QuickCheck test into something that can be run+-- with testrunner. This type-level indirection is necessary to please the+-- type checker.+data RunWithQuickCheck where+ RunWithQuickCheck :: QC.Testable a => a -> RunWithQuickCheck++-- | QuickCheck properties can be run by testrunner.+-- You do lose a lot of information on the result though; only whether the+-- test succeeded or not is returned.+instance RunnableTest RunWithQuickCheck where+ run t = runWithArgs QC.stdArgs t+ runWithArgs args (RunWithQuickCheck t) = do+ r <- QC.quickCheckWithResult args t+ return $ case r of+ QC.Failure seed size reason _ -> Just (reason ++ " (seed: " +++ show seed ++ ", size: "+ ++ show size ++ ")")+ _ -> Nothing++-- | HUnit @Test@s can be run by testrunner.+instance RunnableTest Test where+ run t = do+ (counts, messages) <- runTestText recordMessage t+ return $ if errors counts == 0 && failures counts == 0+ then Nothing+ else Just (concat $ reverse $ intersperse "\n" messages)+ where recordMessage = PutText (\msg _ msgs -> return (msg : msgs)) []++-- | A TestRunnerTest is a data type that hides the actual type of the test -+-- QuickCheck, plain IO action, or any other RunnableTest. This is required to+-- be able to put tests of different types in a single list.+data TestRunnerTest where+ TestRunnerTest :: (RunnableTest a) => a -> TestRunnerTest++-- | Convenience function to go from something testable by QuickCheck to a+-- @TestRunnerTest@ in one step.+runWithQuickCheck :: (QC.Testable a) => a -> TestRunnerTest+runWithQuickCheck = TestRunnerTest . RunWithQuickCheck+
+ Test/Runner/Driver.hs view
@@ -0,0 +1,132 @@+-- Copyright (C) 2009 Reinier Lamers+--+-- This program is free software; you can redistribute it and/or modify+-- it under the terms of the GNU General Public License as published by+-- the Free Software Foundation; either version 2, or (at your option)+-- any later version.+--+-- This program is distributed in the hope that it will be useful,+-- but WITHOUT ANY WARRANTY; without even the implied warranty of+-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the+-- GNU General Public License for more details.+--+-- You should have received a copy of the GNU General Public License+-- along with this program; see the file COPYING. If not, write to+-- the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor,+-- Boston, MA 02110-1301, USA.++-- | Test.Runner.Driver contains the functions that determine which tests are+-- run, with which parameters and by how many threads.+module Test.Runner.Driver ( runTests, runTestsParallel,+ runTestsWithArgs,+ runTestsParallelWithArgs,+ Result(..),+ ) where++import Data.Maybe ( isJust, isNothing, fromJust )+import Control.Concurrent ( forkIO )+import Control.Concurrent.STM ( newTVar, readTVar, writeTVar, atomically,+ retry, TVar )+import Test.QuickCheck ( Args(..), stdArgs )++import Test.Runner.Backends++-- | Shows a name, runs the test, and then shows whether it failed or not by+-- printing either "OK" or "FAIL!" to standard output.+run_showing_name :: Args -> (String, TestRunnerTest) -> IO (Maybe String)+run_showing_name qcArgs (name, TestRunnerTest t) = do+ putStr (name ++ ": ")+ r <- runWithArgs qcArgs t+ putStrLn (if isNothing r then "OK" else "FAIL!")+ return r+++-- | The result of the test runner mentions how many tests passed, and the names+-- and failure messages of the tests that failed.+data Result = Result { numPassed :: Int+ , failures :: [(String, String)]+ } deriving (Show, Eq, Ord)++-- | Run a list of named tests.+runTests :: [(String, TestRunnerTest)] -> IO Result+runTests = runTestsWithArgs stdArgs++-- | Run a list of named tests, using the given QuickCheck @Args@ for the+-- QuickCHeck tests.+runTestsWithArgs :: Args -> [(String, TestRunnerTest)] -> IO Result+runTestsWithArgs qcArgs namedTests = do+ results <- mapM (run_showing_name qcArgs) namedTests+ let namedResults = zip names results+ passed = length (filter isNothing results)+ failed = map sndFromJust $ filter (isJust . snd) namedResults+ return (Result passed failed)+ where (names, _) = unzip namedTests+ sndFromJust (name, result) = (name, fromJust result)++-- * Running tests in parallel++data RunnerState = RunnerState+ { testsToDo :: [(String, TestRunnerTest)]+ , passedTests :: [String]+ , failedTests :: [(String, String)] -- ^ Names and failure messages+ , numDone :: Int+ }++-- | Creates an initial test runner state given the list of named tests+initial_runner_state :: [(String, TestRunnerTest)] -> RunnerState+initial_runner_state ts = RunnerState ts [] [] 0++-- | Uses multiple threads to run a set of unit tests.+runTestsParallel :: Int -- ^ Number of worker threads to use+ -> [(String, TestRunnerTest)] -- ^ The tests with their names+ -> IO Result+runTestsParallel n namedTests = runTestsParallelWithArgs n stdArgs namedTests++-- | Use multiple threads to run a set of unit tests, and run the QuickCheck+-- tests with the given QuickCheck @Args@.+runTestsParallelWithArgs :: Int -- ^ Number of worker threads to use+ -> Args -- ^ Arguments to QuickCheck+ -> [(String, TestRunnerTest)] -- ^ Tests with names+ -> IO Result+runTestsParallelWithArgs n qcArgs namedTests = do+ let numToDo = length namedTests+ stateRef <- atomically (newTVar (initial_runner_state namedTests))+ sequence_ (replicate n (forkIO (test_runner_thread qcArgs stateRef)))+ -- now wait until the threads have completed+ atomically $ do+ state <- readTVar stateRef+ if numDone state == numToDo+ then return $ Result (length (passedTests state)) (failedTests state)+ else retry++-- | The main loop of a worker thread when doing a parallel run+test_runner_thread :: Args -> TVar RunnerState -> IO ()+test_runner_thread qcArgs stateRef = do+ nextTest <- getNextTest+ case nextTest of+ Just t -> run_one_test qcArgs stateRef t >> test_runner_thread qcArgs stateRef+ Nothing -> return () -- ready+ where getNextTest = atomically $ do+ state <- readTVar stateRef+ let tests_to_do = testsToDo state+ case tests_to_do of+ doNow:doLater -> do writeTVar stateRef (state { testsToDo = doLater })+ return (Just doNow)+ [] -> return Nothing++-- | Runs one test as a part of a parallel test run, and updates the global+-- state after it's done.+run_one_test :: Args -> TVar RunnerState -> (String, TestRunnerTest) -> IO ()+run_one_test qcArgs stateRef (name, TestRunnerTest t) = do+ result <- runWithArgs qcArgs t+ putStrLn (name ++ ": " ++ (if isNothing result then "OK" else "FAIL!"))+ atomically $ do+ state <- readTVar stateRef+ let ps = passedTests state+ fs = failedTests state+ state' = state { numDone = numDone state + 1 }+ state'' = case result of+ Nothing -> state' { passedTests = name : ps }+ Just msg -> state' { failedTests = (name, msg) : fs }+ writeTVar stateRef state''+
+ Test/Runner/Frontend.hs view
@@ -0,0 +1,136 @@+-- Copyright (C) 2009 Reinier Lamers+--+-- This program is free software; you can redistribute it and/or modify+-- it under the terms of the GNU General Public License as published by+-- the Free Software Foundation; either version 2, or (at your option)+-- any later version.+--+-- This program is distributed in the hope that it will be useful,+-- but WITHOUT ANY WARRANTY; without even the implied warranty of+-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the+-- GNU General Public License for more details.+--+-- You should have received a copy of the GNU General Public License+-- along with this program; see the file COPYING. If not, write to+-- the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor,+-- Boston, MA 02110-1301, USA.++-- | Test.Runner.Frontend contains the code for the prefabricated unit test+-- executable, like command-line argument parsing and handling.+module Test.Runner.Frontend ( testRunnerMain ) where++import System.IO ( hPutStrLn, stderr, hSetBuffering, stdout,+ BufferMode ( NoBuffering ) )+import System.Console.GetOpt ( OptDescr(..), ArgDescr(..), getOpt, usageInfo,+ ArgOrder(Permute) )+import System.Environment ( getArgs )+import System.Random ( StdGen )+import System.Exit ( exitWith, ExitCode(ExitSuccess) )+import Data.Maybe ( isJust )+import Text.Regex ( mkRegex, matchRegex )+import Test.QuickCheck ( Args(..), stdArgs )++import Test.Runner.Backends ( TestRunnerTest(..) )+import Test.Runner.Driver ( runTestsParallelWithArgs, Result(..) )++-- | testRunnerMain is intended to be used as the main function of a unit test+-- program. It takes as an argument the complete list of unit tests for a+-- package.+testRunnerMain :: [(String, TestRunnerTest)] -> IO ()+testRunnerMain tests = do+ hSetBuffering stdout NoBuffering+ maybeFlags <- parse_args `fmap` getArgs+ case maybeFlags of+ Nothing -> do hPutStrLn stderr "testrunner: Unrecognized arguments on command line"+ printUsageAndDie+ Just flags ->+ let showHelp = not (null [x | x <- flags,+ case x of+ ShowHelp -> True+ _ -> False])+ in if showHelp+ then printUsageAndDie+ else runWithFlags flags tests++printUsageAndDie :: IO ()+printUsageAndDie = do+ putStr (usageInfo "unit - run darcs unit tests" opts)+ exitWith ExitSuccess++-- | Runs a set of tests according to the given command-line flags. The+-- @ShowHelp@ flag is assumed to have been handled already and is ignored by+-- this function.+runWithFlags :: [UnitFlag] -> [(String, TestRunnerTest)] -> IO ()+runWithFlags flags tests = runAndShowTests numThreads qcArgs matchingTests+ where matchingTests = filter (isJust . (matchRegex matchEx) . fst) tests+ qcArgs = case replayValues of+ Nothing -> stdArgs+ Just (seed, size) -> stdArgs { replay = Just (seed, size) }+ numThreads | null jobsArgs = 1+ | otherwise = last jobsArgs+ jobsArgs = [j | NumJobs j <- flags]+ matchEx | null matchArgs = mkRegex ".*" -- matches any string+ | otherwise = mkRegex (last matchArgs)+ matchArgs = [ex | Matching ex <- flags]+ replayValues | null replayArgs = Nothing+ | otherwise = Just (last replayArgs)+ replayArgs = [(seed, size) | QuickCheckReplay seed size <- flags]++-- | Run tests in a number of threads, and give a summary after all tests have+-- been run+runAndShowTests :: Int -> Args -> [(String, TestRunnerTest)] -> IO ()+runAndShowTests numThreads qcArgs tests = do+ results <- runTestsParallelWithArgs numThreads qcArgs tests+ putStr (show (numPassed results) ++ " tests passed.")+ if not (null (failures results))+ then do putStrLn " Failing tests:"+ mapM_ (putStr . formatFailure) (failures results)+ else putChar '\n'+ where formatFailure (name, output) =+ " " ++ name ++ ":\n" +++ ((unlines . map (" "++) . lines) output)++-- | Data type describing command line flag+data UnitFlag = ShowHelp+ | NumJobs Int+ | Matching String+ | QuickCheckReplay StdGen Int++-- | Parse a string to UnitFlag that describes the number of jobs to run. Exits+-- in case of malformed input.+parse_numjobs :: String -> UnitFlag+parse_numjobs s = case reads s of+ [(x,"")] -> NumJobs x+ _ -> error "Invalid number of Haskell threads given"++parse_qc_replay :: String -> UnitFlag+parse_qc_replay s = QuickCheckReplay seed size+ where (seedString, sizeString) = break (==',') s+ seed = case reads seedString :: [(StdGen, String)] of+ [(seed', "")] -> seed'+ _ -> error "Invalid QuickCheck seed given"+ size = case sizeString of+ [] -> error "Empty QuickCheck size given"+ sizeString' -> case reads (tail sizeString') of+ [(size', "")] -> size'+ _ -> error "Invalid QuickCheck size given"++-- | List of possible command line options+opts :: [OptDescr UnitFlag]+opts = [Option ['j'] ["jobs"] (ReqArg parse_numjobs "NUM") "Number of Haskell threads to run unit tests (you need +RTS -N<NUM> too)"+ ,Option ['m'] ["matching"] (ReqArg Matching "REGEX") "Run only tests matching the given POSIX regular expression"+ ,Option ['r'] ["quickcheck-replay"] (ReqArg parse_qc_replay "SEED,SIZE") "Run QuickCheck tests once with given seed and size"+ ,Option ['h'] ["help"] (NoArg ShowHelp) "Show usage information and exit"+ ]++-- | Parses the command line options with @getOpt@ and returns @Nothing@ in the+-- case of an invalid command line, or the parsed options otherwise.+parse_args :: [String] -- ^ The list of command line args as from+ -- @getArgs@+ -> Maybe [UnitFlag] -- ^ List of options represented by @UnitFlag@+ -- values+parse_args args | not (null nonopts)+ || not (null errs) = Nothing+ | otherwise = Just vals+ where (vals, nonopts, errs) = getOpt Permute opts args+
+ Test/Runner/Test.hs view
@@ -0,0 +1,83 @@+-- Copyright (C) 2009 Reinier Lamers+--+-- This program is free software; you can redistribute it and/or modify+-- it under the terms of the GNU General Public License as published by+-- the Free Software Foundation; either version 2, or (at your option)+-- any later version.+--+-- This program is distributed in the hope that it will be useful,+-- but WITHOUT ANY WARRANTY; without even the implied warranty of+-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the+-- GNU General Public License for more details.+--+-- You should have received a copy of the GNU General Public License+-- along with this program; see the file COPYING. If not, write to+-- the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor,+-- Boston, MA 02110-1301, USA.++module Main where++import System.Exit ( exitWith, ExitCode(..) )+import Test.HUnit ( runTestTT, Test(..), Counts(..), Assertion, assertEqual,+ assertFailure, assertBool )++import Test.Runner.Backends ( TestRunnerTest(..), runWithQuickCheck )+import Test.Runner.Driver ( runTests, Result(Result), runTestsParallel )++main :: IO ()+main = do+ counts <- runTestTT tests+ if errors counts == 0 && failures counts == 0+ then exitWith ExitSuccess+ else exitWith (ExitFailure 1)++tests :: Test+tests = TestList [ TestLabel "Checking that empty list of tests does not fail" (TestCase emptyListTest)+ , TestLabel "Checking singleton success" (TestCase singletonSuccess)+ , TestLabel "Checking singleton failure" (TestCase singletonFailure)+ , TestLabel "Checking parallel testing" (TestCase parallelNumber)+ ]+++-- | Verify that testrunner does the right thing when given an empty list+emptyListTest :: Assertion+emptyListTest = do+ res <- runTests [] + assertEqual "result for empty list" (Result 0 []) res++singletonSuccess :: Assertion+singletonSuccess = do+ res <- runTests [("success", TestRunnerTest True)]+ assertEqual "result for singleton success" (Result 1 []) res+ resHUnit <- runTests [("HUnit success", TestRunnerTest (TestCase (assertBool "success" True)))]+ assertEqual "result for singleton success with HUnit" (Result 1 []) resHUnit+ resQC <- runTests [("QuickCheck success", runWithQuickCheck True)]+ assertEqual "result for singleton success with QuickCheck" (Result 1 []) resQC+ resIO <- runTests [("IO success", TestRunnerTest (return True :: IO Bool))]+ assertEqual "result for singleton success with IO" (Result 1 []) resIO++singletonFailure :: Assertion+singletonFailure = do+ (Result numPassed fails) <- runTests [("failure", TestRunnerTest False)]+ assertEqual "no successes in singleton failure list" 0 numPassed+ assertEqual "one failure in singleton failure list" 1 (length fails)+ (Result numPassedHU failsHU) <- runTests [("HUnit failure", TestRunnerTest (TestCase (assertFailure "failure")))]+ assertEqual "no successes in singleton failure list for HUnit" 0 numPassedHU+ assertEqual "one failure in singleton failure list for HUnit" 1 (length failsHU)+ (Result numPassedQC failsQC) <- runTests [("QuickCheck failure", runWithQuickCheck False)]+ assertEqual "no successes in singleton failure list for QuickCheck" 0 numPassedQC+ assertEqual "one failure in singleton failure list for QuickCheck" 1 (length failsQC)+ (Result numPassedIO failsIO) <- runTests [("IO failure", TestRunnerTest (return False :: IO Bool))]+ assertEqual "no success in singleton failure list for IO" 0 numPassedIO+ assertEqual "one failure in singleton failure list for QuickCheck" 1 (length failsIO)++-- | Checks that the number of tests executed by the parallel runner is equal to+-- the number of tests given (this may easily go wrong in the case of race+-- conditions).+parallelNumber :: Assertion+parallelNumber = do+ let twoTests = [("success", TestRunnerTest True), ("failure", TestRunnerTest False)]+ (Result numPassed fails) <- runTestsParallel 8 (concat $ replicate 500 twoTests)+ let numRun = numPassed + length fails+ assertEqual "Number of tests run equals number of tests given" 1000 numRun+
+ homepage.html view
@@ -0,0 +1,52 @@+<?xml version="1.0" encoding="utf-8"?>+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">+<html xmlns="http://www.w3.org/1999/xhtml">+ <head>+ <title>testrunner</title>+ <style type="text/css">+ * { font-family: verdana, sans-serif; }+ body { margin-left: 20%;+ margin-right: 20%; }+ img { border: none;+ margin: 12px; }+ code { font-family: monospace; }+ </style>+ </head>+ <body>+ <h1>testrunner</h1>+ <p>testrunner is a <a href="http://www.haskell.org/">Haskell</a> library for+ running unit tests. It has the+ following distinguishing features:</p>+ <ul>+ <li>It can run unit tests in parallel.</li>+ <li>It can run QuickCheck and HUnit tests as well as simple boolean+ expressions.</li>+ <li>It comes with a ready-made main function for your unit test+ executable.</li>+ <li>This main function recognizes command-line arguments to select tests+ by name and replay QuickCheck tests.</li>+ </ul>++ <p>testrunner was spun off the <a href="http://darcs.net/">darcs</a>+ project.</p>++ <h2>Using testrunner</h2>+ <p>A tutorial for testrunner can be found <a+ href="using-testrunner.html">here</a>.</p>+ <h2>Getting testrunner</h2>+ <p>You can download the 0.9 release of testrunner <a+ href="releases/testrunner-0.9.tar.gz">here</a>.</p>+ <p>The darcs repository of testrunner is <a+ href="http://code.haskell.org/testrunner/"><code>http://code.haskell.org/testrunner/</code></a>.+ Thus, use <code>darcs get http://code.haskell.org/testrunner/</code> to check+ out the latest code.</p>+ <div id="bottombanner">+ <a href="http://www.haskell.org/hpc"><img src="hpcbadge.jpg" alt="hpc+ badge"/></a>+ <a href="http://validator.w3.org/check?uri=referer">+ <img src="http://www.w3.org/Icons/valid-xhtml10-blue" alt="Valid XHTML 1.0 Strict" height="31" width="88" />+ </a>+ </div>+ </body>+</html>+
+ testrunner.cabal view
@@ -0,0 +1,37 @@+Name: testrunner+Version: 0.9+Category: Testing+Synopsis: Easy unit test driver framework+Description: testrunner is a framework to run unit tests. It has the+ following distinguishing features:++ * It can run unit tests in parallel.++ * It can run QuickCheck and HUnit tests as well as simple+ boolean expressions.++ * It comes with a ready-made main function for your unit test+ executable.++ * This main function recognizes command-line arguments to select+ tests by name and replay QuickCheck tests.+License: GPL+License-file: LICENSE+Author: Reinier Lamers <tux_rocker@reinier.de>+Maintainer: Reinier Lamers <tux_rocker@reinier.de>+Build-Type: Simple+Cabal-Version: >=1.2++Library+ Build-Depends: QuickCheck>=2.1,+ HUnit,+ stm,+ base >=3 && <5,+ regex-compat,+ random+ Exposed-modules: Test.Runner,+ Test.Runner.Frontend,+ Test.Runner.Driver,+ Test.Runner.Backends+ ghc-options: -Wall+
+ using-testrunner.html view
@@ -0,0 +1,160 @@+<?xml version="1.0" encoding="utf-8"?>+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">+<html xmlns="http://www.w3.org/1999/xhtml">+ <head>+ <title>Using testrunner</title>+ <style type="text/css">+ * { font-family: verdana, sans-serif; }+ body { margin-left: 20%;+ margin-right: 20%; }+ code { font-family: monospace; }+ pre { font-family: monospace; }+ </style>+ </head>+ <body>+ <h1>Using testrunner</h1>+ <p>This is a tutorial for the <em>testrunner</em> library. For the details of the exported API, please+ consult the <a href="haddock/">haddock+ documentation.</a></p>++ <h2>Two ways to use it</h2>++ <p>+ There are two ways you can use the testrunner library. The first is to put+ all the unit tests of your application in a big list, and then have a+ one-line <code>main</code> function for your unit test program that calls+ the <code>testRunnerMain</code> function from <code>Test.Runner</code> to+ run the unit tests. </p>+ <p>If you use testrunner this way, you get a unit test program with a couple+ of nice features almost for free. You can replay <a+ href="http://www.cs.chalmers.se/~rjmh/QuickCheck/">QuickCheck</a> tests,+ select which tests to run, and choose how many tests to run at the same+ time from the command line. If you wish to extend the functionality of the unit test+ program however, you have to modify the testrunner library.</p>+ <p>That is why there is another way to use testrunner: you can use the+ functions exported by <code>Test.Runner.Driver</code> and+ <code>Test.Runner.Backends</code> to run unit tests. You can tell testrunner+ how many tests to run in parallel, or tell it the QuickCheck arguments to+ use when running QuickCheck tests. You get back a data structure that you+ can examine to see which tests failed and which succeeded. This way of using+ testrunner is not further described in this document, but see the <a+ href="haddock/Test-Runner-Driver.html">haddock</a>.</p>++ <h2>Using testrunner the first way, using+ <code>testRunnerMain</code></h2>+ <p>Say you have a program called <code>hello</code>, which contains the+ simple source file <code>Hello.hs</code> shown here:</p>+ <pre>+ module Hello where++ helloWorld :: String+ helloWorld = hello "world"++ hello :: String -> String+ hello s = "hello " ++ s+ </pre>++ <p>Now we want to define unit tests for this module. Let's say we come up+ with the following. I realize that they are silly, but this is+ not a guide to testing the right properties, this is a guide to using+ testrunner. Here is the test code, in a file <code>Test.hs</code>:</p>+ <pre>+ module Main where++ import Test.HUnit+ import Test.QuickCheck++ import Hello++ -- use HUnit to assert that helloWorld produces "hello world"+ hunitTest :: Test+ hunitTest = TestCase $ do+ assertEqual "hello world" "hello world" helloWorld++ -- use QuickCheck to check the length of hello's result+ helloLength :: String -> Bool+ helloLength s = length (hello s) == length "hello " + length s++ -- A simple boolean expression that states that hello of an empty string is+ -- "hello"+ helloEmpty :: Bool+ helloEmpty = hello "" == "hello "+ </pre>+ <p>The only thing lacking from this test module is a main function. Of+ course, you could easily write your own, but the simple version would not+ support parallel test execution, selecting unit tests to execute, or running+ the QuickCheck test with the same random sample as a previous run.</p>++ <p>+ testrunner lets you write your main function more concise and gives you+ those nice properties for free. Here is <code>Test.hs</code> again, with the+ testrunner-based main function:+ </p>+ <pre>+ module Main where++ import Test.HUnit+ import Test.QuickCheck+ import Test.Runner++ import Hello++ -- use HUnit to assert that helloWorld produces "hello world"+ hunitTest :: Test+ hunitTest = TestCase $ do+ assertEqual "hello world" "hello world" helloWorld++ -- use QuickCheck to check the length of hello's result+ helloLength :: String -> Bool+ helloLength s = length (hello s) == length "hello " + length s++ -- A simple boolean expression that states that hello of an empty string is+ -- "hello"+ helloEmpty :: Bool+ helloEmpty = hello "" == "hello "++ tests :: [(String, TestRunnerTest)]+ tests = [("helloWorld value", TestRunnerTest hunitTest),+ ("hello length", runWithQuickCheck helloLength),+ ("value of hello applied to empty string", TestRunnerTest helloEmpty)]+ main :: IO ()+ main = testRunnerMain tests+ </pre>+ <p>Here, we put all the unit tests in a list of type <code>[(String,+ TestRunnerTest)]</code>, where the first element of every tuple is the+ name of the test. We use the <code>TestRunnerTest</code> constructor+ to create a <code>TestRunnerTest</code> from HUnit tests and boolean+ expressions, and use <code>runWithQuickCheck</code> to turn QuickCheck+ <code>Testable</code>s into <code>TestRunnerTest</code>s. Then our main+ function is a one-liner that calls <code>testRunnerMain</code> on this+ list.</p>++ <p>Now your powerful unit test program is ready! For example, try the+ commands:</p>+ <pre>+ ghc --make -threaded Test.hs -o test+ ./test+ ./test -m length+ ./test -r '1387922338 2147483372,86'+ ./test -r '1387922338 2147483372,86' -m length+ ./test -j 3 +RTS -N3+ </pre>+ <p>The first command compiles the <code>Test.hs</code> file to an executable+ named <code>test</code>. The second command just runs all the tests and+ reports the result. The second command runs only the test whose name matches+ the regular expression 'length'. The third command tries the QuickCheck test+ (<code>helloLength</code>) with the random seed of 1387922338 2147483372 and+ size 86, a combination that makes a lot of darcs unit tests fail. The fourth+ command runs only the QuickCheck tests like the second, and runs it with the+ random seed and size of the third command. The fifth command runs all the+ unit tests in parallel.</p>++ <p>You may notice that screen output becomes garbled when running tests in+ parallel. This is partly due to the QuickCheck API, and that is why it is+ not fixed. The report at the end (the single line "3 tests passed" in this+ example) is printed after the worker threads have+ quit, and thus will always be readable.</p>++ <p>That's it folks, enjoy!</p>+ </body>+</html>