#+TITLE: Tasty SUGAR - Search Using Golden Answer References
#+AUTHOR: Kevin Quick <kquick@galois.com>
* Tasty SUGAR - Search Using Golden Answer References
The ~tasty-sugar~ package extends the tasty testing framework with
the ability to generate tests based on golden answer files found for
specific inputs. Multiple answers may be specified with different
parameterization, and there can be associated files that are
presented to the test as well.
The primary use of tasty-sugar is to _generate test cases_ based on
the contents of a directory, where the presence of various files
determine which tests are generated.
** Elements of tasty-sugar:
* Tasty.Sugar.CUBE :: Configuration Using Base Expectations
Describes the configuration for tasty-sugar tests, including
where they are located and what syntax the files should have.
* Tasty.Sugar.Sweets :: Specifications With Existing Expected Testing Samples
The tasty-sugar library uses one or more CUBE's to generate a
set of test configurations based on the existing files found,
outputting a list of Sweets representing the existing test
data.
# * KWQ: something to generate the tests
# * KWQ: name of the test runner to show the tests
# * KWQ: a test runner or option to write new outputs
* How to use tasty-sugar
Full information on tasty-sugar features and capabilities will be
provided in a later [[id:de026768-f805-4d30-8299-522cdd70926b][Detailed Information]] section, but this is a
quick introduction describing various testing use cases and how
tasty-sugar can be used in those use cases.
For these motivational use cases, the example scenario basis is that
the target to be tested is a tool to parse binary ELF files and
generate various output information about those files (e.g. similar
to objdump, but in a Haskell library form).
** Single Expected Output per Test
* Scenario :: When running a test, it generates output that should
be compared to the expected data maintained in a file.
There is a simple, single expected output for each
test, and no inputs other than the test name.
For the example scenario, several actual ELF binary files were
collected and placed in the ~test/samples~ directory:
#+BEGIN_EXAMPLE
$ ls test/samples
empty.txt
fibonacci.c
fibonacci
foo.tar
hello.c
hello
ls.c
ls
tmux.c
tmux
$
#+END_EXAMPLE
Note that there are a couple of non-ELF files in there as well, to
verify the errors generated by our library when given
The actual outputs aren't known yet, but tasty-sugar can help with
that. Setup the tasty-sugar CUBE configuration as follows:
#+BEGIN_EXAMPLE
cube = mkCUBE { inputDirs = [ "test/samples" ]
, rootName = "*.c"
, expectedSuffix = "exp"
, associatedNames = [ ("binary", "") ]
}
main = do testSweets <- findSugar cube
tests <- withSugarGroups testSweets testGroup $
\sweets expIdx expectation -> return $
testCase (rootMatchName sweets <> " #" <> show expIdx) $ do
let Just binaryName = lookup "binary" $ associated expectation
r <- runTestOn binaryName
e <- readFile $ expectedFile expectation
r @?= e
defaultMain $ testGroup "elf" tests
runTestOn :: FilePath -> IO String
runTestOn f = ...
#+END_EXAMPLE
The tasty-sugar framework does not provide the actual testing: that
is still provided by the developer. Instead, the tasty-sugar
framework reads the contents of the ~test/samples~ directory and
analyses the available files to create a list of tests that should
be run. The tasty-sugar package also provides a function that can
organize the tests and invoke the user's test function once for
each test configuration.
If the tests are run at this point, the tasty-sugar framework will
do nothing.
Why?
The tasty-sugar framework will ignore any files in the target
directory that do not have an associated expected file describing
the expected output. This can be confirmed by running the tests
with the ~--showsearch~ argument, which will use an alternate tasty
ingredient that does not actually run the tests but write out the
search process and search results.
To get actual tests to run, simply create an expected file for each
of the input candidates. The contents of the file can be empty, or
any random data.
Running the tests now will result in a test created for each input
file that has a corresponding ~*.exp~ file. Note that tasty-sweet
doesn't actually read in any of the files, just invokes the test
creation function with the Sweets and Expectation data structures
that let the test do whatever is appropriate.
#+BEGIN_EXAMPLE
$ ls test/samples
empty.txt
empty.txt.exp
fibonacci
finonacci.c
fibonacci.exp
foo.tar
foo.tar.exp
hello
hello.c
hello.exp
ls
ls.c
ls.exp
tmux
tmux.c
$
#+END_EXAMPLE
Note that ~empty.txt~ and ~foo.tar~ will be ignored, even though
there is an ~.exp~ file for them because they don't match the
source target of ~*.c~. Similarly, ~tmux~ will be ignored because
there is no ~.exp~ file for it.
At this point, any changes to the target library that cause output
changes will be identified when running the tests.
** Single Input and Output per Test
* Scenario :: Similar to the previous scenario, but there is a file
containing the expected input needed by the test to
generate the output.
To extend the previous example, let us assume that in addition to
the pre-existing binaries that we will be generating a number of
"interesting" binaries to run the target library on. These will be
kept in a different directory where a different part of the build
will compile the sources to generate the binaries for testing:
#+BEGIN_EXAMPLE
$ ls test/src_samples
foo.c
foo
foo.expct
simple.c
simple
simple.expct
recursive.rs
recursive
recursive.expct
functional.hs
functional
functional.expct
$
#+END_EXAMPLE
Note that there are several different source types (C, Rust,
Haskell) involved, but each of them has an associated output binary
that the target library should be tested on.
In an initial approach, the source files can be ignored by the
testing code: simply create a ~FILE.expct~ file for each of the
binaries and use the same ~Tasty.Sugar.CUBE~ configuration for this
directory as for the previous directory.
However however an approach where the actual test written by the
user needs access to the source file itself for some reason. This
can be handled by specifying an "associated" file in the
~Tasty.Sugar.CUBE~ configuration:
#+BEGIN_EXAMPLE
cube = mkCUBE { inputDirs = [ "test/samples" ]
, rootName = "*.exe"
, expectedSuffix = "expct"
, associatedNames = [ ("c-source", ".c")
, ("rust-source", ".rs")
, ("haskell", ".hs)
]
}
ingredients = includingOptions sugarOptions :
sugarIngredients [cube] <> defaultIngredients
main = do testSweets <- findSugar cube
tests <- withSugarGroups testSweets testGroup $
\sweets expIdx expectation -> return $
testCase (rootMatchName sweets <> " #" <> show expIdx) $ do
e <- readFile $ expectedFile expectation
let assoc = associated expectation
f = rootFile sweets
r <- case lookup "c-source" assoc of
Just c -> runCTestOn f
Nothing ->
case lookup "rust-source" assoc of
Just r -> runRustTestOn f
Nothing ->
runHaskellTestOn f
r @?= e
defaultMainWithIngredients ingredients $
testGroup "elf" tests
runCTestOn :: FilePath -> IO String
runCTestOn f = ...
runRustTestOn :: FilePath -> IO String
runRustTestOn f = ...
runHaskellTestOn :: FilePath -> IO String
runHaskellTestOn f = ...
#+END_EXAMPLE
Now when tasty-sugar generates the test configurations, each test
will have a name, a source file, an expected file, and a single
associated file. The test is free to use these files in any way it
sees fit. For the configuration above, there would be 4 test
configurations provided to the test:
| Test Name | Input File | Expected File | Associated Files |
|------------+------------+------------------+---------------------------------|
| simple | simple | simple.expct | ("c-source", "simple.c") |
| foo | foo | foo.expct | ("c-source", "foo.c") |
| recursive | recursive | recursive.expct | ("rust-source", "recursive.rs") |
| functional | functional | functional.expct | ("haskell", "functional.hs") |
Note that if both "simple.c" and "simple.hs" files existed, then the
simple test configuration would get both as associated files.
** Single Input with different parameters producing different outputs
* Scenario :: For each input file, multiple tests should be run,
each with different parameters, and the expected
output may or may not depend on the parameter.
Using the previous example scenario, let's now assume that for each
of the source sample files, two different executables were built:
one with and one without optimization. Additionally, if they were a
C source file, then there was a version built with GCC and a version
built with Clang. The output executables are now named accordingly:
#+BEGIN_EXAMPLE
$ ls test/src_samples
foo.c
foo.noopt.clang.exe
foo.O0.gcc.exe
foo.opt.clang.exe
foo.O2.gcc.exe
foo.O3.gcc.exe
simple.c
simple.noopt.clang.exe
simple.noopt.gcc.exe
simple.opt-clang.exe
simple-opt.gcc-exe
recursive.rs
recursive.noopt.exe
recursive.opt.exe
functional.hs
functional.noopt.exe
functional.opt.exe
$
#+END_EXAMPLE
While the filenames are fairly regular, there are different numbers
of executables and different naming conventions for different files.
The opt/noopt/O0/O2/O3 and gcc/clang information is known to
tasty-sugar as a "parameter". Parameters can appear in the filename
in a specific order, and each parameter may have one of a set of
valid values (e.g. gcc or clang) or it may have any (free-form)
value (as with the optimization specification).
The ~Tasty.Sugar.CUBE~ confguration for is scenario is updated to:
#+BEGIN_EXAMPLE
cube = mkCUBE { inputDirs = [ "test/samples" ]
, rootName = "*"
, separators = "-."
, expectedSuffix = "expct"
, associatedNames = [ ("c-source", ".c")
, ("rust-source", ".rs")
, ("haskell", ".hs")
]
, validParams = [
("optimization", Nothing)
,("c-compiler", Just ["gcc", "clang"])
]
}
ingredients = includingOptions sugarOptions :
sugarIngredients [cube] <> defaultIngredients
main = do testSweets <- findSugar cube
tests <- withSugarGroups testSweets testGroup $
\sweets expIdx expectation -> return $
testCase (rootMatchName sweets <> " #" <> show expIdx) $ do
e <- readFile $ expectedFile expectation
let assoc = associated expectation
f = rootFile sweets
r <- case lookup "c-source" assoc of
Just c -> runCTestOn f
Nothing ->
case lookup "rust-source" assoc of
Just r -> runRustTestOn f
Nothing -> runHaskellTestOn f
r @?= e
defaultMainWithIngredients ingredients $
testGroup "elf" tests
runCTestOn :: FilePath -> String
runCTestOn f = ...
runRustTestOn :: FilePath -> String
runRustTestOn f = ...
runHaskellTestOn :: FilePath -> String
runHaskellTestOn f = ...
#+END_EXAMPLE
Parameters are separated by designated separator characters and must
appear in the order declared. The default separators are "." and
"-" (e.g. both of the two optimized executable files for the
simple.c source above are accepted).
Filenames may omit later parameter values: the file is assumed to
apply to all unspecified parameter values if there is no more
specific override. This can be very useful to avoid repetition and
copying when specifying test files.
In the above example, a ~simple.expected~ file would be used for all
four executables, but if there was also a
~simple.noopt-gcc.expected~ and a ~simple-opt.expected~ then the
former would be used only for the ~simple.noopt.gcc.exe~ and the
latter would be used for both the ~gcc~ and the ~clang~ executables,
leaving the ~sample.expected~ to be used only for the
~simple.opt-clang.exe~ file.
** Parameter ranges
In some cases, there are a large number of possible parameter values, but the
expected files change for only some of the parameter variations. While this
could normally be handled by an expected file with no corresponding parameter
value in its name to represent the default case, this is somewhat more
difficult when there are multiple "defaults". This is handled by _parameter
ranges_.
As an example parameter range, consider a situation that might test the output
of the llvm assembler. The llvm assembler output tends to be unchanged over a
couple of versions, but then there will be a change (e.g. adding a new LLVM
assembly instruction in llvm version 14) that will change the output, but the
new format will become the default output.
Conventionally, having a ~target.ll~ assembly file as the original default
expected file means that for every version starting at 14, there will be a
specific expected file (e.g. ~target-llvm14.ll~, ~target-llvm15.ll~,
~target-llvm16.ll~, ...) even though all of those files are identical.
To address this, the parameter values can specify an upper or lower range limit
and use the ~rangedParamAdjuster~ to adjust the Expectations generated. Using
this, the expected files could be something like ~target-llvm_pre12.ll~,
~target-llvm_pre14.ll~, ~target.ll~, along with an indication that the
parameter value of ~llvm_preN~ formed an upper limit. When used this way, the
~target-llvm_pre12.ll~ file will be chosen for versions 10 and 11 of llvm, the
~target-llvm_pre14.ll~ file would be chosen for versions 12 and 13 of llvm, and
the ~target.ll~ file would be chosen for llvm version 14 or higher.
For more information, see the documentation for the ~rangedParamAdjuster~
helper function, and the corresponding tests.
# ** Multiple Inputs with different parameters producing different outputs
#
# KWQ...
* Comparisons
** tasty-KAT
* The tasty-KAT package reads both the inputs and the outputs from a
single file, instaed of allowing the inputs to be a separate file
that can be processed by the target under test.
+ The tasty-sugar package allows inputs and outputs to be in
separate files, and additional "associated" files to be provided
as inputs to the test.
* The tasty-KAT package inputs and outputs must be specifiable in a
file with other KAT markup; this does not easily handle text
markup conflicts and binary inputs/outputs.
+ The tasty-sugar package does not attempt to interpret the
contents of the files, but simply passes them to the test
itself.
* The tasty-KAT package does not allow auxiliary files, or different
parameterized tests.
+ As mentioned above, tasty-sugar allows multiple auxiliary files
per tests, and allows test inputs and expected outputs to be
filename parameterized (with either constrained or free-form
parameter values).
** tasty-golden
* The tasty-golden package requires a 1:1 association between tests
and corresponding golden expected output files; it does not
support file-provided inputs, or associated files.
+ The tasty-sugar package allows multiple associated files in
addition to the primary input file.
+ The tasty-sugar package supports parameterization of expected
results (and associated files) as part of the filenames to allow
multiple tests per input.
* The tasty-golden package will write the expected results if the
expected file is missing.
+ The tasty-sugar package does not actually read or write any of
the files identified, it simply provides the names of the files
to the test generator function. The user's test code is
responsible for all file processing.
** tasty-silver
Similar to tasty-golden in functionality.
** Features unique to tasty-sugar
* Multiple potential outputs, parameterized by filename elements.
* Multiple associated input files.
* Search analysis mode showing how tests are generated based on the
available files.
* Automatic grouping of generated tests by parameter values.
* Limitations
* Huge numbers of directories or files will cause performance slowdowns
* Will throw any exception that the listDirectory function can throw.
* Detailed Information
:PROPERTIES:
:ID: de026768-f805-4d30-8299-522cdd70926b
:END:
** Requirements
* There must be a root (input) file to feed to the test
* There must be one or more "expected" results files for a root file
* There may be associated files for the root file required for the test
* All three groups of files may be parameterized by additional fields.
* All fields are represented by a common basename with optional
parameters and required associated suffixes, separated by
allowable separators.
All of the above may utilize globbing as provided by System.FilePath.Glob
* Examples
** Example:
For example, a test which would verify that the size of a compiled
file meets the expectations would specify:
#+BEGIN_EXAMPLE
CUBE =
{ inputDirs = [ "tests/samples" ] -- relative to cabal file
, separators = ".-"
, rootName = "*.c"
, associatedNames = [ ("exe", "exe")
, ("object", "o")
]
, expectedSuffix = "expected"
, validParams = [ ("arch" : Just ["ppc", "x86_64"]) ]
}
#+END_EXAMPLE
And given the following directory configuration:
#+BEGIN_EXAMPLE
tests/samples/
foo.c
bar.c
bar.exe
bar.ppc.exe
bar.expected
cow.c
cow.ppc.exe
cow.x86_64.exe
cow.expected
cow.ppc-expected
cow.x86.expected
moo.c
moo.exe
moo-expected
dog.exe
dog.expected
#+END_EXAMPLE
The result would be:
#+BEGIN_EXAMPLE
sweets =
[ Sweets
{ rootMatchName = "bar"
, rootBaseName = "bar"
, rootFile = "tests/samples/bar.c"
, expected =
[ Expectation
{ expectedFile = "tests/samples/bar.expected"
, associated = [ ("exe", "tests/samples/bar.exe") ]
, expParamsMatch = []
}
, Expectation
{ expectedFile = "tests/samples/bar.expected"
, associated = [ ("exe", "tests/samples/bar.ppc.exe") ]
, expParamsMatch = [ ("arch", "ppc") ]
}
]
},
, Sweets
{ rootMatchName = "cow"
, rootBaseName = "cow"
, rootFile = "tests/samples/cow.c"
, expected =
[ Expectation
{ expectedFile = "tests/samples/cow.ppc-expected"
, associated = [ ("exe", "tests/samples/cow.ppc.exe") ]
, expParamsMatch = [ { "arch", "ppc" } ]
}
, Expectedfile
{ expectedFile = "tests/samples/cow.expected"
, associated = [ ("exe", "tests/samples/cow.x86_64.exe") ]
, expParamsMatch = [ ("arch", "x86_64") ]
}
]
},
, Sweets
{ rootMatchName = "moo"
, rootBaseName = "moo"
, rootFile = "tests/samples/moo.c"
, expected =
[ Expectation
{ expectedFile = "tests/samples/moo-expected"
, associated = [ ("exe", "tests/samples/moo.exe") ]
, expParamsMatch = []
}
]
},
#+END_EXAMPLE
* FAQ
Why do the configurations need to be described by a ~Tasty.Sugar.CUBE~
data object? Why can't they be passed in on the command-line?
* Answer :: They could be, but there are a couple of issues that
would make that more awkward:
1. There would need to be a number of command-line
arguments to describe all of the CUBE information.
2. The tasty framework provides command-line parsing and
argument handling (and expects to do so). Handling
some command-line arguments prior to tasty and some
within tasty would be difficult and brittle (and also
note that the set of all tests must be known *prior*
to invoking the tasty main code; they cannot be added
dynamically after that point).
3. Testing should be consistent: relying on user-input during
testing would return different results for different input.