diff --git a/CHANGELOG.md b/CHANGELOG.md --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,11 +1,17 @@ # Revision history for erebos-tester +## 0.2.2 -- 2024-05-17 + +* Fix unshare failing with newer compilers +* Documentation and helptext updates +* Compatibility with GHC up to 9.10 + ## 0.2.1 -- 2024-05-14 -* Selection of test from test file path on command line using ':' charater -* Added --repeat option to run the tests multiple times -* Added --wait option to wait at the end of each test -* Added 'flush' command +* Selection of test from test file path on command line using '`:`' charater +* Added `--repeat` option to run the tests multiple times +* Added `--wait` option to wait at the end of each test +* Added `flush` command * Show record selectors in failure reports * Compatibility with GHC up to 9.8 diff --git a/README.md b/README.md new file mode 100644 --- /dev/null +++ b/README.md @@ -0,0 +1,328 @@ +Erebos Tester +============= + +Developed for testing the [Erebos protocol implementation](https://erebosprotocol.net/erebos) +and [C++ library](https://erebosprotocol.net/cpp), generally intended mainly +for testing networking code – it provides an easy way to execute the tested +program simultaneously on multiple nodes within a virtual network. These nodes +can be configured in a single or multiple subnets, and their properties, like +whether they are connected, can be changed during test run according to a given +script. + +The test framework uses two components from the tested project: + +1. Test tool – executable that accepts commands on standard input, executes the + tested functionality based on those, and provides output on standard output. + The output can be asynchronous, i.e. some events may trigger a message to be + generated while waiting for output of a command; interpretation is up to the + provided test script. + + The test tool can be set either: + * by the `--tool` command-line parameter of `erebos-tester`, or + * in the `erebos-tester.yaml` configuration file, or + * by the `EREBOS_TEST_TOOL` environment variable. + +2. Test script – defines how to run the instances of test tool and in what kind + of network topology. Contains commands to send to the test tool instances + and rules to interpret the responses. The script is written in a custom + language described below. + +Usage +----- + +The `erebos-tester` tool, when executed without any arguments, +looks for a `erebos-tester.yaml` file in the current or any parent directory (see below for details). +Run `erebos-tester --help` for details about command-line parameters. + +The tester can be installed from sources or directly via cabal: +``` +cabal install erebos-tester +``` + +When available in the `PATH`, it can be run to test the [Haskell Erebos implementation](https://erebosprotocol.net/erebos): +``` +git clone git://erebosprotocol.net/erebos +cd erebos +cabal build +erebos-tester --tool="$(cabal list-bin erebos) test" --verbose +``` + +or the [C++ one](https://erebosprotocol.net/cpp): +``` +git clone git://erebosprotocol.net/cpp +cd cpp +cmake -B build +cmake --build build +erebos-tester --verbose +``` + +To run tests from a given test file, pass it as command-line argument: +``` +erebos-tester path/to/script.test +``` + +To select single test from a file, use `:` separator: +``` +erebos-tester path/to/script.test:TestName +``` + +Configuration +------------- + +To allow running `erebos-tester` without the need to supply project-specific configuration on command line, +per-project configuration can be done using `erebos-tester.yaml` file placed in the root of the project +(or other directory from which the tests will be executed). +This is a YAML file with following fields: + +* `tool`: path to the test tool, which may be overridden by the `--tool` command-line option. +* `tests`: glob pattern that expands to all the test script files that should be used. + +Script language +--------------- + +The test script language uses indentation to define the command blocks, e.g. to +define a test body or denote the scope of variables. Each command is on its own +line, terminated by newline. Commands accept arguments preceded by name/keyword +in arbitrary order, to make the behavior clear without the need to know +the expected order of the parameters. + +For examples, see tests within the +[Erebos implementation repository](https://code.erebosprotocol.net/erebos/tree/test). + +Each test script file consists of one or more test cases, started with `test` +keyword, with its body within indented block: + +``` +test []: + +``` + +Test name is optional, but if present can be used to run the single test from +a file that contains multiple tests. + +### Types + +The script language is strictly typed without any implicit conversions, +although types can not be (as of now) declared explicitly and are always inferred. +Each expression has specific concrete type, polymorphic types are not supported (yet). + +#### integer + +Integer numbers. Entered as decimal literals and used in arithmetic expressions: +``` +let x = 2 +let y = 3 +let z = x * 2 + y +``` + +#### number + +Arbitrary-precision numbers. Entered as literals with decimal point or percentage and used in arithmetic expressions: +``` +let x = 2.1 +let y = 34% +let z = x * 2.0 + y +``` + +#### string + +String literals are enclosed in double quotes (`"`), +using backslash to escape special characters (`"`, `\` and `$`) +and to represent some others (`\n` for newline). +``` +let s = "some text" +``` + +Dollar sign (`$`) can be used to expand variables (numbers are expanded to decimal representation). +``` +let a = "abc" +let b = 4 +let c = "$a $b" # = "abc 4" +``` + +Arbitrary expression can be used within additional curly braces: +``` +let a = 2 +let b = 3 +let s = "abc ${2*a + b}" # = "abc 7" +``` + +#### regex + +Regular expression literals are enclosed in slash characters (`/`): +``` +let re = /a.*/ # match any string starting with 'a' +``` + +Dollar-expansion can be used here as well. +Strings expand to regular expressions matching the exact string, +regular expressions expand are used directly. +``` +let str = "." +let re1 = /./ +let re2 = "$str$re1" # match '.' followed by any character +``` + +#### boolean + +Result of comparison operators `==` and `/=`. + +#### network + +Represents network/subnet, created by `subnet` command and used by `subnet`, `node`, `spawn` and network configuration commands. + +#### node + +Represents network node, created by `node` command and used by `spawn` or network configuration commands. + +Members: + +`ip`: string representation of node's IP address. + +#### process + +Represents running process. Created by `spawn`, used by `send` and `expect` commands. + +Members: + +`node`: node on which the process is running + +#### list + +Lists are written using bracket notation: +``` +let numbers = [1, 2, 4] +``` + +List elements can be of any type, but all elements of a particular list must have the same type. + +Used in the `for` command. + +### Build-in commands + +``` +subnet [of ] +``` + +Create a subnet within a `` (or context network if omitted) and assign the new network to the variable ``. + +``` +node [on ] +``` + +Create a node on network `` (or context network if omitted) and assign the new node to the variable ``. + +``` +spawn as [on ( | )] +``` + +Spawn a new test process on `` or `` (or one from context) and assign the new process to variable ``. +When spawning on network, create a new node for this process. + +The process is terminated when the variable `` goes out of scope (at the end of the block in which it was created) by closing its stdin. +When the process fails to terminate successfully within a timeout, the test fails. + +``` +send to +``` +Send line with `` to the standard input of ``. + +``` +expect from [capture [, ... ]] +``` +Check whether `` produces line matching `` on standard output, and if this does not happen within current timeout, the test fails. +Output lines produced before starting this command and not matched by some previous `expect` are accepted as well. +Output lines not matching `` are ignored by this `expect` call, i.e. do not cause the `expect` call to fail. + +Regular expressions are anchored on both sides, so must match the entire line. +If e.g. only the beginning should be matched, the passed regular expression needs to end with `.*`. + +The regular expression can contain capture groups – parts enclosed in parentheses (`(`, `)`). +In that case the expect command has to have the `capture` clause with matching number of variable names. +Results of the captures are then assigned to the newly created variables as strings. + +``` +flush [from ] +``` + +Flush memory of `` output, so no following `expect` command will match anything produced up to this point. + +``` +guard +``` + +Check whether boolean expression `` is true; if not, the test fails. + +``` +disconnect_node [] +``` + +Disconnect `` from network – state of the veth network link from the node is set to down. +The effect lasts until the end of the block. + +``` +disconnect_nodes [] +``` + +Disconnect all nodes of ``. The network bridge interface state is set to down. +The effect lasts until the end of the block. + +``` +disconnect_upstream [] +``` + +Disconnect network upstream – state of the veth network link connecting network bridge to the upstream network is set to down. +The effect lasts until the end of the block. + +``` +packet_loss [on ] +``` + +Set the packet loss rate on the node's veth link to `` as decimal number or percentage, e.g. `0.2` or `20%` for 20% packet loss rate. +The effect lasts until the end of the block. + +``` +for in : + +``` + +Execute `` for each element of list ``, with current element assigned to ``. + +``` +local: + +``` + +Execute `` in a new local scope. Used to restrict scope of variables or duration of effects. + +``` +with : + +``` + +Execute `` with `` as context. + +``` +wait +``` + +Wait for user input before continuing. Useful mostly for debugging or test development. + + +Optional dependencies +--------------------- + +The test framework can use some other tools to help with debugging or development. + +### GDB + +If GDB is installed, it's possible to use `--gdb` command-line switch of the `erebos-tester` tool to use the debugger. +The GDB session is started in background and tester uses the GDB machine interface to communicate with it. +Whenever a new process is spawned, it is attached to the debugger as a new inferior. +In case any process is terminated by a signal, e.g. crashes with segfault, interactive GDB session is opened. + +### tcpdump + +If `tcpdump` binary is found in the `PATH`, +it is used to generate network log in the pcap capture format within the test directory, +separately for each virtual network (specifically its bridge interface). diff --git a/erebos-tester.cabal b/erebos-tester.cabal --- a/erebos-tester.cabal +++ b/erebos-tester.cabal @@ -1,7 +1,7 @@ -cabal-version: 2.2 +cabal-version: 3.0 name: erebos-tester -version: 0.2.1 +version: 0.2.2 synopsis: Test framework with virtual network using Linux namespaces description: This framework is intended mainly for networking libraries/applications and @@ -18,6 +18,7 @@ -- copyright: category: Testing extra-doc-files: + README.md CHANGELOG.md flag ci @@ -40,18 +41,28 @@ -- sometimes needed for backward/forward compatibility: -Wno-error=unused-imports + build-depends: + base ^>= { 4.15, 4.16, 4.17, 4.18, 4.19, 4.20 }, + executable erebos-tester import: common + ghc-options: + -- disable interval timer to avoid spawing thread that breaks unshare(CLONE_NEWUSER) + -with-rtsopts=-V0 + if impl(ghc >= 9.8) + ghc-options: + -- no multithreading is allowed for unshare(CLONE_NEWUSER) + -single-threaded main-is: Wrapper.hs -- other-modules: -- other-extensions: - build-depends: base >=4.13 && <5, - directory >=1.3 && <1.4, - filepath ^>=1.4.2.1, - linux-namespaces ^>=0.1.3, - process ^>=1.6.9, - unix >=2.7 && <2.9, + build-depends: + directory >=1.3 && <1.4, + filepath ^>= { 1.4.2.1, 1.5.2 }, + linux-namespaces^>=0.1.3, + process ^>=1.6.9, + unix >=2.7 && <2.9, hs-source-dirs: src default-language: Haskell2010 @@ -99,26 +110,24 @@ TypeApplications TypeFamilies TypeOperators - build-depends: base >=4.13 && <5, - bytestring >=0.10 && <0.12, - containers ^>=0.6.2.1, - directory ^>=1.3.6.0, - filepath ^>=1.4.2.1, - generic-deriving >=1.14 && <1.15, - Glob >=0.10 && <0.11, - haskeline >=0.8 && <0.9, - HsYAML >=0.2 && <0.3, - lens >=5.0 && <5.3, - megaparsec >=9.0 && <10, - mtl >=2.2 && <2.4, - parser-combinators >=1.3 && <1.4, - process ^>=1.6.9, - regex-tdfa ^>=1.3.1.0, - scientific >=0.3 && < 0.4, - stm ^>=2.5.0.1, - template-haskell >=2.17 && <2.22, - text >=1.2 && <2.1, - th-compat >=0.1 && <0.2, - unix >=2.7 && <2.9, + build-depends: + bytestring ^>= { 0.10, 0.11, 0.12 }, + containers ^>= { 0.6.2.1, 0.7 }, + directory ^>=1.3.6.0, + filepath ^>= { 1.4.2.1, 1.5.2 }, + Glob >=0.10 && <0.11, + haskeline >=0.8 && <0.9, + HsYAML >=0.2 && <0.3, + megaparsec >=9.0 && <10, + mtl >=2.2 && <2.4, + parser-combinators >=1.3 && <1.4, + process ^>=1.6.9, + regex-tdfa ^>=1.3.1.0, + scientific >=0.3 && < 0.4, + stm ^>=2.5.0.1, + template-haskell^>= { 2.17, 2.18, 2.19, 2.20, 2.21, 2.22 }, + text ^>= { 1.2, 2.0, 2.1 }, + th-compat >=0.1 && <0.2, + unix >=2.7 && <2.9, hs-source-dirs: src default-language: Haskell2010 diff --git a/src/Main.hs b/src/Main.hs --- a/src/Main.hs +++ b/src/Main.hs @@ -47,7 +47,7 @@ (ReqArg (\str -> to $ \opts -> case break (==':') str of (path, []) -> opts { optDefaultTool = path } (pname, (_:path)) -> opts { optProcTools = (ProcName (T.pack pname), path) : optProcTools opts } - ) "PATH") + ) "") "test tool to be used" , Option ['v'] ["verbose"] (NoArg (\opts -> opts { optVerbose = True })) @@ -55,7 +55,7 @@ , Option ['t'] ["timeout"] (ReqArg (\str -> to $ \opts -> case readMaybe str of Just timeout -> opts { optTimeout = timeout } - Nothing -> error "timeout must be a number") "SECONDS") + Nothing -> error "timeout must be a number") "") "default timeout in seconds with microsecond precision" , Option ['g'] ["gdb"] (NoArg $ to $ \opts -> opts { optGDB = True }) @@ -67,7 +67,7 @@ (NoArg $ to $ \opts -> opts { optKeep = True }) "keep test directory even if all tests succeed" , Option ['r'] ["repeat"] - (ReqArg (\str opts -> opts { optRepeat = read str }) "COUNT") + (ReqArg (\str opts -> opts { optRepeat = read str }) "") "number of times to repeat the test(s)" , Option [] ["wait"] (NoArg $ to $ \opts -> opts { optWait = True }) @@ -106,7 +106,13 @@ exitFailure when (optShowHelp opts) $ do - let header = "Usage: erebos-tester [OPTION...]" + let header = unlines + [ "Usage: erebos-tester [