erebos-tester 0.2.1 → 0.2.2
raw patch · 4 files changed
+386/−37 lines, 4 filesdep −generic-derivingdep −lensdep ~basedep ~bytestringdep ~containers
Dependencies removed: generic-deriving, lens
Dependency ranges changed: base, bytestring, containers, filepath, template-haskell, text
Files
- CHANGELOG.md +10/−4
- README.md +328/−0
- erebos-tester.cabal +38/−29
- src/Main.hs +10/−4
CHANGELOG.md view
@@ -1,11 +1,17 @@ # Revision history for erebos-tester +## 0.2.2 -- 2024-05-17++* Fix unshare failing with newer compilers+* Documentation and helptext updates+* Compatibility with GHC up to 9.10+ ## 0.2.1 -- 2024-05-14 -* Selection of test from test file path on command line using ':' charater-* Added --repeat option to run the tests multiple times-* Added --wait option to wait at the end of each test-* Added 'flush' command+* Selection of test from test file path on command line using '`:`' charater+* Added `--repeat` option to run the tests multiple times+* Added `--wait` option to wait at the end of each test+* Added `flush` command * Show record selectors in failure reports * Compatibility with GHC up to 9.8
+ README.md view
@@ -0,0 +1,328 @@+Erebos Tester+=============++Developed for testing the [Erebos protocol implementation](https://erebosprotocol.net/erebos)+and [C++ library](https://erebosprotocol.net/cpp), generally intended mainly+for testing networking code – it provides an easy way to execute the tested+program simultaneously on multiple nodes within a virtual network. These nodes+can be configured in a single or multiple subnets, and their properties, like+whether they are connected, can be changed during test run according to a given+script.++The test framework uses two components from the tested project:++1. Test tool – executable that accepts commands on standard input, executes the+ tested functionality based on those, and provides output on standard output.+ The output can be asynchronous, i.e. some events may trigger a message to be+ generated while waiting for output of a command; interpretation is up to the+ provided test script.++ The test tool can be set either:+ * by the `--tool` command-line parameter of `erebos-tester`, or+ * in the `erebos-tester.yaml` configuration file, or+ * by the `EREBOS_TEST_TOOL` environment variable.++2. Test script – defines how to run the instances of test tool and in what kind+ of network topology. Contains commands to send to the test tool instances+ and rules to interpret the responses. The script is written in a custom+ language described below.++Usage+-----++The `erebos-tester` tool, when executed without any arguments,+looks for a `erebos-tester.yaml` file in the current or any parent directory (see below for details).+Run `erebos-tester --help` for details about command-line parameters.++The tester can be installed from sources or directly via cabal:+```+cabal install erebos-tester+```++When available in the `PATH`, it can be run to test the [Haskell Erebos implementation](https://erebosprotocol.net/erebos):+```+git clone git://erebosprotocol.net/erebos+cd erebos+cabal build+erebos-tester --tool="$(cabal list-bin erebos) test" --verbose+```++or the [C++ one](https://erebosprotocol.net/cpp):+```+git clone git://erebosprotocol.net/cpp+cd cpp+cmake -B build+cmake --build build+erebos-tester --verbose+```++To run tests from a given test file, pass it as command-line argument:+```+erebos-tester path/to/script.test+```++To select single test from a file, use `:` separator:+```+erebos-tester path/to/script.test:TestName+```++Configuration+-------------++To allow running `erebos-tester` without the need to supply project-specific configuration on command line,+per-project configuration can be done using `erebos-tester.yaml` file placed in the root of the project+(or other directory from which the tests will be executed).+This is a YAML file with following fields:++* `tool`: path to the test tool, which may be overridden by the `--tool` command-line option.+* `tests`: glob pattern that expands to all the test script files that should be used.++Script language+---------------++The test script language uses indentation to define the command blocks, e.g. to+define a test body or denote the scope of variables. Each command is on its own+line, terminated by newline. Commands accept arguments preceded by name/keyword+in arbitrary order, to make the behavior clear without the need to know+the expected order of the parameters.++For examples, see tests within the+[Erebos implementation repository](https://code.erebosprotocol.net/erebos/tree/test).++Each test script file consists of one or more test cases, started with `test`+keyword, with its body within indented block:++```+test [<name>]:+ <test block>+```++Test name is optional, but if present can be used to run the single test from+a file that contains multiple tests.++### Types++The script language is strictly typed without any implicit conversions,+although types can not be (as of now) declared explicitly and are always inferred.+Each expression has specific concrete type, polymorphic types are not supported (yet).++#### integer++Integer numbers. Entered as decimal literals and used in arithmetic expressions:+```+let x = 2+let y = 3+let z = x * 2 + y+```++#### number++Arbitrary-precision numbers. Entered as literals with decimal point or percentage and used in arithmetic expressions:+```+let x = 2.1+let y = 34%+let z = x * 2.0 + y+```++#### string++String literals are enclosed in double quotes (`"`),+using backslash to escape special characters (`"`, `\` and `$`)+and to represent some others (`\n` for newline).+```+let s = "some text"+```++Dollar sign (`$`) can be used to expand variables (numbers are expanded to decimal representation).+```+let a = "abc"+let b = 4+let c = "$a $b" # = "abc 4"+```++Arbitrary expression can be used within additional curly braces:+```+let a = 2+let b = 3+let s = "abc ${2*a + b}" # = "abc 7"+```++#### regex++Regular expression literals are enclosed in slash characters (`/`):+```+let re = /a.*/ # match any string starting with 'a'+```++Dollar-expansion can be used here as well.+Strings expand to regular expressions matching the exact string,+regular expressions expand are used directly.+```+let str = "."+let re1 = /./+let re2 = "$str$re1" # match '.' followed by any character+```++#### boolean++Result of comparison operators `==` and `/=`.++#### network++Represents network/subnet, created by `subnet` command and used by `subnet`, `node`, `spawn` and network configuration commands.++#### node++Represents network node, created by `node` command and used by `spawn` or network configuration commands.++Members:++`ip`: string representation of node's IP address.++#### process++Represents running process. Created by `spawn`, used by `send` and `expect` commands.++Members:++`node`: node on which the process is running++#### list++Lists are written using bracket notation:+```+let numbers = [1, 2, 4]+```++List elements can be of any type, but all elements of a particular list must have the same type.++Used in the `for` command.++### Build-in commands++```+subnet <name> [of <network>]+```++Create a subnet within a `<network>` (or context network if omitted) and assign the new network to the variable `<name>`.++```+node <name> [on <network>]+```++Create a node on network `<network>` (or context network if omitted) and assign the new node to the variable `<name>`.++```+spawn as <name> [on (<node> | <network>)]+```++Spawn a new test process on `<node>` or `<network>` (or one from context) and assign the new process to variable `<name>`.+When spawning on network, create a new node for this process.++The process is terminated when the variable `<name>` goes out of scope (at the end of the block in which it was created) by closing its stdin.+When the process fails to terminate successfully within a timeout, the test fails.++```+send <string> to <process>+```+Send line with `<string>` to the standard input of `<process>`.++```+expect <regex> from <process> [capture <var1> [, <var2> ... ]]+```+Check whether `<process>` produces line matching `<regex>` on standard output, and if this does not happen within current timeout, the test fails.+Output lines produced before starting this command and not matched by some previous `expect` are accepted as well.+Output lines not matching `<regex>` are ignored by this `expect` call, i.e. do not cause the `expect` call to fail.++Regular expressions are anchored on both sides, so must match the entire line.+If e.g. only the beginning should be matched, the passed regular expression needs to end with `.*`.++The regular expression can contain capture groups – parts enclosed in parentheses (`(`, `)`).+In that case the expect command has to have the `capture` clause with matching number of variable names.+Results of the captures are then assigned to the newly created variables as strings.++```+flush [from <proc>]+```++Flush memory of `<proc>` output, so no following `expect` command will match anything produced up to this point.++```+guard <expr>+```++Check whether boolean expression `<expr>` is true; if not, the test fails.++```+disconnect_node [<node>]+```++Disconnect `<node>` from network – state of the veth network link from the node is set to down.+The effect lasts until the end of the block.++```+disconnect_nodes [<network>]+```++Disconnect all nodes of `<network>`. The network bridge interface state is set to down.+The effect lasts until the end of the block.++```+disconnect_upstream [<network>]+```++Disconnect network upstream – state of the veth network link connecting network bridge to the upstream network is set to down.+The effect lasts until the end of the block.++```+packet_loss <rate> [on <node>]+```++Set the packet loss rate on the node's veth link to `<rate>` as decimal number or percentage, e.g. `0.2` or `20%` for 20% packet loss rate.+The effect lasts until the end of the block.++```+for <var> in <expr>:+ <test block>+```++Execute `<test block>` for each element of list `<expr>`, with current element assigned to `<var>`.++```+local:+ <test block>+```++Execute `<test block>` in a new local scope. Used to restrict scope of variables or duration of effects.++```+with <expr>:+ <test block>+```++Execute `<test block>` with `<expr>` as context.++```+wait+```++Wait for user input before continuing. Useful mostly for debugging or test development.+++Optional dependencies+---------------------++The test framework can use some other tools to help with debugging or development.++### GDB++If GDB is installed, it's possible to use `--gdb` command-line switch of the `erebos-tester` tool to use the debugger.+The GDB session is started in background and tester uses the GDB machine interface to communicate with it.+Whenever a new process is spawned, it is attached to the debugger as a new inferior.+In case any process is terminated by a signal, e.g. crashes with segfault, interactive GDB session is opened.++### tcpdump++If `tcpdump` binary is found in the `PATH`,+it is used to generate network log in the pcap capture format within the test directory,+separately for each virtual network (specifically its bridge interface).
erebos-tester.cabal view
@@ -1,7 +1,7 @@-cabal-version: 2.2+cabal-version: 3.0 name: erebos-tester-version: 0.2.1+version: 0.2.2 synopsis: Test framework with virtual network using Linux namespaces description: This framework is intended mainly for networking libraries/applications and@@ -18,6 +18,7 @@ -- copyright: category: Testing extra-doc-files:+ README.md CHANGELOG.md flag ci@@ -40,18 +41,28 @@ -- sometimes needed for backward/forward compatibility: -Wno-error=unused-imports + build-depends:+ base ^>= { 4.15, 4.16, 4.17, 4.18, 4.19, 4.20 },+ executable erebos-tester import: common+ ghc-options:+ -- disable interval timer to avoid spawing thread that breaks unshare(CLONE_NEWUSER)+ -with-rtsopts=-V0+ if impl(ghc >= 9.8)+ ghc-options:+ -- no multithreading is allowed for unshare(CLONE_NEWUSER)+ -single-threaded main-is: Wrapper.hs -- other-modules: -- other-extensions:- build-depends: base >=4.13 && <5,- directory >=1.3 && <1.4,- filepath ^>=1.4.2.1,- linux-namespaces ^>=0.1.3,- process ^>=1.6.9,- unix >=2.7 && <2.9,+ build-depends:+ directory >=1.3 && <1.4,+ filepath ^>= { 1.4.2.1, 1.5.2 },+ linux-namespaces^>=0.1.3,+ process ^>=1.6.9,+ unix >=2.7 && <2.9, hs-source-dirs: src default-language: Haskell2010 @@ -99,26 +110,24 @@ TypeApplications TypeFamilies TypeOperators- build-depends: base >=4.13 && <5,- bytestring >=0.10 && <0.12,- containers ^>=0.6.2.1,- directory ^>=1.3.6.0,- filepath ^>=1.4.2.1,- generic-deriving >=1.14 && <1.15,- Glob >=0.10 && <0.11,- haskeline >=0.8 && <0.9,- HsYAML >=0.2 && <0.3,- lens >=5.0 && <5.3,- megaparsec >=9.0 && <10,- mtl >=2.2 && <2.4,- parser-combinators >=1.3 && <1.4,- process ^>=1.6.9,- regex-tdfa ^>=1.3.1.0,- scientific >=0.3 && < 0.4,- stm ^>=2.5.0.1,- template-haskell >=2.17 && <2.22,- text >=1.2 && <2.1,- th-compat >=0.1 && <0.2,- unix >=2.7 && <2.9,+ build-depends:+ bytestring ^>= { 0.10, 0.11, 0.12 },+ containers ^>= { 0.6.2.1, 0.7 },+ directory ^>=1.3.6.0,+ filepath ^>= { 1.4.2.1, 1.5.2 },+ Glob >=0.10 && <0.11,+ haskeline >=0.8 && <0.9,+ HsYAML >=0.2 && <0.3,+ megaparsec >=9.0 && <10,+ mtl >=2.2 && <2.4,+ parser-combinators >=1.3 && <1.4,+ process ^>=1.6.9,+ regex-tdfa ^>=1.3.1.0,+ scientific >=0.3 && < 0.4,+ stm ^>=2.5.0.1,+ template-haskell^>= { 2.17, 2.18, 2.19, 2.20, 2.21, 2.22 },+ text ^>= { 1.2, 2.0, 2.1 },+ th-compat >=0.1 && <0.2,+ unix >=2.7 && <2.9, hs-source-dirs: src default-language: Haskell2010
src/Main.hs view
@@ -47,7 +47,7 @@ (ReqArg (\str -> to $ \opts -> case break (==':') str of (path, []) -> opts { optDefaultTool = path } (pname, (_:path)) -> opts { optProcTools = (ProcName (T.pack pname), path) : optProcTools opts }- ) "PATH")+ ) "<path>") "test tool to be used" , Option ['v'] ["verbose"] (NoArg (\opts -> opts { optVerbose = True }))@@ -55,7 +55,7 @@ , Option ['t'] ["timeout"] (ReqArg (\str -> to $ \opts -> case readMaybe str of Just timeout -> opts { optTimeout = timeout }- Nothing -> error "timeout must be a number") "SECONDS")+ Nothing -> error "timeout must be a number") "<seconds>") "default timeout in seconds with microsecond precision" , Option ['g'] ["gdb"] (NoArg $ to $ \opts -> opts { optGDB = True })@@ -67,7 +67,7 @@ (NoArg $ to $ \opts -> opts { optKeep = True }) "keep test directory even if all tests succeed" , Option ['r'] ["repeat"]- (ReqArg (\str opts -> opts { optRepeat = read str }) "COUNT")+ (ReqArg (\str opts -> opts { optRepeat = read str }) "<count>") "number of times to repeat the test(s)" , Option [] ["wait"] (NoArg $ to $ \opts -> opts { optWait = True })@@ -106,7 +106,13 @@ exitFailure when (optShowHelp opts) $ do- let header = "Usage: erebos-tester [OPTION...]"+ let header = unlines+ [ "Usage: erebos-tester [<option>...] [<script>[:<test>]...]"+ , " <script> path to test script file"+ , " <test> name of the test to run"+ , ""+ ]+ <> "Options are:" putStrLn $ usageInfo header options exitSuccess