# goldplate 🏅
$ goldplate -j2 tests/
Found 32 specs
Running 49 executions in 2 jobs
Finished in 0.84s
Ran 32 specs, 49 executions, 146 asserts, all A-OK!
`goldplate` is a cute and simple opaque [golden test] runner for CLI
applications. You place your test cases in a directory, annotate them with
`.goldplate` files, and that's it. It is completely language agnostic. And
perhaps its best feature is that it can automatically✨🪄 fix your tests outputs!
Give `goldplate` a try if:
- You are testing a CLI application
- The application produces deterministic output (e.g. compilers, prettifiers,
convertors)
- The application is quick to start (a process is created for every test)
- Your output is text-based and not huge in size
At [Fugue](https://fugue.co), we've been using internal versions of this tool
since 2016, so it should be pretty stable.
## Table of Contents
- [Tutorial](#tutorial)
- [Creating a first test](#creating-a-first-test)
- [Feeding input on stdin](#feeding-input-on-stdin)
- [Setting environment
variables](#setting-environment-variables)
- [Globbing input files](#globbing-input-files)
- [Post processing: find and
replace](#post-processing-find-and-replace)
- [Post processing: prettify
JSON](#post-processing-prettify-json)
- [Created files and
directories](#created-files-and-directories)
- [Installation](#installation)
- [Using stack](#using-stack)
- [Using Cabal](#using-cabal)
- [Reference](#reference)
- [Syntax](#syntax)
- [Environment variables](#environment-variables)
- [Similar projects](#similar-projects)
## Tutorial
### Creating a first test
Imagine we are up to testing the behaviour of `echo` command. In this very
simple example, we run `echo "Hello, world!"` and expect it to print `Hello,
world!` to the stdout stream as a result.
Create a new file `echo.goldplate` and add the following content:
```json
{
"command": "echo",
"arguments": ["Hello, world!"],
"asserts": [
{"exit_code": 0},
{"stdout": "hello-world.txt"}
]
}
```
Let's go through this bit by bit.
The test invocation is specified by the `command` and `arguments` fields. We
are invoking the `echo` process with a single argument, `"Hello, world!"`.
The expected results of our test live in the `asserts` field. This simple test
has two asserts:
1. We verify that the exit code is 0 (success).
2. We check the `stdout` (output) of the command against the contents of
the file `hello-world.txt`.
We haven't created `hello-world.txt` yet, but that's not a problem. We can
invoke `goldplate --fix` to create it:
$ goldplate echo.goldplate --pretty-diff --fix
...
echo.goldplate: stdout: does not match
echo.goldplate: fixed ./hello-world.txt
...
Ran 1 specs, 1 executions, 2 asserts, 1 failed.
After `hello-world.txt` has been created with proper contents, subsequent
`goldplate` invocations will pass:
$ goldplate echo.goldplate
...
Ran 1 specs, 1 executions, 2 asserts, all A-OK!
You can view the full example here:
- [`tests/echo.goldplate`](tests/echo.goldplate)
- [`tests/hello-world.txt`](tests/hello-world.txt)
### Feeding input on stdin
View example:
- [`tests/cat.goldplate`](tests/cat.goldplate)
- [`tests/cat.stdout`](tests/cat.stdout)
You can pass one or multiple lines of input to the command by using the `stdin`
field.
### Setting environment variables
View example:
- [`tests/env.goldplate`](tests/env.goldplate)
- [`tests/env.stdout`](tests/env.stdout)
The `environment` field can be used to set environment variables for the
program.
`goldplate` also sets [a number of environment
variables](#environment-variables). You can use these directly within the
configuration JSON. In this example, we use:
{"stdout": "${GOLDPLATE_NAME}.stdout"}
Rather than:
{"stdout": "env.stdout"}
We found this to be good practice, it makes mass-renaming of tests much easier.
### Globbing input files
View example:
- [`tests/glob.goldplate`](tests/glob.goldplate)
- [`tests/glob-1.txt`](tests/glob-1.txt)
- [`tests/glob-1.stdout`](tests/glob-1.stdout)
- [`tests/glob-2.txt`](tests/glob-2.txt)
- [`tests/glob-2.stdout`](tests/glob-2.stdout)
`.goldplate` files are fairly small but if you have a whole directory of files
that you just want to run the same command on, they can get very repetitive.
This is why `goldplate` provides a simple way to pull in multiple input files.
If the `input_files` field is set to a glob, all asserts will be ran for _every_
matching input file. `goldplate` will set the following variables:
- `${GOLDPLATE_INPUT_FILE}`: the path to the input file
- `${GOLDPLATE_INPUT_NAME}`: the input file without extension
### Post processing: find and replace
View example:
- [`tests/replace.goldplate`](tests/replace.goldplate)
- [`tests/replace.stdout`](tests/replace.stdout)
Sometimes you may want to do a find-and-replace on the actual output, for
example to filter out timestamps or other information that you do not expect to
match up against the expected output.
### Post processing: prettify JSON
View example:
- [`tests/prettify-json.goldplate`](tests/prettify-json.goldplate)
- [`tests/prettify-json.json`](tests/prettify-json.json)
Many modern CLI tools output JSON. You can use the `prettify_json` post
processor to make sure the JSON is converted to a normalized form with sorted
keys and consistent indentation.
### Created files and directories
View example:
- [`tests/create.goldplate`](tests/create.goldplate)
- [`tests/create.file`](tests/create.file)
`goldplate` is geared towards checking the `stdout` and `stderr` outputs of a
program, but you can also check that files were created with specific contents.
If you do this, `goldplate` will remove these files and directories afterwards
to leave a clean repository behind.
## Installation
Installation through source is done using standard Haskell tooling -- [Cabal]
and [stack] both work well.
### Using stack
1. Install [stack] for your platform.
2. Clone this repository and `cd` into it.
3. Run `stack install`.
4. Make sure `$HOME/.local/bin` is in your `$PATH`.
### Using Cabal
1. Install [Cabal] for your platform.
2. Clone this repository and `cd` into it.
3. Run `cabal install`.
4. Make sure `$HOME/.cabal/bin` is in your `$PATH`.
## Reference
### Syntax
Environment variables can be spliced into the configuration using `${VAR}`
syntax within strings. To escape this syntax, use `$${VAR}` to get a literal
`${VAR}`, `$$${VAR}` to get a literal `$${VAR}`, and so on.
### Environment variables
The test is always executed in the directory that holds the `.goldplate` file.
`goldplate` will always set the following environment variables:
- `GOLDPLATE_FILE`: The filename of the `.goldplate` file.
- `GOLDPLATE_NAME`: The filename of the `.goldplate` file _without_ the
extension.
- `GOLDPLATE_BASENAME`: The basename (filename without directory) of
the `.goldplate` file.
When dealing with [multiple input files](#globbing-input-files), the following
additional variables are set:
- `GOLDPLATE_INPUT_FILE`: The input file name (relative to the current
directory).
- `GOLDPLATE_INPUT_NAME`: The same as `GOLDPLATE_INPUT_FILE` but without
the extension.
- `GOLDPLATE_INPUT_BASENAME`: The basename (filename without directory) of
the input file.
Here is an example that outputs all of these environment variables:
- [`tests/builtin.goldplate`](tests/builtin.goldplate)
- [`tests/builtin.stdout`](tests/builtin.stdout)
## Similar projects
A similar project is [smoke]. I think `goldplate` has two major advantages
over smoke:
- It can fix the output files automatically using `--fix`! This is very
useful if you make a change to your tool that affects _a lot_ of test
files. You still need to manually review the diff, but at least you don't
need to manually update the specs.
- You can avoid most repetitive configs by using
[simple globbing](#globbing-input-files).
[Cabal]: https://www.haskell.org/cabal/
[golden test]: https://ro-che.info/articles/2017-12-04-golden-tests
[stack]: https://docs.haskellstack.org/en/stable/README/
[smoke]: https://github.com/SamirTalwar/smoke