setdown-0.2.1.0: README.markdown
# Setdown
Line-based set manipulation from the command line.
[](https://github.com/robertmassaioli/setdown/actions/workflows/test.yml)
[][7]
Author: [Robert Massaioli][6] · Created in 2015
## What is setdown?
Setdown treats text files as sets — one element per line — and lets you combine them with
intersection, union, difference, and symmetric difference. You describe the operations once in a
**`.setdown`** file (think of it like a `Makefile` for sets), and setdown resolves the whole
dependency graph, computes every definition, and writes one result file per definition.
```setdown
InternalStaff: "admins.txt" \/ "developers.txt"
AllUsers: InternalStaff \/ "contractors.txt"
ContractorsOnly: "contractors.txt" - InternalStaff
```
Run `setdown` in the directory containing that file and you get an `output/` directory with
`InternalStaff.txt`, `AllUsers.txt`, and `ContractorsOnly.txt` — each sorted and de-duplicated.
Input files don't need to be sorted, de-duplicated, or even sets to begin with; setdown normalizes
them as it goes. And setdown is current-working-directory invariant: all paths inside a
**`.setdown`** file are resolved relative to that file, not to wherever you happen to run the
command from, so you can invoke it from anywhere in your project tree and get the same result.
## Installation
### Via nix-shell (quickest, no local setup required)
```shell
$ nix-shell -p haskellPackages.setdown
$ setdown --help
```
### Via Hackage
```shell
stack install setdown
```
This works because [setdown is on Hackage][7]. To build from source instead — for example, to get
the latest unreleased changes — see [Building the code](#building-the-code) below.
## Quick start
Every example below lives under [`examples/`](examples) in this repository — clone the repo and
run any of them directly.
```shell
git clone https://github.com/robertmassaioli/setdown.git
cd setdown/examples/access-control
stack exec -- setdown
```
| Example | What it shows |
|---|---|
| [`standard`](examples/standard) | A tour of intersection, union, and difference, including bracketed precedence |
| [`access-control`](examples/access-control) | Deriving permission groups (internal staff, contractors) from role files |
| [`basic-difference`](examples/basic-difference) | Diffing two API surfaces to find what was added and removed |
| [`data-reconciliation`](examples/data-reconciliation) | Comparing two months of customer lists: retained, new, and lost |
| [`feature-flags`](examples/feature-flags) | Segmenting users by experiment exposure, including a three-way overlap |
| [`software-dependencies`](examples/software-dependencies) | Auditing shared and unique dependencies across two applications |
| [`symmetric-difference`](examples/symmetric-difference) | The `><` operator versus the equivalent longhand expression |
| [`cycle-detection`](examples/cycle-detection) | A deliberately cyclic definition, and the error setdown reports |
| [`parse-error`](examples/parse-error) | A deliberately malformed expression, and the error setdown reports |
## Set operations and precedence
| Operator | ASCII | Unicode |
|----------|-------|---------|
| Intersection | `/\` | `∩` |
| Union | `\/` | `∪` |
| Difference | `-` | |
| Symmetric difference | `><` | `△` |
Intersection, union, and symmetric difference are commutative (`A op B` is the same as
`B op A`). Difference is not (`A - B` ≠ `B - A`).
Symmetric difference (`><` or `△`) yields the elements that appear in exactly one of the two
inputs — those in A but not B, plus those in B but not A. It is equivalent to `(A - B) \/ (B - A)`
but computed in a single pass; see [`examples/symmetric-difference`](examples/symmetric-difference)
for both forms side by side.
```setdown
definition: (A - B) \/ (C /\ D)
changedSubscribers: "january.txt" >< "february.txt"
```
**There is no operator precedence** — you must bracket nested expressions explicitly. Consider:
```setdown
def: A /\ B \/ C
```
Should this parse as `(A /\ B) \/ C` or `A /\ (B \/ C)`? Substitute the empty set for `B` and the
two readings diverge completely (`E` versus `A /\ C`). Because the difference is not cosmetic,
setdown refuses to guess — an unbracketed expression like this is a parse error.
## Language reference
### Identifiers
Definition names may contain letters, digits, hyphens, and underscores: `[a-zA-Z0-9_-]+`. For
example, `mySet`, `result-2`, and `Final_Output` are all valid; spaces and other punctuation are
not permitted.
### Definition ordering
Definitions may appear in any order in a **`.setdown`** file, and a definition may reference
another that's defined later on. Setdown resolves all identifiers by name after parsing the whole
file.
### Circular definitions
Definitions must not form a cycle:
```setdown
A: "file.txt" \/ B
B: A /\ "other.txt"
```
Setdown detects cycles like this and exits with an error before performing any operations — see
[`examples/cycle-detection`](examples/cycle-detection).
### Comments
Add comments with a double-dash (`--`) through the end of the line, anywhere on the line:
```setdown
-- This is a definition for A, created because we wanted to do X
A: "y.txt" - "z.txt"
B: (A \/ C) -- \/ D This is still a comment and \/ D never happens
```
### Full example
```setdown
-- A is the intersection of the file b-1.out and the set B
A: "b-1.out" /\ B
-- B is the union of the file a-1.out and a-2.out
B: "a-1.out" \/ "a-2.out"
-- C is the difference of the file b-1.out and the set B
C: "b-1.out" - B
-- D is the symmetric difference of two files (elements in one but not both)
D: "a-1.out" >< "a-2.out"
```
Files with these definitions are usually suffixed **`.setdown`** and fed to the executable:
```shell
setdown path/to/mydefinitions.setdown
```
## Output
When setdown runs, it creates an `output/` directory next to your **`.setdown`** file. Each named
definition produces a result file in that directory named after the definition with a `.txt`
extension — for example, a definition called `Overlap` produces `output/Overlap.txt`. The file
contains one element per line, sorted and de-duplicated.
Intermediate results for sub-expressions are computed in a scratch `output/processing/` directory,
which is removed automatically once the run finishes. Pass `--keep-processing` to leave it in
place for debugging, and `--show-transient` to have those intermediate results included in the
summary table setdown prints at the end of a run.
You can choose a different output directory with `--output`, given relative to the **`.setdown`**
file, not the current working directory:
```shell
setdown --output=results mydefinitions.setdown
```
## Command-line flags
```text
setdown evaluates a .setdown definitions file to perform set operations
(intersection, union, difference) on line-based text files, writing one
result file per definition to an output directory.
setdown [OPTIONS]
Common flags:
-o --output[=DIR] Directory in which to place output files,
relative to your .setdown file. Defaults to
'output' if omitted.
-i --input=definitions.setdown The .setdown definitions file to evaluate.
If omitted, setdown looks for a single
.setdown file in the current directory and
uses it automatically. Exits with an error if
zero or more than one are found.
--show-transient Also show intermediate results for
sub-expressions generated internally to
evaluate your definitions. Useful for
debugging complex .setdown files.
--keep-processing Keep the processing/ subdirectory after the
run completes instead of deleting it. Useful
for inspecting intermediate files when
debugging.
-? --help Display help message
-V --version Print version information
```
## Troubleshooting
Setdown prints a short error message to stdout and exits with a non-zero code when something goes
wrong:
| Exit code | Cause |
|-----------|-------|
| 1 | The file specified with `--input` does not exist. |
| 2 | Multiple **`.setdown`** files found in the current directory; use `--input` to select one. |
| 3 | No **`.setdown`** files found in the current directory; use `--input` to specify one. |
| 11 | Two or more definitions share the same name. |
| 12 | A definition references an identifier that has not been defined. |
| 13 | One or more input files referenced in the definitions could not be found. |
| 20 | A cyclic dependency was detected between definitions. |
All file paths in error messages are relative to the **`.setdown`** file, not the current working
directory. See [`examples/parse-error`](examples/parse-error) and
[`examples/cycle-detection`](examples/cycle-detection) for these in action.
## Building the code
Setdown is a Haskell library (the set language, parser, and evaluation engine) plus a thin
`setdown` executable, built with [Stack][10]:
```shell
stack build
```
Run the three test suites — unit, property-based, and golden — with:
```shell
stack test
```
The same build-and-test steps run in CI on every push and pull request; see
[`.github/workflows`](.github/workflows).
To run setdown during development, without installing it:
```shell
stack exec -- setdown --help
stack exec -- setdown mydefinitions.setdown
```
## Contributing
Contributions are welcome. The preferred workflow is:
1. Open an issue describing what you intend to fix or improve.
2. Write the code.
3. Open a pull request and ask Robert Massaioli to review it.
4. Iterate until the code is clean and merged.
5. Celebrate!
Design proposals and background for larger changes live under [`ai-planning/`](ai-planning) if
you'd like context before picking something up.
[6]: https://github.com/robertmassaioli
[7]: http://hackage.haskell.org/package/setdown
[10]: https://docs.haskellstack.org/en/stable/