diff --git a/README.markdown b/README.markdown --- a/README.markdown +++ b/README.markdown @@ -1,86 +1,75 @@ -# Setdown - Line based set manipulation +# Setdown -**Version:** 0.1.3.0 | [Hackage][7] +Line-based set manipulation from the command line. -Author: [Robert Massaioli][6] -Created in: 2015 +[![Test](https://github.com/robertmassaioli/setdown/actions/workflows/test.yml/badge.svg)](https://github.com/robertmassaioli/setdown/actions/workflows/test.yml) +[![Hackage](https://img.shields.io/hackage/v/setdown.svg)][7] +Author: [Robert Massaioli][6] · Created in 2015 + +## What is setdown? + +Setdown treats text files as sets — one element per line — and lets you combine them with +intersection, union, difference, and symmetric difference. You describe the operations once in a +**`.setdown`** file (think of it like a `Makefile` for sets), and setdown resolves the whole +dependency graph, computes every definition, and writes one result file per definition. + +```setdown +InternalStaff: "admins.txt" \/ "developers.txt" +AllUsers: InternalStaff \/ "contractors.txt" +ContractorsOnly: "contractors.txt" - InternalStaff +``` + +Run `setdown` in the directory containing that file and you get an `output/` directory with +`InternalStaff.txt`, `AllUsers.txt`, and `ContractorsOnly.txt` — each sorted and de-duplicated. + +Input files don't need to be sorted, de-duplicated, or even sets to begin with; setdown normalizes +them as it goes. And setdown is current-working-directory invariant: all paths inside a +**`.setdown`** file are resolved relative to that file, not to wherever you happen to run the +command from, so you can invoke it from anywhere in your project tree and get the same result. + ## Installation ### Via nix-shell (quickest, no local setup required) -``` shell +```shell $ nix-shell -p haskellPackages.setdown $ setdown --help ``` ### Via Hackage -``` shell +```shell stack install setdown ``` -This works because [setdown is on Hackage][7]. - -## What is setdown and how does it work? - -Setdown is a command line tool for line based set operations. To use setdown you write a "setdown -definitions file" often suffixed with **.setdown**. If you are familiar with [Make][3] then you can think -of this **.setdown** file much like a Makefile. Inside that file you write a number of -definitions of the form: - -``` setdown -definitionName: "file-1.txt" /\ "file-2.txt" -``` - -This line says that "definitionName" is a new set definition that is a label for the intersection of -"file-1.txt" and "file-2.txt". You can write more complicated expressions than this. - -### Example Setdown Projects - -[Checkout the setdown-examples project][2] on Bitbucket; it will show you how setdown works. - -However, to get an in-depth description of setdown and its abilities you should -read the sections below. - -### Input Files - -In setdown *each file is treated as a list of elements where each line -is an element*. Input files do not need to begin as sets; they can contain duplicate and unsorted -elements. Setdown will automatically sort and de-duplicate all input files, turning them into sets. - -Another important point is that of relativity: specifically, if you have a **.setdown** file that -references the input file "some-elements.txt" and you run the setdown executable from a directory that -is not the same directory as the **.setdown** file, where will setdown look for -some-elements.txt? The answer is that setdown always looks for files relative to the **.setdown** -file. That is where you wrote your definitions so the paths are relative to that. It was designed in -this way so that you could run setdown from anywhere in the directory tree and still get the same -result. Setdown has been designed to be current working directory invariant, as opposed to many -other command line programs. Please keep this in mind. - -### Output - -When setdown runs, it creates an `output/` directory next to your **.setdown** file. Each named -definition produces a result file in that directory named after the definition with a `.txt` -extension — for example, a definition called `Overlap` produces `output/Overlap.txt`. The file -contains one element per line, sorted and de-duplicated. +This works because [setdown is on Hackage][7]. To build from source instead — for example, to get +the latest unreleased changes — see [Building the code](#building-the-code) below. -Progress and status messages are written to stdout as setdown works through your definitions. At -the end of a successful run, a summary table is printed showing each definition name, the path to -its result file, and the number of elements it contains. +## Quick start -You can choose a different output directory with the `--output` flag: +Every example below lives under [`examples/`](examples) in this repository — clone the repo and +run any of them directly. -``` shell -setdown --output=results mydefinitions.setdown +```shell +git clone https://github.com/robertmassaioli/setdown.git +cd setdown/examples/access-control +stack exec -- setdown ``` -The path given to `--output` is relative to the **.setdown** file, not the current working -directory. - -### Set Operations and Precedence +| Example | What it shows | +|---|---| +| [`standard`](examples/standard) | A tour of intersection, union, and difference, including bracketed precedence | +| [`access-control`](examples/access-control) | Deriving permission groups (internal staff, contractors) from role files | +| [`basic-difference`](examples/basic-difference) | Diffing two API surfaces to find what was added and removed | +| [`data-reconciliation`](examples/data-reconciliation) | Comparing two months of customer lists: retained, new, and lost | +| [`feature-flags`](examples/feature-flags) | Segmenting users by experiment exposure, including a three-way overlap | +| [`software-dependencies`](examples/software-dependencies) | Auditing shared and unique dependencies across two applications | +| [`symmetric-difference`](examples/symmetric-difference) | The `><` operator versus the equivalent longhand expression | +| [`cycle-detection`](examples/cycle-detection) | A deliberately cyclic definition, and the error setdown reports | +| [`parse-error`](examples/parse-error) | A deliberately malformed expression, and the error setdown reports | -In the setdown language there are a number of supported operators: +## Set operations and precedence | Operator | ASCII | Unicode | |----------|-------|---------| @@ -92,100 +81,66 @@ Intersection, union, and symmetric difference are commutative (`A op B` is the same as `B op A`). Difference is not (`A - B` ≠ `B - A`). -Symmetric difference (`><` or `△`) yields the elements that appear in exactly one of -the two inputs — those in A but not B, plus those in B but not A. It is equivalent to -`(A - B) \/ (B - A)` but computed in a single pass. - -For example, they might be used in the following way: +Symmetric difference (`><` or `△`) yields the elements that appear in exactly one of the two +inputs — those in A but not B, plus those in B but not A. It is equivalent to `(A - B) \/ (B - A)` +but computed in a single pass; see [`examples/symmetric-difference`](examples/symmetric-difference) +for both forms side by side. -``` setdown +```setdown definition: (A - B) \/ (C /\ D) changedSubscribers: "january.txt" >< "february.txt" ``` -You may be wondering what [operator precedence][1] the setdown language uses and the answer is: -there is no operator precedence at all, instead *you must clearly specify the precedence of nested -expressions with brackets*. This is very important because it will result in parsing errors -otherwise. To explain the reasoning for explicit operator precedence: +**There is no operator precedence** — you must bracket nested expressions explicitly. Consider: -``` setdown --- Here is a simple expression +```setdown def: A /\ B \/ C --- Now, should this be parsed as: -defV1: (A /\ B) \/ C --- or as: -defV2: A /\ (B \/ C) --- If you pretend that B is the empty set (E) then you can see that these expressions evaluate --- completely differently. If we simplify them with that assumption then they become: -defV1-bempty: E -defV2-bempty: A /\ C ``` -So as you can see, order of operations really matters for set operations. Because it is so critical -the use of brackets is mandatory. - -### Comments - -In the setdown language you can add comments by writing a double-dash (`--`) and then writing the -comment to the end of the line. Comments can appear anywhere on a line — at the start, or inline -after an expression. - -``` setdown --- This is a definition for A, created because we wanted to do X -A: "y.txt" - "z.txt" - --- This is an example of a comment halfway through an expression -B: (A \/ C) -- \/ D This is still a comment and \/ D never happens -``` - -You can use comments to leave messages for any people that might read your setdown definitions in -the future. - -### Language Reference - -#### Identifier rules +Should this parse as `(A /\ B) \/ C` or `A /\ (B \/ C)`? Substitute the empty set for `B` and the +two readings diverge completely (`E` versus `A /\ C`). Because the difference is not cosmetic, +setdown refuses to guess — an unbracketed expression like this is a parse error. -Definition names (identifiers) may contain letters (upper and lowercase), digits, hyphens, and -underscores: +## Language reference -``` -[a-zA-Z0-9_-]+ -``` +### Identifiers -For example, `mySet`, `result-2`, and `Final_Output` are all valid identifiers. Spaces and -punctuation other than `-` and `_` are not permitted. +Definition names may contain letters, digits, hyphens, and underscores: `[a-zA-Z0-9_-]+`. For +example, `mySet`, `result-2`, and `Final_Output` are all valid; spaces and other punctuation are +not permitted. -#### Definition ordering +### Definition ordering -Definitions may appear in any order in your **.setdown** file. A definition may reference another -that is defined later in the file. Setdown resolves all identifiers by name after parsing the -complete file. +Definitions may appear in any order in a **`.setdown`** file, and a definition may reference +another that's defined later on. Setdown resolves all identifiers by name after parsing the whole +file. -#### Circular definitions +### Circular definitions -Definitions must not form a cycle. For example: +Definitions must not form a cycle: -``` setdown +```setdown A: "file.txt" \/ B B: A /\ "other.txt" ``` -This is invalid because `A` depends on `B` and `B` depends on `A`. Setdown detects cycles and -exits with an error before performing any operations. +Setdown detects cycles like this and exits with an error before performing any operations — see +[`examples/cycle-detection`](examples/cycle-detection). -### Writing your own definitions +### Comments -In the setdown language you can write a definition in the following format: +Add comments with a double-dash (`--`) through the end of the line, anywhere on the line: -``` setdown -: +```setdown +-- This is a definition for A, created because we wanted to do X +A: "y.txt" - "z.txt" + +B: (A \/ C) -- \/ D This is still a comment and \/ D never happens ``` -Where the definition name is the identifier that you give to that expression. An expression is the -application of set operations on identifiers or files. A practical example of what this looks like -should help cement what this means. Here is a valid setdown file: +### Full example -``` setdown +```setdown -- A is the intersection of the file b-1.out and the set B A: "b-1.out" /\ B @@ -199,19 +154,37 @@ D: "a-1.out" >< "a-2.out" ``` -Usually, when you write these definitions you put them in a file that has a suffix of **.setdown**. -You can then feed this file into the setdown executable like so: +Files with these definitions are usually suffixed **`.setdown`** and fed to the executable: -``` shell +```shell setdown path/to/mydefinitions.setdown ``` -### Command-line flags +## Output -``` text +When setdown runs, it creates an `output/` directory next to your **`.setdown`** file. Each named +definition produces a result file in that directory named after the definition with a `.txt` +extension — for example, a definition called `Overlap` produces `output/Overlap.txt`. The file +contains one element per line, sorted and de-duplicated. + +Intermediate results for sub-expressions are computed in a scratch `output/processing/` directory, +which is removed automatically once the run finishes. Pass `--keep-processing` to leave it in +place for debugging, and `--show-transient` to have those intermediate results included in the +summary table setdown prints at the end of a run. + +You can choose a different output directory with `--output`, given relative to the **`.setdown`** +file, not the current working directory: + +```shell +setdown --output=results mydefinitions.setdown +``` + +## Command-line flags + +```text setdown evaluates a .setdown definitions file to perform set operations -(intersection, union, difference, symmetric difference) on line-based text -files, writing one result file per definition to an output directory. +(intersection, union, difference) on line-based text files, writing one +result file per definition to an output directory. setdown [OPTIONS] @@ -228,45 +201,60 @@ sub-expressions generated internally to evaluate your definitions. Useful for debugging complex .setdown files. + --keep-processing Keep the processing/ subdirectory after the + run completes instead of deleting it. Useful + for inspecting intermediate files when + debugging. -? --help Display help message -V --version Print version information ``` -## Building the code - -To build the code for this project, have [Stack][10] installed and then: - -``` shell -stack build -``` - -To run setdown during development: - -``` shell -stack exec -- setdown --help -stack exec -- setdown mydefinitions.setdown -``` - ## Troubleshooting Setdown prints a short error message to stdout and exits with a non-zero code when something goes -wrong. The error codes are: +wrong: | Exit code | Cause | |-----------|-------| | 1 | The file specified with `--input` does not exist. | -| 2 | Multiple **.setdown** files found in the current directory; use `--input` to select one. | -| 3 | No **.setdown** files found in the current directory; use `--input` to specify one. | +| 2 | Multiple **`.setdown`** files found in the current directory; use `--input` to select one. | +| 3 | No **`.setdown`** files found in the current directory; use `--input` to specify one. | | 11 | Two or more definitions share the same name. | | 12 | A definition references an identifier that has not been defined. | | 13 | One or more input files referenced in the definitions could not be found. | | 20 | A cyclic dependency was detected between definitions. | -All file paths in error messages are relative to the **.setdown** file, not the current working -directory. +All file paths in error messages are relative to the **`.setdown`** file, not the current working +directory. See [`examples/parse-error`](examples/parse-error) and +[`examples/cycle-detection`](examples/cycle-detection) for these in action. -## Contributing to the setdown project +## Building the code +Setdown is a Haskell library (the set language, parser, and evaluation engine) plus a thin +`setdown` executable, built with [Stack][10]: + +```shell +stack build +``` + +Run the three test suites — unit, property-based, and golden — with: + +```shell +stack test +``` + +The same build-and-test steps run in CI on every push and pull request; see +[`.github/workflows`](.github/workflows). + +To run setdown during development, without installing it: + +```shell +stack exec -- setdown --help +stack exec -- setdown mydefinitions.setdown +``` + +## Contributing + Contributions are welcome. The preferred workflow is: 1. Open an issue describing what you intend to fix or improve. @@ -275,10 +263,9 @@ 4. Iterate until the code is clean and merged. 5. Celebrate! - [1]: http://en.wikipedia.org/wiki/Order_of_operations - [2]: https://bitbucket.org/robertmassaioli/setdown-examples - [3]: http://www.gnu.org/software/make/ +Design proposals and background for larger changes live under [`ai-planning/`](ai-planning) if +you'd like context before picking something up. + [6]: https://github.com/robertmassaioli [7]: http://hackage.haskell.org/package/setdown - [8]: https://nixos.org/manual/nix/unstable/command-ref/nix-shell.html [10]: https://docs.haskellstack.org/en/stable/ diff --git a/app/Main.hs b/app/Main.hs --- a/app/Main.hs +++ b/app/Main.hs @@ -7,7 +7,7 @@ import qualified Data.Text.Lazy as LT import qualified Data.Text.Lazy.IO as T -import qualified Text.Layout.Table as Tab +import GHC.IO.Encoding (setLocaleEncoding, utf8) import System.Console.CmdArgs import System.Exit @@ -31,6 +31,7 @@ import SetData import SetInput import SetInputVerification +import TableRender (Align (..), renderTable) import DuplicateElimination import PerformOperations @@ -135,6 +136,12 @@ -- for one main :: IO () main = do + -- setdown prints Unicode box-drawing characters in its results tables. Force + -- UTF-8 regardless of the ambient locale so this doesn't crash on systems + -- without one configured (e.g. minimal Debian/Ubuntu installs and most CI + -- images default to the C/POSIX locale). + setLocaleEncoding utf8 + opts <- cmdArgs options inputFilePath <- getInputFileOrFail (setdownFile opts) @@ -324,29 +331,14 @@ wrapInQuotes x = "\"" ++ x ++ "\"" printTabularResults :: [(FilePath, FilePath)] -> IO () -printTabularResults fileMapping = sequence_ . fmap putStrLn $ Tab.tableLines (Tab.columnHeaderTableS columns Tab.unicodeBoldHeaderS headers rows) - where - headers = Tab.titlesH ["From", "To"] - - columns = - [ Tab.column Tab.expand Tab.left Tab.noAlign Tab.noCutMark - , Tab.column Tab.expand Tab.left Tab.noAlign Tab.noCutMark - ] - - rows = [Tab.rowsG $ fmap (\(from, to) -> [from, to]) fileMapping] +printTabularResults fileMapping = mapM_ putStrLn $ + renderTable [AlignLeft, AlignLeft] ["From", "To"] + (fmap (\(from, to) -> [from, to]) fileMapping) printTabularResultsWithCount :: [(String, FilePath, Int)] -> IO () -printTabularResultsWithCount rows = sequence_ . fmap putStrLn $ Tab.tableLines (Tab.columnHeaderTableS columns Tab.unicodeBoldHeaderS headers tableRows) - where - headers = Tab.titlesH ["Name", "File", "Count"] - - columns = - [ Tab.column Tab.expand Tab.left Tab.noAlign Tab.noCutMark - , Tab.column Tab.expand Tab.left Tab.noAlign Tab.noCutMark - , Tab.column Tab.expand Tab.right Tab.noAlign Tab.noCutMark - ] - - tableRows = [Tab.rowsG $ fmap (\(defName, fp, n) -> [defName, fp, show n]) rows] +printTabularResultsWithCount rows = mapM_ putStrLn $ + renderTable [AlignLeft, AlignLeft, AlignRight] ["Name", "File", "Count"] + (fmap (\(defName, fp, n) -> [defName, fp, show n]) rows) printComputedResults :: Options -> [(SimpleDefinition, FilePath, Int)] -> IO () printComputedResults opts results = do diff --git a/man/setdown.1 b/man/setdown.1 new file mode 100644 --- /dev/null +++ b/man/setdown.1 @@ -0,0 +1,349 @@ +.\" Man page for setdown. Keep the .TH date/version in sync with setdown.cabal on release. +.TH SETDOWN 1 "2026-08-22" "setdown 0.2.0.0" "User Commands" +.SH NAME +setdown \- evaluate a \fB.setdown\fR file to perform set operations on line\-based text files +.SH SYNOPSIS +.B setdown +[\fB\-i\fR \fIFILE\fR] +[\fB\-o\fR \fIDIR\fR] +[\fB\-\-show\-transient\fR] +[\fB\-\-keep\-processing\fR] +.br +.B setdown +\fB\-\-help\fR +.br +.B setdown +\fB\-\-version\fR +.SH DESCRIPTION +.B setdown +treats text files as sets \(em one element per line \(em and lets you combine them with +intersection, union, difference, and symmetric difference. The operations are described once in a +\fB.setdown\fR definitions file, similar in spirit to a +.BR Makefile , +and +.B setdown +resolves the whole dependency graph between definitions, computes every one of them, and writes +one result file per definition into an output directory. +.PP +Input files do not need to be sorted, de\-duplicated, or already act like sets; \fBsetdown\fR +normalizes them as it goes. Every path written inside a \fB.setdown\fR file \(em both the input +files it reads and the \fB\-\-output\fR directory it writes to \(em is resolved relative to the +location of that \fB.setdown\fR file, never relative to the current working directory. This makes +a \fB.setdown\fR file's behaviour independent of where you happen to invoke +.B setdown +from. +.PP +If no \fB.setdown\fR file is given explicitly with \fB\-\-input\fR, +.B setdown +looks for exactly one \fB.setdown\fR file in the current directory and uses it automatically. +.SH OPTIONS +.TP +\fB\-o\fR, \fB\-\-output\fR[=\fIDIR\fR] +Directory in which to place output files, given relative to the \fB.setdown\fR file being +evaluated (not the current working directory). Defaults to +.I output +if this option is omitted entirely. +.TP +\fB\-i\fR, \fB\-\-input\fR=\fIFILE\fR +The \fB.setdown\fR definitions file to evaluate. If omitted, +.B setdown +looks for a single \fB.setdown\fR file in the current directory and uses it automatically. Exits +with an error if zero, or more than one, are found (see \fBEXIT STATUS\fR below). +.TP +\fB\-\-show\-transient\fR +Also print intermediate results for the sub\-expressions +.B setdown +generates internally while evaluating your definitions, in addition to the named definitions +themselves. Useful when a definition's expression is complex and you want to see each +intersection/union/difference/symmetric\-difference step that led to the final result. +.TP +\fB\-\-keep\-processing\fR +Keep the scratch +.I processing/ +subdirectory (see \fBFILES\fR below) after the run completes, instead of deleting it. Useful for +inspecting the intermediate files behind a definition when debugging unexpected output. +.TP +\fB\-?\fR, \fB\-\-help\fR +Display a short help message summarising these options, then exit. +.TP +\fB\-V\fR, \fB\-\-version\fR +Print version information, then exit. +.SH THE .setdown FILE FORMAT +A \fB.setdown\fR file is a plain text file containing one or more +.I definitions, +each of the form: +.PP +.RS +.I name\fR: \fIexpression\fR +.RE +.PP +.B setdown +evaluates every definition in the file and writes one result file per definition (see +\fBOUTPUT\fR). Definitions may appear in any order \(em a definition may reference another +definition that is written later in the same file, since +.B setdown +resolves all names after parsing the whole file. +.SS Identifiers +A definition's name may contain letters, digits, hyphens, and underscores +.RI ( [a\-zA\-Z0\-9_\-]+ ). +Spaces and other punctuation are not permitted. Examples of valid names: +.IR mySet ", " result\-2 ", " Final_Output . +.SS Filenames +A file is referenced by writing its path in double quotes, e.g.\& +.IR \(dqusers.txt\(dq . +The path may contain spaces and is resolved relative to the location of the \fB.setdown\fR file +itself, not the current working directory and not any \fB\-\-output\fR directory. +.SS Operators +An expression combines files and other definitions using one of four binary set operators. Each +has an ASCII spelling, which always works, and an equivalent single\-character Unicode symbol, +which may or may not render depending on your terminal and locale: +.TP +.B Intersection +.B /\e +(Unicode: the intersection symbol, U+2229) +(elements present in both operands) +.TP +.B Union +.B \e/ +(Unicode: the union symbol, U+222A) +(elements present in either operand) +.TP +.B Difference +.B \- +(elements in the left operand that are not in the right operand; not commutative \(em +.I A +\- +.I B +is not the same as +.I B +\- +.IR A ) +.TP +.B Symmetric difference +.B >< +(Unicode: the white up\-pointing small triangle, U+25B3) +(elements present in exactly one of the two operands; equivalent to +.RI ( A " \- " B ")" " \e/ " "(" B " \- " A ), +but computed in a single pass) +.PP +Intersection, union, and symmetric difference are commutative +.RI ( A " op " B +is the same as +.IR B " op " A ). +Difference is the only operator that is not. +.SS Bracketing and precedence +.B setdown +defines +.B no +operator precedence. An expression combining more than one operator without brackets, such as +.PP +.RS +.nf +def: A /\e B \e/ C +.fi +.RE +.PP +is a parse error, because there is no single unambiguous reading: it could mean +.RI "(" A " /\e " B ") \e/ " C +or +.RI "" A " /\e (" B " \e/ " C ")" , +and substituting the empty set for +.I B +gives two genuinely different results depending on which reading is intended. Rather than pick one +silently, +.B setdown +requires every such expression to be bracketed explicitly: +.PP +.RS +.nf +def: (A /\e B) \e/ C +.fi +.RE +.SS Referencing other definitions +An expression's operand may be a quoted filename, another definition's name, or a bracketed +sub\-expression. Because a bare identifier is itself a valid expression, one definition may simply +alias another: +.PP +.RS +.nf +Combined: "a.txt" \e/ "b.txt" +Alias: Combined +.fi +.RE +.SS Comments +Everything from a double dash +.RB ( \-\- ) +to the end of the line is a comment, wherever it appears on the line: +.PP +.RS +.nf +\-\- This is a definition for A, created because we wanted to do X +A: "y.txt" \- "z.txt" + +B: (A \e/ C) \-\- \e/ D This is still a comment and \e/ D never happens +.fi +.RE +.SS Circular definitions +Definitions must not form a cycle: +.PP +.RS +.nf +A: "file.txt" \e/ B +B: A /\e "other.txt" +.fi +.RE +.PP +.B setdown +detects cycles like this before performing any set operations and exits with status 20 (see +\fBEXIT STATUS\fR), printing the cyclic chain of definition names it found. +.SS A complete example +.RS +.nf +\-\- A is the intersection of the file b\-1.out and the set B +A: "b\-1.out" /\e B + +\-\- B is the union of the files a\-1.out and a\-2.out +B: "a\-1.out" \e/ "a\-2.out" + +\-\- C is the difference of the file b\-1.out and the set B +C: "b\-1.out" \- B + +\-\- D is the symmetric difference of two files +D: "a\-1.out" >< "a\-2.out" +.fi +.RE +.PP +Save this as, for example, +.IR mydefinitions.setdown , +then run: +.PP +.RS +.nf +setdown \-\-input=mydefinitions.setdown +.fi +.RE +.SH OUTPUT +.B setdown +creates an output directory next to the \fB.setdown\fR file (named +.I output +by default; see \fB\-\-output\fR). Each definition produces one result file in that directory, +named after the definition with a +.I .txt +extension \(em a definition called +.I Overlap +produces +.IR output/Overlap.txt . +Every result file contains one element per line, sorted and de\-duplicated. +.PP +While evaluating sub\-expressions, +.B setdown +writes intermediate files to a scratch +.I output/processing/ +subdirectory, which is removed automatically once the run finishes successfully. Pass +\fB\-\-keep\-processing\fR to leave it in place for debugging, and \fB\-\-show\-transient\fR to +have those intermediate results included in the summary table +.B setdown +prints at the end of a run. +.SH EXIT STATUS +.TP +.B 0 +Success. +.TP +.B 1 +The file given with \fB\-\-input\fR does not exist. +.TP +.B 2 +More than one \fB.setdown\fR file was found in the current directory and no \fB\-\-input\fR was +given; use \fB\-\-input\fR to select one. +.TP +.B 3 +No \fB.setdown\fR file was found in the current directory and no \fB\-\-input\fR was given; use +\fB\-\-input\fR to specify one. +.TP +.B 11 +Two or more definitions in the file share the same name. +.TP +.B 12 +A definition references an identifier that has not been defined anywhere in the file. +.TP +.B 13 +One or more input files referenced by the definitions could not be found on disk. +.TP +.B 20 +A cyclic dependency was detected between two or more definitions. +.PP +All file paths reported in error messages are relative to the \fB.setdown\fR file, not the +current working directory. +.SH EXAMPLES +Given two role files listing usernames, one per line: +.PP +.RS +.nf +$ cat admins.txt +alice +bob + +$ cat developers.txt +bob +carol +.fi +.RE +.PP +and a \fB.setdown\fR file: +.PP +.RS +.nf +$ cat access.setdown +InternalStaff: "admins.txt" \e/ "developers.txt" +.fi +.RE +.PP +running +.B setdown +in that directory: +.PP +.RS +.nf +$ setdown +==> Using setdown file: access.setdown +\&... +$ cat output/InternalStaff.txt +alice +bob +carol +.fi +.RE +.PP +produces the union of the two files, sorted and de\-duplicated, in +.IR output/InternalStaff.txt . +More worked examples, covering every operator and several common use cases (access control, +API diffing, feature\-flag segmentation, dependency auditing), are distributed with the setdown +source under +.IR examples/ . +.SH FILES +.TP +.I *.setdown +A definitions file, as described in \fBTHE .setdown FILE FORMAT\fR above. Conventionally suffixed +.IR .setdown , +though +.B setdown +does not require this except when auto\-detecting a file in the current directory (see +\fB\-\-input\fR). +.TP +.I output/ +The default output directory; see \fBOUTPUT\fR. +.TP +.I output/processing/ +Scratch space for intermediate results; see \fBOUTPUT\fR and \fB\-\-keep\-processing\fR. +.SH SEE ALSO +The setdown source repository, including the full set of worked examples referenced above, is at +.BR https://github.com/robertmassaioli/setdown . +.SH AUTHOR +Robert Massaioli +.SH REPORTING BUGS +Report bugs at +.BR https://github.com/robertmassaioli/setdown/issues . +.SH COPYRIGHT +Copyright \(co 2015 Robert Massaioli. Licensed under the 3\-clause BSD license; see the +.I LICENSE +file distributed with the source for the full text. diff --git a/setdown.cabal b/setdown.cabal --- a/setdown.cabal +++ b/setdown.cabal @@ -10,7 +10,7 @@ -- PVP summary: +-+------- breaking API changes -- | | +----- non-breaking API additions -- | | | +--- code changes with no API change -version: 0.2.0.0 +version: 0.2.1.0 -- A short (one-line) description of the package. synopsis: Treating files as sets to perform rapid set manipulation. @@ -23,8 +23,8 @@ built with the intention that you would use it in conjunction with version control tools to manage your set data and set description file. -homepage: http://bitbucket.org/robertmassaioli/setdown -bug-reports: https://bitbucket.org/robertmassaioli/setdown/issues +homepage: https://github.com/robertmassaioli/setdown +bug-reports: https://github.com/robertmassaioli/setdown/issues -- The license under which the package is released. license: BSD3 @@ -37,7 +37,7 @@ -- An email address to which users can send suggestions, bug reports, and -- patches. -maintainer: robertmassaioli@gmail.com +maintainer: setdown@rmdir.app -- A copyright notice. copyright: (c) 2015 Robert Massaioli @@ -49,13 +49,14 @@ -- Extra files to be distributed with the package, such as examples or a -- README. extra-source-files: README.markdown + , man/setdown.1 -- Constraint on the version of Cabal needed to build this package. cabal-version: >=1.10 source-repository head type: git - location: git@bitbucket.org:robertmassaioli/setdown.git + location: https://github.com/robertmassaioli/setdown.git library exposed-modules: SetLanguage @@ -71,20 +72,21 @@ , SetInput , SetInputVerification , SimpleDefinitionCycles + , TableRender build-depends: base >=4.7 && < 5 -- Lexing dependencies - , array >= 0.5 && < 0.6 - , bytestring >= 0.10 && < 0.13 - , text >= 1.2 && < 2.2 + , array >= 0.5 && < 1 + , bytestring >= 0.10 && < 1 + , text >= 1.2 && < 3 -- Module Dependencies , filepath >= 1.2 && < 3 , directory >= 1.1 && < 3 - , containers >= 0.6 && < 0.8 - , uuid >= 1.3 && < 1.4 - , split == 0.2.* - , mtl >= 2.2 && < 2.4 - , async >= 2.2 && < 2.3 + , containers >= 0.6 && < 1 + , uuid >= 1.3 && < 2 + , split >= 0.2 && < 1 + , mtl >= 2.2 && < 3 + , async >= 2.2 && < 3 build-tools: alex, happy @@ -99,16 +101,15 @@ build-depends: base >=4.7 && < 5 , setdown - , bytestring >= 0.10 && < 0.13 - , text >= 1.2 && < 2.2 - , containers >= 0.6 && < 0.8 + , bytestring >= 0.10 && < 1 + , text >= 1.2 && < 3 + , containers >= 0.6 && < 1 , directory >= 1.1 && < 3 , filepath >= 1.2 && < 3 - , cmdargs >= 0.10 && < 0.11 - , table-layout >= 0.8 && < 1.1 + , cmdargs >= 0.10 && < 1 if !os(windows) - build-depends: unix >= 2.7 && < 2.9 + build-depends: unix >= 2.7 && < 3 hs-source-dirs: app @@ -123,10 +124,10 @@ build-depends: base >=4.7 && < 5 , setdown - , tasty >= 1.4 && < 1.6 - , tasty-hunit >= 0.10 && < 0.11 - , text >= 1.2 && < 2.2 - , bytestring >= 0.10 && < 0.13 + , tasty >= 1.4 && < 2 + , tasty-hunit >= 0.10 && < 1 + , text >= 1.2 && < 3 + , bytestring >= 0.10 && < 1 default-language: Haskell2010 @@ -139,10 +140,10 @@ build-depends: base >=4.7 && < 5 , setdown - , tasty >= 1.4 && < 1.6 - , tasty-quickcheck >= 0.10 && < 0.12 - , QuickCheck >= 2.14 && < 2.16 - , text >= 1.2 && < 2.2 + , tasty >= 1.4 && < 2 + , tasty-quickcheck >= 0.10 && < 1 + , QuickCheck >= 2.14 && < 3 + , text >= 1.2 && < 3 default-language: Haskell2010 @@ -154,10 +155,11 @@ hs-source-dirs: test build-depends: base >=4.7 && < 5 - , tasty >= 1.4 && < 1.6 - , tasty-golden >= 2.3 && < 2.4 + , tasty >= 1.4 && < 2 + , tasty-golden >= 2.3 && < 3 + , tasty-hunit >= 0.10 && < 1 , filepath >= 1.2 && < 3 - , process >= 1.6 && < 1.7 + , process >= 1.6 && < 2 default-language: Haskell2010 diff --git a/src/TableRender.hs b/src/TableRender.hs new file mode 100644 --- /dev/null +++ b/src/TableRender.hs @@ -0,0 +1,49 @@ +module TableRender + ( Align(..) + , renderTable + ) where + +import Data.List (intercalate, transpose) + +-- | How a column's cells are padded to the column width. +data Align = AlignLeft | AlignRight + +-- | Render a header row and body rows as a Unicode box-drawing table: a bold +-- (heavy-lined) top border and header separator, single-lined body borders, +-- one alignment per column, and column widths sized to the widest cell +-- (including the header). This reproduces the exact visual style setdown has +-- always used for its results tables. +-- +-- Column widths and the header row always account for the header text, even +-- when there are zero body rows, so headers never disappear on empty input. +renderTable :: [Align] -> [String] -> [[String]] -> [String] +renderTable aligns headers rows = + [topBorder, headerRow, headerSeparator] ++ map bodyRow rows ++ [bottomBorder] + where + widths = zipWith columnWidth headers (columnsOf headers rows) + columnWidth header column = maximum (length header : map length column) + + columnsOf hs [] = replicate (length hs) [] + columnsOf _ rs = transpose rs + + topBorder = border '┏' '┳' '┓' '━' + headerSeparator = border '┡' '╇' '┩' '━' + bottomBorder = border '└' '┴' '┘' '─' + + border left mid right line = + [left] ++ intercalate [mid] (map (\w -> replicate (w + 2) line) widths) ++ [right] + + headerRow = rowOf '┃' (zipWith padCenter widths headers) + bodyRow cells = rowOf '│' (zipWith3 padAlign widths aligns cells) + + rowOf sep cells = + [sep] ++ intercalate [sep] (map (\c -> ' ' : c ++ " ") cells) ++ [sep] + + padCenter width s = + let extra = width - length s + left = extra `div` 2 + right = extra - left + in replicate left ' ' ++ s ++ replicate right ' ' + + padAlign width AlignLeft s = s ++ replicate (width - length s) ' ' + padAlign width AlignRight s = replicate (width - length s) ' ' ++ s diff --git a/test/GoldenTests.hs b/test/GoldenTests.hs --- a/test/GoldenTests.hs +++ b/test/GoldenTests.hs @@ -2,8 +2,11 @@ import Test.Tasty import Test.Tasty.Golden -import System.FilePath (()) -import System.Process (callProcess) +import Test.Tasty.HUnit +import System.FilePath (()) +import System.Process (callProcess, readProcessWithExitCode) +import System.Exit (ExitCode(..)) +import Data.List (isInfixOf) main :: IO () main = defaultMain tests @@ -14,6 +17,9 @@ , goldenTest "union" , goldenTest "difference" , goldenTest "symmetric-difference" + , goldenTest "single-element-distinct" + , goldenTest "single-element-identical" + , errorDetectionTests ] -- | Run setdown on a fixture directory and compare the Result.txt output @@ -42,3 +48,31 @@ runSetdown :: FilePath -> IO () runSetdown inputFile = callProcess "stack" ["exec", "--", "setdown", "-i", inputFile] + +-- --------------------------------------------------------------------------- +-- Error detection (CLI integration) +-- --------------------------------------------------------------------------- + +-- | Each fixture lives under test/golden/errors//example.setdown and is +-- expected to make setdown exit with a specific failure code, printing a +-- message containing the given fragment. There is no golden output file +-- here, since setdown exits before writing any results. +errorDetectionTests :: TestTree +errorDetectionTests = testGroup "error detection" + [ errorTest "duplicate-definition" 11 "Duplicate definitions found" + , errorTest "unknown-identifier" 12 "Unknown identifiers used" + , errorTest "missing-file" 13 "the following files could not be found" + ] + +errorTest :: String -> Int -> String -> TestTree +errorTest name expectedCode expectedFragment = + testCase name $ do + (exitCode, stdout, _stderr) <- readProcessWithExitCode "stack" + ["exec", "--", "setdown", "-i", fixtureDir "example.setdown"] + "" + exitCode @?= ExitFailure expectedCode + assertBool + ("expected stdout to mention \"" ++ expectedFragment ++ "\", got:\n" ++ stdout) + (expectedFragment `isInfixOf` stdout) + where + fixtureDir = "test" "golden" "errors" name diff --git a/test/UnitTests.hs b/test/UnitTests.hs --- a/test/UnitTests.hs +++ b/test/UnitTests.hs @@ -6,12 +6,16 @@ import qualified Data.Text.Lazy as T import qualified Data.ByteString.Lazy.Char8 as BC +import Data.List (isInfixOf) +import Control.Exception (SomeException, evaluate, try) import PerformOperations (linesSetOperation, operatorTools) import SetData import SimpleDefinitionCycles (getCyclesInSimpleDefinitions) import DuplicateElimination (eliminateDuplicates, orderDefinitions) import SetInput (parse) +import SetInputVerification (duplicateDefinitionName, unknownIdentifier) +import TableRender (Align (..), renderTable) main :: IO () main = defaultMain tests @@ -27,6 +31,9 @@ , cycleDetectionTests , duplicateEliminationTests , parseTests + , parseErrorTests + , verificationTests + , tableRenderTests ] -- --------------------------------------------------------------------------- @@ -85,6 +92,10 @@ lso Or ["a", "b"] [] @?= ["a", "b"] , testCase "both empty → empty" $ lso Or [] [] @?= [] + , testCase "single distinct elements → both kept" $ + lso Or ["x"] ["y"] @?= ["x", "y"] + , testCase "single identical elements → one kept" $ + lso Or ["x"] ["x"] @?= ["x"] ] -- --------------------------------------------------------------------------- @@ -106,6 +117,10 @@ , testCase "A - B ≠ B - A (not commutative)" $ do lso Difference ["a", "b"] ["a"] @?= ["b"] lso Difference ["a"] ["a", "b"] @?= [] + , testCase "single identical elements → empty" $ + lso Difference ["x"] ["x"] @?= [] + , testCase "single distinct elements → left unchanged" $ + lso Difference ["x"] ["y"] @?= ["x"] ] -- --------------------------------------------------------------------------- @@ -132,6 +147,10 @@ b = ["b", "c", "d"] in lso SymmetricDifference a b @?= lso Or (lso Difference a b) (lso Difference b a) + , testCase "single identical elements → empty" $ + lso SymmetricDifference ["x"] ["x"] @?= [] + , testCase "single distinct elements → both kept" $ + lso SymmetricDifference ["x"] ["y"] @?= ["x", "y"] ] -- --------------------------------------------------------------------------- @@ -223,4 +242,179 @@ length (parse (BC.pack "A: (\"a.txt\" /\\ \"b.txt\")")) @?= 1 , testCase "comment is ignored" $ length (parse (BC.pack "-- just a comment\nA: \"a.txt\"")) @?= 1 + ] + +-- --------------------------------------------------------------------------- +-- Parse errors +-- --------------------------------------------------------------------------- + +-- | Force full evaluation of a parse so that any error thrown from within +-- the lazy parse tree is raised here, where we can catch it. +parseFailure :: String -> IO (Either SomeException String) +parseFailure input = try (evaluate (show (parse (BC.pack input)))) + +assertParseFailureContains :: String -> [String] -> IO () +assertParseFailureContains input expectedFragments = do + result <- parseFailure input + case result of + Left e -> + let msg = show e + in mapM_ (\frag -> assertBool + ("expected \"" ++ frag ++ "\" in error message: " ++ msg) + (frag `isInfixOf` msg)) + expectedFragments + Right _ -> assertFailure "expected a parse error, but parsing succeeded" + +parseErrorTests :: TestTree +parseErrorTests = testGroup "parse errors" + [ testCase "two filenames with no operator reports line and column" $ + assertParseFailureContains + "A: \"a.txt\" \"b.txt\"" + ["line 1", "column 12", "unexpected"] + , testCase "unexpected token reports the correct line in multi-line input" $ + assertParseFailureContains + "A: \"a.txt\"\nB: \"b.txt\" \"c.txt\"" + ["line 2", "column 12"] + , testCase "unrecognised character is reported as a lexical error" $ + assertParseFailureContains + "A: @" + ["lexical error", "line 1", "column 4"] + , testCase "unrecognised character on a later line reports that line" $ + assertParseFailureContains + "A: \"a.txt\"\nB: \"b.txt\"\nC: @" + ["lexical error", "line 3", "column 4"] + ] + +-- --------------------------------------------------------------------------- +-- Verification: duplicate names and unknown identifiers +-- --------------------------------------------------------------------------- + +mkRawDef :: String -> Expression -> Definition +mkRawDef name expr = Definition (T.pack name) expr + +verificationTests :: TestTree +verificationTests = testGroup "verification" + [ testGroup "duplicate definition names" + [ testCase "no duplicates → no errors" $ + duplicateDefinitionName + [ mkRawDef "A" (FileExpression "a.txt") + , mkRawDef "B" (FileExpression "b.txt") + ] + @?= [] + , testCase "one name defined twice → one error" $ + length (duplicateDefinitionName + [ mkRawDef "A" (FileExpression "a.txt") + , mkRawDef "A" (FileExpression "b.txt") + ]) + @?= 1 + , testCase "one name defined three times → still one grouped error" $ + length (duplicateDefinitionName + [ mkRawDef "A" (FileExpression "a.txt") + , mkRawDef "A" (FileExpression "b.txt") + , mkRawDef "A" (FileExpression "c.txt") + ]) + @?= 1 + ] + , testGroup "unknown identifiers" + [ testCase "no references → no errors" $ + unknownIdentifier + [ mkRawDef "A" (FileExpression "a.txt") ] + @?= [] + , testCase "reference to a defined identifier → no errors" $ + unknownIdentifier + [ mkRawDef "A" (FileExpression "a.txt") + , mkRawDef "B" (IdentifierExpression "A") + ] + @?= [] + , testCase "reference to an undefined identifier → one error" $ + length (unknownIdentifier + [ mkRawDef "A" (IdentifierExpression "B") ]) + @?= 1 + , testCase "undefined identifier used inside a binary expression → one error" $ + length (unknownIdentifier + [ mkRawDef "A" (BinaryExpression And + (FileExpression "a.txt") + (IdentifierExpression "Missing")) + ]) + @?= 1 + ] + ] + +-- --------------------------------------------------------------------------- +-- Table rendering +-- +-- The expected outputs for "two left-aligned columns" and "left, left, +-- right" below were captured verbatim from setdown running against the +-- real, upstream table-layout library (columnHeaderTableS/unicodeBoldHeaderS) +-- before it was replaced by TableRender, to guarantee this reimplementation +-- is byte-for-byte compatible with the table style setdown has always used. +-- --------------------------------------------------------------------------- + +tableRenderTests :: TestTree +tableRenderTests = testGroup "table rendering" + [ testCase "two left-aligned columns, matches table-layout output exactly" $ + renderTable [AlignLeft, AlignLeft] ["From", "To"] + [ ["api-v1.txt", "./output/api-v1.txt.1.split.sorted"] + , ["api-v2.txt", "./output/api-v2.txt.1.split.sorted"] + ] + @?= + [ "┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓" + , "┃ From ┃ To ┃" + , "┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩" + , "│ api-v1.txt │ ./output/api-v1.txt.1.split.sorted │" + , "│ api-v2.txt │ ./output/api-v2.txt.1.split.sorted │" + , "└────────────┴────────────────────────────────────┘" + ] + , testCase "left, left, right columns, matches table-layout output exactly" $ + renderTable [AlignLeft, AlignLeft, AlignRight] ["Name", "File", "Count"] + [ ["Added", "./output/Added.txt", "4"] + , ["Removed", "./output/Removed.txt", "2"] + ] + @?= + [ "┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┓" + , "┃ Name ┃ File ┃ Count ┃" + , "┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━┩" + , "│ Added │ ./output/Added.txt │ 4 │" + , "│ Removed │ ./output/Removed.txt │ 2 │" + , "└─────────┴──────────────────────┴───────┘" + ] + , testCase "column width grows to fit a right-aligned multi-digit value" $ + renderTable [AlignLeft, AlignRight] ["Name", "Count"] + [ ["AllDeps", "12"] + , ["Shared", "7"] + ] + @?= + [ "┏━━━━━━━━━┳━━━━━━━┓" + , "┃ Name ┃ Count ┃" + , "┡━━━━━━━━━╇━━━━━━━┩" + , "│ AllDeps │ 12 │" + , "│ Shared │ 7 │" + , "└─────────┴───────┘" + ] + , testCase "cell narrower than its header is padded to the header width" $ + renderTable [AlignLeft, AlignLeft] ["From", "To"] [["a", "b"]] + @?= + [ "┏━━━━━━┳━━━━┓" + , "┃ From ┃ To ┃" + , "┡━━━━━━╇━━━━┩" + , "│ a │ b │" + , "└──────┴────┘" + ] + , testCase "empty string cell is padded like any other cell" $ + renderTable [AlignLeft, AlignLeft] ["From", "To"] [["", "x"]] + @?= + [ "┏━━━━━━┳━━━━┓" + , "┃ From ┃ To ┃" + , "┡━━━━━━╇━━━━┩" + , "│ │ x │" + , "└──────┴────┘" + ] + , testCase "no rows still renders headers (deliberately unlike upstream table-layout, which drops header text and collapses width to zero when the row list is empty)" $ + renderTable [AlignLeft, AlignLeft] ["From", "To"] [] + @?= + [ "┏━━━━━━┳━━━━┓" + , "┃ From ┃ To ┃" + , "┡━━━━━━╇━━━━┩" + , "└──────┴────┘" + ] ]