setdown 0.2.0.0 → 0.2.1.0
raw patch · 7 files changed
+821/−214 lines, 7 filesdep −table-layoutdep ~QuickCheckdep ~arraydep ~asyncPVP ok
version bump matches the API change (PVP)
Dependencies removed: table-layout
Dependency ranges changed: QuickCheck, array, async, bytestring, cmdargs, containers, mtl, process, split, tasty, tasty-golden, tasty-hunit, tasty-quickcheck, text, unix, uuid
API changes (from Hackage documentation)
+ TableRender: AlignLeft :: Align
+ TableRender: AlignRight :: Align
+ TableRender: data Align
+ TableRender: renderTable :: [Align] -> [String] -> [[String]] -> [String]
Files
- README.markdown +147/−160
- app/Main.hs +14/−22
- man/setdown.1 +349/−0
- setdown.cabal +32/−30
- src/TableRender.hs +49/−0
- test/GoldenTests.hs +36/−2
- test/UnitTests.hs +194/−0
README.markdown view
@@ -1,86 +1,75 @@-# Setdown - Line based set manipulation+# Setdown -**Version:** 0.1.3.0 | [Hackage][7]+Line-based set manipulation from the command line. -Author: [Robert Massaioli][6]-Created in: 2015+[](https://github.com/robertmassaioli/setdown/actions/workflows/test.yml)+[][7] +Author: [Robert Massaioli][6] · Created in 2015++## What is setdown?++Setdown treats text files as sets — one element per line — and lets you combine them with+intersection, union, difference, and symmetric difference. You describe the operations once in a+**`.setdown`** file (think of it like a `Makefile` for sets), and setdown resolves the whole+dependency graph, computes every definition, and writes one result file per definition.++```setdown+InternalStaff: "admins.txt" \/ "developers.txt"+AllUsers: InternalStaff \/ "contractors.txt"+ContractorsOnly: "contractors.txt" - InternalStaff+```++Run `setdown` in the directory containing that file and you get an `output/` directory with+`InternalStaff.txt`, `AllUsers.txt`, and `ContractorsOnly.txt` — each sorted and de-duplicated.++Input files don't need to be sorted, de-duplicated, or even sets to begin with; setdown normalizes+them as it goes. And setdown is current-working-directory invariant: all paths inside a+**`.setdown`** file are resolved relative to that file, not to wherever you happen to run the+command from, so you can invoke it from anywhere in your project tree and get the same result.+ ## Installation ### Via nix-shell (quickest, no local setup required) -``` shell+```shell $ nix-shell -p haskellPackages.setdown $ setdown --help ``` ### Via Hackage -``` shell+```shell stack install setdown ``` -This works because [setdown is on Hackage][7].--## What is setdown and how does it work?--Setdown is a command line tool for line based set operations. To use setdown you write a "setdown-definitions file" often suffixed with **.setdown**. If you are familiar with [Make][3] then you can think-of this **.setdown** file much like a Makefile. Inside that file you write a number of-definitions of the form:--``` setdown-definitionName: "file-1.txt" /\ "file-2.txt"-```--This line says that "definitionName" is a new set definition that is a label for the intersection of-"file-1.txt" and "file-2.txt". You can write more complicated expressions than this.--### Example Setdown Projects--[Checkout the setdown-examples project][2] on Bitbucket; it will show you how setdown works.--However, to get an in-depth description of setdown and its abilities you should-read the sections below.--### Input Files--In setdown *each file is treated as a list of elements where each line-is an element*. Input files do not need to begin as sets; they can contain duplicate and unsorted-elements. Setdown will automatically sort and de-duplicate all input files, turning them into sets.--Another important point is that of relativity: specifically, if you have a **.setdown** file that-references the input file "some-elements.txt" and you run the setdown executable from a directory that-is not the same directory as the **.setdown** file, where will setdown look for-some-elements.txt? The answer is that setdown always looks for files relative to the **.setdown**-file. That is where you wrote your definitions so the paths are relative to that. It was designed in-this way so that you could run setdown from anywhere in the directory tree and still get the same-result. Setdown has been designed to be current working directory invariant, as opposed to many-other command line programs. Please keep this in mind.--### Output--When setdown runs, it creates an `output/` directory next to your **.setdown** file. Each named-definition produces a result file in that directory named after the definition with a `.txt`-extension — for example, a definition called `Overlap` produces `output/Overlap.txt`. The file-contains one element per line, sorted and de-duplicated.+This works because [setdown is on Hackage][7]. To build from source instead — for example, to get+the latest unreleased changes — see [Building the code](#building-the-code) below. -Progress and status messages are written to stdout as setdown works through your definitions. At-the end of a successful run, a summary table is printed showing each definition name, the path to-its result file, and the number of elements it contains.+## Quick start -You can choose a different output directory with the `--output` flag:+Every example below lives under [`examples/`](examples) in this repository — clone the repo and+run any of them directly. -``` shell-setdown --output=results mydefinitions.setdown+```shell+git clone https://github.com/robertmassaioli/setdown.git+cd setdown/examples/access-control+stack exec -- setdown ``` -The path given to `--output` is relative to the **.setdown** file, not the current working-directory.--### Set Operations and Precedence+| Example | What it shows |+|---|---|+| [`standard`](examples/standard) | A tour of intersection, union, and difference, including bracketed precedence |+| [`access-control`](examples/access-control) | Deriving permission groups (internal staff, contractors) from role files |+| [`basic-difference`](examples/basic-difference) | Diffing two API surfaces to find what was added and removed |+| [`data-reconciliation`](examples/data-reconciliation) | Comparing two months of customer lists: retained, new, and lost |+| [`feature-flags`](examples/feature-flags) | Segmenting users by experiment exposure, including a three-way overlap |+| [`software-dependencies`](examples/software-dependencies) | Auditing shared and unique dependencies across two applications |+| [`symmetric-difference`](examples/symmetric-difference) | The `><` operator versus the equivalent longhand expression |+| [`cycle-detection`](examples/cycle-detection) | A deliberately cyclic definition, and the error setdown reports |+| [`parse-error`](examples/parse-error) | A deliberately malformed expression, and the error setdown reports | -In the setdown language there are a number of supported operators:+## Set operations and precedence | Operator | ASCII | Unicode | |----------|-------|---------|@@ -92,100 +81,66 @@ Intersection, union, and symmetric difference are commutative (`A op B` is the same as `B op A`). Difference is not (`A - B` ≠ `B - A`). -Symmetric difference (`><` or `△`) yields the elements that appear in exactly one of-the two inputs — those in A but not B, plus those in B but not A. It is equivalent to-`(A - B) \/ (B - A)` but computed in a single pass.--For example, they might be used in the following way:+Symmetric difference (`><` or `△`) yields the elements that appear in exactly one of the two+inputs — those in A but not B, plus those in B but not A. It is equivalent to `(A - B) \/ (B - A)`+but computed in a single pass; see [`examples/symmetric-difference`](examples/symmetric-difference)+for both forms side by side. -``` setdown+```setdown definition: (A - B) \/ (C /\ D) changedSubscribers: "january.txt" >< "february.txt" ``` -You may be wondering what [operator precedence][1] the setdown language uses and the answer is:-there is no operator precedence at all, instead *you must clearly specify the precedence of nested-expressions with brackets*. This is very important because it will result in parsing errors-otherwise. To explain the reasoning for explicit operator precedence:+**There is no operator precedence** — you must bracket nested expressions explicitly. Consider: -``` setdown--- Here is a simple expression+```setdown def: A /\ B \/ C--- Now, should this be parsed as:-defV1: (A /\ B) \/ C--- or as:-defV2: A /\ (B \/ C)--- If you pretend that B is the empty set (E) then you can see that these expressions evaluate--- completely differently. If we simplify them with that assumption then they become:-defV1-bempty: E-defV2-bempty: A /\ C ``` -So as you can see, order of operations really matters for set operations. Because it is so critical-the use of brackets is mandatory.--### Comments--In the setdown language you can add comments by writing a double-dash (`--`) and then writing the-comment to the end of the line. Comments can appear anywhere on a line — at the start, or inline-after an expression.--``` setdown--- This is a definition for A, created because we wanted to do X-A: "y.txt" - "z.txt"---- This is an example of a comment halfway through an expression-B: (A \/ C) -- \/ D This is still a comment and \/ D never happens-```--You can use comments to leave messages for any people that might read your setdown definitions in-the future.--### Language Reference--#### Identifier rules+Should this parse as `(A /\ B) \/ C` or `A /\ (B \/ C)`? Substitute the empty set for `B` and the+two readings diverge completely (`E` versus `A /\ C`). Because the difference is not cosmetic,+setdown refuses to guess — an unbracketed expression like this is a parse error. -Definition names (identifiers) may contain letters (upper and lowercase), digits, hyphens, and-underscores:+## Language reference -```-[a-zA-Z0-9_-]+-```+### Identifiers -For example, `mySet`, `result-2`, and `Final_Output` are all valid identifiers. Spaces and-punctuation other than `-` and `_` are not permitted.+Definition names may contain letters, digits, hyphens, and underscores: `[a-zA-Z0-9_-]+`. For+example, `mySet`, `result-2`, and `Final_Output` are all valid; spaces and other punctuation are+not permitted. -#### Definition ordering+### Definition ordering -Definitions may appear in any order in your **.setdown** file. A definition may reference another-that is defined later in the file. Setdown resolves all identifiers by name after parsing the-complete file.+Definitions may appear in any order in a **`.setdown`** file, and a definition may reference+another that's defined later on. Setdown resolves all identifiers by name after parsing the whole+file. -#### Circular definitions+### Circular definitions -Definitions must not form a cycle. For example:+Definitions must not form a cycle: -``` setdown+```setdown A: "file.txt" \/ B B: A /\ "other.txt" ``` -This is invalid because `A` depends on `B` and `B` depends on `A`. Setdown detects cycles and-exits with an error before performing any operations.+Setdown detects cycles like this and exits with an error before performing any operations — see+[`examples/cycle-detection`](examples/cycle-detection). -### Writing your own definitions+### Comments -In the setdown language you can write a definition in the following format:+Add comments with a double-dash (`--`) through the end of the line, anywhere on the line: -``` setdown-<definitionName>: <expression>+```setdown+-- This is a definition for A, created because we wanted to do X+A: "y.txt" - "z.txt"++B: (A \/ C) -- \/ D This is still a comment and \/ D never happens ``` -Where the definition name is the identifier that you give to that expression. An expression is the-application of set operations on identifiers or files. A practical example of what this looks like-should help cement what this means. Here is a valid setdown file:+### Full example -``` setdown+```setdown -- A is the intersection of the file b-1.out and the set B A: "b-1.out" /\ B @@ -199,19 +154,37 @@ D: "a-1.out" >< "a-2.out" ``` -Usually, when you write these definitions you put them in a file that has a suffix of **.setdown**.-You can then feed this file into the setdown executable like so:+Files with these definitions are usually suffixed **`.setdown`** and fed to the executable: -``` shell+```shell setdown path/to/mydefinitions.setdown ``` -### Command-line flags+## Output -``` text+When setdown runs, it creates an `output/` directory next to your **`.setdown`** file. Each named+definition produces a result file in that directory named after the definition with a `.txt`+extension — for example, a definition called `Overlap` produces `output/Overlap.txt`. The file+contains one element per line, sorted and de-duplicated.++Intermediate results for sub-expressions are computed in a scratch `output/processing/` directory,+which is removed automatically once the run finishes. Pass `--keep-processing` to leave it in+place for debugging, and `--show-transient` to have those intermediate results included in the+summary table setdown prints at the end of a run.++You can choose a different output directory with `--output`, given relative to the **`.setdown`**+file, not the current working directory:++```shell+setdown --output=results mydefinitions.setdown+```++## Command-line flags++```text setdown evaluates a .setdown definitions file to perform set operations-(intersection, union, difference, symmetric difference) on line-based text-files, writing one result file per definition to an output directory.+(intersection, union, difference) on line-based text files, writing one+result file per definition to an output directory. setdown [OPTIONS] @@ -228,45 +201,60 @@ sub-expressions generated internally to evaluate your definitions. Useful for debugging complex .setdown files.+ --keep-processing Keep the processing/ subdirectory after the+ run completes instead of deleting it. Useful+ for inspecting intermediate files when+ debugging. -? --help Display help message -V --version Print version information ``` -## Building the code--To build the code for this project, have [Stack][10] installed and then:--``` shell-stack build-```--To run setdown during development:--``` shell-stack exec -- setdown --help-stack exec -- setdown mydefinitions.setdown-```- ## Troubleshooting Setdown prints a short error message to stdout and exits with a non-zero code when something goes-wrong. The error codes are:+wrong: | Exit code | Cause | |-----------|-------| | 1 | The file specified with `--input` does not exist. |-| 2 | Multiple **.setdown** files found in the current directory; use `--input` to select one. |-| 3 | No **.setdown** files found in the current directory; use `--input` to specify one. |+| 2 | Multiple **`.setdown`** files found in the current directory; use `--input` to select one. |+| 3 | No **`.setdown`** files found in the current directory; use `--input` to specify one. | | 11 | Two or more definitions share the same name. | | 12 | A definition references an identifier that has not been defined. | | 13 | One or more input files referenced in the definitions could not be found. | | 20 | A cyclic dependency was detected between definitions. | -All file paths in error messages are relative to the **.setdown** file, not the current working-directory.+All file paths in error messages are relative to the **`.setdown`** file, not the current working+directory. See [`examples/parse-error`](examples/parse-error) and+[`examples/cycle-detection`](examples/cycle-detection) for these in action. -## Contributing to the setdown project+## Building the code +Setdown is a Haskell library (the set language, parser, and evaluation engine) plus a thin+`setdown` executable, built with [Stack][10]:++```shell+stack build+```++Run the three test suites — unit, property-based, and golden — with:++```shell+stack test+```++The same build-and-test steps run in CI on every push and pull request; see+[`.github/workflows`](.github/workflows).++To run setdown during development, without installing it:++```shell+stack exec -- setdown --help+stack exec -- setdown mydefinitions.setdown+```++## Contributing+ Contributions are welcome. The preferred workflow is: 1. Open an issue describing what you intend to fix or improve.@@ -275,10 +263,9 @@ 4. Iterate until the code is clean and merged. 5. Celebrate! - [1]: http://en.wikipedia.org/wiki/Order_of_operations- [2]: https://bitbucket.org/robertmassaioli/setdown-examples- [3]: http://www.gnu.org/software/make/+Design proposals and background for larger changes live under [`ai-planning/`](ai-planning) if+you'd like context before picking something up.+ [6]: https://github.com/robertmassaioli [7]: http://hackage.haskell.org/package/setdown- [8]: https://nixos.org/manual/nix/unstable/command-ref/nix-shell.html [10]: https://docs.haskellstack.org/en/stable/
app/Main.hs view
@@ -7,7 +7,7 @@ import qualified Data.Text.Lazy as LT import qualified Data.Text.Lazy.IO as T-import qualified Text.Layout.Table as Tab+import GHC.IO.Encoding (setLocaleEncoding, utf8) import System.Console.CmdArgs import System.Exit @@ -31,6 +31,7 @@ import SetData import SetInput import SetInputVerification+import TableRender (Align (..), renderTable) import DuplicateElimination import PerformOperations@@ -135,6 +136,12 @@ -- for one main :: IO () main = do+ -- setdown prints Unicode box-drawing characters in its results tables. Force+ -- UTF-8 regardless of the ambient locale so this doesn't crash on systems+ -- without one configured (e.g. minimal Debian/Ubuntu installs and most CI+ -- images default to the C/POSIX locale).+ setLocaleEncoding utf8+ opts <- cmdArgs options inputFilePath <- getInputFileOrFail (setdownFile opts)@@ -324,29 +331,14 @@ wrapInQuotes x = "\"" ++ x ++ "\"" printTabularResults :: [(FilePath, FilePath)] -> IO ()-printTabularResults fileMapping = sequence_ . fmap putStrLn $ Tab.tableLines (Tab.columnHeaderTableS columns Tab.unicodeBoldHeaderS headers rows)- where- headers = Tab.titlesH ["From", "To"]-- columns =- [ Tab.column Tab.expand Tab.left Tab.noAlign Tab.noCutMark- , Tab.column Tab.expand Tab.left Tab.noAlign Tab.noCutMark- ]-- rows = [Tab.rowsG $ fmap (\(from, to) -> [from, to]) fileMapping]+printTabularResults fileMapping = mapM_ putStrLn $+ renderTable [AlignLeft, AlignLeft] ["From", "To"]+ (fmap (\(from, to) -> [from, to]) fileMapping) printTabularResultsWithCount :: [(String, FilePath, Int)] -> IO ()-printTabularResultsWithCount rows = sequence_ . fmap putStrLn $ Tab.tableLines (Tab.columnHeaderTableS columns Tab.unicodeBoldHeaderS headers tableRows)- where- headers = Tab.titlesH ["Name", "File", "Count"]-- columns =- [ Tab.column Tab.expand Tab.left Tab.noAlign Tab.noCutMark- , Tab.column Tab.expand Tab.left Tab.noAlign Tab.noCutMark- , Tab.column Tab.expand Tab.right Tab.noAlign Tab.noCutMark- ]-- tableRows = [Tab.rowsG $ fmap (\(defName, fp, n) -> [defName, fp, show n]) rows]+printTabularResultsWithCount rows = mapM_ putStrLn $+ renderTable [AlignLeft, AlignLeft, AlignRight] ["Name", "File", "Count"]+ (fmap (\(defName, fp, n) -> [defName, fp, show n]) rows) printComputedResults :: Options -> [(SimpleDefinition, FilePath, Int)] -> IO () printComputedResults opts results = do
+ man/setdown.1 view
@@ -0,0 +1,349 @@+.\" Man page for setdown. Keep the .TH date/version in sync with setdown.cabal on release.+.TH SETDOWN 1 "2026-08-22" "setdown 0.2.0.0" "User Commands"+.SH NAME+setdown \- evaluate a \fB.setdown\fR file to perform set operations on line\-based text files+.SH SYNOPSIS+.B setdown+[\fB\-i\fR \fIFILE\fR]+[\fB\-o\fR \fIDIR\fR]+[\fB\-\-show\-transient\fR]+[\fB\-\-keep\-processing\fR]+.br+.B setdown+\fB\-\-help\fR+.br+.B setdown+\fB\-\-version\fR+.SH DESCRIPTION+.B setdown+treats text files as sets \(em one element per line \(em and lets you combine them with+intersection, union, difference, and symmetric difference. The operations are described once in a+\fB.setdown\fR definitions file, similar in spirit to a+.BR Makefile ,+and+.B setdown+resolves the whole dependency graph between definitions, computes every one of them, and writes+one result file per definition into an output directory.+.PP+Input files do not need to be sorted, de\-duplicated, or already act like sets; \fBsetdown\fR+normalizes them as it goes. Every path written inside a \fB.setdown\fR file \(em both the input+files it reads and the \fB\-\-output\fR directory it writes to \(em is resolved relative to the+location of that \fB.setdown\fR file, never relative to the current working directory. This makes+a \fB.setdown\fR file's behaviour independent of where you happen to invoke+.B setdown+from.+.PP+If no \fB.setdown\fR file is given explicitly with \fB\-\-input\fR,+.B setdown+looks for exactly one \fB.setdown\fR file in the current directory and uses it automatically.+.SH OPTIONS+.TP+\fB\-o\fR, \fB\-\-output\fR[=\fIDIR\fR]+Directory in which to place output files, given relative to the \fB.setdown\fR file being+evaluated (not the current working directory). Defaults to+.I output+if this option is omitted entirely.+.TP+\fB\-i\fR, \fB\-\-input\fR=\fIFILE\fR+The \fB.setdown\fR definitions file to evaluate. If omitted,+.B setdown+looks for a single \fB.setdown\fR file in the current directory and uses it automatically. Exits+with an error if zero, or more than one, are found (see \fBEXIT STATUS\fR below).+.TP+\fB\-\-show\-transient\fR+Also print intermediate results for the sub\-expressions+.B setdown+generates internally while evaluating your definitions, in addition to the named definitions+themselves. Useful when a definition's expression is complex and you want to see each+intersection/union/difference/symmetric\-difference step that led to the final result.+.TP+\fB\-\-keep\-processing\fR+Keep the scratch+.I processing/+subdirectory (see \fBFILES\fR below) after the run completes, instead of deleting it. Useful for+inspecting the intermediate files behind a definition when debugging unexpected output.+.TP+\fB\-?\fR, \fB\-\-help\fR+Display a short help message summarising these options, then exit.+.TP+\fB\-V\fR, \fB\-\-version\fR+Print version information, then exit.+.SH THE .setdown FILE FORMAT+A \fB.setdown\fR file is a plain text file containing one or more+.I definitions,+each of the form:+.PP+.RS+.I name\fR: \fIexpression\fR+.RE+.PP+.B setdown+evaluates every definition in the file and writes one result file per definition (see+\fBOUTPUT\fR). Definitions may appear in any order \(em a definition may reference another+definition that is written later in the same file, since+.B setdown+resolves all names after parsing the whole file.+.SS Identifiers+A definition's name may contain letters, digits, hyphens, and underscores+.RI ( [a\-zA\-Z0\-9_\-]+ ).+Spaces and other punctuation are not permitted. Examples of valid names:+.IR mySet ", " result\-2 ", " Final_Output .+.SS Filenames+A file is referenced by writing its path in double quotes, e.g.\&+.IR \(dqusers.txt\(dq .+The path may contain spaces and is resolved relative to the location of the \fB.setdown\fR file+itself, not the current working directory and not any \fB\-\-output\fR directory.+.SS Operators+An expression combines files and other definitions using one of four binary set operators. Each+has an ASCII spelling, which always works, and an equivalent single\-character Unicode symbol,+which may or may not render depending on your terminal and locale:+.TP+.B Intersection+.B /\e+(Unicode: the intersection symbol, U+2229)+(elements present in both operands)+.TP+.B Union+.B \e/+(Unicode: the union symbol, U+222A)+(elements present in either operand)+.TP+.B Difference+.B \-+(elements in the left operand that are not in the right operand; not commutative \(em+.I A+\-+.I B+is not the same as+.I B+\-+.IR A )+.TP+.B Symmetric difference+.B ><+(Unicode: the white up\-pointing small triangle, U+25B3)+(elements present in exactly one of the two operands; equivalent to+.RI ( A " \- " B ")" " \e/ " "(" B " \- " A ),+but computed in a single pass)+.PP+Intersection, union, and symmetric difference are commutative+.RI ( A " op " B+is the same as+.IR B " op " A ).+Difference is the only operator that is not.+.SS Bracketing and precedence+.B setdown+defines+.B no+operator precedence. An expression combining more than one operator without brackets, such as+.PP+.RS+.nf+def: A /\e B \e/ C+.fi+.RE+.PP+is a parse error, because there is no single unambiguous reading: it could mean+.RI "(" A " /\e " B ") \e/ " C+or+.RI "" A " /\e (" B " \e/ " C ")" ,+and substituting the empty set for+.I B+gives two genuinely different results depending on which reading is intended. Rather than pick one+silently,+.B setdown+requires every such expression to be bracketed explicitly:+.PP+.RS+.nf+def: (A /\e B) \e/ C+.fi+.RE+.SS Referencing other definitions+An expression's operand may be a quoted filename, another definition's name, or a bracketed+sub\-expression. Because a bare identifier is itself a valid expression, one definition may simply+alias another:+.PP+.RS+.nf+Combined: "a.txt" \e/ "b.txt"+Alias: Combined+.fi+.RE+.SS Comments+Everything from a double dash+.RB ( \-\- )+to the end of the line is a comment, wherever it appears on the line:+.PP+.RS+.nf+\-\- This is a definition for A, created because we wanted to do X+A: "y.txt" \- "z.txt"++B: (A \e/ C) \-\- \e/ D This is still a comment and \e/ D never happens+.fi+.RE+.SS Circular definitions+Definitions must not form a cycle:+.PP+.RS+.nf+A: "file.txt" \e/ B+B: A /\e "other.txt"+.fi+.RE+.PP+.B setdown+detects cycles like this before performing any set operations and exits with status 20 (see+\fBEXIT STATUS\fR), printing the cyclic chain of definition names it found.+.SS A complete example+.RS+.nf+\-\- A is the intersection of the file b\-1.out and the set B+A: "b\-1.out" /\e B++\-\- B is the union of the files a\-1.out and a\-2.out+B: "a\-1.out" \e/ "a\-2.out"++\-\- C is the difference of the file b\-1.out and the set B+C: "b\-1.out" \- B++\-\- D is the symmetric difference of two files+D: "a\-1.out" >< "a\-2.out"+.fi+.RE+.PP+Save this as, for example,+.IR mydefinitions.setdown ,+then run:+.PP+.RS+.nf+setdown \-\-input=mydefinitions.setdown+.fi+.RE+.SH OUTPUT+.B setdown+creates an output directory next to the \fB.setdown\fR file (named+.I output+by default; see \fB\-\-output\fR). Each definition produces one result file in that directory,+named after the definition with a+.I .txt+extension \(em a definition called+.I Overlap+produces+.IR output/Overlap.txt .+Every result file contains one element per line, sorted and de\-duplicated.+.PP+While evaluating sub\-expressions,+.B setdown+writes intermediate files to a scratch+.I output/processing/+subdirectory, which is removed automatically once the run finishes successfully. Pass+\fB\-\-keep\-processing\fR to leave it in place for debugging, and \fB\-\-show\-transient\fR to+have those intermediate results included in the summary table+.B setdown+prints at the end of a run.+.SH EXIT STATUS+.TP+.B 0+Success.+.TP+.B 1+The file given with \fB\-\-input\fR does not exist.+.TP+.B 2+More than one \fB.setdown\fR file was found in the current directory and no \fB\-\-input\fR was+given; use \fB\-\-input\fR to select one.+.TP+.B 3+No \fB.setdown\fR file was found in the current directory and no \fB\-\-input\fR was given; use+\fB\-\-input\fR to specify one.+.TP+.B 11+Two or more definitions in the file share the same name.+.TP+.B 12+A definition references an identifier that has not been defined anywhere in the file.+.TP+.B 13+One or more input files referenced by the definitions could not be found on disk.+.TP+.B 20+A cyclic dependency was detected between two or more definitions.+.PP+All file paths reported in error messages are relative to the \fB.setdown\fR file, not the+current working directory.+.SH EXAMPLES+Given two role files listing usernames, one per line:+.PP+.RS+.nf+$ cat admins.txt+alice+bob++$ cat developers.txt+bob+carol+.fi+.RE+.PP+and a \fB.setdown\fR file:+.PP+.RS+.nf+$ cat access.setdown+InternalStaff: "admins.txt" \e/ "developers.txt"+.fi+.RE+.PP+running+.B setdown+in that directory:+.PP+.RS+.nf+$ setdown+==> Using setdown file: access.setdown+\&...+$ cat output/InternalStaff.txt+alice+bob+carol+.fi+.RE+.PP+produces the union of the two files, sorted and de\-duplicated, in+.IR output/InternalStaff.txt .+More worked examples, covering every operator and several common use cases (access control,+API diffing, feature\-flag segmentation, dependency auditing), are distributed with the setdown+source under+.IR examples/ .+.SH FILES+.TP+.I *.setdown+A definitions file, as described in \fBTHE .setdown FILE FORMAT\fR above. Conventionally suffixed+.IR .setdown ,+though+.B setdown+does not require this except when auto\-detecting a file in the current directory (see+\fB\-\-input\fR).+.TP+.I output/+The default output directory; see \fBOUTPUT\fR.+.TP+.I output/processing/+Scratch space for intermediate results; see \fBOUTPUT\fR and \fB\-\-keep\-processing\fR.+.SH SEE ALSO+The setdown source repository, including the full set of worked examples referenced above, is at+.BR https://github.com/robertmassaioli/setdown .+.SH AUTHOR+Robert Massaioli <setdown@rmdir.app>+.SH REPORTING BUGS+Report bugs at+.BR https://github.com/robertmassaioli/setdown/issues .+.SH COPYRIGHT+Copyright \(co 2015 Robert Massaioli. Licensed under the 3\-clause BSD license; see the+.I LICENSE+file distributed with the source for the full text.
setdown.cabal view
@@ -10,7 +10,7 @@ -- PVP summary: +-+------- breaking API changes -- | | +----- non-breaking API additions -- | | | +--- code changes with no API change-version: 0.2.0.0+version: 0.2.1.0 -- A short (one-line) description of the package. synopsis: Treating files as sets to perform rapid set manipulation.@@ -23,8 +23,8 @@ built with the intention that you would use it in conjunction with version control tools to manage your set data and set description file. -homepage: http://bitbucket.org/robertmassaioli/setdown-bug-reports: https://bitbucket.org/robertmassaioli/setdown/issues+homepage: https://github.com/robertmassaioli/setdown+bug-reports: https://github.com/robertmassaioli/setdown/issues -- The license under which the package is released. license: BSD3@@ -37,7 +37,7 @@ -- An email address to which users can send suggestions, bug reports, and -- patches.-maintainer: robertmassaioli@gmail.com+maintainer: setdown@rmdir.app -- A copyright notice. copyright: (c) 2015 Robert Massaioli@@ -49,13 +49,14 @@ -- Extra files to be distributed with the package, such as examples or a -- README. extra-source-files: README.markdown+ , man/setdown.1 -- Constraint on the version of Cabal needed to build this package. cabal-version: >=1.10 source-repository head type: git- location: git@bitbucket.org:robertmassaioli/setdown.git+ location: https://github.com/robertmassaioli/setdown.git library exposed-modules: SetLanguage@@ -71,20 +72,21 @@ , SetInput , SetInputVerification , SimpleDefinitionCycles+ , TableRender build-depends: base >=4.7 && < 5 -- Lexing dependencies- , array >= 0.5 && < 0.6- , bytestring >= 0.10 && < 0.13- , text >= 1.2 && < 2.2+ , array >= 0.5 && < 1+ , bytestring >= 0.10 && < 1+ , text >= 1.2 && < 3 -- Module Dependencies , filepath >= 1.2 && < 3 , directory >= 1.1 && < 3- , containers >= 0.6 && < 0.8- , uuid >= 1.3 && < 1.4- , split == 0.2.*- , mtl >= 2.2 && < 2.4- , async >= 2.2 && < 2.3+ , containers >= 0.6 && < 1+ , uuid >= 1.3 && < 2+ , split >= 0.2 && < 1+ , mtl >= 2.2 && < 3+ , async >= 2.2 && < 3 build-tools: alex, happy @@ -99,16 +101,15 @@ build-depends: base >=4.7 && < 5 , setdown- , bytestring >= 0.10 && < 0.13- , text >= 1.2 && < 2.2- , containers >= 0.6 && < 0.8+ , bytestring >= 0.10 && < 1+ , text >= 1.2 && < 3+ , containers >= 0.6 && < 1 , directory >= 1.1 && < 3 , filepath >= 1.2 && < 3- , cmdargs >= 0.10 && < 0.11- , table-layout >= 0.8 && < 1.1+ , cmdargs >= 0.10 && < 1 if !os(windows)- build-depends: unix >= 2.7 && < 2.9+ build-depends: unix >= 2.7 && < 3 hs-source-dirs: app @@ -123,10 +124,10 @@ build-depends: base >=4.7 && < 5 , setdown- , tasty >= 1.4 && < 1.6- , tasty-hunit >= 0.10 && < 0.11- , text >= 1.2 && < 2.2- , bytestring >= 0.10 && < 0.13+ , tasty >= 1.4 && < 2+ , tasty-hunit >= 0.10 && < 1+ , text >= 1.2 && < 3+ , bytestring >= 0.10 && < 1 default-language: Haskell2010 @@ -139,10 +140,10 @@ build-depends: base >=4.7 && < 5 , setdown- , tasty >= 1.4 && < 1.6- , tasty-quickcheck >= 0.10 && < 0.12- , QuickCheck >= 2.14 && < 2.16- , text >= 1.2 && < 2.2+ , tasty >= 1.4 && < 2+ , tasty-quickcheck >= 0.10 && < 1+ , QuickCheck >= 2.14 && < 3+ , text >= 1.2 && < 3 default-language: Haskell2010 @@ -154,10 +155,11 @@ hs-source-dirs: test build-depends: base >=4.7 && < 5- , tasty >= 1.4 && < 1.6- , tasty-golden >= 2.3 && < 2.4+ , tasty >= 1.4 && < 2+ , tasty-golden >= 2.3 && < 3+ , tasty-hunit >= 0.10 && < 1 , filepath >= 1.2 && < 3- , process >= 1.6 && < 1.7+ , process >= 1.6 && < 2 default-language: Haskell2010
+ src/TableRender.hs view
@@ -0,0 +1,49 @@+module TableRender+ ( Align(..)+ , renderTable+ ) where++import Data.List (intercalate, transpose)++-- | How a column's cells are padded to the column width.+data Align = AlignLeft | AlignRight++-- | Render a header row and body rows as a Unicode box-drawing table: a bold+-- (heavy-lined) top border and header separator, single-lined body borders,+-- one alignment per column, and column widths sized to the widest cell+-- (including the header). This reproduces the exact visual style setdown has+-- always used for its results tables.+--+-- Column widths and the header row always account for the header text, even+-- when there are zero body rows, so headers never disappear on empty input.+renderTable :: [Align] -> [String] -> [[String]] -> [String]+renderTable aligns headers rows =+ [topBorder, headerRow, headerSeparator] ++ map bodyRow rows ++ [bottomBorder]+ where+ widths = zipWith columnWidth headers (columnsOf headers rows)+ columnWidth header column = maximum (length header : map length column)++ columnsOf hs [] = replicate (length hs) []+ columnsOf _ rs = transpose rs++ topBorder = border '┏' '┳' '┓' '━'+ headerSeparator = border '┡' '╇' '┩' '━'+ bottomBorder = border '└' '┴' '┘' '─'++ border left mid right line =+ [left] ++ intercalate [mid] (map (\w -> replicate (w + 2) line) widths) ++ [right]++ headerRow = rowOf '┃' (zipWith padCenter widths headers)+ bodyRow cells = rowOf '│' (zipWith3 padAlign widths aligns cells)++ rowOf sep cells =+ [sep] ++ intercalate [sep] (map (\c -> ' ' : c ++ " ") cells) ++ [sep]++ padCenter width s =+ let extra = width - length s+ left = extra `div` 2+ right = extra - left+ in replicate left ' ' ++ s ++ replicate right ' '++ padAlign width AlignLeft s = s ++ replicate (width - length s) ' '+ padAlign width AlignRight s = replicate (width - length s) ' ' ++ s
test/GoldenTests.hs view
@@ -2,8 +2,11 @@ import Test.Tasty import Test.Tasty.Golden-import System.FilePath ((</>))-import System.Process (callProcess)+import Test.Tasty.HUnit+import System.FilePath ((</>))+import System.Process (callProcess, readProcessWithExitCode)+import System.Exit (ExitCode(..))+import Data.List (isInfixOf) main :: IO () main = defaultMain tests@@ -14,6 +17,9 @@ , goldenTest "union" , goldenTest "difference" , goldenTest "symmetric-difference"+ , goldenTest "single-element-distinct"+ , goldenTest "single-element-identical"+ , errorDetectionTests ] -- | Run setdown on a fixture directory and compare the Result.txt output@@ -42,3 +48,31 @@ runSetdown :: FilePath -> IO () runSetdown inputFile = callProcess "stack" ["exec", "--", "setdown", "-i", inputFile]++-- ---------------------------------------------------------------------------+-- Error detection (CLI integration)+-- ---------------------------------------------------------------------------++-- | Each fixture lives under test/golden/errors/<name>/example.setdown and is+-- expected to make setdown exit with a specific failure code, printing a+-- message containing the given fragment. There is no golden output file+-- here, since setdown exits before writing any results.+errorDetectionTests :: TestTree+errorDetectionTests = testGroup "error detection"+ [ errorTest "duplicate-definition" 11 "Duplicate definitions found"+ , errorTest "unknown-identifier" 12 "Unknown identifiers used"+ , errorTest "missing-file" 13 "the following files could not be found"+ ]++errorTest :: String -> Int -> String -> TestTree+errorTest name expectedCode expectedFragment =+ testCase name $ do+ (exitCode, stdout, _stderr) <- readProcessWithExitCode "stack"+ ["exec", "--", "setdown", "-i", fixtureDir </> "example.setdown"]+ ""+ exitCode @?= ExitFailure expectedCode+ assertBool+ ("expected stdout to mention \"" ++ expectedFragment ++ "\", got:\n" ++ stdout)+ (expectedFragment `isInfixOf` stdout)+ where+ fixtureDir = "test" </> "golden" </> "errors" </> name
test/UnitTests.hs view
@@ -6,12 +6,16 @@ import qualified Data.Text.Lazy as T import qualified Data.ByteString.Lazy.Char8 as BC+import Data.List (isInfixOf)+import Control.Exception (SomeException, evaluate, try) import PerformOperations (linesSetOperation, operatorTools) import SetData import SimpleDefinitionCycles (getCyclesInSimpleDefinitions) import DuplicateElimination (eliminateDuplicates, orderDefinitions) import SetInput (parse)+import SetInputVerification (duplicateDefinitionName, unknownIdentifier)+import TableRender (Align (..), renderTable) main :: IO () main = defaultMain tests@@ -27,6 +31,9 @@ , cycleDetectionTests , duplicateEliminationTests , parseTests+ , parseErrorTests+ , verificationTests+ , tableRenderTests ] -- ---------------------------------------------------------------------------@@ -85,6 +92,10 @@ lso Or ["a", "b"] [] @?= ["a", "b"] , testCase "both empty → empty" $ lso Or [] [] @?= []+ , testCase "single distinct elements → both kept" $+ lso Or ["x"] ["y"] @?= ["x", "y"]+ , testCase "single identical elements → one kept" $+ lso Or ["x"] ["x"] @?= ["x"] ] -- ---------------------------------------------------------------------------@@ -106,6 +117,10 @@ , testCase "A - B ≠ B - A (not commutative)" $ do lso Difference ["a", "b"] ["a"] @?= ["b"] lso Difference ["a"] ["a", "b"] @?= []+ , testCase "single identical elements → empty" $+ lso Difference ["x"] ["x"] @?= []+ , testCase "single distinct elements → left unchanged" $+ lso Difference ["x"] ["y"] @?= ["x"] ] -- ---------------------------------------------------------------------------@@ -132,6 +147,10 @@ b = ["b", "c", "d"] in lso SymmetricDifference a b @?= lso Or (lso Difference a b) (lso Difference b a)+ , testCase "single identical elements → empty" $+ lso SymmetricDifference ["x"] ["x"] @?= []+ , testCase "single distinct elements → both kept" $+ lso SymmetricDifference ["x"] ["y"] @?= ["x", "y"] ] -- ---------------------------------------------------------------------------@@ -223,4 +242,179 @@ length (parse (BC.pack "A: (\"a.txt\" /\\ \"b.txt\")")) @?= 1 , testCase "comment is ignored" $ length (parse (BC.pack "-- just a comment\nA: \"a.txt\"")) @?= 1+ ]++-- ---------------------------------------------------------------------------+-- Parse errors+-- ---------------------------------------------------------------------------++-- | Force full evaluation of a parse so that any error thrown from within+-- the lazy parse tree is raised here, where we can catch it.+parseFailure :: String -> IO (Either SomeException String)+parseFailure input = try (evaluate (show (parse (BC.pack input))))++assertParseFailureContains :: String -> [String] -> IO ()+assertParseFailureContains input expectedFragments = do+ result <- parseFailure input+ case result of+ Left e ->+ let msg = show e+ in mapM_ (\frag -> assertBool+ ("expected \"" ++ frag ++ "\" in error message: " ++ msg)+ (frag `isInfixOf` msg))+ expectedFragments+ Right _ -> assertFailure "expected a parse error, but parsing succeeded"++parseErrorTests :: TestTree+parseErrorTests = testGroup "parse errors"+ [ testCase "two filenames with no operator reports line and column" $+ assertParseFailureContains+ "A: \"a.txt\" \"b.txt\""+ ["line 1", "column 12", "unexpected"]+ , testCase "unexpected token reports the correct line in multi-line input" $+ assertParseFailureContains+ "A: \"a.txt\"\nB: \"b.txt\" \"c.txt\""+ ["line 2", "column 12"]+ , testCase "unrecognised character is reported as a lexical error" $+ assertParseFailureContains+ "A: @"+ ["lexical error", "line 1", "column 4"]+ , testCase "unrecognised character on a later line reports that line" $+ assertParseFailureContains+ "A: \"a.txt\"\nB: \"b.txt\"\nC: @"+ ["lexical error", "line 3", "column 4"]+ ]++-- ---------------------------------------------------------------------------+-- Verification: duplicate names and unknown identifiers+-- ---------------------------------------------------------------------------++mkRawDef :: String -> Expression -> Definition+mkRawDef name expr = Definition (T.pack name) expr++verificationTests :: TestTree+verificationTests = testGroup "verification"+ [ testGroup "duplicate definition names"+ [ testCase "no duplicates → no errors" $+ duplicateDefinitionName+ [ mkRawDef "A" (FileExpression "a.txt")+ , mkRawDef "B" (FileExpression "b.txt")+ ]+ @?= []+ , testCase "one name defined twice → one error" $+ length (duplicateDefinitionName+ [ mkRawDef "A" (FileExpression "a.txt")+ , mkRawDef "A" (FileExpression "b.txt")+ ])+ @?= 1+ , testCase "one name defined three times → still one grouped error" $+ length (duplicateDefinitionName+ [ mkRawDef "A" (FileExpression "a.txt")+ , mkRawDef "A" (FileExpression "b.txt")+ , mkRawDef "A" (FileExpression "c.txt")+ ])+ @?= 1+ ]+ , testGroup "unknown identifiers"+ [ testCase "no references → no errors" $+ unknownIdentifier+ [ mkRawDef "A" (FileExpression "a.txt") ]+ @?= []+ , testCase "reference to a defined identifier → no errors" $+ unknownIdentifier+ [ mkRawDef "A" (FileExpression "a.txt")+ , mkRawDef "B" (IdentifierExpression "A")+ ]+ @?= []+ , testCase "reference to an undefined identifier → one error" $+ length (unknownIdentifier+ [ mkRawDef "A" (IdentifierExpression "B") ])+ @?= 1+ , testCase "undefined identifier used inside a binary expression → one error" $+ length (unknownIdentifier+ [ mkRawDef "A" (BinaryExpression And+ (FileExpression "a.txt")+ (IdentifierExpression "Missing"))+ ])+ @?= 1+ ]+ ]++-- ---------------------------------------------------------------------------+-- Table rendering+--+-- The expected outputs for "two left-aligned columns" and "left, left,+-- right" below were captured verbatim from setdown running against the+-- real, upstream table-layout library (columnHeaderTableS/unicodeBoldHeaderS)+-- before it was replaced by TableRender, to guarantee this reimplementation+-- is byte-for-byte compatible with the table style setdown has always used.+-- ---------------------------------------------------------------------------++tableRenderTests :: TestTree+tableRenderTests = testGroup "table rendering"+ [ testCase "two left-aligned columns, matches table-layout output exactly" $+ renderTable [AlignLeft, AlignLeft] ["From", "To"]+ [ ["api-v1.txt", "./output/api-v1.txt.1.split.sorted"]+ , ["api-v2.txt", "./output/api-v2.txt.1.split.sorted"]+ ]+ @?=+ [ "┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓"+ , "┃ From ┃ To ┃"+ , "┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩"+ , "│ api-v1.txt │ ./output/api-v1.txt.1.split.sorted │"+ , "│ api-v2.txt │ ./output/api-v2.txt.1.split.sorted │"+ , "└────────────┴────────────────────────────────────┘"+ ]+ , testCase "left, left, right columns, matches table-layout output exactly" $+ renderTable [AlignLeft, AlignLeft, AlignRight] ["Name", "File", "Count"]+ [ ["Added", "./output/Added.txt", "4"]+ , ["Removed", "./output/Removed.txt", "2"]+ ]+ @?=+ [ "┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┓"+ , "┃ Name ┃ File ┃ Count ┃"+ , "┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━┩"+ , "│ Added │ ./output/Added.txt │ 4 │"+ , "│ Removed │ ./output/Removed.txt │ 2 │"+ , "└─────────┴──────────────────────┴───────┘"+ ]+ , testCase "column width grows to fit a right-aligned multi-digit value" $+ renderTable [AlignLeft, AlignRight] ["Name", "Count"]+ [ ["AllDeps", "12"]+ , ["Shared", "7"]+ ]+ @?=+ [ "┏━━━━━━━━━┳━━━━━━━┓"+ , "┃ Name ┃ Count ┃"+ , "┡━━━━━━━━━╇━━━━━━━┩"+ , "│ AllDeps │ 12 │"+ , "│ Shared │ 7 │"+ , "└─────────┴───────┘"+ ]+ , testCase "cell narrower than its header is padded to the header width" $+ renderTable [AlignLeft, AlignLeft] ["From", "To"] [["a", "b"]]+ @?=+ [ "┏━━━━━━┳━━━━┓"+ , "┃ From ┃ To ┃"+ , "┡━━━━━━╇━━━━┩"+ , "│ a │ b │"+ , "└──────┴────┘"+ ]+ , testCase "empty string cell is padded like any other cell" $+ renderTable [AlignLeft, AlignLeft] ["From", "To"] [["", "x"]]+ @?=+ [ "┏━━━━━━┳━━━━┓"+ , "┃ From ┃ To ┃"+ , "┡━━━━━━╇━━━━┩"+ , "│ │ x │"+ , "└──────┴────┘"+ ]+ , testCase "no rows still renders headers (deliberately unlike upstream table-layout, which drops header text and collapses width to zero when the row list is empty)" $+ renderTable [AlignLeft, AlignLeft] ["From", "To"] []+ @?=+ [ "┏━━━━━━┳━━━━┓"+ , "┃ From ┃ To ┃"+ , "┡━━━━━━╇━━━━┩"+ , "└──────┴────┘"+ ] ]