hledger-flow 0.11.0.0 → 0.11.1.0

raw patch · 11 files changed

+755/−617 lines, 11 filesdep +optparse-applicativePVP: major bump suggested

API removals or changes: PVP suggests a major version bump

Dependencies added: optparse-applicative

API changes (from Hackage documentation)

Files

• ChangeLog.md +8/−0
• README.md +0/−578
• README.org +689/−0
• app/Main.hs +18/−9
• hledger-flow.cabal +7/−5
• src/Hledger/Flow/CSVImport.hs +5/−5
• src/Hledger/Flow/Common.hs +15/−13
• src/Hledger/Flow/Import/Types.hs +3/−3
• src/Hledger/Flow/Report/Types.hs +3/−3
• src/Hledger/Flow/Types.hs +6/−0
• test/TestHelpers.hs +1/−1

@@ -1,5 +1,13 @@ # Changelog for [hledger-flow](https://github.com/apauley/hledger-flow) +## 0.11.1++- Create statically linked executables on Linux - https://github.com/apauley/hledger-flow/releases+- Add an option to disable parallel processing+- Log the exit status of shell commands.+- Upgrade to LTS 13.16 for GHC 8.6.4.++ ## 0.11  - Change the name from `hledger-makeitso` to `hledger-flow`.

@@ -1,578 +0,0 @@-# Hledger Flow--# What is it?--`hledger-flow` is a command-line executable program that gives you a-guided hledger workflow. It focuses on automated processing of-electronic statements as much as possible, as opposed to manually adding-your own hledger journal entries.--Manual entries are still possible, we just think it saves time in the-long run to automatically process a statement whenever one is available.--It started when I realized that the scripts I wrote while playing around-with the ideas in [adept's Full-fledged Hledger](https://github.com/adept/full-fledged-hledger/wiki) isn't-really specific to my own finances, and can be shared.--# Overview of the Basic Workflow--1.  Save an input CSV file to a [specific directory](https://github.com/apauley/hledger-flow#input-files).-2.  Add an hledger [rules file](https://github.com/apauley/hledger-flow#rules-files).-    Include some classification rules if you want.-3.  Run `hledger-flow import`--Add all your files to your favourite version control system.--The generated journal that you most likely want to use as your-`LEDGER_FILE` is called `all-years.journal`. This has include directives-to all the automatically imported journals, as well as includes for your-own manually managed journal entries.--In a typical software project we don't add generated files to version-control, but in this case I think it is a good idea to add all the-generated files to version control as well - when you inevitably change-something, e.g. how you classify transactions in your rules file, then-you can easily see if your change had the desired effect by looking at a-diff.--# Who should use this?--`hledger-flow` is intended for you if:--  - You are interested in getting started with-    [hledger](http://hledger.org/) and you wouldn't mind pointers to the-    right docs along the way.-  - You want a way to organise your finances into a structure that will-    be maintainable over the long term.-  - You want to automate as much as possible when dealing with your-    financial life.-  - You don't mind writing some scripts when needed, as long as it saves-    you time over the long term.-  - You want the ability to model your entire financial life in one-    tool, as opposed to just the parts that some online tool currently-    supports.-  - You appreciate the fact that all your financial information stays-    within your control.--# How Stable is it?--We're not close to a 1.0 release yet, which means that we can still make-changes if needed.--That being said, some parts have been used and tested extensively and-are likely to remain stable. Have a look at the "Stability of this-Feature" sections in the feature reference below.--I add future work, ideas and thoughts as [Github-issues](https://github.com/apauley/hledger-flow/issues) and in-[TODO.org](TODO.org), so have a look there for more clues as to what may-likely change.--Let me know if you can think of some improvements.--# Detailed Step-By-Step Guide--Have a look at the [detailed step-by-step instructions](docs/README.org)-and the files in the [documentation directory](docs/).--For a visual overview, check out the slide show version of the same-step-by-step instructions:--<https://pauley.org.za/hledger-flow/>--You can see the example imported financial transactions as it was-generated by the step-by-step instructions here:--<https://github.com/apauley/hledger-flow-example-finances>--# After Cloning This Repository--This repository has some submodules included, mostly related to the-examples in the documentation.--You need to initialise and update the submodules:--``` bash-git submodule init-git submodule update-```--# Build Instructions--[![CircleCI](https://circleci.com/gh/apauley/hledger-flow.svg?style=svg)](https://circleci.com/gh/apauley/hledger-flow) [![Build Status](https://travis-ci.com/apauley/hledger-flow.svg?branch=master)](https://travis-ci.com/apauley/hledger-flow)--You need a recent version of-[stack](https://docs.haskellstack.org/en/stable/README/) installed.--Then run:--``` bash-stack test-stack install-```--Which should end with this:--``` org-Copied executables to ~/.local/bin:-- hledger-flow-```--Ensure that `${HOME}/.local/bin` is in your `PATH`.--Usually this means adding this to your `~/.bashrc`:--``` bash-PATH="${HOME}/.local/bin:${PATH}"-```--## Building with older Haskell Versions--To build using an older version of GHC and related dependencies, point-stack to one of the other yaml files:--``` bash-stack test --stack-yaml stack-8.4.4.yaml-stack test --stack-yaml stack-8.2.2.yaml-```--# Feature Reference--## Input Files--Your input files will probably be CSV files with a line for each-transaction, although other file types will work fine if you use a-`preprocess` or a `construct` script that can read them. These scripts-are explained later.--We mostly use conventions based on a predefined directory structure for-your input statements.--For example, assuming you have a `savings` account at `mybank`, you'll-put your first CSV statement here:-`import/john/mybank/savings/1-in/2018/123456789_2018-06-30.csv`.--Some people may want to include accounts belonging to their spouse as-part of the household finances:-`import/spouse/otherbank/checking/1-in/2018/987654321_2018-06-30.csv`.--### More About Input Files--All files and directories under the `import` directory is related to the-automatic importing and classification of transactions.--The directory directly under `import` is meant to indicate the owner or-custodian of the accounts below it. It mostly has an impact on-reporting. You may want to have separate reports for `import/mycompany`-and `import/personal`.--Below the directory for the owner we can indicate where an account is-held. For a bank account you may choose to name it `import/john/mybank`.--If your underground bunker filled with gold has CSV statements linked to-it, then you can absolutely create `import/john/secret-treasure-room`.--Under the directory for the financial institution, you'll have a-directory for each account at that institution, e.g.-`import/mycompany/bigbankinc/customer-deposits` and-`import/mycompany/bigbankinc/expense-account`.--Next you'll create a directory named `1-in`. This is to distinguish it-from `2-preprocessed` and `3-journal` which will be auto-generated-later.--Under `1-in` you'll create a directory for the year, e.g. `2018`, and-within that you can copy the statements for that year:-`import/john/mybank/savings/1-in/2018/123456789_2018-06-30.csv`--### Stability of this Feature--The basic owner/bank/account/year structure has been used and tested-fairly extensively, I don't expect a need for it to change.--I'm open to suggestions for improvement though.--## Rules Files--If your input file is in CSV format, or converted to CSV by your-`preprocess` script, then you'll need an [hledger rules-file](http://hledger.org/csv.html).--`hledger-flow` will try to find a rules file for each statement in a-few places. The same rules file is typically used for all statements of-a specific account, or even for all accounts of the same specific bank.--  - A global rules file for any `mybank` statement can be saved here:-    `import/mybank.rules`-  - A rules file for all statements of a specific account:-    `import/spouse/bigbankinc/savings/bigbankinc-savings.rules`--### Statement-specific Rules Files--What happens if some of the statements for an account has a different-format than the others?--This can happen if you normally get your statements directly from your-bank, but some statements you had to download from somewhere else, like-Mint, because your bank is being daft with older statements.--In order to tell `hledger-flow` that you want to override the rules-file for a specific statement, you need to add a suffix, separated by an-underscore (`_`) and starting with the letters `rfo` (rules file-override) to the filename of that statement.--For example: assuming you've named your statement-`99966633_20171223_1844_rfo-mint.csv`.--`hledger-flow` will look for a rules file named `rfo-mint.rules` in-the following places:--  - in the import directory, e.g. `import/rfo-mint.rules`-  - in the bank directory, e.g. `import/john/mybank/rfo-mint.rules`-  - in the account directory, e.g.-    `import/john/mybank/savings/rfo-mint.rules`--### Example rules file usage--A common scenario is multiple accounts that share the same file format,-but have different `account1` directives.--One possible approach would be to include a shared rules file in your-account-specific rules file.--If you are lucky enough that all statements at `mybank` share a common-format across all accounts, then you can `include` a rules file that-just defines the parts that are shared across accounts.--Two accounts at `mybank` may have rules files similar to these.--A checking account at mybank:--``` hledger-# Saved as: import/john/mybank/checking/mybank-checking.rules-include ../../../mybank-shared.rules-account1 Assets:Current:John:MyBank:Checking-```--Another account at mybank:--``` hledger-# Saved as: import/alice/mybank/savings/mybank-savings.rules-include ../../../mybank-shared.rules-account1 Assets:Current:Alice:MyBank:Savings-```--Where `import/mybank-shared.rules` may define some shared attributes:--``` hledger-skip 1--fields date, description, amount, balance--date-format %Y-%m-%d-currency $-```--Another possible approach could be to use your `preprocess` script to-write out a CSV file that has extra fields for `account1` and-`account2`.--You could then create the above mentioned global `import/mybank.rules`-with the fields defined more or less like this:--``` hledger-fields date, description, amount, balance, account1, account2-```--### Stability of this Feature--Rules files are a stable feature within [hledger](http://hledger.org/),-and we're just using the normal hledger rules files. The account, bank-and statement-specific rules files have been used and tested fairly-extensively, I don't expect this to change.--Let me know if you think it should change.--## Opening and Closing Balances--### Opening Balances--`hledger-flow` looks for a file named `YEAR-opening.journal` in each-account directory, where `YEAR` corresponds to an actual year directory,-eg. **1983** (if you have electronic statements [dating back-to 1983](https://en.wikipedia.org/wiki/Online_banking#First_online_banking_services_in_the_United_States)).-Example: `import/john/mybank/savings/1983-opening.journal`--If it exists the file will automatically be included at the beginning of-the generated journal include file for that year.--You need to edit this file for each account to specify the opening-balance at the date of the first available transaction.--An opening balance may look something like this:--``` hledger-2018-06-01 Savings Account Opening Balance-assets:Current:MyBank:Savings               $102.01-equity:Opening Balances:MyBank:Savings-```--### Closing Balances--Similar to opening balances, `hledger-flow` looks for an optional-file named `YEAR-closing.journal` in each account directory. Example:-`import/john/mybank/savings/1983-closing.journal`--If it exists the file will automatically be included at the end of the-generated journal include file for that year.--A closing balance may look something like this:--``` hledger-2018-06-01 Savings Account Closing Balance-assets:Current:MyBank:Savings               $-234.56 = $0.00-equity:Closing Balances:MyBank:Savings-```--### Example Opening and Closing Journal Files--As an example, assuming that the relevant year is `2019` and-`hledger-flow` is about to generate-`import/john/mybank/savings/2019-include.journal`, then one or both of-the following files will be added to the include file if they exist:--1.  `import/john/mybank/savings/2019-opening.journal`-2.  `import/john/mybank/savings/2019-closing.journal`--The `opening.journal` will be included just before the other included-entries, while the `closing.journal` will be included just after the-other entries in that include file.--An include file may look like this:--``` bash-cat import/john/mybank/savings/2019-include.journal-```--``` hledger-### Generated by hledger-flow - DO NOT EDIT ###--!include 2019-opening.journal-!include 3-journal/2019/123456789_2019-01-30-!include 2019-closing.journal-```--### Stability of this Feature--The opening balances file works well in my opinion, I don't expect it to-change. I'm only using closing balances in one or two places, so maybe-that could do with some suggestions from people who use this more than-myself.--## The `preprocess` Script--Sometimes the statements you get from your bank is [less than-suitable](https://github.com/apauley/fnb-csv-demoronizer) for automatic-processing. Or maybe you just want to make it easier for the hledger-rules file to do its thing by adding some useful columns.--If you put a script called `preprocess` in the account directory, e.g.-`import/john/mybank/savings/preprocess`, then `hledger-flow` will-call that script for each input statement.--The `preprocess` script will be called with 4 positional parameters:--1.  The path to the input statement, e.g.-    `import/john/mybank/savings/1-in/2018/123456789_2018-06-30.csv`-2.  The path to an output file that can be sent to `hledger`, e.g.-    `import/john/mybank/savings/2-preprocessed/2018/123456789_2018-06-30.csv`-3.  The name of the bank, e.g. `mybank`-4.  The name of the account, e.g. `savings`-5.  The name of the owner, e.g. `john`--Your `preprocess` script is expected to:--  - read the input file-  - write a new output file at the supplied path that works with your-    rules file-  - be idempotent. Running `preprocess` multiple times on the same files-    will produce the same result.--### Stability of this Feature--Stable and tested.--## The `construct` Script--If you need even more power and flexibility than what you can get from-the `preprocess` script and `hledger`'s CSV import functionality, then-you can create your own custom script to `construct` transactions-exactly as you need them.--At the expense of more construction work for you, of course.--As an example, `hledger`'s CSV import currently [only supports two-postings per-transaction](https://github.com/simonmichael/hledger/issues/627), even-though `hledger` itself is perfectly happy with transactions containing-more than two postings, e.g.:--``` hledger-2019-02-01 Mortgage Payment-Liabilities:Mortgage                                $1000.00-Expenses:Interest:Real Estate                         $833.33-Assets:Cash                                         -$1833.33-```--The `construct` script can be used in addition to the `preprocess`-script, or on it's own. But since the `construct` script is more-powerful than the `preprocess` script, you could tell your `construct`-script to do anything that the `preprocess` script would have done.--Save your `construct` script in the account directory, e.g.-`import/john/mybank/savings/construct`.--`hledger-flow` will call your `construct` script with 4 positional-parameters:--1.  The path to the input statement, e.g.-    `import/john/mybank/savings/1-in/2018/123456789_2018-06-30.csv`-2.  A "-" (indicating that output should be sent to `stdout`)-3.  The name of the bank, e.g. `mybank`-4.  The name of the account, e.g. `savings`-5.  The name of the owner, e.g. `john`--Your `construct` script is expected to:--  - read the input file-  - generate your own `hledger` journal transactions-  - be idempotent. Running `construct` multiple times on the same files-    should produce the same result.-  - send all output to `stdout`. `hledger-flow` will pipe your-    output into `hledger` which will format it and save it to an output-    file.--### Stability of this Feature--Stable and tested.--## Manually Managed Journals--Not every transaction in your life comes with CSV statements.--Sometimes you just need to add a transaction for that time you loaned a-friend some money.--`hledger-flow` looks for `pre-import` and `post-import` files-related to each generated include file as part of the import.--You can enter your own transactions manually into these files.--You can run `hledger-flow import --verbose` to see exactly which-files are being looked for.--As an example, assuming that the relevant year is `2019` and-`hledger-flow` is about to generate-`import/john/2019-include.journal`, then one or both of the following-files will be added to the include file if they exist:--1.  `import/john/_manual_/2019/pre-import.journal`-2.  `import/john/_manual_/2019/post-import.journal`--The `pre-import.journal` will be included just before the other included-entries, while the `post-import.journal` will be included just after the-other entries in that include file.--An include file may look like this:--``` bash-cat import/john/2019-include.journal-```--``` hledger-### Generated by hledger-flow - DO NOT EDIT ###--!include _manual_/2019/pre-import.journal-!include mybank/2019-include.journal-!include otherbank/2019-include.journal-!include _manual_/2019/post-import.journal-```--### Stability of this Feature--It works, but the naming of `_manual_` looks a bit weird. Should it be-changed?--# Compatibility with Ledger--When writing out the journal include files, `hledger-flow` sorts the-include statements by filename.--[Ledger](https://www.ledger-cli.org/) fails any balance assertions when-the transactions aren't included in chronological order.--An easy way around this is to name your input files so that March's-statement is listed before December's statement.--Another option is to add `--permissive` to any-[ledger](https://www.ledger-cli.org/) command.--So you should easily be able to use both `ledger` and `hledger` on these-journals.--# Project Goals--My `hledger` files started to collect a bunch of supporting code that-weren't really specific to my financial situation.--I want to extract and share as much as possible of that supporting code.--[Adept's](https://github.com/adept/full-fledged-hledger/wiki) goals also-resonated with me:--  - Tracking expenses should take as little time, effort and manual work-    as possible-  - Eventual consistency should be achievable: even if I can't record-    something precisely right now, maybe I would be able to do it later,-    so I should be able to leave things half-done and pick them up later-  - Ability to refactor is a must. I want to be able to go back and-    change the way I am doing things, with as little effort as possible-    and without fear of irrevocably breaking things.--I've given [a talk](https://pauley.org.za/functional-finance-hledger/)-at [Lambda Luminaries-Johannesburg](https://www.meetup.com/lambda-luminaries/events/qklkvpyxmbnb/)-featuring hledger and hledger-flow.--# FAQ--## How does `hledger-flow` differ from `Full-fledged Hledger`?--[Full-fledged Hledger](https://github.com/adept/full-fledged-hledger/wiki#full-fledged-hledger-tutorial)-is a brilliant system, and hledger-flow continues to learn much from-it.--It has great documentation that does an excellent job of not only-showing **how** things can be done, but also **why** it is such a great-idea.--hledger-flow can be seen as a specific implementation of the-Full-fledged Hledger system, with a few implementation details that are-different.--| Full-fledged Hledger                                                                                                                                                                                                       | Hledger: Make It So                                                                                                                                                                                                                                |-| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |-|                                                                                                                                                                                                                            |                                                                                                                                                                                                                                                    |-| FFH describes itself as a [tutorial](https://github.com/adept/full-fledged-hledger/wiki#full-fledged-hledger-tutorial) with helper scripts that you can start using and adapt to your needs.                               | I started by following the FFH tutorial, and changed bits and pieces over time to suit my needs. The "owner/bank/account" structure for example.                                                                                                   |-|                                                                                                                                                                                                                            |                                                                                                                                                                                                                                                    |-| FFH is more open-ended: you can start with the basic scripts and over time turn it into something that solves your needs exactly. But you'll also end up with more code that you need to maintain yourself.                | Hledger Flow is less open-ended. For example, you have to adopt the "owner/bank/account" structure precisely as specified. But this allows Hledger Flow to do more work for you.                                                                                   |-|                                                                                                                                                                                                                            |                                                                                                                                                                                                                                                    |-| Maintaining include files are currently part of the user's responsibility.                                                                                                                                                 | Hledger Flow generates flexible include files for you, and automatically includes opening/closing journals if the appropriately named files are present on disk.                                                                                           |-|                                                                                                                                                                                                                            |                                                                                                                                                                                                                                                    |-| FFH actually generates some useful reports right now. Hledger Flow still [plans to get this done](https://github.com/apauley/hledger-flow/pull/4) one day.                                                                     | The "owner/bank/account" structure may look a bit much at first, but it allows us to run separate queries/reports for me/my spouse/my business etc and also allows for reports covering all of it in a an overall view.                            |-|                                                                                                                                                                                                                            |                                                                                                                                                                                                                                                    |-| FFH chose scripts and build files that you can easily modify as you go along, but this requires a Haskell runtime to be installed everywhere it needs to run. The included docker image helps to make it less of an issue. | Hledger Flow distributes a compiled binary. This means users or deployment targets don't need extra dependencies installed, they can just run a CLI program. This also provides a clearer distinction between what is provided, and what users need to do. |-|                                                                                                                                                                                                                            |                                                                                                                                                                                                                                                    |-| The FFH build scripts requires familiarity with Haskell and the Shake build system.                                                                                                                                        | Users may need to write `preprocess` or `construct` hooks, but in a language of their choice.                                                                                                                                                      |-|                                                                                                                                                                                                                            |                                                                                                                                                                                                                                                    |-| Input files are assumed to always be CSV files.                                                                                                                                                                            | Hledger Flow de-empasises the need that input files must be in CSV format. Input files can be in any format that a `preprocess` or `construct` hook can read.                                                                                              |

@@ -0,0 +1,689 @@+#+STARTUP: showall+#+PROPERTY: header-args:sh :prologue exec 2>&1 :epilogue echo :++* Hledger Flow+  :PROPERTIES:+  :CUSTOM_ID: hledger-flow+  :END:++| TravisCI                                                     | CircleCI                                                   |+|--------------------------------------------------------------+------------------------------------------------------------|+| [[https://travis-ci.org/apauley/hledger-flow][https://travis-ci.org/apauley/hledger-flow.svg?branch=master]] | [[https://circleci.com/gh/apauley/hledger-flow][https://circleci.com/gh/apauley/hledger-flow.svg?style=svg]] |++* What is it?+  :PROPERTIES:+  :CUSTOM_ID: what-is-it+  :END:++=hledger-flow= is a command-line program that gives you a guided+[[https://hledger.org/][Hledger]] workflow. It focuses on automated+processing of electronic statements as much as possible, as opposed to+manually adding your own hledger journal entries.++Manual entries are still possible, we just think it saves time in the+long run to automatically process a statement whenever one is available.++Within =hledger-flow= you will keep your original bank statements around+permanently as input, and generate classified Hledger journals each time+you run the program. The classification is done with rules files, or+your own script hooks.++Keeping the original statements means that you never have to worry too+much about "am I doing this accounting thing right?" or "what happens if+I make a mistake?". If you want to change your mind about some+classification, or if you made a mistake, you just change your+classification rules, and run the program again.++It started when I realized that the scripts I wrote while playing around+with the ideas in [[https://github.com/adept/full-fledged-hledger/wiki][adept's Full-fledged Hledger]] aren't really specific to+my own finances, and can be shared.++* Overview of the Basic Workflow+  :PROPERTIES:+  :CUSTOM_ID: overview-of-the-basic-workflow+  :END:++1. Save an input transaction file (typically CSV) to a [[#input-files][specific directory]].+2. Add an hledger [[#rules-files][rules file]].+   Include some classification rules if you want.+3. Run =hledger-flow import=++Add all your files to your favourite version control system.++The generated journal that you most likely want to use as your+=LEDGER_FILE= is called =all-years.journal=. This has include directives+to all the automatically imported journals, as well as includes for your+own manually managed journal entries.++In a typical software project we don't add generated files to version+control, but in this case I think it is a good idea to add all the+generated files to version control as well - when you inevitably change+something, e.g. how you classify transactions in your rules file, then+you can easily see if your change had the desired effect by looking at a+diff.++* Who should use this?+  :PROPERTIES:+  :CUSTOM_ID: who-should-use-this+  :END:++=hledger-flow= is intended for you if:++- You are interested in getting started with+  [[http://hledger.org/][hledger]] and you wouldn't mind pointers to the+  right docs along the way.+- You want a way to organise your finances into a structure that will be+  maintainable over the long term.+- You want to automate as much as possible when dealing with your+  financial life.+- You don't mind writing some scripts when needed, as long as it saves+  you time over the long term.+- You want the ability to model your entire financial life in one tool,+  as opposed to just the parts that some online tool currently supports.+- You appreciate the fact that all your financial information stays+  within your control.++* How Stable is it?+  :PROPERTIES:+  :CUSTOM_ID: how-stable-is-it+  :END:++We're not close to a 1.0 release yet, which means that we can still make+changes if needed.++That being said, some parts have been used and tested extensively and+are likely to remain stable. Have a look at the "Stability of this+Feature" sections in the feature reference below.++I add future work, ideas and thoughts as+[[https://github.com/apauley/hledger-flow/issues][Github issues]] and in+[[file:TODO.org]], so have a look there for more clues as to what may+likely change.++Let me know if you can think of some improvements.++* Detailed Step-By-Step Guide+  :PROPERTIES:+  :CUSTOM_ID: detailed-step-by-step-guide+  :END:++Have a look at the [[file:docs/README.org][detailed step-by-step+instructions]].++For a visual overview, check out the slide show version of the same+step-by-step instructions:++[[https://pauley.org.za/hledger-flow/][https://pauley.org.za/hledger-flow/]]++You can see the example imported financial transactions as it was+generated by the step-by-step instructions here:++[[https://github.com/apauley/hledger-flow-example-finances][https://github.com/apauley/hledger-flow-example-finances]]++* After Cloning This Repository+  :PROPERTIES:+  :CUSTOM_ID: after-cloning-this-repository+  :END:++This repository has some submodules included, mostly related to the+examples in the documentation.++You need to initialise and update the submodules:++#+BEGIN_SRC sh+    git submodule init+    git submodule update+#+END_SRC++* Build Instructions+  :PROPERTIES:+  :CUSTOM_ID: build-instructions+  :END:++You need a recent version of [[https://docs.haskellstack.org/en/stable/README/][stack]] installed.++Then run:++#+NAME: stack-build+#+BEGIN_SRC sh :results none :exports both+    stack test+    stack install+#+END_SRC++Which should end with this:++#+BEGIN_SRC org+    Copied executables to ~/.local/bin:+    - hledger-flow+#+END_SRC++Ensure that =${HOME}/.local/bin= is in your =PATH=.++Usually this means adding this to your =~/.bashrc=:++#+BEGIN_SRC sh+    PATH="${HOME}/.local/bin:${PATH}"+#+END_SRC++** Building with older Haskell Versions+   :PROPERTIES:+   :CUSTOM_ID: building-with-older-haskell-versions+   :END:++To build using an older version of GHC and related dependencies, point+stack to one of the other yaml files:++#+NAME: stack-build-versions+#+BEGIN_SRC sh :results none :exports both+    stack test --stack-yaml stack-8.4.4.yaml+    stack test --stack-yaml stack-8.2.2.yaml+#+END_SRC++* Feature Reference+  :PROPERTIES:+  :CUSTOM_ID: feature-reference+  :END:++** Input Files+   :PROPERTIES:+   :CUSTOM_ID: input-files+   :END:++Your input files will probably be CSV files with a line for each+transaction, although other file types will work fine if you use a+=preprocess= or a =construct= script that can read them. These scripts+are explained later.++We mostly use conventions based on a predefined directory structure for+your input statements.++For example, assuming you have a =savings= account at =mybank=, you'll+put your first CSV statement here:+=import/john/mybank/savings/1-in/2018/123456789_2018-06-30.csv=.++Some people may want to include accounts belonging to their spouse as+part of the household finances:+=import/spouse/otherbank/checking/1-in/2018/987654321_2018-06-30.csv=.++*** More About Input Files+    :PROPERTIES:+    :CUSTOM_ID: more-about-input-files+    :END:++All files and directories under the =import= directory is related to the+automatic importing and classification of transactions.++The directory directly under =import= is meant to indicate the owner or+custodian of the accounts below it. It mostly has an impact on+reporting. You may want to have separate reports for =import/mycompany=+and =import/personal=.++Below the directory for the owner we can indicate where an account is+held. For a bank account you may choose to name it =import/john/mybank=.++If your underground bunker filled with gold has CSV statements linked to+it, then you can absolutely create =import/john/secret-treasure-room=.++Under the directory for the financial institution, you'll have a+directory for each account at that institution, e.g.+=import/mycompany/bigbankinc/customer-deposits= and+=import/mycompany/bigbankinc/expense-account=.++Next you'll create a directory named =1-in=. This is to distinguish it+from =2-preprocessed= and =3-journal= which will be auto-generated+later.++Under =1-in= you'll create a directory for the year, e.g. =2018=, and+within that you can copy the statements for that year:+=import/john/mybank/savings/1-in/2018/123456789_2018-06-30.csv=++*** Stability of this Feature+    :PROPERTIES:+    :CUSTOM_ID: stability-of-this-feature+    :END:++The basic owner/bank/account/year structure has been used and tested+fairly extensively, I don't expect a need for it to change.++I'm open to suggestions for improvement though.++** Rules Files+   :PROPERTIES:+   :CUSTOM_ID: rules-files+   :END:++If your input file is in CSV format, or converted to CSV by your+=preprocess= script, then you'll need an+[[http://hledger.org/csv.html][hledger rules file]].++=hledger-flow= will try to find a rules file for each statement in a few+places. The same rules file is typically used for all statements of a+specific account, or even for all accounts of the same specific bank.++- A global rules file for any =mybank= statement can be saved here:+  =import/mybank.rules=+- A rules file for all statements of a specific account:+  =import/spouse/bigbankinc/savings/bigbankinc-savings.rules=++*** Statement-specific Rules Files+    :PROPERTIES:+    :CUSTOM_ID: statement-specific-rules-files+    :END:++What happens if some of the statements for an account has a different+format than the others?++This can happen if you normally get your statements directly from your+bank, but some statements you had to download from somewhere else, like+Mint, because your bank is being daft with older statements.++In order to tell =hledger-flow= that you want to override the rules file+for a specific statement, you need to add a suffix, separated by an+underscore (=_=) and starting with the letters =rfo= (rules file+override) to the filename of that statement.++For example: assuming you've named your statement+=99966633_20171223_1844_rfo-mint.csv=.++=hledger-flow= will look for a rules file named =rfo-mint.rules= in the+following places:++- in the import directory, e.g. =import/rfo-mint.rules=+- in the bank directory, e.g. =import/john/mybank/rfo-mint.rules=+- in the account directory, e.g.+  =import/john/mybank/savings/rfo-mint.rules=++*** Example rules file usage+    :PROPERTIES:+    :CUSTOM_ID: example-rules-file-usage+    :END:++A common scenario is multiple accounts that share the same file format,+but have different =account1= directives.++One possible approach would be to include a shared rules file in your+account-specific rules file.++If you are lucky enough that all statements at =mybank= share a common+format across all accounts, then you can =include= a rules file that+just defines the parts that are shared across accounts.++Two accounts at =mybank= may have rules files similar to these.++A checking account at mybank:++#+BEGIN_EXAMPLE+    # Saved as: import/john/mybank/checking/mybank-checking.rules+    include ../../../mybank-shared.rules+    account1 Assets:Current:John:MyBank:Checking+#+END_EXAMPLE++Another account at mybank:++#+BEGIN_EXAMPLE+    # Saved as: import/alice/mybank/savings/mybank-savings.rules+    include ../../../mybank-shared.rules+    account1 Assets:Current:Alice:MyBank:Savings+#+END_EXAMPLE++Where =import/mybank-shared.rules= may define some shared attributes:++#+BEGIN_EXAMPLE+    skip 1++    fields date, description, amount, balance++    date-format %Y-%m-%d+    currency $+#+END_EXAMPLE++Another possible approach could be to use your =preprocess= script to+write out a CSV file that has extra fields for =account1= and+=account2=.++You could then create the above mentioned global =import/mybank.rules=+with the fields defined more or less like this:++#+BEGIN_EXAMPLE+    fields date, description, amount, balance, account1, account2+#+END_EXAMPLE++*** Stability of this Feature+    :PROPERTIES:+    :CUSTOM_ID: stability-of-this-feature-1+    :END:++Rules files are a stable feature within+[[http://hledger.org/][hledger]], and we're just using the normal+hledger rules files. The account, bank and statement-specific rules+files have been used and tested fairly extensively, I don't expect this+to change.++Let me know if you think it should change.++** Opening and Closing Balances+   :PROPERTIES:+   :CUSTOM_ID: opening-and-closing-balances+   :END:++*** Opening Balances+    :PROPERTIES:+    :CUSTOM_ID: opening-balances+    :END:++=hledger-flow= looks for a file named =YEAR-opening.journal= in each+account directory, where =YEAR= corresponds to an actual year directory,+eg. *1983* (if you have electronic statements+[[https://en.wikipedia.org/wiki/Online_banking#First_online_banking_services_in_the_United_States][dating+back to 1983]]). Example:+=import/john/mybank/savings/1983-opening.journal=++If it exists the file will automatically be included at the beginning of+the generated journal include file for that year.++You need to edit this file for each account to specify the opening+balance at the date of the first available transaction.++An opening balance may look something like this:++#+BEGIN_EXAMPLE+    2018-06-01 Savings Account Opening Balance+    assets:Current:MyBank:Savings               $102.01+    equity:Opening Balances:MyBank:Savings+#+END_EXAMPLE++*** Closing Balances+    :PROPERTIES:+    :CUSTOM_ID: closing-balances+    :END:++Similar to opening balances, =hledger-flow= looks for an optional file+named =YEAR-closing.journal= in each account directory. Example:+=import/john/mybank/savings/1983-closing.journal=++If it exists the file will automatically be included at the end of the+generated journal include file for that year.++A closing balance may look something like this:++#+BEGIN_EXAMPLE+    2018-06-01 Savings Account Closing Balance+    assets:Current:MyBank:Savings               $-234.56 = $0.00+    equity:Closing Balances:MyBank:Savings+#+END_EXAMPLE++*** Example Opening and Closing Journal Files+    :PROPERTIES:+    :CUSTOM_ID: example-opening-and-closing-journal-files+    :END:++As an example, assuming that the relevant year is =2019= and+=hledger-flow= is about to generate+=import/john/mybank/savings/2019-include.journal=, then one or both of+the following files will be added to the include file if they exist:++1. =import/john/mybank/savings/2019-opening.journal=+2. =import/john/mybank/savings/2019-closing.journal=++The =opening.journal= will be included just before the other included+entries, while the =closing.journal= will be included just after the+other entries in that include file.++An include file may look like this:++#+BEGIN_SRC sh+    cat import/john/mybank/savings/2019-include.journal+#+END_SRC++#+BEGIN_EXAMPLE+    ### Generated by hledger-flow - DO NOT EDIT ###++    !include 2019-opening.journal+    !include 3-journal/2019/123456789_2019-01-30+    !include 2019-closing.journal+#+END_EXAMPLE++*** Stability of this Feature+    :PROPERTIES:+    :CUSTOM_ID: stability-of-this-feature-2+    :END:++The opening balances file works well in my opinion, I don't expect it to+change. I'm only using closing balances in one or two places, so maybe+that could do with some suggestions from people who use this more than+myself.++** The =preprocess= Script+   :PROPERTIES:+   :CUSTOM_ID: the-preprocess-script+   :END:++Sometimes the statements you get from your bank is+[[https://github.com/apauley/fnb-csv-demoronizer][less than suitable]]+for automatic processing. Or maybe you just want to make it easier for+the hledger rules file to do its thing by adding some useful columns.++If you put a script called =preprocess= in the account directory, e.g.+=import/john/mybank/savings/preprocess=, then =hledger-flow= will call+that script for each input statement.++The =preprocess= script will be called with 4 positional parameters:++1. The path to the input statement, e.g.+   =import/john/mybank/savings/1-in/2018/123456789_2018-06-30.csv=+2. The path to an output file that can be sent to =hledger=, e.g.+   =import/john/mybank/savings/2-preprocessed/2018/123456789_2018-06-30.csv=+3. The name of the bank, e.g. =mybank=+4. The name of the account, e.g. =savings=+5. The name of the owner, e.g. =john=++Your =preprocess= script is expected to:++- read the input file+- write a new output file at the supplied path that works with your+  rules file+- be idempotent. Running =preprocess= multiple times on the same files+  will produce the same result.++*** Stability of this Feature+    :PROPERTIES:+    :CUSTOM_ID: stability-of-this-feature-3+    :END:++Stable and tested.++** The =construct= Script+   :PROPERTIES:+   :CUSTOM_ID: the-construct-script+   :END:++If you need even more power and flexibility than what you can get from+the =preprocess= script and =hledger='s CSV import functionality, then+you can create your own custom script to =construct= transactions+exactly as you need them.++At the expense of more construction work for you, of course.++As an example, =hledger='s CSV import currently+[[https://github.com/simonmichael/hledger/issues/627][only supports two+postings per transaction]], even though =hledger= itself is perfectly+happy with transactions containing more than two postings, e.g.:++#+BEGIN_EXAMPLE+    2019-02-01 Mortgage Payment+    Liabilities:Mortgage                                $1000.00+    Expenses:Interest:Real Estate                         $833.33+    Assets:Cash                                         -$1833.33+#+END_EXAMPLE++The =construct= script can be used in addition to the =preprocess=+script, or on it's own. But since the =construct= script is more+powerful than the =preprocess= script, you could tell your =construct=+script to do anything that the =preprocess= script would have done.++Save your =construct= script in the account directory, e.g.+=import/john/mybank/savings/construct=.++=hledger-flow= will call your =construct= script with 4 positional+parameters:++1. The path to the input statement, e.g.+   =import/john/mybank/savings/1-in/2018/123456789_2018-06-30.csv=+2. A "-" (indicating that output should be sent to =stdout=)+3. The name of the bank, e.g. =mybank=+4. The name of the account, e.g. =savings=+5. The name of the owner, e.g. =john=++Your =construct= script is expected to:++- read the input file+- generate your own =hledger= journal transactions+- be idempotent. Running =construct= multiple times on the same files+  should produce the same result.+- send all output to =stdout=. =hledger-flow= will pipe your output into+  =hledger= which will format it and save it to an output file.++*** Stability of this Feature+    :PROPERTIES:+    :CUSTOM_ID: stability-of-this-feature-4+    :END:++Stable and tested.++** Manually Managed Journals+   :PROPERTIES:+   :CUSTOM_ID: manually-managed-journals+   :END:++Not every transaction in your life comes with CSV statements.++Sometimes you just need to add a transaction for that time you loaned a+friend some money.++=hledger-flow= looks for =pre-import= and =post-import= files related to+each generated include file as part of the import.++You can enter your own transactions manually into these files.++You can run =hledger-flow import --verbose= to see exactly which files+are being looked for.++As an example, assuming that the relevant year is =2019= and+=hledger-flow= is about to generate =import/john/2019-include.journal=,+then one or both of the following files will be added to the include+file if they exist:++1. =import/john/_manual_/2019/pre-import.journal=+2. =import/john/_manual_/2019/post-import.journal=++The =pre-import.journal= will be included just before the other included+entries, while the =post-import.journal= will be included just after the+other entries in that include file.++An include file may look like this:++#+BEGIN_SRC sh+    cat import/john/2019-include.journal+#+END_SRC++#+BEGIN_EXAMPLE+    ### Generated by hledger-flow - DO NOT EDIT ###++    !include _manual_/2019/pre-import.journal+    !include mybank/2019-include.journal+    !include otherbank/2019-include.journal+    !include _manual_/2019/post-import.journal+#+END_EXAMPLE++*** Stability of this Feature+    :PROPERTIES:+    :CUSTOM_ID: stability-of-this-feature-5+    :END:++It works, but the naming of =_manual_= looks a bit weird. Should it be+changed?++* Compatibility with Ledger+  :PROPERTIES:+  :CUSTOM_ID: compatibility-with-ledger+  :END:++When writing out the journal include files, =hledger-flow= sorts the+include statements by filename.++[[https://www.ledger-cli.org/][Ledger]] fails any balance assertions+when the transactions aren't included in chronological order.++An easy way around this is to name your input files so that March's+statement is listed before December's statement.++Another option is to add =--permissive= to any+[[https://www.ledger-cli.org/][ledger]] command.++So you should easily be able to use both =ledger= and =hledger= on these+journals.++* Project Goals+  :PROPERTIES:+  :CUSTOM_ID: project-goals+  :END:++My =hledger= files started to collect a bunch of supporting code that+weren't really specific to my financial situation.++I want to extract and share as much as possible of that supporting code.++[[https://github.com/adept/full-fledged-hledger/wiki][Adept's]] goals+also resonated with me:++- Tracking expenses should take as little time, effort and manual work+  as possible+- Eventual consistency should be achievable: even if I can't record+  something precisely right now, maybe I would be able to do it later,+  so I should be able to leave things half-done and pick them up later+- Ability to refactor is a must. I want to be able to go back and change+  the way I am doing things, with as little effort as possible and+  without fear of irrevocably breaking things.++I've given [[https://pauley.org.za/functional-finance-hledger/][a talk]]+at+[[https://www.meetup.com/lambda-luminaries/events/qklkvpyxmbnb/][Lambda+Luminaries Johannesburg]] featuring hledger and hledger-flow.++* FAQ+  :PROPERTIES:+  :CUSTOM_ID: faq+  :END:++** How does =hledger-flow= differ from =Full-fledged Hledger=?+   :PROPERTIES:+   :CUSTOM_ID: how-does-hledger-flow-differ-from-full-fledged-hledger+   :END:++[[https://github.com/adept/full-fledged-hledger/wiki#full-fledged-hledger-tutorial][Full-fledged+Hledger]] is a brilliant system, and hledger-flow continues to learn+much from it.++It has great documentation that does an excellent job of not only+showing *how* things can be done, but also *why* it is such a great+idea.++hledger-flow can be seen as a specific implementation of the+Full-fledged Hledger system, with a few implementation details that are+different.++| Full-fledged Hledger                                                                                                                                                                                                       | Hledger Flow                                                                                                                                                                                                                                               |+|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|+| FFH describes itself as a [[https://github.com/adept/full-fledged-hledger/wiki#full-fledged-hledger-tutorial][tutorial]] with helper scripts that you can start using and adapt to your needs.                                                                                                                   | I started by following the FFH tutorial, and changed bits and pieces over time to suit my needs. The "owner/bank/account" structure for example.                                                                                                           |+|                                                                                                                                                                                                                            |                                                                                                                                                                                                                                                            |+| FFH is more open-ended: you can start with the basic scripts and over time turn it into something that solves your needs exactly. But you'll also end up with more code that you need to maintain yourself.                | Hledger Flow is more opinionated and less open-ended. For example, you have to adopt the "owner/bank/account" structure precisely as specified. But this allows Hledger Flow to do more work for you.                                                      |+|                                                                                                                                                                                                                            |                                                                                                                                                                                                                                                            |+| Maintaining include files are currently part of the user's responsibility.                                                                                                                                                 | Hledger Flow generates flexible include files for you, and automatically includes opening/closing journals if the appropriately named files are present on disk.                                                                                           |+|                                                                                                                                                                                                                            |                                                                                                                                                                                                                                                            |+| FFH actually generates some useful reports right now. Hledger Flow still [[https://github.com/apauley/hledger-flow/pull/4][plans to get this done]] one day.                                                                                                                   | The "owner/bank/account" structure may look a bit much at first, but it allows us to run separate queries/reports for me/my spouse/my business etc and also allows for reports covering all of it in a an overall view.                                    |+|                                                                                                                                                                                                                            |                                                                                                                                                                                                                                                            |+| FFH chose scripts and build files that you can easily modify as you go along, but this requires a Haskell runtime to be installed everywhere it needs to run. The included docker image helps to make it less of an issue. | Hledger Flow distributes a compiled binary. This means users or deployment targets don't need extra dependencies installed, they can just run a CLI program. This also provides a clearer distinction between what is provided, and what users need to do. |+|                                                                                                                                                                                                                            |                                                                                                                                                                                                                                                            |+| The FFH build scripts requires familiarity with Haskell and the Shake build system.                                                                                                                                        | Users may need to write =preprocess= or =construct= hooks, but in a language of their choice.                                                                                                                                                              |+|                                                                                                                                                                                                                            |                                                                                                                                                                                                                                                            |+| Input files are assumed to always be CSV files.                                                                                                                                                                            | Hledger Flow de-empasises the need that input files must be in CSV format. Input files can be in any format that a =preprocess= or =construct= hook can read.                                                                                              |

@@ -2,15 +2,19 @@  module Main where -import Turtle+import Turtle hiding (switch) import Prelude hiding (FilePath, putStrLn)++import Options.Applicative+import Data.Semigroup ((<>))+ import qualified Hledger.Flow.Import.Types as IT import qualified Hledger.Flow.Report.Types as RT import Hledger.Flow.Common import Hledger.Flow.Reports import Hledger.Flow.CSVImport -type SubcommandParams = (Maybe FilePath, Bool)+type SubcommandParams = (Maybe FilePath, Bool, Bool) data Command = Version (Maybe Text) | Import SubcommandParams | Report SubcommandParams deriving (Show)  main :: IO ()@@ -22,20 +26,25 @@     Report subParams -> toReportOptions subParams >>= generateReports  toImportOptions :: SubcommandParams -> IO IT.ImportOptions-toImportOptions (maybeBaseDir, verbose) = do+toImportOptions (maybeBaseDir, verbose, sequential) = do   bd <- dirOrPwd maybeBaseDir-  return IT.ImportOptions {IT.baseDir = bd, IT.verbose = verbose}+  return IT.ImportOptions {IT.baseDir = bd, IT.verbose = verbose, IT.sequential = sequential}  toReportOptions :: SubcommandParams -> IO RT.ReportOptions-toReportOptions (maybeBaseDir, verbose) = do+toReportOptions (maybeBaseDir, verbose, sequential) = do   bd <- dirOrPwd maybeBaseDir-  return RT.ReportOptions {RT.baseDir = bd, RT.verbose = verbose}+  return RT.ReportOptions {RT.baseDir = bd, RT.verbose = verbose, RT.sequential = sequential}  parser :: Parser Command parser = fmap Import (subcommand "import" "Converts CSV transactions into categorised journal files" subcommandParser)   <|> fmap Report (subcommand "report" "Generate Reports" subcommandParser)-  <|> fmap Version (subcommand "version" "Display version information" (optional (argText "" "")))+  <|> fmap Version (subcommand "version" "Display version information" noArgs)  subcommandParser :: Parser SubcommandParams-subcommandParser = (,) <$> optional (argPath "basedir" "The hledger-flow base directory")-                       <*> switch  "verbose" 'v' "Print more verbose output"+subcommandParser = (,,)+  <$> optional (argPath "basedir" "The hledger-flow base directory")+  <*> switch (long "verbose" <> short 'v' <> help "Print more verbose output")+  <*> switch (long "sequential" <> help "Disable parallel processing")++noArgs :: Parser (Maybe Text)+noArgs = optional (argText "" "")

@@ -1,13 +1,13 @@ cabal-version: 1.12 --- This file has been generated from package.yaml by hpack version 0.31.1.+-- This file has been generated from package.yaml by hpack version 0.31.2. -- -- see: https://github.com/sol/hpack ----- hash: 29e37b663362aa563be3deed457271c77c4a0a359c0e7f91da8e160675f2f3a9+-- hash: dff74bf821d03440baa4b4f2c9e36afa00f7ce580680c1c94e598f56f06bf1bd  name:           hledger-flow-version:        0.11.0.0+version:        0.11.1.0 synopsis:       An hledger workflow focusing on automated statement import and classification. description:    Please see the README on GitHub at <https://github.com/apauley/hledger-flow#readme> category:       Finance, Console@@ -20,7 +20,7 @@ license-file:   LICENSE build-type:     Simple extra-source-files:-    README.md+    README.org     ChangeLog.md  source-repository head@@ -56,10 +56,12 @@       Paths_hledger_flow   hs-source-dirs:       app-  ghc-options: -threaded -rtsopts -with-rtsopts=-N -Wall+  ghc-options: -threaded -rtsopts -with-rtsopts=-N -Wall -fPIC+  ld-options: -dynamic   build-depends:       base >=4.7 && <5     , hledger-flow+    , optparse-applicative     , text     , turtle   default-language: Haskell2010

@@ -39,9 +39,9 @@       exit $ ExitFailure 1     else     do-      logVerbose opts ch $ format ("Found "%d%" input files in "%s%". Proceeding with import...") fileCount (repr diff)+      channelOut ch $ format ("Found "%d%" input files in "%s%". Proceeding with import...") fileCount (repr diff)       let actions = map (extractAndImport opts ch) inputFiles :: [IO FilePath]-      importedJournals <- single . shellToList $ parallel actions+      importedJournals <- if (sequential opts) then sequence actions else single . shellToList $ parallel actions       sh $ writeIncludesUpTo opts ch "import" importedJournals       return importedJournals @@ -81,7 +81,7 @@   let csvOut = changePathAndExtension "2-preprocessed" "csv" src   mktree $ directory csvOut   let script' = format fp script :: Text-  let action = procs script' [format fp src, format fp csvOut, lineToText bank, lineToText account, lineToText owner] empty+  let action = proc script' [format fp src, format fp csvOut, lineToText bank, lineToText account, lineToText owner] empty   let relScript = relativeToBase opts script   let relSrc = relativeToBase opts src   let msg = format ("executing '"%fp%"' on '"%fp%"'") relScript relSrc@@ -104,7 +104,7 @@   case maybeRulesFile of     Just rf -> do       let relRules = relativeToBase opts rf-      let action = procs "hledger" ["print", "--rules-file", format fp rf, "--file", format fp csvSrc, "--output-file", format fp journalOut] empty+      let action = proc "hledger" ["print", "--rules-file", format fp rf, "--file", format fp csvSrc, "--output-file", format fp journalOut] empty       let msg = format ("importing '"%fp%"' using rules file '"%fp%"'") relCSV relRules       _ <- logVerboseTime opts ch msg action       return journalOut@@ -152,7 +152,7 @@ customConstruct opts ch constructScript bank account owner csvSrc journalOut = do   let script = format fp constructScript :: Text   let importOut = inproc script [format fp csvSrc, "-", lineToText bank, lineToText account, lineToText owner] empty-  let action = procs "hledger" ["print", "--ignore-assertions", "--file", "-", "--output-file", format fp journalOut] importOut+  let action = proc "hledger" ["print", "--ignore-assertions", "--file", "-", "--output-file", format fp journalOut] importOut   let relScript = relativeToBase opts constructScript   let relSrc = relativeToBase opts csvSrc   let msg = format ("executing '"%fp%"' on '"%fp%"'") relScript relSrc

@@ -60,6 +60,8 @@ import Paths_hledger_flow (version) import qualified Data.Version as Version (showVersion) +type InputFileBundle = Map.Map FilePath [FilePath]+ versionInfo :: NE.NonEmpty Line versionInfo = textToLines $ T.pack ("hledger-flow " ++ Version.showVersion version) @@ -103,11 +105,11 @@ logVerbose :: HasVerbosity o => o -> TChan LogMessage -> Text -> IO () logVerbose opts ch msg = if (verbose opts) then logToChannel ch msg else return () -logVerboseTime :: HasVerbosity o => o -> TChan LogMessage -> Text -> IO a -> IO (a, NominalDiffTime)+logVerboseTime :: (HasVerbosity o, HasExitCode a) => o -> TChan LogMessage -> Text -> IO a -> IO (a, NominalDiffTime) logVerboseTime opts ch msg action = do   logVerbose opts ch $ format ("Begin: "%s) msg   (result, diff) <- time action-  logVerbose opts ch $ format ("End:   "%s%" ("%s%")") msg $ repr diff+  logVerbose opts ch $ format ("End:   "%s%" "%s%" ("%s%")") msg (repr $ exitCode result) (repr diff)   return (result, diff)  verboseTestFile :: (HasVerbosity o, HasBaseDir o) => o -> TChan LogMessage -> FilePath -> IO Bool@@ -150,19 +152,19 @@ allYearsPath' :: (FilePath -> FilePath) -> FilePath -> FilePath allYearsPath' dir p = dir p </> "all-years.journal" -groupIncludeFiles :: [FilePath] -> (Map.Map FilePath [FilePath], Map.Map FilePath [FilePath])+groupIncludeFiles :: [FilePath] -> (InputFileBundle, InputFileBundle) groupIncludeFiles = allYearIncludeFiles . groupIncludeFilesPerYear -groupIncludeFilesPerYear :: [FilePath] -> Map.Map FilePath [FilePath]+groupIncludeFilesPerYear :: [FilePath] -> InputFileBundle groupIncludeFilesPerYear [] = Map.empty groupIncludeFilesPerYear ps@(p:_) = case extractImportDirs p of     Right _ -> (groupValuesBy initialIncludeFilePath) ps     Left  _ -> (groupValuesBy parentIncludeFilePath)  ps -allYearIncludeFiles :: Map.Map FilePath [FilePath] -> (Map.Map FilePath [FilePath], Map.Map FilePath [FilePath])+allYearIncludeFiles :: InputFileBundle -> (InputFileBundle, InputFileBundle) allYearIncludeFiles m = (m, yearsIncludeMap $ Map.keys m) -yearsIncludeMap :: [FilePath] -> Map.Map FilePath [FilePath]+yearsIncludeMap :: [FilePath] -> InputFileBundle yearsIncludeMap = groupValuesBy allYearsPath  docURL :: Line -> Text@@ -232,22 +234,22 @@ includeFileName :: FilePath -> FilePath includeFileName = (<.> "journal") . fromText . (format (fp%"-include")) . dirname -toIncludeFiles :: (HasBaseDir o, HasVerbosity o) => o -> TChan LogMessage -> Map.Map FilePath [FilePath] -> Shell (Map.Map FilePath Text)+toIncludeFiles :: (HasBaseDir o, HasVerbosity o) => o -> TChan LogMessage -> InputFileBundle -> Shell (Map.Map FilePath Text) toIncludeFiles opts ch m = do   preMap  <- extraIncludes opts ch (Map.keys m) ["opening.journal"] ["pre-import.journal"]   postMap <- extraIncludes opts ch (Map.keys m) ["closing.journal"] ["post-import.journal"]   return $ (addPreamble . toIncludeFiles' preMap postMap) m -extraIncludes :: (HasBaseDir o, HasVerbosity o) => o -> TChan LogMessage -> [FilePath] -> [Text] -> [FilePath] -> Shell (Map.Map FilePath [FilePath])+extraIncludes :: (HasBaseDir o, HasVerbosity o) => o -> TChan LogMessage -> [FilePath] -> [Text] -> [FilePath] -> Shell (InputFileBundle) extraIncludes opts ch = extraIncludes' opts ch Map.empty -extraIncludes' :: (HasBaseDir o, HasVerbosity o) => o -> TChan LogMessage -> Map.Map FilePath [FilePath] -> [FilePath] -> [Text] -> [FilePath] -> Shell (Map.Map FilePath [FilePath])+extraIncludes' :: (HasBaseDir o, HasVerbosity o) => o -> TChan LogMessage -> InputFileBundle -> [FilePath] -> [Text] -> [FilePath] -> Shell (InputFileBundle) extraIncludes' _ _ acc [] _ _ = return acc extraIncludes' opts ch acc (file:files) extraSuffixes manualFiles = do   extra <- extraIncludesForFile opts ch file extraSuffixes manualFiles   extraIncludes' opts ch (Map.unionWith (++) acc extra) files extraSuffixes manualFiles -extraIncludesForFile :: (HasVerbosity o, HasBaseDir o) => o -> TChan LogMessage -> FilePath -> [Text] -> [FilePath] -> Shell (Map.Map FilePath [FilePath])+extraIncludesForFile :: (HasVerbosity o, HasBaseDir o) => o -> TChan LogMessage -> FilePath -> [Text] -> [FilePath] -> Shell (InputFileBundle) extraIncludesForFile opts ch file extraSuffixes manualFiles = do   let dirprefix = fromText $ fst $ T.breakOn "-" $ format fp $ basename file   let fileNames = map (\suff -> fromText $ format (fp%"-"%s) dirprefix suff) extraSuffixes@@ -264,7 +266,7 @@ relativeFilesAsText :: HasBaseDir o => o -> [FilePath] -> [Text] relativeFilesAsText opts ps = map ((format fp) . (relativeToBase opts)) ps -toIncludeFiles' :: Map.Map FilePath [FilePath] -> Map.Map FilePath [FilePath] -> Map.Map FilePath [FilePath] -> Map.Map FilePath Text+toIncludeFiles' :: InputFileBundle -> InputFileBundle -> InputFileBundle -> Map.Map FilePath Text toIncludeFiles' preMap postMap = Map.mapWithKey $ generatedIncludeText preMap postMap  addPreamble :: Map.Map FilePath Text -> Map.Map FilePath Text@@ -273,7 +275,7 @@ toIncludeLine :: FilePath -> FilePath -> Text toIncludeLine base file = format ("!include "%fp) $ relativeToBase' base file -generatedIncludeText :: Map.Map FilePath [FilePath] -> Map.Map FilePath [FilePath] -> FilePath -> [FilePath] -> Text+generatedIncludeText :: InputFileBundle -> InputFileBundle -> FilePath -> [FilePath] -> Text generatedIncludeText preMap postMap outputFile fs = do   let preFiles = fromMaybe [] $ Map.lookup outputFile preMap   let files = List.nub . List.sort $ fs@@ -300,7 +302,7 @@ writeTextMap :: Map.Map FilePath Text -> IO () writeTextMap = Map.foldlWithKey (\a k v -> a <> writeTextFile k v) (return ()) -writeFileMap :: (HasBaseDir o, HasVerbosity o) => o -> TChan LogMessage -> (Map.Map FilePath [FilePath], Map.Map FilePath [FilePath]) -> Shell [FilePath]+writeFileMap :: (HasBaseDir o, HasVerbosity o) => o -> TChan LogMessage -> (InputFileBundle, InputFileBundle) -> Shell [FilePath] writeFileMap opts ch (m, allYears) = do   _ <- writeFiles' $ (addPreamble . toIncludeFiles' Map.empty Map.empty) allYears   writeFiles . (toIncludeFiles opts ch) $ m

@@ -5,14 +5,14 @@ import Prelude hiding (FilePath, putStrLn) import Hledger.Flow.Types -data ImportOptions = ImportOptions { baseDir :: FilePath, verbose :: Bool }+data ImportOptions = ImportOptions { baseDir :: FilePath, verbose :: Bool, sequential :: Bool }   deriving (Show)  instance HasVerbosity ImportOptions where-  verbose (ImportOptions _ v) = v+  verbose (ImportOptions _ v _) = v  instance HasBaseDir ImportOptions where-  baseDir (ImportOptions bd _) = bd+  baseDir (ImportOptions bd _ _) = bd  data ImportDirs = ImportDirs { importDir  :: FilePath                              , ownerDir   :: FilePath

@@ -5,11 +5,11 @@ import Prelude hiding (FilePath, putStrLn) import Hledger.Flow.Types -data ReportOptions = ReportOptions { baseDir :: FilePath, verbose :: Bool }+data ReportOptions = ReportOptions { baseDir :: FilePath, verbose :: Bool, sequential :: Bool }   deriving (Show)  instance HasVerbosity ReportOptions where-  verbose (ReportOptions _ v) = v+  verbose (ReportOptions _ v _) = v  instance HasBaseDir ReportOptions where-  baseDir (ReportOptions bd _) = bd+  baseDir (ReportOptions bd _ _) = bd

@@ -11,3 +11,9 @@  class HasBaseDir a where   baseDir :: a -> FilePath++class HasExitCode a where+  exitCode :: a -> ExitCode++instance HasExitCode ExitCode where+  exitCode c = c

@@ -50,7 +50,7 @@ hiddenFiles = [".hiddenfile", "checking/.DS_Store", "import/john/bogartbank/savings/1-in/.anotherhiddenfile", "import/john/bogartbank/checking/1-in/2018/.hidden"] :: [FilePath]  defaultOpts :: FilePath -> ImportOptions-defaultOpts bd = ImportOptions bd False+defaultOpts bd = ImportOptions bd False False  toJournals :: [FilePath] -> [FilePath] toJournals = map (changePathAndExtension "3-journal" "journal")