hledger-makeitso 0.9.0.1 → 0.10.0.0
raw patch · 7 files changed
+674/−400 lines, 7 filesPVP ok
version bump matches the API change (PVP)
API changes (from Hackage documentation)
+ Hledger.MakeItSo.Common: versionInfo :: NonEmpty Line
Files
- ChangeLog.md +70/−2
- README.md +578/−0
- README.org +0/−387
- app/Main.hs +4/−2
- hledger-makeitso.cabal +5/−5
- src/Hledger/MakeItSo/CSVImport.hs +1/−1
- src/Hledger/MakeItSo/Common.hs +16/−3
ChangeLog.md view
@@ -1,3 +1,71 @@-# Changelog for hledger-makeitso+# Changelog for [hledger-makeitso](https://github.com/apauley/hledger-makeitso) -## Unreleased changes+## 0.10++- Add a `version` subcommand.+ Create [issue #15](https://github.com/apauley/hledger-makeitso/issues/15)+ to change it into a `--version` flag later.+- Fix a minor issue where yearly include files were generated at the top-level+ of the directtory structure, even though the same content was available in the+ `import` directory.+ The top-level `all-years.journal` now just includes the years within the+ `import` directory.+- Upgrade to LTS 13.15 for GHC 8.6.4+- Add CircleCI and TravisCI build instructions. Switch the README.org to a+ README.md in order to better support CI status badges.+++## 0.9.0.1++First hackage release. Minor changes to fix `stack sdist` warnings and errors, in preperation of+the hackage upload.++1bf817c "Merge pull request #9 from apauley/hackage-upload" Mar 31 21:51:38 2019 +0200++## 0.9++Process all statements in parallel.++This has a significant speed improvement on multi-processor machines when dealing with lots of input files.+a906bb5 "Merge pull request #8 from apauley/parallel-import" 2019-03-27 22:45:23 +0200++## 0.8++Generate an all-years.journal on each level which includes all the available years for that level. Replace the old makeitso.journal with the top-level version of this:++06f2127 "Merge pull request #5 from apauley/all-years-includes" 2019-03-22 00:09:27 +0200++## 0.7++Change the way include files are aggregated.++It used to be by owner/bank/account, now each of those levels (owner/bank/account) are aggregated per year:+eb17fed "Merge pull request #3 from apauley/annual-includes" 2019-03-12 23:09:17 +0200++## 0.6++61c71d6 "Upgrade to lts-13.9 (GHC 8.6.3)" 2019-03-01 11:31:23 +0200++## 0.5++3a7a39e "Upgrade to lts-13.6 (GHC 8.6.3)" 2019-02-16 09:54:49 +0200++## 0.4++213552d "Upgrade to lts-12.16 (GHC 8.4.4)" 2018-11-03 20:00:21 +0200++## 0.3++5e2d45f "Update from lts-12.1 to lts-12.11" 2018-10-01 23:07:21 +0200++## 0.2++First support for the construct script, when it was confusingly named an import script:++24ac4c7 "Support a fully custom import script" 2018-09-16 16:11:53 +0200++## 0.1++The first semi-useful version, replacing a previous bash script:++131f8af "Write journal" 2018-07-23 16:08:44 +0200
+ README.md view
@@ -0,0 +1,578 @@+# hledger: Make It So++# What is it?++`hledger-makeitso` is a command-line executable program that gives you a+guided hledger workflow. It focuses on automated processing of+electronic statements as much as possible, as opposed to manually adding+your own hledger journal entries.++Manual entries are still possible, we just think it saves time in the+long run to automatically process a statement whenever one is available.++It started when I realized that the scripts I wrote while playing around+with the ideas in [adept's Full-fledged Hledger](https://github.com/adept/full-fledged-hledger/wiki) isn't+really specific to my own finances, and can be shared.++# Overview of the Basic Workflow++1. Save an input CSV file to a [specific directory](https://github.com/apauley/hledger-makeitso#input-files).+2. Add an hledger [rules file](https://github.com/apauley/hledger-makeitso#rules-files).+ Include some classification rules if you want.+3. Run `hledger-makeitso import`++Add all your files to your favourite version control system.++The generated journal that you most likely want to use as your+`LEDGER_FILE` is called `all-years.journal`. This has include directives+to all the automatically imported journals, as well as includes for your+own manually managed journal entries.++In a typical software project we don't add generated files to version+control, but in this case I think it is a good idea to add all the+generated files to version control as well - when you inevitably change+something, e.g. how you classify transactions in your rules file, then+you can easily see if your change had the desired effect by looking at a+diff.++# Who should use this?++`hledger-makeitso` is intended for you if:++ - You are interested in getting started with+ [hledger](http://hledger.org/) and you wouldn't mind pointers to the+ right docs along the way.+ - You want a way to organise your finances into a structure that will+ be maintainable over the long term.+ - You want to automate as much as possible when dealing with your+ financial life.+ - You don't mind writing some scripts when needed, as long as it saves+ you time over the long term.+ - You want the ability to model your entire financial life in one+ tool, as opposed to just the parts that some online tool currently+ supports.+ - You appreciate the fact that all your financial information stays+ within your control.++# How Stable is it?++We're not close to a 1.0 release yet, which means that we can still make+changes if needed.++That being said, some parts have been used and tested extensively and+are likely to remain stable. Have a look at the "Stability of this+Feature" sections in the feature reference below.++I add future work, ideas and thoughts as [Github+issues](https://github.com/apauley/hledger-makeitso/issues) and in+[TODO.org](TODO.org), so have a look there for more clues as to what may+likely change.++Let me know if you can think of some improvements.++# Detailed Step-By-Step Guide++Have a look at the [detailed step-by-step instructions](docs/README.org)+and the files in the [documentation directory](docs/).++For a visual overview, check out the slide show version of the same+step-by-step instructions:++<https://pauley.org.za/hledger-makeitso/>++You can see the example imported financial transactions as it was+generated by the step-by-step instructions here:++<https://github.com/apauley/hledger-makeitso-example-finances>++# After Cloning This Repository++This repository has some submodules included, mostly related to the+examples in the documentation.++You need to initialise and update the submodules:++``` bash+git submodule init+git submodule update+```++# Build Instructions++[](https://circleci.com/gh/apauley/hledger-makeitso) [](https://travis-ci.com/apauley/hledger-makeitso)++You need a recent version of+[stack](https://docs.haskellstack.org/en/stable/README/) installed.++Then run:++``` bash+stack test+stack install+```++Which should end with this:++``` org+Copied executables to ~/.local/bin:+- hledger-makeitso+```++Ensure that `${HOME}/.local/bin` is in your `PATH`.++Usually this means adding this to your `~/.bashrc`:++``` bash+PATH="${HOME}/.local/bin:${PATH}"+```++## Building with older Haskell Versions++To build using an older version of GHC and related dependencies, point+stack to one of the other yaml files:++``` bash+stack test --stack-yaml stack-8.4.4.yaml+stack test --stack-yaml stack-8.2.2.yaml+```++# Feature Reference++## Input Files++Your input files will probably be CSV files with a line for each+transaction, although other file types will work fine if you use a+`preprocess` or a `construct` script that can read them. These scripts+are explained later.++We mostly use conventions based on a predefined directory structure for+your input statements.++For example, assuming you have a `savings` account at `mybank`, you'll+put your first CSV statement here:+`import/john/mybank/savings/1-in/2018/123456789_2018-06-30.csv`.++Some people may want to include accounts belonging to their spouse as+part of the household finances:+`import/spouse/otherbank/checking/1-in/2018/987654321_2018-06-30.csv`.++### More About Input Files++All files and directories under the `import` directory is related to the+automatic importing and classification of transactions.++The directory directly under `import` is meant to indicate the owner or+custodian of the accounts below it. It mostly has an impact on+reporting. You may want to have separate reports for `import/mycompany`+and `import/personal`.++Below the directory for the owner we can indicate where an account is+held. For a bank account you may choose to name it `import/john/mybank`.++If your underground bunker filled with gold has CSV statements linked to+it, then you can absolutely create `import/john/secret-treasure-room`.++Under the directory for the financial institution, you'll have a+directory for each account at that institution, e.g.+`import/mycompany/bigbankinc/customer-deposits` and+`import/mycompany/bigbankinc/expense-account`.++Next you'll create a directory named `1-in`. This is to distinguish it+from `2-preprocessed` and `3-journal` which will be auto-generated+later.++Under `1-in` you'll create a directory for the year, e.g. `2018`, and+within that you can copy the statements for that year:+`import/john/mybank/savings/1-in/2018/123456789_2018-06-30.csv`++### Stability of this Feature++The basic owner/bank/account/year structure has been used and tested+fairly extensively, I don't expect a need for it to change.++I'm open to suggestions for improvement though.++## Rules Files++If your input file is in CSV format, or converted to CSV by your+`preprocess` script, then you'll need an [hledger rules+file](http://hledger.org/csv.html).++`hledger-makeitso` will try to find a rules file for each statement in a+few places. The same rules file is typically used for all statements of+a specific account, or even for all accounts of the same specific bank.++ - A global rules file for any `mybank` statement can be saved here:+ `import/mybank.rules`+ - A rules file for all statements of a specific account:+ `import/spouse/bigbankinc/savings/bigbankinc-savings.rules`++### Statement-specific Rules Files++What happens if some of the statements for an account has a different+format than the others?++This can happen if you normally get your statements directly from your+bank, but some statements you had to download from somewhere else, like+Mint, because your bank is being daft with older statements.++In order to tell `hledger-makeitso` that you want to override the rules+file for a specific statement, you need to add a suffix, separated by an+underscore (`_`) and starting with the letters `rfo` (rules file+override) to the filename of that statement.++For example: assuming you've named your statement+`99966633_20171223_1844_rfo-mint.csv`.++`hledger-makeitso` will look for a rules file named `rfo-mint.rules` in+the following places:++ - in the import directory, e.g. `import/rfo-mint.rules`+ - in the bank directory, e.g. `import/john/mybank/rfo-mint.rules`+ - in the account directory, e.g.+ `import/john/mybank/savings/rfo-mint.rules`++### Example rules file usage++A common scenario is multiple accounts that share the same file format,+but have different `account1` directives.++One possible approach would be to include a shared rules file in your+account-specific rules file.++If you are lucky enough that all statements at `mybank` share a common+format across all accounts, then you can `include` a rules file that+just defines the parts that are shared across accounts.++Two accounts at `mybank` may have rules files similar to these.++A checking account at mybank:++``` hledger+# Saved as: import/john/mybank/checking/mybank-checking.rules+include ../../../mybank-shared.rules+account1 Assets:Current:John:MyBank:Checking+```++Another account at mybank:++``` hledger+# Saved as: import/alice/mybank/savings/mybank-savings.rules+include ../../../mybank-shared.rules+account1 Assets:Current:Alice:MyBank:Savings+```++Where `import/mybank-shared.rules` may define some shared attributes:++``` hledger+skip 1++fields date, description, amount, balance++date-format %Y-%m-%d+currency $+```++Another possible approach could be to use your `preprocess` script to+write out a CSV file that has extra fields for `account1` and+`account2`.++You could then create the above mentioned global `import/mybank.rules`+with the fields defined more or less like this:++``` hledger+fields date, description, amount, balance, account1, account2+```++### Stability of this Feature++Rules files are a stable feature within [hledger](http://hledger.org/),+and we're just using the normal hledger rules files. The account, bank+and statement-specific rules files have been used and tested fairly+extensively, I don't expect this to change.++Let me know if you think it should change.++## Opening and Closing Balances++### Opening Balances++`hledger-makeitso` looks for a file named `YEAR-opening.journal` in each+account directory, where `YEAR` corresponds to an actual year directory,+eg. **1983** (if you have electronic statements [dating back+to 1983](https://en.wikipedia.org/wiki/Online_banking#First_online_banking_services_in_the_United_States)).+Example: `import/john/mybank/savings/1983-opening.journal`++If it exists the file will automatically be included at the beginning of+the generated journal include file for that year.++You need to edit this file for each account to specify the opening+balance at the date of the first available transaction.++An opening balance may look something like this:++``` hledger+2018-06-01 Savings Account Opening Balance+assets:Current:MyBank:Savings $102.01+equity:Opening Balances:MyBank:Savings+```++### Closing Balances++Similar to opening balances, `hledger-makeitso` looks for an optional+file named `YEAR-closing.journal` in each account directory. Example:+`import/john/mybank/savings/1983-closing.journal`++If it exists the file will automatically be included at the end of the+generated journal include file for that year.++A closing balance may look something like this:++``` hledger+2018-06-01 Savings Account Closing Balance+assets:Current:MyBank:Savings $-234.56 = $0.00+equity:Closing Balances:MyBank:Savings+```++### Example Opening and Closing Journal Files++As an example, assuming that the relevant year is `2019` and+`hledger-makeitso` is about to generate+`import/john/mybank/savings/2019-include.journal`, then one or both of+the following files will be added to the include file if they exist:++1. `import/john/mybank/savings/2019-opening.journal`+2. `import/john/mybank/savings/2019-closing.journal`++The `opening.journal` will be included just before the other included+entries, while the `closing.journal` will be included just after the+other entries in that include file.++An include file may look like this:++``` bash+cat import/john/mybank/savings/2019-include.journal+```++``` hledger+### Generated by hledger-makeitso - DO NOT EDIT ###++!include 2019-opening.journal+!include 3-journal/2019/123456789_2019-01-30+!include 2019-closing.journal+```++### Stability of this Feature++The opening balances file works well in my opinion, I don't expect it to+change. I'm only using closing balances in one or two places, so maybe+that could do with some suggestions from people who use this more than+myself.++## The `preprocess` Script++Sometimes the statements you get from your bank is [less than+suitable](https://github.com/apauley/fnb-csv-demoronizer) for automatic+processing. Or maybe you just want to make it easier for the hledger+rules file to do its thing by adding some useful columns.++If you put a script called `preprocess` in the account directory, e.g.+`import/john/mybank/savings/preprocess`, then `hledger-makeitso` will+call that script for each input statement.++The `preprocess` script will be called with 4 positional parameters:++1. The path to the input statement, e.g.+ `import/john/mybank/savings/1-in/2018/123456789_2018-06-30.csv`+2. The path to an output file that can be sent to `hledger`, e.g.+ `import/john/mybank/savings/2-preprocessed/2018/123456789_2018-06-30.csv`+3. The name of the bank, e.g. `mybank`+4. The name of the account, e.g. `savings`+5. The name of the owner, e.g. `john`++Your `preprocess` script is expected to:++ - read the input file+ - write a new output file at the supplied path that works with your+ rules file+ - be idempotent. Running `preprocess` multiple times on the same files+ will produce the same result.++### Stability of this Feature++Stable and tested.++## The `construct` Script++If you need even more power and flexibility than what you can get from+the `preprocess` script and `hledger`'s CSV import functionality, then+you can create your own custom script to `construct` transactions+exactly as you need them.++At the expense of more construction work for you, of course.++As an example, `hledger`'s CSV import currently [only supports two+postings per+transaction](https://github.com/simonmichael/hledger/issues/627), even+though `hledger` itself is perfectly happy with transactions containing+more than two postings, e.g.:++``` hledger+2019-02-01 Mortgage Payment+Liabilities:Mortgage $1000.00+Expenses:Interest:Real Estate $833.33+Assets:Cash -$1833.33+```++The `construct` script can be used in addition to the `preprocess`+script, or on it's own. But since the `construct` script is more+powerful than the `preprocess` script, you could tell your `construct`+script to do anything that the `preprocess` script would have done.++Save your `construct` script in the account directory, e.g.+`import/john/mybank/savings/construct`.++`hledger-makeitso` will call your `construct` script with 4 positional+parameters:++1. The path to the input statement, e.g.+ `import/john/mybank/savings/1-in/2018/123456789_2018-06-30.csv`+2. A "-" (indicating that output should be sent to `stdout`)+3. The name of the bank, e.g. `mybank`+4. The name of the account, e.g. `savings`+5. The name of the owner, e.g. `john`++Your `construct` script is expected to:++ - read the input file+ - generate your own `hledger` journal transactions+ - be idempotent. Running `construct` multiple times on the same files+ should produce the same result.+ - send all output to `stdout`. `hledger-makeitso` will pipe your+ output into `hledger` which will format it and save it to an output+ file.++### Stability of this Feature++Stable and tested.++## Manually Managed Journals++Not every transaction in your life comes with CSV statements.++Sometimes you just need to add a transaction for that time you loaned a+friend some money.++`hledger-makeitso` looks for `pre-import` and `post-import` files+related to each generated include file as part of the import.++You can enter your own transactions manually into these files.++You can run `hledger-makeitso import --verbose` to see exactly which+files are being looked for.++As an example, assuming that the relevant year is `2019` and+`hledger-makeitso` is about to generate+`import/john/2019-include.journal`, then one or both of the following+files will be added to the include file if they exist:++1. `import/john/_manual_/2019/pre-import.journal`+2. `import/john/_manual_/2019/post-import.journal`++The `pre-import.journal` will be included just before the other included+entries, while the `post-import.journal` will be included just after the+other entries in that include file.++An include file may look like this:++``` bash+cat import/john/2019-include.journal+```++``` hledger+### Generated by hledger-makeitso - DO NOT EDIT ###++!include _manual_/2019/pre-import.journal+!include mybank/2019-include.journal+!include otherbank/2019-include.journal+!include _manual_/2019/post-import.journal+```++### Stability of this Feature++It works, but the naming of `_manual_` looks a bit weird. Should it be+changed?++# Compatibility with Ledger++When writing out the journal include files, `hledger-makeitso` sorts the+include statements by filename.++[Ledger](https://www.ledger-cli.org/) fails any balance assertions when+the transactions aren't included in chronological order.++An easy way around this is to name your input files so that March's+statement is listed before December's statement.++Another option is to add `--permissive` to any+[ledger](https://www.ledger-cli.org/) command.++So you should easily be able to use both `ledger` and `hledger` on these+journals.++# Project Goals++My `hledger` files started to collect a bunch of supporting code that+weren't really specific to my financial situation.++I want to extract and share as much as possible of that supporting code.++[Adept's](https://github.com/adept/full-fledged-hledger/wiki) goals also+resonated with me:++ - Tracking expenses should take as little time, effort and manual work+ as possible+ - Eventual consistency should be achievable: even if I can't record+ something precisely right now, maybe I would be able to do it later,+ so I should be able to leave things half-done and pick them up later+ - Ability to refactor is a must. I want to be able to go back and+ change the way I am doing things, with as little effort as possible+ and without fear of irrevocably breaking things.++I've given [a talk](https://pauley.org.za/functional-finance-hledger/)+at [Lambda Luminaries+Johannesburg](https://www.meetup.com/lambda-luminaries/events/qklkvpyxmbnb/)+featuring hledger and hledger-makeitso.++# FAQ++## How does `hledger-makeitso` differ from `Full-fledged Hledger`?++[Full-fledged Hledger](https://github.com/adept/full-fledged-hledger/wiki#full-fledged-hledger-tutorial)+is a brilliant system, and hledger-makeitso continues to learn much from+it.++It has great documentation that does an excellent job of not only+showing **how** things can be done, but also **why** it is such a great+idea.++hledger-makeitso can be seen as a specific implementation of the+Full-fledged Hledger system, with a few implementation details that are+different.++| Full-fledged Hledger | Hledger: Make It So |+| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |+| | |+| FFH describes itself as a [tutorial](https://github.com/adept/full-fledged-hledger/wiki#full-fledged-hledger-tutorial) with helper scripts that you can start using and adapt to your needs. | I started by following the FFH tutorial, and changed bits and pieces over time to suit my needs. The "owner/bank/account" structure for example. |+| | |+| FFH is more open-ended: you can start with the basic scripts and over time turn it into something that solves your needs exactly. But you'll also end up with more code that you need to maintain yourself. | HMIS is less open-ended. For example, you have to adopt the "owner/bank/account" structure precisely as specified. But this allows HMIS to do more work for you. |+| | |+| Maintaining include files are currently part of the user's responsibility. | HMIS generates flexible include files for you, and automatically includes opening/closing journals if the appropriately named files are present on disk. |+| | |+| FFH actually generates some useful reports right now. HMIS still [plans to get this done](https://github.com/apauley/hledger-makeitso/pull/4) one day. | The "owner/bank/account" structure may look a bit much at first, but it allows us to run separate queries/reports for me/my spouse/my business etc and also allows for reports covering all of it in a an overall view. |+| | |+| FFH chose scripts and build files that you can easily modify as you go along, but this requires a Haskell runtime to be installed everywhere it needs to run. The included docker image helps to make it less of an issue. | HMIS distributes a compiled binary. This means users or deployment targets don't need extra dependencies installed, they can just run a CLI program. This also provides a clearer distinction between what is provided, and what users need to do. |+| | |+| The FFH build scripts requires familiarity with Haskell and the Shake build system. | Users may need to write `preprocess` or `construct` hooks, but in a language of their choice. |+| | |+| Input files are assumed to always be CSV files. | HMIS de-empasises the need that input files must be in CSV format. Input files can be in any format that a `preprocess` or `construct` hook can read. |
− README.org
@@ -1,387 +0,0 @@-#+STARTUP: showall-#+PROPERTY: header-args:sh :prologue exec 2>&1 :epilogue echo :--* hledger: Make It So-- Inspired by the structure and ideas of [[https://github.com/adept/full-fledged-hledger/wiki][adept's Full-fledged Hledger]].-- I've given [[https://pauley.org.za/functional-finance-hledger/][a talk]] at [[https://www.meetup.com/lambda-luminaries/events/qklkvpyxmbnb/][Lambda Luminaries Johannesburg]] featuring hledger and hledger-makeitso.--* Who should use this?-- =hledger-makeitso= is intended for you if:-- - You are interested in getting started with [[http://hledger.org/][hledger]] and you wouldn't mind pointers to the right docs along the way.- - You want a way to organise your finances into a structure that will be maintainable over the long term.- - You want to be able to automate as much as possible when dealing with your financial life.- - You don't mind writing some scripts when needed, as long as it saves you time over the long term.- - You want the ability to model your entire financial life in one tool, as opposed to just the parts that some online tool currently supports.- - You appreciate the fact that all your financial information stays within your control.-- At the moment this software is in a fairly stable beta.- It is still possible that I'll make a breaking change to enable some feature, but I hope not.-- Let me know if you are actively using this - I'm more likely to proceed with caution if I know I'm not the only active user.--* After Cloning This Repository-- This repository has some submodules included, mostly related to the examples in the documentation.-- You need to initialise and update the submodules:- #+BEGIN_SRC sh- git submodule init- git submodule update- #+END_SRC--* Build Instructions-- You need a recent version of [[https://docs.haskellstack.org/en/stable/README/][stack]] installed.-- Then run:- #+NAME: stack-build- #+BEGIN_SRC sh :results none :exports both- stack test- stack install- #+END_SRC-- Which should end with this:- #+BEGIN_SRC org- Copied executables to ~/.local/bin:- - hledger-makeitso- #+END_SRC-- Ensure that =${HOME}/.local/bin= is in your =PATH=.-- Usually this means adding this to your =~/.bashrc=:- #+BEGIN_SRC sh :results none :exports both- PATH="${HOME}/.local/bin:${PATH}"- #+END_SRC--* How to Use It--** Overview of the Basic Workflow-- 1. Save an input CSV file to a [[https://github.com/apauley/hledger-makeitso#input-files][specific directory]].- 2. Add an hledger [[https://github.com/apauley/hledger-makeitso#rules-files][rules file]]. Include some classification rules if you want.- 3. Run =hledger-makeitso import=-- Add all your files to your favourite version control system.-- The generated journal that you most likely want to use as your =LEDGER_FILE= is called =all-years.journal=.- This has include directives to all the automatically imported journals, as well as includes for your- own manually managed journal entries.-- In a typical software project we don't add generated files to version control, but in this case I think it is a good idea- to add all the generated files to version control as well - when you inevitably change something, e.g. how you classify transactions- in your rules file, then you can easily see if your change had the desired effect by looking at a diff.--** Detailed Step-By-Step Guide-- Have a look at the [[file:docs/README.org][detailed step-by-step instructions]] and the files in the [[file:docs/][documentation directory]].-- For a visual overview, check out the slide show version of the same step-by-step instructions:-- https://pauley.org.za/hledger-makeitso/-- You can see the example imported financial transactions as it was generated by the step-by-step- instructions here:-- https://github.com/apauley/hledger-makeitso-example-finances--* Feature Reference--** Input Files-- Your input files will probably be CSV files with a line for each transaction,- although other file types will work fine if you use a =preprocess= or a =construct= script that can read them.- These scripts are explained later.-- We mostly use conventions based on a predefined directory structure for your input statements.-- For example, assuming you have a =savings= account at =mybank=, you'll put your first CSV statement here:- =import/john/mybank/savings/1-in/2018/123456789_2018-06-30.csv=.-- Some people may want to include accounts belonging to their spouse as part of the household finances:- =import/spouse/otherbank/checking/1-in/2018/987654321_2018-06-30.csv=.--*** More About Input Files-- All files and directories under the =import= directory is related to the automatic importing and classification of transactions.-- The directory directly under =import= is meant to indicate the owner or custodian of the accounts below it.- It mostly has an impact on reporting. You may want to have separate reports for =import/mycompany= and =import/personal=.-- Below the directory for the owner we can indicate where an account is held.- For a bank account you may choose to name it =import/john/mybank=.-- If your underground bunker filled with gold has CSV statements linked to it,- then you can absolutely create =import/john/secret-treasure-room=.-- Under the directory for the financial institution, you'll have a directory for each account- at that institution, e.g. =import/mycompany/bigbankinc/customer-deposits= and- =import/mycompany/bigbankinc/expense-account=.-- Next you'll create a directory named =1-in=.- This is to distinguish it from =2-preprocessed= and =3-journal= which will be auto-generated later.-- Under =1-in= you'll create a directory for the year, e.g. =2018=, and within that you can copy the statements for that year:- =import/john/mybank/savings/1-in/2018/123456789_2018-06-30.csv=--** Rules Files-- If your input file is in CSV format, or converted to CSV by your =preprocess= script,- then you'll need an [[http://hledger.org/csv.html][hledger rules file]].-- =hledger-makeitso= will try to find a rules file for each statement in a few places.- The same rules file is typically used for all statements of a specific account, or even for all accounts- of the same specific bank.-- - A global rules file for any =mybank= statement can be saved here: =import/mybank.rules=- - A rules file for all statements of a specific account: =import/spouse/bigbankinc/savings/bigbankinc-savings.rules=--*** Statement-specific Rules Files-- What happens if some of the statements for an account has a different format than the others?-- This can happen if you normally get your statements directly from your bank, but some- statements you had to download from somewhere else, like Mint, because your bank is being daft- with older statements.-- In order to tell =hledger-makeitso= that you want to override the rules file for a specific statement,- you need to add a suffix, separated by an underscore (=_=) and starting with the letters =rfo= (rules file override)- to the filename of that statement.-- For example: assuming you've named your statement =99966633_20171223_1844_rfo-mint.csv=.-- =hledger-makeitso= will look for a rules file named =rfo-mint.rules= in the following places:- - in the import directory, e.g. =import/rfo-mint.rules=- - in the bank directory, e.g. =import/john/mybank/rfo-mint.rules=- - in the account directory, e.g. =import/john/mybank/savings/rfo-mint.rules=--*** Example rules file usage-- A common scenario is multiple accounts that share the same file format, but have different =account1= directives.-- One possible approach would be to include a shared rules file in your account-specific rules file.-- If you are lucky enough that all statements at =mybank= share a common format across all accounts,- then you can =include= a rules file that just defines the parts that are shared across accounts.-- Two accounts at =mybank= may have rules files similar to these.-- A checking account at mybank:- #+BEGIN_SRC hledger- # Saved as: import/john/mybank/checking/mybank-checking.rules- include ../../../mybank-shared.rules- account1 Assets:Current:John:MyBank:Checking- #+END_SRC-- Another account at mybank:- #+BEGIN_SRC hledger- # Saved as: import/alice/mybank/savings/mybank-savings.rules- include ../../../mybank-shared.rules- account1 Assets:Current:Alice:MyBank:Savings- #+END_SRC-- Where =import/mybank-shared.rules= may define some shared attributes:- #+BEGIN_SRC hledger- skip 1-- fields date, description, amount, balance-- date-format %Y-%m-%d- currency $- #+END_SRC-- Another possible approach could be to use your =preprocess= script to write out a CSV file- that has extra fields for =account1= and =account2=.-- You could then create the above mentioned global =import/mybank.rules= with the fields defined more or less like this:- #+BEGIN_SRC hledger- fields date, description, amount, balance, account1, account2- #+END_SRC--** Opening and Closing Balances--*** Opening Balances-- =hledger-makeitso= looks for a file named =YEAR-opening.journal= in each account directory, where =YEAR=- corresponds to an actual year directory, eg. *1983*- (if you have electronic statements [[https://en.wikipedia.org/wiki/Online_banking#First_online_banking_services_in_the_United_States][dating back to 1983]]).- Example: =import/john/mybank/savings/1983-opening.journal=-- If it exists the file will automatically be included at the beginning of the generated journal include file for that year.-- You need to edit this file for each account to specify the opening balance at the date of the first available transaction.-- An opening balance may look something like this:- #+BEGIN_SRC hledger- 2018-06-01 Savings Account Opening Balance- assets:Current:MyBank:Savings $102.01- equity:Opening Balances:MyBank:Savings- #+END_SRC--*** Closing Balances-- Similar to opening balances, =hledger-makeitso= looks for an optional file named =YEAR-closing.journal= in each account directory.- Example: =import/john/mybank/savings/1983-closing.journal=-- If it exists the file will automatically be included at the end of the generated journal include file for that year.-- A closing balance may look something like this:- #+BEGIN_SRC hledger- 2018-06-01 Savings Account Closing Balance- assets:Current:MyBank:Savings $-234.56 = $0.00- equity:Closing Balances:MyBank:Savings- #+END_SRC--*** Example Opening and Closing Journal Files-- As an example, assuming that the relevant year is =2019= and =hledger-makeitso= is about to generate- =import/john/mybank/savings/2019-include.journal=, then one or both of the following files will be added to the include file if they exist:-- 1. =import/john/mybank/savings/2019-opening.journal=- 2. =import/john/mybank/savings/2019-closing.journal=-- The =opening.journal= will be included just before the other included entries, while the =closing.journal=- will be included just after the other entries in that include file.-- An include file may look like this:-- #+BEGIN_SRC sh- cat import/john/mybank/savings/2019-include.journal- #+END_SRC-- #+BEGIN_SRC hledger- ### Generated by hledger-makeitso - DO NOT EDIT ###-- !include 2019-opening.journal- !include 3-journal/2019/123456789_2019-01-30- !include 2019-closing.journal- #+END_SRC--** The =preprocess= Script-- Sometimes the statements you get from your bank is [[https://github.com/apauley/fnb-csv-demoronizer][less than suitable]] for automatic processing.- Or maybe you just want to make it easier for the hledger rules file to do its thing by adding some useful columns.-- If you put a script called =preprocess= in the account directory, e.g. =import/john/mybank/savings/preprocess=,- then =hledger-makeitso= will call that script for each input statement.-- The =preprocess= script will be called with 4 positional parameters:- 1. The path to the input statement, e.g. =import/john/mybank/savings/1-in/2018/123456789_2018-06-30.csv=- 2. The path to an output file that can be sent to =hledger=, e.g. =import/john/mybank/savings/2-preprocessed/2018/123456789_2018-06-30.csv=- 3. The name of the bank, e.g. =mybank=- 4. The name of the account, e.g. =savings=- 5. The name of the owner, e.g. =john=-- Your =preprocess= script is expected to:- - read the input file- - write a new output file at the supplied path that works with your rules file- - be idempotent. Running =preprocess= multiple times on the same files will produce the same result.--** The =construct= Script-- If you need even more power and flexibility than what you can get from the =preprocess= script and =hledger='s CSV import functionality,- then you can create your own custom script to =construct= transactions exactly as you need them.-- At the expense of more construction work for you, of course.-- As an example, =hledger='s CSV import currently [[https://github.com/simonmichael/hledger/issues/627][only supports two postings per transaction]], even though =hledger= itself- is perfectly happy with transactions containing more than two postings, e.g.:-- #+BEGIN_SRC hledger- 2019-02-01 Mortgage Payment- Liabilities:Mortgage 1,000.00- Expenses:Interest:Real Estate 833.33- Assets:Cash -1833.33- #+END_SRC-- The =construct= script can be used in addition to the =preprocess= script, or on it's own.- But since the =construct= script is more powerful than the =preprocess= script, you could tell your =construct= script to do- anything that the =preprocess= script would have done.-- Save your =construct= script in the account directory, e.g. =import/john/mybank/savings/construct=.-- =hledger-makeitso= will call your =construct= script with 4 positional parameters:- 1. The path to the input statement, e.g. =import/john/mybank/savings/1-in/2018/123456789_2018-06-30.csv=- 2. A "-" (indicating that output should be sent to =stdout=)- 3. The name of the bank, e.g. =mybank=- 4. The name of the account, e.g. =savings=- 5. The name of the owner, e.g. =john=-- Your =construct= script is expected to:- - read the input file- - generate your own =hledger= journal transactions- - be idempotent. Running =construct= multiple times on the same files should produce the same result.- - send all output to =stdout=. =hledger-makeitso= will pipe your output into =hledger= which will format it and save it to an output file.---** Manually Managed Journals-- Not every transaction in your life comes with CSV statements.-- Sometimes you just need to add a transaction for that time you loaned a friend some money.-- =hledger-makeitso= looks for =pre-import= and =post-import= files related to each generated include file as part of the import.-- You can enter your own transactions manually into these files.-- You can run =hledger-makeitso import --verbose= to see exactly which files are being looked for.-- As an example, assuming that the relevant year is =2019= and =hledger-makeitso= is about to generate- =import/john/2019-include.journal=, then one or both of the following files will be added to the include file if they exist:-- 1. =import/john/_manual_/2019/pre-import.journal=- 2. =import/john/_manual_/2019/post-import.journal=-- The =pre-import.journal= will be included just before the other included entries, while the =post-import.journal=- will be included just after the other entries in that include file.-- An include file may look like this:-- #+BEGIN_SRC sh- cat import/john/2019-include.journal- #+END_SRC-- #+BEGIN_SRC hledger- ### Generated by hledger-makeitso - DO NOT EDIT ###-- !include _manual_/2019/pre-import.journal- !include mybank/2019-include.journal- !include otherbank/2019-include.journal- !include _manual_/2019/post-import.journal- #+END_SRC--* Compatibility with Ledger-- When writing out the journal include files, =hledger-makeitso= sorts the include statements by filename.-- [[https://www.ledger-cli.org/][Ledger]] fails any balance assertions when the transactions aren't included in chronological order.-- An easy way around this is to name your input files so that March's statement is listed before December's statement.-- Another option is to add =--permissive= to any [[https://www.ledger-cli.org/][ledger]] command.-- So you should easily be able to use both =ledger= and =hledger= on these journals.--* Project Goals-- My =hledger= files started to collect a bunch of supporting code that weren't really- specific to my financial situation.-- I want to extract and share as much as possible of that supporting code.-- Adept's goals also resonated with me:-- - Tracking expenses should take as little time, effort and manual work as possible- - Eventual consistency should be achievable: even if I can't record something precisely right now,- maybe I would be able to do it later, so I should be able to leave things half-done and pick them up later- - Ability to refactor is a must. I want to be able to go back and change the way I am doing things,- with as little effort as possible and without fear of irrevocably breaking things.--* Still To Be Done-- I add ideas and thoughts in [[file:TODO.org][TODO.org]]-- Let me know if you can think of some improvements.
app/Main.hs view
@@ -11,12 +11,13 @@ import Hledger.MakeItSo.CSVImport type SubcommandParams = (Maybe FilePath, Bool)-data Command = Import SubcommandParams | Report SubcommandParams deriving (Show)+data Command = Version (Maybe Text) | Import SubcommandParams | Report SubcommandParams deriving (Show) main :: IO () main = do cmd <- options "An hledger workflow focusing on automated statement import and classification:\nhttps://github.com/apauley/hledger-makeitso#readme" parser case cmd of+ Version _ -> stdout $ select versionInfo Import subParams -> toImportOptions subParams >>= importCSVs Report subParams -> toReportOptions subParams >>= generateReports @@ -32,7 +33,8 @@ parser :: Parser Command parser = fmap Import (subcommand "import" "Converts CSV transactions into categorised journal files" subcommandParser)- <|> fmap Report (subcommand "report" "Generate Reports" subcommandParser)+ <|> fmap Report (subcommand "report" "Generate Reports" subcommandParser)+ <|> fmap Version (subcommand "version" "Display version information" (optional (argText "" ""))) subcommandParser :: Parser SubcommandParams subcommandParser = (,) <$> optional (argPath "basedir" "The hledger-makeitso base directory")
hledger-makeitso.cabal view
@@ -4,23 +4,23 @@ -- -- see: https://github.com/sol/hpack ----- hash: 59ade1a7ba88d7f66d95d1181ca0947b0c46281f3b7909755578659ead785124+-- hash: 50dfd4532a9271fb43604b13777e11ea8b183289ada3795f017879911383784c name: hledger-makeitso-version: 0.9.0.1+version: 0.10.0.0 synopsis: An hledger workflow focusing on automated statement import and classification. description: Please see the README on GitHub at <https://github.com/apauley/hledger-makeitso#readme> category: Finance, Console homepage: https://github.com/apauley/hledger-makeitso#readme bug-reports: https://github.com/apauley/hledger-makeitso/issues-author: Andreas Pauley <andreas@pauley.org.za>-maintainer: Andreas Pauley <andreas@pauley.org.za>+author: Andreas Pauley <hledger-makeitso@pauley.org.za>+maintainer: Andreas Pauley <hledger-makeitso@pauley.org.za> copyright: 2018 Andreas Pauley license: GPL-3 license-file: LICENSE build-type: Simple extra-source-files:- README.org+ README.md ChangeLog.md source-repository head
src/Hledger/MakeItSo/CSVImport.hs view
@@ -28,7 +28,7 @@ importCSVs' :: ImportOptions -> TChan LogMessage -> IO [FilePath] importCSVs' opts ch = do channelOut ch "Collecting input files..."- (inputFiles, diff) <- time $ single . shellToList . onlyFiles $ find (has (suffix "1-in/")) $ baseDir opts+ (inputFiles, diff) <- time $ single . shellToList . onlyFiles $ find (has (suffix "1-in")) $ baseDir opts let fileCount = length inputFiles if (fileCount == 0) then do
src/Hledger/MakeItSo/Common.hs view
@@ -2,6 +2,7 @@ module Hledger.MakeItSo.Common ( docURL+ , versionInfo , showCmdArgs , consoleChannelLoop , terminateChannelLoop@@ -55,6 +56,13 @@ import qualified Hledger.MakeItSo.Import.Types as IT import Control.Concurrent.STM +import qualified Data.List.NonEmpty as NE+import Paths_hledger_makeitso (version)+import qualified Data.Version as Version (showVersion)++versionInfo :: NE.NonEmpty Line+versionInfo = textToLines $ T.pack ("hledger-makeitso " ++ Version.showVersion version)+ showCmdArgs :: [Text] -> Text showCmdArgs args = T.intercalate " " (map escapeArg args) @@ -137,8 +145,11 @@ parentIncludeFilePath p = (parent . parent) p </> (filename p) allYearsPath :: FilePath -> FilePath-allYearsPath p = directory p </> "all-years.journal"+allYearsPath = allYearsPath' directory +allYearsPath' :: (FilePath -> FilePath) -> FilePath -> FilePath+allYearsPath' dir p = dir p </> "all-years.journal"+ groupIncludeFiles :: [FilePath] -> (Map.Map FilePath [FilePath], Map.Map FilePath [FilePath]) groupIncludeFiles = allYearIncludeFiles . groupIncludeFilesPerYear @@ -299,8 +310,10 @@ writeIncludesUpTo opts ch stopAt paths = do let shouldStop = any (\dir -> dir == stopAt) $ map dirname paths if shouldStop- then groupAndWriteIncludeFiles opts ch paths else- do+ then do+ let allTop = groupValuesBy (allYearsPath' (parent . parent)) paths+ writeFileMap opts ch (Map.empty, allTop)+ else do newPaths <- groupAndWriteIncludeFiles opts ch paths writeIncludesUpTo opts ch stopAt newPaths