#+STARTUP: showall
#+PROPERTY: header-args:sh :prologue exec 2>&1 :epilogue echo :
* hledger: Make It So
Inspired by the structure and ideas of [[https://github.com/adept/full-fledged-hledger/wiki][adept's Full-fledged Hledger]].
I've given [[https://pauley.org.za/functional-finance-hledger/][a talk]] at [[https://www.meetup.com/lambda-luminaries/events/qklkvpyxmbnb/][Lambda Luminaries Johannesburg]] featuring hledger and hledger-makeitso.
* Who should use this?
=hledger-makeitso= is intended for you if:
- You are interested in getting started with [[http://hledger.org/][hledger]] and you wouldn't mind pointers to the right docs along the way.
- You want a way to organise your finances into a structure that will be maintainable over the long term.
- You want to be able to automate as much as possible when dealing with your financial life.
- You don't mind writing some scripts when needed, as long as it saves you time over the long term.
- You want the ability to model your entire financial life in one tool, as opposed to just the parts that some online tool currently supports.
- You appreciate the fact that all your financial information stays within your control.
At the moment this software is in a fairly stable beta.
It is still possible that I'll make a breaking change to enable some feature, but I hope not.
Let me know if you are actively using this - I'm more likely to proceed with caution if I know I'm not the only active user.
* After Cloning This Repository
This repository has some submodules included, mostly related to the examples in the documentation.
You need to initialise and update the submodules:
#+BEGIN_SRC sh
git submodule init
git submodule update
#+END_SRC
* Build Instructions
You need a recent version of [[https://docs.haskellstack.org/en/stable/README/][stack]] installed.
Then run:
#+NAME: stack-build
#+BEGIN_SRC sh :results none :exports both
stack test
stack install
#+END_SRC
Which should end with this:
#+BEGIN_SRC org
Copied executables to ~/.local/bin:
- hledger-makeitso
#+END_SRC
Ensure that =${HOME}/.local/bin= is in your =PATH=.
Usually this means adding this to your =~/.bashrc=:
#+BEGIN_SRC sh :results none :exports both
PATH="${HOME}/.local/bin:${PATH}"
#+END_SRC
* How to Use It
** Overview of the Basic Workflow
1. Save an input CSV file to a [[https://github.com/apauley/hledger-makeitso#input-files][specific directory]].
2. Add an hledger [[https://github.com/apauley/hledger-makeitso#rules-files][rules file]]. Include some classification rules if you want.
3. Run =hledger-makeitso import=
Add all your files to your favourite version control system.
The generated journal that you most likely want to use as your =LEDGER_FILE= is called =all-years.journal=.
This has include directives to all the automatically imported journals, as well as includes for your
own manually managed journal entries.
In a typical software project we don't add generated files to version control, but in this case I think it is a good idea
to add all the generated files to version control as well - when you inevitably change something, e.g. how you classify transactions
in your rules file, then you can easily see if your change had the desired effect by looking at a diff.
** Detailed Step-By-Step Guide
Have a look at the [[file:docs/README.org][detailed step-by-step instructions]] and the files in the [[file:docs/][documentation directory]].
For a visual overview, check out the slide show version of the same step-by-step instructions:
https://pauley.org.za/hledger-makeitso/
You can see the example imported financial transactions as it was generated by the step-by-step
instructions here:
https://github.com/apauley/hledger-makeitso-example-finances
* Feature Reference
** Input Files
Your input files will probably be CSV files with a line for each transaction,
although other file types will work fine if you use a =preprocess= or a =construct= script that can read them.
These scripts are explained later.
We mostly use conventions based on a predefined directory structure for your input statements.
For example, assuming you have a =savings= account at =mybank=, you'll put your first CSV statement here:
=import/john/mybank/savings/1-in/2018/123456789_2018-06-30.csv=.
Some people may want to include accounts belonging to their spouse as part of the household finances:
=import/spouse/otherbank/checking/1-in/2018/987654321_2018-06-30.csv=.
*** More About Input Files
All files and directories under the =import= directory is related to the automatic importing and classification of transactions.
The directory directly under =import= is meant to indicate the owner or custodian of the accounts below it.
It mostly has an impact on reporting. You may want to have separate reports for =import/mycompany= and =import/personal=.
Below the directory for the owner we can indicate where an account is held.
For a bank account you may choose to name it =import/john/mybank=.
If your underground bunker filled with gold has CSV statements linked to it,
then you can absolutely create =import/john/secret-treasure-room=.
Under the directory for the financial institution, you'll have a directory for each account
at that institution, e.g. =import/mycompany/bigbankinc/customer-deposits= and
=import/mycompany/bigbankinc/expense-account=.
Next you'll create a directory named =1-in=.
This is to distinguish it from =2-preprocessed= and =3-journal= which will be auto-generated later.
Under =1-in= you'll create a directory for the year, e.g. =2018=, and within that you can copy the statements for that year:
=import/john/mybank/savings/1-in/2018/123456789_2018-06-30.csv=
** Rules Files
If your input file is in CSV format, or converted to CSV by your =preprocess= script,
then you'll need an [[http://hledger.org/csv.html][hledger rules file]].
=hledger-makeitso= will try to find a rules file for each statement in a few places.
The same rules file is typically used for all statements of a specific account, or even for all accounts
of the same specific bank.
- A global rules file for any =mybank= statement can be saved here: =import/mybank.rules=
- A rules file for all statements of a specific account: =import/spouse/bigbankinc/savings/bigbankinc-savings.rules=
*** Statement-specific Rules Files
What happens if some of the statements for an account has a different format than the others?
This can happen if you normally get your statements directly from your bank, but some
statements you had to download from somewhere else, like Mint, because your bank is being daft
with older statements.
In order to tell =hledger-makeitso= that you want to override the rules file for a specific statement,
you need to add a suffix, separated by an underscore (=_=) and starting with the letters =rfo= (rules file override)
to the filename of that statement.
For example: assuming you've named your statement =99966633_20171223_1844_rfo-mint.csv=.
=hledger-makeitso= will look for a rules file named =rfo-mint.rules= in the following places:
- in the import directory, e.g. =import/rfo-mint.rules=
- in the bank directory, e.g. =import/john/mybank/rfo-mint.rules=
- in the account directory, e.g. =import/john/mybank/savings/rfo-mint.rules=
*** Example rules file usage
A common scenario is multiple accounts that share the same file format, but have different =account1= directives.
One possible approach would be to include a shared rules file in your account-specific rules file.
If you are lucky enough that all statements at =mybank= share a common format across all accounts,
then you can =include= a rules file that just defines the parts that are shared across accounts.
Two accounts at =mybank= may have rules files similar to these.
A checking account at mybank:
#+BEGIN_SRC hledger
# Saved as: import/john/mybank/checking/mybank-checking.rules
include ../../../mybank-shared.rules
account1 Assets:Current:John:MyBank:Checking
#+END_SRC
Another account at mybank:
#+BEGIN_SRC hledger
# Saved as: import/alice/mybank/savings/mybank-savings.rules
include ../../../mybank-shared.rules
account1 Assets:Current:Alice:MyBank:Savings
#+END_SRC
Where =import/mybank-shared.rules= may define some shared attributes:
#+BEGIN_SRC hledger
skip 1
fields date, description, amount, balance
date-format %Y-%m-%d
currency $
#+END_SRC
Another possible approach could be to use your =preprocess= script to write out a CSV file
that has extra fields for =account1= and =account2=.
You could then create the above mentioned global =import/mybank.rules= with the fields defined more or less like this:
#+BEGIN_SRC hledger
fields date, description, amount, balance, account1, account2
#+END_SRC
** Opening and Closing Balances
*** Opening Balances
=hledger-makeitso= looks for a file named =YEAR-opening.journal= in each account directory, where =YEAR=
corresponds to an actual year directory, eg. *1983*
(if you have electronic statements [[https://en.wikipedia.org/wiki/Online_banking#First_online_banking_services_in_the_United_States][dating back to 1983]]).
Example: =import/john/mybank/savings/1983-opening.journal=
If it exists the file will automatically be included at the beginning of the generated journal include file for that year.
You need to edit this file for each account to specify the opening balance at the date of the first available transaction.
An opening balance may look something like this:
#+BEGIN_SRC hledger
2018-06-01 Savings Account Opening Balance
assets:Current:MyBank:Savings $102.01
equity:Opening Balances:MyBank:Savings
#+END_SRC
*** Closing Balances
Similar to opening balances, =hledger-makeitso= looks for an optional file named =YEAR-closing.journal= in each account directory.
Example: =import/john/mybank/savings/1983-closing.journal=
If it exists the file will automatically be included at the end of the generated journal include file for that year.
A closing balance may look something like this:
#+BEGIN_SRC hledger
2018-06-01 Savings Account Closing Balance
assets:Current:MyBank:Savings $-234.56 = $0.00
equity:Closing Balances:MyBank:Savings
#+END_SRC
*** Example Opening and Closing Journal Files
As an example, assuming that the relevant year is =2019= and =hledger-makeitso= is about to generate
=import/john/mybank/savings/2019-include.journal=, then one or both of the following files will be added to the include file if they exist:
1. =import/john/mybank/savings/2019-opening.journal=
2. =import/john/mybank/savings/2019-closing.journal=
The =opening.journal= will be included just before the other included entries, while the =closing.journal=
will be included just after the other entries in that include file.
An include file may look like this:
#+BEGIN_SRC sh
cat import/john/mybank/savings/2019-include.journal
#+END_SRC
#+BEGIN_SRC hledger
### Generated by hledger-makeitso - DO NOT EDIT ###
!include 2019-opening.journal
!include 3-journal/2019/123456789_2019-01-30
!include 2019-closing.journal
#+END_SRC
** The =preprocess= Script
Sometimes the statements you get from your bank is [[https://github.com/apauley/fnb-csv-demoronizer][less than suitable]] for automatic processing.
Or maybe you just want to make it easier for the hledger rules file to do its thing by adding some useful columns.
If you put a script called =preprocess= in the account directory, e.g. =import/john/mybank/savings/preprocess=,
then =hledger-makeitso= will call that script for each input statement.
The =preprocess= script will be called with 4 positional parameters:
1. The path to the input statement, e.g. =import/john/mybank/savings/1-in/2018/123456789_2018-06-30.csv=
2. The path to an output file that can be sent to =hledger=, e.g. =import/john/mybank/savings/2-preprocessed/2018/123456789_2018-06-30.csv=
3. The name of the bank, e.g. =mybank=
4. The name of the account, e.g. =savings=
5. The name of the owner, e.g. =john=
Your =preprocess= script is expected to:
- read the input file
- write a new output file at the supplied path that works with your rules file
- be idempotent. Running =preprocess= multiple times on the same files will produce the same result.
** The =construct= Script
If you need even more power and flexibility than what you can get from the =preprocess= script and =hledger='s CSV import functionality,
then you can create your own custom script to =construct= transactions exactly as you need them.
At the expense of more construction work for you, of course.
As an example, =hledger='s CSV import currently [[https://github.com/simonmichael/hledger/issues/627][only supports two postings per transaction]], even though =hledger= itself
is perfectly happy with transactions containing more than two postings, e.g.:
#+BEGIN_SRC hledger
2019-02-01 Mortgage Payment
Liabilities:Mortgage 1,000.00
Expenses:Interest:Real Estate 833.33
Assets:Cash -1833.33
#+END_SRC
The =construct= script can be used in addition to the =preprocess= script, or on it's own.
But since the =construct= script is more powerful than the =preprocess= script, you could tell your =construct= script to do
anything that the =preprocess= script would have done.
Save your =construct= script in the account directory, e.g. =import/john/mybank/savings/construct=.
=hledger-makeitso= will call your =construct= script with 4 positional parameters:
1. The path to the input statement, e.g. =import/john/mybank/savings/1-in/2018/123456789_2018-06-30.csv=
2. A "-" (indicating that output should be sent to =stdout=)
3. The name of the bank, e.g. =mybank=
4. The name of the account, e.g. =savings=
5. The name of the owner, e.g. =john=
Your =construct= script is expected to:
- read the input file
- generate your own =hledger= journal transactions
- be idempotent. Running =construct= multiple times on the same files should produce the same result.
- send all output to =stdout=. =hledger-makeitso= will pipe your output into =hledger= which will format it and save it to an output file.
** Manually Managed Journals
Not every transaction in your life comes with CSV statements.
Sometimes you just need to add a transaction for that time you loaned a friend some money.
=hledger-makeitso= looks for =pre-import= and =post-import= files related to each generated include file as part of the import.
You can enter your own transactions manually into these files.
You can run =hledger-makeitso import --verbose= to see exactly which files are being looked for.
As an example, assuming that the relevant year is =2019= and =hledger-makeitso= is about to generate
=import/john/2019-include.journal=, then one or both of the following files will be added to the include file if they exist:
1. =import/john/_manual_/2019/pre-import.journal=
2. =import/john/_manual_/2019/post-import.journal=
The =pre-import.journal= will be included just before the other included entries, while the =post-import.journal=
will be included just after the other entries in that include file.
An include file may look like this:
#+BEGIN_SRC sh
cat import/john/2019-include.journal
#+END_SRC
#+BEGIN_SRC hledger
### Generated by hledger-makeitso - DO NOT EDIT ###
!include _manual_/2019/pre-import.journal
!include mybank/2019-include.journal
!include otherbank/2019-include.journal
!include _manual_/2019/post-import.journal
#+END_SRC
* Compatibility with Ledger
When writing out the journal include files, =hledger-makeitso= sorts the include statements by filename.
[[https://www.ledger-cli.org/][Ledger]] fails any balance assertions when the transactions aren't included in chronological order.
An easy way around this is to name your input files so that March's statement is listed before December's statement.
Another option is to add =--permissive= to any [[https://www.ledger-cli.org/][ledger]] command.
So you should easily be able to use both =ledger= and =hledger= on these journals.
* Project Goals
My =hledger= files started to collect a bunch of supporting code that weren't really
specific to my financial situation.
I want to extract and share as much as possible of that supporting code.
Adept's goals also resonated with me:
- Tracking expenses should take as little time, effort and manual work as possible
- Eventual consistency should be achievable: even if I can't record something precisely right now,
maybe I would be able to do it later, so I should be able to leave things half-done and pick them up later
- Ability to refactor is a must. I want to be able to go back and change the way I am doing things,
with as little effort as possible and without fear of irrevocably breaking things.
* Still To Be Done
I add ideas and thoughts in [[file:TODO.org][TODO.org]]
Let me know if you can think of some improvements.