hledger-flow 0.14.4 → 0.16.1
raw patch · 29 files changed
+1691/−1851 lines, 29 filesPVP ok
version bump matches the API change (PVP)
API changes (from Hackage documentation)
+ Hledger.Flow.Common: directivesFile :: HasBaseDir o => o -> TurtlePath
+ Hledger.Flow.RuntimeOptions: [prettyReports] :: RuntimeOptions -> Bool
+ Hledger.Flow.RuntimeOptions: instance Hledger.Flow.Types.HasPrettyReports Hledger.Flow.RuntimeOptions.RuntimeOptions
+ Hledger.Flow.Types: class HasPrettyReports a
+ Hledger.Flow.Types: prettyReports :: HasPrettyReports a => a -> Bool
- Hledger.Flow.Common: logTimedAction :: HasVerbosity o => o -> TChan LogMessage -> Text -> [Text] -> (TChan LogMessage -> Text -> IO ()) -> (TChan LogMessage -> Text -> IO ()) -> IO FullOutput -> IO FullTimedOutput
+ Hledger.Flow.Common: logTimedAction :: HasVerbosity o => o -> TChan LogMessage -> Text -> [Text] -> Text -> [Text] -> (TChan LogMessage -> Text -> IO ()) -> (TChan LogMessage -> Text -> IO ()) -> IO FullOutput -> IO FullTimedOutput
- Hledger.Flow.RuntimeOptions: RuntimeOptions :: BaseDir -> RunDir -> Maybe Integer -> Bool -> Text -> HledgerInfo -> SystemInfo -> Bool -> Bool -> Bool -> Int -> RuntimeOptions
+ Hledger.Flow.RuntimeOptions: RuntimeOptions :: BaseDir -> RunDir -> Maybe Integer -> Bool -> Text -> HledgerInfo -> SystemInfo -> Bool -> Bool -> Bool -> Int -> Bool -> RuntimeOptions
Files
- ChangeLog.md +29/−1
- README.org +35/−568
- app/Main.hs +8/−3
- hledger-flow.cabal +6/−4
- src/Hledger/Flow/BaseDir.hs +29/−27
- src/Hledger/Flow/Common.hs +116/−75
- src/Hledger/Flow/DateTime.hs +2/−2
- src/Hledger/Flow/DocHelpers.hs +2/−2
- src/Hledger/Flow/Import/CSVImport.hs +116/−76
- src/Hledger/Flow/Import/ImportHelpers.hs +20/−17
- src/Hledger/Flow/Import/ImportHelpersTurtle.hs +60/−53
- src/Hledger/Flow/Import/Types.hs +11/−8
- src/Hledger/Flow/Internals.hs +33/−22
- src/Hledger/Flow/Logging.hs +5/−7
- src/Hledger/Flow/PathHelpers.hs +40/−32
- src/Hledger/Flow/Reports.hs +55/−53
- src/Hledger/Flow/RuntimeOptions.hs +25/−21
- src/Hledger/Flow/Types.hs +13/−6
- test/BaseDir/Integration.hs +28/−27
- test/CSVImport/ImportHelperTests.hs +91/−95
- test/CSVImport/ImportHelperTurtleTests.hs +532/−374
- test/CSVImport/Integration.hs +240/−191
- test/CSVImport/Unit.hs +1/−2
- test/Common/Integration.hs +41/−40
- test/Common/Unit.hs +43/−40
- test/PathHelpers/Unit.hs +16/−15
- test/Spec.hs +7/−8
- test/TestHelpers.hs +48/−48
- test/TestHelpersTurtle.hs +39/−34
ChangeLog.md view
@@ -1,5 +1,33 @@ # Changelog for [hledger-flow](https://github.com/apauley/hledger-flow) +## 0.16.1++- Fix documentation URL: https://github.com/apauley/hledger-flow/tree/master/docs#feature-reference++## 0.16.0++- Switched back to `hledger print` during statement import [#126](https://github.com/apauley/hledger-flow/issues/126)+- Move feature reference to https://github.com/apauley/hledger-flow/docs+- Fix preprocessing logic for CSV files [#123](https://github.com/apauley/hledger-flow/issues/123)+- Add --ascii-reports flag [#115](https://github.com/apauley/hledger-flow/pull/115)++Other changes:+ - Switched to Stackage lts-24.8 (GHC 9.10.2)++## 0.15.0++Made some changes that will result in formatting changes of generated files:++ - Removed the obsolete exclamation mark from the `include` directive+ - Switched from `hledger print` to `hledger import` during statement import.++ `hledger import` uses your preferred commodity styles from your+ [directives.journal](https://github.com/apauley/hledger-flow#hledger-directives) to generate journals.++Other changes:++ - Switched to GHC 9.0.1+ ## 0.14.4 Add an option to process files in batches.@@ -86,7 +114,7 @@ The use case has been described in [issue 64](https://github.com/apauley/hledger-flow/issues/64). It is experimental, because the only problem it currently solves is getting hledger-flow to fail fast.-One of the current side effects of doing so is that the generated include files are then written to only +One of the current side effects of doing so is that the generated include files are then written to only include the subset of files that were processed. But as soon as you do a full run again, the include files will be correctly re-generated as before.
README.org view
@@ -1,15 +1,18 @@ #+STARTUP: showall -* Hledger Flow+* hledger-flow :PROPERTIES: :CUSTOM_ID: hledger-flow :END: -| TravisCI | CircleCI |-|--------------------------------------------------------------+------------------------------------------------------------|-| [[https://travis-ci.com/apauley/hledger-flow][https://travis-ci.com/apauley/hledger-flow.svg?branch=master]] | [[https://circleci.com/gh/apauley/hledger-flow][https://circleci.com/gh/apauley/hledger-flow.svg?style=svg]] |+** Project Status -* What is it?+I haven't been able to spend much time on this project in the last few years,+and looking at my current responsibilities this is likely to remain the case.++I do still try to respond to bug reports as they happen.++** What is it? :PROPERTIES: :CUSTOM_ID: what-is-it :END:@@ -26,7 +29,7 @@ a statement whenever one is available. Within =hledger-flow= you will keep your original bank statements around-permanently as input, and generate classified Hledger journals each time+permanently as input, and generate (h)ledger journals each time you run the program. The classification is done with [[https://hledger.org/csv.html][hledger's rules files]], and/or your own script hooks. @@ -40,7 +43,7 @@ [[https://github.com/adept/full-fledged-hledger/wiki][adept's Full-fledged Hledger]] aren't really specific to my own finances, and can be shared. -* Overview of the Basic Workflow+** Overview of the Basic Workflow :PROPERTIES: :CUSTOM_ID: overview-of-the-basic-workflow :END:@@ -64,45 +67,34 @@ you can easily see if your change had the desired effect by looking at a diff. -* Who should use this?+** Who should use this? :PROPERTIES: :CUSTOM_ID: who-should-use-this :END: =hledger-flow= is intended for you if: -- You are interested in getting started with- [[http://hledger.org/][hledger]] or [[https://www.ledger-cli.org/][ledger-cli]] and you wouldn't mind pointers to the- right docs along the way. - You want a way to organise your finances into a structure that will be maintainable over the long term.+- You like the idea of treating your source transactions (typically CSV files)+ as input, and having your hledger journals (mostly) being generated as output. - You want to automate as much as possible when dealing with your financial life. - You don't mind writing some scripts when needed, as long as it saves you time over the long term.-- You want the ability to model your entire financial life in one tool,- as opposed to just the parts that some online tool currently supports.-- You appreciate the fact that all your financial information stays- within your control. -* About This Documentation--The rest of this file documents how to use =hledger-flow=, and it has probably outgrown what should be in a README file.--If you can spare some time to contribute to this project, please consider converting these docs to something more suitable,-such as [[https://github.com/apauley/hledger-flow/issues/47][Read the Docs]].--* How do I install it?+** How do I install it? :PROPERTIES: :CUSTOM_ID: how-do-i-install-it :END: -The easiest way to get it running is to download [[https://github.com/apauley/hledger-flow/releases][the latest release]]-for your OS (Linux or Mac OS X), and copy the =hledger-flow= executable to a directory in your PATH.-Then just run it and see what it tells you to do.+If you can compile it yourself, please do so by following the [[https://github.com/apauley/hledger-flow/blob/master/CONTRIBUTING.org#build-the-project][build instructions]]. -You can also compile it yourself by following the [[https://github.com/apauley/hledger-flow/blob/master/CONTRIBUTING.org#build-the-project][build instructions]].+Otherwise download [[https://github.com/apauley/hledger-flow/releases][the latest release]] for your OS (Linux or Mac OS X), and copy the =hledger-flow= executable to a directory in your PATH. +On Linux you should just be able to run the executable.+On Mac you may see a warning.+ ** Windows Support Currently =hledger-flow= does not work on Windows.@@ -114,535 +106,36 @@ Please send me some pull requests if you would like =hledger-flow= to work on Windows. -* How Stable is it?- :PROPERTIES:- :CUSTOM_ID: how-stable-is-it- :END:--We're not close to a 1.0 release yet, which means that we can still make-changes if needed.--As an example, the command-line switches we use will probably change over time.-Some switches change the behaviour of the program - the default behaviour will probably change between releases.-The names of these command-line options can change, or they can be removed when it is no longer needed.--That being said, some parts have been used and tested extensively and-are likely to remain stable. Have a look at the "Stability of this-Feature" sections in the feature reference below.--I add future work, ideas and thoughts as-[[https://github.com/apauley/hledger-flow/issues][Github issues]] and in-[[file:TODO.org]], so have a look there for more clues as to what may-likely change.--Let me know if you can think of some improvements.--* Getting Started+** Getting Started :PROPERTIES: :CUSTOM_ID: getting-started :END: -Have a look at the [[file:docs/README.org][detailed step-by-step-instructions]].--For a visual overview, check out the slide show version of the same-step-by-step instructions:--[[https://pauley.org.za/hledger-flow/][https://pauley.org.za/hledger-flow/]]+Have a look at the [[file:step-by-step/README.org][detailed step-by-step instructions]] and the [[file:docs/README.org][feature reference]]. You can see the example imported financial transactions as it was generated by the step-by-step instructions here: [[https://github.com/apauley/hledger-flow-example][https://github.com/apauley/hledger-flow-example]] -* Feature Reference- :PROPERTIES:- :CUSTOM_ID: feature-reference- :END:--** Input Files- :PROPERTIES:- :CUSTOM_ID: input-files- :END:--Your input files will probably be CSV files with a line for each-transaction, although other file types will work fine if you use a-=preprocess= or a =construct= script that can read them. These scripts-are explained later.--We mostly use conventions based on a predefined directory structure for-your input statements.--For example, assuming you have a =savings= account at =mybank=, you'll-put your first CSV statement here:-=import/john/mybank/savings/1-in/2018/123456789_2018-06-30.csv=.--Some people may want to include accounts belonging to their spouse as-part of the household finances:-=import/spouse/otherbank/checking/1-in/2018/987654321_2018-06-30.csv=.--*** More About Input Files- :PROPERTIES:- :CUSTOM_ID: more-about-input-files- :END:--All files and directories under the =import= directory is related to the-automatic importing and classification of transactions.--The directory directly under =import= is meant to indicate the owner or-custodian of the accounts below it. It mostly has an impact on-reporting. You may want to have separate reports for =import/mycompany=-and =import/personal=.--Below the directory for the owner we can indicate where an account is-held. For a bank account you may choose to name it =import/john/mybank=.--If your underground bunker filled with gold has CSV statements linked to-it, then you can absolutely create =import/john/secret-treasure-room=.--Under the directory for the financial institution, you'll have a-directory for each account at that institution, e.g.-=import/mycompany/bigbankinc/customer-deposits= and-=import/mycompany/bigbankinc/expense-account=.--Next you'll create a directory named =1-in=. This is to distinguish it-from =2-preprocessed= and =3-journal= which will be auto-generated-later.--Under =1-in= you'll create a directory for the year, e.g. =2018=, and-within that you can copy the statements for that year:-=import/john/mybank/savings/1-in/2018/123456789_2018-06-30.csv=--*** Stability of this Feature- :PROPERTIES:- :CUSTOM_ID: stability-of-this-feature- :END:--The basic owner/bank/account/year structure has been used and tested-fairly extensively, I don't expect a need for it to change.--I'm open to suggestions for improvement though.--** Rules Files- :PROPERTIES:- :CUSTOM_ID: rules-files- :END:--If your input file is in CSV format, or converted to CSV by your-=preprocess= script, then you'll need an-[[http://hledger.org/csv.html][hledger rules file]].--=hledger-flow= will try to find a rules file for each statement in a few-places. The same rules file is typically used for all statements of a-specific account, or even for all accounts of the same specific bank.--- A global rules file for any =mybank= statement can be saved here:- =import/mybank.rules=-- A rules file for all statements of a specific account:- =import/spouse/bigbankinc/savings/bigbankinc-savings.rules=--*** Statement-specific Rules Files- :PROPERTIES:- :CUSTOM_ID: statement-specific-rules-files- :END:--What happens if some of the statements for an account has a different-format than the others?--This can happen if you normally get your statements directly from your-bank, but some statements you had to download from somewhere else, like-Mint, because your bank is being daft with older statements.--In order to tell =hledger-flow= that you want to override the rules file-for a specific statement, you need to add a suffix, separated by an-underscore (=_=) and starting with the letters =rfo= (rules file-override) to the filename of that statement.--For example: assuming you've named your statement-=99966633_20171223_1844_rfo-mint.csv=.--=hledger-flow= will look for a rules file named =rfo-mint.rules= in the-following places:--- in the import directory, e.g. =import/rfo-mint.rules=-- in the bank directory, e.g. =import/john/mybank/rfo-mint.rules=-- in the account directory, e.g.- =import/john/mybank/savings/rfo-mint.rules=--*** Example rules file usage- :PROPERTIES:- :CUSTOM_ID: example-rules-file-usage- :END:--A common scenario is multiple accounts that share the same file format,-but have different =account1= directives.--One possible approach would be to include a shared rules file in your-account-specific rules file.--If you are lucky enough that all statements at =mybank= share a common-format across all accounts, then you can =include= a rules file that-just defines the parts that are shared across accounts.--Two accounts at =mybank= may have rules files similar to these.--A checking account at mybank:--#+BEGIN_EXAMPLE-# Saved as: import/john/mybank/checking/mybank-checking.rules-include ../../../mybank-shared.rules-account1 Assets:Current:John:MyBank:Checking-#+END_EXAMPLE--Another account at mybank:--#+BEGIN_EXAMPLE-# Saved as: import/alice/mybank/savings/mybank-savings.rules-include ../../../mybank-shared.rules-account1 Assets:Current:Alice:MyBank:Savings-#+END_EXAMPLE--Where =import/mybank-shared.rules= may define some shared attributes:--#+BEGIN_EXAMPLE- skip 1-- fields date, description, amount, balance-- date-format %Y-%m-%d- currency $-#+END_EXAMPLE--Another possible approach could be to use your =preprocess= script to-write out a CSV file that has extra fields for =account1= and-=account2=.--You could then create the above mentioned global =import/mybank.rules=-with the fields defined more or less like this:--#+BEGIN_EXAMPLE-fields date, description, amount, balance, account1, account2-#+END_EXAMPLE--*** Stability of this Feature- :PROPERTIES:- :CUSTOM_ID: stability-of-this-feature-1- :END:--Rules files are a stable feature within-[[http://hledger.org/][hledger]], and we're just using the normal-hledger rules files. The account, bank and statement-specific rules-files have been used and tested fairly extensively, I don't expect this-to change.--Let me know if you think it should change.--** Opening and Closing Balances- :PROPERTIES:- :CUSTOM_ID: opening-and-closing-balances- :END:--*** Opening Balances- :PROPERTIES:- :CUSTOM_ID: opening-balances- :END:--=hledger-flow= looks for a file named =YEAR-opening.journal= in each-account directory, where =YEAR= corresponds to an actual year directory,-eg. *1983* (if you have electronic statements-[[https://en.wikipedia.org/wiki/Online_banking#First_online_banking_services_in_the_United_States][dating-back to 1983]]). Example:-=import/john/mybank/savings/1983-opening.journal=--If it exists the file will automatically be included at the beginning of-the generated journal include file for that year.--You need to edit this file for each account to specify the opening-balance at the date of the first available transaction.--An opening balance may look something like this:--#+BEGIN_EXAMPLE-2018-06-01 Savings Account Opening Balance- assets:Current:MyBank:Savings $102.01- equity:Opening Balances:MyBank:Savings-#+END_EXAMPLE---*** A Note of Caution Regarding Closing Balances--When closing your balances it may result in some =hledger= queries showing zero-values, or there could be issues with balance assertions.--Please have a look at the upstream =hledger= documentation on closing balances, e.g here:-https://hledger.org/hledger.html#close-usage--Some of the gotchas you may run into are also described in [[https://github.com/apauley/hledger-flow/issues/79][this hledger-flow issue]].--*** Closing Balances- :PROPERTIES:- :CUSTOM_ID: closing-balances- :END:--Similar to opening balances, =hledger-flow= looks for an optional file-named =YEAR-closing.journal= in each account directory. Example:-=import/john/mybank/savings/1983-closing.journal=--If it exists the file will automatically be included at the end of the-generated journal include file for that year.--A closing balance may look something like this:--#+BEGIN_EXAMPLE-2018-06-01 Savings Account Closing Balance- assets:Current:MyBank:Savings $-234.56 = $0.00- equity:Closing Balances:MyBank:Savings-#+END_EXAMPLE--*** Example Opening and Closing Journal Files- :PROPERTIES:- :CUSTOM_ID: example-opening-and-closing-journal-files- :END:--As an example, assuming that the relevant year is =2019= and-=hledger-flow= is about to generate-=import/john/mybank/savings/2019-include.journal=, then one or both of-the following files will be added to the include file if they exist:--1. =import/john/mybank/savings/2019-opening.journal=-2. =import/john/mybank/savings/2019-closing.journal=--The =opening.journal= will be included just before the other included-entries, while the =closing.journal= will be included just after the-other entries in that include file.--An include file may look like this:--#+BEGIN_SRC sh- cat import/john/mybank/savings/2019-include.journal-#+END_SRC--#+BEGIN_EXAMPLE- ### Generated by hledger-flow - DO NOT EDIT ###-- !include 2019-opening.journal- !include 3-journal/2019/123456789_2019-01-30- !include 2019-closing.journal-#+END_EXAMPLE--*** Stability of this Feature- :PROPERTIES:- :CUSTOM_ID: stability-of-this-feature-2- :END:--Closing balances sometimes result in [[https://github.com/apauley/hledger-flow/issues/79][unexpected query results]].-In future we may change how/where the generated files include the closing journal.--We may also need to suggest some naming conventions for opening and closing balances so that reports can exclude-some of these transactions.--It is also possible that we might want to change the name/location of the closing journal,-but we'll try to avoid this if possible, because that would require users to rename their existing files.--** Price Files- :PROPERTIES:- :CUSTOM_ID: price-files- :END:--=hledger-flow= looks for [[https://hledger.org/journal.html#market-prices][price files]] to include in each yearly include file.--For example, the presence of a file named =${BASE}/prices/2020/prices.journal= will result in some extra include file magic.--The rest of this section assumes you'll have a file named =prices/2020/prices.journal= which contains price data for the year 2020.-The =prices= directory should be right at the top of your =hledger-flow= base directory, next to the =import= directory.--=hledger-flow= does not care how the price files got there, it only cares that you should have a separate file per year,-and that it follows the above naming convention.--Here is an example script which downloads prices and follows the naming convention:-https://gist.github.com/apauley/398fa031c202733959af76b3b8ce8197--After running an import with available price files you'll see a line has been added to =import/2020-include.journal=:--#+BEGIN_EXAMPLE-!include ../prices/2020/prices.journal-#+END_EXAMPLE--** The =preprocess= Script- :PROPERTIES:- :CUSTOM_ID: the-preprocess-script- :END:--Sometimes the statements you get from your bank is-[[https://github.com/apauley/fnb-csv-demoronizer][less than suitable]]-for automatic processing. Or maybe you just want to make it easier for-the hledger rules file to do its thing by adding some useful columns.--If you put a script called =preprocess= in the account directory, e.g.-=import/john/mybank/savings/preprocess=, then =hledger-flow= will call-that script for each input statement.--The =preprocess= script will be called with 4 positional parameters:--1. The path to the input statement, e.g.- =import/john/mybank/savings/1-in/2018/123456789_2018-06-30.csv=-2. The path to an output file that can be sent to =hledger=, e.g.- =import/john/mybank/savings/2-preprocessed/2018/123456789_2018-06-30.csv=-3. The name of the bank, e.g. =mybank=-4. The name of the account, e.g. =savings=-5. The name of the owner, e.g. =john=--Your =preprocess= script is expected to:--- read the input file-- write a new output file at the supplied path that works with your- rules file-- be idempotent. Running =preprocess= multiple times on the same files- will produce the same result.--*** Stability of this Feature- :PROPERTIES:- :CUSTOM_ID: stability-of-this-feature-3- :END:--Stable and tested.--** The =construct= Script- :PROPERTIES:- :CUSTOM_ID: the-construct-script- :END:--If you need even more power and flexibility than what you can get from-the =preprocess= script and =hledger='s [[https://hledger.org/csv.html][CSV import functionality]], then-you can create your own custom script to =construct= transactions-exactly as you need them.--At the expense of more construction work for you, of course.--The =construct= script can be used in addition to the =preprocess=-script, or on it's own. But since the =construct= script is more-powerful than the =preprocess= script, you could tell your =construct=-script to do anything that the =preprocess= script would have done.--Save your =construct= script in the account directory, e.g.-=import/john/mybank/savings/construct=.--=hledger-flow= will call your =construct= script with 5 positional-parameters:--1. The path to the input statement, e.g.- =import/john/mybank/savings/1-in/2018/123456789_2018-06-30.csv=-2. A "-" (indicating that output should be sent to =stdout=)-3. The name of the bank, e.g. =mybank=-4. The name of the account, e.g. =savings=-5. The name of the owner, e.g. =john=--Your =construct= script is expected to:--- read the input file-- generate your own =hledger= journal transactions-- be idempotent. Running =construct= multiple times on the same files- should produce the same result.-- send all journals to =stdout=. =hledger-flow= will pipe your standard output into- =hledger= which will format it and save it to an output file.--You can still use =stderr= in your construct script for any other output that you may want to see.--*** Stability of this Feature- :PROPERTIES:- :CUSTOM_ID: stability-of-this-feature-4- :END:--Stable and tested.--** Manually Managed Journals- :PROPERTIES:- :CUSTOM_ID: manually-managed-journals- :END:--Not every transaction in your life comes with CSV statements.--Sometimes you just need to add a transaction for that time you loaned a-friend some money.--=hledger-flow= looks for =pre-import= and =post-import= files related to-each generated include file as part of the import.--You can enter your own transactions manually into these files.--You can run =hledger-flow import --verbose= to see exactly which files-are being looked for.--As an example, assuming that the relevant year is =2019= and-=hledger-flow= is about to generate =import/john/2019-include.journal=,-then one or both of the following files will be added to the include-file if they exist:--1. =import/john/_manual_/2019/pre-import.journal=-2. =import/john/_manual_/2019/post-import.journal=--The =pre-import.journal= will be included just before the other included-entries, while the =post-import.journal= will be included just after the-other entries in that include file.--An include file may look like this:--#+BEGIN_SRC sh- cat import/john/2019-include.journal-#+END_SRC--#+BEGIN_EXAMPLE- ### Generated by hledger-flow - DO NOT EDIT ###-- !include _manual_/2019/pre-import.journal- !include mybank/2019-include.journal- !include otherbank/2019-include.journal- !include _manual_/2019/post-import.journal-#+END_EXAMPLE--*** Stability of this Feature- :PROPERTIES:- :CUSTOM_ID: stability-of-this-feature-5- :END:--It works, but the naming of =_manual_= looks a bit weird. Should it be-changed?--* Validating an hledger repository using Github Actions--The following example was contributed by [[https://github.com/amitaibu][Amitai Burstein]]:--#+BEGIN_SRC yaml-# .github/workflows/hledger-flow.yml--name: Validate hledger-flow--on: [push]--jobs:- build:-- runs-on: ubuntu-latest- steps:- - uses: actions/checkout@v1-- - name: Install hledger- run: docker pull dastapov/hledger+** Compatibility with hledger - - name: Install hledger-flow- run: curl -L https://github.com/apauley/hledger-flow/releases/download/v0.12.4.0/hledger-flow_Linux_x86_64_v0.12.4.0_4b9b027.tar.gz | tar xvz && mv hledger-flow_Linux_x86_64_v0.12.4.0_4b9b027/hledger-flow .+=hledger-flow= should work with any recent version of =hledger=. - - name: Grant permissions to create files- run: chmod 777 -R ./my-finances+The most recent version that it was tested with is [[https://hledger.org/relnotes.html#2025-09-03-hledger-150][hledger-1.50]] (September 2025). - - name: Test hledger file- run: docker run --name="ledger" -v $(pwd):/data dastapov/hledger ./hledger-flow import ./my-finances-#+END_SRC yaml+Note to future readers: if you are using =hledger-flow= with a more recent version of =hledger= than mentioned above,+please submit a small pull request with the updated version of =hledger=. -* Compatibility with Ledger+** Compatibility with ledger-cli :PROPERTIES: :CUSTOM_ID: compatibility-with-ledger :END: +=hledger-flow= uses =hledger= to produce journal's, so this page about [[https://hledger.org/ledger.html][hledger and Ledger]] should be relevant for all =hledger-flow= users.++That said, here are some observations that are specifically relevant to =hledger-flow=:+ When writing out the journal include files, =hledger-flow= sorts the include statements by filename. @@ -656,9 +149,10 @@ [[https://www.ledger-cli.org/][ledger]] command. So you should easily be able to use both =ledger= and =hledger= on these-journals.+journals if you take care to [[https://hledger.org/faq.html#how-is-hledger-different-from-ledger-][avoid the few incompatibilities]] which exists+(eg in your rules files or manual journals). -* Project Goals+** Project Goals :PROPERTIES: :CUSTOM_ID: project-goals :END:@@ -680,11 +174,8 @@ the way I am doing things, with as little effort as possible and without fear of irrevocably breaking things. -I've given [[https://pauley.org.za/functional-finance-hledger/][a talk]] at-[[https://www.meetup.com/lambda-luminaries/events/qklkvpyxmbnb/][Lambda Luminaries Johannesburg]]-featuring hledger and hledger-flow. -* Contributing to Hledger Flow+* Contributing to hledger-flow Have a look at the [[file:CONTRIBUTING.org][contribution guidelines]]. @@ -770,27 +261,3 @@ [[https://github.com/adept/full-fledged-hledger/wiki/Adding-more-accounts#lets-make-sure-that-transfers-are-not-double-counted]] The question was first asked in [[https://github.com/apauley/hledger-flow/issues/51][issue #51]].--** How does =hledger-flow= differ from =Full-fledged Hledger=?- :PROPERTIES:- :CUSTOM_ID: how-does-hledger-flow-differ-from-full-fledged-hledger- :END:--[[https://github.com/adept/full-fledged-hledger/wiki#full-fledged-hledger-tutorial][Full-fledged Hledger]]-is a brilliant system, and hledger-flow continues to learn much from it.--It has great documentation that does an excellent job of not only-showing *how* things can be done, but also *why* it is such a great-idea.--hledger-flow can be seen as a specific implementation of the-Full-fledged Hledger system, with a few implementation details that are-different.--| Full-fledged Hledger | Hledger Flow |-|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-| FFH is a [[https://github.com/adept/full-fledged-hledger/wiki#full-fledged-hledger-tutorial][tutorial with helper scripts]] that you can start using and adapt to your needs. | I started with FFH, and changed bits and pieces over time to suit my needs. The "owner/bank/account" structure for example. |-| | |-| FFH is more open-ended: you can start with the basic scripts and over time turn it into something that solves your needs exactly. But you'll also end up with more code that you need to maintain yourself. | Hledger Flow is more opinionated and less open-ended. For example, you have to adopt the "owner/bank/account" structure precisely as specified. But this allows Hledger Flow to do more work for you. |-| | |-| FFH uses scripts and [[https://shakebuild.com/][Haskell/Shake build files]] that you can easily modify as you go along, but this requires a Haskell runtime to be installed everywhere it needs to run. The included docker image helps to make it less of an issue. | Hledger Flow [[https://github.com/apauley/hledger-flow/releases][distributes a compiled binary]]. This means users or deployment targets don't need extra dependencies installed, they can just run a CLI program. This also provides a clearer distinction between what is provided, and what users need to do. |
app/Main.hs view
@@ -38,7 +38,9 @@ , onlyNewFiles :: Bool } deriving (Show) -newtype ReportParams = ReportParams {maybeReportBaseDir :: Maybe TurtlePath} deriving Show+data ReportParams = ReportParams { maybeReportBaseDir :: Maybe TurtlePath+ , asciiReports :: Bool+ } deriving (Show) data Command = Import ImportParams | Report ReportParams deriving (Show) @@ -61,7 +63,7 @@ Command mainParams' (Report subParams) -> toRuntimeOptionsReport mainParams' subParams >>= generateReports defaultBatchSize :: Int-defaultBatchSize = 200+defaultBatchSize = 20 determineBatchSize :: MainParams -> IO Int determineBatchSize mainParams' =@@ -88,6 +90,7 @@ , RT.showOptions = showOpts mainParams' , RT.sequential = sequential mainParams' , RT.batchSize = size+ , RT.prettyReports = True } toRuntimeOptionsReport :: MainParams -> ReportParams -> IO RT.RuntimeOptions@@ -108,6 +111,7 @@ , RT.showOptions = showOpts mainParams' , RT.sequential = sequential mainParams' , RT.batchSize = size+ , RT.prettyReports = not(asciiReports subParams') } baseCommandParser :: Parser BaseCommand@@ -128,10 +132,11 @@ subcommandParserImport :: Parser ImportParams subcommandParserImport = ImportParams- <$> optional (Turtle.argPath "dir" "The directory to import. Use the base directory for a full import or a sub-directory for a partial import. Defaults to the current directory. This behaviour is changing: see --enable-future-rundir")+ <$> optional (Turtle.argPath "dir" "The directory to import. Use the base directory for a full import or a sub-directory for a partial import. Defaults to the current directory.") <*> optional (option str (long "start-year" <> metavar "YEAR" <> help "Import only from the specified year and onwards, ignoring previous years. By default all available years are imported. Valid values include a 4-digit year or 'current' for the current year")) <*> switch (long "new-files-only" <> help "Don't regenerate transaction files if they are already present. This applies to hledger journal files as well as files produced by the preprocess and construct scripts.") subcommandParserReport :: Parser ReportParams subcommandParserReport = ReportParams <$> optional (Turtle.argPath "basedir" "The hledger-flow base directory")+ <*> switch (long "ascii-reports" <> help "If to avoid using hledger --pretty-tables flag when generating reports.")
hledger-flow.cabal view
@@ -1,11 +1,13 @@ cabal-version: 1.12 --- This file has been generated from package.yaml by hpack version 0.34.4.+-- This file has been generated from package.yaml by hpack version 0.37.0. -- -- see: https://github.com/sol/hpack+--+-- hash: 9711cc7907045ee1194c92bdb910d265677ae1a32a1395e49da6dbd1099c92ca name: hledger-flow-version: 0.14.4+version: 0.16.1 synopsis: An hledger workflow focusing on automated statement import and classification. description: Please see the README on GitHub at <https://github.com/apauley/hledger-flow#readme> category: Finance, Console@@ -13,7 +15,7 @@ bug-reports: https://github.com/apauley/hledger-flow/issues author: Andreas Pauley <hledger-flow@pauley.org.za> maintainer: Andreas Pauley <hledger-flow@pauley.org.za>-copyright: 2020 Andreas Pauley+copyright: 2023 Andreas Pauley license: GPL-3 license-file: LICENSE build-type: Simple@@ -69,7 +71,7 @@ hs-source-dirs: app ghc-options: -threaded -rtsopts -with-rtsopts=-N -Wall -fPIC- ld-options: -static+ ld-options: -dynamic build-depends: base >=4.7 && <5 , hledger-flow
src/Hledger/Flow/BaseDir.hs view
@@ -1,28 +1,26 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE QuasiQuotes #-} -module Hledger.Flow.BaseDir (- determineBaseDir- , relativeToBase- , relativeToBase'- , turtleBaseDir- , effectiveRunDir-) where--import Path-import Path.IO-import Hledger.Flow.Types (HasBaseDir, BaseDir, RunDir, baseDir)-import Hledger.Flow.PathHelpers--import Data.Maybe+module Hledger.Flow.BaseDir+ ( determineBaseDir,+ relativeToBase,+ relativeToBase',+ turtleBaseDir,+ effectiveRunDir,+ )+where +import Control.Monad (when) import Control.Monad.Catch (MonadThrow, throwM) import Control.Monad.IO.Class (MonadIO)-import Control.Monad (when)--import qualified Turtle (liftIO, repr, stripPrefix)+import Data.Maybe import qualified Data.Text as T import qualified Data.Text.IO as T+import Hledger.Flow.PathHelpers+import Hledger.Flow.Types (BaseDir, HasBaseDir, RunDir, baseDir)+import Path+import Path.IO+import qualified Turtle (liftIO, repr, stripPrefix) determineBaseDir :: Maybe TurtlePath -> IO (BaseDir, RunDir) determineBaseDir suppliedDir = do@@ -35,15 +33,15 @@ determineBaseDirFromStartDir absDir determineBaseDir' pwd Nothing = determineBaseDirFromStartDir pwd -determineBaseDirFromStartDir :: AbsDir -> IO (BaseDir, RunDir)+determineBaseDirFromStartDir :: AbsDir -> IO (BaseDir, RunDir) determineBaseDirFromStartDir startDir = determineBaseDirFromStartDir' startDir startDir determineBaseDirFromStartDir' :: (MonadIO m, MonadThrow m) => AbsDir -> AbsDir -> m (BaseDir, RunDir) determineBaseDirFromStartDir' startDir possibleBaseDir = do Control.Monad.when (parent possibleBaseDir == possibleBaseDir) $ throwM (MissingBaseDir startDir) foundBaseDir <- doesDirExist $ possibleBaseDir </> [reldir|import|]- if foundBaseDir then- do+ if foundBaseDir+ then do runDir <- limitRunDir possibleBaseDir startDir return (possibleBaseDir, runDir) else determineBaseDirFromStartDir' startDir $ parent possibleBaseDir@@ -64,18 +62,22 @@ return newRunDir composeN :: Int -> (a -> a) -> (a -> a)-composeN n f | n < 1 = id- | n == 1 = f- | otherwise = composeN (n-1) (f . f)+composeN n f+ | n < 1 = id+ | n == 1 = f+ | otherwise = composeN (n - 1) (f . f) -relativeToBase :: HasBaseDir o => o -> TurtlePath -> TurtlePath+relativeToBase :: (HasBaseDir o) => o -> TurtlePath -> TurtlePath relativeToBase opts = relativeToBase' $ pathToTurtle (baseDir opts) relativeToBase' :: TurtlePath -> TurtlePath -> TurtlePath-relativeToBase' bd p = if forceTrailingSlash bd == forceTrailingSlash p then "./" else- fromMaybe p $ Turtle.stripPrefix (forceTrailingSlash bd) p+relativeToBase' bd p =+ if forceTrailingSlash bd == forceTrailingSlash p+ then "./"+ else+ fromMaybe p $ Turtle.stripPrefix (forceTrailingSlash bd) p -turtleBaseDir :: HasBaseDir o => o -> TurtlePath+turtleBaseDir :: (HasBaseDir o) => o -> TurtlePath turtleBaseDir opts = pathToTurtle $ baseDir opts effectiveRunDir :: BaseDir -> RunDir -> AbsDir
src/Hledger/Flow/Common.hs view
@@ -3,53 +3,48 @@ module Hledger.Flow.Common where -import Path (absfile, relfile)-import qualified Path.IO as Path--import qualified Turtle-import Turtle ((%), (</>), (<.>))--import Prelude hiding (putStrLn)-import qualified Data.Text as T-import qualified Data.Text.IO as T-import qualified Data.Text.Read as T-import qualified GHC.IO.Handle.FD as H-+import Control.Concurrent.STM+import qualified Control.Foldl as Fold import Data.Char (isDigit) import Data.Either--import qualified Control.Foldl as Fold-import qualified Data.Map.Strict as Map- import Data.Function (on)-import qualified Data.List as List (null, sortBy, groupBy)+import qualified Data.List as List (groupBy, null, sortBy)+import qualified Data.Map.Strict as Map import Data.Ord (comparing)--import Hledger.Flow.Types-+import qualified Data.Text as T+import qualified Data.Text.IO as T+import qualified Data.Text.Read as T+import qualified GHC.IO.Handle.FD as H+import Hledger.Flow.BaseDir (relativeToBase, turtleBaseDir) import Hledger.Flow.Logging import Hledger.Flow.PathHelpers (AbsFile, TurtlePath, fromTurtleAbsFile, pathToTurtle)-import Hledger.Flow.BaseDir (turtleBaseDir, relativeToBase)--import Control.Concurrent.STM+import Hledger.Flow.Types+import Path (absfile, relfile)+import qualified Path.IO as Path+import Turtle ((%), (<.>), (</>))+import qualified Turtle+import Prelude hiding (putStrLn, readFile, writeFile) hledgerPathFromOption :: Maybe TurtlePath -> IO AbsFile hledgerPathFromOption pathOption = do case pathOption of- Just h -> do+ Just h -> do hlAbs <- fromTurtleAbsFile h isOnDisk <- Path.doesFileExist hlAbs- if isOnDisk then return hlAbs else do- let msg = Turtle.format ("Unable to find hledger at "%Turtle.fp) h- errExit' 1 (T.hPutStrLn H.stderr) msg hlAbs+ if isOnDisk+ then return hlAbs+ else do+ let msg = Turtle.format ("Unable to find hledger at " % Turtle.fp) h+ errExit' 1 (T.hPutStrLn H.stderr) msg hlAbs Nothing -> do maybeH <- Path.findExecutable [relfile|hledger|] case maybeH of- Just h -> return h+ Just h -> return h Nothing -> do- let msg = "Unable to find hledger in your path.\n"- <> "You need to either install hledger, or add it to your PATH, or provide the path to an hledger executable.\n\n"- <> "There are a number of installation options on the hledger website: https://hledger.org/download.html"+ let msg =+ "Unable to find hledger in your path.\n"+ <> "You need to either install hledger, or add it to your PATH, or provide the path to an hledger executable.\n\n"+ <> "There are a number of installation options on the hledger website: https://hledger.org/download.html" errExit' 1 (T.hPutStrLn H.stderr) msg [absfile|/hledger|] hledgerVersionFromPath :: TurtlePath -> IO T.Text@@ -80,43 +75,84 @@ descriptiveOutput :: T.Text -> T.Text -> T.Text descriptiveOutput outputLabel outTxt = do if not (T.null outTxt)- then Turtle.format (Turtle.s%":\n"%Turtle.s%"\n") outputLabel outTxt+ then Turtle.format (Turtle.s % ":\n" % Turtle.s % "\n") outputLabel outTxt else "" -logTimedAction :: HasVerbosity o => o -> TChan LogMessage -> T.Text -> [T.Text]- -> (TChan LogMessage -> T.Text -> IO ()) -> (TChan LogMessage -> T.Text -> IO ())- -> IO FullOutput- -> IO FullTimedOutput-logTimedAction opts ch cmdLabel extraCmdLabels stdoutLogger stderrLogger action = do- logVerbose opts ch $ Turtle.format ("Begin: "%Turtle.s) cmdLabel+logTimedAction ::+ (HasVerbosity o) =>+ o ->+ TChan LogMessage ->+ T.Text ->+ [T.Text] ->+ T.Text ->+ [T.Text] ->+ (TChan LogMessage -> T.Text -> IO ()) ->+ (TChan LogMessage -> T.Text -> IO ()) ->+ IO FullOutput ->+ IO FullTimedOutput+logTimedAction opts ch cmdLabel extraCmdLabels cmd args stdoutLogger stderrLogger action = do+ logVerbose opts ch $ Turtle.format ("Begin: " % Turtle.s) cmdLabel+ logVerbose opts ch $ Turtle.format (Turtle.s % " " % Turtle.s) cmd (T.intercalate " " args) if (List.null extraCmdLabels) then return () else logVerbose opts ch $ T.intercalate "\n" extraCmdLabels timed@((ec, stdOut, stdErr), diff) <- Turtle.time action stdoutLogger ch stdOut stderrLogger ch stdErr- logVerbose opts ch $ Turtle.format ("End: "%Turtle.s%" "%Turtle.s%" ("%Turtle.s%")") cmdLabel (Turtle.repr ec) (Turtle.repr diff)+ logVerbose opts ch $ Turtle.format ("End: " % Turtle.s % " " % Turtle.s % " (" % Turtle.s % ")") cmdLabel (Turtle.repr ec) (Turtle.repr diff) return timed -timeAndExitOnErr :: (HasSequential o, HasVerbosity o) => o -> TChan LogMessage -> T.Text- -> (TChan LogMessage -> T.Text -> IO ()) -> (TChan LogMessage -> T.Text -> IO ())- -> ProcFun -> ProcInput- -> IO FullTimedOutput+timeAndExitOnErr ::+ (HasSequential o, HasVerbosity o) =>+ o ->+ TChan LogMessage ->+ T.Text ->+ (TChan LogMessage -> T.Text -> IO ()) ->+ (TChan LogMessage -> T.Text -> IO ()) ->+ ProcFun ->+ ProcInput ->+ IO FullTimedOutput timeAndExitOnErr opts ch cmdLabel = timeAndExitOnErr' opts ch cmdLabel [] -timeAndExitOnErr' :: (HasSequential o, HasVerbosity o) => o -> TChan LogMessage -> T.Text -> [T.Text]- -> (TChan LogMessage -> T.Text -> IO ()) -> (TChan LogMessage -> T.Text -> IO ())- -> ProcFun -> ProcInput- -> IO FullTimedOutput+timeAndExitOnErr' ::+ (HasSequential o, HasVerbosity o) =>+ o ->+ TChan LogMessage ->+ T.Text ->+ [T.Text] ->+ (TChan LogMessage -> T.Text -> IO ()) ->+ (TChan LogMessage -> T.Text -> IO ()) ->+ ProcFun ->+ ProcInput ->+ IO FullTimedOutput timeAndExitOnErr' opts ch cmdLabel extraCmdLabels stdoutLogger stderrLogger procFun (cmd, args, stdInput) = do let action = procFun cmd args stdInput- timed@((ec, stdOut, stdErr), _) <- logTimedAction opts ch cmdLabel extraCmdLabels stdoutLogger stderrLogger action+ timed@((ec, stdOut, stdErr), _) <- logTimedAction opts ch cmdLabel extraCmdLabels cmd args stdoutLogger stderrLogger action case ec of Turtle.ExitFailure i -> do- let cmdText = Turtle.format (Turtle.s%" "%Turtle.s) cmd $ showCmdArgs args+ let cmdText = Turtle.format (Turtle.s % " " % Turtle.s) cmd $ showCmdArgs args let msgOut = descriptiveOutput "Standard output" stdOut let msgErr = descriptiveOutput "Error output" stdErr - let exitMsg = Turtle.format ("\n=== Begin Error: "%Turtle.s%" ===\nExternal command:\n"%Turtle.s%"\nExit code "%Turtle.d%"\n"- %Turtle.s%Turtle.s%"=== End Error: "%Turtle.s%" ===\n") cmdLabel cmdText i msgOut msgErr cmdLabel+ let exitMsg =+ Turtle.format+ ( "\n=== Begin Error: "+ % Turtle.s+ % " ===\nExternal command:\n"+ % Turtle.s+ % "\nExit code "+ % Turtle.d+ % "\n"+ % Turtle.s+ % Turtle.s+ % "=== End Error: "+ % Turtle.s+ % " ===\n"+ )+ cmdLabel+ cmdText+ i+ msgOut+ msgErr+ cmdLabel errExit i ch exitMsg timed Turtle.ExitSuccess -> return timed @@ -125,7 +161,7 @@ ec <- Turtle.proc cmd args stdinput return (ec, T.empty, T.empty) -parAwareProc :: HasSequential o => o -> ProcFun+parAwareProc :: (HasSequential o) => o -> ProcFun parAwareProc opts = if (sequential opts) then procWithEmptyOutput else Turtle.procStrictWithErr parAwareActions :: (HasSequential o, HasBatchSize o) => o -> [IO a] -> IO [a]@@ -139,13 +175,12 @@ doneNow <- (Turtle.single . shellToList . Turtle.parallel) doNow parBatchedActions batch (done ++ doneNow) remaining - inprocWithErrFun :: (T.Text -> IO ()) -> ProcInput -> Turtle.Shell Turtle.Line inprocWithErrFun errFun (cmd, args, standardInput) = do result <- Turtle.inprocWithErr cmd args standardInput case result of Right ln -> return ln- Left ln -> do+ Left ln -> do (Turtle.liftIO . errFun . Turtle.lineToText) ln Turtle.empty @@ -154,13 +189,15 @@ fileExists <- Turtle.testfile p let rel = relativeToBase opts p if fileExists- then logVerbose opts ch $ Turtle.format ("Found '"%Turtle.fp%"'") rel- else logVerbose opts ch $ Turtle.format ("Looked for but did not find '"%Turtle.fp%"'") rel+ then logVerbose opts ch $ Turtle.format ("Found '" % Turtle.fp % "'") rel+ else logVerbose opts ch $ Turtle.format ("Looked for but did not find '" % Turtle.fp % "'") rel return fileExists groupPairs' :: (Eq a, Ord a) => [(a, b)] -> [(a, [b])]-groupPairs' = map (\ll -> (fst . head $ ll, map snd ll)) . List.groupBy ((==) `on` fst)- . List.sortBy (comparing fst)+groupPairs' =+ map (\ll -> (fst . head $ ll, map snd ll))+ . List.groupBy ((==) `on` fst)+ . List.sortBy (comparing fst) groupPairs :: (Eq a, Ord a) => [(a, b)] -> Map.Map a [b] groupPairs = Map.fromList . groupPairs'@@ -174,6 +211,9 @@ allYearsFileName :: TurtlePath allYearsFileName = "all-years" <.> "journal" +directivesFile :: (HasBaseDir o) => o -> TurtlePath+directivesFile opts = turtleBaseDir opts </> "directives" <.> "journal"+ lsDirs :: TurtlePath -> Turtle.Shell TurtlePath lsDirs = onlyDirs . Turtle.ls @@ -191,9 +231,9 @@ filterPathsByFileStatus' :: (Turtle.FileStatus -> Bool) -> [TurtlePath] -> [TurtlePath] -> Turtle.Shell [TurtlePath] filterPathsByFileStatus' _ acc [] = return acc-filterPathsByFileStatus' filepred acc (file:files) = do+filterPathsByFileStatus' filepred acc (file : files) = do filestat <- Turtle.stat file- let filtered = if (filepred filestat) then file:acc else acc+ let filtered = if (filepred filestat) then file : acc else acc filterPathsByFileStatus' filepred filtered files filterPaths :: (TurtlePath -> IO Bool) -> [TurtlePath] -> Turtle.Shell [TurtlePath]@@ -201,9 +241,9 @@ filterPaths' :: [TurtlePath] -> (TurtlePath -> IO Bool) -> [TurtlePath] -> Turtle.Shell [TurtlePath] filterPaths' acc _ [] = return acc-filterPaths' acc filepred (file:files) = do+filterPaths' acc filepred (file : files) = do shouldInclude <- Turtle.liftIO $ filepred file- let filtered = if shouldInclude then file:acc else acc+ let filtered = if shouldInclude then file : acc else acc filterPaths' filtered filepred files excludeHiddenFiles :: Turtle.Shell TurtlePath -> Turtle.Shell TurtlePath@@ -211,7 +251,7 @@ p <- paths case (Turtle.match (Turtle.prefix ".") $ Turtle.format Turtle.fp $ Turtle.filename p) of [] -> Turtle.select [p]- _ -> Turtle.select []+ _ -> Turtle.select [] excludeWeirdPaths :: Turtle.Shell TurtlePath -> Turtle.Shell TurtlePath excludeWeirdPaths = Turtle.findtree (Turtle.suffix $ Turtle.noneOf "_")@@ -219,18 +259,18 @@ firstExistingFile :: [TurtlePath] -> IO (Maybe TurtlePath) firstExistingFile files = do case files of- [] -> return Nothing- file:fs -> do+ [] -> return Nothing+ file : fs -> do exists <- Turtle.testfile file if exists then return (Just file) else firstExistingFile fs basenameLine :: TurtlePath -> Turtle.Shell Turtle.Line basenameLine path = case (Turtle.textToLine $ Turtle.format Turtle.fp $ Turtle.basename path) of- Nothing -> Turtle.die $ Turtle.format ("Unable to determine basename from path: "%Turtle.fp%"\n") path+ Nothing -> Turtle.die $ Turtle.format ("Unable to determine basename from path: " % Turtle.fp % "\n") path Just bn -> return bn buildFilename :: [Turtle.Line] -> T.Text -> TurtlePath-buildFilename identifiers ext = Turtle.fromText (T.intercalate "-" (map Turtle.lineToText identifiers)) <.> ext+buildFilename identifiers ext = T.unpack (T.intercalate "-" (map Turtle.lineToText identifiers)) Turtle.<.> (T.unpack ext) shellToList :: Turtle.Shell a -> Turtle.Shell [a] shellToList files = Turtle.fold files Fold.list@@ -246,29 +286,30 @@ return $ Map.keys fileMap writeTextMap :: Map.Map TurtlePath T.Text -> IO ()-writeTextMap = Map.foldlWithKey (\a k v -> a <> Turtle.writeTextFile k v) (return ())+writeTextMap = Map.foldlWithKey (\a k v -> a <> T.writeFile k v) (return ()) changeExtension :: T.Text -> TurtlePath -> TurtlePath-changeExtension ext path = (Turtle.dropExtension path) <.> ext+changeExtension ext path = (Turtle.dropExtension path) Turtle.<.> (T.unpack ext) changePathAndExtension :: TurtlePath -> T.Text -> TurtlePath -> TurtlePath changePathAndExtension newOutputLocation newExt = (changeOutputPath newOutputLocation) . (changeExtension newExt) changeOutputPath :: TurtlePath -> TurtlePath -> TurtlePath changeOutputPath newOutputLocation srcFile = mconcat $ map changeSrcDir $ Turtle.splitDirectories srcFile- where changeSrcDir file = if file == "1-in/" || file == "2-preprocessed/" then newOutputLocation else file+ where+ changeSrcDir file = if file == "1-in/" || file == "2-preprocessed/" then newOutputLocation else file -listOwners :: HasBaseDir o => o -> Turtle.Shell TurtlePath+listOwners :: (HasBaseDir o) => o -> Turtle.Shell TurtlePath listOwners opts = fmap Turtle.basename $ lsDirs $ (turtleBaseDir opts) </> "import" intPath :: Integer -> TurtlePath-intPath = Turtle.fromText . (Turtle.format Turtle.d)+intPath = T.unpack . (Turtle.format Turtle.d) includeYears :: TChan LogMessage -> TurtlePath -> IO [Integer] includeYears ch includeFile = do- txt <- Turtle.readTextFile includeFile+ txt <- T.readFile includeFile case includeYears' txt of- Left msg -> do+ Left msg -> do channelErrLn ch msg return [] Right years -> return years@@ -276,7 +317,7 @@ includeYears' :: T.Text -> Either T.Text [Integer] includeYears' txt = case partitionEithers (includeYears'' txt) of (errors, []) -> do- let msg = Turtle.format ("Unable to extract years from the following text:\n"%Turtle.s%"\nErrors:\n"%Turtle.s) txt (T.intercalate "\n" $ map T.pack errors)+ let msg = Turtle.format ("Unable to extract years from the following text:\n" % Turtle.s % "\nErrors:\n" % Turtle.s) txt (T.intercalate "\n" $ map T.pack errors) Left msg (_, years) -> Right years
src/Hledger/Flow/DateTime.hs view
@@ -1,9 +1,9 @@ module Hledger.Flow.DateTime where -import Data.Time.Clock import Data.Time.Calendar+import Data.Time.Clock -currentDate :: IO (Integer,Int,Int) -- :: (year,month,day)+currentDate :: IO (Integer, Int, Int) -- :: (year,month,day) currentDate = toGregorian . utctDay <$> getCurrentTime currentYear :: IO Integer
src/Hledger/Flow/DocHelpers.hs view
@@ -3,8 +3,8 @@ module Hledger.Flow.DocHelpers where import qualified Data.Text as T (Text)-import qualified Turtle as Turtle (Line, format, l) import Turtle ((%))+import qualified Turtle as Turtle (Line, format, l) docURL :: Turtle.Line -> T.Text-docURL = Turtle.format ("https://github.com/apauley/hledger-flow#"%Turtle.l)+docURL = Turtle.format ("https://github.com/apauley/hledger-flow/tree/master/docs#" % Turtle.l)
src/Hledger/Flow/Import/CSVImport.hs view
@@ -1,68 +1,77 @@ {-# LANGUAGE OverloadedStrings #-} module Hledger.Flow.Import.CSVImport- ( importCSVs- ) where+ ( importCSVs,+ )+where -import qualified Turtle hiding (stdout, stderr, proc, procStrictWithErr)-import Turtle ((%), (</>), (<.>))-import Prelude hiding (putStrLn, take)-import qualified Data.Text as T+import Control.Concurrent.STM+import Control.Monad import qualified Data.List.NonEmpty as NonEmpty-import qualified Hledger.Flow.Types as FlowTypes-import Hledger.Flow.Import.Types-import Hledger.Flow.BaseDir (relativeToBase, effectiveRunDir)+import Data.Maybe (fromMaybe, isNothing)+import qualified Data.Text as T+import qualified Data.Text.IO as T+import Hledger.Flow.BaseDir (effectiveRunDir, relativeToBase)+import Hledger.Flow.Common+import Hledger.Flow.DocHelpers (docURL) import Hledger.Flow.Import.ImportHelpers import Hledger.Flow.Import.ImportHelpersTurtle (extractImportDirs, writeIncludesUpTo, writeToplevelAllYearsInclude)-import Hledger.Flow.PathHelpers (TurtlePath, pathToTurtle)-import Hledger.Flow.DocHelpers (docURL)-import Hledger.Flow.Common+import Hledger.Flow.Import.Types import Hledger.Flow.Logging+import Hledger.Flow.PathHelpers (TurtlePath, pathToTurtle) import Hledger.Flow.RuntimeOptions-import Control.Concurrent.STM-import Control.Monad-import Data.Maybe (fromMaybe, isNothing)+import qualified Hledger.Flow.Types as FlowTypes+import Turtle ((%), (<.>), (</>))+import qualified Turtle hiding (proc, procStrictWithErr, stderr, stdout)+import Prelude hiding (putStrLn, take, writeFile) type FileWasGenerated = Bool importCSVs :: RuntimeOptions -> IO ()-importCSVs opts = Turtle.sh (- do- ch <- Turtle.liftIO newTChanIO- logHandle <- Turtle.fork $ consoleChannelLoop ch- Turtle.liftIO $ when (showOptions opts) (channelOutLn ch (Turtle.repr opts))- Turtle.liftIO $ logVerbose opts ch "Starting import"- (journals, diff) <- Turtle.time $ Turtle.liftIO $ importCSVs' opts ch- let generatedJournals = filter snd journals- Turtle.liftIO $ channelOutLn ch $ Turtle.format ("Imported "%Turtle.d%"/"%Turtle.d%" journals in "%Turtle.s) (length generatedJournals) (length journals) $ Turtle.repr diff- Turtle.liftIO $ terminateChannelLoop ch- Turtle.wait logHandle- )+importCSVs opts =+ Turtle.sh+ ( do+ ch <- Turtle.liftIO newTChanIO+ logHandle <- Turtle.fork $ consoleChannelLoop ch+ Turtle.liftIO $ when (showOptions opts) (channelOutLn ch (Turtle.repr opts))+ Turtle.liftIO $ logVerbose opts ch "Starting import"+ (journals, diff) <- Turtle.time $ Turtle.liftIO $ importCSVs' opts ch+ let generatedJournals = filter snd journals+ Turtle.liftIO $ channelOutLn ch $ Turtle.format ("Imported " % Turtle.d % "/" % Turtle.d % " journals in " % Turtle.s) (length generatedJournals) (length journals) $ Turtle.repr diff+ Turtle.liftIO $ terminateChannelLoop ch+ Turtle.wait logHandle+ ) importCSVs' :: RuntimeOptions -> TChan FlowTypes.LogMessage -> IO [(TurtlePath, FileWasGenerated)] importCSVs' opts ch = do let effectiveDir = effectiveRunDir (baseDir opts) (importRunDir opts) let startYearMsg = maybe " " (Turtle.format (" (for the year " % Turtle.d % " and onwards) ")) (importStartYear opts)- channelOutLn ch $ Turtle.format ("Collecting input files"%Turtle.s%"from "%Turtle.fp) startYearMsg (pathToTurtle effectiveDir)+ channelOutLn ch $ Turtle.format ("Collecting input files" % Turtle.s % "from " % Turtle.fp) startYearMsg (pathToTurtle effectiveDir) (inputFiles, diff) <- Turtle.time $ findInputFiles (fromMaybe 0 $ importStartYear opts) effectiveDir let fileCount = length inputFiles- if fileCount == 0 && isNothing (importStartYear opts) then- do- let msg = Turtle.format ("I couldn't find any input files underneath "%Turtle.fp- %"\n\nhledger-flow expects to find its input files in specifically\nnamed directories.\n\n"%- "Have a look at the documentation for a detailed explanation:\n"%Turtle.s) (pathToTurtle effectiveDir) (docURL "input-files")+ if fileCount == 0 && isNothing (importStartYear opts)+ then do+ let msg =+ Turtle.format+ ( "I couldn't find any input files underneath "+ % Turtle.fp+ % "\n\nhledger-flow expects to find its input files in specifically\nnamed directories.\n\n"+ % "Have a look at the documentation for a detailed explanation:\n"+ % Turtle.s+ )+ (pathToTurtle effectiveDir)+ (docURL "input-files") errExit 1 ch msg []- else- do- channelOutLn ch $ Turtle.format ("Found "%Turtle.d%" input files"%Turtle.s%"in "%Turtle.s%". Proceeding with import...") fileCount startYearMsg (Turtle.repr diff)+ else do+ channelOutLn ch $ Turtle.format ("Found " % Turtle.d % " input files" % Turtle.s % "in " % Turtle.s % ". Proceeding with import...") fileCount startYearMsg (Turtle.repr diff) let actions = map (extractAndImport opts ch . pathToTurtle) inputFiles :: [IO (TurtlePath, FileWasGenerated)] importedJournals <- parAwareActions opts actions (journalsOnDisk, journalFindTime) <- Turtle.time $ findJournalFiles effectiveDir (_, writeIncludeTime1) <- Turtle.time $ writeIncludesUpTo opts ch (pathToTurtle effectiveDir) $ fmap pathToTurtle journalsOnDisk (_, writeIncludeTime2) <- Turtle.time $ writeToplevelAllYearsInclude opts let includeGenTime = journalFindTime + writeIncludeTime1 + writeIncludeTime2- channelOutLn ch $ Turtle.format ("Wrote include files for "%Turtle.d%" journals in "%Turtle.s) (length journalsOnDisk) (Turtle.repr includeGenTime)+ channelOutLn ch $ Turtle.format ("Wrote include files for " % Turtle.d % " journals in " % Turtle.s) (length journalsOnDisk) (Turtle.repr includeGenTime) return importedJournals extractAndImport :: RuntimeOptions -> TChan FlowTypes.LogMessage -> TurtlePath -> IO (TurtlePath, FileWasGenerated)@@ -80,16 +89,18 @@ let accountName = importDirLine accountDir importDirs let ownerName = importDirLine ownerDir importDirs (csvFile, preprocessHappened) <- preprocessIfNeeded opts ch preprocessScript bankName accountName ownerName srcFile- let journalOut = changePathAndExtension "3-journal" "journal" csvFile- shouldImport <- if onlyNewFiles opts && not preprocessHappened- then not <$> verboseTestFile opts ch journalOut- else return True+ let journalOut = changePathAndExtension "3-journal/" "journal" csvFile+ shouldImport <-+ if onlyNewFiles opts && not preprocessHappened+ then not <$> verboseTestFile opts ch journalOut+ else return True - importFun <- if shouldImport- then constructOrImport opts ch constructScript bankName accountName ownerName- else do- _ <- logNewFileSkip opts ch "import" journalOut- return $ \_p1 _p2 -> return journalOut+ importFun <-+ if shouldImport+ then constructOrImport opts ch constructScript bankName accountName ownerName+ else do+ _ <- logNewFileSkip opts ch "import" journalOut+ return $ \_p1 _p2 -> return journalOut Turtle.mktree $ Turtle.directory journalOut out <- importFun csvFile journalOut return (out, shouldImport)@@ -103,31 +114,37 @@ preprocessIfNeeded :: RuntimeOptions -> TChan FlowTypes.LogMessage -> TurtlePath -> Turtle.Line -> Turtle.Line -> Turtle.Line -> TurtlePath -> IO (TurtlePath, Bool) preprocessIfNeeded opts ch script bank account owner src = do- let csvOut = changePathAndExtension "2-preprocessed" "csv" src+ let csvOut = changePathAndExtension "2-preprocessed/" "csv" src scriptExists <- verboseTestFile opts ch script- shouldProceed <- if onlyNewFiles opts - then do- targetExists <- verboseTestFile opts ch csvOut- return $ scriptExists && not targetExists- else return scriptExists+ targetExists <- verboseTestFile opts ch csvOut+ shouldProceed <-+ if onlyNewFiles opts+ then return $ scriptExists && not targetExists+ else return scriptExists if shouldProceed then do- out <- preprocess opts ch script bank account owner src csvOut- return (out, True)+ out <- preprocess opts ch script bank account owner src csvOut+ return (out, True) else do _ <- logNewFileSkip opts ch "preprocess" csvOut- return (src, False)+ if targetExists+ then return (csvOut, False)+ else return (src, False) logNewFileSkip :: RuntimeOptions -> TChan FlowTypes.LogMessage -> T.Text -> TurtlePath -> IO () logNewFileSkip opts ch logIdentifier absTarget = Control.Monad.when (onlyNewFiles opts) $ do- let relativeTarget = relativeToBase opts absTarget- logVerbose opts ch- $ Turtle.format- ("Skipping " % Turtle.s- % " - only creating new files and this output file already exists: '"- % Turtle.fp- % "'") logIdentifier relativeTarget+ let relativeTarget = relativeToBase opts absTarget+ logVerbose opts ch $+ Turtle.format+ ( "Skipping "+ % Turtle.s+ % " - only creating new files and this output file already exists: '"+ % Turtle.fp+ % "'"+ )+ logIdentifier+ relativeTarget preprocess :: RuntimeOptions -> TChan FlowTypes.LogMessage -> TurtlePath -> Turtle.Line -> Turtle.Line -> Turtle.Line -> TurtlePath -> TurtlePath -> IO TurtlePath preprocess opts ch script bank account owner src csvOut = do@@ -135,7 +152,7 @@ let args = [Turtle.format Turtle.fp src, Turtle.format Turtle.fp csvOut, Turtle.lineToText bank, Turtle.lineToText account, Turtle.lineToText owner] let relScript = relativeToBase opts script let relSrc = relativeToBase opts src- let cmdLabel = Turtle.format ("executing '"%Turtle.fp%"' on '"%Turtle.fp%"'") relScript relSrc+ let cmdLabel = Turtle.format ("executing '" % Turtle.fp % "' on '" % Turtle.fp % "'") relScript relSrc _ <- timeAndExitOnErr opts ch cmdLabel channelOut channelErr (parAwareProc opts) (Turtle.format Turtle.fp script, args, Turtle.empty) return csvOut @@ -155,18 +172,34 @@ Just rf -> do let relRules = relativeToBase opts rf let hledger = Turtle.format Turtle.fp $ pathToTurtle . FlowTypes.hlPath . hledgerInfo $ opts :: T.Text- let args = ["print", "--rules-file", Turtle.format Turtle.fp rf, "--file", Turtle.format Turtle.fp csvSrc, "--output-file", Turtle.format Turtle.fp journalOut]- let cmdLabel = Turtle.format ("importing '"%Turtle.fp%"' using rules file '"%Turtle.fp%"'") relCSV relRules- _ <- timeAndExitOnErr opts ch cmdLabel channelOut channelErr (parAwareProc opts) (hledger, args, Turtle.empty)+ directivesExist <- Turtle.testfile $ directivesFile opts+ let directivesArgs = if directivesExist then ["--file", Turtle.format Turtle.fp (directivesFile opts)] else []+ let csvArgs = ["--file", Turtle.format Turtle.fp csvSrc, "--rules-file", Turtle.format Turtle.fp rf]+ let args = ["print", "--explicit"] ++ directivesArgs ++ csvArgs++ let cmdLabel = Turtle.format ("importing '" % Turtle.fp % "' using rules file '" % Turtle.fp % "'") relCSV relRules+ ((_, stdOut, _), _) <- timeAndExitOnErr opts ch cmdLabel dummyLogger channelErr (parAwareProc opts) (hledger, args, Turtle.empty)+ _ <- T.writeFile journalOut stdOut return journalOut Nothing -> do let relativeCandidates = map (relativeToBase opts) candidates let candidatesTxt = T.intercalate "\n" $ map (Turtle.format Turtle.fp) relativeCandidates- let msg = Turtle.format ("I couldn't find an hledger rules file while trying to import\n"%Turtle.fp- %"\n\nI will happily use the first rules file I can find from any one of these "%Turtle.d%" files:\n"%Turtle.s- %"\n\nHere is a bit of documentation about rules files that you may find helpful:\n"%Turtle.s)- relCSV (length candidates) candidatesTxt (docURL "rules-files")+ let msg =+ Turtle.format+ ( "I couldn't find an hledger rules file while trying to import\n"+ % Turtle.fp+ % "\n\nI will happily use the first rules file I can find from any one of these "+ % Turtle.d+ % " files:\n"+ % Turtle.s+ % "\n\nHere is a bit of documentation about rules files that you may find helpful:\n"+ % Turtle.s+ )+ relCSV+ (length candidates)+ candidatesTxt+ (docURL "rules-files") errExit 1 ch msg csvSrc rulesFileCandidates :: TurtlePath -> ImportDirs -> [TurtlePath]@@ -192,9 +225,8 @@ let srcSuffix = snd $ T.breakOnEnd "_" (Turtle.format Turtle.fp (Turtle.basename csvSrc)) if ((T.take 3 srcSuffix) == "rfo")- then- do- let srcSpecificFilename = Turtle.fromText srcSuffix <.> "rules"+ then do+ let srcSpecificFilename = T.unpack srcSuffix <.> "rules" map (</> srcSpecificFilename) [accountDir importDirs, bankDir importDirs, importDir importDirs] else [] @@ -203,11 +235,19 @@ let script = Turtle.format Turtle.fp constructScript :: T.Text let relScript = relativeToBase opts constructScript let constructArgs = [Turtle.format Turtle.fp csvSrc, "-", Turtle.lineToText bank, Turtle.lineToText account, Turtle.lineToText owner]- let constructCmdText = Turtle.format ("Running: "%Turtle.fp%" "%Turtle.s) relScript (showCmdArgs constructArgs)+ let constructCmdText = Turtle.format ("Running: " % Turtle.fp % " " % Turtle.s) relScript (showCmdArgs constructArgs) let stdLines = inprocWithErrFun (channelErrLn ch) (script, constructArgs, Turtle.empty) let hledger = Turtle.format Turtle.fp $ pathToTurtle . FlowTypes.hlPath . hledgerInfo $ opts :: T.Text- let args = ["print", "--ignore-assertions", "--file", "-", "--output-file", Turtle.format Turtle.fp journalOut]+ let args =+ [ "print",+ "--ignore-assertions",+ "--file",+ "-",+ "--output-file",+ Turtle.format Turtle.fp journalOut+ ]+ let relSrc = relativeToBase opts csvSrc- let cmdLabel = Turtle.format ("executing '"%Turtle.fp%"' on '"%Turtle.fp%"'") relScript relSrc+ let cmdLabel = Turtle.format ("executing '" % Turtle.fp % "' on '" % Turtle.fp % "'") relScript relSrc _ <- timeAndExitOnErr' opts ch cmdLabel [constructCmdText] channelOut channelErr (parAwareProc opts) (hledger, args, stdLines) return journalOut
src/Hledger/Flow/Import/ImportHelpers.hs view
@@ -2,16 +2,14 @@ module Hledger.Flow.Import.ImportHelpers (findInputFiles, findJournalFiles, groupIncludesUpTo, includeFileName) where -import Path import Data.Char (isDigit)+import qualified Data.Map.Strict as Map import Data.Maybe (fromMaybe)-import System.FilePath (dropTrailingPathSeparator)- import Hledger.Flow.Common (groupValuesBy)-import Hledger.Flow.PathHelpers (AbsDir, AbsFile, RelDir, RelFile, findFilesIn, pathSize) import Hledger.Flow.Import.Types (InputFileBundle)--import qualified Data.Map.Strict as Map+import Hledger.Flow.PathHelpers (AbsDir, AbsFile, RelDir, RelFile, findFilesIn, pathSize)+import Path+import System.FilePath (dropTrailingPathSeparator) findInputFiles :: Integer -> AbsDir -> IO [AbsFile] findInputFiles startYear = do@@ -25,11 +23,13 @@ -- | Include only files directly underneath parentDir/yearDir, e.g. 1-in/2020/* or 3-journal/2020/* includeYearFilesForParent :: RelDir -> Integer -> AbsDir -> Bool-includeYearFilesForParent parentDir startYear d = (dirname . parent) d == parentDir- && length shortDirName == 4- && all isDigit shortDirName- && read shortDirName >= startYear- where shortDirName = dirToStringNoSlash d+includeYearFilesForParent parentDir startYear d =+ (dirname . parent) d == parentDir+ && length shortDirName == 4+ && all isDigit shortDirName+ && read shortDirName >= startYear+ where+ shortDirName = dirToStringNoSlash d dirToStringNoSlash :: AbsDir -> String dirToStringNoSlash = init . Path.toFilePath . Path.dirname@@ -45,15 +45,18 @@ groupIncludesUpTo' acc stopAt journals = do let dirs = map parent journals :: [RelDir] let shouldStop = stopAt `elem` dirs- if shouldStop then acc else do- let grouped = groupIncludeFilesPerYear journals- groupIncludesUpTo' (acc <> grouped) stopAt (Map.keys grouped)+ if shouldStop+ then acc+ else do+ let grouped = groupIncludeFilesPerYear journals+ groupIncludesUpTo' (acc <> grouped) stopAt (Map.keys grouped) groupIncludeFilesPerYear :: [RelFile] -> InputFileBundle groupIncludeFilesPerYear [] = Map.empty-groupIncludeFilesPerYear ps@(p:_) = if pathSize (parent p) == 6- then groupValuesBy initialIncludeFilePath ps- else groupValuesBy parentIncludeFilePath ps+groupIncludeFilesPerYear ps@(p : _) =+ if pathSize (parent p) == 6+ then groupValuesBy initialIncludeFilePath ps+ else groupValuesBy parentIncludeFilePath ps initialIncludeFilePath :: RelFile -> RelFile initialIncludeFilePath p = (parent . parent . parent) p </> includeFileName p
src/Hledger/Flow/Import/ImportHelpersTurtle.hs view
@@ -1,59 +1,62 @@ {-# LANGUAGE OverloadedStrings #-} -- | Functions which currently uses TurtlePath and will be replaced with Path eventually-module Hledger.Flow.Import.ImportHelpersTurtle (- allYearIncludeFiles- , extractImportDirs- , extraIncludesForFile- , groupIncludeFiles- , groupAndWriteIncludeFiles- , includePreamble- , toIncludeFiles- , toIncludeLine- , writeIncludesUpTo- , writeToplevelAllYearsInclude- , yearsIncludeMap+module Hledger.Flow.Import.ImportHelpersTurtle+ ( allYearIncludeFiles,+ extractImportDirs,+ extraIncludesForFile,+ groupIncludeFiles,+ groupAndWriteIncludeFiles,+ includePreamble,+ toIncludeFiles,+ toIncludeLine,+ writeIncludesUpTo,+ writeToplevelAllYearsInclude,+ yearsIncludeMap, )- where+where -import Hledger.Flow.PathHelpers (TurtlePath)-import Hledger.Flow.DocHelpers (docURL)-import Hledger.Flow.Common (allYearsFileName, filterPaths, groupValuesBy, writeFiles, writeFiles')+import Control.Concurrent.STM (TChan)+import qualified Data.List as List (nub, sort)+import qualified Data.Map.Strict as Map+import Data.Maybe (fromMaybe)+import qualified Data.Text as T import Hledger.Flow.BaseDir (relativeToBase, relativeToBase', turtleBaseDir)+import Hledger.Flow.Common (allYearsFileName, directivesFile, filterPaths, groupValuesBy, writeFiles, writeFiles')+import Hledger.Flow.DocHelpers (docURL)+import Hledger.Flow.Import.Types import Hledger.Flow.Logging (logVerbose)-+import Hledger.Flow.PathHelpers (TurtlePath) import Hledger.Flow.Types-import Hledger.Flow.Import.Types--import qualified Data.List as List (nub, sort)-import qualified Data.Text as T-import qualified Data.Map.Strict as Map-+import Turtle ((%), (<.>), (</>)) import qualified Turtle-import Turtle ((%), (</>), (<.>)) -import Control.Concurrent.STM (TChan)-import Data.Maybe (fromMaybe)- extractImportDirs :: TurtlePath -> Either T.Text ImportDirs extractImportDirs inputFile = do case importDirBreakdown inputFile of- [bd,owner,bank,account,filestate,year] -> Right $ ImportDirs bd owner bank account filestate year+ [bd, owner, bank, account, filestate, year] -> Right $ ImportDirs bd owner bank account filestate year _ -> do- Left $ Turtle.format ("I couldn't find the right number of directories between \"import\" and the input file:\n"%Turtle.fp- %"\n\nhledger-flow expects to find input files in this structure:\n"%- "import/owner/bank/account/filestate/year/trxfile\n\n"%- "Have a look at the documentation for a detailed explanation:\n"%Turtle.s) inputFile (docURL "input-files")+ Left $+ Turtle.format+ ( "I couldn't find the right number of directories between \"import\" and the input file:\n"+ % Turtle.fp+ % "\n\nhledger-flow expects to find input files in this structure:\n"+ % "import/owner/bank/account/filestate/year/trxfile\n\n"+ % "Have a look at the documentation for a detailed explanation:\n"+ % Turtle.s+ )+ inputFile+ (docURL "input-files") -importDirBreakdown :: TurtlePath -> [TurtlePath]+importDirBreakdown :: TurtlePath -> [TurtlePath] importDirBreakdown = importDirBreakdown' [] importDirBreakdown' :: [TurtlePath] -> TurtlePath -> [TurtlePath] importDirBreakdown' acc path = do let dir = Turtle.directory path if Turtle.dirname dir == "import" || (Turtle.dirname dir == "")- then dir:acc- else importDirBreakdown' (dir:acc) $ Turtle.parent dir+ then dir : acc+ else importDirBreakdown' (dir : acc) $ Turtle.parent dir groupIncludeFiles :: [TurtlePath] -> (TurtleFileBundle, TurtleFileBundle) groupIncludeFiles = allYearIncludeFiles . groupIncludeFilesPerYear . filter isJournalFile@@ -74,13 +77,13 @@ allYearsPath' dir p = dir p </> allYearsFileName includeFileName :: TurtlePath -> TurtlePath-includeFileName = (<.> "journal") . Turtle.fromText . (Turtle.format (Turtle.fp%"-include")) . Turtle.dirname+includeFileName = (<.> "journal") . T.unpack . (Turtle.format (Turtle.fp % "-include")) . Turtle.dirname groupIncludeFilesPerYear :: [TurtlePath] -> TurtleFileBundle groupIncludeFilesPerYear [] = Map.empty-groupIncludeFilesPerYear ps@(p:_) = case extractImportDirs p of- Right _ -> groupValuesBy initialIncludeFilePath ps- Left _ -> groupValuesBy parentIncludeFilePath ps+groupIncludeFilesPerYear ps@(p : _) = case extractImportDirs p of+ Right _ -> groupValuesBy initialIncludeFilePath ps+ Left _ -> groupValuesBy parentIncludeFilePath ps initialIncludeFilePath :: TurtlePath -> TurtlePath initialIncludeFilePath p = (Turtle.parent . Turtle.parent . Turtle.parent) p </> includeFileName p@@ -90,7 +93,7 @@ toIncludeFiles :: (HasBaseDir o, HasVerbosity o) => o -> TChan LogMessage -> TurtleFileBundle -> IO (Map.Map TurtlePath T.Text) toIncludeFiles opts ch m = do- preMap <- extraIncludes opts ch (Map.keys m) ["opening.journal"] ["pre-import.journal"] []+ preMap <- extraIncludes opts ch (Map.keys m) ["opening.journal"] ["pre-import.journal"] [] postMap <- extraIncludes opts ch (Map.keys m) ["closing.journal"] ["post-import.journal"] ["prices.journal"] return $ (addPreamble . toIncludeFiles' preMap postMap) m @@ -101,7 +104,7 @@ addPreamble = Map.map (\txt -> includePreamble <> "\n" <> txt) toIncludeLine :: TurtlePath -> TurtlePath -> T.Text-toIncludeLine base file = Turtle.format ("!include "%Turtle.fp) $ relativeToBase' base file+toIncludeLine base file = Turtle.format ("include " % Turtle.fp) $ relativeToBase' base file generatedIncludeText :: TurtleFileBundle -> TurtleFileBundle -> TurtlePath -> [TurtlePath] -> T.Text generatedIncludeText preMap postMap outputFile fs = do@@ -134,9 +137,8 @@ writeToplevelAllYearsInclude :: (HasBaseDir o, HasVerbosity o) => o -> IO [TurtlePath] writeToplevelAllYearsInclude opts = do- let directivesFile = turtleBaseDir opts </> "directives" <.> "journal"- directivesExists <- Turtle.testfile directivesFile- let preMap = if directivesExists then Map.singleton (turtleBaseDir opts </> allYearsFileName) [directivesFile] else Map.empty+ directivesExists <- Turtle.testfile (directivesFile opts)+ let preMap = if directivesExists then Map.singleton (turtleBaseDir opts </> allYearsFileName) [directivesFile opts] else Map.empty let allTop = Map.singleton (turtleBaseDir opts </> allYearsFileName) ["import" </> allYearsFileName] writeFiles' $ (addPreamble . toIncludeFiles' preMap Map.empty) allTop @@ -145,24 +147,29 @@ extraIncludes' :: (HasBaseDir o, HasVerbosity o) => o -> TChan LogMessage -> TurtleFileBundle -> [TurtlePath] -> [T.Text] -> [TurtlePath] -> [TurtlePath] -> IO TurtleFileBundle extraIncludes' _ _ acc [] _ _ _ = return acc-extraIncludes' opts ch acc (file:files) extraSuffixes manualFiles prices = do+extraIncludes' opts ch acc (file : files) extraSuffixes manualFiles prices = do extra <- extraIncludesForFile opts ch file extraSuffixes manualFiles prices extraIncludes' opts ch (Map.unionWith (++) acc extra) files extraSuffixes manualFiles prices extraIncludesForFile :: (HasVerbosity o, HasBaseDir o) => o -> TChan LogMessage -> TurtlePath -> [T.Text] -> [TurtlePath] -> [TurtlePath] -> IO TurtleFileBundle extraIncludesForFile opts ch file extraSuffixes manualFiles prices = do- let dirprefix = Turtle.fromText $ fst $ T.breakOn "-" $ Turtle.format Turtle.fp $ Turtle.basename file- let fileNames = map (Turtle.fromText . Turtle.format (Turtle.fp % "-" % Turtle.s) dirprefix) extraSuffixes+ let dirprefix = T.unpack $ fst $ T.breakOn "-" $ Turtle.format Turtle.fp $ Turtle.basename file+ let fileNames = map (T.unpack . Turtle.format (Turtle.fp % "-" % Turtle.s) dirprefix) extraSuffixes let suffixFiles = map (Turtle.directory file </>) fileNames- let suffixDirFiles = map (Turtle.directory file </> "_manual_" </> dirprefix </>) manualFiles- let priceFiles = map (Turtle.directory file </> ".." </> "prices" </> dirprefix </>) prices+ let suffixDirFiles = map (((Turtle.directory file </> "_manual_") </> dirprefix) </>) manualFiles+ let priceFiles = map ((((Turtle.directory file </> "..") </> "prices") </> dirprefix) </>) prices let extraFiles = suffixFiles ++ suffixDirFiles ++ priceFiles filtered <- Turtle.single $ filterPaths Turtle.testfile extraFiles- let logMsg = Turtle.format ("Looking for possible extra include files for '"%Turtle.fp%"' among these "%Turtle.d%" options: "%Turtle.s%". Found "%Turtle.d%": "%Turtle.s)- (relativeToBase opts file) (length extraFiles) (Turtle.repr $ relativeFilesAsText opts extraFiles)- (length filtered) (Turtle.repr $ relativeFilesAsText opts filtered)+ let logMsg =+ Turtle.format+ ("Looking for possible extra include files for '" % Turtle.fp % "' among these " % Turtle.d % " options: " % Turtle.s % ". Found " % Turtle.d % ": " % Turtle.s)+ (relativeToBase opts file)+ (length extraFiles)+ (Turtle.repr $ relativeFilesAsText opts extraFiles)+ (length filtered)+ (Turtle.repr $ relativeFilesAsText opts filtered) logVerbose opts ch logMsg return $ Map.fromList [(file, filtered)] -relativeFilesAsText :: HasBaseDir o => o -> [TurtlePath] -> [T.Text]+relativeFilesAsText :: (HasBaseDir o) => o -> [TurtlePath] -> [T.Text] relativeFilesAsText opts = map (Turtle.format Turtle.fp . relativeToBase opts)
src/Hledger/Flow/Import/Types.hs view
@@ -1,15 +1,18 @@ module Hledger.Flow.Import.Types where -import Hledger.Flow.PathHelpers (RelFile, TurtlePath) import qualified Data.Map.Strict as Map+import Hledger.Flow.PathHelpers (RelFile, TurtlePath) type TurtleFileBundle = Map.Map TurtlePath [TurtlePath]+ type InputFileBundle = Map.Map RelFile [RelFile] -data ImportDirs = ImportDirs { importDir :: TurtlePath- , ownerDir :: TurtlePath- , bankDir :: TurtlePath- , accountDir :: TurtlePath- , stateDir :: TurtlePath- , yearDir :: TurtlePath- } deriving (Show)+data ImportDirs = ImportDirs+ { importDir :: TurtlePath,+ ownerDir :: TurtlePath,+ bankDir :: TurtlePath,+ accountDir :: TurtlePath,+ stateDir :: TurtlePath,+ yearDir :: TurtlePath+ }+ deriving (Show)
src/Hledger/Flow/Internals.hs view
@@ -2,39 +2,50 @@ module Hledger.Flow.Internals where -import GHC.Conc (getNumCapabilities, getNumProcessors)-import Development.GitRev+import qualified Data.Text as T import Data.Version (Version, showVersion)+import Development.GitRev+import GHC.Conc (getNumCapabilities, getNumProcessors) import Paths_hledger_flow (version)--import qualified Data.Text as T import qualified System.Info as Sys -data SystemInfo = SystemInfo { os :: String- , arch :: String- , compilerName :: String- , compilerVersion :: Version- , cores :: Int- , availableCores :: Int- }- deriving (Show)+data SystemInfo = SystemInfo+ { os :: String,+ arch :: String,+ compilerName :: String,+ compilerVersion :: Version,+ cores :: Int,+ availableCores :: Int+ }+ deriving (Show) versionInfo :: SystemInfo -> T.Text-versionInfo sysInfo = T.pack ("hledger-flow " ++ showVersion version ++ " " ++- os sysInfo ++ " " ++ arch sysInfo ++ " " ++- compilerName sysInfo ++ " " ++- showVersion (compilerVersion sysInfo) ++- " " ++ $(gitHash))+versionInfo sysInfo =+ T.pack+ ( "hledger-flow "+ ++ showVersion version+ ++ " "+ ++ os sysInfo+ ++ " "+ ++ arch sysInfo+ ++ " "+ ++ compilerName sysInfo+ ++ " "+ ++ showVersion (compilerVersion sysInfo)+ ++ " "+ ++ $(gitHash)+ ) systemInfo :: IO SystemInfo systemInfo = do- processors <- getNumProcessors- available <- getNumCapabilities- return SystemInfo {- os = Sys.os,+ processors <- getNumProcessors+ available <- getNumCapabilities+ return+ SystemInfo+ { os = Sys.os, arch = Sys.arch, compilerName = Sys.compilerName, compilerVersion = Sys.compilerVersion, cores = processors, availableCores = available- }+ }
src/Hledger/Flow/Logging.hs view
@@ -2,17 +2,15 @@ module Hledger.Flow.Logging where -import Hledger.Flow.Types import Control.Concurrent.STM import Control.Monad (when)- import qualified Data.Text as T import qualified Data.Text.IO as T import Data.Time.LocalTime (getZonedTime)- import qualified GHC.IO.Handle.FD as H-import qualified Turtle+import Hledger.Flow.Types import Turtle ((%))+import qualified Turtle dummyLogger :: TChan LogMessage -> T.Text -> IO () dummyLogger _ _ = return ()@@ -37,7 +35,7 @@ timestampPrefix :: T.Text -> IO T.Text timestampPrefix txt = do t <- getZonedTime- return $ Turtle.format (Turtle.s%"\thledger-flow "%Turtle.s) (Turtle.repr t) txt+ return $ Turtle.format (Turtle.s % "\thledger-flow " % Turtle.s) (Turtle.repr t) txt consoleChannelLoop :: TChan LogMessage -> IO () consoleChannelLoop ch = do@@ -49,10 +47,10 @@ StdErr msg -> do T.hPutStr H.stderr msg consoleChannelLoop ch- Terminate -> return ()+ Terminate -> return () terminateChannelLoop :: TChan LogMessage -> IO () terminateChannelLoop ch = atomically $ writeTChan ch Terminate -logVerbose :: HasVerbosity o => o -> TChan LogMessage -> T.Text -> IO ()+logVerbose :: (HasVerbosity o) => o -> TChan LogMessage -> T.Text -> IO () logVerbose opts ch msg = when (verbose opts) $ logToChannel ch msg
src/Hledger/Flow/PathHelpers.hs view
@@ -2,58 +2,60 @@ module Hledger.Flow.PathHelpers where -import Control.Monad.Catch (MonadThrow, Exception, throwM)+import Control.Monad.Catch (Exception, MonadThrow, throwM) import Control.Monad.IO.Class (MonadIO)- import qualified Data.Text as T-+import Hledger.Flow.DocHelpers (docURL) import Path ((</>)) import qualified Path import qualified Path.IO as Path import qualified Turtle -import Hledger.Flow.DocHelpers (docURL)- type TurtlePath = Turtle.FilePath type AbsFile = Path.Path Path.Abs Path.File+ type RelFile = Path.Path Path.Rel Path.File+ type AbsDir = Path.Path Path.Abs Path.Dir+ type RelDir = Path.Path Path.Rel Path.Dir data PathException = MissingBaseDir AbsDir | InvalidTurtleDir TurtlePath deriving (Eq) instance Show PathException where- show (MissingBaseDir d) = "Unable to find an import directory at " ++ show d ++- " (or in any of its parent directories).\n\n" ++- "Have a look at the documentation for more information:\n" ++- T.unpack (docURL "getting-started")- show (InvalidTurtleDir d) = "Expected a directory but got this instead: " ++ Turtle.encodeString d+ show (MissingBaseDir d) =+ "Unable to find an import directory at "+ ++ show d+ ++ " (or in any of its parent directories).\n\n"+ ++ "Have a look at the documentation for more information:\n"+ ++ T.unpack (docURL "getting-started")+ show (InvalidTurtleDir d) = "Expected a directory but got this instead: " ++ d instance Exception PathException -fromTurtleAbsFile :: MonadThrow m => TurtlePath -> m AbsFile-fromTurtleAbsFile turtlePath = Path.parseAbsFile $ Turtle.encodeString turtlePath+fromTurtleAbsFile :: (MonadThrow m) => TurtlePath -> m AbsFile+fromTurtleAbsFile turtlePath = Path.parseAbsFile turtlePath -fromTurtleRelFile :: MonadThrow m => TurtlePath -> m RelFile-fromTurtleRelFile turtlePath = Path.parseRelFile $ Turtle.encodeString turtlePath+fromTurtleRelFile :: (MonadThrow m) => TurtlePath -> m RelFile+fromTurtleRelFile turtlePath = Path.parseRelFile turtlePath -fromTurtleAbsDir :: MonadThrow m => TurtlePath -> m AbsDir-fromTurtleAbsDir turtlePath = Path.parseAbsDir $ Turtle.encodeString turtlePath+fromTurtleAbsDir :: (MonadThrow m) => TurtlePath -> m AbsDir+fromTurtleAbsDir turtlePath = Path.parseAbsDir turtlePath -fromTurtleRelDir :: MonadThrow m => TurtlePath -> m RelDir-fromTurtleRelDir turtlePath = Path.parseRelDir $ Turtle.encodeString turtlePath+fromTurtleRelDir :: (MonadThrow m) => TurtlePath -> m RelDir+fromTurtleRelDir turtlePath = Path.parseRelDir turtlePath turtleToAbsDir :: (MonadIO m, MonadThrow m) => AbsDir -> TurtlePath -> m AbsDir turtleToAbsDir baseDir p = do isDir <- Turtle.testdir p if isDir- then Path.resolveDir baseDir $ Turtle.encodeString p+ then Path.resolveDir baseDir p else throwM $ InvalidTurtleDir p pathToTurtle :: Path.Path b t -> TurtlePath-pathToTurtle = Turtle.decodeString . Path.toFilePath+pathToTurtle = Path.toFilePath forceTrailingSlash :: TurtlePath -> TurtlePath forceTrailingSlash p = Turtle.directory (p Turtle.</> "temp")@@ -62,21 +64,27 @@ pathSize p = pathSize' p 0 pathSize' :: Path.Path b Path.Dir -> Int -> Int-pathSize' p count = if Path.parent p == p then count else pathSize' (Path.parent p) (count+1)+pathSize' p count = if Path.parent p == p then count else pathSize' (Path.parent p) (count + 1) -- | Do a recursive search starting from the given directory. -- Return all files contained in each directory which matches the given predicate.-findFilesIn :: MonadIO m- => (AbsDir -> Bool) -- ^ Do we want the files in this directory?- -> [RelDir] -- ^ Exclude these directory names- -> AbsDir -- ^ Top of the search tree- -> m [AbsFile] -- ^ Absolute paths to all files in the directories which match the predicate+findFilesIn ::+ (MonadIO m) =>+ -- | Do we want the files in this directory?+ (AbsDir -> Bool) ->+ -- | Exclude these directory names+ [RelDir] ->+ -- | Top of the search tree+ AbsDir ->+ -- | Absolute paths to all files in the directories which match the predicate+ m [AbsFile] findFilesIn includePred excludeDirs = Path.walkDirAccum (Just excludeHandler) accumulator- where excludeHandler currentDir _ _ = return $ Path.WalkExclude (map (currentDir </>) excludeDirs)- accumulator currentDir _ files =- if includePred currentDir- then return $ excludeHiddenFiles files- else return []+ where+ excludeHandler currentDir _ _ = return $ Path.WalkExclude (map (currentDir </>) excludeDirs)+ accumulator currentDir _ files =+ if includePred currentDir+ then return $ excludeHiddenFiles files+ else return [] excludeHiddenFiles :: [AbsFile] -> [AbsFile]-excludeHiddenFiles = filter (\ f -> head (Path.toFilePath (Path.filename f)) /= '.')+excludeHiddenFiles = filter (\f -> head (Path.toFilePath (Path.filename f)) /= '.')
src/Hledger/Flow/Reports.hs view
@@ -1,70 +1,74 @@ {-# LANGUAGE OverloadedStrings #-} module Hledger.Flow.Reports- ( generateReports- ) where--import qualified Turtle as Turtle hiding (stdout, stderr, proc)-import Turtle ((%), (</>), (<.>))-import Prelude hiding (putStrLn, writeFile)--import Hledger.Flow.RuntimeOptions-import Hledger.Flow.Common-import Hledger.Flow.BaseDir (turtleBaseDir, relativeToBase)+ ( generateReports,+ )+where import Control.Concurrent.STM import Data.Either+import qualified Data.List as List import Data.Maybe- import qualified Data.Text as T-import qualified Hledger.Flow.Types as FlowTypes-import Hledger.Flow.PathHelpers (TurtlePath, pathToTurtle)+import qualified Data.Text.IO as T+import Hledger.Flow.BaseDir (relativeToBase, turtleBaseDir)+import Hledger.Flow.Common import Hledger.Flow.Logging-import qualified Data.List as List+import Hledger.Flow.PathHelpers (TurtlePath, pathToTurtle)+import Hledger.Flow.RuntimeOptions+import qualified Hledger.Flow.Types as FlowTypes+import Turtle ((%), (<.>), (</>))+import qualified Turtle as Turtle hiding (proc, stderr, stdout)+import Prelude hiding (putStrLn, readFile, writeFile) -data ReportParams = ReportParams { ledgerFile :: TurtlePath- , reportYears :: [Integer]- , outputDir :: TurtlePath- }- deriving (Show)+data ReportParams = ReportParams+ { ledgerFile :: TurtlePath,+ reportYears :: [Integer],+ outputDir :: TurtlePath+ }+ deriving (Show)+ type ReportGenerator = RuntimeOptions -> TChan FlowTypes.LogMessage -> TurtlePath -> TurtlePath -> Integer -> IO (Either TurtlePath TurtlePath) generateReports :: RuntimeOptions -> IO ()-generateReports opts = Turtle.sh (- do- ch <- Turtle.liftIO newTChanIO- logHandle <- Turtle.fork $ consoleChannelLoop ch- Turtle.liftIO $ if (showOptions opts) then channelOutLn ch (Turtle.repr opts) else return ()- (reports, diff) <- Turtle.time $ Turtle.liftIO $ generateReports' opts ch- let failedAttempts = lefts reports- let failedText = if List.null failedAttempts then "" else Turtle.format ("(and attempted to write "%Turtle.d%" more) ") $ length failedAttempts- Turtle.liftIO $ channelOutLn ch $ Turtle.format ("Generated "%Turtle.d%" reports "%Turtle.s%"in "%Turtle.s) (length (rights reports)) failedText $ Turtle.repr diff- Turtle.liftIO $ terminateChannelLoop ch- Turtle.wait logHandle- )+generateReports opts =+ Turtle.sh+ ( do+ ch <- Turtle.liftIO newTChanIO+ logHandle <- Turtle.fork $ consoleChannelLoop ch+ Turtle.liftIO $ if (showOptions opts) then channelOutLn ch (Turtle.repr opts) else return ()+ (reports, diff) <- Turtle.time $ Turtle.liftIO $ generateReports' opts ch+ let failedAttempts = lefts reports+ let failedText = if List.null failedAttempts then "" else Turtle.format ("(and attempted to write " % Turtle.d % " more) ") $ length failedAttempts+ Turtle.liftIO $ channelOutLn ch $ Turtle.format ("Generated " % Turtle.d % " reports " % Turtle.s % "in " % Turtle.s) (length (rights reports)) failedText $ Turtle.repr diff+ Turtle.liftIO $ terminateChannelLoop ch+ Turtle.wait logHandle+ ) generateReports' :: RuntimeOptions -> TChan FlowTypes.LogMessage -> IO [Either TurtlePath TurtlePath] generateReports' opts ch = do- let wipMsg = "Report generation is still a work-in-progress - please let me know how this can be more useful.\n\n"- <> "Keep an eye out for report-related pull requests and issues, and feel free to submit some of your own:\n"- <> "https://github.com/apauley/hledger-flow/pulls\n"- <> "https://github.com/apauley/hledger-flow/issues\n"+ let wipMsg =+ "These reports can be used as a starting point for more tailored reports.\n"+ <> "The first line of each report contains the command used - change the parameters and use it in your own reports.\n" channelOutLn ch wipMsg owners <- Turtle.single $ shellToList $ listOwners opts ledgerEnvValue <- Turtle.need "LEDGER_FILE" :: IO (Maybe T.Text)- let hledgerJournal = fromMaybe (turtleBaseDir opts </> allYearsFileName) $ fmap Turtle.fromText ledgerEnvValue+ let hledgerJournal = fromMaybe (turtleBaseDir opts </> allYearsFileName) $ fmap T.unpack ledgerEnvValue hledgerJournalExists <- Turtle.testfile hledgerJournal- _ <- if not hledgerJournalExists then Turtle.die $ Turtle.format ("Unable to find journal file: "%Turtle.fp%"\nIs your LEDGER_FILE environment variable set correctly?") hledgerJournal else return ()+ _ <- if not hledgerJournalExists then Turtle.die $ Turtle.format ("Unable to find journal file: " % Turtle.fp % "\nIs your LEDGER_FILE environment variable set correctly?") hledgerJournal else return () let journalWithYears = journalFile opts [] let aggregateReportDir = outputReportDir opts ["all"] aggregateYears <- includeYears ch journalWithYears- let aggregateParams = ReportParams { ledgerFile = hledgerJournal- , reportYears = aggregateYears- , outputDir = aggregateReportDir}+ let aggregateParams =+ ReportParams+ { ledgerFile = hledgerJournal,+ reportYears = aggregateYears,+ outputDir = aggregateReportDir+ } let aggregateOnlyReports = reportActions opts ch [transferBalance] aggregateParams ownerParams <- ownerParameters opts ch owners let ownerWithAggregateParams = (if length owners > 1 then [aggregateParams] else []) ++ ownerParams- let sharedOptions = ["--pretty-tables", "--depth", "2"]+ let sharedOptions = (if prettyReports opts then ["--pretty-tables"] else []) ++ ["--depth", "2"] let ownerWithAggregateReports = List.concat $ fmap (reportActions opts ch [incomeStatement sharedOptions, incomeMonthlyStatement sharedOptions, balanceSheet sharedOptions]) ownerWithAggregateParams let ownerOnlyReports = List.concat $ fmap (reportActions opts ch [accountList, unknownTransactions]) ownerParams parAwareActions opts (aggregateOnlyReports ++ ownerWithAggregateReports ++ ownerOnlyReports)@@ -101,7 +105,7 @@ transferBalance :: ReportGenerator transferBalance opts ch journal baseOutDir year = do- let reportArgs = ["balance", "--pretty-tables", "--quarterly", "--flat", "--no-total", "transfer"]+ let reportArgs = ["balance"] ++ (if prettyReports opts then ["--pretty-tables"] else []) ++ ["--quarterly", "--flat", "--no-total", "transfer"] generateReport opts ch journal year (baseOutDir </> intPath year) ("transfer-balance" <.> "txt") reportArgs (\txt -> (length $ T.lines txt) > 4) generateReport :: RuntimeOptions -> TChan FlowTypes.LogMessage -> TurtlePath -> Integer -> TurtlePath -> TurtlePath -> [T.Text] -> (T.Text -> Bool) -> IO (Either TurtlePath TurtlePath)@@ -113,33 +117,31 @@ let reportArgs = ["--file", Turtle.format Turtle.fp journal, "--period", Turtle.repr year] ++ args let reportDisplayArgs = ["--file", Turtle.format Turtle.fp relativeJournal, "--period", Turtle.repr year] ++ args let hledger = Turtle.format Turtle.fp $ pathToTurtle . FlowTypes.hlPath . hledgerInfo $ opts :: T.Text- let cmdLabel = Turtle.format ("hledger "%Turtle.s) $ showCmdArgs reportDisplayArgs+ let cmdLabel = Turtle.format ("hledger " % Turtle.s) $ showCmdArgs reportDisplayArgs ((exitCode, stdOut, _), _) <- timeAndExitOnErr opts ch cmdLabel dummyLogger channelErr Turtle.procStrictWithErr (hledger, reportArgs, mempty) if (successCheck stdOut)- then- do- Turtle.writeTextFile outputFile (cmdLabel <> "\n\n"<> stdOut)- logVerbose opts ch $ Turtle.format ("Wrote "%Turtle.fp) $ relativeOutputFile+ then do+ T.writeFile outputFile (cmdLabel <> "\n\n" <> stdOut)+ logVerbose opts ch $ Turtle.format ("Wrote " % Turtle.fp) $ relativeOutputFile return $ Right outputFile- else- do- channelErrLn ch $ Turtle.format ("Did not write '"%Turtle.fp%"' ("%Turtle.s%") "%Turtle.s) relativeOutputFile cmdLabel (Turtle.repr exitCode)+ else do+ channelErrLn ch $ Turtle.format ("Did not write '" % Turtle.fp % "' (" % Turtle.s % ") " % Turtle.s) relativeOutputFile cmdLabel (Turtle.repr exitCode) exists <- Turtle.testfile outputFile if exists then Turtle.rm outputFile else return () return $ Left outputFile journalFile :: RuntimeOptions -> [TurtlePath] -> TurtlePath-journalFile opts dirs = (foldl (</>) (turtleBaseDir opts) ("import":dirs)) </> allYearsFileName+journalFile opts dirs = (foldl (</>) (turtleBaseDir opts) ("import" : dirs)) </> allYearsFileName outputReportDir :: RuntimeOptions -> [TurtlePath] -> TurtlePath-outputReportDir opts dirs = foldl (</>) (turtleBaseDir opts) ("reports":dirs)+outputReportDir opts dirs = foldl (</>) (turtleBaseDir opts) ("reports" : dirs) ownerParameters :: RuntimeOptions -> TChan FlowTypes.LogMessage -> [TurtlePath] -> IO [ReportParams] ownerParameters opts ch owners = do let actions = map (ownerParameters' opts ch) owners parAwareActions opts actions -ownerParameters' :: RuntimeOptions -> TChan FlowTypes.LogMessage -> TurtlePath-> IO ReportParams+ownerParameters' :: RuntimeOptions -> TChan FlowTypes.LogMessage -> TurtlePath -> IO ReportParams ownerParameters' opts ch owner = do let ownerJournal = journalFile opts [owner] years <- includeYears ch ownerJournal
src/Hledger/Flow/RuntimeOptions.hs view
@@ -1,36 +1,40 @@-module Hledger.Flow.RuntimeOptions-where+module Hledger.Flow.RuntimeOptions where import qualified Data.Text as T-import Prelude hiding (putStrLn)-import Hledger.Flow.Types import Hledger.Flow.Internals (SystemInfo)+import Hledger.Flow.Types+import Prelude hiding (putStrLn) -data RuntimeOptions = RuntimeOptions { baseDir :: BaseDir- , importRunDir :: RunDir- , importStartYear :: Maybe Integer- , onlyNewFiles :: Bool- , hfVersion :: T.Text- , hledgerInfo :: HledgerInfo- , sysInfo :: SystemInfo- , verbose :: Bool- , showOptions :: Bool- , sequential :: Bool- , batchSize :: Int- }+data RuntimeOptions = RuntimeOptions+ { baseDir :: BaseDir,+ importRunDir :: RunDir,+ importStartYear :: Maybe Integer,+ onlyNewFiles :: Bool,+ hfVersion :: T.Text,+ hledgerInfo :: HledgerInfo,+ sysInfo :: SystemInfo,+ verbose :: Bool,+ showOptions :: Bool,+ sequential :: Bool,+ batchSize :: Int,+ prettyReports :: Bool+ } deriving (Show) instance HasVerbosity RuntimeOptions where- verbose (RuntimeOptions _ _ _ _ _ _ _ v _ _ _) = v+ verbose (RuntimeOptions _ _ _ _ _ _ _ v _ _ _ _) = v instance HasSequential RuntimeOptions where- sequential (RuntimeOptions _ _ _ _ _ _ _ _ _ sq _) = sq+ sequential (RuntimeOptions _ _ _ _ _ _ _ _ _ sq _ _) = sq instance HasBatchSize RuntimeOptions where- batchSize (RuntimeOptions _ _ _ _ _ _ _ _ _ _ bs) = bs+ batchSize (RuntimeOptions _ _ _ _ _ _ _ _ _ _ bs _) = bs instance HasBaseDir RuntimeOptions where- baseDir (RuntimeOptions bd _ _ _ _ _ _ _ _ _ _) = bd+ baseDir (RuntimeOptions bd _ _ _ _ _ _ _ _ _ _ _) = bd instance HasRunDir RuntimeOptions where- importRunDir (RuntimeOptions _ rd _ _ _ _ _ _ _ _ _) = rd+ importRunDir (RuntimeOptions _ rd _ _ _ _ _ _ _ _ _ _) = rd++instance HasPrettyReports RuntimeOptions where+ prettyReports (RuntimeOptions _ _ _ _ _ _ _ _ _ _ _ pr) = pr
src/Hledger/Flow/Types.hs view
@@ -2,25 +2,29 @@ module Hledger.Flow.Types where -import qualified Turtle (ExitCode, NominalDiffTime, Shell, Line) import qualified Data.Text as T- import Hledger.Flow.PathHelpers+import qualified Turtle (ExitCode, Line, NominalDiffTime, Shell) type BaseDir = AbsDir+ type RunDir = RelDir data LogMessage = StdOut T.Text | StdErr T.Text | Terminate deriving (Show)+ type FullOutput = (Turtle.ExitCode, T.Text, T.Text)+ type FullTimedOutput = (FullOutput, Turtle.NominalDiffTime) type ProcFun = T.Text -> [T.Text] -> Turtle.Shell Turtle.Line -> IO FullOutput+ type ProcInput = (T.Text, [T.Text], Turtle.Shell Turtle.Line) -data HledgerInfo = HledgerInfo { hlPath :: AbsFile- , hlVersion :: T.Text- }- deriving (Show)+data HledgerInfo = HledgerInfo+ { hlPath :: AbsFile,+ hlVersion :: T.Text+ }+ deriving (Show) class HasVerbosity a where verbose :: a -> Bool@@ -36,3 +40,6 @@ class HasBatchSize a where batchSize :: a -> Int++class HasPrettyReports a where+ prettyReports :: a -> Bool
test/BaseDir/Integration.hs view
@@ -1,22 +1,21 @@-{-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE OverloadedLists #-}+{-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE QuasiQuotes #-} module BaseDir.Integration (tests) where import Control.Exception (try)-import Test.HUnit--import Path-import Path.IO--import qualified Turtle import qualified Data.Text as T--import Hledger.Flow.Common-import Hledger.Flow.Types (BaseDir, RunDir)+import qualified Data.Text.IO as T import Hledger.Flow.BaseDir (determineBaseDir)+import Hledger.Flow.Common import Hledger.Flow.PathHelpers+import Hledger.Flow.Types (BaseDir, RunDir)+import Path+import Path.IO+import Test.HUnit+import qualified Turtle+import Prelude hiding (readFile, writeFile) assertSubDirsForDetermineBaseDir :: AbsDir -> BaseDir -> [Path.Path b Dir] -> IO () assertSubDirsForDetermineBaseDir initialPwd expectedBaseDir importDirs = do@@ -40,14 +39,14 @@ setCurrentDir initialPwd let msg dir = "determineBaseDir searches from pwd upwards until it finds a dir containing 'import' - " ++ show dir- sequence_ $ map (\ dir -> assertEqual (msg dir) expectedBaseDir dir) [bd1, bd2, bd3, bd4]+ sequence_ $ map (\dir -> assertEqual (msg dir) expectedBaseDir dir) [bd1, bd2, bd3, bd4] assertFindTestFileUsingRundir :: BaseDir -> RunDir -> IO () assertFindTestFileUsingRundir baseDir runDir = do let absRunDir = baseDir </> runDir found <- Turtle.single $ fmap head $ shellToList $ Turtle.find (Turtle.has "test-file.txt") $ pathToTurtle absRunDir- fileContents <- Turtle.readTextFile found+ fileContents <- T.readFile found assertEqual "We should find our test file by searching from the returned runDir" (T.pack $ "The expected base dir is " ++ show baseDir) fileContents assertCurrentDirVariations :: AbsDir -> RelDir -> IO ()@@ -61,7 +60,7 @@ (bd4, runDir4) <- determineBaseDir $ Just $ pathToTurtle absBaseDir let msg label dir = "When pwd is the base dir, determineBaseDir returns the same " ++ label ++ ", regardless of the input variation. " ++ show dir- sequence_ $ map (\ dir -> assertEqual (msg "baseDir" dir) absBaseDir dir) [bd1, bd2, bd3, bd4]+ sequence_ $ map (\dir -> assertEqual (msg "baseDir" dir) absBaseDir dir) [bd1, bd2, bd3, bd4] sequence_ $ map (\dir -> assertEqual (msg "runDir" dir) [reldir|.|] dir) [runDir1, runDir2, runDir3, runDir4] testBaseDirWithTempDir :: AbsDir -> AbsDir -> IO ()@@ -98,7 +97,7 @@ let absoluteBaseDir = absoluteTempDir </> baseDir - Turtle.writeTextFile (pathToTurtle $ absoluteTempDir </> yearDir </> [relfile|test-file.txt|]) (T.pack $ "The expected base dir is " ++ show absoluteBaseDir)+ T.writeFile (pathToTurtle $ absoluteTempDir </> yearDir </> [relfile|test-file.txt|]) (T.pack $ "The expected base dir is " ++ show absoluteBaseDir) assertSubDirsForDetermineBaseDir absoluteTempDir absoluteBaseDir subDirs assertSubDirsForDetermineBaseDir absoluteTempDir absoluteBaseDir absoluteSubDirs@@ -131,21 +130,23 @@ withCurrentDir baseDir $ assertRunDirs accDir [accDir, bankDir, ownerDir, importDir] [yearDir, inDir] testRunDirs :: Test-testRunDirs = TestCase (- do- initialPwd <- getCurrentDir- let tmpbase = initialPwd </> [reldir|test|] </> [reldir|tmp|]- withTempDir tmpbase "hlflowtest" testRunDirsWithTempDir- )+testRunDirs =+ TestCase+ ( do+ initialPwd <- getCurrentDir+ let tmpbase = initialPwd </> [reldir|test|] </> [reldir|tmp|]+ withTempDir tmpbase "hlflowtest" testRunDirsWithTempDir+ ) testDetermineBaseDir :: Test-testDetermineBaseDir = TestCase (- do- initialPwd <- getCurrentDir- let tmpbase = initialPwd </> [reldir|test|] </> [reldir|tmp|]- createDirIfMissing True tmpbase- withTempDir tmpbase "hlflowtest" $ testBaseDirWithTempDir initialPwd- )+testDetermineBaseDir =+ TestCase+ ( do+ initialPwd <- getCurrentDir+ let tmpbase = initialPwd </> [reldir|test|] </> [reldir|tmp|]+ createDirIfMissing True tmpbase+ withTempDir tmpbase "hlflowtest" $ testBaseDirWithTempDir initialPwd+ ) tests :: Test tests = TestList [testDetermineBaseDir, testRunDirs]
test/CSVImport/ImportHelperTests.hs view
@@ -1,119 +1,115 @@-{-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE OverloadedLists #-}+{-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE QuasiQuotes #-} module CSVImport.ImportHelperTests where -import Test.HUnit--import Hledger.Flow.Import.Types (InputFileBundle) import Hledger.Flow.Import.ImportHelpers (groupIncludesUpTo, includeFileName)-+import Hledger.Flow.Import.Types (InputFileBundle) import Path+import Test.HUnit import TestHelpers testToJournal :: Test-testToJournal = TestCase (- do- let journal = toJournal [relfile|import/jane/bogartbank/investment/1-in/2020/2020-09-30.csv|]- let expected = [relfile|import/jane/bogartbank/investment/3-journal/2020/2020-09-30.journal|]- assertEqual "toJournal" expected journal- )+testToJournal =+ TestCase+ ( do+ let journal = toJournal [relfile|import/jane/bogartbank/investment/1-in/2020/2020-09-30.csv|]+ let expected = [relfile|import/jane/bogartbank/investment/3-journal/2020/2020-09-30.journal|]+ assertEqual "toJournal" expected journal+ ) testIncludeFileName :: Test-testIncludeFileName = TestCase (- do- let includeFile = includeFileName [relfile|import/jane/bogartbank/investment/3-journals/2020/2020-09-30.journal|]- assertEqual "includeFileName" [relfile|2020-include.journal|] includeFile- )+testIncludeFileName =+ TestCase+ ( do+ let includeFile = includeFileName [relfile|import/jane/bogartbank/investment/3-journals/2020/2020-09-30.journal|]+ assertEqual "includeFileName" [relfile|2020-include.journal|] includeFile+ ) testGroupIncludesUpToTinySet :: Test-testGroupIncludesUpToTinySet = TestCase (- do- let expected = [- ([relfile|import/jane/bogartbank/savings/2017-include.journal|], [janeSavingsJournal2017])- , ([relfile|import/jane/bogartbank/2017-include.journal|], [[relfile|import/jane/bogartbank/savings/2017-include.journal|]])- , ([relfile|import/jane/2017-include.journal|], [[relfile|import/jane/bogartbank/2017-include.journal|]])- ] :: InputFileBundle+testGroupIncludesUpToTinySet =+ TestCase+ ( do+ let expected =+ [ ([relfile|import/jane/bogartbank/savings/2017-include.journal|], [janeSavingsJournal2017]),+ ([relfile|import/jane/bogartbank/2017-include.journal|], [[relfile|import/jane/bogartbank/savings/2017-include.journal|]]),+ ([relfile|import/jane/2017-include.journal|], [[relfile|import/jane/bogartbank/2017-include.journal|]])+ ] ::+ InputFileBundle - let grouped = groupIncludesUpTo [reldir|import/jane|] [janeSavingsJournal2017]- assertEqual "groupIncludesUpTo: A single journal file grouping" expected grouped- )+ let grouped = groupIncludesUpTo [reldir|import/jane|] [janeSavingsJournal2017]+ assertEqual "groupIncludesUpTo: A single journal file grouping" expected grouped+ ) testGroupIncludesUpToSmallSet :: Test-testGroupIncludesUpToSmallSet = TestCase (- do- let expected = [- ([relfile|import/jane/bogartbank/savings/2017-include.journal|], [janeSavingsJournal2017])- , ([relfile|import/jane/bogartbank/savings/2018-include.journal|], janeSavingsJournals2018)- , ([relfile|import/jane/bogartbank/savings/2019-include.journal|], [janeSavingsJournal2019])- , ([relfile|import/jane/bogartbank/2017-include.journal|], [[relfile|import/jane/bogartbank/savings/2017-include.journal|]])- , ([relfile|import/jane/bogartbank/2018-include.journal|], [[relfile|import/jane/bogartbank/savings/2018-include.journal|]])- , ([relfile|import/jane/bogartbank/2019-include.journal|], [[relfile|import/jane/bogartbank/savings/2019-include.journal|]])- , ([relfile|import/jane/2017-include.journal|], [[relfile|import/jane/bogartbank/2017-include.journal|]])- , ([relfile|import/jane/2018-include.journal|], [[relfile|import/jane/bogartbank/2018-include.journal|]])- , ([relfile|import/jane/2019-include.journal|], [[relfile|import/jane/bogartbank/2019-include.journal|]])- ] :: InputFileBundle+testGroupIncludesUpToSmallSet =+ TestCase+ ( do+ let expected =+ [ ([relfile|import/jane/bogartbank/savings/2017-include.journal|], [janeSavingsJournal2017]),+ ([relfile|import/jane/bogartbank/savings/2018-include.journal|], janeSavingsJournals2018),+ ([relfile|import/jane/bogartbank/savings/2019-include.journal|], [janeSavingsJournal2019]),+ ([relfile|import/jane/bogartbank/2017-include.journal|], [[relfile|import/jane/bogartbank/savings/2017-include.journal|]]),+ ([relfile|import/jane/bogartbank/2018-include.journal|], [[relfile|import/jane/bogartbank/savings/2018-include.journal|]]),+ ([relfile|import/jane/bogartbank/2019-include.journal|], [[relfile|import/jane/bogartbank/savings/2019-include.journal|]]),+ ([relfile|import/jane/2017-include.journal|], [[relfile|import/jane/bogartbank/2017-include.journal|]]),+ ([relfile|import/jane/2018-include.journal|], [[relfile|import/jane/bogartbank/2018-include.journal|]]),+ ([relfile|import/jane/2019-include.journal|], [[relfile|import/jane/bogartbank/2019-include.journal|]])+ ] ::+ InputFileBundle - let grouped = groupIncludesUpTo [reldir|import/jane|] janeSavingsJournals- assertEqual "groupIncludesUpTo: A small set of journal files - same account over 3 years" expected grouped- )+ let grouped = groupIncludesUpTo [reldir|import/jane|] janeSavingsJournals+ assertEqual "groupIncludesUpTo: A small set of journal files - same account over 3 years" expected grouped+ ) testGroupIncludesUpTo :: Test-testGroupIncludesUpTo = TestCase (- do- let expected = [- ([relfile|import/john/bogartbank/savings/2017-include.journal|], johnSavingsJournals2017)- , ([relfile|import/john/bogartbank/savings/2018-include.journal|], johnSavingsJournals2018)- , ([relfile|import/john/bogartbank/checking/2018-include.journal|], johnCheckingJournals2018)- , ([relfile|import/john/bogartbank/checking/2019-include.journal|], johnCheckingJournals2019)-- , ([relfile|import/john/otherbank/creditcard/2017-include.journal|], [johnCCJournal2017])- , ([relfile|import/john/otherbank/creditcard/2018-include.journal|], [johnCCJournal2018])- , ([relfile|import/john/otherbank/investments/2018-include.journal|], [johnInvestJournal2018])- , ([relfile|import/john/otherbank/investments/2019-include.journal|], [johnInvestJournal2019])-- , ([relfile|import/jane/bogartbank/savings/2017-include.journal|], [janeSavingsJournal2017])- , ([relfile|import/jane/bogartbank/savings/2018-include.journal|], janeSavingsJournals2018)- , ([relfile|import/jane/bogartbank/savings/2019-include.journal|], [janeSavingsJournal2019])-- , ([relfile|import/jane/otherbank/creditcard/2017-include.journal|], [janeCCJournal2017])- , ([relfile|import/jane/otherbank/creditcard/2018-include.journal|], [janeCCJournal2018])- , ([relfile|import/jane/otherbank/investments/2018-include.journal|], [janeInvestJournal2018])- , ([relfile|import/jane/otherbank/investments/2019-include.journal|], [janeInvestJournal2019])-- , ([relfile|import/john/bogartbank/2017-include.journal|], [[relfile|import/john/bogartbank/savings/2017-include.journal|]])- , ([relfile|import/john/bogartbank/2018-include.journal|], [[relfile|import/john/bogartbank/checking/2018-include.journal|], [relfile|import/john/bogartbank/savings/2018-include.journal|]])- , ([relfile|import/john/bogartbank/2019-include.journal|], [[relfile|import/john/bogartbank/checking/2019-include.journal|]])-- , ([relfile|import/john/otherbank/2017-include.journal|], [[relfile|import/john/otherbank/creditcard/2017-include.journal|]])- , ([relfile|import/john/otherbank/2018-include.journal|], [[relfile|import/john/otherbank/creditcard/2018-include.journal|], [relfile|import/john/otherbank/investments/2018-include.journal|]])- , ([relfile|import/john/otherbank/2019-include.journal|], [[relfile|import/john/otherbank/investments/2019-include.journal|]])-- , ([relfile|import/jane/bogartbank/2017-include.journal|], [[relfile|import/jane/bogartbank/savings/2017-include.journal|]])- , ([relfile|import/jane/bogartbank/2018-include.journal|], [[relfile|import/jane/bogartbank/savings/2018-include.journal|]])- , ([relfile|import/jane/bogartbank/2019-include.journal|], [[relfile|import/jane/bogartbank/savings/2019-include.journal|]])-- , ([relfile|import/jane/otherbank/2017-include.journal|], [[relfile|import/jane/otherbank/creditcard/2017-include.journal|]])- , ([relfile|import/jane/otherbank/2018-include.journal|], [[relfile|import/jane/otherbank/creditcard/2018-include.journal|], [relfile|import/jane/otherbank/investments/2018-include.journal|]])- , ([relfile|import/jane/otherbank/2019-include.journal|], [[relfile|import/jane/otherbank/investments/2019-include.journal|]])-- , ([relfile|import/john/2017-include.journal|], [[relfile|import/john/bogartbank/2017-include.journal|], [relfile|import/john/otherbank/2017-include.journal|]])- , ([relfile|import/john/2018-include.journal|], [[relfile|import/john/bogartbank/2018-include.journal|], [relfile|import/john/otherbank/2018-include.journal|]])- , ([relfile|import/john/2019-include.journal|], [[relfile|import/john/bogartbank/2019-include.journal|], [relfile|import/john/otherbank/2019-include.journal|]])-- , ([relfile|import/jane/2017-include.journal|], [[relfile|import/jane/bogartbank/2017-include.journal|], [relfile|import/jane/otherbank/2017-include.journal|]])- , ([relfile|import/jane/2018-include.journal|], [[relfile|import/jane/bogartbank/2018-include.journal|], [relfile|import/jane/otherbank/2018-include.journal|]])- , ([relfile|import/jane/2019-include.journal|], [[relfile|import/jane/bogartbank/2019-include.journal|], [relfile|import/jane/otherbank/2019-include.journal|]])-- , ([relfile|import/2017-include.journal|], [[relfile|import/jane/2017-include.journal|], [relfile|import/john/2017-include.journal|]])- , ([relfile|import/2018-include.journal|], [[relfile|import/jane/2018-include.journal|], [relfile|import/john/2018-include.journal|]])- , ([relfile|import/2019-include.journal|], [[relfile|import/jane/2019-include.journal|], [relfile|import/john/2019-include.journal|]])- ] :: InputFileBundle+testGroupIncludesUpTo =+ TestCase+ ( do+ let expected =+ [ ([relfile|import/john/bogartbank/savings/2017-include.journal|], johnSavingsJournals2017),+ ([relfile|import/john/bogartbank/savings/2018-include.journal|], johnSavingsJournals2018),+ ([relfile|import/john/bogartbank/checking/2018-include.journal|], johnCheckingJournals2018),+ ([relfile|import/john/bogartbank/checking/2019-include.journal|], johnCheckingJournals2019),+ ([relfile|import/john/otherbank/creditcard/2017-include.journal|], [johnCCJournal2017]),+ ([relfile|import/john/otherbank/creditcard/2018-include.journal|], [johnCCJournal2018]),+ ([relfile|import/john/otherbank/investments/2018-include.journal|], [johnInvestJournal2018]),+ ([relfile|import/john/otherbank/investments/2019-include.journal|], [johnInvestJournal2019]),+ ([relfile|import/jane/bogartbank/savings/2017-include.journal|], [janeSavingsJournal2017]),+ ([relfile|import/jane/bogartbank/savings/2018-include.journal|], janeSavingsJournals2018),+ ([relfile|import/jane/bogartbank/savings/2019-include.journal|], [janeSavingsJournal2019]),+ ([relfile|import/jane/otherbank/creditcard/2017-include.journal|], [janeCCJournal2017]),+ ([relfile|import/jane/otherbank/creditcard/2018-include.journal|], [janeCCJournal2018]),+ ([relfile|import/jane/otherbank/investments/2018-include.journal|], [janeInvestJournal2018]),+ ([relfile|import/jane/otherbank/investments/2019-include.journal|], [janeInvestJournal2019]),+ ([relfile|import/john/bogartbank/2017-include.journal|], [[relfile|import/john/bogartbank/savings/2017-include.journal|]]),+ ([relfile|import/john/bogartbank/2018-include.journal|], [[relfile|import/john/bogartbank/checking/2018-include.journal|], [relfile|import/john/bogartbank/savings/2018-include.journal|]]),+ ([relfile|import/john/bogartbank/2019-include.journal|], [[relfile|import/john/bogartbank/checking/2019-include.journal|]]),+ ([relfile|import/john/otherbank/2017-include.journal|], [[relfile|import/john/otherbank/creditcard/2017-include.journal|]]),+ ([relfile|import/john/otherbank/2018-include.journal|], [[relfile|import/john/otherbank/creditcard/2018-include.journal|], [relfile|import/john/otherbank/investments/2018-include.journal|]]),+ ([relfile|import/john/otherbank/2019-include.journal|], [[relfile|import/john/otherbank/investments/2019-include.journal|]]),+ ([relfile|import/jane/bogartbank/2017-include.journal|], [[relfile|import/jane/bogartbank/savings/2017-include.journal|]]),+ ([relfile|import/jane/bogartbank/2018-include.journal|], [[relfile|import/jane/bogartbank/savings/2018-include.journal|]]),+ ([relfile|import/jane/bogartbank/2019-include.journal|], [[relfile|import/jane/bogartbank/savings/2019-include.journal|]]),+ ([relfile|import/jane/otherbank/2017-include.journal|], [[relfile|import/jane/otherbank/creditcard/2017-include.journal|]]),+ ([relfile|import/jane/otherbank/2018-include.journal|], [[relfile|import/jane/otherbank/creditcard/2018-include.journal|], [relfile|import/jane/otherbank/investments/2018-include.journal|]]),+ ([relfile|import/jane/otherbank/2019-include.journal|], [[relfile|import/jane/otherbank/investments/2019-include.journal|]]),+ ([relfile|import/john/2017-include.journal|], [[relfile|import/john/bogartbank/2017-include.journal|], [relfile|import/john/otherbank/2017-include.journal|]]),+ ([relfile|import/john/2018-include.journal|], [[relfile|import/john/bogartbank/2018-include.journal|], [relfile|import/john/otherbank/2018-include.journal|]]),+ ([relfile|import/john/2019-include.journal|], [[relfile|import/john/bogartbank/2019-include.journal|], [relfile|import/john/otherbank/2019-include.journal|]]),+ ([relfile|import/jane/2017-include.journal|], [[relfile|import/jane/bogartbank/2017-include.journal|], [relfile|import/jane/otherbank/2017-include.journal|]]),+ ([relfile|import/jane/2018-include.journal|], [[relfile|import/jane/bogartbank/2018-include.journal|], [relfile|import/jane/otherbank/2018-include.journal|]]),+ ([relfile|import/jane/2019-include.journal|], [[relfile|import/jane/bogartbank/2019-include.journal|], [relfile|import/jane/otherbank/2019-include.journal|]]),+ ([relfile|import/2017-include.journal|], [[relfile|import/jane/2017-include.journal|], [relfile|import/john/2017-include.journal|]]),+ ([relfile|import/2018-include.journal|], [[relfile|import/jane/2018-include.journal|], [relfile|import/john/2018-include.journal|]]),+ ([relfile|import/2019-include.journal|], [[relfile|import/jane/2019-include.journal|], [relfile|import/john/2019-include.journal|]])+ ] ::+ InputFileBundle - let grouped = groupIncludesUpTo [reldir|import|] journalFiles- assertEqual "groupIncludesUpTo: A full set of journal files" expected grouped- )+ let grouped = groupIncludesUpTo [reldir|import|] journalFiles+ assertEqual "groupIncludesUpTo: A full set of journal files" expected grouped+ ) tests :: Test tests = TestList [testToJournal, testIncludeFileName, testGroupIncludesUpToTinySet, testGroupIncludesUpToSmallSet, testGroupIncludesUpTo]
test/CSVImport/ImportHelperTurtleTests.hs view
@@ -1,432 +1,590 @@-{-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE OverloadedLists #-}+{-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE QuasiQuotes #-} module CSVImport.ImportHelperTurtleTests where -import Test.HUnit-import Path-import Turtle-import qualified Data.Map.Strict as Map import Control.Concurrent.STM--import TestHelpers (defaultOpts)-import TestHelpersTurtle-import Hledger.Flow.PathHelpers (TurtlePath)-import Hledger.Flow.Common-import Hledger.Flow.Import.Types (TurtleFileBundle)-import Hledger.Flow.Import.ImportHelpersTurtle (allYearIncludeFiles, groupIncludeFiles, toIncludeFiles, toIncludeLine, yearsIncludeMap)- import Data.Either+import qualified Data.Map.Strict as Map import qualified Data.Text as T+import Hledger.Flow.Common+import Hledger.Flow.Import.ImportHelpersTurtle (allYearIncludeFiles, groupIncludeFiles, toIncludeFiles, toIncludeLine, yearsIncludeMap)+import Hledger.Flow.Import.Types (TurtleFileBundle)+import Hledger.Flow.PathHelpers (TurtlePath)+import Path+import Test.HUnit+import TestHelpers (defaultOpts)+import TestHelpersTurtle+import Turtle groupedJaneBogart :: TurtleFileBundle-groupedJaneBogart = [- ("./import/jane/bogartbank/checking/2018-include.journal",- ["import/jane/bogartbank/checking/3-journal/2018/2018-12-30.journal"]),- ("./import/jane/bogartbank/checking/2019-include.journal",- ["import/jane/bogartbank/checking/3-journal/2019/2019-01-30.journal"]),- ("./import/jane/bogartbank/savings/2017-include.journal",- ["import/jane/bogartbank/savings/3-journal/2017/2017-12-30.journal"]),- ("./import/jane/bogartbank/savings/2018-include.journal",- ["import/jane/bogartbank/savings/3-journal/2018/2018-01-30.journal"])]+groupedJaneBogart =+ [ ( "./import/jane/bogartbank/checking/2018-include.journal",+ ["import/jane/bogartbank/checking/3-journal/2018/2018-12-30.journal"]+ ),+ ( "./import/jane/bogartbank/checking/2019-include.journal",+ ["import/jane/bogartbank/checking/3-journal/2019/2019-01-30.journal"]+ ),+ ( "./import/jane/bogartbank/savings/2017-include.journal",+ ["import/jane/bogartbank/savings/3-journal/2017/2017-12-30.journal"]+ ),+ ( "./import/jane/bogartbank/savings/2018-include.journal",+ ["import/jane/bogartbank/savings/3-journal/2018/2018-01-30.journal"]+ )+ ] groupedJaneOther :: TurtleFileBundle-groupedJaneOther = [- ("./import/jane/otherbank/creditcard/2017-include.journal",- ["import/jane/otherbank/creditcard/3-journal/2017/2017-12-30.journal"]),- ("./import/jane/otherbank/creditcard/2018-include.journal",- ["import/jane/otherbank/creditcard/3-journal/2018/2018-01-30.journal"]),- ("./import/jane/otherbank/investments/2018-include.journal",- ["import/jane/otherbank/investments/3-journal/2018/2018-12-30.journal"]),- ("./import/jane/otherbank/investments/2019-include.journal",- ["import/jane/otherbank/investments/3-journal/2019/2019-01-30.journal"])]+groupedJaneOther =+ [ ( "./import/jane/otherbank/creditcard/2017-include.journal",+ ["import/jane/otherbank/creditcard/3-journal/2017/2017-12-30.journal"]+ ),+ ( "./import/jane/otherbank/creditcard/2018-include.journal",+ ["import/jane/otherbank/creditcard/3-journal/2018/2018-01-30.journal"]+ ),+ ( "./import/jane/otherbank/investments/2018-include.journal",+ ["import/jane/otherbank/investments/3-journal/2018/2018-12-30.journal"]+ ),+ ( "./import/jane/otherbank/investments/2019-include.journal",+ ["import/jane/otherbank/investments/3-journal/2019/2019-01-30.journal"]+ )+ ] groupedJohnBogart :: TurtleFileBundle-groupedJohnBogart = [- ("./import/john/bogartbank/checking/2018-include.journal",- ["import/john/bogartbank/checking/3-journal/2018/2018-11-30.journal",- "import/john/bogartbank/checking/3-journal/2018/2018-10-30.journal",- "import/john/bogartbank/checking/3-journal/2018/2018-12-30.journal"]),- ("./import/john/bogartbank/checking/2019-include.journal",- ["import/john/bogartbank/checking/3-journal/2019/2019-01-30.journal",- "import/john/bogartbank/checking/3-journal/2019/2019-02-30.journal"]),- ("./import/john/bogartbank/savings/2017-include.journal",- ["import/john/bogartbank/savings/3-journal/2017/2017-11-30.journal",- "import/john/bogartbank/savings/3-journal/2017/2017-12-30.journal"]),- ("./import/john/bogartbank/savings/2018-include.journal",- ["import/john/bogartbank/savings/3-journal/2018/2018-02-30.journal",- "import/john/bogartbank/savings/3-journal/2018/2018-01-30.journal"])]+groupedJohnBogart =+ [ ( "./import/john/bogartbank/checking/2018-include.journal",+ [ "import/john/bogartbank/checking/3-journal/2018/2018-11-30.journal",+ "import/john/bogartbank/checking/3-journal/2018/2018-10-30.journal",+ "import/john/bogartbank/checking/3-journal/2018/2018-12-30.journal"+ ]+ ),+ ( "./import/john/bogartbank/checking/2019-include.journal",+ [ "import/john/bogartbank/checking/3-journal/2019/2019-01-30.journal",+ "import/john/bogartbank/checking/3-journal/2019/2019-02-30.journal"+ ]+ ),+ ( "./import/john/bogartbank/savings/2017-include.journal",+ [ "import/john/bogartbank/savings/3-journal/2017/2017-11-30.journal",+ "import/john/bogartbank/savings/3-journal/2017/2017-12-30.journal"+ ]+ ),+ ( "./import/john/bogartbank/savings/2018-include.journal",+ [ "import/john/bogartbank/savings/3-journal/2018/2018-02-30.journal",+ "import/john/bogartbank/savings/3-journal/2018/2018-01-30.journal"+ ]+ )+ ] groupedJohnOther :: TurtleFileBundle-groupedJohnOther = [- ("./import/john/otherbank/creditcard/2017-include.journal",- ["import/john/otherbank/creditcard/3-journal/2017/2017-12-30.journal"]),- ("./import/john/otherbank/creditcard/2018-include.journal",- ["import/john/otherbank/creditcard/3-journal/2018/2018-01-30.journal"]),- ("./import/john/otherbank/investments/2018-include.journal",- ["import/john/otherbank/investments/3-journal/2018/2018-12-30.journal"]),- ("./import/john/otherbank/investments/2019-include.journal",- ["import/john/otherbank/investments/3-journal/2019/2019-01-30.journal"])]+groupedJohnOther =+ [ ( "./import/john/otherbank/creditcard/2017-include.journal",+ ["import/john/otherbank/creditcard/3-journal/2017/2017-12-30.journal"]+ ),+ ( "./import/john/otherbank/creditcard/2018-include.journal",+ ["import/john/otherbank/creditcard/3-journal/2018/2018-01-30.journal"]+ ),+ ( "./import/john/otherbank/investments/2018-include.journal",+ ["import/john/otherbank/investments/3-journal/2018/2018-12-30.journal"]+ ),+ ( "./import/john/otherbank/investments/2019-include.journal",+ ["import/john/otherbank/investments/3-journal/2019/2019-01-30.journal"]+ )+ ] groupedIncludeFiles :: TurtleFileBundle-groupedIncludeFiles = groupedJaneBogart <> groupedJaneOther <>- groupedJohnBogart <> groupedJohnOther+groupedIncludeFiles =+ groupedJaneBogart+ <> groupedJaneOther+ <> groupedJohnBogart+ <> groupedJohnOther testYearsIncludeMap :: Test-testYearsIncludeMap = TestCase (- do- let maps = allYearIncludeFiles groupedJohnOther- let yearGrouping = [("./import/john/otherbank/creditcard/all-years.journal",- ["./import/john/otherbank/creditcard/2017-include.journal",- "./import/john/otherbank/creditcard/2018-include.journal"]),- ("./import/john/otherbank/investments/all-years.journal",- ["./import/john/otherbank/investments/2018-include.journal",- "./import/john/otherbank/investments/2019-include.journal"])]- let expected = (groupedJohnOther, yearGrouping)- assertEqual "An augmented map with grouped years per level added" expected maps- )+testYearsIncludeMap =+ TestCase+ ( do+ let maps = allYearIncludeFiles groupedJohnOther+ let yearGrouping =+ [ ( "./import/john/otherbank/creditcard/all-years.journal",+ [ "./import/john/otherbank/creditcard/2017-include.journal",+ "./import/john/otherbank/creditcard/2018-include.journal"+ ]+ ),+ ( "./import/john/otherbank/investments/all-years.journal",+ [ "./import/john/otherbank/investments/2018-include.journal",+ "./import/john/otherbank/investments/2019-include.journal"+ ]+ )+ ]+ let expected = (groupedJohnOther, yearGrouping)+ assertEqual "An augmented map with grouped years per level added" expected maps+ ) testYearsIncludeGrouping :: Test-testYearsIncludeGrouping = TestCase (- do- let yearsMap = yearsIncludeMap (Map.keys groupedJohnOther)- let expected = [("./import/john/otherbank/creditcard/all-years.journal",- ["./import/john/otherbank/creditcard/2017-include.journal",- "./import/john/otherbank/creditcard/2018-include.journal"]),- ("./import/john/otherbank/investments/all-years.journal",- ["./import/john/otherbank/investments/2018-include.journal",- "./import/john/otherbank/investments/2019-include.journal"])]- assertEqual "A basic map with grouped years per level" expected yearsMap- )+testYearsIncludeGrouping =+ TestCase+ ( do+ let yearsMap = yearsIncludeMap (Map.keys groupedJohnOther)+ let expected =+ [ ( "./import/john/otherbank/creditcard/all-years.journal",+ [ "./import/john/otherbank/creditcard/2017-include.journal",+ "./import/john/otherbank/creditcard/2018-include.journal"+ ]+ ),+ ( "./import/john/otherbank/investments/all-years.journal",+ [ "./import/john/otherbank/investments/2018-include.journal",+ "./import/john/otherbank/investments/2019-include.journal"+ ]+ )+ ]+ assertEqual "A basic map with grouped years per level" expected yearsMap+ ) testGroupIncludeFilesTinySet :: Test-testGroupIncludeFilesTinySet = TestCase (- do- let journals1 = [ "import/jane/bogartbank/savings/3-journals/2017/2017-12-30.journal"]- let expected1 = [("./import/jane/bogartbank/savings/2017-include.journal", journals1)] :: TurtleFileBundle- let expectedAllYears1 = [("./import/jane/bogartbank/savings/all-years.journal", ["./import/jane/bogartbank/savings/2017-include.journal"])]- let (group1, allYears1) = groupIncludeFiles journals1- assertEqual "groupIncludeFiles small allYears 1" expectedAllYears1 allYears1- assertEqual "groupIncludeFiles small set 1" expected1 group1+testGroupIncludeFilesTinySet =+ TestCase+ ( do+ let journals1 = ["import/jane/bogartbank/savings/3-journals/2017/2017-12-30.journal"]+ let expected1 = [("./import/jane/bogartbank/savings/2017-include.journal", journals1)] :: TurtleFileBundle+ let expectedAllYears1 = [("./import/jane/bogartbank/savings/all-years.journal", ["./import/jane/bogartbank/savings/2017-include.journal"])]+ let (group1, allYears1) = groupIncludeFiles journals1+ assertEqual "groupIncludeFiles small allYears 1" expectedAllYears1 allYears1+ assertEqual "groupIncludeFiles small set 1" expected1 group1 - let journals2 = [(fst . head . Map.toList) expected1] :: [TurtlePath]- let expected2 = [("./import/jane/bogartbank/2017-include.journal", journals2)] :: TurtleFileBundle- let expectedAllYears2 = [("./import/jane/bogartbank/all-years.journal", ["./import/jane/bogartbank/2017-include.journal"])]- let (group2, allYears2) = groupIncludeFiles journals2- assertEqual "groupIncludeFiles small allYears 2" expectedAllYears2 allYears2- assertEqual "groupIncludeFiles small set 2" expected2 group2+ let journals2 = [(fst . head . Map.toList) expected1] :: [TurtlePath]+ let expected2 = [("./import/jane/bogartbank/2017-include.journal", journals2)] :: TurtleFileBundle+ let expectedAllYears2 = [("./import/jane/bogartbank/all-years.journal", ["./import/jane/bogartbank/2017-include.journal"])]+ let (group2, allYears2) = groupIncludeFiles journals2+ assertEqual "groupIncludeFiles small allYears 2" expectedAllYears2 allYears2+ assertEqual "groupIncludeFiles small set 2" expected2 group2 - let journals3 = [(fst . head . Map.toList) expected2] :: [TurtlePath]- let expected3 = [("./import/jane/2017-include.journal", journals3)] :: TurtleFileBundle- let expectedAllYears3 = [("./import/jane/all-years.journal", ["./import/jane/2017-include.journal"])]- let (group3, allYears3) = groupIncludeFiles journals3- assertEqual "groupIncludeFiles small allYears 3" expectedAllYears3 allYears3- assertEqual "groupIncludeFiles small set 3" expected3 group3- )+ let journals3 = [(fst . head . Map.toList) expected2] :: [TurtlePath]+ let expected3 = [("./import/jane/2017-include.journal", journals3)] :: TurtleFileBundle+ let expectedAllYears3 = [("./import/jane/all-years.journal", ["./import/jane/2017-include.journal"])]+ let (group3, allYears3) = groupIncludeFiles journals3+ assertEqual "groupIncludeFiles small allYears 3" expectedAllYears3 allYears3+ assertEqual "groupIncludeFiles small set 3" expected3 group3+ ) testGroupIncludeFilesSmallSet :: Test-testGroupIncludeFilesSmallSet = TestCase (- do- let (group1, allYears1) = groupIncludeFiles (toJournals inputJaneBogart)- let expectedAllYears1 = [- ("./import/jane/bogartbank/checking/all-years.journal",- ["./import/jane/bogartbank/checking/2018-include.journal",- "./import/jane/bogartbank/checking/2019-include.journal"]),- ("./import/jane/bogartbank/savings/all-years.journal",- ["./import/jane/bogartbank/savings/2017-include.journal",- "./import/jane/bogartbank/savings/2018-include.journal"])]+testGroupIncludeFilesSmallSet =+ TestCase+ ( do+ let (group1, allYears1) = groupIncludeFiles $ toJournals inputJaneBogart+ let expectedAllYears1 =+ [ ( "./import/jane/bogartbank/checking/all-years.journal",+ [ "./import/jane/bogartbank/checking/2018-include.journal",+ "./import/jane/bogartbank/checking/2019-include.journal"+ ]+ ),+ ( "./import/jane/bogartbank/savings/all-years.journal",+ [ "./import/jane/bogartbank/savings/2017-include.journal",+ "./import/jane/bogartbank/savings/2018-include.journal"+ ]+ )+ ] - assertEqual "groupIncludeFiles Jane AllYears 1" expectedAllYears1 allYears1- assertEqual "groupIncludeFiles Jane 1" groupedJaneBogart group1+ assertEqual "groupIncludeFiles Jane AllYears 1" expectedAllYears1 allYears1+ assertEqual "groupIncludeFiles Jane 1" groupedJaneBogart group1 - let (group2, allYears2) = groupIncludeFiles (Map.keys group1)- let expectedAllYears2 = [- ("./import/jane/bogartbank/all-years.journal",- ["./import/jane/bogartbank/2017-include.journal",- "./import/jane/bogartbank/2018-include.journal",- "./import/jane/bogartbank/2019-include.journal"])]- assertEqual "groupIncludeFiles Jane AllYears 2" expectedAllYears2 allYears2- let expectedGroup2 = [- ("./import/jane/bogartbank/2017-include.journal",- ["./import/jane/bogartbank/savings/2017-include.journal"]),- ("./import/jane/bogartbank/2018-include.journal",- ["./import/jane/bogartbank/checking/2018-include.journal",- "./import/jane/bogartbank/savings/2018-include.journal"]),- ("./import/jane/bogartbank/2019-include.journal",- ["./import/jane/bogartbank/checking/2019-include.journal"])]- assertEqual "groupIncludeFiles Jane 2" expectedGroup2 group2+ let (group2, allYears2) = groupIncludeFiles (Map.keys group1)+ let expectedAllYears2 =+ [ ( "./import/jane/bogartbank/all-years.journal",+ [ "./import/jane/bogartbank/2017-include.journal",+ "./import/jane/bogartbank/2018-include.journal",+ "./import/jane/bogartbank/2019-include.journal"+ ]+ )+ ]+ assertEqual "groupIncludeFiles Jane AllYears 2" expectedAllYears2 allYears2+ let expectedGroup2 =+ [ ( "./import/jane/bogartbank/2017-include.journal",+ ["./import/jane/bogartbank/savings/2017-include.journal"]+ ),+ ( "./import/jane/bogartbank/2018-include.journal",+ [ "./import/jane/bogartbank/checking/2018-include.journal",+ "./import/jane/bogartbank/savings/2018-include.journal"+ ]+ ),+ ( "./import/jane/bogartbank/2019-include.journal",+ ["./import/jane/bogartbank/checking/2019-include.journal"]+ )+ ]+ assertEqual "groupIncludeFiles Jane 2" expectedGroup2 group2 - let (group3, allYears3) = groupIncludeFiles (Map.keys group2)- let expectedAllYears3 = [- ("./import/jane/all-years.journal",- ["./import/jane/2017-include.journal",- "./import/jane/2018-include.journal",- "./import/jane/2019-include.journal"])]- assertEqual "groupIncludeFiles Jane AllYears 3" expectedAllYears3 allYears3- let expectedGroup3 = [- ("./import/jane/2017-include.journal",- ["./import/jane/bogartbank/2017-include.journal"]),- ("./import/jane/2018-include.journal",- ["./import/jane/bogartbank/2018-include.journal"]),- ("./import/jane/2019-include.journal",- ["./import/jane/bogartbank/2019-include.journal"])]- assertEqual "groupIncludeFiles Jane 3" expectedGroup3 group3+ let (group3, allYears3) = groupIncludeFiles (Map.keys group2)+ let expectedAllYears3 =+ [ ( "./import/jane/all-years.journal",+ [ "./import/jane/2017-include.journal",+ "./import/jane/2018-include.journal",+ "./import/jane/2019-include.journal"+ ]+ )+ ]+ assertEqual "groupIncludeFiles Jane AllYears 3" expectedAllYears3 allYears3+ let expectedGroup3 =+ [ ( "./import/jane/2017-include.journal",+ ["./import/jane/bogartbank/2017-include.journal"]+ ),+ ( "./import/jane/2018-include.journal",+ ["./import/jane/bogartbank/2018-include.journal"]+ ),+ ( "./import/jane/2019-include.journal",+ ["./import/jane/bogartbank/2019-include.journal"]+ )+ ]+ assertEqual "groupIncludeFiles Jane 3" expectedGroup3 group3 - let (group4, allYears4) = groupIncludeFiles (Map.keys group3)- let expectedAllYears4 = [- ("./import/all-years.journal",- ["./import/2017-include.journal",- "./import/2018-include.journal",- "./import/2019-include.journal"])]- assertEqual "groupIncludeFiles Jane AllYears 4" expectedAllYears4 allYears4- let expectedGroup4 = [- ("./import/2017-include.journal",- ["./import/jane/2017-include.journal"]),- ("./import/2018-include.journal",- ["./import/jane/2018-include.journal"]),- ("./import/2019-include.journal",- ["./import/jane/2019-include.journal"])]- assertEqual "groupIncludeFiles Jane 4" expectedGroup4 group4+ let (group4, allYears4) = groupIncludeFiles (Map.keys group3)+ let expectedAllYears4 =+ [ ( "./import/all-years.journal",+ [ "./import/2017-include.journal",+ "./import/2018-include.journal",+ "./import/2019-include.journal"+ ]+ )+ ]+ assertEqual "groupIncludeFiles Jane AllYears 4" expectedAllYears4 allYears4+ let expectedGroup4 =+ [ ( "./import/2017-include.journal",+ ["./import/jane/2017-include.journal"]+ ),+ ( "./import/2018-include.journal",+ ["./import/jane/2018-include.journal"]+ ),+ ( "./import/2019-include.journal",+ ["./import/jane/2019-include.journal"]+ )+ ]+ assertEqual "groupIncludeFiles Jane 4" expectedGroup4 group4 - let (group5, allYears5) = groupIncludeFiles (Map.keys group4)- let expectedAllYears5 = [("./all-years.journal", ["./2017-include.journal", "./2018-include.journal", "./2019-include.journal"])]- assertEqual "groupIncludeFiles Jane AllYears 5" expectedAllYears5 allYears5- let expectedGroup5 = [- ("./2017-include.journal",- ["./import/2017-include.journal"]),- ("./2018-include.journal",- ["./import/2018-include.journal"]),- ("./2019-include.journal",- ["./import/2019-include.journal"])]- assertEqual "groupIncludeFiles Jane 5" expectedGroup5 group5- )+ let (group5, allYears5) = groupIncludeFiles (Map.keys group4)+ let expectedAllYears5 = [("./all-years.journal", ["./2017-include.journal", "./2018-include.journal", "./2019-include.journal"])]+ assertEqual "groupIncludeFiles Jane AllYears 5" expectedAllYears5 allYears5+ let expectedGroup5 =+ [ ( "./2017-include.journal",+ ["./import/2017-include.journal"]+ ),+ ( "./2018-include.journal",+ ["./import/2018-include.journal"]+ ),+ ( "./2019-include.journal",+ ["./import/2019-include.journal"]+ )+ ]+ assertEqual "groupIncludeFiles Jane 5" expectedGroup5 group5+ ) testGroupIncludeFiles :: Test-testGroupIncludeFiles = TestCase (- do- let (group1, allYears1) = groupIncludeFiles journalFiles- let expectedAllYears1 = [- ("./import/jane/bogartbank/checking/all-years.journal",- ["./import/jane/bogartbank/checking/2018-include.journal",- "./import/jane/bogartbank/checking/2019-include.journal"]),- ("./import/jane/bogartbank/savings/all-years.journal",- ["./import/jane/bogartbank/savings/2017-include.journal",- "./import/jane/bogartbank/savings/2018-include.journal"]),- ("./import/jane/otherbank/creditcard/all-years.journal",- ["./import/jane/otherbank/creditcard/2017-include.journal",- "./import/jane/otherbank/creditcard/2018-include.journal"]),- ("./import/jane/otherbank/investments/all-years.journal",- ["./import/jane/otherbank/investments/2018-include.journal",- "./import/jane/otherbank/investments/2019-include.journal"]),- ("./import/john/bogartbank/checking/all-years.journal",- ["./import/john/bogartbank/checking/2018-include.journal",- "./import/john/bogartbank/checking/2019-include.journal"]),- ("./import/john/bogartbank/savings/all-years.journal",- ["./import/john/bogartbank/savings/2017-include.journal",- "./import/john/bogartbank/savings/2018-include.journal"]),- ("./import/john/otherbank/creditcard/all-years.journal",- ["./import/john/otherbank/creditcard/2017-include.journal",- "./import/john/otherbank/creditcard/2018-include.journal"]),- ("./import/john/otherbank/investments/all-years.journal",- ["./import/john/otherbank/investments/2018-include.journal",- "./import/john/otherbank/investments/2019-include.journal"])]- assertEqual "groupIncludeFiles Jane AllYears 1" expectedAllYears1 allYears1- assertEqual "groupIncludeFiles 1" groupedIncludeFiles group1-- let (group2, allYears2) = groupIncludeFiles (Map.keys group1)- let expectedAllYears2 = [- ("./import/jane/bogartbank/all-years.journal",- ["./import/jane/bogartbank/2017-include.journal",- "./import/jane/bogartbank/2018-include.journal",- "./import/jane/bogartbank/2019-include.journal"]),- ("./import/jane/otherbank/all-years.journal",- ["./import/jane/otherbank/2017-include.journal",- "./import/jane/otherbank/2018-include.journal",- "./import/jane/otherbank/2019-include.journal"]),- ("./import/john/bogartbank/all-years.journal",- ["./import/john/bogartbank/2017-include.journal",- "./import/john/bogartbank/2018-include.journal",- "./import/john/bogartbank/2019-include.journal"]),- ("./import/john/otherbank/all-years.journal",- ["./import/john/otherbank/2017-include.journal",- "./import/john/otherbank/2018-include.journal",- "./import/john/otherbank/2019-include.journal"])]-- assertEqual "groupIncludeFiles Jane AllYears 2" expectedAllYears2 allYears2- let expectedGroup2 = [("./import/jane/bogartbank/2017-include.journal",- ["./import/jane/bogartbank/savings/2017-include.journal"]),- ("./import/jane/bogartbank/2018-include.journal",- ["./import/jane/bogartbank/checking/2018-include.journal",- "./import/jane/bogartbank/savings/2018-include.journal"]),- ("./import/jane/bogartbank/2019-include.journal",- ["./import/jane/bogartbank/checking/2019-include.journal"]),-- ("./import/jane/otherbank/2017-include.journal",- ["./import/jane/otherbank/creditcard/2017-include.journal"]),- ("./import/jane/otherbank/2018-include.journal",- ["./import/jane/otherbank/creditcard/2018-include.journal",- "./import/jane/otherbank/investments/2018-include.journal"]),- ("./import/jane/otherbank/2019-include.journal",- ["./import/jane/otherbank/investments/2019-include.journal"]),+testGroupIncludeFiles =+ TestCase+ ( do+ let (group1, allYears1) = groupIncludeFiles journalFiles+ let expectedAllYears1 =+ [ ( "./import/jane/bogartbank/checking/all-years.journal",+ [ "./import/jane/bogartbank/checking/2018-include.journal",+ "./import/jane/bogartbank/checking/2019-include.journal"+ ]+ ),+ ( "./import/jane/bogartbank/savings/all-years.journal",+ [ "./import/jane/bogartbank/savings/2017-include.journal",+ "./import/jane/bogartbank/savings/2018-include.journal"+ ]+ ),+ ( "./import/jane/otherbank/creditcard/all-years.journal",+ [ "./import/jane/otherbank/creditcard/2017-include.journal",+ "./import/jane/otherbank/creditcard/2018-include.journal"+ ]+ ),+ ( "./import/jane/otherbank/investments/all-years.journal",+ [ "./import/jane/otherbank/investments/2018-include.journal",+ "./import/jane/otherbank/investments/2019-include.journal"+ ]+ ),+ ( "./import/john/bogartbank/checking/all-years.journal",+ [ "./import/john/bogartbank/checking/2018-include.journal",+ "./import/john/bogartbank/checking/2019-include.journal"+ ]+ ),+ ( "./import/john/bogartbank/savings/all-years.journal",+ [ "./import/john/bogartbank/savings/2017-include.journal",+ "./import/john/bogartbank/savings/2018-include.journal"+ ]+ ),+ ( "./import/john/otherbank/creditcard/all-years.journal",+ [ "./import/john/otherbank/creditcard/2017-include.journal",+ "./import/john/otherbank/creditcard/2018-include.journal"+ ]+ ),+ ( "./import/john/otherbank/investments/all-years.journal",+ [ "./import/john/otherbank/investments/2018-include.journal",+ "./import/john/otherbank/investments/2019-include.journal"+ ]+ )+ ]+ assertEqual "groupIncludeFiles Jane AllYears 1" expectedAllYears1 allYears1+ assertEqual "groupIncludeFiles 1" groupedIncludeFiles group1 - ("./import/john/bogartbank/2017-include.journal",- ["./import/john/bogartbank/savings/2017-include.journal"]),- ("./import/john/bogartbank/2018-include.journal",- ["./import/john/bogartbank/checking/2018-include.journal",- "./import/john/bogartbank/savings/2018-include.journal"]),- ("./import/john/bogartbank/2019-include.journal",- ["./import/john/bogartbank/checking/2019-include.journal"]),+ let (group2, allYears2) = groupIncludeFiles (Map.keys group1)+ let expectedAllYears2 =+ [ ( "./import/jane/bogartbank/all-years.journal",+ [ "./import/jane/bogartbank/2017-include.journal",+ "./import/jane/bogartbank/2018-include.journal",+ "./import/jane/bogartbank/2019-include.journal"+ ]+ ),+ ( "./import/jane/otherbank/all-years.journal",+ [ "./import/jane/otherbank/2017-include.journal",+ "./import/jane/otherbank/2018-include.journal",+ "./import/jane/otherbank/2019-include.journal"+ ]+ ),+ ( "./import/john/bogartbank/all-years.journal",+ [ "./import/john/bogartbank/2017-include.journal",+ "./import/john/bogartbank/2018-include.journal",+ "./import/john/bogartbank/2019-include.journal"+ ]+ ),+ ( "./import/john/otherbank/all-years.journal",+ [ "./import/john/otherbank/2017-include.journal",+ "./import/john/otherbank/2018-include.journal",+ "./import/john/otherbank/2019-include.journal"+ ]+ )+ ] - ("./import/john/otherbank/2017-include.journal",- ["./import/john/otherbank/creditcard/2017-include.journal"]),- ("./import/john/otherbank/2018-include.journal",- ["./import/john/otherbank/creditcard/2018-include.journal",- "./import/john/otherbank/investments/2018-include.journal"]),- ("./import/john/otherbank/2019-include.journal",- ["./import/john/otherbank/investments/2019-include.journal"])]- assertEqual "groupIncludeFiles 2 - diff 1" [] (expectedGroup2 Map.\\ group2)- assertEqual "groupIncludeFiles 2 - diff 2" [] (group2 Map.\\ expectedGroup2)- assertEqual "groupIncludeFiles 2" expectedGroup2 group2+ assertEqual "groupIncludeFiles Jane AllYears 2" expectedAllYears2 allYears2+ let expectedGroup2 =+ [ ( "./import/jane/bogartbank/2017-include.journal",+ ["./import/jane/bogartbank/savings/2017-include.journal"]+ ),+ ( "./import/jane/bogartbank/2018-include.journal",+ [ "./import/jane/bogartbank/checking/2018-include.journal",+ "./import/jane/bogartbank/savings/2018-include.journal"+ ]+ ),+ ( "./import/jane/bogartbank/2019-include.journal",+ ["./import/jane/bogartbank/checking/2019-include.journal"]+ ),+ ( "./import/jane/otherbank/2017-include.journal",+ ["./import/jane/otherbank/creditcard/2017-include.journal"]+ ),+ ( "./import/jane/otherbank/2018-include.journal",+ [ "./import/jane/otherbank/creditcard/2018-include.journal",+ "./import/jane/otherbank/investments/2018-include.journal"+ ]+ ),+ ( "./import/jane/otherbank/2019-include.journal",+ ["./import/jane/otherbank/investments/2019-include.journal"]+ ),+ ( "./import/john/bogartbank/2017-include.journal",+ ["./import/john/bogartbank/savings/2017-include.journal"]+ ),+ ( "./import/john/bogartbank/2018-include.journal",+ [ "./import/john/bogartbank/checking/2018-include.journal",+ "./import/john/bogartbank/savings/2018-include.journal"+ ]+ ),+ ( "./import/john/bogartbank/2019-include.journal",+ ["./import/john/bogartbank/checking/2019-include.journal"]+ ),+ ( "./import/john/otherbank/2017-include.journal",+ ["./import/john/otherbank/creditcard/2017-include.journal"]+ ),+ ( "./import/john/otherbank/2018-include.journal",+ [ "./import/john/otherbank/creditcard/2018-include.journal",+ "./import/john/otherbank/investments/2018-include.journal"+ ]+ ),+ ( "./import/john/otherbank/2019-include.journal",+ ["./import/john/otherbank/investments/2019-include.journal"]+ )+ ]+ assertEqual "groupIncludeFiles 2 - diff 1" [] (expectedGroup2 Map.\\ group2)+ assertEqual "groupIncludeFiles 2 - diff 2" [] (group2 Map.\\ expectedGroup2)+ assertEqual "groupIncludeFiles 2" expectedGroup2 group2 - let (group3, allYears3) = groupIncludeFiles (Map.keys group2)- let expectedAllYears3 = [- ("./import/jane/all-years.journal",- ["./import/jane/2017-include.journal",- "./import/jane/2018-include.journal",- "./import/jane/2019-include.journal"]),- ("./import/john/all-years.journal",- ["./import/john/2017-include.journal",- "./import/john/2018-include.journal",- "./import/john/2019-include.journal"])]+ let (group3, allYears3) = groupIncludeFiles (Map.keys group2)+ let expectedAllYears3 =+ [ ( "./import/jane/all-years.journal",+ [ "./import/jane/2017-include.journal",+ "./import/jane/2018-include.journal",+ "./import/jane/2019-include.journal"+ ]+ ),+ ( "./import/john/all-years.journal",+ [ "./import/john/2017-include.journal",+ "./import/john/2018-include.journal",+ "./import/john/2019-include.journal"+ ]+ )+ ] - assertEqual "groupIncludeFiles Jane AllYears 3" expectedAllYears3 allYears3- let expectedGroup3 = [("./import/jane/2017-include.journal",- ["./import/jane/bogartbank/2017-include.journal",- "./import/jane/otherbank/2017-include.journal"]),- ("./import/jane/2018-include.journal",- ["./import/jane/bogartbank/2018-include.journal",- "./import/jane/otherbank/2018-include.journal"]),- ("./import/jane/2019-include.journal",- ["./import/jane/bogartbank/2019-include.journal",- "./import/jane/otherbank/2019-include.journal"]),- ("./import/john/2017-include.journal",- ["./import/john/bogartbank/2017-include.journal",- "./import/john/otherbank/2017-include.journal"]),- ("./import/john/2018-include.journal",- ["./import/john/bogartbank/2018-include.journal",- "./import/john/otherbank/2018-include.journal"]),- ("./import/john/2019-include.journal",- ["./import/john/bogartbank/2019-include.journal",- "./import/john/otherbank/2019-include.journal"])]- assertEqual "groupIncludeFiles 3 - diff 1" [] (expectedGroup3 Map.\\ group3)- assertEqual "groupIncludeFiles 3 - diff 2" [] (group3 Map.\\ expectedGroup3)- assertEqual "groupIncludeFiles 3" expectedGroup3 group3+ assertEqual "groupIncludeFiles Jane AllYears 3" expectedAllYears3 allYears3+ let expectedGroup3 =+ [ ( "./import/jane/2017-include.journal",+ [ "./import/jane/bogartbank/2017-include.journal",+ "./import/jane/otherbank/2017-include.journal"+ ]+ ),+ ( "./import/jane/2018-include.journal",+ [ "./import/jane/bogartbank/2018-include.journal",+ "./import/jane/otherbank/2018-include.journal"+ ]+ ),+ ( "./import/jane/2019-include.journal",+ [ "./import/jane/bogartbank/2019-include.journal",+ "./import/jane/otherbank/2019-include.journal"+ ]+ ),+ ( "./import/john/2017-include.journal",+ [ "./import/john/bogartbank/2017-include.journal",+ "./import/john/otherbank/2017-include.journal"+ ]+ ),+ ( "./import/john/2018-include.journal",+ [ "./import/john/bogartbank/2018-include.journal",+ "./import/john/otherbank/2018-include.journal"+ ]+ ),+ ( "./import/john/2019-include.journal",+ [ "./import/john/bogartbank/2019-include.journal",+ "./import/john/otherbank/2019-include.journal"+ ]+ )+ ]+ assertEqual "groupIncludeFiles 3 - diff 1" [] (expectedGroup3 Map.\\ group3)+ assertEqual "groupIncludeFiles 3 - diff 2" [] (group3 Map.\\ expectedGroup3)+ assertEqual "groupIncludeFiles 3" expectedGroup3 group3 - let (group4, allYears4) = groupIncludeFiles (Map.keys group3)- let expectedAllYears4 = [- ("./import/all-years.journal",- ["./import/2017-include.journal",- "./import/2018-include.journal",- "./import/2019-include.journal"])]- assertEqual "groupIncludeFiles Jane AllYears 4" expectedAllYears4 allYears4- let expectedGroup4 = [("./import/2017-include.journal",- ["./import/jane/2017-include.journal",- "./import/john/2017-include.journal"]),- ("./import/2018-include.journal",- ["./import/jane/2018-include.journal",- "./import/john/2018-include.journal"]),- ("./import/2019-include.journal",- ["./import/jane/2019-include.journal",- "./import/john/2019-include.journal"])]- assertEqual "groupIncludeFiles 4 - diff 1" [] (expectedGroup4 Map.\\ group4)- assertEqual "groupIncludeFiles 4 - diff 2" [] (group4 Map.\\ expectedGroup4)- assertEqual "groupIncludeFiles 4" expectedGroup4 group4+ let (group4, allYears4) = groupIncludeFiles (Map.keys group3)+ let expectedAllYears4 =+ [ ( "./import/all-years.journal",+ [ "./import/2017-include.journal",+ "./import/2018-include.journal",+ "./import/2019-include.journal"+ ]+ )+ ]+ assertEqual "groupIncludeFiles Jane AllYears 4" expectedAllYears4 allYears4+ let expectedGroup4 =+ [ ( "./import/2017-include.journal",+ [ "./import/jane/2017-include.journal",+ "./import/john/2017-include.journal"+ ]+ ),+ ( "./import/2018-include.journal",+ [ "./import/jane/2018-include.journal",+ "./import/john/2018-include.journal"+ ]+ ),+ ( "./import/2019-include.journal",+ [ "./import/jane/2019-include.journal",+ "./import/john/2019-include.journal"+ ]+ )+ ]+ assertEqual "groupIncludeFiles 4 - diff 1" [] (expectedGroup4 Map.\\ group4)+ assertEqual "groupIncludeFiles 4 - diff 2" [] (group4 Map.\\ expectedGroup4)+ assertEqual "groupIncludeFiles 4" expectedGroup4 group4 - let (group5, allYears5) = groupIncludeFiles (Map.keys group4)- let expectedAllYears5 = [("./all-years.journal", ["./2017-include.journal", "./2018-include.journal", "./2019-include.journal"])]- assertEqual "groupIncludeFiles Jane AllYears 5" expectedAllYears5 allYears5- let expectedGroup5 = [- ("./2017-include.journal",- ["./import/2017-include.journal"]),- ("./2018-include.journal",- ["./import/2018-include.journal"]),- ("./2019-include.journal",- ["./import/2019-include.journal"])]- assertEqual "groupIncludeFiles 5 - diff 1" [] (expectedGroup5 Map.\\ group5)- assertEqual "groupIncludeFiles 5 - diff 2" [] (group5 Map.\\ expectedGroup5)- assertEqual "groupIncludeFiles 5" expectedGroup5 group5- )+ let (group5, allYears5) = groupIncludeFiles (Map.keys group4)+ let expectedAllYears5 = [("./all-years.journal", ["./2017-include.journal", "./2018-include.journal", "./2019-include.journal"])]+ assertEqual "groupIncludeFiles Jane AllYears 5" expectedAllYears5 allYears5+ let expectedGroup5 =+ [ ( "./2017-include.journal",+ ["./import/2017-include.journal"]+ ),+ ( "./2018-include.journal",+ ["./import/2018-include.journal"]+ ),+ ( "./2019-include.journal",+ ["./import/2019-include.journal"]+ )+ ]+ assertEqual "groupIncludeFiles 5 - diff 1" [] (expectedGroup5 Map.\\ group5)+ assertEqual "groupIncludeFiles 5 - diff 2" [] (group5 Map.\\ expectedGroup5)+ assertEqual "groupIncludeFiles 5" expectedGroup5 group5+ ) testIncludeYears :: Test-testIncludeYears = TestCase (- do- let txterr = "Some text without years"- let expectederr = ["Unable to extract years from the following text:", txterr, "Errors:"]- let actualerr = (init . head) $ map (T.lines) $ lefts [includeYears' txterr] :: [Text]- assertEqual "Get a list of years from an include file - error case" expectederr actualerr+testIncludeYears =+ TestCase+ ( do+ let txterr = "Some text without years"+ let expectederr = ["Unable to extract years from the following text:", txterr, "Errors:"]+ let actualerr = (init . head) $ map T.lines $ lefts [includeYears' txterr] :: [Text]+ assertEqual "Get a list of years from an include file - error case" expectederr actualerr - let txt1 = "### Generated by hledger-flow - DO NOT EDIT ###\n\n" <>- "!include import/2014-include.journal\n" <>- "!include import/2015-include.journal\n" <>- "!include import/2016-include.journal\n" <>- "!include import/2017-include.journal\n" <>- "!include import/2018-include.journal\n" <>- "!include import/2019-include.journal"+ let txt1 =+ "### Generated by hledger-flow - DO NOT EDIT ###\n\n"+ <> "include import/2014-include.journal\n"+ <> "include import/2015-include.journal\n"+ <> "include import/2016-include.journal\n"+ <> "include import/2017-include.journal\n"+ <> "include import/2018-include.journal\n"+ <> "include import/2019-include.journal" - let expected1 = Right [2014..2019]- let actual1 = includeYears' txt1- assertEqual "Get a list of years from an include file - success case 1" expected1 actual1+ let expected1 = Right [2014 .. 2019]+ let actual1 = includeYears' txt1+ assertEqual "Get a list of years from an include file - success case 1" expected1 actual1 - let txt2 = "!include 2019-include.journal"+ let txt2 = "include 2019-include.journal" - let expected2 = Right [2019]- let actual2 = includeYears' txt2- assertEqual "Get a list of years from an include file - success case 2" expected2 actual2- )+ let expected2 = Right [2019]+ let actual2 = includeYears' txt2+ assertEqual "Get a list of years from an include file - success case 2" expected2 actual2+ ) testToIncludeLine :: Test-testToIncludeLine = TestCase (- do- let expected = "!include file1.journal"- let relativeWithTrailingSlash = toIncludeLine "./base/dir/" "./base/dir/file1.journal"- assertEqual "Include line - relative base dir with trailing slash" expected relativeWithTrailingSlash+testToIncludeLine =+ TestCase+ ( do+ let expected = "include file1.journal"+ let relativeWithTrailingSlash = toIncludeLine "./base/dir/" "./base/dir/file1.journal"+ assertEqual "Include line - relative base dir with trailing slash" expected relativeWithTrailingSlash - let relativeNoTrailingSlash = toIncludeLine "./base/dir" "./base/dir/file1.journal"- assertEqual "Include line - relative base dir without a trailing slash" expected relativeNoTrailingSlash+ let relativeNoTrailingSlash = toIncludeLine "./base/dir" "./base/dir/file1.journal"+ assertEqual "Include line - relative base dir without a trailing slash" expected relativeNoTrailingSlash - let absoluteWithTrailingSlash = toIncludeLine "/base/dir/" "/base/dir/file1.journal"- assertEqual "Include line - absolute base dir with trailing slash" expected absoluteWithTrailingSlash+ let absoluteWithTrailingSlash = toIncludeLine "/base/dir/" "/base/dir/file1.journal"+ assertEqual "Include line - absolute base dir with trailing slash" expected absoluteWithTrailingSlash - let absoluteNoTrailingSlash = toIncludeLine "/base/dir" "/base/dir/file1.journal"- assertEqual "Include line - absolute base dir without a trailing slash" expected absoluteNoTrailingSlash- )+ let absoluteNoTrailingSlash = toIncludeLine "/base/dir" "/base/dir/file1.journal"+ assertEqual "Include line - absolute base dir without a trailing slash" expected absoluteNoTrailingSlash+ ) testToIncludeFiles :: Test-testToIncludeFiles = TestCase (- do- let expected = [- ("./import/john/bogartbank/checking/2018-include.journal",- "### Generated by hledger-flow - DO NOT EDIT ###\n\n" <>- "!include import/john/bogartbank/checking/3-journal/2018/2018-10-30.journal\n" <>- "!include import/john/bogartbank/checking/3-journal/2018/2018-11-30.journal\n" <>- "!include import/john/bogartbank/checking/3-journal/2018/2018-12-30.journal\n"),- ("./import/john/bogartbank/checking/2019-include.journal",- "### Generated by hledger-flow - DO NOT EDIT ###\n\n" <>- "!include import/john/bogartbank/checking/3-journal/2019/2019-01-30.journal\n" <>- "!include import/john/bogartbank/checking/3-journal/2019/2019-02-30.journal\n"),- ("./import/john/bogartbank/savings/2017-include.journal",- "### Generated by hledger-flow - DO NOT EDIT ###\n\n" <>- "!include import/john/bogartbank/savings/3-journal/2017/2017-11-30.journal\n" <>- "!include import/john/bogartbank/savings/3-journal/2017/2017-12-30.journal\n"),- ("./import/john/bogartbank/savings/2018-include.journal",- "### Generated by hledger-flow - DO NOT EDIT ###\n\n" <>- "!include import/john/bogartbank/savings/3-journal/2018/2018-01-30.journal\n" <>- "!include import/john/bogartbank/savings/3-journal/2018/2018-02-30.journal\n")]+testToIncludeFiles =+ TestCase+ ( do+ let expected =+ [ ( "./import/john/bogartbank/checking/2018-include.journal",+ "### Generated by hledger-flow - DO NOT EDIT ###\n\n"+ <> "include import/john/bogartbank/checking/3-journal/2018/2018-10-30.journal\n"+ <> "include import/john/bogartbank/checking/3-journal/2018/2018-11-30.journal\n"+ <> "include import/john/bogartbank/checking/3-journal/2018/2018-12-30.journal\n"+ ),+ ( "./import/john/bogartbank/checking/2019-include.journal",+ "### Generated by hledger-flow - DO NOT EDIT ###\n\n"+ <> "include import/john/bogartbank/checking/3-journal/2019/2019-01-30.journal\n"+ <> "include import/john/bogartbank/checking/3-journal/2019/2019-02-30.journal\n"+ ),+ ( "./import/john/bogartbank/savings/2017-include.journal",+ "### Generated by hledger-flow - DO NOT EDIT ###\n\n"+ <> "include import/john/bogartbank/savings/3-journal/2017/2017-11-30.journal\n"+ <> "include import/john/bogartbank/savings/3-journal/2017/2017-12-30.journal\n"+ ),+ ( "./import/john/bogartbank/savings/2018-include.journal",+ "### Generated by hledger-flow - DO NOT EDIT ###\n\n"+ <> "include import/john/bogartbank/savings/3-journal/2018/2018-01-30.journal\n"+ <> "include import/john/bogartbank/savings/3-journal/2018/2018-02-30.journal\n"+ )+ ] - ch <- newTChanIO- txt <- toIncludeFiles (defaultOpts [absdir|/|]) ch groupedJohnBogart- assertEqual "Convert a grouped map of paths, to a map with text contents for each file" expected txt)+ ch <- newTChanIO+ txt <- toIncludeFiles (defaultOpts [absdir|/|]) ch groupedJohnBogart+ assertEqual "Convert a grouped map of paths, to a map with text contents for each file" expected txt+ ) tests :: Test tests = TestList [testYearsIncludeMap, testYearsIncludeGrouping, testGroupIncludeFilesTinySet, testGroupIncludeFilesSmallSet, testGroupIncludeFiles, testIncludeYears, testToIncludeLine, testToIncludeFiles]
test/CSVImport/Integration.hs view
@@ -1,248 +1,297 @@-{-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE OverloadedLists #-}+{-# LANGUAGE OverloadedStrings #-} module CSVImport.Integration (tests) where -import Test.HUnit-import Turtle-import qualified Data.Map.Strict as Map+import Control.Concurrent.STM import qualified Data.List as List (sort)--import TestHelpers (defaultOpts)-import TestHelpersTurtle (journalFiles, hiddenFiles, touchAll, extraFiles)+import qualified Data.Map.Strict as Map+import qualified Data.Text.IO as T import Hledger.Flow.Common import Hledger.Flow.Import.ImportHelpersTurtle (extraIncludesForFile, groupAndWriteIncludeFiles, includePreamble, toIncludeFiles) import Hledger.Flow.PathHelpers--import Control.Concurrent.STM+import Test.HUnit+import TestHelpers (defaultOpts)+import TestHelpersTurtle (extraFiles, hiddenFiles, journalFiles, touchAll)+import Turtle+import Prelude hiding (readFile, writeFile) testExtraIncludesForFile :: Test-testExtraIncludesForFile = TestCase (- sh (- do- currentDir <- pwd- tmpdir <- using (mktempdir currentDir "hlflow")- tmpdirAbsPath <- fromTurtleAbsDir tmpdir+testExtraIncludesForFile =+ TestCase+ ( sh+ ( do+ currentDir <- pwd+ tmpdir <- using (mktempdir currentDir "hlflow")+ tmpdirAbsPath <- fromTurtleAbsDir tmpdir - let importedJournals = map (tmpdir </>) journalFiles :: [TurtlePath]- let accountDir = "import/john/bogartbank/savings"- let opening = tmpdir </> accountDir </> "2017-opening.journal"- let closing = tmpdir </> accountDir </> "2017-closing.journal"- let hidden = map (tmpdir </>) hiddenFiles :: [TurtlePath]- touchAll $ importedJournals ++ hidden+ let importedJournals = map (tmpdir </>) journalFiles :: [TurtlePath]+ let accountDir = "import/john/bogartbank/savings"+ let opening = tmpdir </> accountDir </> "2017-opening.journal"+ let closing = tmpdir </> accountDir </> "2017-closing.journal"+ let hidden = map (tmpdir </>) hiddenFiles :: [TurtlePath]+ touchAll $ importedJournals ++ hidden - let accountInclude = tmpdir </> accountDir </> "2017-include.journal"- let expectedEmpty = [(accountInclude, [])]+ let accountInclude = tmpdir </> accountDir </> "2017-include.journal"+ let expectedEmpty = [(accountInclude, [])] - ch <- liftIO newTChanIO+ ch <- liftIO newTChanIO - extraOpening1 <- liftIO $ extraIncludesForFile (defaultOpts tmpdirAbsPath) ch accountInclude ["opening.journal"] [] []- liftIO $ assertEqual "The opening journal should not be included when it is not on disk" expectedEmpty extraOpening1+ extraOpening1 <- liftIO $ extraIncludesForFile (defaultOpts tmpdirAbsPath) ch accountInclude ["opening.journal"] [] []+ liftIO $ assertEqual "The opening journal should not be included when it is not on disk" expectedEmpty extraOpening1 - extraClosing1 <- liftIO $ extraIncludesForFile (defaultOpts tmpdirAbsPath) ch accountInclude ["closing.journal"] [] []- liftIO $ assertEqual "The closing journal should not be included when it is not on disk" expectedEmpty extraClosing1+ extraClosing1 <- liftIO $ extraIncludesForFile (defaultOpts tmpdirAbsPath) ch accountInclude ["closing.journal"] [] []+ liftIO $ assertEqual "The closing journal should not be included when it is not on disk" expectedEmpty extraClosing1 - touchAll [opening, closing]+ touchAll [opening, closing] - extraOpening2 <- liftIO $ extraIncludesForFile (defaultOpts tmpdirAbsPath) ch accountInclude ["opening.journal"] [] []- liftIO $ assertEqual "The opening journal should be included when it is on disk" [(accountInclude, [opening])] extraOpening2+ extraOpening2 <- liftIO $ extraIncludesForFile (defaultOpts tmpdirAbsPath) ch accountInclude ["opening.journal"] [] []+ liftIO $ assertEqual "The opening journal should be included when it is on disk" [(accountInclude, [opening])] extraOpening2 - extraClosing2 <- liftIO $ extraIncludesForFile (defaultOpts tmpdirAbsPath) ch accountInclude ["closing.journal"] [] []- liftIO $ assertEqual "The closing journal should be included when it is on disk" [(accountInclude, [closing])] extraClosing2- ))+ extraClosing2 <- liftIO $ extraIncludesForFile (defaultOpts tmpdirAbsPath) ch accountInclude ["closing.journal"] [] []+ liftIO $ assertEqual "The closing journal should be included when it is on disk" [(accountInclude, [closing])] extraClosing2+ )+ ) testExtraIncludesPrices :: Test-testExtraIncludesPrices = TestCase (- sh (- do- currentDir <- pwd- tmpdir <- using (mktempdir currentDir "hlflow")- tmpdirAbsPath <- fromTurtleAbsDir tmpdir+testExtraIncludesPrices =+ TestCase+ ( sh+ ( do+ currentDir <- pwd+ tmpdir <- using (mktempdir currentDir "hlflow")+ tmpdirAbsPath <- fromTurtleAbsDir tmpdir - let importedJournals = map (tmpdir </>) journalFiles :: [TurtlePath]- touchAll $ importedJournals+ let importedJournals = map (tmpdir </>) journalFiles :: [TurtlePath]+ touchAll $ importedJournals - let priceFile = "prices" </> "2020" </> "prices.journal"+ let priceFile = "prices" </> "2020" </> "prices.journal" - let includeFile = tmpdir </> "import" </> "2020-include.journal"- let expectedEmpty = [(includeFile, [])]+ let includeFile = tmpdir </> "import" </> "2020-include.journal"+ let expectedEmpty = [(includeFile, [])] - ch <- liftIO newTChanIO+ ch <- liftIO newTChanIO - price1 <- liftIO $ extraIncludesForFile (defaultOpts tmpdirAbsPath) ch includeFile [] [] ["prices.journal"]- liftIO $ assertEqual "The price file should not be included when it is not on disk" expectedEmpty price1+ price1 <- liftIO $ extraIncludesForFile (defaultOpts tmpdirAbsPath) ch includeFile [] [] ["prices.journal"]+ liftIO $ assertEqual "The price file should not be included when it is not on disk" expectedEmpty price1 - touchAll [tmpdir </> priceFile]- let expectedPricePath = tmpdir </> "import" </> ".." </> priceFile+ touchAll [tmpdir </> priceFile]+ let expectedPricePath = tmpdir </> "import" </> ".." </> priceFile - price2 <- liftIO $ extraIncludesForFile (defaultOpts tmpdirAbsPath) ch includeFile [] [] ["prices.journal"]- liftIO $ assertEqual "The price file should be included when it is on disk" [(includeFile, [expectedPricePath])] price2- ))+ price2 <- liftIO $ extraIncludesForFile (defaultOpts tmpdirAbsPath) ch includeFile [] [] ["prices.journal"]+ liftIO $ assertEqual "The price file should be included when it is on disk" [(includeFile, [expectedPricePath])] price2+ )+ ) testIncludesPrePost :: Test-testIncludesPrePost = TestCase (- sh (- do- currentDir <- pwd- tmpdir <- using (mktempdir currentDir "hlflow")- tmpdirAbsPath <- fromTurtleAbsDir tmpdir+testIncludesPrePost =+ TestCase+ ( sh+ ( do+ currentDir <- pwd+ tmpdir <- using (mktempdir currentDir "hlflow")+ tmpdirAbsPath <- fromTurtleAbsDir tmpdir - let ownerDir = tmpdir </> "import" </> "john"- let includeFile = ownerDir </> "2019-include.journal"- let pre = ownerDir </> "_manual_" </> "2019" </> "pre-import.journal"- let post = ownerDir </> "_manual_" </> "2019" </> "post-import.journal"- touchAll [pre, post]+ let ownerDir = tmpdir </> "import" </> "john"+ let includeFile = ownerDir </> "2019-include.journal"+ let pre = ownerDir </> "_manual_" </> "2019" </> "pre-import.journal"+ let post = ownerDir </> "_manual_" </> "2019" </> "post-import.journal"+ touchAll [pre, post] - let includeMap = Map.singleton includeFile [ownerDir </> "bank1" </> "2019-include.journal",- ownerDir </> "bank2" </> "2019-include.journal"]+ let includeMap =+ Map.singleton+ includeFile+ [ ownerDir </> "bank1" </> "2019-include.journal",+ ownerDir </> "bank2" </> "2019-include.journal"+ ] - ch <- liftIO newTChanIO- fileMap <- liftIO $ toIncludeFiles (defaultOpts tmpdirAbsPath) ch includeMap- let expectedText = includePreamble <> "\n"- <> "!include _manual_/2019/pre-import.journal\n"- <> "!include bank1/2019-include.journal\n"- <> "!include bank2/2019-include.journal\n"- <> "!include _manual_/2019/post-import.journal\n"- let expectedMap = Map.singleton includeFile expectedText- liftIO $ assertEqual "All pre/post files on disk should be included" expectedMap fileMap- ))+ ch <- liftIO newTChanIO+ fileMap <- liftIO $ toIncludeFiles (defaultOpts tmpdirAbsPath) ch includeMap+ let expectedText =+ includePreamble+ <> "\n"+ <> "include _manual_/2019/pre-import.journal\n"+ <> "include bank1/2019-include.journal\n"+ <> "include bank2/2019-include.journal\n"+ <> "include _manual_/2019/post-import.journal\n"+ let expectedMap = Map.singleton includeFile expectedText+ liftIO $ assertEqual "All pre/post files on disk should be included" expectedMap fileMap+ )+ ) testIncludesOpeningClosing :: Test-testIncludesOpeningClosing = TestCase (- sh (- do- currentDir <- pwd- tmpdir <- using (mktempdir currentDir "hlflow")- tmpdirAbsPath <- fromTurtleAbsDir tmpdir+testIncludesOpeningClosing =+ TestCase+ ( sh+ ( do+ currentDir <- pwd+ tmpdir <- using (mktempdir currentDir "hlflow")+ tmpdirAbsPath <- fromTurtleAbsDir tmpdir - let ownerDir = tmpdir </> "import/john"- let accountDir = ownerDir </> "bank1" </> "savings"- let includeFile = accountDir </> "2019-include.journal"- let opening = accountDir </> "2019-opening.journal"- let closing = accountDir </> "2019-closing.journal"- touchAll [opening, closing]+ let ownerDir = tmpdir </> "import/john"+ let accountDir = ownerDir </> "bank1" </> "savings"+ let includeFile = accountDir </> "2019-include.journal"+ let opening = accountDir </> "2019-opening.journal"+ let closing = accountDir </> "2019-closing.journal"+ touchAll [opening, closing] - let includeMap = Map.singleton includeFile [accountDir </> "3-journal" </> "2019" </> "2019-01-30.journal"]+ let includeMap = Map.singleton includeFile [accountDir </> "3-journal" </> "2019" </> "2019-01-30.journal"] - ch <- liftIO newTChanIO- fileMap <- liftIO $ toIncludeFiles (defaultOpts tmpdirAbsPath) ch includeMap- let expectedText = includePreamble <> "\n"- <> "!include 2019-opening.journal\n"- <> "!include 3-journal/2019/2019-01-30.journal\n"- <> "!include 2019-closing.journal\n"- let expectedMap = Map.singleton includeFile expectedText- liftIO $ assertEqual "All opening/closing files on disk should be included" expectedMap fileMap- ))+ ch <- liftIO newTChanIO+ fileMap <- liftIO $ toIncludeFiles (defaultOpts tmpdirAbsPath) ch includeMap+ let expectedText =+ includePreamble+ <> "\n"+ <> "include 2019-opening.journal\n"+ <> "include 3-journal/2019/2019-01-30.journal\n"+ <> "include 2019-closing.journal\n"+ let expectedMap = Map.singleton includeFile expectedText+ liftIO $ assertEqual "All opening/closing files on disk should be included" expectedMap fileMap+ )+ ) testIncludesPrices :: Test-testIncludesPrices = TestCase (- sh (- do- currentDir <- pwd- tmpdir <- using (mktempdir currentDir "hlflow")- tmpdirAbsPath <- fromTurtleAbsDir tmpdir+testIncludesPrices =+ TestCase+ ( sh+ ( do+ currentDir <- pwd+ tmpdir <- using (mktempdir currentDir "hlflow")+ tmpdirAbsPath <- fromTurtleAbsDir tmpdir - let importDir = tmpdir </> "import"- let includeFile = importDir </> "2020-include.journal"- let prices = tmpdir </> "prices" </> "2020" </> "prices.journal"- let pre = importDir </> "_manual_" </> "2020" </> "pre-import.journal"- let post = importDir </> "_manual_" </> "2020" </> "post-import.journal"- touchAll [prices, pre, post]+ let importDir = tmpdir </> "import"+ let includeFile = importDir </> "2020-include.journal"+ let prices = tmpdir </> "prices" </> "2020" </> "prices.journal"+ let pre = importDir </> "_manual_" </> "2020" </> "pre-import.journal"+ let post = importDir </> "_manual_" </> "2020" </> "post-import.journal"+ touchAll [prices, pre, post] - let includeMap = Map.singleton includeFile [importDir </> "john" </> "2020-include.journal"]+ let includeMap = Map.singleton includeFile [importDir </> "john" </> "2020-include.journal"] - ch <- liftIO newTChanIO- fileMap <- liftIO $ toIncludeFiles (defaultOpts tmpdirAbsPath) ch includeMap- let expectedText = includePreamble <> "\n"- <> "!include _manual_/2020/pre-import.journal\n"- <> "!include john/2020-include.journal\n"- <> "!include ../prices/2020/prices.journal\n"- <> "!include _manual_/2020/post-import.journal\n"- let expectedMap = Map.singleton includeFile expectedText- liftIO $ assertEqual "The price file should be included together with any pre/post files" expectedMap fileMap- ))+ ch <- liftIO newTChanIO+ fileMap <- liftIO $ toIncludeFiles (defaultOpts tmpdirAbsPath) ch includeMap+ let expectedText =+ includePreamble+ <> "\n"+ <> "include _manual_/2020/pre-import.journal\n"+ <> "include john/2020-include.journal\n"+ <> "include ../prices/2020/prices.journal\n"+ <> "include _manual_/2020/post-import.journal\n"+ let expectedMap = Map.singleton includeFile expectedText+ liftIO $ assertEqual "The price file should be included together with any pre/post files" expectedMap fileMap+ )+ ) testWriteIncludeFiles :: Test-testWriteIncludeFiles = TestCase (- sh (- do- currentDir <- pwd- tmpdir <- using (mktempdir currentDir "hlflow")- tmpdirAbsPath <- fromTurtleAbsDir tmpdir+testWriteIncludeFiles =+ TestCase+ ( sh+ ( do+ currentDir <- pwd+ tmpdir <- using (mktempdir currentDir "hlflow")+ tmpdirAbsPath <- fromTurtleAbsDir tmpdir - let importedJournals = map (tmpdir </>) journalFiles :: [TurtlePath]- let extras = map (tmpdir </>) extraFiles :: [TurtlePath]- let hidden = map (tmpdir </>) hiddenFiles :: [TurtlePath]- touchAll $ importedJournals ++ extras ++ hidden+ let importedJournals = map (tmpdir </>) journalFiles :: [TurtlePath]+ let extras = map (tmpdir </>) extraFiles :: [TurtlePath]+ let hidden = map (tmpdir </>) hiddenFiles :: [TurtlePath]+ touchAll $ importedJournals ++ extras ++ hidden - let jane1 = tmpdir </> "import/jane/bogartbank/checking/2018-include.journal"- let jane2 = tmpdir </> "import/jane/bogartbank/checking/2019-include.journal"- let jane3 = tmpdir </> "import/jane/bogartbank/savings/2017-include.journal"- let jane4 = tmpdir </> "import/jane/bogartbank/savings/2018-include.journal"- let jane5 = tmpdir </> "import/jane/otherbank/creditcard/2017-include.journal"- let jane6 = tmpdir </> "import/jane/otherbank/creditcard/2018-include.journal"- let jane7 = tmpdir </> "import/jane/otherbank/investments/2018-include.journal"- let jane8 = tmpdir </> "import/jane/otherbank/investments/2019-include.journal"+ let jane1 = tmpdir </> "import/jane/bogartbank/checking/2018-include.journal"+ let jane2 = tmpdir </> "import/jane/bogartbank/checking/2019-include.journal"+ let jane3 = tmpdir </> "import/jane/bogartbank/savings/2017-include.journal"+ let jane4 = tmpdir </> "import/jane/bogartbank/savings/2018-include.journal"+ let jane5 = tmpdir </> "import/jane/otherbank/creditcard/2017-include.journal"+ let jane6 = tmpdir </> "import/jane/otherbank/creditcard/2018-include.journal"+ let jane7 = tmpdir </> "import/jane/otherbank/investments/2018-include.journal"+ let jane8 = tmpdir </> "import/jane/otherbank/investments/2019-include.journal" - let john1 = tmpdir </> "import/john/bogartbank/checking/2018-include.journal"- let john2 = tmpdir </> "import/john/bogartbank/checking/2019-include.journal"- let john3 = tmpdir </> "import/john/bogartbank/savings/2017-include.journal"- let john4 = tmpdir </> "import/john/bogartbank/savings/2018-include.journal"- let john5 = tmpdir </> "import/john/otherbank/creditcard/2017-include.journal"- let john6 = tmpdir </> "import/john/otherbank/creditcard/2018-include.journal"- let john7 = tmpdir </> "import/john/otherbank/investments/2018-include.journal"- let john8 = tmpdir </> "import/john/otherbank/investments/2019-include.journal"- let expectedIncludes = [jane1, jane2, jane3, jane4, jane5, jane6, jane7, jane8,- john1, john2, john3, john4, john5, john6, john7, john8]+ let john1 = tmpdir </> "import/john/bogartbank/checking/2018-include.journal"+ let john2 = tmpdir </> "import/john/bogartbank/checking/2019-include.journal"+ let john3 = tmpdir </> "import/john/bogartbank/savings/2017-include.journal"+ let john4 = tmpdir </> "import/john/bogartbank/savings/2018-include.journal"+ let john5 = tmpdir </> "import/john/otherbank/creditcard/2017-include.journal"+ let john6 = tmpdir </> "import/john/otherbank/creditcard/2018-include.journal"+ let john7 = tmpdir </> "import/john/otherbank/investments/2018-include.journal"+ let john8 = tmpdir </> "import/john/otherbank/investments/2019-include.journal"+ let expectedIncludes =+ [ jane1,+ jane2,+ jane3,+ jane4,+ jane5,+ jane6,+ jane7,+ jane8,+ john1,+ john2,+ john3,+ john4,+ john5,+ john6,+ john7,+ john8+ ] - ch <- liftIO newTChanIO- reportedAsWritten <- liftIO $ groupAndWriteIncludeFiles (defaultOpts tmpdirAbsPath) ch importedJournals- liftIO $ assertEqual "groupAndWriteIncludeFiles should return which files it wrote" expectedIncludes reportedAsWritten+ ch <- liftIO newTChanIO+ reportedAsWritten <- liftIO $ groupAndWriteIncludeFiles (defaultOpts tmpdirAbsPath) ch importedJournals+ liftIO $ assertEqual "groupAndWriteIncludeFiles should return which files it wrote" expectedIncludes reportedAsWritten - let allYears = [tmpdir </> "import/jane/bogartbank/checking/all-years.journal",- tmpdir </> "import/jane/bogartbank/savings/all-years.journal",- tmpdir </> "import/jane/otherbank/creditcard/all-years.journal",- tmpdir </> "import/jane/otherbank/investments/all-years.journal",- tmpdir </> "import/john/bogartbank/checking/all-years.journal",- tmpdir </> "import/john/bogartbank/savings/all-years.journal",- tmpdir </> "import/john/otherbank/creditcard/all-years.journal",- tmpdir </> "import/john/otherbank/investments/all-years.journal"]- let expectedOnDisk = List.sort $ reportedAsWritten ++ extras ++ importedJournals ++ allYears- allFilesOnDisk <- single $ sort $ onlyFiles $ lstree tmpdir- liftIO $ assertEqual "The actual files on disk should match what groupAndWriteIncludeFiles reported" expectedOnDisk allFilesOnDisk+ let allYears =+ [ tmpdir </> "import/jane/bogartbank/checking/all-years.journal",+ tmpdir </> "import/jane/bogartbank/savings/all-years.journal",+ tmpdir </> "import/jane/otherbank/creditcard/all-years.journal",+ tmpdir </> "import/jane/otherbank/investments/all-years.journal",+ tmpdir </> "import/john/bogartbank/checking/all-years.journal",+ tmpdir </> "import/john/bogartbank/savings/all-years.journal",+ tmpdir </> "import/john/otherbank/creditcard/all-years.journal",+ tmpdir </> "import/john/otherbank/investments/all-years.journal"+ ]+ let expectedOnDisk = List.sort $ reportedAsWritten ++ extras ++ importedJournals ++ allYears+ allFilesOnDisk <- single $ sort $ onlyFiles $ lstree tmpdir+ liftIO $ assertEqual "The actual files on disk should match what groupAndWriteIncludeFiles reported" expectedOnDisk allFilesOnDisk - let expectedJohn1Contents = includePreamble <> "\n"- <> "!include 3-journal/2018/2018-10-30.journal\n"- <> "!include 3-journal/2018/2018-11-30.journal\n"- <> "!include 3-journal/2018/2018-12-30.journal\n"- actualJohn1Contents <- liftIO $ readTextFile john1- liftIO $ assertEqual "John1: The include file contents should be the journal files" expectedJohn1Contents actualJohn1Contents+ let expectedJohn1Contents =+ includePreamble+ <> "\n"+ <> "include 3-journal/2018/2018-10-30.journal\n"+ <> "include 3-journal/2018/2018-11-30.journal\n"+ <> "include 3-journal/2018/2018-12-30.journal\n"+ actualJohn1Contents <- liftIO $ T.readFile john1+ liftIO $ assertEqual "John1: The include file contents should be the journal files" expectedJohn1Contents actualJohn1Contents - let expectedJohn2Contents = includePreamble <> "\n"- <> "!include 3-journal/2019/2019-01-30.journal\n"- <> "!include 3-journal/2019/2019-02-30.journal\n"- actualJohn2Contents <- liftIO $ readTextFile john2- liftIO $ assertEqual "John2: The include file contents should be the journal files" expectedJohn2Contents actualJohn2Contents+ let expectedJohn2Contents =+ includePreamble+ <> "\n"+ <> "include 3-journal/2019/2019-01-30.journal\n"+ <> "include 3-journal/2019/2019-02-30.journal\n"+ actualJohn2Contents <- liftIO $ T.readFile john2+ liftIO $ assertEqual "John2: The include file contents should be the journal files" expectedJohn2Contents actualJohn2Contents - let expectedJohn3Contents = includePreamble <> "\n"- <> "!include 2017-opening.journal\n"- <> "!include 3-journal/2017/2017-11-30.journal\n"- <> "!include 3-journal/2017/2017-12-30.journal\n"- actualJohn3Contents <- liftIO $ readTextFile john3- liftIO $ assertEqual "John3: The include file contents should be the journal files" expectedJohn3Contents actualJohn3Contents+ let expectedJohn3Contents =+ includePreamble+ <> "\n"+ <> "include 2017-opening.journal\n"+ <> "include 3-journal/2017/2017-11-30.journal\n"+ <> "include 3-journal/2017/2017-12-30.journal\n"+ actualJohn3Contents <- liftIO $ T.readFile john3+ liftIO $ assertEqual "John3: The include file contents should be the journal files" expectedJohn3Contents actualJohn3Contents - let expectedJohn4Contents = includePreamble <> "\n"- <> "!include 3-journal/2018/2018-01-30.journal\n"- <> "!include 3-journal/2018/2018-02-30.journal\n"- actualJohn4Contents <- liftIO $ readTextFile john4- liftIO $ assertEqual "John4: The include file contents should be the journal files" expectedJohn4Contents actualJohn4Contents+ let expectedJohn4Contents =+ includePreamble+ <> "\n"+ <> "include 3-journal/2018/2018-01-30.journal\n"+ <> "include 3-journal/2018/2018-02-30.journal\n"+ actualJohn4Contents <- liftIO $ T.readFile john4+ liftIO $ assertEqual "John4: The include file contents should be the journal files" expectedJohn4Contents actualJohn4Contents - let expectedJane7Contents = includePreamble <> "\n"- <> "!include 3-journal/2018/2018-12-30.journal\n"- actualJane7Contents <- liftIO $ readTextFile jane7- liftIO $ assertEqual "Jane7: The include file contents should be the journal files" expectedJane7Contents actualJane7Contents- )- )+ let expectedJane7Contents =+ includePreamble+ <> "\n"+ <> "include 3-journal/2018/2018-12-30.journal\n"+ actualJane7Contents <- liftIO $ T.readFile jane7+ liftIO $ assertEqual "Jane7: The include file contents should be the journal files" expectedJane7Contents actualJane7Contents+ )+ ) tests :: Test tests = TestList [testExtraIncludesForFile, testExtraIncludesPrices, testIncludesPrePost, testIncludesOpeningClosing, testIncludesPrices, testWriteIncludeFiles]
test/CSVImport/Unit.hs view
@@ -1,9 +1,8 @@ module CSVImport.Unit where -import Test.HUnit- import qualified CSVImport.ImportHelperTests import qualified CSVImport.ImportHelperTurtleTests+import Test.HUnit tests :: Test tests = TestList [CSVImport.ImportHelperTests.tests, CSVImport.ImportHelperTurtleTests.tests]
test/Common/Integration.hs view
@@ -1,54 +1,55 @@-{-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE OverloadedLists #-}+{-# LANGUAGE OverloadedStrings #-} module Common.Integration (tests) where -import Test.HUnit-import Turtle import qualified Data.List as List (sort)--import TestHelpersTurtle-import Hledger.Flow.PathHelpers (TurtlePath) import Hledger.Flow.Common+import Hledger.Flow.PathHelpers (TurtlePath)+import Test.HUnit+import TestHelpersTurtle+import Turtle testHiddenFiles :: Test-testHiddenFiles = TestCase (- sh (- do- let tmpbase = "." </> "test" </> "tmp"- mktree tmpbase- tmpdir <- using (mktempdir tmpbase "hlflowtest")- let tmpJournals = map (tmpdir </>) journalFiles :: [TurtlePath]- let tmpExtras = map (tmpdir </>) extraFiles :: [TurtlePath]- let tmpHidden = map (tmpdir </>) hiddenFiles :: [TurtlePath]- let onDisk = List.sort $ tmpJournals ++ tmpExtras ++ tmpHidden- touchAll onDisk- filtered <- fmap List.sort $ shellToList $ onlyFiles $ select onDisk- let expected = List.sort $ tmpExtras ++ tmpJournals- liftIO $ assertEqual "Hidden files should be excluded" expected filtered- )- )+testHiddenFiles =+ TestCase+ ( sh+ ( do+ let tmpbase = "." </> "test" </> "tmp"+ mktree tmpbase+ tmpdir <- using (mktempdir tmpbase "hlflowtest")+ let tmpJournals = map (tmpdir </>) journalFiles :: [TurtlePath]+ let tmpExtras = map (tmpdir </>) extraFiles :: [TurtlePath]+ let tmpHidden = map (tmpdir </>) hiddenFiles :: [TurtlePath]+ let onDisk = List.sort $ tmpJournals ++ tmpExtras ++ tmpHidden+ touchAll onDisk+ filtered <- fmap List.sort $ shellToList $ onlyFiles $ select onDisk+ let expected = List.sort $ tmpExtras ++ tmpJournals+ liftIO $ assertEqual "Hidden files should be excluded" expected filtered+ )+ ) testFilterPaths :: Test-testFilterPaths = TestCase (- sh (- do- let tmpbase = "." </> "test" </> "tmp"- mktree tmpbase- tmpdir <- using (mktempdir tmpbase "hlflowtest")- let tmpJournals = map (tmpdir </>) journalFiles :: [TurtlePath]- let tmpExtras = map (tmpdir </>) extraFiles :: [TurtlePath]- let tmpHidden = map (tmpdir </>) hiddenFiles :: [TurtlePath]- let onDisk = List.sort $ tmpJournals ++ tmpExtras ++ tmpHidden- touchAll onDisk+testFilterPaths =+ TestCase+ ( sh+ ( do+ let tmpbase = "." </> "test" </> "tmp"+ mktree tmpbase+ tmpdir <- using (mktempdir tmpbase "hlflowtest")+ let tmpJournals = map (tmpdir </>) journalFiles :: [TurtlePath]+ let tmpExtras = map (tmpdir </>) extraFiles :: [TurtlePath]+ let tmpHidden = map (tmpdir </>) hiddenFiles :: [TurtlePath]+ let onDisk = List.sort $ tmpJournals ++ tmpExtras ++ tmpHidden+ touchAll onDisk - let nonExistant = map (tmpdir </>) ["where", "is", "my", "mind"]- let toFilter = nonExistant ++ onDisk- filtered <- single $ filterPaths testfile toFilter- let actual = List.sort filtered- liftIO $ assertEqual "The filtered paths should exclude files not actually on disk" onDisk actual- )- )+ let nonExistant = map (tmpdir </>) ["where", "is", "my", "mind"]+ let toFilter = nonExistant ++ onDisk+ filtered <- single $ filterPaths testfile toFilter+ let actual = List.sort filtered+ liftIO $ assertEqual "The filtered paths should exclude files not actually on disk" onDisk actual+ )+ ) tests :: Test tests = TestList [testHiddenFiles, testFilterPaths]
test/Common/Unit.hs view
@@ -1,65 +1,68 @@-{-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE OverloadedLists #-}+{-# LANGUAGE OverloadedStrings #-} module Common.Unit where -import Test.HUnit- import Hledger.Flow.BaseDir (relativeToBase') import Hledger.Flow.Common+import Test.HUnit testShowCmdArgs :: Test-testShowCmdArgs = TestCase (- do- let opts = ["--number", "/tmp/file with spaces"]- let expected = "--number '/tmp/file with spaces'"- let actual = showCmdArgs opts- assertEqual "Convert command-line arguments to text" expected actual)+testShowCmdArgs =+ TestCase+ ( do+ let opts = ["--number", "/tmp/file with spaces"]+ let expected = "--number '/tmp/file with spaces'"+ let actual = showCmdArgs opts+ assertEqual "Convert command-line arguments to text" expected actual+ ) testRelativeToBase :: Test-testRelativeToBase = TestCase (- do- let expected = "file1.journal"- let relativeWithTrailingSlash = relativeToBase' "./base/dir/" "./base/dir/file1.journal"- assertEqual "relative base dir with trailing slash" expected relativeWithTrailingSlash+testRelativeToBase =+ TestCase+ ( do+ let expected = "file1.journal"+ let relativeWithTrailingSlash = relativeToBase' "./base/dir/" "./base/dir/file1.journal"+ assertEqual "relative base dir with trailing slash" expected relativeWithTrailingSlash - let relativeNoTrailingSlash = relativeToBase' "./base/dir" "./base/dir/file1.journal"- assertEqual "relative base dir without a trailing slash" expected relativeNoTrailingSlash+ let relativeNoTrailingSlash = relativeToBase' "./base/dir" "./base/dir/file1.journal"+ assertEqual "relative base dir without a trailing slash" expected relativeNoTrailingSlash - let absoluteWithTrailingSlash = relativeToBase' "/base/dir/" "/base/dir/file1.journal"- assertEqual "absolute base dir with trailing slash" expected absoluteWithTrailingSlash+ let absoluteWithTrailingSlash = relativeToBase' "/base/dir/" "/base/dir/file1.journal"+ assertEqual "absolute base dir with trailing slash" expected absoluteWithTrailingSlash - let absoluteNoTrailingSlash = relativeToBase' "/base/dir" "/base/dir/file1.journal"- assertEqual "absolute base dir without a trailing slash" expected absoluteNoTrailingSlash+ let absoluteNoTrailingSlash = relativeToBase' "/base/dir" "/base/dir/file1.journal"+ assertEqual "absolute base dir without a trailing slash" expected absoluteNoTrailingSlash - let absoluteTwiceNoTrailingSlash = relativeToBase' "/base/dir" "/base/dir"- assertEqual "absolute base dir without a trailing slash supplied twice" "./" absoluteTwiceNoTrailingSlash+ let absoluteTwiceNoTrailingSlash = relativeToBase' "/base/dir" "/base/dir"+ assertEqual "absolute base dir without a trailing slash supplied twice" "./" absoluteTwiceNoTrailingSlash - let absoluteTwiceWithTrailingSlash = relativeToBase' "/base/dir/" "/base/dir/"- assertEqual "absolute base dir with a trailing slash supplied twice" "./" absoluteTwiceWithTrailingSlash+ let absoluteTwiceWithTrailingSlash = relativeToBase' "/base/dir/" "/base/dir/"+ assertEqual "absolute base dir with a trailing slash supplied twice" "./" absoluteTwiceWithTrailingSlash - let absoluteTwiceNoTrailingSlashOnSecondParam = relativeToBase' "/base/dir/" "/base/dir"- assertEqual "absolute base dir supplied twice, but the second param has no slash" "./" absoluteTwiceNoTrailingSlashOnSecondParam+ let absoluteTwiceNoTrailingSlashOnSecondParam = relativeToBase' "/base/dir/" "/base/dir"+ assertEqual "absolute base dir supplied twice, but the second param has no slash" "./" absoluteTwiceNoTrailingSlashOnSecondParam - let mismatch = relativeToBase' "/base/dir" "/unrelated/dir/file1.journal"- assertEqual "A basedir with no shared prefix should return the supplied file unchanged" "/unrelated/dir/file1.journal" mismatch- )+ let mismatch = relativeToBase' "/base/dir" "/unrelated/dir/file1.journal"+ assertEqual "A basedir with no shared prefix should return the supplied file unchanged" "/unrelated/dir/file1.journal" mismatch+ ) testExtractDigits :: Test-testExtractDigits = TestCase (- do- let txt1 = "A number: 321\nAnother number is 42, so is 0"+testExtractDigits =+ TestCase+ ( do+ let txt1 = "A number: 321\nAnother number is 42, so is 0" - let expected1 = Right 321420- let actual1 = extractDigits txt1- assertEqual "Extract digits from text 1" expected1 actual1+ let expected1 = Right 321420+ let actual1 = extractDigits txt1+ assertEqual "Extract digits from text 1" expected1 actual1 - let txt2 = "No numbers in this line"+ let txt2 = "No numbers in this line" - let expected2 = Left "input does not start with a digit"- let actual2 = extractDigits txt2- assertEqual "Extract digits from text 2" expected2 actual2- )+ let expected2 = Left "input does not start with a digit"+ let actual2 = extractDigits txt2+ assertEqual "Extract digits from text 2" expected2 actual2+ ) tests :: Test tests = TestList [testShowCmdArgs, testRelativeToBase, testExtractDigits]
test/PathHelpers/Unit.hs view
@@ -2,24 +2,25 @@ module PathHelpers.Unit where -import Test.HUnit-import Path import Hledger.Flow.PathHelpers+import Path+import Test.HUnit testPathSize :: Test-testPathSize = TestCase (- do- let d0 = [reldir|.|]- let d1 = [reldir|d1|]- let d1ond0 = d0 </> [reldir|d1|]- let d2 = d1 </> [reldir|d2|]- let d3 = d2 </> [reldir|d3|]- assertEqual "Calculate the path size correctly" 0 (pathSize d0)- assertEqual "Calculate the path size correctly" 1 (pathSize d1)- assertEqual "Calculate the path size correctly" 1 (pathSize d1ond0)- assertEqual "Calculate the path size correctly" 2 (pathSize d2)- assertEqual "Calculate the path size correctly" 3 (pathSize d3)- )+testPathSize =+ TestCase+ ( do+ let d0 = [reldir|.|]+ let d1 = [reldir|d1|]+ let d1ond0 = d0 </> [reldir|d1|]+ let d2 = d1 </> [reldir|d2|]+ let d3 = d2 </> [reldir|d3|]+ assertEqual "Calculate the path size correctly" 0 (pathSize d0)+ assertEqual "Calculate the path size correctly" 1 (pathSize d1)+ assertEqual "Calculate the path size correctly" 1 (pathSize d1ond0)+ assertEqual "Calculate the path size correctly" 2 (pathSize d2)+ assertEqual "Calculate the path size correctly" 3 (pathSize d3)+ ) tests :: Test tests = TestList [testPathSize]
test/Spec.hs view
@@ -1,17 +1,16 @@-{-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE OverloadedLists #-}+{-# LANGUAGE OverloadedStrings #-} module Main where -import Test.HUnit-import Turtle--import qualified Common.Unit-import qualified PathHelpers.Unit-import qualified Common.Integration import qualified BaseDir.Integration-import qualified CSVImport.Unit import qualified CSVImport.Integration+import qualified CSVImport.Unit+import qualified Common.Integration+import qualified Common.Unit+import qualified PathHelpers.Unit+import Test.HUnit+import Turtle tests :: Test tests = TestList [Common.Unit.tests, Common.Integration.tests, PathHelpers.Unit.tests, BaseDir.Integration.tests, CSVImport.Unit.tests, CSVImport.Integration.tests]
test/TestHelpers.hs view
@@ -3,44 +3,44 @@ module TestHelpers where -import Path- import Data.Maybe (fromMaybe)--import Hledger.Flow.Internals ( versionInfo, SystemInfo(..) )+import Hledger.Flow.Internals (SystemInfo (..), versionInfo) import Hledger.Flow.PathHelpers (RelFile)--import qualified Hledger.Flow.Types as FlowTypes import Hledger.Flow.RuntimeOptions+import qualified Hledger.Flow.Types as FlowTypes+import Path import qualified System.Info as Sys defaultHlInfo :: FlowTypes.HledgerInfo defaultHlInfo = FlowTypes.HledgerInfo [absfile|/path/to/hledger|] "1.2.3" testSystemInfo :: SystemInfo-testSystemInfo = SystemInfo {- os = Sys.os,- arch = Sys.arch,- compilerName = Sys.compilerName,- compilerVersion = Sys.compilerVersion,- cores = 1,- availableCores = 1- }+testSystemInfo =+ SystemInfo+ { os = Sys.os,+ arch = Sys.arch,+ compilerName = Sys.compilerName,+ compilerVersion = Sys.compilerVersion,+ cores = 1,+ availableCores = 1+ } defaultOpts :: FlowTypes.BaseDir -> RuntimeOptions-defaultOpts bd = RuntimeOptions {- baseDir = bd- , importRunDir = [reldir|./|]- , importStartYear = Nothing- , onlyNewFiles = False- , hfVersion = versionInfo testSystemInfo- , hledgerInfo = defaultHlInfo- , sysInfo = testSystemInfo- , verbose = False- , showOptions = False- , sequential = False- , batchSize = 1-}+defaultOpts bd =+ RuntimeOptions+ { baseDir = bd,+ importRunDir = [reldir|./|],+ importStartYear = Nothing,+ onlyNewFiles = False,+ hfVersion = versionInfo testSystemInfo,+ hledgerInfo = defaultHlInfo,+ sysInfo = testSystemInfo,+ verbose = False,+ showOptions = False,+ sequential = False,+ batchSize = 1,+ prettyReports = True+ } toJournal :: RelFile -> RelFile toJournal inFile = do@@ -50,37 +50,37 @@ (parent . parent . parent) inFile </> [reldir|3-journal|] </> yearDir </> journalName inputJohnSavings2017 :: [RelFile]-inputJohnSavings2017 = [- [relfile|import/john/bogartbank/savings/1-in/2017/2017-11-30.csv|],- [relfile|import/john/bogartbank/savings/1-in/2017/2017-12-30.csv|]+inputJohnSavings2017 =+ [ [relfile|import/john/bogartbank/savings/1-in/2017/2017-11-30.csv|],+ [relfile|import/john/bogartbank/savings/1-in/2017/2017-12-30.csv|] ] johnSavingsJournals2017 :: [RelFile] johnSavingsJournals2017 = map toJournal inputJohnSavings2017 inputJohnSavings2018 :: [RelFile]-inputJohnSavings2018 = [- [relfile|import/john/bogartbank/savings/1-in/2018/2018-02-30.csv|],- [relfile|import/john/bogartbank/savings/1-in/2018/2018-01-30.csv|]+inputJohnSavings2018 =+ [ [relfile|import/john/bogartbank/savings/1-in/2018/2018-02-30.csv|],+ [relfile|import/john/bogartbank/savings/1-in/2018/2018-01-30.csv|] ] johnSavingsJournals2018 :: [RelFile] johnSavingsJournals2018 = map toJournal inputJohnSavings2018 inputJohnChecking2018 :: [RelFile]-inputJohnChecking2018 = [- [relfile|import/john/bogartbank/checking/1-in/2018/2018-11-30.csv|],- [relfile|import/john/bogartbank/checking/1-in/2018/2018-10-30.csv|],- [relfile|import/john/bogartbank/checking/1-in/2018/2018-12-30.csv|]+inputJohnChecking2018 =+ [ [relfile|import/john/bogartbank/checking/1-in/2018/2018-11-30.csv|],+ [relfile|import/john/bogartbank/checking/1-in/2018/2018-10-30.csv|],+ [relfile|import/john/bogartbank/checking/1-in/2018/2018-12-30.csv|] ] johnCheckingJournals2018 :: [RelFile] johnCheckingJournals2018 = map toJournal inputJohnChecking2018 inputJohnChecking2019 :: [RelFile]-inputJohnChecking2019 = [- [relfile|import/john/bogartbank/checking/1-in/2019/2019-01-30.csv|],- [relfile|import/john/bogartbank/checking/1-in/2019/2019-02-30.csv|]+inputJohnChecking2019 =+ [ [relfile|import/john/bogartbank/checking/1-in/2019/2019-01-30.csv|],+ [relfile|import/john/bogartbank/checking/1-in/2019/2019-02-30.csv|] ] johnCheckingJournals2019 :: [RelFile]@@ -120,20 +120,20 @@ janeSavings2017 = [relfile|import/jane/bogartbank/savings/1-in/2017/2017-12-30.csv|] janeSavings2018 :: [RelFile]-janeSavings2018 = [- [relfile|import/jane/bogartbank/savings/1-in/2018/2018-01-30.csv|]- , [relfile|import/jane/bogartbank/savings/1-in/2018/2018-12-30.csv|]+janeSavings2018 =+ [ [relfile|import/jane/bogartbank/savings/1-in/2018/2018-01-30.csv|],+ [relfile|import/jane/bogartbank/savings/1-in/2018/2018-12-30.csv|] ] janeSavings2019 :: RelFile janeSavings2019 = [relfile|import/jane/bogartbank/savings/1-in/2019/2019-01-30.csv|] inputJaneBogart :: [RelFile]-inputJaneBogart = [- janeSavings2017- , [relfile|import/jane/bogartbank/savings/3-journals/2018/2018-01-30.journal|]- , [relfile|import/jane/bogartbank/savings/3-journals/2018/2018-12-30.journal|]- , janeSavings2019+inputJaneBogart =+ [ janeSavings2017,+ [relfile|import/jane/bogartbank/savings/3-journals/2018/2018-01-30.journal|],+ [relfile|import/jane/bogartbank/savings/3-journals/2018/2018-12-30.journal|],+ janeSavings2019 ] janeSavingsJournal2017 :: RelFile
test/TestHelpersTurtle.hs view
@@ -2,43 +2,46 @@ module TestHelpersTurtle where -import Turtle--import Hledger.Flow.PathHelpers (TurtlePath) import Hledger.Flow.Common (changePathAndExtension)+import Hledger.Flow.PathHelpers (TurtlePath)+import Turtle inputJohnBogart :: [TurtlePath]-inputJohnBogart = [- "import/john/bogartbank/savings/1-in/2017/2017-11-30.csv",- "import/john/bogartbank/savings/1-in/2017/2017-12-30.csv",- "import/john/bogartbank/savings/1-in/2018/2018-02-30.csv",- "import/john/bogartbank/savings/1-in/2018/2018-01-30.csv",- "import/john/bogartbank/checking/1-in/2018/2018-11-30.csv",- "import/john/bogartbank/checking/1-in/2018/2018-10-30.csv",- "import/john/bogartbank/checking/1-in/2018/2018-12-30.csv",- "import/john/bogartbank/checking/1-in/2019/2019-01-30.csv",- "import/john/bogartbank/checking/1-in/2019/2019-02-30.csv"]+inputJohnBogart =+ [ "import/john/bogartbank/savings/1-in/2017/2017-11-30.csv",+ "import/john/bogartbank/savings/1-in/2017/2017-12-30.csv",+ "import/john/bogartbank/savings/1-in/2018/2018-02-30.csv",+ "import/john/bogartbank/savings/1-in/2018/2018-01-30.csv",+ "import/john/bogartbank/checking/1-in/2018/2018-11-30.csv",+ "import/john/bogartbank/checking/1-in/2018/2018-10-30.csv",+ "import/john/bogartbank/checking/1-in/2018/2018-12-30.csv",+ "import/john/bogartbank/checking/1-in/2019/2019-01-30.csv",+ "import/john/bogartbank/checking/1-in/2019/2019-02-30.csv"+ ] inputJohnOther :: [TurtlePath]-inputJohnOther = [- "import/john/otherbank/creditcard/1-in/2017/2017-12-30.csv",- "import/john/otherbank/creditcard/1-in/2018/2018-01-30.csv",- "import/john/otherbank/investments/1-in/2018/2018-12-30.csv",- "import/john/otherbank/investments/1-in/2019/2019-01-30.csv"]+inputJohnOther =+ [ "import/john/otherbank/creditcard/1-in/2017/2017-12-30.csv",+ "import/john/otherbank/creditcard/1-in/2018/2018-01-30.csv",+ "import/john/otherbank/investments/1-in/2018/2018-12-30.csv",+ "import/john/otherbank/investments/1-in/2019/2019-01-30.csv"+ ] inputJaneBogart :: [TurtlePath]-inputJaneBogart = [- "import/jane/bogartbank/savings/1-in/2017/2017-12-30.csv",- "import/jane/bogartbank/savings/1-in/2018/2018-01-30.csv",- "import/jane/bogartbank/checking/1-in/2018/2018-12-30.csv",- "import/jane/bogartbank/checking/1-in/2019/2019-01-30.csv"]+inputJaneBogart =+ [ "import/jane/bogartbank/savings/1-in/2017/2017-12-30.csv",+ "import/jane/bogartbank/savings/1-in/2018/2018-01-30.csv",+ "import/jane/bogartbank/checking/1-in/2018/2018-12-30.csv",+ "import/jane/bogartbank/checking/1-in/2019/2019-01-30.csv"+ ] inputJaneOther :: [TurtlePath]-inputJaneOther = [- "import/jane/otherbank/creditcard/1-in/2017/2017-12-30.csv",- "import/jane/otherbank/creditcard/1-in/2018/2018-01-30.csv",- "import/jane/otherbank/investments/1-in/2018/2018-12-30.csv",- "import/jane/otherbank/investments/1-in/2019/2019-01-30.csv"]+inputJaneOther =+ [ "import/jane/otherbank/creditcard/1-in/2017/2017-12-30.csv",+ "import/jane/otherbank/creditcard/1-in/2018/2018-01-30.csv",+ "import/jane/otherbank/investments/1-in/2018/2018-12-30.csv",+ "import/jane/otherbank/investments/1-in/2019/2019-01-30.csv"+ ] inputFiles :: [TurtlePath] inputFiles = inputJohnBogart <> inputJohnOther <> inputJaneBogart <> inputJaneOther@@ -53,14 +56,16 @@ extraFiles = ["import/john/bogartbank/savings/2017-opening.journal"] hiddenFiles :: [TurtlePath]-hiddenFiles = [".hiddenfile",- "checking/.DS_Store",- "import/john/bogartbank/savings/1-in/.anotherhiddenfile",- "import/john/bogartbank/checking/1-in/2018/.hidden",- "import/john/bogartbank/checking/3-journal/2018/.hidden"]+hiddenFiles =+ [ ".hiddenfile",+ "checking/.DS_Store",+ "import/john/bogartbank/savings/1-in/.anotherhiddenfile",+ "import/john/bogartbank/checking/1-in/2018/.hidden",+ "import/john/bogartbank/checking/3-journal/2018/.hidden"+ ] toJournals :: [TurtlePath] -> [TurtlePath]-toJournals = map (changePathAndExtension "3-journal" "journal")+toJournals = map (changePathAndExtension "3-journal/" "journal") touchAll :: [TurtlePath] -> Shell () touchAll = foldl (\acc file -> acc <> superTouch file) (return ())