hledger-flow 0.12.2.1 → 0.12.3.0

raw patch · 15 files changed

+255/−162 lines, 15 filesPVP: major bump suggested

API removals or changes: PVP suggests a major version bump

API changes (from Hackage documentation)

Files

• ChangeLog.md +8/−0
• README.org +94/−18
• app/Main.hs +12/−25
• hledger-flow.cabal +3/−3
• src/Hledger/Flow/CSVImport.hs +10/−9
• src/Hledger/Flow/Import/Types.hs +0/−19
• src/Hledger/Flow/Report/Types.hs +0/−25
• src/Hledger/Flow/Reports.hs +54/−34
• src/Hledger/Flow/RuntimeOptions.hs +25/−0
• test/CSVImport/Integration.hs +5/−3
• test/CSVImport/Unit.hs +9/−3
• test/Common/Integration.hs +5/−6
• test/Common/Unit.hs +6/−4
• test/Spec.hs +4/−3
• test/TestHelpers.hs +20/−10

@@ -1,5 +1,13 @@ # Changelog for [hledger-flow](https://github.com/apauley/hledger-flow) +## 0.12.3++Add more reports:++- Balance Sheet per owner per year, and for all owners per year+- Unknown transactions per owner per year+- A transfer balance overview per year+ ## 0.12.2.1  Fix resolver extraction and hledger-flow --version in release-tarball script

@@ -1,5 +1,4 @@ #+STARTUP: showall-#+PROPERTY: header-args:sh :prologue exec 2>&1 :epilogue echo :  * Hledger Flow   :PROPERTIES:@@ -20,8 +19,8 @@ processing of electronic statements as much as possible, as opposed to manually adding your own hledger journal entries. -Manual entries are still possible, we just think it saves time in the-long run to automatically process a statement whenever one is available.+Manual entries are still possible, it just saves time in the long run to+automatically process a statement whenever one is available.  Within =hledger-flow= you will keep your original bank statements around permanently as input, and generate classified Hledger journals each time@@ -38,17 +37,6 @@ with the ideas in [[https://github.com/adept/full-fledged-hledger/wiki][adept's Full-fledged Hledger]] aren't really specific to my own finances, and can be shared. -* How do I install it?-  :PROPERTIES:-  :CUSTOM_ID: how-do-i-install-it-  :END:--The easiest way to get it running is to download [[https://github.com/apauley/hledger-flow/releases][the latest release]]-for your OS, and copy the =hledger-flow= executable to a directory in your PATH.-Then just run it and see what it tells you to do.--You can also compile it yourself by following the [[https://github.com/apauley/hledger-flow/blob/master/CONTRIBUTING.org#build-the-project][build instructions]].- * Overview of the Basic Workflow   :PROPERTIES:   :CUSTOM_ID: overview-of-the-basic-workflow@@ -94,6 +82,17 @@ - You appreciate the fact that all your financial information stays   within your control. +* How do I install it?+  :PROPERTIES:+  :CUSTOM_ID: how-do-i-install-it+  :END:++The easiest way to get it running is to download [[https://github.com/apauley/hledger-flow/releases][the latest release]]+for your OS, and copy the =hledger-flow= executable to a directory in your PATH.+Then just run it and see what it tells you to do.++You can also compile it yourself by following the [[https://github.com/apauley/hledger-flow/blob/master/CONTRIBUTING.org#build-the-project][build instructions]].+ * How Stable is it?   :PROPERTIES:   :CUSTOM_ID: how-stable-is-it@@ -612,14 +611,91 @@   :CUSTOM_ID: faq   :END: +** How do you balance transfers between 2 accounts when you have statements for both accounts?+   :PROPERTIES:+   :CUSTOM_ID: transfer-2-accounts+   :END:++*** The Problem++In your primary bank account you've happily been classifying transfers to a+secondary account as just =Expenses:OtherAccount=.++But you've recently started processing the statements from the second account as+well so that you can classify those expenses more accurately.++And now the balances of these two accounts are all wrong when the statements of+each account deals with money transferred between these two accounts.++In =bank1.journal=, imported from =bank1.csv=:+#+BEGIN_EXAMPLE+2018/11/09 Transfer from primary account to secondary account+    Assets:Bank1:Primary    $-200+    Assets:Bank2:Secondary+#+END_EXAMPLE++In =bank2.journal=, imported from =bank2.csv=:+#+BEGIN_EXAMPLE+2018/11/09 Transfer from primary account to secondary account+    Assets:Bank2:Secondary  $200+    Assets:Bank1:Primary+#+END_EXAMPLE++*** The Solution++As soon as you start importing statements for both accounts you will have to+introduce an intermediate account for classification between these two accounts.++I use =Assets:Transfers:*=.++And we may have reports looking at these transfers accounts at some point, you+should consider using the same names.++The above example then becomes as follows.++In =bank1.journal=, imported from =bank1.csv=:+#+BEGIN_EXAMPLE+2019-05-18 Transfer from primary account to secondary account+    Assets:Bank1:Primary         $-200+    Assets:Transfers:Bank1Bank2+#+END_EXAMPLE++In =bank2.journal=, imported from =bank2.csv=:+#+BEGIN_EXAMPLE+2019-05-18 Transfer from primary account to secondary account+    Assets:Bank2:Secondary       $200+    Assets:Transfers:Bank1Bank2+#+END_EXAMPLE++Any posting to =Assets:Transfers:*= indicates an in "in-flight" amount.+You would expect the balance of =Assets:Transfers= to be zero most of the time.+Whenever it isn't zero it means that you either don't yet have the other side of+the transfer, or that something is wrong in your rules.++You could theoretically just use =Assets:Transfers= without any subaccounts, but+I found it useful to use subaccounts. Because then the subaccounts can show me+where I should look for any missing transfer transaction.++I typically use sorted names as the subaccount (Python code sample):++#+BEGIN_SRC python+    "Assets:Transfers:" + "".join(sorted(["Bank2", "Bank1"]))+#+END_SRC++*** External references++    This approach is based on what is described in Full-fledged hledger:+    [[https://github.com/adept/full-fledged-hledger/wiki/Adding-more-accounts#lets-make-sure-that-transfers-are-not-double-counted]]++    The question was first asked in [[https://github.com/apauley/hledger-flow/issues/51][issue #51]].+ ** How does =hledger-flow= differ from =Full-fledged Hledger=?    :PROPERTIES:    :CUSTOM_ID: how-does-hledger-flow-differ-from-full-fledged-hledger    :END: -[[https://github.com/adept/full-fledged-hledger/wiki#full-fledged-hledger-tutorial][Full-fledged-Hledger]] is a brilliant system, and hledger-flow continues to learn-much from it.+[[https://github.com/adept/full-fledged-hledger/wiki#full-fledged-hledger-tutorial][Full-fledged Hledger]]+is a brilliant system, and hledger-flow continues to learn much from it.  It has great documentation that does an excellent job of not only showing *how* things can be done, but also *why* it is such a great@@ -637,7 +713,7 @@ |                                                                                                                                                                                                                            |                                                                                                                                                                                                                                                            | | Maintaining include files are currently part of the user's responsibility.                                                                                                                                                 | Hledger Flow generates flexible include files for you, and automatically includes opening/closing journals if the appropriately named files are present on disk.                                                                                           | |                                                                                                                                                                                                                            |                                                                                                                                                                                                                                                            |-| FFH actually generates some useful reports right now. Hledger Flow still [[https://github.com/apauley/hledger-flow/pull/4][plans to get this done]] one day.                                                                                                                   | The "owner/bank/account" structure may look a bit much at first, but it allows us to run separate queries/reports for me/my spouse/my business etc and also allows for reports covering all of it in a an overall view.                                    |+| FFH generates some of the same reports as Hledger Flow.                                                                                                                                                                    | The "owner/bank/account" structure may look a bit much at first, but it allows us to run separate queries/reports for "me" vs "my spouse" vs "my business" etc and also allows for reports covering all of it in a an overall view.                       | |                                                                                                                                                                                                                            |                                                                                                                                                                                                                                                            | | FFH chose scripts and build files that you can easily modify as you go along, but this requires a Haskell runtime to be installed everywhere it needs to run. The included docker image helps to make it less of an issue. | Hledger Flow distributes a compiled binary. This means users or deployment targets don't need extra dependencies installed, they can just run a CLI program. This also provides a clearer distinction between what is provided, and what users need to do. | |                                                                                                                                                                                                                            |                                                                                                                                                                                                                                                            |

@@ -8,9 +8,8 @@ import Options.Applicative import Data.Semigroup ((<>)) -import qualified Hledger.Flow.Import.Types as IT-import qualified Hledger.Flow.Report.Types as RT import Hledger.Flow.Common+import qualified Hledger.Flow.RuntimeOptions as RT import Hledger.Flow.Reports import Hledger.Flow.CSVImport @@ -29,32 +28,20 @@   cmd <- options "An hledger workflow focusing on automated statement import and classification:\nhttps://github.com/apauley/hledger-flow#readme" baseCommandParser   case cmd of     Version                                -> stdout $ select versionInfo-    Command mainParams' (Import subParams) -> toImportOptions mainParams' subParams >>= importCSVs-    Command mainParams' (Report subParams) -> toReportOptions mainParams' subParams >>= generateReports--toImportOptions :: MainParams -> SubcommandParams -> IO IT.ImportOptions-toImportOptions mainParams' subParams' = do-  bd <- determineBaseDir $ maybeBaseDir subParams'-  hli <- hledgerInfoFromPath $ hledgerPathOpt mainParams'-  return IT.ImportOptions { IT.baseDir = bd-                          , IT.hfVersion = versionInfo'-                          , IT.hledgerInfo = hli-                          , IT.sysInfo = systemInfo-                          , IT.verbose = verbose mainParams'-                          , IT.showOptions = showOpts mainParams'-                          , IT.sequential = sequential mainParams' }+    Command mainParams' (Import subParams) -> toRuntimeOptions mainParams' subParams >>= importCSVs+    Command mainParams' (Report subParams) -> toRuntimeOptions mainParams' subParams >>= generateReports -toReportOptions :: MainParams -> SubcommandParams -> IO RT.ReportOptions-toReportOptions mainParams' subParams' = do+toRuntimeOptions :: MainParams -> SubcommandParams -> IO RT.RuntimeOptions+toRuntimeOptions mainParams' subParams' = do   bd <- determineBaseDir $ maybeBaseDir subParams'   hli <- hledgerInfoFromPath $ hledgerPathOpt mainParams'-  return RT.ReportOptions { RT.baseDir = bd-                          , RT.hfVersion = versionInfo'-                          , RT.hledgerInfo = hli-                          , RT.sysInfo = systemInfo-                          , RT.verbose = verbose mainParams'-                          , RT.showOptions = showOpts mainParams'-                          , RT.sequential = sequential mainParams' }+  return RT.RuntimeOptions { RT.baseDir = bd+                           , RT.hfVersion = versionInfo'+                           , RT.hledgerInfo = hli+                           , RT.sysInfo = systemInfo+                           , RT.verbose = verbose mainParams'+                           , RT.showOptions = showOpts mainParams'+                           , RT.sequential = sequential mainParams' }  baseCommandParser :: Parser BaseCommand baseCommandParser = (Command <$> verboseParser <*> commandParser)

@@ -4,10 +4,10 @@ -- -- see: https://github.com/sol/hpack ----- hash: 88668a61a38818c6bcecc7aa0b1e04eda0c03c931ff0a2b9dedcd23cd2bd646d+-- hash: 238554ba5f9f4fbeff3effd7709b051f2e6d5e4386d56f57aa1c7de2241a6727  name:           hledger-flow-version:        0.12.2.1+version:        0.12.3.0 synopsis:       An hledger workflow focusing on automated statement import and classification. description:    Please see the README on GitHub at <https://github.com/apauley/hledger-flow#readme> category:       Finance, Console@@ -32,8 +32,8 @@       Hledger.Flow.Common       Hledger.Flow.CSVImport       Hledger.Flow.Import.Types-      Hledger.Flow.Report.Types       Hledger.Flow.Reports+      Hledger.Flow.RuntimeOptions       Hledger.Flow.Types   other-modules:       Paths_hledger_flow

@@ -11,9 +11,10 @@ import qualified Hledger.Flow.Types as FlowTypes import Hledger.Flow.Import.Types import Hledger.Flow.Common+import Hledger.Flow.RuntimeOptions import Control.Concurrent.STM -importCSVs :: ImportOptions -> IO ()+importCSVs :: RuntimeOptions -> IO () importCSVs opts = sh (   do     ch <- liftIO newTChanIO@@ -32,7 +33,7 @@ inputFilePattern :: Pattern Text inputFilePattern = contains (once (oneOf pathSeparators) <> asciiCI "1-in" <> once (oneOf pathSeparators) <> plus digit <> once (oneOf pathSeparators)) -importCSVs' :: ImportOptions -> TChan FlowTypes.LogMessage -> IO [FilePath]+importCSVs' :: RuntimeOptions -> TChan FlowTypes.LogMessage -> IO [FilePath] importCSVs' opts ch = do   channelOutLn ch "Collecting input files..."   (inputFiles, diff) <- time $ single . shellToList . onlyFiles $ find inputFilePattern $ baseDir opts@@ -51,14 +52,14 @@       sh $ writeIncludesUpTo opts ch "import" importedJournals       return importedJournals -extractAndImport :: ImportOptions -> TChan FlowTypes.LogMessage -> FilePath -> IO FilePath+extractAndImport :: RuntimeOptions -> TChan FlowTypes.LogMessage -> FilePath -> IO FilePath extractAndImport opts ch inputFile = do   case extractImportDirs inputFile of     Right importDirs -> importCSV opts ch importDirs inputFile     Left errorMessage -> do       errExit 1 ch errorMessage inputFile -importCSV :: ImportOptions -> TChan FlowTypes.LogMessage -> ImportDirs -> FilePath -> IO FilePath+importCSV :: RuntimeOptions -> TChan FlowTypes.LogMessage -> ImportDirs -> FilePath -> IO FilePath importCSV opts ch importDirs srcFile = do   let preprocessScript = accountDir importDirs </> "preprocess"   let constructScript = accountDir importDirs </> "construct"@@ -74,14 +75,14 @@   mktree $ directory journalOut   importFun csvFile journalOut -preprocessIfNeeded :: ImportOptions -> TChan FlowTypes.LogMessage -> FilePath -> Line -> Line -> Line -> FilePath -> IO FilePath+preprocessIfNeeded :: RuntimeOptions -> TChan FlowTypes.LogMessage -> FilePath -> Line -> Line -> Line -> FilePath -> IO FilePath preprocessIfNeeded opts ch script bank account owner src = do   shouldPreprocess <- verboseTestFile opts ch script   if shouldPreprocess     then preprocess opts ch script bank account owner src     else return src -preprocess :: ImportOptions -> TChan FlowTypes.LogMessage -> FilePath -> Line -> Line -> Line -> FilePath -> IO FilePath+preprocess :: RuntimeOptions -> TChan FlowTypes.LogMessage -> FilePath -> Line -> Line -> Line -> FilePath -> IO FilePath preprocess opts ch script bank account owner src = do   let csvOut = changePathAndExtension "2-preprocessed" "csv" src   mktree $ directory csvOut@@ -92,14 +93,14 @@   _ <- timeAndExitOnErr opts ch cmdLabel channelOut channelErr (parAwareProc opts) (format fp script, args, empty)   return csvOut -hledgerImport :: ImportOptions -> TChan FlowTypes.LogMessage -> FilePath -> FilePath -> IO FilePath+hledgerImport :: RuntimeOptions -> TChan FlowTypes.LogMessage -> FilePath -> FilePath -> IO FilePath hledgerImport opts ch csvSrc journalOut = do   case extractImportDirs csvSrc of     Right importDirs -> hledgerImport' opts ch importDirs csvSrc journalOut     Left errorMessage -> do       errExit 1 ch errorMessage csvSrc -hledgerImport' :: ImportOptions -> TChan FlowTypes.LogMessage -> ImportDirs -> FilePath -> FilePath -> IO FilePath+hledgerImport' :: RuntimeOptions -> TChan FlowTypes.LogMessage -> ImportDirs -> FilePath -> FilePath -> IO FilePath hledgerImport' opts ch importDirs csvSrc journalOut = do   let candidates = rulesFileCandidates csvSrc importDirs   maybeRulesFile <- firstExistingFile candidates@@ -151,7 +152,7 @@       map (</> srcSpecificFilename) [accountDir importDirs, bankDir importDirs, importDir importDirs]     else [] -customConstruct :: ImportOptions -> TChan FlowTypes.LogMessage -> FilePath -> Line -> Line -> Line -> FilePath -> FilePath -> IO FilePath+customConstruct :: RuntimeOptions -> TChan FlowTypes.LogMessage -> FilePath -> Line -> Line -> Line -> FilePath -> FilePath -> IO FilePath customConstruct opts ch constructScript bank account owner csvSrc journalOut = do   let script = format fp constructScript :: Text   let relScript = relativeToBase opts constructScript

@@ -3,25 +3,6 @@  import Turtle import Prelude hiding (FilePath, putStrLn)-import Hledger.Flow.Types--data ImportOptions = ImportOptions { baseDir :: FilePath-                                   , hfVersion :: Text-                                   , hledgerInfo :: HledgerInfo-                                   , sysInfo :: SystemInfo-                                   , verbose :: Bool-                                   , showOptions :: Bool-                                   , sequential :: Bool }-  deriving (Show)--instance HasVerbosity ImportOptions where-  verbose (ImportOptions _ _ _ _ v _ _) = v--instance HasSequential ImportOptions where-  sequential (ImportOptions _ _ _ _ _ _ sq) = sq--instance HasBaseDir ImportOptions where-  baseDir (ImportOptions bd _ _ _ _ _  _) = bd  data ImportDirs = ImportDirs { importDir  :: FilePath                              , ownerDir   :: FilePath

@@ -1,25 +0,0 @@-module Hledger.Flow.Report.Types-where--import Turtle-import Prelude hiding (FilePath, putStrLn)-import Hledger.Flow.Types--data ReportOptions = ReportOptions { baseDir :: FilePath-                                   , hfVersion :: Text-                                   , hledgerInfo :: HledgerInfo-                                   , sysInfo :: SystemInfo-                                   , verbose :: Bool-                                   , showOptions :: Bool-                                   , sequential :: Bool-                                   }-  deriving (Show)--instance HasVerbosity ReportOptions where-  verbose (ReportOptions _ _ _ _ v _ _) = v--instance HasSequential ReportOptions where-  sequential (ReportOptions _ _ _ _ _ _ sq) = sq--instance HasBaseDir ReportOptions where-  baseDir (ReportOptions bd _ _ _ _ _ _) = bd

@@ -6,7 +6,7 @@  import Turtle hiding (stdout, stderr, proc) import Prelude hiding (FilePath, putStrLn, writeFile)-import Hledger.Flow.Report.Types+import Hledger.Flow.RuntimeOptions import Hledger.Flow.Common import Control.Concurrent.STM import Data.Either@@ -20,8 +20,9 @@                                  , outputDir :: FilePath                                  }                   deriving (Show)+type ReportGenerator = RuntimeOptions -> TChan FlowTypes.LogMessage -> FilePath -> FilePath -> Integer -> IO (Either FilePath FilePath) -generateReports :: ReportOptions -> IO ()+generateReports :: RuntimeOptions -> IO () generateReports opts = sh (   do     ch <- liftIO newTChanIO@@ -35,7 +36,7 @@     wait logHandle   ) -generateReports' :: ReportOptions -> TChan FlowTypes.LogMessage -> IO [Either FilePath FilePath]+generateReports' :: RuntimeOptions -> TChan FlowTypes.LogMessage -> IO [Either FilePath FilePath] generateReports' opts ch = do   let wipMsg = "Report generation is still a work-in-progress - please let me know how this can be more useful.\n\n"                <> "Keep an eye out for report-related pull requests and issues, and feel free to submit some of your own:\n"@@ -43,66 +44,85 @@                <> "https://github.com/apauley/hledger-flow/issues\n"   channelOutLn ch wipMsg   owners <- single $ shellToList $ listOwners opts-  let baseJournal = journalFile opts []-  let baseReportDir = outputReportDir opts ["all"]-  baseYears <- includeYears ch baseJournal-  let baseParams = if length owners > 1 then [(ReportParams baseJournal baseYears baseReportDir)] else []+  let aggregateJournal = journalFile opts []+  let aggregateReportDir = outputReportDir opts ["all"]+  aggregateYears <- includeYears ch aggregateJournal+  let aggregateParams = ReportParams aggregateJournal aggregateYears aggregateReportDir+  let aggregateOnlyReports = reportActions opts ch [transferBalance] aggregateParams   ownerParams <- ownerParameters opts ch owners-  let reportParams = baseParams ++ ownerParams-  let actions = List.concat $ fmap (generateReports'' opts ch) reportParams-  parAwareActions opts actions+  let ownerWithAggregateParams = (if length owners > 1 then [aggregateParams] else []) ++ ownerParams+  let sharedOptions = ["--depth", "2", "--pretty-tables", "not:equity"]+  let ownerWithAggregateReports = List.concat $ fmap (reportActions opts ch [incomeStatement sharedOptions, balanceSheet sharedOptions]) ownerWithAggregateParams+  let ownerOnlyReports = List.concat $ fmap (reportActions opts ch [accountList, unknownTransactions]) ownerParams+  parAwareActions opts (aggregateOnlyReports ++ ownerWithAggregateReports ++ ownerOnlyReports) -generateReports'' :: ReportOptions -> TChan FlowTypes.LogMessage -> ReportParams -> [IO (Either FilePath FilePath)]-generateReports'' opts ch (ReportParams journal years reportsDir) = do+reportActions :: RuntimeOptions -> TChan FlowTypes.LogMessage -> [ReportGenerator] -> ReportParams -> [IO (Either FilePath FilePath)]+reportActions opts ch reports (ReportParams journal years reportsDir) = do   y <- years-  let actions = map (\r -> r opts ch journal reportsDir y) [accountList, incomeStatement]-  map (fmap fst) actions--incomeStatement :: ReportOptions -> TChan FlowTypes.LogMessage -> FilePath -> FilePath -> Integer -> IO (Either FilePath FilePath, FlowTypes.FullTimedOutput)-incomeStatement opts ch journal reportsDir year = do-  let sharedOptions = ["--depth", "2", "--pretty-tables", "not:equity", "--cost", "--value"]-  let reportArgs = ["incomestatement"] ++ sharedOptions-  generateReport opts ch journal reportsDir year ("income-expenses" <.> "txt") reportArgs+  map (\r -> r opts ch journal reportsDir y) reports -accountList :: ReportOptions -> TChan FlowTypes.LogMessage -> FilePath -> FilePath -> Integer -> IO (Either FilePath FilePath, FlowTypes.FullTimedOutput)+accountList :: ReportGenerator accountList opts ch journal reportsDir year = do   let reportArgs = ["accounts"]-  generateReport opts ch journal reportsDir year ("accounts" <.> "txt") reportArgs+  generateReport opts ch journal reportsDir year ("accounts" <.> "txt") reportArgs (not . T.null) -generateReport :: ReportOptions -> TChan FlowTypes.LogMessage -> FilePath -> FilePath -> Integer -> FilePath -> [Text] -> IO (Either FilePath FilePath, FlowTypes.FullTimedOutput)-generateReport opts ch journal baseOutDir year fileName args = do+unknownTransactions :: ReportGenerator+unknownTransactions opts ch journal reportsDir year = do+  let reportArgs = ["print", "unknown"]+  generateReport opts ch journal reportsDir year ("unknown-transactions" <.> "txt") reportArgs (not . T.null)++incomeStatement :: [Text] -> ReportGenerator+incomeStatement sharedOptions opts ch journal reportsDir year = do+  let reportArgs = ["incomestatement"] ++ sharedOptions ++ ["--cost", "--value"]+  generateReport opts ch journal reportsDir year ("income-expenses" <.> "txt") reportArgs (not . T.null)++balanceSheet :: [Text] -> ReportGenerator+balanceSheet sharedOptions opts ch journal reportsDir year = do+  let reportArgs = ["balancesheet"] ++ sharedOptions ++ ["--cost", "--flat"]+  generateReport opts ch journal reportsDir year ("balance-sheet" <.> "txt") reportArgs (not . T.null)++transferBalance :: ReportGenerator+transferBalance opts ch journal reportsDir year = do+  let reportArgs = ["balance", "--pretty-tables", "--quarterly", "--flat", "--no-total", "transfer"]+  generateReport opts ch journal reportsDir year ("transfer-balance" <.> "txt") reportArgs (\txt -> (length $ T.lines txt) > 4)++generateReport :: RuntimeOptions -> TChan FlowTypes.LogMessage -> FilePath -> FilePath -> Integer -> FilePath -> [Text] -> (Text -> Bool) -> IO (Either FilePath FilePath)+generateReport opts ch journal baseOutDir year fileName args successCheck = do   let reportsDir = baseOutDir </> intPath year   mktree reportsDir   let outputFile = reportsDir </> fileName   let relativeJournal = relativeToBase opts journal+  let relativeOutputFile = relativeToBase opts outputFile   let reportArgs = ["--file", format fp journal, "--period", repr year] ++ args   let reportDisplayArgs = ["--file", format fp relativeJournal, "--period", repr year] ++ args   let hledger = format fp $ FlowTypes.hlPath . hledgerInfo $ opts :: Text   let cmdLabel = format ("hledger "%s) $ showCmdArgs reportDisplayArgs-  result@((exitCode, stdOut, _), _) <- timeAndExitOnErr opts ch cmdLabel dummyLogger channelErr procStrictWithErr (hledger, reportArgs, empty)-  if not (T.null stdOut)+  ((exitCode, stdOut, _), _) <- timeAndExitOnErr opts ch cmdLabel dummyLogger channelErr procStrictWithErr (hledger, reportArgs, empty)+  if (successCheck stdOut)     then     do       writeTextFile outputFile (cmdLabel <> "\n\n"<> stdOut)-      logVerbose opts ch $ format ("Wrote "%fp) $ relativeToBase opts outputFile-      return (Right outputFile, result)+      logVerbose opts ch $ format ("Wrote "%fp) $ relativeOutputFile+      return $ Right outputFile     else     do-      channelErrLn ch $ format ("No report output for '"%s%"' "%s) cmdLabel (repr exitCode)-      return (Left outputFile, result)+      channelErrLn ch $ format ("Did not write '"%fp%"' ("%s%") "%s) relativeOutputFile cmdLabel (repr exitCode)+      exists <- testfile outputFile+      if exists then rm outputFile else return ()+      return $ Left outputFile -journalFile :: ReportOptions -> [FilePath] -> FilePath+journalFile :: RuntimeOptions -> [FilePath] -> FilePath journalFile opts dirs = (foldl (</>) (baseDir opts) dirs) </> "all-years" <.> "journal" -outputReportDir :: ReportOptions -> [FilePath] -> FilePath+outputReportDir :: RuntimeOptions -> [FilePath] -> FilePath outputReportDir opts dirs = foldl (</>) (baseDir opts) ("reports":dirs) -ownerParameters :: ReportOptions -> TChan FlowTypes.LogMessage -> [FilePath] -> IO [ReportParams]+ownerParameters :: RuntimeOptions -> TChan FlowTypes.LogMessage -> [FilePath] -> IO [ReportParams] ownerParameters opts ch owners = do   let actions = map (ownerParameters' opts ch) owners   parAwareActions opts actions -ownerParameters' :: ReportOptions -> TChan FlowTypes.LogMessage -> FilePath -> IO ReportParams+ownerParameters' :: RuntimeOptions -> TChan FlowTypes.LogMessage -> FilePath -> IO ReportParams ownerParameters' opts ch owner = do   let ownerJournal = journalFile opts ["import", owner]   years <- includeYears ch ownerJournal

@@ -0,0 +1,25 @@+module Hledger.Flow.RuntimeOptions+where++import Turtle+import Prelude hiding (FilePath, putStrLn)+import Hledger.Flow.Types++data RuntimeOptions = RuntimeOptions { baseDir :: FilePath+                                     , hfVersion :: Text+                                     , hledgerInfo :: HledgerInfo+                                     , sysInfo :: SystemInfo+                                     , verbose :: Bool+                                     , showOptions :: Bool+                                     , sequential :: Bool+                                     }+  deriving (Show)++instance HasVerbosity RuntimeOptions where+  verbose (RuntimeOptions _ _ _ _ v _ _) = v++instance HasSequential RuntimeOptions where+  sequential (RuntimeOptions _ _ _ _ _ _ sq) = sq++instance HasBaseDir RuntimeOptions where+  baseDir (RuntimeOptions bd _ _ _ _ _ _) = bd

@@ -7,15 +7,13 @@ import Turtle import Prelude hiding (FilePath) import qualified Data.Map.Strict as Map-import qualified Control.Foldl as Fold-import qualified Data.Text as T import qualified Data.List as List (sort)  import TestHelpers-import Hledger.Flow.Import.Types import Hledger.Flow.Common import Control.Concurrent.STM +testExtraIncludesForFile :: Test testExtraIncludesForFile = TestCase (   sh (       do@@ -47,6 +45,7 @@         liftIO $ assertEqual "The closing journal should be included when it is on disk" [(accountInclude, [closing])] extraClosing2      )) +testIncludesPrePost :: Test testIncludesPrePost = TestCase (   sh (       do@@ -71,6 +70,7 @@         liftIO $ assertEqual "All pre/post files on disk should be included" expectedMap fileMap      )) +testIncludesOpeningClosing :: Test testIncludesOpeningClosing = TestCase (   sh (       do@@ -94,6 +94,7 @@         liftIO $ assertEqual "All pre/post files on disk should be included" expectedMap fileMap      )) +testWriteIncludeFiles :: Test testWriteIncludeFiles = TestCase (   sh (       do@@ -172,4 +173,5 @@      )   ) +tests :: Test tests = TestList [testExtraIncludesForFile, testIncludesPrePost, testIncludesOpeningClosing, testWriteIncludeFiles]

@@ -7,9 +7,6 @@ import Turtle import Prelude hiding (FilePath) import qualified Data.Map.Strict as Map-import qualified Control.Foldl as Fold-import qualified Data.Text as T-import qualified CSVImport.Integration import Control.Concurrent.STM  import TestHelpers@@ -68,6 +65,7 @@ groupedIncludeFiles = groupedJaneBogart <> groupedJaneOther <>                       groupedJohnBogart <> groupedJohnOther +testYearsIncludeMap :: Test testYearsIncludeMap = TestCase (   do     let maps = allYearIncludeFiles groupedJohnOther@@ -81,6 +79,7 @@     assertEqual "An augmented map with grouped years per level added" expected maps   ) +testYearsIncludeGrouping :: Test testYearsIncludeGrouping = TestCase (   do     let yearsMap = yearsIncludeMap (Map.keys groupedJohnOther)@@ -93,6 +92,7 @@     assertEqual "A basic map with grouped years per level" expected yearsMap   ) +testGroupIncludeFilesTinySet :: Test testGroupIncludeFilesTinySet = TestCase (   do     let journals1 = [   "import/jane/bogartbank/savings/3-journals/2017/2017-12-30.journal"]@@ -117,6 +117,7 @@     assertEqual "groupIncludeFiles small set 3" expected3 group3   ) +testGroupIncludeFilesSmallSet :: Test testGroupIncludeFilesSmallSet = TestCase (   do     let (group1, allYears1) = groupIncludeFiles (toJournals inputJaneBogart)@@ -193,6 +194,7 @@     assertEqual "groupIncludeFiles Jane 5" expectedGroup5 group5  ) +testGroupIncludeFiles :: Test testGroupIncludeFiles = TestCase (   do     let (group1, allYears1) = groupIncludeFiles journalFiles@@ -348,6 +350,7 @@     assertEqual "groupIncludeFiles 5" expectedGroup5 group5   ) +testRelativeToBase :: Test testRelativeToBase = TestCase (   do     let expected = "file1.journal"@@ -367,6 +370,7 @@     assertEqual "A basedir with no shared prefix should return the supplied file unchanged" "/unrelated/dir/file1.journal" mismatch   ) +testToIncludeLine :: Test testToIncludeLine = TestCase (   do     let expected = "!include file1.journal"@@ -383,6 +387,7 @@     assertEqual "Include line - absolute base dir without a trailing slash" expected absoluteNoTrailingSlash   ) +testToIncludeFiles :: Test testToIncludeFiles = TestCase (   do     let expected = [@@ -408,4 +413,5 @@     txt <- single $ toIncludeFiles (defaultOpts ".") ch groupedJohnBogart     assertEqual "Convert a grouped map of paths, to a map with text contents for each file" expected txt) +tests :: Test tests = TestList [testYearsIncludeMap, testYearsIncludeGrouping, testGroupIncludeFilesTinySet, testGroupIncludeFilesSmallSet, testGroupIncludeFiles, testRelativeToBase, testToIncludeLine, testToIncludeFiles]

@@ -6,16 +6,12 @@ import Test.HUnit import Turtle import Prelude hiding (FilePath)-import qualified Data.Map.Strict as Map-import qualified Control.Foldl as Fold-import qualified Data.Text as T import qualified Data.List as List (sort)  import TestHelpers-import Hledger.Flow.Import.Types import Hledger.Flow.Common-import Control.Concurrent.STM +testHiddenFiles :: Test testHiddenFiles = TestCase (   sh (       do@@ -31,6 +27,7 @@      )   ) +testDirOrPwd :: Test testDirOrPwd = TestCase (   sh (       do@@ -52,6 +49,7 @@      )   ) +testDetermineBaseDir :: Test testDetermineBaseDir = TestCase (   sh (       do@@ -113,6 +111,7 @@      )   ) +testFilterPaths :: Test testFilterPaths = TestCase (   sh (       do@@ -131,5 +130,5 @@      )   ) -+tests :: Test tests = TestList [testDirOrPwd, testDetermineBaseDir, testHiddenFiles, testFilterPaths]

@@ -7,20 +7,20 @@ import Turtle import Prelude hiding (FilePath) -import TestHelpers import Hledger.Flow.Common  import Data.Either import qualified Data.Text as T-import qualified Data.List as List +testShowCmdArgs :: Test testShowCmdArgs = TestCase (   do-    let options = ["--number", "/tmp/file with spaces"]+    let opts = ["--number", "/tmp/file with spaces"]     let expected = "--number '/tmp/file with spaces'"-    let actual = showCmdArgs options+    let actual = showCmdArgs opts     assertEqual "Convert command-line arguments to text" expected actual) +testIncludeYears :: Test testIncludeYears = TestCase (   do     let txterr = "Some text without years"@@ -47,6 +47,7 @@     assertEqual "Get a list of years from an include file - success case 2" expected2 actual2   ) +testExtractDigits :: Test testExtractDigits = TestCase (   do     let txt1 = "A number: 321\nAnother number is 42, so is 0"@@ -62,4 +63,5 @@     assertEqual "Extract digits from text 2" expected2 actual2   ) +tests :: Test tests = TestList [testShowCmdArgs, testIncludeYears, testExtractDigits]

@@ -12,11 +12,12 @@ import qualified CSVImport.Unit import qualified CSVImport.Integration +tests :: Test tests = TestList [Common.Unit.tests, Common.Integration.tests, CSVImport.Unit.tests, CSVImport.Integration.tests]  main :: IO Counts main = do-  counts <- runTestTT tests-  if (errors counts > 0 || failures counts > 0)+  errCounts <- runTestTT tests+  if (errors errCounts > 0 || failures errCounts > 0)     then exit $ ExitFailure 1-    else return counts+    else return errCounts

@@ -7,9 +7,10 @@ import Prelude hiding (FilePath)  import qualified Hledger.Flow.Types as FlowTypes-import Hledger.Flow.Import.Types import Hledger.Flow.Common+import Hledger.Flow.RuntimeOptions +inputJohnBogart :: [FilePath] inputJohnBogart = [   "import/john/bogartbank/savings/1-in/2017/2017-11-30.csv",   "import/john/bogartbank/savings/1-in/2017/2017-12-30.csv",@@ -19,37 +20,46 @@   "import/john/bogartbank/checking/1-in/2018/2018-10-30.csv",   "import/john/bogartbank/checking/1-in/2018/2018-12-30.csv",   "import/john/bogartbank/checking/1-in/2019/2019-01-30.csv",-  "import/john/bogartbank/checking/1-in/2019/2019-02-30.csv"] :: [FilePath]+  "import/john/bogartbank/checking/1-in/2019/2019-02-30.csv"] +inputJohnOther :: [FilePath] inputJohnOther = [   "import/john/otherbank/creditcard/1-in/2017/2017-12-30.csv",   "import/john/otherbank/creditcard/1-in/2018/2018-01-30.csv",   "import/john/otherbank/investments/1-in/2018/2018-12-30.csv",-  "import/john/otherbank/investments/1-in/2019/2019-01-30.csv"] :: [FilePath]+  "import/john/otherbank/investments/1-in/2019/2019-01-30.csv"] +inputJaneBogart :: [FilePath] inputJaneBogart = [   "import/jane/bogartbank/savings/1-in/2017/2017-12-30.csv",   "import/jane/bogartbank/savings/1-in/2018/2018-01-30.csv",   "import/jane/bogartbank/checking/1-in/2018/2018-12-30.csv",-  "import/jane/bogartbank/checking/1-in/2019/2019-01-30.csv"] :: [FilePath]+  "import/jane/bogartbank/checking/1-in/2019/2019-01-30.csv"] +inputJaneOther :: [FilePath] inputJaneOther = [   "import/jane/otherbank/creditcard/1-in/2017/2017-12-30.csv",   "import/jane/otherbank/creditcard/1-in/2018/2018-01-30.csv",   "import/jane/otherbank/investments/1-in/2018/2018-12-30.csv",-  "import/jane/otherbank/investments/1-in/2019/2019-01-30.csv"] :: [FilePath]+  "import/jane/otherbank/investments/1-in/2019/2019-01-30.csv"] +inputFiles :: [FilePath] inputFiles = inputJohnBogart <> inputJohnOther <> inputJaneBogart <> inputJaneOther -journalFiles = toJournals inputFiles :: [FilePath]-extraFiles = ["import/john/bogartbank/savings/2017-opening.journal"] :: [FilePath]-hiddenFiles = [".hiddenfile", "checking/.DS_Store", "import/john/bogartbank/savings/1-in/.anotherhiddenfile", "import/john/bogartbank/checking/1-in/2018/.hidden"] :: [FilePath]+journalFiles :: [FilePath]+journalFiles = toJournals inputFiles +extraFiles :: [FilePath]+extraFiles = ["import/john/bogartbank/savings/2017-opening.journal"]++hiddenFiles :: [FilePath]+hiddenFiles = [".hiddenfile", "checking/.DS_Store", "import/john/bogartbank/savings/1-in/.anotherhiddenfile", "import/john/bogartbank/checking/1-in/2018/.hidden"]+ defaultHlInfo :: FlowTypes.HledgerInfo defaultHlInfo = FlowTypes.HledgerInfo "/path/to/hledger" "1.2.3" -defaultOpts :: FilePath -> ImportOptions-defaultOpts bd = ImportOptions bd versionInfo' defaultHlInfo systemInfo False False False+defaultOpts :: FilePath -> RuntimeOptions+defaultOpts bd = RuntimeOptions bd versionInfo' defaultHlInfo systemInfo False False False  toJournals :: [FilePath] -> [FilePath] toJournals = map (changePathAndExtension "3-journal" "journal")