diff --git a/ChangeLog.md b/ChangeLog.md --- a/ChangeLog.md +++ b/ChangeLog.md @@ -1,5 +1,13 @@ # Changelog for [hledger-flow](https://github.com/apauley/hledger-flow) +## 0.12.3 + +Add more reports: + +- Balance Sheet per owner per year, and for all owners per year +- Unknown transactions per owner per year +- A transfer balance overview per year + ## 0.12.2.1 Fix resolver extraction and hledger-flow --version in release-tarball script diff --git a/README.org b/README.org --- a/README.org +++ b/README.org @@ -1,5 +1,4 @@ #+STARTUP: showall -#+PROPERTY: header-args:sh :prologue exec 2>&1 :epilogue echo : * Hledger Flow :PROPERTIES: @@ -20,8 +19,8 @@ processing of electronic statements as much as possible, as opposed to manually adding your own hledger journal entries. -Manual entries are still possible, we just think it saves time in the -long run to automatically process a statement whenever one is available. +Manual entries are still possible, it just saves time in the long run to +automatically process a statement whenever one is available. Within =hledger-flow= you will keep your original bank statements around permanently as input, and generate classified Hledger journals each time @@ -38,17 +37,6 @@ with the ideas in [[https://github.com/adept/full-fledged-hledger/wiki][adept's Full-fledged Hledger]] aren't really specific to my own finances, and can be shared. -* How do I install it? - :PROPERTIES: - :CUSTOM_ID: how-do-i-install-it - :END: - -The easiest way to get it running is to download [[https://github.com/apauley/hledger-flow/releases][the latest release]] -for your OS, and copy the =hledger-flow= executable to a directory in your PATH. -Then just run it and see what it tells you to do. - -You can also compile it yourself by following the [[https://github.com/apauley/hledger-flow/blob/master/CONTRIBUTING.org#build-the-project][build instructions]]. - * Overview of the Basic Workflow :PROPERTIES: :CUSTOM_ID: overview-of-the-basic-workflow @@ -94,6 +82,17 @@ - You appreciate the fact that all your financial information stays within your control. +* How do I install it? + :PROPERTIES: + :CUSTOM_ID: how-do-i-install-it + :END: + +The easiest way to get it running is to download [[https://github.com/apauley/hledger-flow/releases][the latest release]] +for your OS, and copy the =hledger-flow= executable to a directory in your PATH. +Then just run it and see what it tells you to do. + +You can also compile it yourself by following the [[https://github.com/apauley/hledger-flow/blob/master/CONTRIBUTING.org#build-the-project][build instructions]]. + * How Stable is it? :PROPERTIES: :CUSTOM_ID: how-stable-is-it @@ -612,14 +611,91 @@ :CUSTOM_ID: faq :END: +** How do you balance transfers between 2 accounts when you have statements for both accounts? + :PROPERTIES: + :CUSTOM_ID: transfer-2-accounts + :END: + +*** The Problem + +In your primary bank account you've happily been classifying transfers to a +secondary account as just =Expenses:OtherAccount=. + +But you've recently started processing the statements from the second account as +well so that you can classify those expenses more accurately. + +And now the balances of these two accounts are all wrong when the statements of +each account deals with money transferred between these two accounts. + +In =bank1.journal=, imported from =bank1.csv=: +#+BEGIN_EXAMPLE +2018/11/09 Transfer from primary account to secondary account + Assets:Bank1:Primary $-200 + Assets:Bank2:Secondary +#+END_EXAMPLE + +In =bank2.journal=, imported from =bank2.csv=: +#+BEGIN_EXAMPLE +2018/11/09 Transfer from primary account to secondary account + Assets:Bank2:Secondary $200 + Assets:Bank1:Primary +#+END_EXAMPLE + +*** The Solution + +As soon as you start importing statements for both accounts you will have to +introduce an intermediate account for classification between these two accounts. + +I use =Assets:Transfers:*=. + +And we may have reports looking at these transfers accounts at some point, you +should consider using the same names. + +The above example then becomes as follows. + +In =bank1.journal=, imported from =bank1.csv=: +#+BEGIN_EXAMPLE +2019-05-18 Transfer from primary account to secondary account + Assets:Bank1:Primary $-200 + Assets:Transfers:Bank1Bank2 +#+END_EXAMPLE + +In =bank2.journal=, imported from =bank2.csv=: +#+BEGIN_EXAMPLE +2019-05-18 Transfer from primary account to secondary account + Assets:Bank2:Secondary $200 + Assets:Transfers:Bank1Bank2 +#+END_EXAMPLE + +Any posting to =Assets:Transfers:*= indicates an in "in-flight" amount. +You would expect the balance of =Assets:Transfers= to be zero most of the time. +Whenever it isn't zero it means that you either don't yet have the other side of +the transfer, or that something is wrong in your rules. + +You could theoretically just use =Assets:Transfers= without any subaccounts, but +I found it useful to use subaccounts. Because then the subaccounts can show me +where I should look for any missing transfer transaction. + +I typically use sorted names as the subaccount (Python code sample): + +#+BEGIN_SRC python + "Assets:Transfers:" + "".join(sorted(["Bank2", "Bank1"])) +#+END_SRC + +*** External references + + This approach is based on what is described in Full-fledged hledger: + [[https://github.com/adept/full-fledged-hledger/wiki/Adding-more-accounts#lets-make-sure-that-transfers-are-not-double-counted]] + + The question was first asked in [[https://github.com/apauley/hledger-flow/issues/51][issue #51]]. + ** How does =hledger-flow= differ from =Full-fledged Hledger=? :PROPERTIES: :CUSTOM_ID: how-does-hledger-flow-differ-from-full-fledged-hledger :END: -[[https://github.com/adept/full-fledged-hledger/wiki#full-fledged-hledger-tutorial][Full-fledged -Hledger]] is a brilliant system, and hledger-flow continues to learn -much from it. +[[https://github.com/adept/full-fledged-hledger/wiki#full-fledged-hledger-tutorial][Full-fledged Hledger]] +is a brilliant system, and hledger-flow continues to learn much from it. It has great documentation that does an excellent job of not only showing *how* things can be done, but also *why* it is such a great @@ -637,7 +713,7 @@ | | | | Maintaining include files are currently part of the user's responsibility. | Hledger Flow generates flexible include files for you, and automatically includes opening/closing journals if the appropriately named files are present on disk. | | | | -| FFH actually generates some useful reports right now. Hledger Flow still [[https://github.com/apauley/hledger-flow/pull/4][plans to get this done]] one day. | The "owner/bank/account" structure may look a bit much at first, but it allows us to run separate queries/reports for me/my spouse/my business etc and also allows for reports covering all of it in a an overall view. | +| FFH generates some of the same reports as Hledger Flow. | The "owner/bank/account" structure may look a bit much at first, but it allows us to run separate queries/reports for "me" vs "my spouse" vs "my business" etc and also allows for reports covering all of it in a an overall view. | | | | | FFH chose scripts and build files that you can easily modify as you go along, but this requires a Haskell runtime to be installed everywhere it needs to run. The included docker image helps to make it less of an issue. | Hledger Flow distributes a compiled binary. This means users or deployment targets don't need extra dependencies installed, they can just run a CLI program. This also provides a clearer distinction between what is provided, and what users need to do. | | | | diff --git a/app/Main.hs b/app/Main.hs --- a/app/Main.hs +++ b/app/Main.hs @@ -8,9 +8,8 @@ import Options.Applicative import Data.Semigroup ((<>)) -import qualified Hledger.Flow.Import.Types as IT -import qualified Hledger.Flow.Report.Types as RT import Hledger.Flow.Common +import qualified Hledger.Flow.RuntimeOptions as RT import Hledger.Flow.Reports import Hledger.Flow.CSVImport @@ -29,32 +28,20 @@ cmd <- options "An hledger workflow focusing on automated statement import and classification:\nhttps://github.com/apauley/hledger-flow#readme" baseCommandParser case cmd of Version -> stdout $ select versionInfo - Command mainParams' (Import subParams) -> toImportOptions mainParams' subParams >>= importCSVs - Command mainParams' (Report subParams) -> toReportOptions mainParams' subParams >>= generateReports - -toImportOptions :: MainParams -> SubcommandParams -> IO IT.ImportOptions -toImportOptions mainParams' subParams' = do - bd <- determineBaseDir $ maybeBaseDir subParams' - hli <- hledgerInfoFromPath $ hledgerPathOpt mainParams' - return IT.ImportOptions { IT.baseDir = bd - , IT.hfVersion = versionInfo' - , IT.hledgerInfo = hli - , IT.sysInfo = systemInfo - , IT.verbose = verbose mainParams' - , IT.showOptions = showOpts mainParams' - , IT.sequential = sequential mainParams' } + Command mainParams' (Import subParams) -> toRuntimeOptions mainParams' subParams >>= importCSVs + Command mainParams' (Report subParams) -> toRuntimeOptions mainParams' subParams >>= generateReports -toReportOptions :: MainParams -> SubcommandParams -> IO RT.ReportOptions -toReportOptions mainParams' subParams' = do +toRuntimeOptions :: MainParams -> SubcommandParams -> IO RT.RuntimeOptions +toRuntimeOptions mainParams' subParams' = do bd <- determineBaseDir $ maybeBaseDir subParams' hli <- hledgerInfoFromPath $ hledgerPathOpt mainParams' - return RT.ReportOptions { RT.baseDir = bd - , RT.hfVersion = versionInfo' - , RT.hledgerInfo = hli - , RT.sysInfo = systemInfo - , RT.verbose = verbose mainParams' - , RT.showOptions = showOpts mainParams' - , RT.sequential = sequential mainParams' } + return RT.RuntimeOptions { RT.baseDir = bd + , RT.hfVersion = versionInfo' + , RT.hledgerInfo = hli + , RT.sysInfo = systemInfo + , RT.verbose = verbose mainParams' + , RT.showOptions = showOpts mainParams' + , RT.sequential = sequential mainParams' } baseCommandParser :: Parser BaseCommand baseCommandParser = (Command <$> verboseParser <*> commandParser) diff --git a/hledger-flow.cabal b/hledger-flow.cabal --- a/hledger-flow.cabal +++ b/hledger-flow.cabal @@ -4,10 +4,10 @@ -- -- see: https://github.com/sol/hpack -- --- hash: 88668a61a38818c6bcecc7aa0b1e04eda0c03c931ff0a2b9dedcd23cd2bd646d +-- hash: 238554ba5f9f4fbeff3effd7709b051f2e6d5e4386d56f57aa1c7de2241a6727 name: hledger-flow -version: 0.12.2.1 +version: 0.12.3.0 synopsis: An hledger workflow focusing on automated statement import and classification. description: Please see the README on GitHub at category: Finance, Console @@ -32,8 +32,8 @@ Hledger.Flow.Common Hledger.Flow.CSVImport Hledger.Flow.Import.Types - Hledger.Flow.Report.Types Hledger.Flow.Reports + Hledger.Flow.RuntimeOptions Hledger.Flow.Types other-modules: Paths_hledger_flow diff --git a/src/Hledger/Flow/CSVImport.hs b/src/Hledger/Flow/CSVImport.hs --- a/src/Hledger/Flow/CSVImport.hs +++ b/src/Hledger/Flow/CSVImport.hs @@ -11,9 +11,10 @@ import qualified Hledger.Flow.Types as FlowTypes import Hledger.Flow.Import.Types import Hledger.Flow.Common +import Hledger.Flow.RuntimeOptions import Control.Concurrent.STM -importCSVs :: ImportOptions -> IO () +importCSVs :: RuntimeOptions -> IO () importCSVs opts = sh ( do ch <- liftIO newTChanIO @@ -32,7 +33,7 @@ inputFilePattern :: Pattern Text inputFilePattern = contains (once (oneOf pathSeparators) <> asciiCI "1-in" <> once (oneOf pathSeparators) <> plus digit <> once (oneOf pathSeparators)) -importCSVs' :: ImportOptions -> TChan FlowTypes.LogMessage -> IO [FilePath] +importCSVs' :: RuntimeOptions -> TChan FlowTypes.LogMessage -> IO [FilePath] importCSVs' opts ch = do channelOutLn ch "Collecting input files..." (inputFiles, diff) <- time $ single . shellToList . onlyFiles $ find inputFilePattern $ baseDir opts @@ -51,14 +52,14 @@ sh $ writeIncludesUpTo opts ch "import" importedJournals return importedJournals -extractAndImport :: ImportOptions -> TChan FlowTypes.LogMessage -> FilePath -> IO FilePath +extractAndImport :: RuntimeOptions -> TChan FlowTypes.LogMessage -> FilePath -> IO FilePath extractAndImport opts ch inputFile = do case extractImportDirs inputFile of Right importDirs -> importCSV opts ch importDirs inputFile Left errorMessage -> do errExit 1 ch errorMessage inputFile -importCSV :: ImportOptions -> TChan FlowTypes.LogMessage -> ImportDirs -> FilePath -> IO FilePath +importCSV :: RuntimeOptions -> TChan FlowTypes.LogMessage -> ImportDirs -> FilePath -> IO FilePath importCSV opts ch importDirs srcFile = do let preprocessScript = accountDir importDirs "preprocess" let constructScript = accountDir importDirs "construct" @@ -74,14 +75,14 @@ mktree $ directory journalOut importFun csvFile journalOut -preprocessIfNeeded :: ImportOptions -> TChan FlowTypes.LogMessage -> FilePath -> Line -> Line -> Line -> FilePath -> IO FilePath +preprocessIfNeeded :: RuntimeOptions -> TChan FlowTypes.LogMessage -> FilePath -> Line -> Line -> Line -> FilePath -> IO FilePath preprocessIfNeeded opts ch script bank account owner src = do shouldPreprocess <- verboseTestFile opts ch script if shouldPreprocess then preprocess opts ch script bank account owner src else return src -preprocess :: ImportOptions -> TChan FlowTypes.LogMessage -> FilePath -> Line -> Line -> Line -> FilePath -> IO FilePath +preprocess :: RuntimeOptions -> TChan FlowTypes.LogMessage -> FilePath -> Line -> Line -> Line -> FilePath -> IO FilePath preprocess opts ch script bank account owner src = do let csvOut = changePathAndExtension "2-preprocessed" "csv" src mktree $ directory csvOut @@ -92,14 +93,14 @@ _ <- timeAndExitOnErr opts ch cmdLabel channelOut channelErr (parAwareProc opts) (format fp script, args, empty) return csvOut -hledgerImport :: ImportOptions -> TChan FlowTypes.LogMessage -> FilePath -> FilePath -> IO FilePath +hledgerImport :: RuntimeOptions -> TChan FlowTypes.LogMessage -> FilePath -> FilePath -> IO FilePath hledgerImport opts ch csvSrc journalOut = do case extractImportDirs csvSrc of Right importDirs -> hledgerImport' opts ch importDirs csvSrc journalOut Left errorMessage -> do errExit 1 ch errorMessage csvSrc -hledgerImport' :: ImportOptions -> TChan FlowTypes.LogMessage -> ImportDirs -> FilePath -> FilePath -> IO FilePath +hledgerImport' :: RuntimeOptions -> TChan FlowTypes.LogMessage -> ImportDirs -> FilePath -> FilePath -> IO FilePath hledgerImport' opts ch importDirs csvSrc journalOut = do let candidates = rulesFileCandidates csvSrc importDirs maybeRulesFile <- firstExistingFile candidates @@ -151,7 +152,7 @@ map ( srcSpecificFilename) [accountDir importDirs, bankDir importDirs, importDir importDirs] else [] -customConstruct :: ImportOptions -> TChan FlowTypes.LogMessage -> FilePath -> Line -> Line -> Line -> FilePath -> FilePath -> IO FilePath +customConstruct :: RuntimeOptions -> TChan FlowTypes.LogMessage -> FilePath -> Line -> Line -> Line -> FilePath -> FilePath -> IO FilePath customConstruct opts ch constructScript bank account owner csvSrc journalOut = do let script = format fp constructScript :: Text let relScript = relativeToBase opts constructScript diff --git a/src/Hledger/Flow/Import/Types.hs b/src/Hledger/Flow/Import/Types.hs --- a/src/Hledger/Flow/Import/Types.hs +++ b/src/Hledger/Flow/Import/Types.hs @@ -3,25 +3,6 @@ import Turtle import Prelude hiding (FilePath, putStrLn) -import Hledger.Flow.Types - -data ImportOptions = ImportOptions { baseDir :: FilePath - , hfVersion :: Text - , hledgerInfo :: HledgerInfo - , sysInfo :: SystemInfo - , verbose :: Bool - , showOptions :: Bool - , sequential :: Bool } - deriving (Show) - -instance HasVerbosity ImportOptions where - verbose (ImportOptions _ _ _ _ v _ _) = v - -instance HasSequential ImportOptions where - sequential (ImportOptions _ _ _ _ _ _ sq) = sq - -instance HasBaseDir ImportOptions where - baseDir (ImportOptions bd _ _ _ _ _ _) = bd data ImportDirs = ImportDirs { importDir :: FilePath , ownerDir :: FilePath diff --git a/src/Hledger/Flow/Report/Types.hs b/src/Hledger/Flow/Report/Types.hs deleted file mode 100644 --- a/src/Hledger/Flow/Report/Types.hs +++ /dev/null @@ -1,25 +0,0 @@ -module Hledger.Flow.Report.Types -where - -import Turtle -import Prelude hiding (FilePath, putStrLn) -import Hledger.Flow.Types - -data ReportOptions = ReportOptions { baseDir :: FilePath - , hfVersion :: Text - , hledgerInfo :: HledgerInfo - , sysInfo :: SystemInfo - , verbose :: Bool - , showOptions :: Bool - , sequential :: Bool - } - deriving (Show) - -instance HasVerbosity ReportOptions where - verbose (ReportOptions _ _ _ _ v _ _) = v - -instance HasSequential ReportOptions where - sequential (ReportOptions _ _ _ _ _ _ sq) = sq - -instance HasBaseDir ReportOptions where - baseDir (ReportOptions bd _ _ _ _ _ _) = bd diff --git a/src/Hledger/Flow/Reports.hs b/src/Hledger/Flow/Reports.hs --- a/src/Hledger/Flow/Reports.hs +++ b/src/Hledger/Flow/Reports.hs @@ -6,7 +6,7 @@ import Turtle hiding (stdout, stderr, proc) import Prelude hiding (FilePath, putStrLn, writeFile) -import Hledger.Flow.Report.Types +import Hledger.Flow.RuntimeOptions import Hledger.Flow.Common import Control.Concurrent.STM import Data.Either @@ -20,8 +20,9 @@ , outputDir :: FilePath } deriving (Show) +type ReportGenerator = RuntimeOptions -> TChan FlowTypes.LogMessage -> FilePath -> FilePath -> Integer -> IO (Either FilePath FilePath) -generateReports :: ReportOptions -> IO () +generateReports :: RuntimeOptions -> IO () generateReports opts = sh ( do ch <- liftIO newTChanIO @@ -35,7 +36,7 @@ wait logHandle ) -generateReports' :: ReportOptions -> TChan FlowTypes.LogMessage -> IO [Either FilePath FilePath] +generateReports' :: RuntimeOptions -> TChan FlowTypes.LogMessage -> IO [Either FilePath FilePath] generateReports' opts ch = do let wipMsg = "Report generation is still a work-in-progress - please let me know how this can be more useful.\n\n" <> "Keep an eye out for report-related pull requests and issues, and feel free to submit some of your own:\n" @@ -43,66 +44,85 @@ <> "https://github.com/apauley/hledger-flow/issues\n" channelOutLn ch wipMsg owners <- single $ shellToList $ listOwners opts - let baseJournal = journalFile opts [] - let baseReportDir = outputReportDir opts ["all"] - baseYears <- includeYears ch baseJournal - let baseParams = if length owners > 1 then [(ReportParams baseJournal baseYears baseReportDir)] else [] + let aggregateJournal = journalFile opts [] + let aggregateReportDir = outputReportDir opts ["all"] + aggregateYears <- includeYears ch aggregateJournal + let aggregateParams = ReportParams aggregateJournal aggregateYears aggregateReportDir + let aggregateOnlyReports = reportActions opts ch [transferBalance] aggregateParams ownerParams <- ownerParameters opts ch owners - let reportParams = baseParams ++ ownerParams - let actions = List.concat $ fmap (generateReports'' opts ch) reportParams - parAwareActions opts actions + let ownerWithAggregateParams = (if length owners > 1 then [aggregateParams] else []) ++ ownerParams + let sharedOptions = ["--depth", "2", "--pretty-tables", "not:equity"] + let ownerWithAggregateReports = List.concat $ fmap (reportActions opts ch [incomeStatement sharedOptions, balanceSheet sharedOptions]) ownerWithAggregateParams + let ownerOnlyReports = List.concat $ fmap (reportActions opts ch [accountList, unknownTransactions]) ownerParams + parAwareActions opts (aggregateOnlyReports ++ ownerWithAggregateReports ++ ownerOnlyReports) -generateReports'' :: ReportOptions -> TChan FlowTypes.LogMessage -> ReportParams -> [IO (Either FilePath FilePath)] -generateReports'' opts ch (ReportParams journal years reportsDir) = do +reportActions :: RuntimeOptions -> TChan FlowTypes.LogMessage -> [ReportGenerator] -> ReportParams -> [IO (Either FilePath FilePath)] +reportActions opts ch reports (ReportParams journal years reportsDir) = do y <- years - let actions = map (\r -> r opts ch journal reportsDir y) [accountList, incomeStatement] - map (fmap fst) actions - -incomeStatement :: ReportOptions -> TChan FlowTypes.LogMessage -> FilePath -> FilePath -> Integer -> IO (Either FilePath FilePath, FlowTypes.FullTimedOutput) -incomeStatement opts ch journal reportsDir year = do - let sharedOptions = ["--depth", "2", "--pretty-tables", "not:equity", "--cost", "--value"] - let reportArgs = ["incomestatement"] ++ sharedOptions - generateReport opts ch journal reportsDir year ("income-expenses" <.> "txt") reportArgs + map (\r -> r opts ch journal reportsDir y) reports -accountList :: ReportOptions -> TChan FlowTypes.LogMessage -> FilePath -> FilePath -> Integer -> IO (Either FilePath FilePath, FlowTypes.FullTimedOutput) +accountList :: ReportGenerator accountList opts ch journal reportsDir year = do let reportArgs = ["accounts"] - generateReport opts ch journal reportsDir year ("accounts" <.> "txt") reportArgs + generateReport opts ch journal reportsDir year ("accounts" <.> "txt") reportArgs (not . T.null) -generateReport :: ReportOptions -> TChan FlowTypes.LogMessage -> FilePath -> FilePath -> Integer -> FilePath -> [Text] -> IO (Either FilePath FilePath, FlowTypes.FullTimedOutput) -generateReport opts ch journal baseOutDir year fileName args = do +unknownTransactions :: ReportGenerator +unknownTransactions opts ch journal reportsDir year = do + let reportArgs = ["print", "unknown"] + generateReport opts ch journal reportsDir year ("unknown-transactions" <.> "txt") reportArgs (not . T.null) + +incomeStatement :: [Text] -> ReportGenerator +incomeStatement sharedOptions opts ch journal reportsDir year = do + let reportArgs = ["incomestatement"] ++ sharedOptions ++ ["--cost", "--value"] + generateReport opts ch journal reportsDir year ("income-expenses" <.> "txt") reportArgs (not . T.null) + +balanceSheet :: [Text] -> ReportGenerator +balanceSheet sharedOptions opts ch journal reportsDir year = do + let reportArgs = ["balancesheet"] ++ sharedOptions ++ ["--cost", "--flat"] + generateReport opts ch journal reportsDir year ("balance-sheet" <.> "txt") reportArgs (not . T.null) + +transferBalance :: ReportGenerator +transferBalance opts ch journal reportsDir year = do + let reportArgs = ["balance", "--pretty-tables", "--quarterly", "--flat", "--no-total", "transfer"] + generateReport opts ch journal reportsDir year ("transfer-balance" <.> "txt") reportArgs (\txt -> (length $ T.lines txt) > 4) + +generateReport :: RuntimeOptions -> TChan FlowTypes.LogMessage -> FilePath -> FilePath -> Integer -> FilePath -> [Text] -> (Text -> Bool) -> IO (Either FilePath FilePath) +generateReport opts ch journal baseOutDir year fileName args successCheck = do let reportsDir = baseOutDir intPath year mktree reportsDir let outputFile = reportsDir fileName let relativeJournal = relativeToBase opts journal + let relativeOutputFile = relativeToBase opts outputFile let reportArgs = ["--file", format fp journal, "--period", repr year] ++ args let reportDisplayArgs = ["--file", format fp relativeJournal, "--period", repr year] ++ args let hledger = format fp $ FlowTypes.hlPath . hledgerInfo $ opts :: Text let cmdLabel = format ("hledger "%s) $ showCmdArgs reportDisplayArgs - result@((exitCode, stdOut, _), _) <- timeAndExitOnErr opts ch cmdLabel dummyLogger channelErr procStrictWithErr (hledger, reportArgs, empty) - if not (T.null stdOut) + ((exitCode, stdOut, _), _) <- timeAndExitOnErr opts ch cmdLabel dummyLogger channelErr procStrictWithErr (hledger, reportArgs, empty) + if (successCheck stdOut) then do writeTextFile outputFile (cmdLabel <> "\n\n"<> stdOut) - logVerbose opts ch $ format ("Wrote "%fp) $ relativeToBase opts outputFile - return (Right outputFile, result) + logVerbose opts ch $ format ("Wrote "%fp) $ relativeOutputFile + return $ Right outputFile else do - channelErrLn ch $ format ("No report output for '"%s%"' "%s) cmdLabel (repr exitCode) - return (Left outputFile, result) + channelErrLn ch $ format ("Did not write '"%fp%"' ("%s%") "%s) relativeOutputFile cmdLabel (repr exitCode) + exists <- testfile outputFile + if exists then rm outputFile else return () + return $ Left outputFile -journalFile :: ReportOptions -> [FilePath] -> FilePath +journalFile :: RuntimeOptions -> [FilePath] -> FilePath journalFile opts dirs = (foldl () (baseDir opts) dirs) "all-years" <.> "journal" -outputReportDir :: ReportOptions -> [FilePath] -> FilePath +outputReportDir :: RuntimeOptions -> [FilePath] -> FilePath outputReportDir opts dirs = foldl () (baseDir opts) ("reports":dirs) -ownerParameters :: ReportOptions -> TChan FlowTypes.LogMessage -> [FilePath] -> IO [ReportParams] +ownerParameters :: RuntimeOptions -> TChan FlowTypes.LogMessage -> [FilePath] -> IO [ReportParams] ownerParameters opts ch owners = do let actions = map (ownerParameters' opts ch) owners parAwareActions opts actions -ownerParameters' :: ReportOptions -> TChan FlowTypes.LogMessage -> FilePath -> IO ReportParams +ownerParameters' :: RuntimeOptions -> TChan FlowTypes.LogMessage -> FilePath -> IO ReportParams ownerParameters' opts ch owner = do let ownerJournal = journalFile opts ["import", owner] years <- includeYears ch ownerJournal diff --git a/src/Hledger/Flow/RuntimeOptions.hs b/src/Hledger/Flow/RuntimeOptions.hs new file mode 100644 --- /dev/null +++ b/src/Hledger/Flow/RuntimeOptions.hs @@ -0,0 +1,25 @@ +module Hledger.Flow.RuntimeOptions +where + +import Turtle +import Prelude hiding (FilePath, putStrLn) +import Hledger.Flow.Types + +data RuntimeOptions = RuntimeOptions { baseDir :: FilePath + , hfVersion :: Text + , hledgerInfo :: HledgerInfo + , sysInfo :: SystemInfo + , verbose :: Bool + , showOptions :: Bool + , sequential :: Bool + } + deriving (Show) + +instance HasVerbosity RuntimeOptions where + verbose (RuntimeOptions _ _ _ _ v _ _) = v + +instance HasSequential RuntimeOptions where + sequential (RuntimeOptions _ _ _ _ _ _ sq) = sq + +instance HasBaseDir RuntimeOptions where + baseDir (RuntimeOptions bd _ _ _ _ _ _) = bd diff --git a/test/CSVImport/Integration.hs b/test/CSVImport/Integration.hs --- a/test/CSVImport/Integration.hs +++ b/test/CSVImport/Integration.hs @@ -7,15 +7,13 @@ import Turtle import Prelude hiding (FilePath) import qualified Data.Map.Strict as Map -import qualified Control.Foldl as Fold -import qualified Data.Text as T import qualified Data.List as List (sort) import TestHelpers -import Hledger.Flow.Import.Types import Hledger.Flow.Common import Control.Concurrent.STM +testExtraIncludesForFile :: Test testExtraIncludesForFile = TestCase ( sh ( do @@ -47,6 +45,7 @@ liftIO $ assertEqual "The closing journal should be included when it is on disk" [(accountInclude, [closing])] extraClosing2 )) +testIncludesPrePost :: Test testIncludesPrePost = TestCase ( sh ( do @@ -71,6 +70,7 @@ liftIO $ assertEqual "All pre/post files on disk should be included" expectedMap fileMap )) +testIncludesOpeningClosing :: Test testIncludesOpeningClosing = TestCase ( sh ( do @@ -94,6 +94,7 @@ liftIO $ assertEqual "All pre/post files on disk should be included" expectedMap fileMap )) +testWriteIncludeFiles :: Test testWriteIncludeFiles = TestCase ( sh ( do @@ -172,4 +173,5 @@ ) ) +tests :: Test tests = TestList [testExtraIncludesForFile, testIncludesPrePost, testIncludesOpeningClosing, testWriteIncludeFiles] diff --git a/test/CSVImport/Unit.hs b/test/CSVImport/Unit.hs --- a/test/CSVImport/Unit.hs +++ b/test/CSVImport/Unit.hs @@ -7,9 +7,6 @@ import Turtle import Prelude hiding (FilePath) import qualified Data.Map.Strict as Map -import qualified Control.Foldl as Fold -import qualified Data.Text as T -import qualified CSVImport.Integration import Control.Concurrent.STM import TestHelpers @@ -68,6 +65,7 @@ groupedIncludeFiles = groupedJaneBogart <> groupedJaneOther <> groupedJohnBogart <> groupedJohnOther +testYearsIncludeMap :: Test testYearsIncludeMap = TestCase ( do let maps = allYearIncludeFiles groupedJohnOther @@ -81,6 +79,7 @@ assertEqual "An augmented map with grouped years per level added" expected maps ) +testYearsIncludeGrouping :: Test testYearsIncludeGrouping = TestCase ( do let yearsMap = yearsIncludeMap (Map.keys groupedJohnOther) @@ -93,6 +92,7 @@ assertEqual "A basic map with grouped years per level" expected yearsMap ) +testGroupIncludeFilesTinySet :: Test testGroupIncludeFilesTinySet = TestCase ( do let journals1 = [ "import/jane/bogartbank/savings/3-journals/2017/2017-12-30.journal"] @@ -117,6 +117,7 @@ assertEqual "groupIncludeFiles small set 3" expected3 group3 ) +testGroupIncludeFilesSmallSet :: Test testGroupIncludeFilesSmallSet = TestCase ( do let (group1, allYears1) = groupIncludeFiles (toJournals inputJaneBogart) @@ -193,6 +194,7 @@ assertEqual "groupIncludeFiles Jane 5" expectedGroup5 group5 ) +testGroupIncludeFiles :: Test testGroupIncludeFiles = TestCase ( do let (group1, allYears1) = groupIncludeFiles journalFiles @@ -348,6 +350,7 @@ assertEqual "groupIncludeFiles 5" expectedGroup5 group5 ) +testRelativeToBase :: Test testRelativeToBase = TestCase ( do let expected = "file1.journal" @@ -367,6 +370,7 @@ assertEqual "A basedir with no shared prefix should return the supplied file unchanged" "/unrelated/dir/file1.journal" mismatch ) +testToIncludeLine :: Test testToIncludeLine = TestCase ( do let expected = "!include file1.journal" @@ -383,6 +387,7 @@ assertEqual "Include line - absolute base dir without a trailing slash" expected absoluteNoTrailingSlash ) +testToIncludeFiles :: Test testToIncludeFiles = TestCase ( do let expected = [ @@ -408,4 +413,5 @@ txt <- single $ toIncludeFiles (defaultOpts ".") ch groupedJohnBogart assertEqual "Convert a grouped map of paths, to a map with text contents for each file" expected txt) +tests :: Test tests = TestList [testYearsIncludeMap, testYearsIncludeGrouping, testGroupIncludeFilesTinySet, testGroupIncludeFilesSmallSet, testGroupIncludeFiles, testRelativeToBase, testToIncludeLine, testToIncludeFiles] diff --git a/test/Common/Integration.hs b/test/Common/Integration.hs --- a/test/Common/Integration.hs +++ b/test/Common/Integration.hs @@ -6,16 +6,12 @@ import Test.HUnit import Turtle import Prelude hiding (FilePath) -import qualified Data.Map.Strict as Map -import qualified Control.Foldl as Fold -import qualified Data.Text as T import qualified Data.List as List (sort) import TestHelpers -import Hledger.Flow.Import.Types import Hledger.Flow.Common -import Control.Concurrent.STM +testHiddenFiles :: Test testHiddenFiles = TestCase ( sh ( do @@ -31,6 +27,7 @@ ) ) +testDirOrPwd :: Test testDirOrPwd = TestCase ( sh ( do @@ -52,6 +49,7 @@ ) ) +testDetermineBaseDir :: Test testDetermineBaseDir = TestCase ( sh ( do @@ -113,6 +111,7 @@ ) ) +testFilterPaths :: Test testFilterPaths = TestCase ( sh ( do @@ -131,5 +130,5 @@ ) ) - +tests :: Test tests = TestList [testDirOrPwd, testDetermineBaseDir, testHiddenFiles, testFilterPaths] diff --git a/test/Common/Unit.hs b/test/Common/Unit.hs --- a/test/Common/Unit.hs +++ b/test/Common/Unit.hs @@ -7,20 +7,20 @@ import Turtle import Prelude hiding (FilePath) -import TestHelpers import Hledger.Flow.Common import Data.Either import qualified Data.Text as T -import qualified Data.List as List +testShowCmdArgs :: Test testShowCmdArgs = TestCase ( do - let options = ["--number", "/tmp/file with spaces"] + let opts = ["--number", "/tmp/file with spaces"] let expected = "--number '/tmp/file with spaces'" - let actual = showCmdArgs options + let actual = showCmdArgs opts assertEqual "Convert command-line arguments to text" expected actual) +testIncludeYears :: Test testIncludeYears = TestCase ( do let txterr = "Some text without years" @@ -47,6 +47,7 @@ assertEqual "Get a list of years from an include file - success case 2" expected2 actual2 ) +testExtractDigits :: Test testExtractDigits = TestCase ( do let txt1 = "A number: 321\nAnother number is 42, so is 0" @@ -62,4 +63,5 @@ assertEqual "Extract digits from text 2" expected2 actual2 ) +tests :: Test tests = TestList [testShowCmdArgs, testIncludeYears, testExtractDigits] diff --git a/test/Spec.hs b/test/Spec.hs --- a/test/Spec.hs +++ b/test/Spec.hs @@ -12,11 +12,12 @@ import qualified CSVImport.Unit import qualified CSVImport.Integration +tests :: Test tests = TestList [Common.Unit.tests, Common.Integration.tests, CSVImport.Unit.tests, CSVImport.Integration.tests] main :: IO Counts main = do - counts <- runTestTT tests - if (errors counts > 0 || failures counts > 0) + errCounts <- runTestTT tests + if (errors errCounts > 0 || failures errCounts > 0) then exit $ ExitFailure 1 - else return counts + else return errCounts diff --git a/test/TestHelpers.hs b/test/TestHelpers.hs --- a/test/TestHelpers.hs +++ b/test/TestHelpers.hs @@ -7,9 +7,10 @@ import Prelude hiding (FilePath) import qualified Hledger.Flow.Types as FlowTypes -import Hledger.Flow.Import.Types import Hledger.Flow.Common +import Hledger.Flow.RuntimeOptions +inputJohnBogart :: [FilePath] inputJohnBogart = [ "import/john/bogartbank/savings/1-in/2017/2017-11-30.csv", "import/john/bogartbank/savings/1-in/2017/2017-12-30.csv", @@ -19,37 +20,46 @@ "import/john/bogartbank/checking/1-in/2018/2018-10-30.csv", "import/john/bogartbank/checking/1-in/2018/2018-12-30.csv", "import/john/bogartbank/checking/1-in/2019/2019-01-30.csv", - "import/john/bogartbank/checking/1-in/2019/2019-02-30.csv"] :: [FilePath] + "import/john/bogartbank/checking/1-in/2019/2019-02-30.csv"] +inputJohnOther :: [FilePath] inputJohnOther = [ "import/john/otherbank/creditcard/1-in/2017/2017-12-30.csv", "import/john/otherbank/creditcard/1-in/2018/2018-01-30.csv", "import/john/otherbank/investments/1-in/2018/2018-12-30.csv", - "import/john/otherbank/investments/1-in/2019/2019-01-30.csv"] :: [FilePath] + "import/john/otherbank/investments/1-in/2019/2019-01-30.csv"] +inputJaneBogart :: [FilePath] inputJaneBogart = [ "import/jane/bogartbank/savings/1-in/2017/2017-12-30.csv", "import/jane/bogartbank/savings/1-in/2018/2018-01-30.csv", "import/jane/bogartbank/checking/1-in/2018/2018-12-30.csv", - "import/jane/bogartbank/checking/1-in/2019/2019-01-30.csv"] :: [FilePath] + "import/jane/bogartbank/checking/1-in/2019/2019-01-30.csv"] +inputJaneOther :: [FilePath] inputJaneOther = [ "import/jane/otherbank/creditcard/1-in/2017/2017-12-30.csv", "import/jane/otherbank/creditcard/1-in/2018/2018-01-30.csv", "import/jane/otherbank/investments/1-in/2018/2018-12-30.csv", - "import/jane/otherbank/investments/1-in/2019/2019-01-30.csv"] :: [FilePath] + "import/jane/otherbank/investments/1-in/2019/2019-01-30.csv"] +inputFiles :: [FilePath] inputFiles = inputJohnBogart <> inputJohnOther <> inputJaneBogart <> inputJaneOther -journalFiles = toJournals inputFiles :: [FilePath] -extraFiles = ["import/john/bogartbank/savings/2017-opening.journal"] :: [FilePath] -hiddenFiles = [".hiddenfile", "checking/.DS_Store", "import/john/bogartbank/savings/1-in/.anotherhiddenfile", "import/john/bogartbank/checking/1-in/2018/.hidden"] :: [FilePath] +journalFiles :: [FilePath] +journalFiles = toJournals inputFiles +extraFiles :: [FilePath] +extraFiles = ["import/john/bogartbank/savings/2017-opening.journal"] + +hiddenFiles :: [FilePath] +hiddenFiles = [".hiddenfile", "checking/.DS_Store", "import/john/bogartbank/savings/1-in/.anotherhiddenfile", "import/john/bogartbank/checking/1-in/2018/.hidden"] + defaultHlInfo :: FlowTypes.HledgerInfo defaultHlInfo = FlowTypes.HledgerInfo "/path/to/hledger" "1.2.3" -defaultOpts :: FilePath -> ImportOptions -defaultOpts bd = ImportOptions bd versionInfo' defaultHlInfo systemInfo False False False +defaultOpts :: FilePath -> RuntimeOptions +defaultOpts bd = RuntimeOptions bd versionInfo' defaultHlInfo systemInfo False False False toJournals :: [FilePath] -> [FilePath] toJournals = map (changePathAndExtension "3-journal" "journal")