hledger-flow 0.12.4.0 → 0.13.0.0

raw patch · 8 files changed

+85/−72 lines, 8 filesPVP ok

version bump matches the API change (PVP)

API changes (from Hackage documentation)

Files

• ChangeLog.md +14/−0
• README.org +53/−39
• app/Main.hs +4/−1
• hledger-flow.cabal +2/−2
• src/Hledger/Flow/CSVImport.hs +6/−3
• src/Hledger/Flow/RuntimeOptions.hs +4/−3
• test/Common/Integration.hs +1/−23
• test/TestHelpers.hs +1/−1

@@ -1,5 +1,19 @@ # Changelog for [hledger-flow](https://github.com/apauley/hledger-flow) +## 0.13.0++- Add an experimental rundir option for imports++The experimental rundir is an attempt to restrict hledger-flow into processing just a subset of files, primarily to quickly get feedback/failures while adding new accounts to an existing set of accounts.++The use case has been described in [issue 64](https://github.com/apauley/hledger-flow/issues/64).++It is experimental, because the only problem it currently solves is getting hledger-flow to fail fast.+One of the current side effects of doing so is that the generated include files are then written to only +include the subset of files that were processed.++But as soon as you do a full run again, the include files will be correctly re-generated as before.+ ## 0.12.4.0  - Update usage of hledger to reflect updated command-line flags of hledger version 1.15

@@ -14,18 +14,21 @@   :CUSTOM_ID: what-is-it   :END: -=hledger-flow= is a command-line program that gives you a guided-[[https://hledger.org/][Hledger]] workflow. It focuses on automated-processing of electronic statements as much as possible, as opposed to-manually adding your own hledger journal entries.+=hledger-flow= is a command-line program that gives you a guided [[https://hledger.org/][Hledger]]+workflow. It is important to note that most of the heavy lifting is done by the+upstream =hledger= project. For example, =hledger-flow= cares about where you+put your files for long-term maintainability, but the actual conversion to+classified accounting journals are done by =hledger=. -Manual entries are still possible, it just saves time in the long run to-automatically process a statement whenever one is available.+=hledger-flow= focuses on automated processing of electronic statements as much as possible,+as opposed to manually adding your own hledger journal entries. Manual entries+are still possible, it just saves time in the long run to automatically process+a statement whenever one is available.  Within =hledger-flow= you will keep your original bank statements around permanently as input, and generate classified Hledger journals each time-you run the program. The classification is done with rules files, or-your own script hooks.+you run the program. The classification is done with [[https://hledger.org/csv.html][hledger's rules files]],+and/or your own script hooks.  Keeping the original statements means that you never have to worry too much about "am I doing this accounting thing right?" or "what happens if@@ -33,9 +36,9 @@ classification, or if you made a mistake, you just change your classification rules, and run the program again. -It started when I realized that the scripts I wrote while playing around-with the ideas in [[https://github.com/adept/full-fledged-hledger/wiki][adept's Full-fledged Hledger]] aren't really specific to-my own finances, and can be shared.+It started when I realized that the scripts I wrote while playing around with+[[https://github.com/adept/full-fledged-hledger/wiki][adept's Full-fledged Hledger]] aren't really specific to my own finances, and can+be shared.  * Overview of the Basic Workflow   :PROPERTIES:@@ -449,24 +452,12 @@    :END:  If you need even more power and flexibility than what you can get from-the =preprocess= script and =hledger='s CSV import functionality, then+the =preprocess= script and =hledger='s [[https://hledger.org/csv.html][CSV import functionality]], then you can create your own custom script to =construct= transactions exactly as you need them.  At the expense of more construction work for you, of course. -As an example, =hledger='s CSV import currently-[[https://github.com/simonmichael/hledger/issues/627][only supports two-postings per transaction]], even though =hledger= itself is perfectly-happy with transactions containing more than two postings, e.g.:--#+BEGIN_EXAMPLE-    2019-02-01 Mortgage Payment-    Liabilities:Mortgage                                $1000.00-    Expenses:Interest:Real Estate                         $833.33-    Assets:Cash                                         -$1833.33-#+END_EXAMPLE- The =construct= script can be used in addition to the =preprocess= script, or on it's own. But since the =construct= script is more powerful than the =preprocess= script, you could tell your =construct=@@ -556,6 +547,37 @@ It works, but the naming of =_manual_= looks a bit weird. Should it be changed? +* Validating an hledger repository using Github Actions++The following example was contributed by [[https://github.com/amitaibu][Amitai Burstein]]:++#+BEGIN_SRC yaml+# .github/workflows/hledger-flow.yml++name: Validate hledger-flow++on: [push]++jobs:+  build:++    runs-on: ubuntu-latest+    steps:+    - uses: actions/checkout@v1++    - name: Install hledger+      run: docker pull dastapov/hledger++    - name: Install hledger-flow+      run: curl -L https://github.com/apauley/hledger-flow/releases/download/v0.12.4.0/hledger-flow_Linux_x86_64_v0.12.4.0_4b9b027.tar.gz | tar xvz && mv hledger-flow_Linux_x86_64_v0.12.4.0_4b9b027/hledger-flow .++    - name: Grant permissions to create files+      run: chmod 777 -R ./my-finances++    - name: Test hledger file+      run: docker run --name="ledger" -v $(pwd):/data dastapov/hledger ./hledger-flow import ./my-finances+#+END_SRC yaml+ * Compatibility with Ledger   :PROPERTIES:   :CUSTOM_ID: compatibility-with-ledger@@ -705,18 +727,10 @@ Full-fledged Hledger system, with a few implementation details that are different. -| Full-fledged Hledger                                                                                                                                                                                                       | Hledger Flow                                                                                                                                                                                                                                               |-|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-| FFH describes itself as a [[https://github.com/adept/full-fledged-hledger/wiki#full-fledged-hledger-tutorial][tutorial]] with helper scripts that you can start using and adapt to your needs.                                                                                                                   | I started by following the FFH tutorial, and changed bits and pieces over time to suit my needs. The "owner/bank/account" structure for example.                                                                                                           |-|                                                                                                                                                                                                                            |                                                                                                                                                                                                                                                            |-| FFH is more open-ended: you can start with the basic scripts and over time turn it into something that solves your needs exactly. But you'll also end up with more code that you need to maintain yourself.                | Hledger Flow is more opinionated and less open-ended. For example, you have to adopt the "owner/bank/account" structure precisely as specified. But this allows Hledger Flow to do more work for you.                                                      |-|                                                                                                                                                                                                                            |                                                                                                                                                                                                                                                            |-| Maintaining include files are currently part of the user's responsibility.                                                                                                                                                 | Hledger Flow generates flexible include files for you, and automatically includes opening/closing journals if the appropriately named files are present on disk.                                                                                           |-|                                                                                                                                                                                                                            |                                                                                                                                                                                                                                                            |-| FFH generates some of the same reports as Hledger Flow.                                                                                                                                                                    | The "owner/bank/account" structure may look a bit much at first, but it allows us to run separate queries/reports for "me" vs "my spouse" vs "my business" etc and also allows for reports covering all of it in a an overall view.                       |-|                                                                                                                                                                                                                            |                                                                                                                                                                                                                                                            |-| FFH chose scripts and build files that you can easily modify as you go along, but this requires a Haskell runtime to be installed everywhere it needs to run. The included docker image helps to make it less of an issue. | Hledger Flow distributes a compiled binary. This means users or deployment targets don't need extra dependencies installed, they can just run a CLI program. This also provides a clearer distinction between what is provided, and what users need to do. |-|                                                                                                                                                                                                                            |                                                                                                                                                                                                                                                            |-| The FFH build scripts requires familiarity with Haskell and the Shake build system.                                                                                                                                        | Users may need to write =preprocess= or =construct= hooks, but in a language of their choice.                                                                                                                                                              |-|                                                                                                                                                                                                                            |                                                                                                                                                                                                                                                            |-| Input files are assumed to always be CSV files.                                                                                                                                                                            | Hledger Flow de-empasises the need that input files must be in CSV format. Input files can be in any format that a =preprocess= or =construct= hook can read.                                                                                              |+| Full-fledged Hledger                                                                                                                                                                                                                    | Hledger Flow                                                                                                                                                                                                                                               |+|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|+| FFH is a [[https://github.com/adept/full-fledged-hledger/wiki#full-fledged-hledger-tutorial][tutorial with helper scripts]] that you can start using and adapt to your needs.                                                                                                                                                 | I started with FFH, and changed bits and pieces over time to suit my needs. The "owner/bank/account" structure for example.                                                                                                                                |+|                                                                                                                                                                                                                                         |                                                                                                                                                                                                                                                            |+| FFH is more open-ended: you can start with the basic scripts and over time turn it into something that solves your needs exactly. But you'll also end up with more code that you need to maintain yourself.                             | Hledger Flow is more opinionated and less open-ended. For example, you have to adopt the "owner/bank/account" structure precisely as specified. But this allows Hledger Flow to do more work for you.                                                      |+|                                                                                                                                                                                                                                         |                                                                                                                                                                                                                                                            |+| FFH uses scripts and [[https://shakebuild.com/][Haskell/Shake build files]] that you can easily modify as you go along, but this requires a Haskell runtime to be installed everywhere it needs to run. The included docker image helps to make it less of an issue. | Hledger Flow [[https://github.com/apauley/hledger-flow/releases][distributes a compiled binary]]. This means users or deployment targets don't need extra dependencies installed, they can just run a CLI program. This also provides a clearer distinction between what is provided, and what users need to do. |

@@ -12,7 +12,8 @@ import Hledger.Flow.Reports import Hledger.Flow.CSVImport -data SubcommandParams = SubcommandParams { maybeBaseDir :: Maybe FilePath } deriving (Show)+data SubcommandParams = SubcommandParams { maybeBaseDir :: Maybe FilePath+                                         , maybeRunDir :: Maybe FilePath } deriving (Show) data Command = Import SubcommandParams | Report SubcommandParams deriving (Show)  data MainParams = MainParams { verbosity :: Int@@ -35,6 +36,7 @@   bd <- determineBaseDir $ maybeBaseDir subParams'   hli <- hledgerInfoFromPath $ hledgerPathOpt mainParams'   return RT.RuntimeOptions { RT.baseDir = bd+                           , RT.runDir = maybeRunDir subParams'                            , RT.hfVersion = versionInfo'                            , RT.hledgerInfo = hli                            , RT.sysInfo = systemInfo@@ -60,3 +62,4 @@ subcommandParser :: Parser SubcommandParams subcommandParser = SubcommandParams   <$> optional (argPath "basedir" "The hledger-flow base directory")+  <*> optional (strOption (long "experimental-rundir" <> help "A subdirectory of the base where the command will restrict itself to"))

@@ -4,10 +4,10 @@ -- -- see: https://github.com/sol/hpack ----- hash: 7a59c0bdbb268fed703e7d2f13e093027bc5ccb2f43d96117aa3496ea0c502b2+-- hash: 57d9fcf2e904cd58e0ec69c82e1177b3bb60e5d401f741bc0f82df8bbfa72143  name:           hledger-flow-version:        0.12.4.0+version:        0.13.0.0 synopsis:       An hledger workflow focusing on automated statement import and classification. description:    Please see the README on GitHub at <https://github.com/apauley/hledger-flow#readme> category:       Finance, Console

@@ -36,13 +36,16 @@ importCSVs' :: RuntimeOptions -> TChan FlowTypes.LogMessage -> IO [FilePath] importCSVs' opts ch = do   channelOutLn ch "Collecting input files..."-  (inputFiles, diff) <- time $ single . shellToList . onlyFiles $ find inputFilePattern $ baseDir opts+  let effectiveDir = case runDir opts of+        Nothing -> (baseDir opts) </> "import"+        Just rd -> rd+  (inputFiles, diff) <- time $ single . shellToList . onlyFiles $ find inputFilePattern effectiveDir   let fileCount = length inputFiles   if (fileCount == 0) then     do       let msg = format ("I couldn't find any input files underneath "%fp-                        %"\n\nhledger-makitso expects to find its input files in specifically\nnamed directories.\n\n"%-                        "Have a look at the documentation for a detailed explanation:\n"%s) (dirname (baseDir opts) </> "import/") (docURL "input-files")+                        %"\n\nhledger-flow expects to find its input files in specifically\nnamed directories.\n\n"%+                        "Have a look at the documentation for a detailed explanation:\n"%s) effectiveDir (docURL "input-files")       errExit 1 ch msg []     else     do

@@ -6,6 +6,7 @@ import Hledger.Flow.Types  data RuntimeOptions = RuntimeOptions { baseDir :: FilePath+                                     , runDir :: Maybe FilePath                                      , hfVersion :: Text                                      , hledgerInfo :: HledgerInfo                                      , sysInfo :: SystemInfo@@ -16,10 +17,10 @@   deriving (Show)  instance HasVerbosity RuntimeOptions where-  verbose (RuntimeOptions _ _ _ _ v _ _) = v+  verbose (RuntimeOptions _ _ _ _ _ v _ _) = v  instance HasSequential RuntimeOptions where-  sequential (RuntimeOptions _ _ _ _ _ _ sq) = sq+  sequential (RuntimeOptions _ _ _ _ _ _ _ sq) = sq  instance HasBaseDir RuntimeOptions where-  baseDir (RuntimeOptions bd _ _ _ _ _ _) = bd+  baseDir (RuntimeOptions bd _ _ _ _ _ _ _) = bd

@@ -28,28 +28,6 @@      )   ) -testDirOrPwd :: Test-testDirOrPwd = TestCase (-  sh (-      do-        currentDir <- fmap (\p -> directory (p </> "t")) pwd-        tmpdir <- using (mktempdir "." "hlflow")-        let fooDir = collapse $ currentDir </> tmpdir </> "foo/"-        let barDir = collapse $ currentDir </> tmpdir </> "bar/"-        mkdir fooDir-        mkdir barDir-        d1 <- liftIO $ dirOrPwd Nothing-        liftIO $ assertEqual "dirOrPwd returns pwd as a fallback" currentDir d1-        liftIO $ assertEqual "dirOrPwd assumes the fallback is a directory" (directory d1) d1-        d2 <- liftIO $ dirOrPwd $ Just $ tmpdir </> "foo"-        liftIO $ assertEqual "dirOrPwd returns the supplied dir - no trailing slash supplied" fooDir d2-        liftIO $ assertEqual "dirOrPwd assumes the supplied dir is a directory - no trailing slash supplied" (directory d2) d2-        d3 <- liftIO $ dirOrPwd $ Just $ tmpdir </> "bar/"-        liftIO $ assertEqual "dirOrPwd returns the supplied dir - trailing slash supplied" barDir d3-        liftIO $ assertEqual "dirOrPwd assumes the supplied dir is a directory - trailing slash supplied" (directory d3) d3-     )-  )- assertSubDirsForDetermineBaseDir :: FilePath -> [FilePath] -> IO () assertSubDirsForDetermineBaseDir expectedBaseDir importDirs = do   _ <- sequence $ map (assertDetermineBaseDir expectedBaseDir) importDirs@@ -127,4 +105,4 @@   )  tests :: Test-tests = TestList [testDirOrPwd, testDetermineBaseDir, testHiddenFiles, testFilterPaths]+tests = TestList [testDetermineBaseDir, testHiddenFiles, testFilterPaths]

@@ -59,7 +59,7 @@ defaultHlInfo = FlowTypes.HledgerInfo "/path/to/hledger" "1.2.3"  defaultOpts :: FilePath -> RuntimeOptions-defaultOpts bd = RuntimeOptions bd versionInfo' defaultHlInfo systemInfo False False False+defaultOpts bd = RuntimeOptions bd Nothing versionInfo' defaultHlInfo systemInfo False False False  toJournals :: [FilePath] -> [FilePath] toJournals = map (changePathAndExtension "3-journal" "journal")