hledger-stockquotes 0.1.0.0 → 0.1.1.0

raw patch · 6 files changed

+257/−86 lines, 6 filesPVP: major bump suggested

API removals or changes: PVP suggests a major version bump

API changes (from Hackage documentation)

Files

• CHANGELOG.md +13/−0
• README.md +80/−23
• app/Main.hs +77/−21
• hledger-stockquotes.cabal +3/−3
• src/Hledger/StockQuotes.hs +37/−17
• src/Web/AlphaVantage.hs +47/−22

@@ -1,5 +1,18 @@ # CHANGELOG +## master+++## v0.1.1.0++* Don't write out a journal file if no prices were successfully fetched.+* Log API errors to `stderr` instead of `stdout`.+* Improve error messages when the AlphaVantage API returns a+  rate-limit-exceeded error.+* Improve documentation in README & `--help` flag.+* Add trailing newline to generated files.++ ## v0.1.0.0  * Initial release

@@ -8,47 +8,104 @@ need a [free AlphaVantage API key](https://www.alphavantage.co/support/#api-key).  -## Running+## Usage -This application is still in early development, so you'll need to clone this-repository first:+`hledger-stockquotes` parses your journal file, determines what commodities are+defined, and queries AlphaVantage for prices on the date range present in the+journal file. +By default, the program will use the HLedger default file location of+`~/.hledger.journal`. A `LEDGER_FILE` environmental variable can be used to+override the location. The `-f` flag can be used to override both the default+and `LEDGER_FILE` locations.++At the bare minimum, you need to set an `ALPHAVANTAGE_KEY` environmental+variable or use the `-a` switch to specify your AlphaVantage key:+ ```-git clone https://github.com/prikhi/hledger-stockquotes.git-cd hledger-stockquotes+hledger-stockquotes -a MY_API_KEY -f accounting.journal ``` -Then you can run the application:+This will print out price directive to a `prices.journal` file. ++### Custom Output Files++The output file can be set with the `-o` flag:+ ```-stack run -- --help+hledger-stockquotes -a MY_API_KEY -o prices/2021.journal ``` -Use the `-a` flag to pass in your API key and optionally pass the path to your-journal file:+NOTE: the contents of the output file will be overwritten if the file already+exists! ++### Excluding Commodities++By default, we query AlphaVantage for all non-USD commodities included in your+journal file. We do not currently support AlphaVantage's FOREX or Crypto API+routes, so if you have those commodities, `stockquotes` will print an error+when fetching them. You can exclude commodities by passing them as arguments to+`hledger-stockquotes`:+ ```-stack run -- -a API_KEY -f accounting.journal+hledger-stockquotes -a MY_API_KEY AUTO TA_VFFVX ``` -If you omit the `-f` flag, the journal file will fallback to the value of the-`LEDGER_FILE` environmental variable. If `LEDGER_FILE` is undefined, a fallback-of `~/.hledger.journal` will be used.+NOTE: hledger defines an `AUTO` commodity if you use the default commodity+directive(`D`). -You can omit the `-a` flag by setting the `ALPHAVANTAGE_KEY` environmental-variable. -The output file defaults to `prices.journal`. You can customize this with the-`-o` flag. Note that the contents of the output file will be overwritten if the-file already exists.+### API Limits -By default, the application will limit itself to 5 API requests a minute, as-specified by the AlphaVantage documentation. You can override this by using the-`-n` flag. You can have the application print the dates and commodities it will-fetch by passing the `--dry-run` flag.+AlphaVantage has an API request limit of 5 requests per minute.+`hledger-stockquotes` enforces this limit on a per-command basis. A single run+will fetch 5 price histories, wait 60 seconds, fetch 5 more, etc. Running+multiple `hledger-stockquotes` commands in sequence will not enforce this limit+over multiple runs and may result in API errors. You can ignore the request+limiting with the `-n` flag. To test a command without hitting the API, pass+the `--dry-run` flag. This will simply print out the commodities and date+ranges that would be queried instead of making requests to AlphaVantage.  -## Manual Builds+### Additional Documentation++The `--help` flag provides more thorough documentation on all available flags:++```+hledger-stockquotes --help+```+++## Build / Install++This project has not yet been packaged for any OSes or Linux distributions, so+you'll have to clone this repository & compile/install the code yourself:++```+git clone https://github.com/prikhi/hledger-stockquotes.git+cd hledger-stockquotes+stack install+```++This will put the `hledger-stockquotes` exe into your `~/.local/bin/`+directory. Ensure that the directory is included in your `PATH` environmental+variable. Then you can run the application:++```+hledger-stockquotes --help+```++Since the executable has the `hledger-` prefix, you can also use it with the+`hledger` command:++```+hledger stockquotes -- --help+```+++## Development/Manual Builds  You can build the project with stack: `stack build` 

@@ -6,32 +6,35 @@ import           Data.Foldable                  ( asum ) import           Data.Maybe                     ( fromMaybe ) import           Data.Time                      ( Day-                                                , formatTime                                                 , defaultTimeLocale+                                                , formatTime                                                 ) import           Data.Version                   ( showVersion ) import           System.Console.CmdArgs         ( (&=)                                                 , Data                                                 , Typeable-                                                , typ-                                                , help+                                                , args+                                                , cmdArgs                                                 , details                                                 , enum-                                                , ignore                                                 , explicit+                                                , help+                                                , helpArg+                                                , ignore                                                 , name-                                                , summary                                                 , program-                                                , helpArg-                                                , cmdArgs-                                                , args+                                                , summary+                                                , typ                                                 ) import           System.Environment             ( lookupEnv ) import           System.Exit                    ( exitFailure )+import           System.IO                      ( hPutStrLn+                                                , stderr+                                                )  import           Hledger.StockQuotes-import           Web.AlphaVantage               ( Config(..) ) import           Paths_hledger_stockquotes      ( version )+import           Web.AlphaVantage               ( Config(..) )  import qualified Data.ByteString.Lazy          as LBS import qualified Data.Text                     as T@@ -45,7 +48,7 @@     apiKey         <- case asum [apiKey_, apiKeyEnv] of         Just k -> return k         Nothing ->-            putStrLn+            logError                     "Error: Pass an AlphaVantage API Key with `-a` or $ALPHAVANTAGE_KEY."                 >> exitFailure @@ -58,7 +61,10 @@     if not dryRun         then do             prices <- fetchPrices cfg commodities start end rateLimit-            LBS.writeFile outputFile $ makePriceDirectives prices+            if null prices+                then logError+                    "Error: No price directives were able to be fetched."+                else LBS.writeFile outputFile $ makePriceDirectives prices         else do             putStrLn                 $  "Querying from "@@ -71,17 +77,19 @@   where     showDate :: Day -> String     showDate = formatTime defaultTimeLocale "%Y-%m-%d"+    logError :: String -> IO ()+    logError = hPutStrLn stderr  -data Args =-    Args-        { apiKey_ :: Maybe String-        , rateLimit :: Bool-        , journalFile_ :: Maybe FilePath-        , outputFile :: FilePath-        , excludedCurrencies :: [String]-        , dryRun :: Bool-        } deriving (Data, Typeable, Show, Eq)+data Args = Args+    { apiKey_            :: Maybe String+    , rateLimit          :: Bool+    , journalFile_       :: Maybe FilePath+    , outputFile         :: FilePath+    , excludedCurrencies :: [String]+    , dryRun             :: Bool+    }+    deriving (Data, Typeable, Show, Eq)  argSpec :: Args argSpec =@@ -113,7 +121,8 @@                 &= typ "FILE"             , outputFile         =                 "prices.journal"-                &= help "File to write prices into. Default: prices.journal"+                &= help+                       "File to write prices into. Existing files will be overwritten. Default: prices.journal"                 &= explicit                 &= name "output-file"                 &= name "o"@@ -136,14 +145,61 @@                , "AlphaVantage stock quote API, and writes a new journal file "                , "containing price directives for each commodity."                , ""+               , ""+               , "DESCRIPTION"+               , ""+               , "By default, we find all non-USD commodities in your "+               , "journal file and query AlphaVantage for their stock prices "+               , "over the date range used in the journal file. Currently, we "+               , "only support public U.S. equities & do not call out to AlphVantage's"+               , "FOREX or Crypto API routes. If you have commodities that are "+               , "not supported by AlphaVantage, hledger-stockquotes will output "+               , "an error when attempting to processing them. To avoid processing "+               , "of unsupported currencies, you can pass in any commodities to "+               , "exclude as arguments. If you use the default commodity directive "+               , "in your journal file, hledger will include an `AUTO` commodity "+               , "when parsing your journal."+               , ""+               , ""+               , "API LIMITS"+               , ""+               , "AlphVantage's API limits users to 5 requests per minute. We respect "+               , "this limit by waiting for 60 seconds after every 5 commities we process. "+               , "You can ignore the rate-limiting by using the `-n` flag, but "+               , "requests are more likely to fail. You can use the `-d` flag to print "+               , "out the dates & currencies that we will fetch to avoid any unecessary "+               , "processing or API requests."+               , ""+               , ""+               , "OUTPUT FILE"+               , ""+               , "You can use the `-o` flag to set the file we will write the "+               , "generated price directives into. By default, we write to "+               , "`prices.journal`."+               , ""                , "Warning: the output file will always be overwritten with the new "                , "price directives. We currently do not support appending to the "                , "output file."                , ""+               , ""+               , "ENVIRONMENTAL VARIABLES"+               , ""                , "If no `-f` flag is passed and the LEDGER_FILE environmental "                , "variable is set, the program will use that as the default "                , "HLedger file. Otherwise ~/.hledger.journal will be used."                , ""                , "Instead of passing the `-a` flag with your AlphaVantage API key, "                , "you can set the ALPHAVANTAGE_KEY environmental variable instead."+               , ""+               , ""+               , "USAGE EXAMPLES"+               , ""+               , "Fetch prices for all commodities in the default journal file:"+               , "    hledger-stockquotes -a <your-api-key>"+               , ""+               , "Output prices into a custom journal file:"+               , "    hledger-stockquotes -a <your-api-key> -o prices/2021.journal"+               , ""+               , "Ignore the default, foreign, & crypto commodities:"+               , "    hledger-stockquotes -a <your-api-key> AUTO BTC ETH EUR"                ]

@@ -1,13 +1,13 @@ cabal-version: 1.12 --- This file has been generated from package.yaml by hpack version 0.33.0.+-- This file has been generated from package.yaml by hpack version 0.34.3. -- -- see: https://github.com/sol/hpack ----- hash: a7089d237d71e7e01442d12003aecf7d101611bfc38701eda170c61077f2fe80+-- hash: 3aa877fb077a2f97f6c9f6cd6234051d71632bb973ebd221b1dac93398f75728  name:           hledger-stockquotes-version:        0.1.0.0+version:        0.1.1.0 synopsis:       Generate HLedger Price Directives From Daily Stock Quotes. description:    @hledger-stockquotes@ is an addon for <https://hledger.org/ hledger> that                 reads your journal file, pulls the historical stock prices for commodities,

@@ -7,27 +7,32 @@ -} module Hledger.StockQuotes where +import           Control.Concurrent             ( threadDelay ) import           Control.Exception              ( SomeException+                                                , displayException                                                 , try                                                 )-import           Control.Concurrent             ( threadDelay ) import           Data.List.Split                ( chunksOf ) import           Data.Maybe                     ( catMaybes )+import           Data.Text.Encoding             ( encodeUtf8 ) import           Data.Time                      ( Day                                                 , UTCTime(utctDay)-                                                , formatTime                                                 , defaultTimeLocale-                                                , getCurrentTime+                                                , formatTime                                                 , fromGregorian+                                                , getCurrentTime                                                 , toGregorian                                                 )-import           Data.Text.Encoding             ( encodeUtf8 ) import           Hledger-import           Safe.Foldable                  ( minimumMay-                                                , maximumMay+import           Safe.Foldable                  ( maximumMay+                                                , minimumMay                                                 )+import           System.IO                      ( hPutStrLn+                                                , stderr+                                                ) -import           Web.AlphaVantage               ( Config+import           Web.AlphaVantage               ( AlphaVantageResponse(..)+                                                , Config                                                 , Prices(..)                                                 , getDailyPrices                                                 )@@ -74,20 +79,35 @@     -> Bool     -> IO [(CommoditySymbol, [(Day, Prices)])] fetchPrices cfg symbols start end rateLimit = do-    let action symbol = try (getDailyPrices cfg symbol start end) >>= \case-            Left (e :: SomeException) -> do-                putStrLn-                    $  "Error Fetching Symbol `"-                    <> T.unpack symbol-                    <> "`: "-                    ++ show e-                return Nothing-            Right prices -> return $ Just (symbol, prices)     if rateLimit         then fmap catMaybes $ rateLimitActions $ map action symbols         else catMaybes <$> mapM action symbols+  where+    action :: CommoditySymbol -> IO (Maybe (CommoditySymbol, [(Day, Prices)]))+    action symbol = try (getDailyPrices cfg symbol start end) >>= \case+        Left (e :: SomeException) -> do+            logError+                $  "Error Fetching Prices for Symbol `"+                <> T.unpack symbol+                <> "`:\n\t"+                ++ displayException e+                ++ "\n"+            return Nothing +        Right (ApiError note) -> do+            logError+                $  "Error Fetching Prices for Symbol `"+                <> T.unpack symbol+                <> "`:\n\t"+                <> T.unpack note+                <> "\n"+            return Nothing +        Right (ApiResponse prices) -> return $ Just (symbol, prices)+    logError :: String -> IO ()+    logError = hPutStrLn stderr++ -- | Perform the actions at a rate of 5 per second, then return all the -- results. --@@ -111,7 +131,7 @@ -- | Build the Price Directives for the Daily Prices of the given -- Commodities. makePriceDirectives :: [(CommoditySymbol, [(Day, Prices)])] -> LBS.ByteString-makePriceDirectives = LBS.intercalate "\n\n" . map makeDirectives+makePriceDirectives = (<> "\n") . LBS.intercalate "\n\n" . map makeDirectives   where     makeDirectives :: (CommoditySymbol, [(Day, Prices)]) -> LBS.ByteString     makeDirectives (symbol, prices) =

@@ -1,3 +1,4 @@+{-# LANGUAGE DeriveFunctor #-} {-# LANGUAGE DeriveGeneric #-} {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE RecordWildCards #-}@@ -9,37 +10,39 @@ -} module Web.AlphaVantage     ( Config(..)+    , AlphaVantageResponse(..)     , Prices(..)     , getDailyPrices-    )-where+    ) where  import           Data.Aeson                     ( (.:)+                                                , (.:?)                                                 , FromJSON(..)+                                                , Value(Object)                                                 , withObject                                                 ) import           Data.Scientific                ( Scientific ) import           Data.Time                      ( Day-                                                , parseTimeM                                                 , defaultTimeLocale+                                                , parseTimeM                                                 ) import           GHC.Generics                   ( Generic ) import           Network.HTTP.Req               ( (/~)                                                 , (=:)                                                 , GET(..)                                                 , NoReqBody(..)-                                                , runReq-                                                , req                                                 , defaultHttpConfig                                                 , https                                                 , jsonResponse+                                                , req                                                 , responseBody+                                                , runReq                                                 ) import           Text.Read                      ( readMaybe ) -import qualified Data.Text                     as T import qualified Data.HashMap.Strict           as HM import qualified Data.List                     as L+import qualified Data.Text                     as T   -- | Configuration for the AlphaVantage API Client.@@ -49,7 +52,22 @@         -- ^ Your API Key.         } deriving (Show, Read, Eq,  Generic) +-- | Wrapper type enumerating between successful responses and error+-- responses with notes.+data AlphaVantageResponse a+    = ApiResponse a+    | ApiError T.Text+    deriving (Show, Read, Eq, Generic, Functor) +-- | Check for errors by attempting to parse a `Note` field. If one does+-- not exist, parse the inner type.+instance FromJSON a => FromJSON (AlphaVantageResponse a) where+    parseJSON = withObject "AlphaVantageResponse" $ \v -> do+        mbErrorNote <- v .:? "Note"+        case mbErrorNote of+            Nothing   -> ApiResponse <$> parseJSON (Object v)+            Just note -> return $ ApiError note+ -- | List of Daily Prices for a Stock. newtype PriceList =     PriceList@@ -66,14 +84,14 @@         where parseDay = parseTimeM True defaultTimeLocale "%F"  -- | The Single-Day Price Quotes & Volume for a Stock,.-data Prices =-    Prices-        { pOpen :: Scientific-        , pHigh :: Scientific-        , pLow :: Scientific-        , pClose :: Scientific-        , pVolume :: Integer-        } deriving (Show, Read, Eq, Generic)+data Prices = Prices+    { pOpen   :: Scientific+    , pHigh   :: Scientific+    , pLow    :: Scientific+    , pClose  :: Scientific+    , pVolume :: Integer+    }+    deriving (Show, Read, Eq, Generic)  instance FromJSON Prices where     parseJSON = withObject "Prices" $ \v -> do@@ -88,12 +106,17 @@             val <- parser             case readMaybe val of                 Just x  -> return x-                Nothing -> fail $ "Could not read: " ++ val+                Nothing -> fail $ "Could not parse number: " ++ val   -- | Fetch the Daily Prices for a Stock, returning only the prices between -- the two given dates.-getDailyPrices :: Config -> T.Text -> Day -> Day -> IO [(Day, Prices)]+getDailyPrices+    :: Config+    -> T.Text+    -> Day+    -> Day+    -> IO (AlphaVantageResponse [(Day, Prices)]) getDailyPrices cfg symbol startDay endDay = do     resp <- runReq defaultHttpConfig $ req         GET@@ -111,9 +134,11 @@         <> "apikey"         =: cApiKey cfg         )-    return-        . takeWhile ((<= endDay) . fst)-        . dropWhile ((< startDay) . fst)-        . L.sortOn fst-        . fromPriceList-        $ responseBody resp+    return . fmap filterByDate $ responseBody resp+  where+    filterByDate :: PriceList -> [(Day, Prices)]+    filterByDate =+        takeWhile ((<= endDay) . fst)+            . dropWhile ((< startDay) . fst)+            . L.sortOn fst+            . fromPriceList