diff --git a/CHANGELOG.md b/CHANGELOG.md --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,18 @@ # CHANGELOG +## master + + +## v0.1.1.0 + +* Don't write out a journal file if no prices were successfully fetched. +* Log API errors to `stderr` instead of `stdout`. +* Improve error messages when the AlphaVantage API returns a + rate-limit-exceeded error. +* Improve documentation in README & `--help` flag. +* Add trailing newline to generated files. + + ## v0.1.0.0 * Initial release diff --git a/README.md b/README.md --- a/README.md +++ b/README.md @@ -8,47 +8,104 @@ need a [free AlphaVantage API key](https://www.alphavantage.co/support/#api-key). -## Running +## Usage -This application is still in early development, so you'll need to clone this -repository first: +`hledger-stockquotes` parses your journal file, determines what commodities are +defined, and queries AlphaVantage for prices on the date range present in the +journal file. +By default, the program will use the HLedger default file location of +`~/.hledger.journal`. A `LEDGER_FILE` environmental variable can be used to +override the location. The `-f` flag can be used to override both the default +and `LEDGER_FILE` locations. + +At the bare minimum, you need to set an `ALPHAVANTAGE_KEY` environmental +variable or use the `-a` switch to specify your AlphaVantage key: + ``` -git clone https://github.com/prikhi/hledger-stockquotes.git -cd hledger-stockquotes +hledger-stockquotes -a MY_API_KEY -f accounting.journal ``` -Then you can run the application: +This will print out price directive to a `prices.journal` file. + +### Custom Output Files + +The output file can be set with the `-o` flag: + ``` -stack run -- --help +hledger-stockquotes -a MY_API_KEY -o prices/2021.journal ``` -Use the `-a` flag to pass in your API key and optionally pass the path to your -journal file: +NOTE: the contents of the output file will be overwritten if the file already +exists! + +### Excluding Commodities + +By default, we query AlphaVantage for all non-USD commodities included in your +journal file. We do not currently support AlphaVantage's FOREX or Crypto API +routes, so if you have those commodities, `stockquotes` will print an error +when fetching them. You can exclude commodities by passing them as arguments to +`hledger-stockquotes`: + ``` -stack run -- -a API_KEY -f accounting.journal +hledger-stockquotes -a MY_API_KEY AUTO TA_VFFVX ``` -If you omit the `-f` flag, the journal file will fallback to the value of the -`LEDGER_FILE` environmental variable. If `LEDGER_FILE` is undefined, a fallback -of `~/.hledger.journal` will be used. +NOTE: hledger defines an `AUTO` commodity if you use the default commodity +directive(`D`). -You can omit the `-a` flag by setting the `ALPHAVANTAGE_KEY` environmental -variable. -The output file defaults to `prices.journal`. You can customize this with the -`-o` flag. Note that the contents of the output file will be overwritten if the -file already exists. +### API Limits -By default, the application will limit itself to 5 API requests a minute, as -specified by the AlphaVantage documentation. You can override this by using the -`-n` flag. You can have the application print the dates and commodities it will -fetch by passing the `--dry-run` flag. +AlphaVantage has an API request limit of 5 requests per minute. +`hledger-stockquotes` enforces this limit on a per-command basis. A single run +will fetch 5 price histories, wait 60 seconds, fetch 5 more, etc. Running +multiple `hledger-stockquotes` commands in sequence will not enforce this limit +over multiple runs and may result in API errors. You can ignore the request +limiting with the `-n` flag. To test a command without hitting the API, pass +the `--dry-run` flag. This will simply print out the commodities and date +ranges that would be queried instead of making requests to AlphaVantage. -## Manual Builds +### Additional Documentation + +The `--help` flag provides more thorough documentation on all available flags: + +``` +hledger-stockquotes --help +``` + + +## Build / Install + +This project has not yet been packaged for any OSes or Linux distributions, so +you'll have to clone this repository & compile/install the code yourself: + +``` +git clone https://github.com/prikhi/hledger-stockquotes.git +cd hledger-stockquotes +stack install +``` + +This will put the `hledger-stockquotes` exe into your `~/.local/bin/` +directory. Ensure that the directory is included in your `PATH` environmental +variable. Then you can run the application: + +``` +hledger-stockquotes --help +``` + +Since the executable has the `hledger-` prefix, you can also use it with the +`hledger` command: + +``` +hledger stockquotes -- --help +``` + + +## Development/Manual Builds You can build the project with stack: `stack build` diff --git a/app/Main.hs b/app/Main.hs --- a/app/Main.hs +++ b/app/Main.hs @@ -6,32 +6,35 @@ import Data.Foldable ( asum ) import Data.Maybe ( fromMaybe ) import Data.Time ( Day - , formatTime , defaultTimeLocale + , formatTime ) import Data.Version ( showVersion ) import System.Console.CmdArgs ( (&=) , Data , Typeable - , typ - , help + , args + , cmdArgs , details , enum - , ignore , explicit + , help + , helpArg + , ignore , name - , summary , program - , helpArg - , cmdArgs - , args + , summary + , typ ) import System.Environment ( lookupEnv ) import System.Exit ( exitFailure ) +import System.IO ( hPutStrLn + , stderr + ) import Hledger.StockQuotes -import Web.AlphaVantage ( Config(..) ) import Paths_hledger_stockquotes ( version ) +import Web.AlphaVantage ( Config(..) ) import qualified Data.ByteString.Lazy as LBS import qualified Data.Text as T @@ -45,7 +48,7 @@ apiKey <- case asum [apiKey_, apiKeyEnv] of Just k -> return k Nothing -> - putStrLn + logError "Error: Pass an AlphaVantage API Key with `-a` or $ALPHAVANTAGE_KEY." >> exitFailure @@ -58,7 +61,10 @@ if not dryRun then do prices <- fetchPrices cfg commodities start end rateLimit - LBS.writeFile outputFile $ makePriceDirectives prices + if null prices + then logError + "Error: No price directives were able to be fetched." + else LBS.writeFile outputFile $ makePriceDirectives prices else do putStrLn $ "Querying from " @@ -71,17 +77,19 @@ where showDate :: Day -> String showDate = formatTime defaultTimeLocale "%Y-%m-%d" + logError :: String -> IO () + logError = hPutStrLn stderr -data Args = - Args - { apiKey_ :: Maybe String - , rateLimit :: Bool - , journalFile_ :: Maybe FilePath - , outputFile :: FilePath - , excludedCurrencies :: [String] - , dryRun :: Bool - } deriving (Data, Typeable, Show, Eq) +data Args = Args + { apiKey_ :: Maybe String + , rateLimit :: Bool + , journalFile_ :: Maybe FilePath + , outputFile :: FilePath + , excludedCurrencies :: [String] + , dryRun :: Bool + } + deriving (Data, Typeable, Show, Eq) argSpec :: Args argSpec = @@ -113,7 +121,8 @@ &= typ "FILE" , outputFile = "prices.journal" - &= help "File to write prices into. Default: prices.journal" + &= help + "File to write prices into. Existing files will be overwritten. Default: prices.journal" &= explicit &= name "output-file" &= name "o" @@ -136,14 +145,61 @@ , "AlphaVantage stock quote API, and writes a new journal file " , "containing price directives for each commodity." , "" + , "" + , "DESCRIPTION" + , "" + , "By default, we find all non-USD commodities in your " + , "journal file and query AlphaVantage for their stock prices " + , "over the date range used in the journal file. Currently, we " + , "only support public U.S. equities & do not call out to AlphVantage's" + , "FOREX or Crypto API routes. If you have commodities that are " + , "not supported by AlphaVantage, hledger-stockquotes will output " + , "an error when attempting to processing them. To avoid processing " + , "of unsupported currencies, you can pass in any commodities to " + , "exclude as arguments. If you use the default commodity directive " + , "in your journal file, hledger will include an `AUTO` commodity " + , "when parsing your journal." + , "" + , "" + , "API LIMITS" + , "" + , "AlphVantage's API limits users to 5 requests per minute. We respect " + , "this limit by waiting for 60 seconds after every 5 commities we process. " + , "You can ignore the rate-limiting by using the `-n` flag, but " + , "requests are more likely to fail. You can use the `-d` flag to print " + , "out the dates & currencies that we will fetch to avoid any unecessary " + , "processing or API requests." + , "" + , "" + , "OUTPUT FILE" + , "" + , "You can use the `-o` flag to set the file we will write the " + , "generated price directives into. By default, we write to " + , "`prices.journal`." + , "" , "Warning: the output file will always be overwritten with the new " , "price directives. We currently do not support appending to the " , "output file." , "" + , "" + , "ENVIRONMENTAL VARIABLES" + , "" , "If no `-f` flag is passed and the LEDGER_FILE environmental " , "variable is set, the program will use that as the default " , "HLedger file. Otherwise ~/.hledger.journal will be used." , "" , "Instead of passing the `-a` flag with your AlphaVantage API key, " , "you can set the ALPHAVANTAGE_KEY environmental variable instead." + , "" + , "" + , "USAGE EXAMPLES" + , "" + , "Fetch prices for all commodities in the default journal file:" + , " hledger-stockquotes -a " + , "" + , "Output prices into a custom journal file:" + , " hledger-stockquotes -a -o prices/2021.journal" + , "" + , "Ignore the default, foreign, & crypto commodities:" + , " hledger-stockquotes -a AUTO BTC ETH EUR" ] diff --git a/hledger-stockquotes.cabal b/hledger-stockquotes.cabal --- a/hledger-stockquotes.cabal +++ b/hledger-stockquotes.cabal @@ -1,13 +1,13 @@ cabal-version: 1.12 --- This file has been generated from package.yaml by hpack version 0.33.0. +-- This file has been generated from package.yaml by hpack version 0.34.3. -- -- see: https://github.com/sol/hpack -- --- hash: a7089d237d71e7e01442d12003aecf7d101611bfc38701eda170c61077f2fe80 +-- hash: 3aa877fb077a2f97f6c9f6cd6234051d71632bb973ebd221b1dac93398f75728 name: hledger-stockquotes -version: 0.1.0.0 +version: 0.1.1.0 synopsis: Generate HLedger Price Directives From Daily Stock Quotes. description: @hledger-stockquotes@ is an addon for that reads your journal file, pulls the historical stock prices for commodities, diff --git a/src/Hledger/StockQuotes.hs b/src/Hledger/StockQuotes.hs --- a/src/Hledger/StockQuotes.hs +++ b/src/Hledger/StockQuotes.hs @@ -7,27 +7,32 @@ -} module Hledger.StockQuotes where +import Control.Concurrent ( threadDelay ) import Control.Exception ( SomeException + , displayException , try ) -import Control.Concurrent ( threadDelay ) import Data.List.Split ( chunksOf ) import Data.Maybe ( catMaybes ) +import Data.Text.Encoding ( encodeUtf8 ) import Data.Time ( Day , UTCTime(utctDay) - , formatTime , defaultTimeLocale - , getCurrentTime + , formatTime , fromGregorian + , getCurrentTime , toGregorian ) -import Data.Text.Encoding ( encodeUtf8 ) import Hledger -import Safe.Foldable ( minimumMay - , maximumMay +import Safe.Foldable ( maximumMay + , minimumMay ) +import System.IO ( hPutStrLn + , stderr + ) -import Web.AlphaVantage ( Config +import Web.AlphaVantage ( AlphaVantageResponse(..) + , Config , Prices(..) , getDailyPrices ) @@ -74,20 +79,35 @@ -> Bool -> IO [(CommoditySymbol, [(Day, Prices)])] fetchPrices cfg symbols start end rateLimit = do - let action symbol = try (getDailyPrices cfg symbol start end) >>= \case - Left (e :: SomeException) -> do - putStrLn - $ "Error Fetching Symbol `" - <> T.unpack symbol - <> "`: " - ++ show e - return Nothing - Right prices -> return $ Just (symbol, prices) if rateLimit then fmap catMaybes $ rateLimitActions $ map action symbols else catMaybes <$> mapM action symbols + where + action :: CommoditySymbol -> IO (Maybe (CommoditySymbol, [(Day, Prices)])) + action symbol = try (getDailyPrices cfg symbol start end) >>= \case + Left (e :: SomeException) -> do + logError + $ "Error Fetching Prices for Symbol `" + <> T.unpack symbol + <> "`:\n\t" + ++ displayException e + ++ "\n" + return Nothing + Right (ApiError note) -> do + logError + $ "Error Fetching Prices for Symbol `" + <> T.unpack symbol + <> "`:\n\t" + <> T.unpack note + <> "\n" + return Nothing + Right (ApiResponse prices) -> return $ Just (symbol, prices) + logError :: String -> IO () + logError = hPutStrLn stderr + + -- | Perform the actions at a rate of 5 per second, then return all the -- results. -- @@ -111,7 +131,7 @@ -- | Build the Price Directives for the Daily Prices of the given -- Commodities. makePriceDirectives :: [(CommoditySymbol, [(Day, Prices)])] -> LBS.ByteString -makePriceDirectives = LBS.intercalate "\n\n" . map makeDirectives +makePriceDirectives = (<> "\n") . LBS.intercalate "\n\n" . map makeDirectives where makeDirectives :: (CommoditySymbol, [(Day, Prices)]) -> LBS.ByteString makeDirectives (symbol, prices) = diff --git a/src/Web/AlphaVantage.hs b/src/Web/AlphaVantage.hs --- a/src/Web/AlphaVantage.hs +++ b/src/Web/AlphaVantage.hs @@ -1,3 +1,4 @@ +{-# LANGUAGE DeriveFunctor #-} {-# LANGUAGE DeriveGeneric #-} {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE RecordWildCards #-} @@ -9,37 +10,39 @@ -} module Web.AlphaVantage ( Config(..) + , AlphaVantageResponse(..) , Prices(..) , getDailyPrices - ) -where + ) where import Data.Aeson ( (.:) + , (.:?) , FromJSON(..) + , Value(Object) , withObject ) import Data.Scientific ( Scientific ) import Data.Time ( Day - , parseTimeM , defaultTimeLocale + , parseTimeM ) import GHC.Generics ( Generic ) import Network.HTTP.Req ( (/~) , (=:) , GET(..) , NoReqBody(..) - , runReq - , req , defaultHttpConfig , https , jsonResponse + , req , responseBody + , runReq ) import Text.Read ( readMaybe ) -import qualified Data.Text as T import qualified Data.HashMap.Strict as HM import qualified Data.List as L +import qualified Data.Text as T -- | Configuration for the AlphaVantage API Client. @@ -49,7 +52,22 @@ -- ^ Your API Key. } deriving (Show, Read, Eq, Generic) +-- | Wrapper type enumerating between successful responses and error +-- responses with notes. +data AlphaVantageResponse a + = ApiResponse a + | ApiError T.Text + deriving (Show, Read, Eq, Generic, Functor) +-- | Check for errors by attempting to parse a `Note` field. If one does +-- not exist, parse the inner type. +instance FromJSON a => FromJSON (AlphaVantageResponse a) where + parseJSON = withObject "AlphaVantageResponse" $ \v -> do + mbErrorNote <- v .:? "Note" + case mbErrorNote of + Nothing -> ApiResponse <$> parseJSON (Object v) + Just note -> return $ ApiError note + -- | List of Daily Prices for a Stock. newtype PriceList = PriceList @@ -66,14 +84,14 @@ where parseDay = parseTimeM True defaultTimeLocale "%F" -- | The Single-Day Price Quotes & Volume for a Stock,. -data Prices = - Prices - { pOpen :: Scientific - , pHigh :: Scientific - , pLow :: Scientific - , pClose :: Scientific - , pVolume :: Integer - } deriving (Show, Read, Eq, Generic) +data Prices = Prices + { pOpen :: Scientific + , pHigh :: Scientific + , pLow :: Scientific + , pClose :: Scientific + , pVolume :: Integer + } + deriving (Show, Read, Eq, Generic) instance FromJSON Prices where parseJSON = withObject "Prices" $ \v -> do @@ -88,12 +106,17 @@ val <- parser case readMaybe val of Just x -> return x - Nothing -> fail $ "Could not read: " ++ val + Nothing -> fail $ "Could not parse number: " ++ val -- | Fetch the Daily Prices for a Stock, returning only the prices between -- the two given dates. -getDailyPrices :: Config -> T.Text -> Day -> Day -> IO [(Day, Prices)] +getDailyPrices + :: Config + -> T.Text + -> Day + -> Day + -> IO (AlphaVantageResponse [(Day, Prices)]) getDailyPrices cfg symbol startDay endDay = do resp <- runReq defaultHttpConfig $ req GET @@ -111,9 +134,11 @@ <> "apikey" =: cApiKey cfg ) - return - . takeWhile ((<= endDay) . fst) - . dropWhile ((< startDay) . fst) - . L.sortOn fst - . fromPriceList - $ responseBody resp + return . fmap filterByDate $ responseBody resp + where + filterByDate :: PriceList -> [(Day, Prices)] + filterByDate = + takeWhile ((<= endDay) . fst) + . dropWhile ((< startDay) . fst) + . L.sortOn fst + . fromPriceList