diff --git a/CHANGELOG.md b/CHANGELOG.md --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,23 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [Unreleased] + +## [0.1.0.2] - 2026-07-15 + +### Added + +- `providers.json` provider catalog: configure provider endpoints, API key env + vars, and protocols without hardcoding providers in Haskell. +- `loadProviderCatalog`, `ProviderCatalogItem`, and `loadGatewaysFromCatalog` + for explicit provider catalog loading. +- Optional `baseUrlEnv` overrides for OpenAI, DeepSeek, and Ollama. + +### Changed + +- Partial `providers.json` files are merged with built-in provider defaults; + file entries add or override providers by `providerName`. + ## [0.1.0.1] - 2026-07-11 ### Changed @@ -42,5 +59,6 @@ DeepSeek), single-shot generation with fallbacks, agent tool loops, structured output, JSON model catalog loading, and workspace-scoped filesystem tools. +[0.1.0.2]: https://github.com/aische/llm-simple/compare/v0.1.0.1...v0.1.0.2 [0.1.0.1]: https://github.com/aische/llm-simple/compare/v0.1.0.0...v0.1.0.1 [0.1.0.0]: https://github.com/aische/llm-simple/releases/tag/v0.1.0.0 diff --git a/Readme.md b/Readme.md --- a/Readme.md +++ b/Readme.md @@ -2,7 +2,7 @@ An experimental Haskell **LLM toolbox** for trying out providers, tool loops, and workspace agents locally. It is meant to get you from zero to a working agent quickly — not to be a production agent platform. -The API stays small on purpose: a JSON model catalog, bundled filesystem tools, and a straight line from `loadModelOrThrow` to `generateText`. Under that surface you still get multi-provider gateways, fallbacks, streaming, structured output, and sandboxed workspace tools — enough to prototype and compare ideas without assembling the pieces yourself. +The API stays small on purpose: JSON model and provider catalogs, bundled filesystem tools, and a straight line from `loadModelOrThrow` to `generateText`. Under that surface you still get multi-provider gateways, fallbacks, streaming, structured output, and sandboxed workspace tools — enough to prototype and compare ideas without assembling the pieces yourself. **Status:** early 0.1.x release. APIs may change. @@ -12,7 +12,7 @@ - **Generate:** single-request text, streaming, and structured object generation with model fallbacks, timeouts, retries, and throttling - **Agent:** multi-round tool loops (`generateText`, `streamText`, `generateObject`) - **Tools:** filesystem tool suite with workspace path sandboxing -- **Load:** JSON model catalog and API-key gateway initialization +- **Load:** JSON model and provider catalogs, API-key gateway initialization - **Observability:** hooks, logging, and generation lifecycle events ## Install @@ -59,7 +59,7 @@ | Field | Description | | ----------------- | ------------------------------------------------------------------------ | | `modelConfigName` | Name used in code to look up this config | -| `providerName` | `"openai"`, `"claude"`, `"gemini"`, `"ollama"`, or `"deepseek"` | +| `providerName` | Provider key defined in `providers.json` | | `modelName` | Provider-specific model identifier | | `pricing` | `pricePerMillionInput` / `pricePerMillionOutput` for usage cost tracking | | `maxTokens` | Max tokens per request | @@ -72,6 +72,33 @@ A provider is only available if its API key is set (except Ollama, which is always available). +### Provider catalog + +Providers are defined in `providers.json` (bundled with the package as a +reference). Built-in providers are always available. When you load a model +catalog, the loader looks for `providers.json` in the same directory; if present, +its entries are merged on top of the built-ins — adding new providers or +overriding built-ins with the same `providerName`. + +| Field | Description | +| ------------- | --------------------------------------------------------------------------- | +| `providerName`| Key referenced by `providerName` in model catalog entries | +| `protocol` | `"openai"`, `"claude"`, `"gemini"`, `"ollama"`, or `"deepseek"` | +| `baseUrl` | Provider host/base URL (OpenAI-compatible protocols append `/v1/chat/completions`) | +| `apiKeyEnv` | Environment variable holding the API key (omit for keyless providers) | +| `baseUrlEnv` | Optional env var that overrides `baseUrl` at runtime | + +To add an OpenAI-compatible provider (OpenRouter, Groq, Together, etc.), add an +entry with `"protocol": "openai"`, set `baseUrl` and `apiKeyEnv`, then reference +`providerName` from your model catalog. + +Optional base URL overrides for built-in providers: `OPENAI_BASE_URL`, +`DEEPSEEK_BASE_URL`, `OLLAMA_BASE_URL`. + +`loadGateways` and `loadGatewaysWithDotenv` always use built-in provider defaults. +Custom providers from a co-located `providers.json` apply when loading models via +`loadModelOrThrow` or `loadModelsOrThrow`. + ### Loading models ```haskell @@ -202,7 +229,7 @@ | `LLM.Generate` | Single-request generation with fallbacks (`generateTextWithFallbacks`, `streamTextWithFallbacks`, `genObject`) | | `LLM.Agent` | Agent loops with tool execution (`generateText`, `streamText`) | | `LLM.Tools` | Built-in filesystem tool definitions | -| `LLM.Load` | Model catalog loading, gateway init, `fsTools` | +| `LLM.Load` | Model and provider catalog loading, gateway init, `fsTools` | Import `LLM` for the common surface, or import sub-modules directly for agent loops, providers, and advanced configuration. Sub-module APIs are exposed but considered experimental in 0.1.x. diff --git a/llm-simple.cabal b/llm-simple.cabal --- a/llm-simple.cabal +++ b/llm-simple.cabal @@ -2,7 +2,7 @@ name: llm-simple -version: 0.1.0.1 +version: 0.1.0.2 synopsis: Multi-provider LLM library with agent tool loops and filesystem tools @@ -13,8 +13,8 @@ . Features multi-provider gateways (OpenAI, Claude, Gemini, Ollama, DeepSeek), single-shot text/stream/object generation with model fallbacks, agent tool - loops, structured output via Autodocodec, JSON model-catalog loading, and - workspace-scoped filesystem tools. + loops, structured output via Autodocodec, JSON model- and provider-catalog + loading, and workspace-scoped filesystem tools. . Status: early 0.1.x release — APIs may change. @@ -40,6 +40,7 @@ extra-source-files: model-catalog.json + providers.json tested-with: GHC ==9.6.7 @@ -127,6 +128,7 @@ LLM.Load LLM.Load.LoadGateways LLM.Load.ModelCatalog + LLM.Load.ProviderCatalog LLM.Load.LoadModels LLM.Load.Types LLM.Load.Utils @@ -239,5 +241,6 @@ hspec >=2.10 && <2.12, heptapod >=1.1 && <1.2, llm-simple, + mtl >=2.3 && <2.4, retry >=0.9 && <0.10 build-tool-depends: hspec-discover:hspec-discover diff --git a/providers.json b/providers.json new file mode 100644 --- /dev/null +++ b/providers.json @@ -0,0 +1,34 @@ +[ + { + "providerName": "openai", + "protocol": "openai", + "baseUrl": "https://api.openai.com", + "apiKeyEnv": "OPENAI_API_KEY", + "baseUrlEnv": "OPENAI_BASE_URL" + }, + { + "providerName": "claude", + "protocol": "claude", + "baseUrl": "https://api.anthropic.com", + "apiKeyEnv": "CLAUDE_API_KEY" + }, + { + "providerName": "gemini", + "protocol": "gemini", + "baseUrl": "https://generativelanguage.googleapis.com", + "apiKeyEnv": "GEMINI_API_KEY" + }, + { + "providerName": "deepseek", + "protocol": "deepseek", + "baseUrl": "https://api.deepseek.com", + "apiKeyEnv": "DEEPSEEK_API_KEY", + "baseUrlEnv": "DEEPSEEK_BASE_URL" + }, + { + "providerName": "ollama", + "protocol": "ollama", + "baseUrl": "http://localhost:11434", + "baseUrlEnv": "OLLAMA_BASE_URL" + } +] diff --git a/src/LLM/Load.hs b/src/LLM/Load.hs --- a/src/LLM/Load.hs +++ b/src/LLM/Load.hs @@ -22,46 +22,54 @@ -- -- @ -- --- available providers: --- --- * "openai" --- --- * "claude" --- --- * "gemini" --- --- * "ollama" --- --- * "deepseek" --- +-- available providers come from built-in defaults. Place an optional +-- @providers.json@ next to your model catalog to add OpenAI-compatible +-- providers or override built-in endpoints; file entries merge on top of +-- the defaults. -- -- If a model (resp. its provider) requires an API key, it must be set in the -- process environment. This module does not load a @.env@ file automatically; -- use @Configuration.Dotenv.loadFile@ in your @main@ or call -- 'loadGatewaysWithDotenv' when building gateways yourself. -- +-- Built-in API key environment variables: +-- -- @ -- GEMINI_API_KEY=... -- CLAUDE_API_KEY=... -- OPENAI_API_KEY=... -- DEEPSEEK_API_KEY=... -- @ +-- +-- Optional base URL overrides: @OPENAI_BASE_URL@, @DEEPSEEK_BASE_URL@, +-- @OLLAMA_BASE_URL@. module LLM.Load ( loadModelsOrThrow, loadModelOrThrow, LoadConfigError (..), loadGateways, loadGatewaysWithDotenv, + loadGatewaysFromCatalog, loadModelCatalog, + loadProviderCatalog, ModelCatalogItem (..), ModelCatalogMap, + ProviderCatalogItem (..), + ProviderCatalogMap, + ProviderProtocol (..), fsTools, fsTools', ) where import LLM.Load.FsTools (fsTools, fsTools') -import LLM.Load.LoadGateways (loadGateways, loadGatewaysWithDotenv) +import LLM.Load.LoadGateways (loadGateways, loadGatewaysFromCatalog, loadGatewaysWithDotenv) import LLM.Load.LoadModels (loadModelOrThrow, loadModelsOrThrow) import LLM.Load.ModelCatalog (ModelCatalogItem (..), ModelCatalogMap, loadModelCatalog) +import LLM.Load.ProviderCatalog + ( ProviderCatalogItem (..), + ProviderCatalogMap, + ProviderProtocol (..), + loadProviderCatalog, + ) import LLM.Load.Types (LoadConfigError (..)) diff --git a/src/LLM/Load/LoadGateways.hs b/src/LLM/Load/LoadGateways.hs --- a/src/LLM/Load/LoadGateways.hs +++ b/src/LLM/Load/LoadGateways.hs @@ -1,46 +1,188 @@ module LLM.Load.LoadGateways ( GatewayMap, + ProviderBaseUrl (..), + buildGateway, loadGateways, loadGatewaysWithDotenv, + loadGatewaysFromCatalog, + loadGatewaysFromCatalogOrThrow, + parseProviderBaseUrl, ) where import Configuration.Dotenv (defaultConfig, loadFile) -import Control.Exception (SomeException, catch) -import Control.Lens ((<&>)) +import Control.Exception (SomeException, catch, throwIO) +import Control.Monad (forM) +import Control.Monad.Except (ExceptT (..), MonadError (throwError), runExceptT) +import Control.Monad.IO.Class (MonadIO (liftIO)) +import Data.Functor ((<&>)) import Data.Map qualified as Map +import Data.ByteString.Char8 qualified as B import Data.Maybe (catMaybes) import Data.Text (Text) import Data.Text qualified as T import LLM.Core.Types (LLMGateway) +import LLM.Load.ProviderCatalog + ( ProviderCatalogItem (..), + ProviderCatalogMap, + ProviderProtocol (..), + defaultProviderCatalogMap, + ) +import LLM.Load.Types (LoadConfigError (LoadModelConfigError)) import LLM.Providers.Claude (claudeGateway) -import LLM.Providers.DeepSeek (deepSeekGateway) +import LLM.Providers.DeepSeek (deepSeekGatewayWith) import LLM.Providers.Gemini (geminiGateway) -import LLM.Providers.Ollama (ollamaGateway) -import LLM.Providers.OpenAI (openAIGateway) +import LLM.Providers.Ollama (ollamaGatewayWith) +import LLM.Providers.OpenAI (openAIGatewayWithName) +import Network.HTTP.Client qualified as HC +import Network.HTTP.Req + ( Option, + Scheme (Http, Https), + Url, + http, + https, + port, + (/:), + ) import System.Environment (lookupEnv) type GatewayMap = Map.Map Text LLMGateway --- | Build a gateway map from the current process environment. --- Does not read a @.env@ file; set variables yourself or use --- 'loadGatewaysWithDotenv'. +data ProviderBaseUrl + = HttpsProviderBase (Url 'Https) (Option 'Https) + | HttpProviderBase (Url 'Http) (Option 'Http) + +parseProviderBaseUrl :: Text -> Either Text ProviderBaseUrl +parseProviderBaseUrl urlText = + case HC.parseRequest (T.unpack urlText) of + Left err -> Left $ "invalid baseUrl: " <> urlText <> ": " <> T.pack (show err) + Right req -> do + let hostText = T.pack (B.unpack (HC.host req)) + reqPort = HC.port req + pathSegments = + map T.pack $ + filter (not . null) $ + splitOnChar '/' $ + B.unpack (HC.path req) + mbPort + | HC.secure req && reqPort == 443 = Nothing + | not (HC.secure req) && reqPort == 80 = Nothing + | otherwise = Just reqPort + if HC.secure req + then + let (url, opts) = httpsBase hostText mbPort + in Right $ HttpsProviderBase (applyPath pathSegments url) opts + else + let (url, opts) = httpBase hostText mbPort + in Right $ HttpProviderBase (applyPath pathSegments url) opts + where + splitOnChar _ [] = [] + splitOnChar c xs = + let (h, t) = break (== c) xs + in h : case dropWhile (== c) t of + [] -> [] + ys -> splitOnChar c ys + httpsBase h Nothing = (https h, mempty) + httpsBase h (Just p) = (https h, port p) + httpBase h Nothing = (http h, mempty) + httpBase h (Just p) = (http h, port p) + applyPath segments url = foldl (/:) url segments + +buildGateway :: ProviderCatalogItem -> ProviderBaseUrl -> Text -> LLMGateway +buildGateway item baseUrl apiKey = + case item.protocol of + ClaudeProtocol -> claudeGateway apiKey + GeminiProtocol -> geminiGateway apiKey + OpenAIProtocol -> case baseUrl of + HttpsProviderBase url opts -> openAIGatewayWithName item.providerName url opts apiKey + HttpProviderBase url opts -> openAIGatewayWithName item.providerName url opts apiKey + DeepSeekProtocol -> case baseUrl of + HttpsProviderBase url opts -> deepSeekGatewayWith url opts apiKey + HttpProviderBase url opts -> deepSeekGatewayWith url opts apiKey + OllamaProtocol -> case baseUrl of + HttpsProviderBase url opts -> ollamaGatewayWith url opts + HttpProviderBase url opts -> ollamaGatewayWith url opts + +resolveBaseUrl :: ProviderCatalogItem -> IO Text +resolveBaseUrl item = + case item.baseUrlEnv of + Nothing -> pure item.baseUrl + Just envVar -> do + mbOverride <- lookupEnv (T.unpack envVar) + pure $ maybe item.baseUrl T.pack mbOverride + +tryBuildGateway :: ProviderCatalogItem -> IO (Maybe (Text, LLMGateway)) +tryBuildGateway item = + case item.apiKeyEnv of + Nothing -> buildWithKey "" + Just envVar -> + lookupEnv (T.unpack envVar) >>= (\case + Nothing -> pure Nothing + Just apiKey -> buildWithKey apiKey) . fmap T.pack + where + buildWithKey apiKey = do + baseUrlText <- resolveBaseUrl item + case parseProviderBaseUrl baseUrlText of + Left _ -> pure Nothing + Right baseUrl -> + pure $ + Just + ( item.providerName, + buildGateway item baseUrl apiKey + ) + +tryBuildGatewayOrThrow :: ProviderCatalogItem -> ExceptT LoadConfigError IO (Maybe (Text, LLMGateway)) +tryBuildGatewayOrThrow item = + case item.apiKeyEnv of + Nothing -> buildWithKey "" + Just envVar -> + liftIO (lookupEnv (T.unpack envVar) <&> fmap T.pack) >>= \case + Nothing -> pure Nothing + Just apiKey -> buildWithKey apiKey + where + buildWithKey apiKey = do + baseUrlText <- liftIO $ resolveBaseUrl item + baseUrl <- case parseProviderBaseUrl baseUrlText of + Left err -> + throwError $ + LoadModelConfigError $ + "Invalid baseUrl for provider " + <> item.providerName + <> ": " + <> err + Right parsed -> pure parsed + pure $ + Just + ( item.providerName, + buildGateway item baseUrl apiKey + ) + +loadGatewaysFromCatalog :: ProviderCatalogMap -> IO GatewayMap +loadGatewaysFromCatalog catalog = do + pairs <- forM (Map.elems catalog) tryBuildGateway + pure $ Map.fromList $ catMaybes pairs + +loadGatewaysFromCatalogOrThrow :: ProviderCatalogMap -> IO GatewayMap +loadGatewaysFromCatalogOrThrow catalog = do + result <- runExceptT $ do + pairs <- forM (Map.elems catalog) tryBuildGatewayOrThrow + pure $ Map.fromList $ catMaybes pairs + case result of + Left err -> throwIO err + Right gatewayMap -> pure gatewayMap + +-- | Build a gateway map from the bundled default provider catalog and the +-- current process environment. Does not read a @.env@ file; set variables +-- yourself or use 'loadGatewaysWithDotenv'. +-- +-- For custom providers, place a @providers.json@ next to your model catalog; +-- 'loadModelOrThrow' loads that file automatically. loadGateways :: IO GatewayMap -loadGateways = loadGatewaysFromEnv +loadGateways = loadGatewaysFromCatalog defaultProviderCatalogMap --- | Load @.env@ (if present) and then build a gateway map from the --- environment. Convenience for local development; prefer explicit env --- injection in production. +-- | Load @.env@ (if present) and then build a gateway map from the bundled +-- default provider catalog. loadGatewaysWithDotenv :: IO GatewayMap loadGatewaysWithDotenv = do loadFile defaultConfig `catch` \(_ :: SomeException) -> pure () - loadGatewaysFromEnv - -loadGatewaysFromEnv :: IO GatewayMap -loadGatewaysFromEnv = do - let ollama = Just ("ollama", ollamaGateway) - openai <- lookupEnv "OPENAI_API_KEY" <&> fmap (("openai",) . openAIGateway . T.pack) - claude <- lookupEnv "CLAUDE_API_KEY" <&> fmap (("claude",) . claudeGateway . T.pack) - gemini <- lookupEnv "GEMINI_API_KEY" <&> fmap (("gemini",) . geminiGateway . T.pack) - deepseek <- lookupEnv "DEEPSEEK_API_KEY" <&> fmap (("deepseek",) . deepSeekGateway . T.pack) - pure $ Map.fromList $ catMaybes [openai, claude, gemini, deepseek, ollama] + loadGateways diff --git a/src/LLM/Load/LoadModels.hs b/src/LLM/Load/LoadModels.hs --- a/src/LLM/Load/LoadModels.hs +++ b/src/LLM/Load/LoadModels.hs @@ -20,15 +20,17 @@ import Data.Text (Text) import LLM.Core.Types (ThinkingMode (..)) import LLM.Generate.ModelConfig (ModelConfig (..)) -import LLM.Load.LoadGateways (GatewayMap, loadGateways) +import LLM.Load.LoadGateways (GatewayMap, loadGatewaysFromCatalogOrThrow) import LLM.Load.ModelCatalog (ModelCatalogItem (..), loadModelCatalog) +import LLM.Load.ProviderCatalog (loadProviderCatalogForModelCatalog) import LLM.Load.Types (LoadConfigError (LoadModelConfigError)) type ModelConfigMap = Map Text ModelConfig loadModelConfigMap :: FilePath -> ExceptT LoadConfigError IO (ModelConfigMap, GatewayMap) loadModelConfigMap filePath = do - gatewayMap <- liftIO loadGateways + providerCatalog <- loadProviderCatalogForModelCatalog filePath + gatewayMap <- liftIO $ loadGatewaysFromCatalogOrThrow providerCatalog modelCatalogMap <- loadModelCatalog filePath modelConfigs <- forM (Map.toList modelCatalogMap) $ \(modelConfigName, item) -> do case Map.lookup item.providerName gatewayMap of diff --git a/src/LLM/Load/ModelCatalog.hs b/src/LLM/Load/ModelCatalog.hs --- a/src/LLM/Load/ModelCatalog.hs +++ b/src/LLM/Load/ModelCatalog.hs @@ -13,7 +13,7 @@ data ModelCatalogItem = ModelCatalogItem { modelConfigName :: Text, - providerName :: Text, -- provider name: "openai", "claude", "gemini", "ollama" + providerName :: Text, -- key from providers.json (built-in or custom) modelName :: Text, pricing :: PricingInfo, maxTokens :: Int, diff --git a/src/LLM/Load/ProviderCatalog.hs b/src/LLM/Load/ProviderCatalog.hs new file mode 100644 --- /dev/null +++ b/src/LLM/Load/ProviderCatalog.hs @@ -0,0 +1,134 @@ +module LLM.Load.ProviderCatalog + ( ProviderProtocol (..), + ProviderCatalogItem (..), + ProviderCatalogMap, + defaultProviderCatalogMap, + loadProviderCatalog, + loadProviderCatalogForModelCatalog, + ) +where + +import Control.Monad.Except (ExceptT (..)) +import Control.Monad.IO.Class (liftIO) +import Data.Aeson (FromJSON (parseJSON), ToJSON (toJSON), withText) +import Data.Map (Map) +import Data.Map qualified as Map +import Data.Text (Text) +import Data.Text qualified as T +import GHC.Generics (Generic) +import LLM.Load.Types (LoadConfigError (..)) +import LLM.Load.Utils (decodeJsonFile) +import System.Directory (doesFileExist) +import System.FilePath ((), takeDirectory) + +data ProviderProtocol + = OpenAIProtocol + | ClaudeProtocol + | GeminiProtocol + | OllamaProtocol + | DeepSeekProtocol + deriving (Show, Eq, Ord) + +instance ToJSON ProviderProtocol where + toJSON = toJSON . protocolToText + +instance FromJSON ProviderProtocol where + parseJSON = withText "ProviderProtocol" $ \t -> + case T.toLower t of + "openai" -> pure OpenAIProtocol + "claude" -> pure ClaudeProtocol + "gemini" -> pure GeminiProtocol + "ollama" -> pure OllamaProtocol + "deepseek" -> pure DeepSeekProtocol + other -> fail $ "unknown provider protocol: " <> T.unpack other + +protocolToText :: ProviderProtocol -> Text +protocolToText = \case + OpenAIProtocol -> "openai" + ClaudeProtocol -> "claude" + GeminiProtocol -> "gemini" + OllamaProtocol -> "ollama" + DeepSeekProtocol -> "deepseek" + +data ProviderCatalogItem = ProviderCatalogItem + { providerName :: Text, + protocol :: ProviderProtocol, + baseUrl :: Text, + apiKeyEnv :: Maybe Text, + baseUrlEnv :: Maybe Text + } + deriving (Show, Eq, Ord, Generic, FromJSON, ToJSON) + +type ProviderCatalogMap = Map Text ProviderCatalogItem + +defaultProviderCatalogMap :: ProviderCatalogMap +defaultProviderCatalogMap = + Map.fromList + [ ("openai", openaiItem), + ("claude", claudeItem), + ("gemini", geminiItem), + ("deepseek", deepseekItem), + ("ollama", ollamaItem) + ] + where + openaiItem = + ProviderCatalogItem + { providerName = "openai", + protocol = OpenAIProtocol, + baseUrl = "https://api.openai.com", + apiKeyEnv = Just "OPENAI_API_KEY", + baseUrlEnv = Just "OPENAI_BASE_URL" + } + claudeItem = + ProviderCatalogItem + { providerName = "claude", + protocol = ClaudeProtocol, + baseUrl = "https://api.anthropic.com", + apiKeyEnv = Just "CLAUDE_API_KEY", + baseUrlEnv = Nothing + } + geminiItem = + ProviderCatalogItem + { providerName = "gemini", + protocol = GeminiProtocol, + baseUrl = "https://generativelanguage.googleapis.com", + apiKeyEnv = Just "GEMINI_API_KEY", + baseUrlEnv = Nothing + } + deepseekItem = + ProviderCatalogItem + { providerName = "deepseek", + protocol = DeepSeekProtocol, + baseUrl = "https://api.deepseek.com", + apiKeyEnv = Just "DEEPSEEK_API_KEY", + baseUrlEnv = Just "DEEPSEEK_BASE_URL" + } + ollamaItem = + ProviderCatalogItem + { providerName = "ollama", + protocol = OllamaProtocol, + baseUrl = "http://localhost:11434", + apiKeyEnv = Nothing, + baseUrlEnv = Just "OLLAMA_BASE_URL" + } + +loadProviderCatalog :: FilePath -> ExceptT LoadConfigError IO ProviderCatalogMap +loadProviderCatalog filePath = + Map.fromList . fmap (\item -> (item.providerName, item)) + <$> decodeJsonFile filePath (LoadProviderCatalogError . T.pack) + +-- | Load providers for a model catalog path. +-- +-- Built-in providers are always available. When @providers.json@ exists in the +-- same directory as the model catalog, its entries are merged on top: file +-- entries add new providers or override built-ins with the same +-- @providerName@. +loadProviderCatalogForModelCatalog :: FilePath -> ExceptT LoadConfigError IO ProviderCatalogMap +loadProviderCatalogForModelCatalog modelCatalogPath = do + let providerPath = takeDirectory modelCatalogPath "providers.json" + exists <- liftIO $ doesFileExist providerPath + fileCatalog <- + if exists + then loadProviderCatalog providerPath + else pure Map.empty + pure $ Map.union fileCatalog defaultProviderCatalogMap diff --git a/src/LLM/Load/Types.hs b/src/LLM/Load/Types.hs --- a/src/LLM/Load/Types.hs +++ b/src/LLM/Load/Types.hs @@ -7,6 +7,8 @@ data LoadConfigError = -- | The catalog file is missing or contains invalid JSON. LoadModelCatalogError Text + | -- | The provider catalog file is missing or contains invalid JSON. + LoadProviderCatalogError Text | -- | A catalog entry references an unknown model or provider. LoadModelConfigError Text | -- | A @file:@ system prompt path could not be read. diff --git a/src/LLM/Providers/Ollama.hs b/src/LLM/Providers/Ollama.hs --- a/src/LLM/Providers/Ollama.hs +++ b/src/LLM/Providers/Ollama.hs @@ -35,7 +35,6 @@ ( Option, POST (POST), ReqBodyJson (ReqBodyJson), - Scheme (Http), Url, http, jsonResponse, @@ -53,7 +52,7 @@ ollamaGateway = toGateway ollamaProvider -- | Create a LLMGateway for a custom Ollama instance. -ollamaGatewayWith :: Url 'Http -> Option 'Http -> LLMGateway +ollamaGatewayWith :: Url scheme -> Option scheme -> LLMGateway ollamaGatewayWith baseUrl baseOpts = toGateway (ollamaProviderWith baseUrl baseOpts) -- | Default Ollama provider at localhost:11434. diff --git a/src/LLM/Providers/OpenAI.hs b/src/LLM/Providers/OpenAI.hs --- a/src/LLM/Providers/OpenAI.hs +++ b/src/LLM/Providers/OpenAI.hs @@ -1,8 +1,10 @@ module LLM.Providers.OpenAI ( openAIGateway, openAIGatewayWith, + openAIGatewayWithName, openAIProvider, openAIProviderWith, + openAIProviderWithName, parseOpenAIResponse, parseOpenAIUsage, buildMessages, @@ -85,16 +87,23 @@ -- | Create an OpenAI-compatible client with a custom base URL. openAIGatewayWith :: Url scheme -> Option scheme -> Text -> LLMGateway -openAIGatewayWith baseUrl baseOpts apiKey = toGateway (openAIProviderWith baseUrl baseOpts apiKey) +openAIGatewayWith = openAIGatewayWithName "openai" +-- | Create an OpenAI-compatible client with a custom provider name and base URL. +openAIGatewayWithName :: Text -> Url scheme -> Option scheme -> Text -> LLMGateway +openAIGatewayWithName name baseUrl baseOpts apiKey = toGateway (openAIProviderWithName name baseUrl baseOpts apiKey) + -- | Create an OpenAI provider. Takes the API key as a parameter. openAIProvider :: Text -> LLMProvider openAIProvider = openAIProviderWith (https "api.openai.com") mempty openAIProviderWith :: Url scheme -> Option scheme -> Text -> LLMProvider -openAIProviderWith baseUrl baseOpts apiKey = +openAIProviderWith = openAIProviderWithName "openai" + +openAIProviderWithName :: Text -> Url scheme -> Option scheme -> Text -> LLMProvider +openAIProviderWithName name baseUrl baseOpts apiKey = LLMProvider - { providerName = "openai", + { providerName = name, buildBody = openAIBuildBody, sendRequest = sendRequest, sendStreamRequest = \body callback -> diff --git a/test/LLM/LoadSpec.hs b/test/LLM/LoadSpec.hs --- a/test/LLM/LoadSpec.hs +++ b/test/LLM/LoadSpec.hs @@ -1,14 +1,23 @@ module LLM.LoadSpec (spec) where import Control.Exception (IOException, SomeException, catch, displayException, fromException, try) +import Control.Monad.Except (runExceptT) import Data.Aeson (Value (Array), eitherDecodeFileStrict') import Data.Map qualified as Map import Data.Maybe (isJust) import Data.Text (Text) import Data.Text qualified as T +import LLM.Core.Types (LLMGateway (..)) import LLM.Generate.ModelConfig (ModelConfig (..)) -import LLM.Load.LoadGateways (loadGateways) +import LLM.Load.LoadGateways (buildGateway, loadGateways, loadGatewaysFromCatalog, parseProviderBaseUrl) import LLM.Load.LoadModels (loadModelOrThrow, loadModelOrThrow_, loadModelsOrThrow) +import LLM.Load.ProviderCatalog + ( ProviderCatalogItem (..), + ProviderProtocol (..), + defaultProviderCatalogMap, + loadProviderCatalog, + loadProviderCatalogForModelCatalog, + ) import LLM.Load.Types (LoadConfigError (..)) import System.FilePath (()) import System.IO.Temp (withSystemTempDirectory) @@ -25,6 +34,12 @@ ollamaCatalog :: FilePath ollamaCatalog = "./test/fixtures/ollama-catalog.json" +providersOllamaOnly :: FilePath +providersOllamaOnly = "./test/fixtures/providers-ollama-only.json" + +providersOpenrouterOnly :: FilePath +providersOpenrouterOnly = "./test/fixtures/providers-openrouter-only.json" + spec :: Spec spec = describe "Load" $ do describe "model catalog JSON" $ do @@ -66,6 +81,66 @@ gateways <- loadGateways Map.member "ollama" gateways `shouldBe` True + it "builds an openai-compatible gateway from a custom provider catalog" $ do + let item = + ProviderCatalogItem + { providerName = "openrouter", + protocol = OpenAIProtocol, + baseUrl = "https://openrouter.ai/api", + apiKeyEnv = Just "OPENROUTER_API_KEY", + baseUrlEnv = Nothing + } + Right baseUrl <- pure $ parseProviderBaseUrl item.baseUrl + let gateway = buildGateway item baseUrl "test-key" + LLMGateway {gwName = name} = gateway + name `shouldBe` "openrouter" + + it "loads gateways from a provider catalog file" $ do + result <- runExceptT $ loadProviderCatalog providersOllamaOnly + case result of + Left err -> expectationFailure $ show err + Right catalog -> do + gateways <- loadGatewaysFromCatalog catalog + Map.keys gateways `shouldSatisfy` ("ollama" `elem`) + + describe "provider catalog JSON" $ do + it "parses the bundled provider catalog" $ do + result <- eitherDecodeFileStrict' "./providers.json" + case result of + Right (Array xs) -> length xs `shouldSatisfy` (> 0) + _ -> expectationFailure "expected a JSON array" + + it "includes built-in providers in the default catalog" $ do + Map.member "openai" defaultProviderCatalogMap `shouldBe` True + Map.member "ollama" defaultProviderCatalogMap `shouldBe` True + + it "merges a partial providers.json with built-in defaults" $ + withProviderCatalogDir providersOpenrouterOnly $ \catalogPath -> do + result <- runExceptT $ loadProviderCatalogForModelCatalog catalogPath + case result of + Left err -> expectationFailure $ show err + Right catalog -> do + Map.member "openrouter" catalog `shouldBe` True + Map.member "openai" catalog `shouldBe` True + Map.member "ollama" catalog `shouldBe` True + + it "lets providers.json override a built-in provider" $ + withProviderCatalogDir providersOllamaOnly $ \catalogPath -> do + result <- runExceptT $ loadProviderCatalogForModelCatalog catalogPath + case result of + Left err -> expectationFailure $ show err + Right catalog -> do + Map.lookup "ollama" catalog + `shouldBe` Just + ProviderCatalogItem + { providerName = "ollama", + protocol = OllamaProtocol, + baseUrl = "http://localhost:11434", + apiKeyEnv = Nothing, + baseUrlEnv = Nothing + } + Map.member "openai" catalog `shouldBe` True + describe "invalid catalog JSON" $ do it "reports a catalog parse/load error" $ withTempCatalog "[not valid json" $ \path -> do result <- try @LoadConfigError (loadModelOrThrow path "x") @@ -80,3 +155,13 @@ let path = root "catalog.json" writeFile path contents act path `catch` \(_ :: IOException) -> act path + +withProviderCatalogDir :: FilePath -> (FilePath -> IO a) -> IO a +withProviderCatalogDir providersFixture act = + withSystemTempDirectory "llm-simple-providers" $ \root -> do + let catalogPath = root "model-catalog.json" + providersPath = root "providers.json" + writeFile catalogPath "[]" + providersJson <- readFile providersFixture + writeFile providersPath providersJson + act catalogPath `catch` \(_ :: IOException) -> act catalogPath