tiktoken 1.0.0 → 1.0.1

raw patch · 3 files changed

+13/−4 lines, 3 filesPVP ok

version bump matches the API change (PVP)

API changes (from Hackage documentation)

Files

• CHANGELOG.md +4/−0
• src/Tiktoken.hs +8/−3
• tiktoken.cabal +1/−1

@@ -1,3 +1,7 @@+1.0.1++- Small fixes to documentation+ 1.0.0  - Initial release

@@ -11,8 +11,8 @@ {-# LANGUAGE RecordWildCards       #-}  -- | You can use this module to convert back and forth between a `ByteString`---   and its corresponding tokens using an existing encoding like @cl100k_base@---   or @o200k_base@+--   and its corresponding tokens using an existing encoding like `cl100k_base`+--   or `o200k_base` -- --   Example usage: --@@ -486,7 +486,12 @@          (prefix, suffix) = split rest --- | Tokenizer that is special-token-aware+{-| Use an `Encoding` to tokenize a `ByteString` into smaller `ByteString`s and+    their associated ranks++    This only fails if you provide an `Encoding` that cannot rank all possible+    1-byte sequences+-} toTokensAndRanks :: Encoding -> ByteString -> Maybe [(Int, ByteString)] toTokensAndRanks encoding@Encoding{..} initialBytes =     foldr cons nil (Map.toList specialTokens) initialBytes

@@ -1,6 +1,6 @@ cabal-version:      2.4 name:               tiktoken-version:            1.0.0+version:            1.0.1 synopsis:           Haskell implementation of tiktoken description:        This packages only implements tokenization.  In other words,                     given an existing encoding (`cl100k_base`) you can tokenize