diff --git a/ChangeLog.md b/ChangeLog.md --- a/ChangeLog.md +++ b/ChangeLog.md @@ -1,5 +1,11 @@ # Changelog and Acknowledgements +## 2.2.1 +* Fixed [#26](https://github.com/sjshuck/hs-pcre2/issues/26) where wide UTF-8 + characters were not handled correctly. +* Docs fully updated for UTF-8 instead of UTF-16. (Docs were deleted from + the 2.2.0 release.) + ## 2.2.0 * Switched to UTF-8 to support `text` 2.0, implementing [#22](https://github.com/sjshuck/hs-pcre2/issues/22). `text` < 2 is no longer diff --git a/README.md b/README.md --- a/README.md +++ b/README.md @@ -48,7 +48,7 @@ compile-once-match-many code. * No [custom typeclasses](https://hackage.haskell.org/package/regex-base/docs/Text-Regex-Base-RegexLike.html#t:RegexContext). * A single datatype for both compile and match options, the `Option` monoid. -* `Text` everywhere. +* `Text` everywhere. See [below](#unicode) for a version guide. * Match success expressed via `Alternative`. * Opt-in Template Haskell facilities for compile-time verification of patterns, indexing captures, and memoizing inline regexes. @@ -59,19 +59,31 @@ Both are first-class. * Vast presentation of PCRE2 functionality. We can even register Haskell callbacks to run during matching! -* Zero-copying of substrings where beneficial. Benchmarks show a 10× - speedup over `pcre-light`, and 20× over `regex-pcre`, for longer - captures. +* Zero-copying of substrings where beneficial. * Few dependencies. -* Bundled, statically-linked UTF-8 build of up-to-date PCRE2 (version 10.40), - with a complete, exposed Haskell binding. +* Bundled, statically-linked UTF-8 (formerly UTF-16) build of up-to-date PCRE2 + (version 10.40), with a complete, exposed Haskell binding. +## Performance +Currently we are slower than other libraries. For example: + +| Operation | `pcre2` | `pcre-light` | `regex-pcre-builtin` | +| :-- | --: | --: | --: | +| Compile and match a regex | 3.9 μs | 1.2 μs | 2.9 μs | + +If it's really regex processing that's causing a bottleneck, +[pcre-light](https://hackage.haskell.org/package/pcre-light)/[-heavy](https://hackage.haskell.org/package/pcre-heavy)/[lens-regex-pcre](https://hackage.haskell.org/package/lens-regex-pcre) +are recommended instead of this library for the very best performance. + +## Unicode +| Encoding | `text` version | `pcre2` version | Code unit representation | +| :-- | :-- | :-- | :-- | +| UTF-8 | ≥ 2 | ≥ 2.2 | `Foreign.C.Types.CUChar` | +| UTF-16 | < 2 | < 2.2 | `Foreign.C.Types.CUShort` | + ## Wishlist -* Many performance optimizations. Currently we are as much as 2–3× - slower than other libraries for some operations, although things are - improving. If it's really regex processing that's causing a bottleneck, - [pcre-light](https://hackage.haskell.org/package/pcre-light)/[-heavy](https://hackage.haskell.org/package/pcre-heavy)/[lens-regex-pcre](https://hackage.haskell.org/package/lens-regex-pcre) - are recommended instead of this library for the very best performance. +* Many performance optimizations. Currently we are slower than other libraries. + For example: * Make use of DFA matching and JIT compilation. * Improve PCRE2 C compile time. * Add splitting support. diff --git a/pcre2.cabal b/pcre2.cabal --- a/pcre2.cabal +++ b/pcre2.cabal @@ -5,7 +5,7 @@ -- see: https://github.com/sol/hpack name: pcre2 -version: 2.2.0 +version: 2.2.1 synopsis: Regular expressions via the PCRE2 C library (included) description: Please see the README on GitHub at category: Text diff --git a/src/hs/Text/Regex/Pcre2.hs b/src/hs/Text/Regex/Pcre2.hs --- a/src/hs/Text/Regex/Pcre2.hs +++ b/src/hs/Text/Regex/Pcre2.hs @@ -7,7 +7,7 @@ to add regular expressions to Haskell programs. All input and output strings are strict `Data.Text.Text`, which maps - directly to how PCRE2 operates on strings of 16-bit-wide code units. + directly to how PCRE2 operates on strings of 8-bit-wide code units. The C API requires pattern strings to be compiled and the compiled patterns to be executed on subject strings in discrete steps. We hide this diff --git a/src/hs/Text/Regex/Pcre2/Foreign.hs b/src/hs/Text/Regex/Pcre2/Foreign.hs --- a/src/hs/Text/Regex/Pcre2/Foreign.hs +++ b/src/hs/Text/Regex/Pcre2/Foreign.hs @@ -475,6 +475,10 @@ constant ''CInt "ERROR_NULL" -- *** UTF-16-specific errors -- +-- /Note: These were accidentally left in for version 2.2.0 which otherwise/ +-- /changed `pcre2` from UTF-16 to UTF-8. They are irrelevant and will be/ +-- /removed in the next minor version./ +-- -- | "Missing low surrogate at end of string" constant ''CInt "ERROR_UTF16_ERR1" -- | "Invalid low surrogate follows high surrogate" diff --git a/src/hs/Text/Regex/Pcre2/Internal.hs b/src/hs/Text/Regex/Pcre2/Internal.hs --- a/src/hs/Text/Regex/Pcre2/Internal.hs +++ b/src/hs/Text/Regex/Pcre2/Internal.hs @@ -641,7 +641,7 @@ codePtr <- pcre2_compile (toCUs pattPtr) (fromIntegral pattCUs) - opts + (opts .|. pcre2_UTF) errorCodePtr errorOffPtr ctxPtr @@ -1286,7 +1286,7 @@ defaultBsr = bsrFromC $ getConfigNumeric pcre2_CONFIG_BSR -- | Which code widths PCRE2 is compiled to operate on. Can be any combination --- of 8, 16, and 32. Should be @[16]@ but provided here for completeness. +-- of 8, 16, and 32. Should be @[8]@ but provided here for completeness. compiledWidths :: [Int] compiledWidths = let bitmap = getConfigNumeric pcre2_CONFIG_COMPILED_WIDTHS diff --git a/test/Spec.hs b/test/Spec.hs --- a/test/Spec.hs +++ b/test/Spec.hs @@ -59,6 +59,10 @@ result `shouldBe` Just "123" result `shouldBe` ["123", "456"] + it ("is in UTF mode" `issue` 26) $ do + matchesOpt (Bsr BsrUnicode) "\\R$" "line separator: \x2028" + `shouldBe` True + describe "lens-powered matching" $ do let _nee :: Traversal' Text Text _nee = _matchOpt (Caseless <> MatchWord) "nee"