diff --git a/ChangeLog.md b/ChangeLog.md --- a/ChangeLog.md +++ b/ChangeLog.md @@ -1,9 +1,40 @@ # Changelog and Acknowledgements +## 2.0.0 +This release introduces significant breaking changes in order to make the API +smaller, more consistent, and safer. +* Implemented [#18](https://github.com/sjshuck/hs-pcre2/issues/18): + * Removed `matchAll`, `matchAllOpt`, `capturesAll`, and `capturesAllOpt`. + * Upgraded `match`, `matchOpt`, `captures`, and `capturesOpt` to offer their + functionality, respectively. + * Renamed `capturesA` and `capturesAOpt` to `captures` and `capturesOpt`, + replacing the latter two functions altogether. `captures`/-`Opt` were + intended to be extreme + [convenience functions](https://hackage.haskell.org/package/pcre2-1.1.5/docs/Text-Regex-Pcre2.html#v:captures) + that required no special datatypes beyond the `Prelude`. However, this + was of doubtful benefit, since that's false anyway—they required + `Text`, not to mention `{-# LANGUAGE OverloadedStrings #-}`. Their names + are simple and valuable, and no other `Alternative`-producing function has + the naming convention "-`A`", so repurposing their names was in order. +* Moved the callout interface to a new module, `Text.Regex.Pcre2.Unsafe`. This + includes the options `UnsafeCompileRecGuard`, `UnsafeCallout`, + `UnsafeSubCallout`, and `AutoCallout`, and the types `CalloutInfo`, + `CalloutIndex`, `CalloutResult`, `SubCalloutInfo`, and `SubCalloutResult`. +* Also moved option `BadEscapeIsLiteral` there. +* Removed the ineffectual options `DupNames` and `Utf`. + +Other improvements with no API impact: +* Updated PCRE2 to 10.37. +* Replaced copied C files with symlinks, diminishing codebase by 1.5K lines and + simplifying future PCRE2 updates. +* Reduced size of Template Haskell splices to make error messages less + obnoxious. +* Moderate refactoring of internals and documentation. + ## 1.1.5 -* Fixed [#17](https://github.com/sjshuck/pcre2/17), where functions returning - `Alternative` containers were not restricted to single results despite their - documentation. +* Fixed [#17](https://github.com/sjshuck/hs-pcre2/issues/17), where functions + returning `Alternative` containers were not restricted to single results + despite their documentation. * Minor improvements to docs and examples. ## 1.1.4 @@ -21,12 +52,12 @@ ## 1.1.2 * Refactored using the `streaming` library. Fixed - [#11](https://github.com/sjshuck/pcre2/11), where large global matches were - very slow. + [#11](https://github.com/sjshuck/hs-pcre2/issues/11), where large global + matches were very slow. ## 1.1.1 -* Fixed [#12](https://github.com/sjshuck/pcre2/4), where some functions returned - too many match results. +* Fixed [#12](https://github.com/sjshuck/hs-pcre2/issues/12), where some + functions returned too many match results. ## 1.1.0 * Added global matching. @@ -35,8 +66,8 @@ * Changed `capturesOptA` to `capturesAOpt` for naming consistency. ## 1.0.2 -* Fixed [#4](https://github.com/sjshuck/pcre2/4), where multiple named captures - were not type-indexed correctly. +* Fixed [#4](https://github.com/sjshuck/hs-pcre2/4), where multiple named + captures were not type-indexed correctly. * Established automated builds using Github Workflows. Thanks amesgen! ## 1.0.1.1 @@ -44,8 +75,8 @@ Hackage. ## 1.0.1 -* Fixed [#1](https://github.com/sjshuck/pcre2/1), where building on Windows - would succeed but not run. Thanks Andrew! +* Fixed [#1](https://github.com/sjshuck/hs-pcre2/issues/1), where building on + Windows would succeed but not run. Thanks Andrew! * Try to adjust dependency version bounds to get it building on Hackage. Thanks snoyberg! diff --git a/README.md b/README.md --- a/README.md +++ b/README.md @@ -7,13 +7,16 @@ ## Teasers ```haskell -embeddedLicensePlate :: Text -> Maybe Text -embeddedLicensePlate = match "[A-Z]{3}[0-9]{3,4}" +licensePlate :: Text -> Maybe Text +licensePlate = match "[A-Z]{3}[0-9]{3,4}" + +licensePlates :: Text -> [Text] +licensePlates = match "[A-Z]{3}[0-9]{3,4}" ``` ```haskell case "The quick brown fox" of - [regex|brown\s+(?\w+)|] -> Text.putStrLn animal - _ -> error "nothing brown" + [regex|\bbrown\s+(?[A-z]+)\b|] -> Text.putStrLn animal + _ -> error "nothing brown" ``` ```haskell let kv'd = lined . packed . [_regex|(?x) # Extended PCRE2 syntax @@ -39,13 +42,14 @@ ``` ## Features -* Quiet functions with simple types—for the most part it's -`Text` _(pattern)_ `-> Text` _(subject)_ `-> result`. -* Use partial application to create performant, compile-once-match-many code. -* Low cognitive overhead—there's just one custom datatype for both compile - and match options, the `Option` monoid. -* `Text` everywhere for interop with both C and the broader Haskell ecosystem. -* Match failures expressed via `Alternative` or pattern match failures. +* No opaque "`Regex`" object. Instead, quiet functions with simple + types—for the most part it's `Text` _(pattern)_ `-> Text` _(subject)_ + `-> result`. Use partial application to create performant, + compile-once-match-many code. +* No [custom typeclasses](https://hackage.haskell.org/package/regex-base/docs/Text-Regex-Base-RegexLike.html#t:RegexContext). +* A single datatype for both compile and match options, the `Option` monoid. +* `Text` everywhere. +* Match success expressed via `Alternative`. * Opt-in Template Haskell facilities for compile-time verification of patterns, indexing captures, and memoizing inline regexes. * Opt-in `lens` support. @@ -55,8 +59,11 @@ Both are first-class. * Vast presentation of PCRE2 functionality. We can even register Haskell callbacks to run during matching! +* Zero-copying of substrings where beneficial. Benchmarks show a 10× + speedup over `pcre-light`, and 20× over `regex-pcre`, for longer + captures. * Few dependencies. -* Bundled, statically-linked UTF-16 build of up-to-date PCRE2 (version 10.36), +* Bundled, statically-linked UTF-16 build of up-to-date PCRE2 (version 10.37), with a complete, exposed Haskell binding. ## Wishlist @@ -64,11 +71,11 @@ slower than other libraries for some operations, although things are improving. (We are already faster than [regex-base](https://hackage.haskell.org/package/regex-base)/[regex-pcre](https://hackage.haskell.org/package/regex-pcre) - when working with `Text`.) If it's really regex processing that's causing a - bottleneck, + when working with `Text`, even without zero-copying.) If it's really regex + processing that's causing a bottleneck, [pcre-light](https://hackage.haskell.org/package/pcre-light)/[-heavy](https://hackage.haskell.org/package/pcre-heavy)/[lens-regex-pcre](https://hackage.haskell.org/package/lens-regex-pcre) are recommended instead of this library for the very best performance. -* Make use of DFA and JIT compilation. +* Make use of DFA matching and JIT compilation. * Improve PCRE2 C compile time. * Add splitting support. @@ -77,4 +84,4 @@ PCRE2 is distributed under the [3-clause BSD](https://www.pcre.org/licence.txt) license. ## Main Author -©2020 Shlomo Shuck +©2020–2021 Shlomo Shuck diff --git a/bench/Bench.hs b/bench/Bench.hs --- a/bench/Bench.hs +++ b/bench/Bench.hs @@ -21,7 +21,7 @@ main = defaultMain [ bgroup "creating single-use regexes" [ bench "pcre2" $ flip nf strings $ \(patt, subj) -> - let Just (_ :| [bar]) = capturesA (Text.pack patt) (Text.pack subj) + let Just (_ :| [bar]) = captures (Text.pack patt) (Text.pack subj) in Text.unpack bar, bench "regex-pcre-builtin" $ flip nf strings $ \(patt, subj) -> @@ -34,9 +34,9 @@ let r = Text.Regex.PCRE.Light.Char8.compile patt [] Just [_, bar] = Text.Regex.PCRE.Light.Char8.match r subj [] in bar], - + bgroup "strings" [ - let r = capturesA $ Text.pack stringPattern + let r = captures $ Text.pack stringPattern in bench "pcre2" $ flip nf stringSubject $ \subj -> let Just (_ :| [bar]) = r $ Text.pack subj in Text.unpack bar, @@ -117,6 +117,6 @@ where textPattern = Text.pack patt textSubject = Text.pack subj - pcre2R = capturesA textPattern + pcre2R = captures textPattern regexBaseR = mkRegexBaseR patt pcreLightR = Text.Regex.PCRE.Light.Char8.compile patt [] diff --git a/pcre2.cabal b/pcre2.cabal --- a/pcre2.cabal +++ b/pcre2.cabal @@ -3,11 +3,9 @@ -- This file has been generated from package.yaml by hpack version 0.34.4. -- -- see: https://github.com/sol/hpack --- --- hash: cb8596609200fe862c181845b59caabfdf1c2a19dc0f410828ecba4f7c2c4fe5 name: pcre2 -version: 1.1.5 +version: 2.0.0 synopsis: Regular expressions via the PCRE2 C library (included) description: Please see the README on GitHub at category: Text @@ -15,7 +13,7 @@ bug-reports: https://github.com/sjshuck/hs-pcre2/issues author: Shlomo Shuck and contributors maintainer: stevenjshuck@gmail.com -copyright: 2020 Shlomo Shuck +copyright: 2020-2021 Shlomo Shuck license: Apache-2.0 license-file: LICENSE build-type: Simple @@ -102,6 +100,7 @@ exposed-modules: Text.Regex.Pcre2 Text.Regex.Pcre2.Foreign + Text.Regex.Pcre2.Unsafe other-modules: Text.Regex.Pcre2.Foreign.TH Text.Regex.Pcre2.Internal @@ -109,7 +108,7 @@ Paths_pcre2 hs-source-dirs: src/hs - ghc-options: -optc=-DPCRE2_CODE_UNIT_WIDTH=16 -optc=-DPCRE2_STATIC=1 -optc=-Wno-discarded-qualifiers -optc=-Wno-incompatible-pointer-types + ghc-options: -optc=-DPCRE2_CODE_UNIT_WIDTH=16 -optc=-DPCRE2_STATIC=1 -optc=-DHAVE_INTTYPES_H=1 -optc=-DHAVE_LIMITS_H=1 -optc=-DHAVE_MEMMOVE_H=1 -optc=-DHAVE_STDINT_H=1 -optc=-DHAVE_STDLIB_H=1 -optc=-DHAVE_STRERROR_H=1 -optc=-DHAVE_STRING_H=1 -optc=-DSUPPORT_JIT=1 -optc=-DSUPPORT_PCRE2_16=1 -optc=-DSUPPORT_UNICODE=1 -optc=-Wno-discarded-qualifiers -optc=-Wno-incompatible-pointer-types cc-options: -DHAVE_CONFIG_H -DPCRE2_CODE_UNIT_WIDTH=16 include-dirs: src/c diff --git a/src/c/config.h b/src/c/config.h --- a/src/c/config.h +++ b/src/c/config.h @@ -1,4 +1,5 @@ -// Copied and edited per the PCRE2 source's NON-AUTOTOOLS-BUILD instructions. +/* src/config.h. Generated from config.h.in by configure. */ +/* src/config.h.in. Generated from configure.ac by autoheader. */ /* PCRE2 is written in Standard C, but there are a few non-standard things it can cope with, allowing it to run on SunOS4 and other "close to standard" @@ -73,19 +74,19 @@ /* #undef HAVE_EDIT_READLINE_READLINE_H */ /* Define to 1 if you have the header file. */ -#define HAVE_INTTYPES_H 1 +/* #undef HAVE_INTTYPES_H */ /* Define to 1 if you have the header file. */ -#define HAVE_LIMITS_H 1 +/* #undef HAVE_LIMITS_H */ /* Define to 1 if you have the `memfd_create' function. */ /* #undef HAVE_MEMFD_CREATE */ /* Define to 1 if you have the `memmove' function. */ -#define HAVE_MEMMOVE 1 +/* #undef HAVE_MEMMOVE */ -/* Define to 1 if you have the header file. */ -/* #undef HAVE_MEMORY_H */ +/* Define to 1 if you have the header file. */ +/* #undef HAVE_MINIX_CONFIG_H */ /* Define to 1 if you have the `mkostemp' function. */ /* #undef HAVE_MKOSTEMP */ @@ -106,19 +107,22 @@ /* #undef HAVE_SECURE_GETENV */ /* Define to 1 if you have the header file. */ -#define HAVE_STDINT_H 1 +/* #undef HAVE_STDINT_H */ +/* Define to 1 if you have the header file. */ +/* #undef HAVE_STDIO_H */ + /* Define to 1 if you have the header file. */ -#define HAVE_STDLIB_H 1 +/* #undef HAVE_STDLIB_H */ /* Define to 1 if you have the `strerror' function. */ -#define HAVE_STRERROR 1 +/* #undef HAVE_STRERROR */ /* Define to 1 if you have the header file. */ /* #undef HAVE_STRINGS_H */ /* Define to 1 if you have the header file. */ -#define HAVE_STRING_H 1 +/* #undef HAVE_STRING_H */ /* Define to 1 if you have the header file. */ /* #undef HAVE_SYS_STAT_H */ @@ -135,6 +139,9 @@ /* Define to 1 if the compiler supports simple visibility declarations. */ /* #undef HAVE_VISIBILITY */ +/* Define to 1 if you have the header file. */ +/* #undef HAVE_WCHAR_H */ + /* Define to 1 if you have the header file. */ /* #undef HAVE_WINDOWS_H */ @@ -223,7 +230,7 @@ #define PACKAGE_NAME "PCRE2" /* Define to the full name and version of this package. */ -#define PACKAGE_STRING "PCRE2 10.36" +#define PACKAGE_STRING "PCRE2 10.37" /* Define to the one symbol short name of this package. */ #define PACKAGE_TARNAME "pcre2" @@ -232,7 +239,7 @@ #define PACKAGE_URL "" /* Define to the version of this package. */ -#define PACKAGE_VERSION "10.36" +#define PACKAGE_VERSION "10.37" /* The value of PARENS_NEST_LIMIT specifies the maximum depth of nested parentheses (of any kind) in a pattern. This limits the amount of system @@ -285,11 +292,13 @@ unless SUPPORT_JIT is also defined. */ /* #undef SLJIT_PROT_EXECUTABLE_ALLOCATOR */ -/* Define to 1 if you have the ANSI C header files. */ -#define STDC_HEADERS 1 +/* Define to 1 if all of the C90 standard headers exist (not just the ones + required in a freestanding environment). This macro is provided for + backward compatibility; new code need not use it. */ +/* #undef STDC_HEADERS */ /* Define to any value to enable support for Just-In-Time compiling. */ -#define SUPPORT_JIT 1 +/* #undef SUPPORT_JIT */ /* Define to any value to allow pcre2grep to be linked with libbz2, so that it is able to handle .bz2 files. */ @@ -318,7 +327,7 @@ /* #undef SUPPORT_PCRE2GREP_JIT */ /* Define to any value to enable the 16 bit PCRE2 library. */ -#define SUPPORT_PCRE2_16 1 +/* #undef SUPPORT_PCRE2_16 */ /* Define to any value to enable the 32 bit PCRE2 library. */ /* #undef SUPPORT_PCRE2_32 */ @@ -330,7 +339,7 @@ will work even in an EBCDIC environment, but it is incompatible with the EBCDIC macro. That is, PCRE2 can support *either* EBCDIC code *or* ASCII/Unicode, but not both at once. */ -#define SUPPORT_UNICODE 1 +/* #undef SUPPORT_UNICODE */ /* Define to any value for valgrind support to find invalid memory reads. */ /* #undef SUPPORT_VALGRIND */ @@ -339,35 +348,91 @@ #ifndef _ALL_SOURCE # define _ALL_SOURCE 1 #endif +/* Enable general extensions on macOS. */ +#ifndef _DARWIN_C_SOURCE +# define _DARWIN_C_SOURCE 1 +#endif +/* Enable general extensions on Solaris. */ +#ifndef __EXTENSIONS__ +# define __EXTENSIONS__ 1 +#endif /* Enable GNU extensions on systems that have them. */ #ifndef _GNU_SOURCE # define _GNU_SOURCE 1 #endif -/* Enable threading extensions on Solaris. */ +/* Enable X/Open compliant socket functions that do not require linking + with -lxnet on HP-UX 11.11. */ +#ifndef _HPUX_ALT_XOPEN_SOCKET_API +# define _HPUX_ALT_XOPEN_SOCKET_API 1 +#endif +/* Identify the host operating system as Minix. + This macro does not affect the system headers' behavior. + A future release of Autoconf may stop defining this macro. */ +#ifndef _MINIX +/* # undef _MINIX */ +#endif +/* Enable general extensions on NetBSD. + Enable NetBSD compatibility extensions on Minix. */ +#ifndef _NETBSD_SOURCE +# define _NETBSD_SOURCE 1 +#endif +/* Enable OpenBSD compatibility extensions on NetBSD. + Oddly enough, this does nothing on OpenBSD. */ +#ifndef _OPENBSD_SOURCE +# define _OPENBSD_SOURCE 1 +#endif +/* Define to 1 if needed for POSIX-compatible behavior. */ +#ifndef _POSIX_SOURCE +/* # undef _POSIX_SOURCE */ +#endif +/* Define to 2 if needed for POSIX-compatible behavior. */ +#ifndef _POSIX_1_SOURCE +/* # undef _POSIX_1_SOURCE */ +#endif +/* Enable POSIX-compatible threading on Solaris. */ #ifndef _POSIX_PTHREAD_SEMANTICS # define _POSIX_PTHREAD_SEMANTICS 1 #endif +/* Enable extensions specified by ISO/IEC TS 18661-5:2014. */ +#ifndef __STDC_WANT_IEC_60559_ATTRIBS_EXT__ +# define __STDC_WANT_IEC_60559_ATTRIBS_EXT__ 1 +#endif +/* Enable extensions specified by ISO/IEC TS 18661-1:2014. */ +#ifndef __STDC_WANT_IEC_60559_BFP_EXT__ +# define __STDC_WANT_IEC_60559_BFP_EXT__ 1 +#endif +/* Enable extensions specified by ISO/IEC TS 18661-2:2015. */ +#ifndef __STDC_WANT_IEC_60559_DFP_EXT__ +# define __STDC_WANT_IEC_60559_DFP_EXT__ 1 +#endif +/* Enable extensions specified by ISO/IEC TS 18661-4:2015. */ +#ifndef __STDC_WANT_IEC_60559_FUNCS_EXT__ +# define __STDC_WANT_IEC_60559_FUNCS_EXT__ 1 +#endif +/* Enable extensions specified by ISO/IEC TS 18661-3:2015. */ +#ifndef __STDC_WANT_IEC_60559_TYPES_EXT__ +# define __STDC_WANT_IEC_60559_TYPES_EXT__ 1 +#endif +/* Enable extensions specified by ISO/IEC TR 24731-2:2010. */ +#ifndef __STDC_WANT_LIB_EXT2__ +# define __STDC_WANT_LIB_EXT2__ 1 +#endif +/* Enable extensions specified by ISO/IEC 24747:2009. */ +#ifndef __STDC_WANT_MATH_SPEC_FUNCS__ +# define __STDC_WANT_MATH_SPEC_FUNCS__ 1 +#endif /* Enable extensions on HP NonStop. */ #ifndef _TANDEM_SOURCE # define _TANDEM_SOURCE 1 #endif -/* Enable general extensions on Solaris. */ -#ifndef __EXTENSIONS__ -# define __EXTENSIONS__ 1 +/* Enable X/Open extensions. Define to 500 only if necessary + to make mbstate_t available. */ +#ifndef _XOPEN_SOURCE +/* # undef _XOPEN_SOURCE */ #endif /* Version number of package */ -#define VERSION "10.36" - -/* Define to 1 if on MINIX. */ -/* #undef _MINIX */ - -/* Define to 2 if the system does not provide POSIX.1 features except with - this defined. */ -/* #undef _POSIX_1_SOURCE */ - -/* Define to 1 if you need to in order for `stat' and other things to work. */ -/* #undef _POSIX_SOURCE */ +#define VERSION "10.37" /* Define to empty if `const' does not conform to ANSI C. */ /* #undef const */ diff --git a/src/c/pcre2.h b/src/c/pcre2.h --- a/src/c/pcre2.h +++ b/src/c/pcre2.h @@ -1,5 +1,3 @@ -// Copied per the PCRE2 source's NON-AUTOTOOLS-BUILD instructions. - /************************************************* * Perl-Compatible Regular Expressions * *************************************************/ @@ -44,9 +42,9 @@ /* The current PCRE version information. */ #define PCRE2_MAJOR 10 -#define PCRE2_MINOR 36 +#define PCRE2_MINOR 37 #define PCRE2_PRERELEASE -#define PCRE2_DATE 2020-12-04 +#define PCRE2_DATE 2021-05-26 /* When an application links to a PCRE DLL in Windows, the symbols that are imported have to be identified as such. When building PCRE2, the appropriate diff --git a/src/c/pcre2/src/pcre2_auto_possess.c b/src/c/pcre2/src/pcre2_auto_possess.c --- a/src/c/pcre2/src/pcre2_auto_possess.c +++ b/src/c/pcre2/src/pcre2_auto_possess.c @@ -7,7 +7,7 @@ Written by Philip Hazel Original API code Copyright (c) 1997-2012 University of Cambridge - New API code Copyright (c) 2016-2020 University of Cambridge + New API code Copyright (c) 2016-2021 University of Cambridge ----------------------------------------------------------------------------- Redistribution and use in source and binary forms, with or without @@ -490,6 +490,7 @@ list[2] = (uint32_t)(end - code); return end; } + return NULL; /* Opcode not accepted */ } @@ -1186,12 +1187,16 @@ c = *repeat_opcode; if (c >= OP_CRSTAR && c <= OP_CRMINRANGE) { - /* end must not be NULL. */ - end = get_chr_property_list(code, utf, ucp, cb->fcc, list); + /* The return from get_chr_property_list() will never be NULL when + *code (aka c) is one of the three class opcodes. However, gcc with + -fanalyzer notes that a NULL return is possible, and grumbles. Hence we + put in a check. */ + end = get_chr_property_list(code, utf, ucp, cb->fcc, list); list[1] = (c & 1) == 0; - if (compare_opcodes(end, utf, ucp, cb, list, end, &rec_limit)) + if (end != NULL && + compare_opcodes(end, utf, ucp, cb, list, end, &rec_limit)) { switch (c) { diff --git a/src/c/pcre2/src/pcre2_compile.c b/src/c/pcre2/src/pcre2_compile.c --- a/src/c/pcre2/src/pcre2_compile.c +++ b/src/c/pcre2/src/pcre2_compile.c @@ -1398,32 +1398,47 @@ read_repeat_counts(PCRE2_SPTR *ptrptr, PCRE2_SPTR ptrend, uint32_t *minp, uint32_t *maxp, int *errorcodeptr) { -PCRE2_SPTR p = *ptrptr; +PCRE2_SPTR p; BOOL yield = FALSE; +BOOL had_comma = FALSE; int32_t min = 0; int32_t max = REPEAT_UNLIMITED; /* This value is larger than MAX_REPEAT_COUNT */ -/* NB read_number() initializes the error code to zero. The only error is for a -number that is too big. */ +/* Check the syntax */ +*errorcodeptr = 0; +for (p = *ptrptr;; p++) + { + uint32_t c; + if (p >= ptrend) return FALSE; + c = *p; + if (IS_DIGIT(c)) continue; + if (c == CHAR_RIGHT_CURLY_BRACKET) break; + if (c == CHAR_COMMA) + { + if (had_comma) return FALSE; + had_comma = TRUE; + } + else return FALSE; + } + +/* The only error from read_number() is for a number that is too big. */ + +p = *ptrptr; if (!read_number(&p, ptrend, -1, MAX_REPEAT_COUNT, ERR5, &min, errorcodeptr)) goto EXIT; -if (p >= ptrend) goto EXIT; - if (*p == CHAR_RIGHT_CURLY_BRACKET) { p++; max = min; } - else { - if (*p++ != CHAR_COMMA || p >= ptrend) goto EXIT; - if (*p != CHAR_RIGHT_CURLY_BRACKET) + if (*(++p) != CHAR_RIGHT_CURLY_BRACKET) { if (!read_number(&p, ptrend, -1, MAX_REPEAT_COUNT, ERR5, &max, - errorcodeptr) || p >= ptrend || *p != CHAR_RIGHT_CURLY_BRACKET) + errorcodeptr)) goto EXIT; if (max < min) { @@ -1438,11 +1453,10 @@ if (minp != NULL) *minp = (uint32_t)min; if (maxp != NULL) *maxp = (uint32_t)max; -/* Update the pattern pointer on success, or after an error, but not when -the result is "not a repeat quantifier". */ +/* Update the pattern pointer */ EXIT: -if (yield || *errorcodeptr != 0) *ptrptr = p; +*ptrptr = p; return yield; } @@ -1776,19 +1790,23 @@ { oldptr = ptr; ptr--; /* Back to the digit */ - if (!read_number(&ptr, ptrend, -1, INT_MAX/10 - 1, ERR61, &s, - errorcodeptr)) - break; - /* \1 to \9 are always back references. \8x and \9x are too; \1x to \7x + /* As we know we are at a digit, the only possible error from + read_number() is a number that is too large to be a group number. In this + case we fall through handle this as not a group reference. If we have + read a small enough number, check for a back reference. + + \1 to \9 are always back references. \8x and \9x are too; \1x to \7x are octal escapes if there are not that many previous captures. */ - if (s < 10 || oldptr[-1] >= CHAR_8 || s <= (int)cb->bracount) + if (read_number(&ptr, ptrend, -1, INT_MAX/10 - 1, 0, &s, errorcodeptr) && + (s < 10 || oldptr[-1] >= CHAR_8 || s <= (int)cb->bracount)) { if (s > (int)MAX_GROUP_NUMBER) *errorcodeptr = ERR61; else escape = -s; /* Indicates a back reference */ break; } + ptr = oldptr; /* Put the pointer back and fall through */ } diff --git a/src/c/pcre2/src/pcre2_jit_compile.c b/src/c/pcre2/src/pcre2_jit_compile.c --- a/src/c/pcre2/src/pcre2_jit_compile.c +++ b/src/c/pcre2/src/pcre2_jit_compile.c @@ -1226,7 +1226,7 @@ return TRUE; } -#define EARLY_FAIL_ENHANCE_MAX (1 + 1) +#define EARLY_FAIL_ENHANCE_MAX (1 + 3) /* start: @@ -1238,6 +1238,7 @@ */ static int detect_early_fail(compiler_common *common, PCRE2_SPTR cc, int *private_data_start, sljit_s32 depth, int start) { +PCRE2_SPTR begin = cc; PCRE2_SPTR next_alt; PCRE2_SPTR end; PCRE2_SPTR accelerated_start; @@ -1475,31 +1476,19 @@ case OP_CBRA: end = cc + GET(cc, 1); - if (*end == OP_KET && PRIVATE_DATA(end) == 0) - { - if (*cc == OP_CBRA) - { - if (common->optimized_cbracket[GET2(cc, 1 + LINK_SIZE)] == 0) - break; - cc += IMM2_SIZE; - } - - cc += 1 + LINK_SIZE; - continue; - } - fast_forward_allowed = FALSE; if (depth >= 4) break; end = bracketend(cc) - (1 + LINK_SIZE); - if (*end != OP_KET || PRIVATE_DATA(end) != 0) - break; - - if (*cc == OP_CBRA && common->optimized_cbracket[GET2(cc, 1 + LINK_SIZE)] == 0) + if (*end != OP_KET || (*cc == OP_CBRA && common->optimized_cbracket[GET2(cc, 1 + LINK_SIZE)] == 0)) break; count = detect_early_fail(common, cc, private_data_start, depth + 1, count); + + if (PRIVATE_DATA(cc) != 0) + common->private_data_ptrs[begin - common->start] = 1; + if (count < EARLY_FAIL_ENHANCE_MAX) { cc = end + (1 + LINK_SIZE); @@ -1555,6 +1544,8 @@ return EARLY_FAIL_ENHANCE_MAX; } + /* Cannot be part of a repeat. */ + common->private_data_ptrs[begin - common->start] = 1; count++; if (count < EARLY_FAIL_ENHANCE_MAX) @@ -1620,11 +1611,12 @@ sljit_s32 min, max, i; /* Detect fixed iterations first. */ -if (end[-(1 + LINK_SIZE)] != OP_KET) +if (end[-(1 + LINK_SIZE)] != OP_KET || PRIVATE_DATA(begin) != 0) return FALSE; -/* Already detected repeat. */ -if (common->private_data_ptrs[end - common->start - LINK_SIZE] != 0) +/* /(?:AB){4,6}/ is currently converted to /(?:AB){3}(?AB){1,3}/ + * Skip the check of the second part. */ +if (PRIVATE_DATA(end - LINK_SIZE) == 0) return TRUE; next = end; @@ -1763,6 +1755,7 @@ if (private_data_ptr > SLJIT_MAX_LOCAL_SIZE) break; + /* When the bracket is prefixed by a zero iteration, skip the repeat check (at this point). */ if (repeat_check && (*cc == OP_ONCE || *cc == OP_BRA || *cc == OP_CBRA || *cc == OP_COND)) { if (detect_repeat(common, cc)) @@ -1813,6 +1806,7 @@ case OP_COND: /* Might be a hidden SCOND. */ + common->private_data_ptrs[cc - common->start] = 0; alternative = cc + GET(cc, 1); if (*alternative == OP_KETRMAX || *alternative == OP_KETRMIN) { @@ -13661,9 +13655,11 @@ memset(common->private_data_ptrs, 0, total_length * sizeof(sljit_s32)); private_data_size = common->cbra_ptr + (re->top_bracket + 1) * sizeof(sljit_sw); -set_private_data_ptrs(common, &private_data_size, ccend); + if ((re->overall_options & PCRE2_ANCHORED) == 0 && (re->overall_options & PCRE2_NO_START_OPTIMIZE) == 0 && !common->has_skip_in_assert_back) detect_early_fail(common, common->start, &private_data_size, 0, 0); + +set_private_data_ptrs(common, &private_data_size, ccend); SLJIT_ASSERT(common->early_fail_start_ptr <= common->early_fail_end_ptr); diff --git a/src/c/pcre2/src/pcre2_jit_simd_inc.h b/src/c/pcre2/src/pcre2_jit_simd_inc.h --- a/src/c/pcre2/src/pcre2_jit_simd_inc.h +++ b/src/c/pcre2/src/pcre2_jit_simd_inc.h @@ -39,8 +39,30 @@ ----------------------------------------------------------------------------- */ -#if (defined SLJIT_CONFIG_X86 && SLJIT_CONFIG_X86) && !(defined SUPPORT_VALGRIND) +#if !(defined SUPPORT_VALGRIND) +#if ((defined SLJIT_CONFIG_X86 && SLJIT_CONFIG_X86) \ + || (defined SLJIT_CONFIG_S390X && SLJIT_CONFIG_S390X)) + +typedef enum { + vector_compare_match1, + vector_compare_match1i, + vector_compare_match2, +} vector_compare_type; + +static SLJIT_INLINE sljit_u32 max_fast_forward_char_pair_offset(void) +{ +#if PCRE2_CODE_UNIT_WIDTH == 8 +return 15; +#elif PCRE2_CODE_UNIT_WIDTH == 16 +return 7; +#elif PCRE2_CODE_UNIT_WIDTH == 32 +return 3; +#else +#error "Unsupported unit width" +#endif +} + #if defined SUPPORT_UNICODE && PCRE2_CODE_UNIT_WIDTH != 32 static struct sljit_jump *jump_if_utf_char_start(struct sljit_compiler *compiler, sljit_s32 reg) { @@ -56,6 +78,10 @@ } #endif +#endif /* SLJIT_CONFIG_X86 || SLJIT_CONFIG_S390X */ + +#if (defined SLJIT_CONFIG_X86 && SLJIT_CONFIG_X86) + static sljit_s32 character_to_int32(PCRE2_UCHAR chr) { sljit_u32 value = chr; @@ -97,13 +123,7 @@ sljit_emit_op_custom(compiler, instruction, 5); } -typedef enum { - sse2_compare_match1, - sse2_compare_match1i, - sse2_compare_match2, -} sse2_compare_type; - -static void fast_forward_char_pair_sse2_compare(struct sljit_compiler *compiler, sse2_compare_type compare_type, +static void fast_forward_char_pair_sse2_compare(struct sljit_compiler *compiler, vector_compare_type compare_type, int step, sljit_s32 dst_ind, sljit_s32 cmp1_ind, sljit_s32 cmp2_ind, sljit_s32 tmp_ind) { sljit_u8 instruction[4]; @@ -112,11 +132,11 @@ SLJIT_ASSERT(step >= 0 && step <= 3); -if (compare_type != sse2_compare_match2) +if (compare_type != vector_compare_match2) { if (step == 0) { - if (compare_type == sse2_compare_match1i) + if (compare_type == vector_compare_match1i) { /* POR xmm1, xmm2/m128 */ /* instruction[0] = 0x66; */ @@ -185,14 +205,14 @@ static void fast_forward_char_simd(compiler_common *common, PCRE2_UCHAR char1, PCRE2_UCHAR char2, sljit_s32 offset) { DEFINE_COMPILER; +sljit_u8 instruction[8]; struct sljit_label *start; #if defined SUPPORT_UNICODE && PCRE2_CODE_UNIT_WIDTH != 32 struct sljit_label *restart; #endif struct sljit_jump *quit; struct sljit_jump *partial_quit[2]; -sse2_compare_type compare_type = sse2_compare_match1; -sljit_u8 instruction[8]; +vector_compare_type compare_type = vector_compare_match1; sljit_s32 tmp1_reg_ind = sljit_get_register_index(TMP1); sljit_s32 str_ptr_reg_ind = sljit_get_register_index(STR_PTR); sljit_s32 data_ind = 0; @@ -207,12 +227,12 @@ if (char1 != char2) { bit = char1 ^ char2; - compare_type = sse2_compare_match1i; + compare_type = vector_compare_match1i; if (!is_powerof2(bit)) { bit = 0; - compare_type = sse2_compare_match2; + compare_type = vector_compare_match2; } } @@ -349,11 +369,11 @@ static jump_list *fast_requested_char_simd(compiler_common *common, PCRE2_UCHAR char1, PCRE2_UCHAR char2) { DEFINE_COMPILER; +sljit_u8 instruction[8]; struct sljit_label *start; struct sljit_jump *quit; jump_list *not_found = NULL; -sse2_compare_type compare_type = sse2_compare_match1; -sljit_u8 instruction[8]; +vector_compare_type compare_type = vector_compare_match1; sljit_s32 tmp1_reg_ind = sljit_get_register_index(TMP1); sljit_s32 str_ptr_reg_ind = sljit_get_register_index(STR_PTR); sljit_s32 data_ind = 0; @@ -366,12 +386,12 @@ if (char1 != char2) { bit = char1 ^ char2; - compare_type = sse2_compare_match1i; + compare_type = vector_compare_match1i; if (!is_powerof2(bit)) { bit = 0; - compare_type = sse2_compare_match2; + compare_type = vector_compare_match2; } } @@ -476,27 +496,15 @@ #ifndef _WIN64 -static SLJIT_INLINE sljit_u32 max_fast_forward_char_pair_offset(void) -{ -#if PCRE2_CODE_UNIT_WIDTH == 8 -return 15; -#elif PCRE2_CODE_UNIT_WIDTH == 16 -return 7; -#elif PCRE2_CODE_UNIT_WIDTH == 32 -return 3; -#else -#error "Unsupported unit width" -#endif -} - #define JIT_HAS_FAST_FORWARD_CHAR_PAIR_SIMD (sljit_has_cpu_feature(SLJIT_HAS_SSE2)) static void fast_forward_char_pair_simd(compiler_common *common, sljit_s32 offs1, PCRE2_UCHAR char1a, PCRE2_UCHAR char1b, sljit_s32 offs2, PCRE2_UCHAR char2a, PCRE2_UCHAR char2b) { DEFINE_COMPILER; -sse2_compare_type compare1_type = sse2_compare_match1; -sse2_compare_type compare2_type = sse2_compare_match1; +sljit_u8 instruction[8]; +vector_compare_type compare1_type = vector_compare_match1; +vector_compare_type compare2_type = vector_compare_match1; sljit_u32 bit1 = 0; sljit_u32 bit2 = 0; sljit_u32 diff = IN_UCHARS(offs1 - offs2); @@ -516,7 +524,6 @@ struct sljit_label *restart; #endif struct sljit_jump *jump[2]; -sljit_u8 instruction[8]; int i; SLJIT_ASSERT(common->mode == PCRE2_JIT_COMPLETE && offs1 > offs2); @@ -549,13 +556,13 @@ bit1 = char1a ^ char1b; if (is_powerof2(bit1)) { - compare1_type = sse2_compare_match1i; + compare1_type = vector_compare_match1i; OP1(SLJIT_MOV, TMP1, 0, SLJIT_IMM, character_to_int32(char1a | bit1)); OP1(SLJIT_MOV, TMP2, 0, SLJIT_IMM, character_to_int32(bit1)); } else { - compare1_type = sse2_compare_match2; + compare1_type = vector_compare_match2; bit1 = 0; OP1(SLJIT_MOV, TMP1, 0, SLJIT_IMM, character_to_int32(char1a)); OP1(SLJIT_MOV, TMP2, 0, SLJIT_IMM, character_to_int32(char1b)); @@ -578,13 +585,13 @@ bit2 = char2a ^ char2b; if (is_powerof2(bit2)) { - compare2_type = sse2_compare_match1i; + compare2_type = vector_compare_match1i; OP1(SLJIT_MOV, TMP1, 0, SLJIT_IMM, character_to_int32(char2a | bit2)); OP1(SLJIT_MOV, TMP2, 0, SLJIT_IMM, character_to_int32(bit2)); } else { - compare2_type = sse2_compare_match2; + compare2_type = vector_compare_match2; bit2 = 0; OP1(SLJIT_MOV, TMP1, 0, SLJIT_IMM, character_to_int32(char2a)); OP1(SLJIT_MOV, TMP2, 0, SLJIT_IMM, character_to_int32(char2b)); @@ -731,9 +738,6 @@ add_jump(compiler, &common->failed_match, CMP(SLJIT_GREATER_EQUAL, STR_PTR, 0, STR_END, 0)); -if (common->match_end_ptr != 0) - OP1(SLJIT_MOV, STR_END, 0, SLJIT_MEM1(SLJIT_SP), common->match_end_ptr); - #if defined SUPPORT_UNICODE && PCRE2_CODE_UNIT_WIDTH != 32 if (common->utf) { @@ -760,7 +764,7 @@ #undef SSE2_COMPARE_TYPE_INDEX -#endif /* SLJIT_CONFIG_X86 && !SUPPORT_VALGRIND */ +#endif /* SLJIT_CONFIG_X86 */ #if (defined SLJIT_CONFIG_ARM_64 && SLJIT_CONFIG_ARM_64 && (defined __ARM_NEON || defined __ARM_NEON__)) @@ -1121,3 +1125,743 @@ } #endif /* SLJIT_CONFIG_ARM_64 && SLJIT_CONFIG_ARM_64 */ + +#if (defined SLJIT_CONFIG_S390X && SLJIT_CONFIG_S390X) + +#if PCRE2_CODE_UNIT_WIDTH == 8 +#define VECTOR_ELEMENT_SIZE 0 +#elif PCRE2_CODE_UNIT_WIDTH == 16 +#define VECTOR_ELEMENT_SIZE 1 +#elif PCRE2_CODE_UNIT_WIDTH == 32 +#define VECTOR_ELEMENT_SIZE 2 +#else +#error "Unsupported unit width" +#endif + +static void load_from_mem_vector(struct sljit_compiler *compiler, BOOL vlbb, sljit_s32 dst_vreg, + sljit_s32 base_reg, sljit_s32 index_reg) +{ +sljit_u16 instruction[3]; + +instruction[0] = (sljit_u16)(0xe700 | (dst_vreg << 4) | index_reg); +instruction[1] = (sljit_u16)(base_reg << 12); +instruction[2] = (sljit_u16)((0x8 << 8) | (vlbb ? 0x07 : 0x06)); + +sljit_emit_op_custom(compiler, instruction, 6); +} + +#if PCRE2_CODE_UNIT_WIDTH == 32 + +static void replicate_imm_vector(struct sljit_compiler *compiler, int step, sljit_s32 dst_vreg, + PCRE2_UCHAR chr, sljit_s32 tmp_general_reg) +{ +sljit_u16 instruction[3]; + +SLJIT_ASSERT(step >= 0 && step <= 1); + +if (chr < 0x7fff) + { + if (step == 1) + return; + + /* VREPI */ + instruction[0] = (sljit_u16)(0xe700 | (dst_vreg << 4)); + instruction[1] = (sljit_u16)chr; + instruction[2] = (sljit_u16)((VECTOR_ELEMENT_SIZE << 12) | (0x8 << 8) | 0x45); + sljit_emit_op_custom(compiler, instruction, 6); + return; + } + +if (step == 0) + { + OP1(SLJIT_MOV, tmp_general_reg, 0, SLJIT_IMM, chr); + + /* VLVG */ + instruction[0] = (sljit_u16)(0xe700 | (dst_vreg << 4) | sljit_get_register_index(tmp_general_reg)); + instruction[1] = 0; + instruction[2] = (sljit_u16)((VECTOR_ELEMENT_SIZE << 12) | (0x8 << 8) | 0x22); + sljit_emit_op_custom(compiler, instruction, 6); + return; + } + +/* VREP */ +instruction[0] = (sljit_u16)(0xe700 | (dst_vreg << 4) | dst_vreg); +instruction[1] = 0; +instruction[2] = (sljit_u16)((VECTOR_ELEMENT_SIZE << 12) | (0xc << 8) | 0x4d); +sljit_emit_op_custom(compiler, instruction, 6); +} + +#endif + +static void fast_forward_char_pair_sse2_compare(struct sljit_compiler *compiler, vector_compare_type compare_type, + int step, sljit_s32 dst_ind, sljit_s32 cmp1_ind, sljit_s32 cmp2_ind, sljit_s32 tmp_ind) +{ +sljit_u16 instruction[3]; + +SLJIT_ASSERT(step >= 0 && step <= 2); + +if (step == 1) + { + /* VCEQ */ + instruction[0] = (sljit_u16)(0xe700 | (dst_ind << 4) | dst_ind); + instruction[1] = (sljit_u16)(cmp1_ind << 12); + instruction[2] = (sljit_u16)((VECTOR_ELEMENT_SIZE << 12) | (0xe << 8) | 0xf8); + sljit_emit_op_custom(compiler, instruction, 6); + return; + } + +if (compare_type != vector_compare_match2) + { + if (step == 0 && compare_type == vector_compare_match1i) + { + /* VO */ + instruction[0] = (sljit_u16)(0xe700 | (dst_ind << 4) | dst_ind); + instruction[1] = (sljit_u16)(cmp2_ind << 12); + instruction[2] = (sljit_u16)((0xe << 8) | 0x6a); + sljit_emit_op_custom(compiler, instruction, 6); + } + return; + } + +switch (step) + { + case 0: + /* VCEQ */ + instruction[0] = (sljit_u16)(0xe700 | (tmp_ind << 4) | dst_ind); + instruction[1] = (sljit_u16)(cmp2_ind << 12); + instruction[2] = (sljit_u16)((VECTOR_ELEMENT_SIZE << 12) | (0xe << 8) | 0xf8); + sljit_emit_op_custom(compiler, instruction, 6); + return; + + case 2: + /* VO */ + instruction[0] = (sljit_u16)(0xe700 | (dst_ind << 4) | dst_ind); + instruction[1] = (sljit_u16)(tmp_ind << 12); + instruction[2] = (sljit_u16)((0xe << 8) | 0x6a); + sljit_emit_op_custom(compiler, instruction, 6); + return; + } +} + +#define JIT_HAS_FAST_FORWARD_CHAR_SIMD 1 + +static void fast_forward_char_simd(compiler_common *common, PCRE2_UCHAR char1, PCRE2_UCHAR char2, sljit_s32 offset) +{ +DEFINE_COMPILER; +sljit_u16 instruction[3]; +struct sljit_label *start; +#if defined SUPPORT_UNICODE && PCRE2_CODE_UNIT_WIDTH != 32 +struct sljit_label *restart; +#endif +struct sljit_jump *quit; +struct sljit_jump *partial_quit[2]; +vector_compare_type compare_type = vector_compare_match1; +sljit_s32 tmp1_reg_ind = sljit_get_register_index(TMP1); +sljit_s32 str_ptr_reg_ind = sljit_get_register_index(STR_PTR); +sljit_s32 data_ind = 0; +sljit_s32 tmp_ind = 1; +sljit_s32 cmp1_ind = 2; +sljit_s32 cmp2_ind = 3; +sljit_s32 zero_ind = 4; +sljit_u32 bit = 0; +int i; + +SLJIT_UNUSED_ARG(offset); + +if (char1 != char2) + { + bit = char1 ^ char2; + compare_type = vector_compare_match1i; + + if (!is_powerof2(bit)) + { + bit = 0; + compare_type = vector_compare_match2; + } + } + +partial_quit[0] = CMP(SLJIT_GREATER_EQUAL, STR_PTR, 0, STR_END, 0); +if (common->mode == PCRE2_JIT_COMPLETE) + add_jump(compiler, &common->failed_match, partial_quit[0]); + +/* First part (unaligned start) */ + +OP2(SLJIT_ADD, TMP2, 0, STR_PTR, 0, SLJIT_IMM, 16); + +#if PCRE2_CODE_UNIT_WIDTH != 32 + +/* VREPI */ +instruction[0] = (sljit_u16)(0xe700 | (cmp1_ind << 4)); +instruction[1] = (sljit_u16)(char1 | bit); +instruction[2] = (sljit_u16)((VECTOR_ELEMENT_SIZE << 12) | (0x8 << 8) | 0x45); +sljit_emit_op_custom(compiler, instruction, 6); + +if (char1 != char2) + { + /* VREPI */ + instruction[0] = (sljit_u16)(0xe700 | (cmp2_ind << 4)); + instruction[1] = (sljit_u16)(bit != 0 ? bit : char2); + /* instruction[2] = (sljit_u16)((VECTOR_ELEMENT_SIZE << 12) | (0x8 << 8) | 0x45); */ + sljit_emit_op_custom(compiler, instruction, 6); + } + +#else /* PCRE2_CODE_UNIT_WIDTH == 32 */ + +for (int i = 0; i < 2; i++) + { + replicate_imm_vector(compiler, i, cmp1_ind, char1 | bit, TMP1); + + if (char1 != char2) + replicate_imm_vector(compiler, i, cmp2_ind, bit != 0 ? bit : char2, TMP1); + } + +#endif /* PCRE2_CODE_UNIT_WIDTH != 32 */ + +if (compare_type == vector_compare_match2) + { + /* VREPI */ + instruction[0] = (sljit_u16)(0xe700 | (zero_ind << 4)); + instruction[1] = 0; + instruction[2] = (sljit_u16)((0x8 << 8) | 0x45); + sljit_emit_op_custom(compiler, instruction, 6); + } + +#if defined SUPPORT_UNICODE && PCRE2_CODE_UNIT_WIDTH != 32 +restart = LABEL(); +#endif + +load_from_mem_vector(compiler, TRUE, data_ind, str_ptr_reg_ind, 0); +OP2(SLJIT_AND, TMP2, 0, TMP2, 0, SLJIT_IMM, ~15); + +if (compare_type != vector_compare_match2) + { + if (compare_type == vector_compare_match1i) + fast_forward_char_pair_sse2_compare(compiler, compare_type, 0, data_ind, cmp1_ind, cmp2_ind, tmp_ind); + + /* VFEE */ + instruction[0] = (sljit_u16)(0xe700 | (data_ind << 4) | data_ind); + instruction[1] = (sljit_u16)((cmp1_ind << 12) | (1 << 4)); + instruction[2] = (sljit_u16)((VECTOR_ELEMENT_SIZE << 12) | (0xe << 8) | 0x80); + sljit_emit_op_custom(compiler, instruction, 6); + } +else + { + for (i = 0; i < 3; i++) + fast_forward_char_pair_sse2_compare(compiler, compare_type, i, data_ind, cmp1_ind, cmp2_ind, tmp_ind); + + /* VFENE */ + instruction[0] = (sljit_u16)(0xe700 | (data_ind << 4) | data_ind); + instruction[1] = (sljit_u16)((zero_ind << 12) | (1 << 4)); + instruction[2] = (sljit_u16)((0xe << 8) | 0x81); + sljit_emit_op_custom(compiler, instruction, 6); + } + +/* TODO: use sljit_set_current_flags */ + +/* VLGVB */ +instruction[0] = (sljit_u16)(0xe700 | (tmp1_reg_ind << 4) | data_ind); +instruction[1] = 7; +instruction[2] = (sljit_u16)((0x4 << 8) | 0x21); +sljit_emit_op_custom(compiler, instruction, 6); + +OP2(SLJIT_ADD, STR_PTR, 0, STR_PTR, 0, TMP1, 0); +quit = CMP(SLJIT_LESS, STR_PTR, 0, TMP2, 0); + +OP2(SLJIT_SUB, STR_PTR, 0, TMP2, 0, SLJIT_IMM, 16); + +/* Second part (aligned) */ +start = LABEL(); + +OP2(SLJIT_ADD, STR_PTR, 0, STR_PTR, 0, SLJIT_IMM, 16); + +partial_quit[1] = CMP(SLJIT_GREATER_EQUAL, STR_PTR, 0, STR_END, 0); +if (common->mode == PCRE2_JIT_COMPLETE) + add_jump(compiler, &common->failed_match, partial_quit[1]); + +load_from_mem_vector(compiler, TRUE, data_ind, str_ptr_reg_ind, 0); + +if (compare_type != vector_compare_match2) + { + if (compare_type == vector_compare_match1i) + fast_forward_char_pair_sse2_compare(compiler, compare_type, 0, data_ind, cmp1_ind, cmp2_ind, tmp_ind); + + /* VFEE */ + instruction[0] = (sljit_u16)(0xe700 | (data_ind << 4) | data_ind); + instruction[1] = (sljit_u16)((cmp1_ind << 12) | (1 << 4)); + instruction[2] = (sljit_u16)((VECTOR_ELEMENT_SIZE << 12) | (0xe << 8) | 0x80); + sljit_emit_op_custom(compiler, instruction, 6); + } +else + { + for (i = 0; i < 3; i++) + fast_forward_char_pair_sse2_compare(compiler, compare_type, i, data_ind, cmp1_ind, cmp2_ind, tmp_ind); + + /* VFENE */ + instruction[0] = (sljit_u16)(0xe700 | (data_ind << 4) | data_ind); + instruction[1] = (sljit_u16)((zero_ind << 12) | (1 << 4)); + instruction[2] = (sljit_u16)((0xe << 8) | 0x81); + sljit_emit_op_custom(compiler, instruction, 6); + } + +/* TODO: use sljit_set_current_flags */ + +/* VLGVB */ +instruction[0] = (sljit_u16)(0xe700 | (tmp1_reg_ind << 4) | data_ind); +instruction[1] = 7; +instruction[2] = (sljit_u16)((0x4 << 8) | 0x21); +sljit_emit_op_custom(compiler, instruction, 6); + +CMPTO(SLJIT_GREATER_EQUAL, TMP1, 0, SLJIT_IMM, 16, start); + +OP2(SLJIT_ADD, STR_PTR, 0, STR_PTR, 0, TMP1, 0); + +JUMPHERE(quit); + +if (common->mode != PCRE2_JIT_COMPLETE) + { + JUMPHERE(partial_quit[0]); + JUMPHERE(partial_quit[1]); + OP2(SLJIT_SUB | SLJIT_SET_GREATER, SLJIT_UNUSED, 0, STR_PTR, 0, STR_END, 0); + CMOV(SLJIT_GREATER, STR_PTR, STR_END, 0); + } +else + add_jump(compiler, &common->failed_match, CMP(SLJIT_GREATER_EQUAL, STR_PTR, 0, STR_END, 0)); + +#if defined SUPPORT_UNICODE && PCRE2_CODE_UNIT_WIDTH != 32 +if (common->utf && offset > 0) + { + SLJIT_ASSERT(common->mode == PCRE2_JIT_COMPLETE); + + OP1(MOV_UCHAR, TMP1, 0, SLJIT_MEM1(STR_PTR), IN_UCHARS(-offset)); + + quit = jump_if_utf_char_start(compiler, TMP1); + + OP2(SLJIT_ADD, STR_PTR, 0, STR_PTR, 0, SLJIT_IMM, IN_UCHARS(1)); + add_jump(compiler, &common->failed_match, CMP(SLJIT_GREATER_EQUAL, STR_PTR, 0, STR_END, 0)); + + OP2(SLJIT_ADD, TMP2, 0, STR_PTR, 0, SLJIT_IMM, 16); + JUMPTO(SLJIT_JUMP, restart); + + JUMPHERE(quit); + } +#endif +} + +#define JIT_HAS_FAST_REQUESTED_CHAR_SIMD 1 + +static jump_list *fast_requested_char_simd(compiler_common *common, PCRE2_UCHAR char1, PCRE2_UCHAR char2) +{ +DEFINE_COMPILER; +sljit_u16 instruction[3]; +struct sljit_label *start; +struct sljit_jump *quit; +jump_list *not_found = NULL; +vector_compare_type compare_type = vector_compare_match1; +sljit_s32 tmp1_reg_ind = sljit_get_register_index(TMP1); +sljit_s32 tmp3_reg_ind = sljit_get_register_index(TMP3); +sljit_s32 data_ind = 0; +sljit_s32 tmp_ind = 1; +sljit_s32 cmp1_ind = 2; +sljit_s32 cmp2_ind = 3; +sljit_s32 zero_ind = 4; +sljit_u32 bit = 0; +int i; + +if (char1 != char2) + { + bit = char1 ^ char2; + compare_type = vector_compare_match1i; + + if (!is_powerof2(bit)) + { + bit = 0; + compare_type = vector_compare_match2; + } + } + +add_jump(compiler, ¬_found, CMP(SLJIT_GREATER_EQUAL, TMP1, 0, STR_END, 0)); + +/* First part (unaligned start) */ + +OP2(SLJIT_ADD, TMP2, 0, TMP1, 0, SLJIT_IMM, 16); + +#if PCRE2_CODE_UNIT_WIDTH != 32 + +/* VREPI */ +instruction[0] = (sljit_u16)(0xe700 | (cmp1_ind << 4)); +instruction[1] = (sljit_u16)(char1 | bit); +instruction[2] = (sljit_u16)((VECTOR_ELEMENT_SIZE << 12) | (0x8 << 8) | 0x45); +sljit_emit_op_custom(compiler, instruction, 6); + +if (char1 != char2) + { + /* VREPI */ + instruction[0] = (sljit_u16)(0xe700 | (cmp2_ind << 4)); + instruction[1] = (sljit_u16)(bit != 0 ? bit : char2); + /* instruction[2] = (sljit_u16)((VECTOR_ELEMENT_SIZE << 12) | (0x8 << 8) | 0x45); */ + sljit_emit_op_custom(compiler, instruction, 6); + } + +#else /* PCRE2_CODE_UNIT_WIDTH == 32 */ + +for (int i = 0; i < 2; i++) + { + replicate_imm_vector(compiler, i, cmp1_ind, char1 | bit, TMP3); + + if (char1 != char2) + replicate_imm_vector(compiler, i, cmp2_ind, bit != 0 ? bit : char2, TMP3); + } + +#endif /* PCRE2_CODE_UNIT_WIDTH != 32 */ + +if (compare_type == vector_compare_match2) + { + /* VREPI */ + instruction[0] = (sljit_u16)(0xe700 | (zero_ind << 4)); + instruction[1] = 0; + instruction[2] = (sljit_u16)((0x8 << 8) | 0x45); + sljit_emit_op_custom(compiler, instruction, 6); + } + +load_from_mem_vector(compiler, TRUE, data_ind, tmp1_reg_ind, 0); +OP2(SLJIT_AND, TMP2, 0, TMP2, 0, SLJIT_IMM, ~15); + +if (compare_type != vector_compare_match2) + { + if (compare_type == vector_compare_match1i) + fast_forward_char_pair_sse2_compare(compiler, compare_type, 0, data_ind, cmp1_ind, cmp2_ind, tmp_ind); + + /* VFEE */ + instruction[0] = (sljit_u16)(0xe700 | (data_ind << 4) | data_ind); + instruction[1] = (sljit_u16)((cmp1_ind << 12) | (1 << 4)); + instruction[2] = (sljit_u16)((VECTOR_ELEMENT_SIZE << 12) | (0xe << 8) | 0x80); + sljit_emit_op_custom(compiler, instruction, 6); + } +else + { + for (i = 0; i < 3; i++) + fast_forward_char_pair_sse2_compare(compiler, compare_type, i, data_ind, cmp1_ind, cmp2_ind, tmp_ind); + + /* VFENE */ + instruction[0] = (sljit_u16)(0xe700 | (data_ind << 4) | data_ind); + instruction[1] = (sljit_u16)((zero_ind << 12) | (1 << 4)); + instruction[2] = (sljit_u16)((0xe << 8) | 0x81); + sljit_emit_op_custom(compiler, instruction, 6); + } + +/* TODO: use sljit_set_current_flags */ + +/* VLGVB */ +instruction[0] = (sljit_u16)(0xe700 | (tmp3_reg_ind << 4) | data_ind); +instruction[1] = 7; +instruction[2] = (sljit_u16)((0x4 << 8) | 0x21); +sljit_emit_op_custom(compiler, instruction, 6); + +OP2(SLJIT_ADD, TMP1, 0, TMP1, 0, TMP3, 0); +quit = CMP(SLJIT_LESS, TMP1, 0, TMP2, 0); + +OP2(SLJIT_SUB, TMP1, 0, TMP2, 0, SLJIT_IMM, 16); + +/* Second part (aligned) */ +start = LABEL(); + +OP2(SLJIT_ADD, TMP1, 0, TMP1, 0, SLJIT_IMM, 16); + +add_jump(compiler, ¬_found, CMP(SLJIT_GREATER_EQUAL, TMP1, 0, STR_END, 0)); + +load_from_mem_vector(compiler, TRUE, data_ind, tmp1_reg_ind, 0); + +if (compare_type != vector_compare_match2) + { + if (compare_type == vector_compare_match1i) + fast_forward_char_pair_sse2_compare(compiler, compare_type, 0, data_ind, cmp1_ind, cmp2_ind, tmp_ind); + + /* VFEE */ + instruction[0] = (sljit_u16)(0xe700 | (data_ind << 4) | data_ind); + instruction[1] = (sljit_u16)((cmp1_ind << 12) | (1 << 4)); + instruction[2] = (sljit_u16)((VECTOR_ELEMENT_SIZE << 12) | (0xe << 8) | 0x80); + sljit_emit_op_custom(compiler, instruction, 6); + } +else + { + for (i = 0; i < 3; i++) + fast_forward_char_pair_sse2_compare(compiler, compare_type, i, data_ind, cmp1_ind, cmp2_ind, tmp_ind); + + /* VFENE */ + instruction[0] = (sljit_u16)(0xe700 | (data_ind << 4) | data_ind); + instruction[1] = (sljit_u16)((zero_ind << 12) | (1 << 4)); + instruction[2] = (sljit_u16)((0xe << 8) | 0x81); + sljit_emit_op_custom(compiler, instruction, 6); + } + +/* TODO: use sljit_set_current_flags */ + +/* VLGVB */ +instruction[0] = (sljit_u16)(0xe700 | (tmp3_reg_ind << 4) | data_ind); +instruction[1] = 7; +instruction[2] = (sljit_u16)((0x4 << 8) | 0x21); +sljit_emit_op_custom(compiler, instruction, 6); + +CMPTO(SLJIT_GREATER_EQUAL, TMP3, 0, SLJIT_IMM, 16, start); + +OP2(SLJIT_ADD, TMP1, 0, TMP1, 0, TMP3, 0); + +JUMPHERE(quit); +add_jump(compiler, ¬_found, CMP(SLJIT_GREATER_EQUAL, TMP1, 0, STR_END, 0)); + +return not_found; +} + +#define JIT_HAS_FAST_FORWARD_CHAR_PAIR_SIMD 1 + +static void fast_forward_char_pair_simd(compiler_common *common, sljit_s32 offs1, + PCRE2_UCHAR char1a, PCRE2_UCHAR char1b, sljit_s32 offs2, PCRE2_UCHAR char2a, PCRE2_UCHAR char2b) +{ +DEFINE_COMPILER; +sljit_u16 instruction[3]; +struct sljit_label *start; +#if defined SUPPORT_UNICODE && PCRE2_CODE_UNIT_WIDTH != 32 +struct sljit_label *restart; +#endif +struct sljit_jump *quit; +struct sljit_jump *jump[2]; +vector_compare_type compare1_type = vector_compare_match1; +vector_compare_type compare2_type = vector_compare_match1; +sljit_u32 bit1 = 0; +sljit_u32 bit2 = 0; +sljit_s32 diff = IN_UCHARS(offs2 - offs1); +sljit_s32 tmp1_reg_ind = sljit_get_register_index(TMP1); +sljit_s32 tmp2_reg_ind = sljit_get_register_index(TMP2); +sljit_s32 str_ptr_reg_ind = sljit_get_register_index(STR_PTR); +sljit_s32 data1_ind = 0; +sljit_s32 data2_ind = 1; +sljit_s32 tmp1_ind = 2; +sljit_s32 tmp2_ind = 3; +sljit_s32 cmp1a_ind = 4; +sljit_s32 cmp1b_ind = 5; +sljit_s32 cmp2a_ind = 6; +sljit_s32 cmp2b_ind = 7; +sljit_s32 zero_ind = 8; +int i; + +SLJIT_ASSERT(common->mode == PCRE2_JIT_COMPLETE && offs1 > offs2); +SLJIT_ASSERT(-diff <= (sljit_s32)IN_UCHARS(max_fast_forward_char_pair_offset())); +SLJIT_ASSERT(tmp1_reg_ind != 0 && tmp2_reg_ind != 0); + +if (char1a != char1b) + { + bit1 = char1a ^ char1b; + compare1_type = vector_compare_match1i; + + if (!is_powerof2(bit1)) + { + bit1 = 0; + compare1_type = vector_compare_match2; + } + } + +if (char2a != char2b) + { + bit2 = char2a ^ char2b; + compare2_type = vector_compare_match1i; + + if (!is_powerof2(bit2)) + { + bit2 = 0; + compare2_type = vector_compare_match2; + } + } + +/* Initialize. */ +if (common->match_end_ptr != 0) + { + OP1(SLJIT_MOV, TMP1, 0, SLJIT_MEM1(SLJIT_SP), common->match_end_ptr); + OP1(SLJIT_MOV, TMP3, 0, STR_END, 0); + OP2(SLJIT_ADD, TMP1, 0, TMP1, 0, SLJIT_IMM, IN_UCHARS(offs1 + 1)); + + OP2(SLJIT_SUB | SLJIT_SET_LESS, SLJIT_UNUSED, 0, TMP1, 0, STR_END, 0); + CMOV(SLJIT_LESS, STR_END, TMP1, 0); + } + +OP2(SLJIT_ADD, STR_PTR, 0, STR_PTR, 0, SLJIT_IMM, IN_UCHARS(offs1)); +add_jump(compiler, &common->failed_match, CMP(SLJIT_GREATER_EQUAL, STR_PTR, 0, STR_END, 0)); +OP2(SLJIT_AND, TMP2, 0, STR_PTR, 0, SLJIT_IMM, ~15); + +#if PCRE2_CODE_UNIT_WIDTH != 32 + +OP2(SLJIT_SUB, TMP1, 0, STR_PTR, 0, SLJIT_IMM, -diff); + +/* VREPI */ +instruction[0] = (sljit_u16)(0xe700 | (cmp1a_ind << 4)); +instruction[1] = (sljit_u16)(char1a | bit1); +instruction[2] = (sljit_u16)((VECTOR_ELEMENT_SIZE << 12) | (0x8 << 8) | 0x45); +sljit_emit_op_custom(compiler, instruction, 6); + +if (char1a != char1b) + { + /* VREPI */ + instruction[0] = (sljit_u16)(0xe700 | (cmp1b_ind << 4)); + instruction[1] = (sljit_u16)(bit1 != 0 ? bit1 : char1b); + /* instruction[2] = (sljit_u16)((VECTOR_ELEMENT_SIZE << 12) | (0x8 << 8) | 0x45); */ + sljit_emit_op_custom(compiler, instruction, 6); + } + +/* VREPI */ +instruction[0] = (sljit_u16)(0xe700 | (cmp2a_ind << 4)); +instruction[1] = (sljit_u16)(char2a | bit2); +/* instruction[2] = (sljit_u16)((VECTOR_ELEMENT_SIZE << 12) | (0x8 << 8) | 0x45); */ +sljit_emit_op_custom(compiler, instruction, 6); + +if (char2a != char2b) + { + /* VREPI */ + instruction[0] = (sljit_u16)(0xe700 | (cmp2b_ind << 4)); + instruction[1] = (sljit_u16)(bit2 != 0 ? bit2 : char2b); + /* instruction[2] = (sljit_u16)((VECTOR_ELEMENT_SIZE << 12) | (0x8 << 8) | 0x45); */ + sljit_emit_op_custom(compiler, instruction, 6); + } + +#else /* PCRE2_CODE_UNIT_WIDTH == 32 */ + +for (int i = 0; i < 2; i++) + { + replicate_imm_vector(compiler, i, cmp1a_ind, char1a | bit1, TMP1); + + if (char1a != char1b) + replicate_imm_vector(compiler, i, cmp1b_ind, bit1 != 0 ? bit1 : char1b, TMP1); + + replicate_imm_vector(compiler, i, cmp2a_ind, char2a | bit2, TMP1); + + if (char2a != char2b) + replicate_imm_vector(compiler, i, cmp2b_ind, bit2 != 0 ? bit2 : char2b, TMP1); + } + +OP2(SLJIT_SUB, TMP1, 0, STR_PTR, 0, SLJIT_IMM, -diff); + +#endif /* PCRE2_CODE_UNIT_WIDTH != 32 */ + +/* VREPI */ +instruction[0] = (sljit_u16)(0xe700 | (zero_ind << 4)); +instruction[1] = 0; +instruction[2] = (sljit_u16)((0x8 << 8) | 0x45); +sljit_emit_op_custom(compiler, instruction, 6); + +#if defined SUPPORT_UNICODE && PCRE2_CODE_UNIT_WIDTH != 32 +restart = LABEL(); +#endif + +jump[0] = CMP(SLJIT_LESS, TMP1, 0, TMP2, 0); +load_from_mem_vector(compiler, TRUE, data2_ind, tmp1_reg_ind, 0); +jump[1] = JUMP(SLJIT_JUMP); +JUMPHERE(jump[0]); +load_from_mem_vector(compiler, FALSE, data2_ind, tmp1_reg_ind, 0); +JUMPHERE(jump[1]); + +load_from_mem_vector(compiler, TRUE, data1_ind, str_ptr_reg_ind, 0); +OP2(SLJIT_ADD, TMP2, 0, TMP2, 0, SLJIT_IMM, 16); + +for (i = 0; i < 3; i++) + { + fast_forward_char_pair_sse2_compare(compiler, compare1_type, i, data1_ind, cmp1a_ind, cmp1b_ind, tmp1_ind); + fast_forward_char_pair_sse2_compare(compiler, compare2_type, i, data2_ind, cmp2a_ind, cmp2b_ind, tmp2_ind); + } + +/* VN */ +instruction[0] = (sljit_u16)(0xe700 | (data1_ind << 4) | data1_ind); +instruction[1] = (sljit_u16)(data2_ind << 12); +instruction[2] = (sljit_u16)((0xe << 8) | 0x68); +sljit_emit_op_custom(compiler, instruction, 6); + +/* VFENE */ +instruction[0] = (sljit_u16)(0xe700 | (data1_ind << 4) | data1_ind); +instruction[1] = (sljit_u16)((zero_ind << 12) | (1 << 4)); +instruction[2] = (sljit_u16)((0xe << 8) | 0x81); +sljit_emit_op_custom(compiler, instruction, 6); + +/* TODO: use sljit_set_current_flags */ + +/* VLGVB */ +instruction[0] = (sljit_u16)(0xe700 | (tmp1_reg_ind << 4) | data1_ind); +instruction[1] = 7; +instruction[2] = (sljit_u16)((0x4 << 8) | 0x21); +sljit_emit_op_custom(compiler, instruction, 6); + +OP2(SLJIT_ADD, STR_PTR, 0, STR_PTR, 0, TMP1, 0); +quit = CMP(SLJIT_LESS, STR_PTR, 0, TMP2, 0); + +OP2(SLJIT_SUB, STR_PTR, 0, TMP2, 0, SLJIT_IMM, 16); +OP1(SLJIT_MOV, TMP1, 0, SLJIT_IMM, diff); + +/* Main loop. */ +start = LABEL(); + +OP2(SLJIT_ADD, STR_PTR, 0, STR_PTR, 0, SLJIT_IMM, 16); +add_jump(compiler, &common->failed_match, CMP(SLJIT_GREATER_EQUAL, STR_PTR, 0, STR_END, 0)); + +load_from_mem_vector(compiler, FALSE, data1_ind, str_ptr_reg_ind, 0); +load_from_mem_vector(compiler, FALSE, data2_ind, str_ptr_reg_ind, tmp1_reg_ind); + +for (i = 0; i < 3; i++) + { + fast_forward_char_pair_sse2_compare(compiler, compare1_type, i, data1_ind, cmp1a_ind, cmp1b_ind, tmp1_ind); + fast_forward_char_pair_sse2_compare(compiler, compare2_type, i, data2_ind, cmp2a_ind, cmp2b_ind, tmp2_ind); + } + +/* VN */ +instruction[0] = (sljit_u16)(0xe700 | (data1_ind << 4) | data1_ind); +instruction[1] = (sljit_u16)(data2_ind << 12); +instruction[2] = (sljit_u16)((0xe << 8) | 0x68); +sljit_emit_op_custom(compiler, instruction, 6); + +/* VFENE */ +instruction[0] = (sljit_u16)(0xe700 | (data1_ind << 4) | data1_ind); +instruction[1] = (sljit_u16)((zero_ind << 12) | (1 << 4)); +instruction[2] = (sljit_u16)((0xe << 8) | 0x81); +sljit_emit_op_custom(compiler, instruction, 6); + +/* TODO: use sljit_set_current_flags */ + +/* VLGVB */ +instruction[0] = (sljit_u16)(0xe700 | (tmp2_reg_ind << 4) | data1_ind); +instruction[1] = 7; +instruction[2] = (sljit_u16)((0x4 << 8) | 0x21); +sljit_emit_op_custom(compiler, instruction, 6); + +CMPTO(SLJIT_GREATER_EQUAL, TMP2, 0, SLJIT_IMM, 16, start); + +OP2(SLJIT_ADD, STR_PTR, 0, STR_PTR, 0, TMP2, 0); + +JUMPHERE(quit); + +add_jump(compiler, &common->failed_match, CMP(SLJIT_GREATER_EQUAL, STR_PTR, 0, STR_END, 0)); + +#if defined SUPPORT_UNICODE && PCRE2_CODE_UNIT_WIDTH != 32 +if (common->utf) + { + SLJIT_ASSERT(common->mode == PCRE2_JIT_COMPLETE); + + OP1(MOV_UCHAR, TMP1, 0, SLJIT_MEM1(STR_PTR), IN_UCHARS(-offs1)); + + quit = jump_if_utf_char_start(compiler, TMP1); + + OP2(SLJIT_ADD, STR_PTR, 0, STR_PTR, 0, SLJIT_IMM, IN_UCHARS(1)); + add_jump(compiler, &common->failed_match, CMP(SLJIT_GREATER_EQUAL, STR_PTR, 0, STR_END, 0)); + + /* TMP1 contains diff. */ + OP2(SLJIT_AND, TMP2, 0, STR_PTR, 0, SLJIT_IMM, ~15); + OP2(SLJIT_SUB, TMP1, 0, STR_PTR, 0, SLJIT_IMM, -diff); + JUMPTO(SLJIT_JUMP, restart); + + JUMPHERE(quit); + } +#endif + +OP2(SLJIT_SUB, STR_PTR, 0, STR_PTR, 0, SLJIT_IMM, IN_UCHARS(offs1)); + +if (common->match_end_ptr != 0) + OP1(SLJIT_MOV, STR_END, 0, TMP3, 0); +} + +#endif /* SLJIT_CONFIG_S390X */ + +#endif /* !SUPPORT_VALGRIND */ diff --git a/src/c/pcre2/src/pcre2_match.c b/src/c/pcre2/src/pcre2_match.c --- a/src/c/pcre2/src/pcre2_match.c +++ b/src/c/pcre2/src/pcre2_match.c @@ -818,10 +818,12 @@ /* N is now the frame of the recursion; the previous frame is at the OP_RECURSE position. Go back there, copying the current subject position - and mark, and move on past the OP_RECURSE. */ + and mark, and the start_match position (\K might have changed it), and + then move on past the OP_RECURSE. */ P->eptr = Feptr; P->mark = Fmark; + P->start_match = Fstart_match; F = P; Fecode += 1 + LINK_SIZE; continue; diff --git a/src/c/pcre2/src/pcre2posix.c b/src/c/pcre2/src/pcre2posix.c --- a/src/c/pcre2/src/pcre2posix.c +++ b/src/c/pcre2/src/pcre2posix.c @@ -7,7 +7,7 @@ Written by Philip Hazel Original API code Copyright (c) 1997-2012 University of Cambridge - New API code Copyright (c) 2016-2019 University of Cambridge + New API code Copyright (c) 2016-2021 University of Cambridge ----------------------------------------------------------------------------- Redistribution and use in source and binary forms, with or without @@ -175,6 +175,20 @@ +#if 0 /* REMOVE THIS CODE */ + +The code below was created for 10.33 (see ChangeLog 10.33 #4) when the +POSIX functions were given pcre2_... names instead of the traditional POSIX +names. However, it has proved to be more troublesome than useful. There have +been at least two cases where a program links with two others, one of which +uses the POSIX library and the other uses the PCRE2 POSIX functions, thus +causing two instances of the POSIX runctions to exist, leading to trouble. For +10.37 this code is commented out. In due course it can be removed if there are +no issues. The only small worry is the comment below about languages that do +not include pcre2posix.h. If there are any such cases, they will have to use +the PCRE2 names. + + /************************************************* * Wrappers with traditional POSIX names * *************************************************/ @@ -218,7 +232,7 @@ { return pcre2_regexec(preg, string, nmatch, pmatch, eflags); } - +#endif /************************************************* diff --git a/src/c/pcre2/src/sljit/sljitConfigInternal.h b/src/c/pcre2/src/sljit/sljitConfigInternal.h --- a/src/c/pcre2/src/sljit/sljitConfigInternal.h +++ b/src/c/pcre2/src/sljit/sljitConfigInternal.h @@ -158,6 +158,8 @@ #define SLJIT_CONFIG_MIPS_64 1 #elif defined(__sparc__) || defined(__sparc) #define SLJIT_CONFIG_SPARC_32 1 +#elif defined(__s390x__) +#define SLJIT_CONFIG_S390X 1 #else /* Unsupported architecture */ #define SLJIT_CONFIG_UNSUPPORTED 1 diff --git a/src/c/pcre2/src/sljit/sljitExecAllocator.c b/src/c/pcre2/src/sljit/sljitExecAllocator.c --- a/src/c/pcre2/src/sljit/sljitExecAllocator.c +++ b/src/c/pcre2/src/sljit/sljitExecAllocator.c @@ -79,6 +79,7 @@ */ #ifdef _WIN32 +#define SLJIT_UPDATE_WX_FLAGS(from, to, enable_exec) static SLJIT_INLINE void* alloc_chunk(sljit_uw size) { @@ -91,96 +92,108 @@ VirtualFree(chunk, 0, MEM_RELEASE); } -#else - -#ifdef __APPLE__ -#ifdef MAP_ANON -/* Configures TARGET_OS_OSX when appropriate */ -#include - -#if TARGET_OS_OSX && defined(MAP_JIT) -#include -#endif /* TARGET_OS_OSX && MAP_JIT */ - -#ifdef MAP_JIT +#else /* POSIX */ +#if defined(__APPLE__) && defined(MAP_JIT) /* On macOS systems, returns MAP_JIT if it is defined _and_ we're running on a - version where it's OK to have more than one JIT block. + version where it's OK to have more than one JIT block or where MAP_JIT is + required. On non-macOS systems, returns MAP_JIT if it is defined. */ +#include +#if TARGET_OS_OSX +#if defined SLJIT_CONFIG_X86 && SLJIT_CONFIG_X86 +#ifdef MAP_ANON +#include +#include + +#define SLJIT_MAP_JIT (get_map_jit_flag()) + static SLJIT_INLINE int get_map_jit_flag() { -#if TARGET_OS_OSX - sljit_sw page_size = get_page_alignment() + 1; + sljit_sw page_size; void *ptr; + struct utsname name; static int map_jit_flag = -1; - /* - The following code is thread safe because multiple initialization - sets map_jit_flag to the same value and the code has no side-effects. - Changing the kernel version witout system restart is (very) unlikely. - */ - if (map_jit_flag == -1) { - struct utsname name; - + if (map_jit_flag < 0) { map_jit_flag = 0; uname(&name); - /* Kernel version for 10.14.0 (Mojave) */ + /* Kernel version for 10.14.0 (Mojave) or later */ if (atoi(name.release) >= 18) { + page_size = get_page_alignment() + 1; /* Only use MAP_JIT if a hardened runtime is used */ - - ptr = mmap(NULL, page_size, PROT_WRITE | PROT_EXEC, MAP_PRIVATE | MAP_ANON, -1, 0); + ptr = mmap(NULL, page_size, PROT_WRITE | PROT_EXEC, + MAP_PRIVATE | MAP_ANON, -1, 0); - if (ptr == MAP_FAILED) { - map_jit_flag = MAP_JIT; - } else { + if (ptr != MAP_FAILED) munmap(ptr, page_size); - } + else + map_jit_flag = MAP_JIT; } } - return map_jit_flag; -#else /* !TARGET_OS_OSX */ - return MAP_JIT; -#endif /* TARGET_OS_OSX */ } - -#endif /* MAP_JIT */ #endif /* MAP_ANON */ -#endif /* __APPLE__ */ +#else /* !SLJIT_CONFIG_X86 */ +#if !(defined SLJIT_CONFIG_ARM && SLJIT_CONFIG_ARM) +#error Unsupported architecture +#endif /* SLJIT_CONFIG_ARM */ +#include +#define SLJIT_MAP_JIT (MAP_JIT) +#define SLJIT_UPDATE_WX_FLAGS(from, to, enable_exec) \ + apple_update_wx_flags(enable_exec) + +static SLJIT_INLINE void apple_update_wx_flags(sljit_s32 enable_exec) +{ + pthread_jit_write_protect_np(enable_exec); +} +#endif /* SLJIT_CONFIG_X86 */ +#else /* !TARGET_OS_OSX */ +#define SLJIT_MAP_JIT (MAP_JIT) +#endif /* TARGET_OS_OSX */ +#endif /* __APPLE__ && MAP_JIT */ +#ifndef SLJIT_UPDATE_WX_FLAGS +#define SLJIT_UPDATE_WX_FLAGS(from, to, enable_exec) +#endif /* !SLJIT_UPDATE_WX_FLAGS */ +#ifndef SLJIT_MAP_JIT +#define SLJIT_MAP_JIT (0) +#endif /* !SLJIT_MAP_JIT */ + static SLJIT_INLINE void* alloc_chunk(sljit_uw size) { void *retval; - const int prot = PROT_READ | PROT_WRITE | PROT_EXEC; - -#ifdef MAP_ANON - - int flags = MAP_PRIVATE | MAP_ANON; + int prot = PROT_READ | PROT_WRITE | PROT_EXEC; + int flags = MAP_PRIVATE; + int fd = -1; -#ifdef MAP_JIT - flags |= get_map_jit_flag(); +#ifdef PROT_MAX + prot |= PROT_MAX(prot); #endif - retval = mmap(NULL, size, prot, flags, -1, 0); +#ifdef MAP_ANON + flags |= MAP_ANON | SLJIT_MAP_JIT; #else /* !MAP_ANON */ if (SLJIT_UNLIKELY((dev_zero < 0) && open_dev_zero())) return NULL; - retval = mmap(NULL, size, prot, MAP_PRIVATE, dev_zero, 0); + fd = dev_zero; #endif /* MAP_ANON */ + retval = mmap(NULL, size, prot, flags, fd, 0); if (retval == MAP_FAILED) - retval = NULL; - else { - if (mprotect(retval, size, prot) < 0) { - munmap(retval, size); - retval = NULL; - } + return NULL; + + if (mprotect(retval, size, PROT_READ | PROT_WRITE | PROT_EXEC) < 0) { + munmap(retval, size); + return NULL; } + SLJIT_UPDATE_WX_FLAGS(retval, (uint8_t *)retval + size, 0); + return retval; } @@ -189,7 +202,7 @@ munmap(chunk, size); } -#endif +#endif /* windows */ /* --------------------------------------------------------------------- */ /* Common functions */ @@ -261,6 +274,7 @@ while (free_block) { if (free_block->size >= size) { chunk_size = free_block->size; + SLJIT_UPDATE_WX_FLAGS(NULL, NULL, 0); if (chunk_size > size + 64) { /* We just cut a block from the end of the free block. */ chunk_size -= size; @@ -326,6 +340,7 @@ allocated_size -= header->size; /* Connecting free blocks together if possible. */ + SLJIT_UPDATE_WX_FLAGS(NULL, NULL, 0); /* If header->prev_size == 0, free_block will equal to header. In this case, free_block->header.size will be > 0. */ @@ -358,6 +373,7 @@ } } + SLJIT_UPDATE_WX_FLAGS(NULL, NULL, 1); SLJIT_ALLOCATOR_UNLOCK(); } @@ -367,6 +383,7 @@ struct free_block* next_free_block; SLJIT_ALLOCATOR_LOCK(); + SLJIT_UPDATE_WX_FLAGS(NULL, NULL, 0); free_block = free_blocks; while (free_block) { @@ -381,5 +398,6 @@ } SLJIT_ASSERT((total_size && free_blocks) || (!total_size && !free_blocks)); + SLJIT_UPDATE_WX_FLAGS(NULL, NULL, 1); SLJIT_ALLOCATOR_UNLOCK(); } diff --git a/src/c/pcre2/src/sljit/sljitNativeS390X.c b/src/c/pcre2/src/sljit/sljitNativeS390X.c --- a/src/c/pcre2/src/sljit/sljitNativeS390X.c +++ b/src/c/pcre2/src/sljit/sljitNativeS390X.c @@ -42,7 +42,7 @@ typedef sljit_uw sljit_ins; /* Instruction tags (most significant halfword). */ -const sljit_ins sljit_ins_const = (sljit_ins)1 << 48; +static const sljit_ins sljit_ins_const = (sljit_ins)1 << 48; static const sljit_u8 reg_map[SLJIT_NUMBER_OF_REGISTERS + 4] = { 14, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 0, 1 @@ -66,22 +66,22 @@ * will be retired ASAP (TODO: carenas) */ -const sljit_gpr r0 = 0; /* reg_map[SLJIT_NUMBER_OF_REGISTERS + 2]: 0 in address calculations; reserved */ -const sljit_gpr r1 = 1; /* reg_map[SLJIT_NUMBER_OF_REGISTERS + 3]: reserved */ -const sljit_gpr r2 = 2; /* reg_map[1]: 1st argument */ -const sljit_gpr r3 = 3; /* reg_map[2]: 2nd argument */ -const sljit_gpr r4 = 4; /* reg_map[3]: 3rd argument */ -const sljit_gpr r5 = 5; /* reg_map[4]: 4th argument */ -const sljit_gpr r6 = 6; /* reg_map[5]: 5th argument; 1st saved register */ -const sljit_gpr r7 = 7; /* reg_map[6] */ -const sljit_gpr r8 = 8; /* reg_map[7] */ -const sljit_gpr r9 = 9; /* reg_map[8] */ -const sljit_gpr r10 = 10; /* reg_map[9] */ -const sljit_gpr r11 = 11; /* reg_map[10] */ -const sljit_gpr r12 = 12; /* reg_map[11]: GOT */ -const sljit_gpr r13 = 13; /* reg_map[12]: Literal Pool pointer */ -const sljit_gpr r14 = 14; /* reg_map[0]: return address and flag register */ -const sljit_gpr r15 = 15; /* reg_map[SLJIT_NUMBER_OF_REGISTERS + 1]: stack pointer */ +static const sljit_gpr r0 = 0; /* reg_map[SLJIT_NUMBER_OF_REGISTERS + 2]: 0 in address calculations; reserved */ +static const sljit_gpr r1 = 1; /* reg_map[SLJIT_NUMBER_OF_REGISTERS + 3]: reserved */ +static const sljit_gpr r2 = 2; /* reg_map[1]: 1st argument */ +static const sljit_gpr r3 = 3; /* reg_map[2]: 2nd argument */ +static const sljit_gpr r4 = 4; /* reg_map[3]: 3rd argument */ +static const sljit_gpr r5 = 5; /* reg_map[4]: 4th argument */ +static const sljit_gpr r6 = 6; /* reg_map[5]: 5th argument; 1st saved register */ +static const sljit_gpr r7 = 7; /* reg_map[6] */ +static const sljit_gpr r8 = 8; /* reg_map[7] */ +static const sljit_gpr r9 = 9; /* reg_map[8] */ +static const sljit_gpr r10 = 10; /* reg_map[9] */ +static const sljit_gpr r11 = 11; /* reg_map[10] */ +static const sljit_gpr r12 = 12; /* reg_map[11]: GOT */ +static const sljit_gpr r13 = 13; /* reg_map[12]: Literal Pool pointer */ +static const sljit_gpr r14 = 14; /* reg_map[0]: return address and flag register */ +static const sljit_gpr r15 = 15; /* reg_map[SLJIT_NUMBER_OF_REGISTERS + 1]: stack pointer */ /* WARNING: r12 and r13 shouldn't be used as per ABI recommendation */ /* TODO(carenas): r12 might conflict in PIC code, reserve? */ @@ -100,8 +100,8 @@ /* Link registers. The normal link register is r14, but since we use that for flags we need to use r0 instead to do fast calls so that flags are preserved. */ -const sljit_gpr link_r = 14; /* r14 */ -const sljit_gpr fast_link_r = 0; /* r0 */ +static const sljit_gpr link_r = 14; /* r14 */ +static const sljit_gpr fast_link_r = 0; /* r0 */ /* Flag register layout: @@ -110,7 +110,7 @@ | ZERO | 0 | 0 | C C |///////| +---------------+---+---+-------+-------+ */ -const sljit_gpr flag_r = 14; /* r14 */ +static const sljit_gpr flag_r = 14; /* r14 */ struct sljit_s390x_const { struct sljit_const const_; /* must be first */ @@ -1465,7 +1465,8 @@ op = GET_OPCODE(op) | (op & SLJIT_I32_OP); switch (op) { case SLJIT_BREAKPOINT: - /* TODO(mundaym): insert real breakpoint? */ + /* The following invalid instruction is emitted by gdb. */ + return push_inst(compiler, 0x0001 /* 2-byte trap */); case SLJIT_NOP: return push_inst(compiler, 0x0700 /* 2-byte nop */); case SLJIT_LMUL_UW: diff --git a/src/c/pcre2/src/sljit/sljitUtils.c b/src/c/pcre2/src/sljit/sljitUtils.c --- a/src/c/pcre2/src/sljit/sljitUtils.c +++ b/src/c/pcre2/src/sljit/sljitUtils.c @@ -48,7 +48,7 @@ static SLJIT_INLINE void allocator_grab_lock(void) { HANDLE lock; - if (SLJIT_UNLIKELY(!allocator_lock)) { + if (SLJIT_UNLIKELY(!InterlockedCompareExchangePointer(&allocator_lock, NULL, NULL))) { lock = CreateMutex(NULL, FALSE, NULL); if (InterlockedCompareExchangePointer(&allocator_lock, lock, NULL)) CloseHandle(lock); @@ -146,9 +146,13 @@ #include static SLJIT_INLINE sljit_sw get_page_alignment(void) { - static sljit_sw sljit_page_align; - if (!sljit_page_align) { + static sljit_sw sljit_page_align = -1; + if (sljit_page_align < 0) { +#ifdef _SC_PAGESIZE sljit_page_align = sysconf(_SC_PAGESIZE); +#else + sljit_page_align = getpagesize(); +#endif /* Should never happen. */ if (sljit_page_align < 0) sljit_page_align = 4096; diff --git a/src/c/pcre2/src/sljit/sljitWXExecAllocator.c b/src/c/pcre2/src/sljit/sljitWXExecAllocator.c --- a/src/c/pcre2/src/sljit/sljitWXExecAllocator.c +++ b/src/c/pcre2/src/sljit/sljitWXExecAllocator.c @@ -121,14 +121,18 @@ static pthread_mutex_t se_lock = PTHREAD_MUTEX_INITIALIZER; #endif static int se_protected = !SLJIT_PROT_WX; + int prot = PROT_READ | PROT_WRITE | SLJIT_PROT_WX; sljit_uw* ptr; if (SLJIT_UNLIKELY(se_protected < 0)) return NULL; +#ifdef PROT_MAX + prot |= PROT_MAX(PROT_READ | PROT_WRITE | PROT_EXEC); +#endif + size += sizeof(sljit_uw); - ptr = (sljit_uw*)mmap(NULL, size, PROT_READ | PROT_WRITE | SLJIT_PROT_WX, - MAP_PRIVATE | MAP_ANON, -1, 0); + ptr = (sljit_uw*)mmap(NULL, size, prot, MAP_PRIVATE | MAP_ANON, -1, 0); if (ptr == MAP_FAILED) return NULL; diff --git a/src/c/pcre2_chartables.c b/src/c/pcre2_chartables.c --- a/src/c/pcre2_chartables.c +++ b/src/c/pcre2_chartables.c @@ -1,5 +1,3 @@ -// Copied per the PCRE2 source's NON-AUTOTOOLS-BUILD instructions. - /************************************************* * Perl-Compatible Regular Expressions * *************************************************/ diff --git a/src/hs/Text/Regex/Pcre2.hs b/src/hs/Text/Regex/Pcre2.hs --- a/src/hs/Text/Regex/Pcre2.hs +++ b/src/hs/Text/Regex/Pcre2.hs @@ -1,7 +1,7 @@ module Text.Regex.Pcre2 ( -- * Matching and substitution {-| - /__Introduction__/ + === __Introduction__ Atop the low-level binding to the C API, we present a high-level interface to add regular expressions to Haskell programs. @@ -45,7 +45,7 @@ cognitive overhead, and (for lenses) library dependencies, so it\'s really a matter of finding the best trade-offs for her case. - /__Definitions__/ + === __Definitions__ [Pattern]: The string defining a regular expression. Refer to syntax [here](https://pcre.org/current/doc/html/pcre2pattern.html). @@ -75,7 +75,7 @@ @(?\...)@. Whether they have names or not, captures are always numbered as described above. - /__Performance__/ + === __Performance__ Each API function is designed such that, when a regex is obtained, the underlying C data generated from the pattern and any options is reused for @@ -99,23 +99,50 @@ Note: Template Haskell regexes are immune from this problem and may be freely inlined; see below. - /__Handling errors__/ + Also of note is the optimization that, for each capture that\'s more than + half the length of the subject, a zero-copy `Text` is produced in constant + time and space. This can yield a large performance boost in many cases, + for example when splitting lines into key-value pairs as in + the [teaser](https://github.com/sjshuck/hs-pcre2#teasers). A downside, + however, is that retaining these slices in memory will carry the overhead + of the dead portions of the subject (still guaranteed to be less than the + slices in total size). - In a few places we use the `Alternative` typeclass to optionally return - match results, expressing success via `pure` and failure via `empty`. - Typically the user will choose the instance `Maybe`, but other useful ones - exist, notably @[STM](https://hackage.haskell.org/package/stm/docs/Control-Monad-STM.html#t:STM)@, + === __Handling results and errors__ + + In contrast to [other](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/match) + [APIs](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/matchAll)—including + previous versions of [this library](https://github.com/sjshuck/hs-pcre2/issues/17)—where + there are separate functions to request single versus global matching, we + accomplish this /(since 2.0.0)/ in a unified fashion using the `Alternative` + typeclass. Typically the user will choose from two instances, `Maybe` and + `[]`: + + > b2 :: (Alternative f) => Text -> f Text + > b2 = match "b.." + > + > -- Zero or one match + > findB2 :: Text -> Maybe Text + > findB2 = b2 + > + > -- Zero or more matches + > findAllB2s :: Text -> [Text] + > findAllB2s = b2 + + Other instances exist for niche uses, + notably @[STM](https://hackage.haskell.org/package/stm/docs/Control-Monad-STM.html#t:STM)@, that of [optparse-applicative](https://hackage.haskell.org/package/optparse-applicative/docs/Options-Applicative.html#t:Parser), and those of parser combinator libraries such as [megaparsec](https://hackage.haskell.org/package/megaparsec/docs/Text-Megaparsec.html#t:ParsecT). By contrast, user errors are thrown purely. If a user error is to be caught, it must be at the site where the match or substitution results are - evaluated—in other words, wherever the regex is applied to a subject. - Even pattern compile errors are deferred to match sites, due to the way this - library employs `System.IO.Unsafe.unsafePerformIO` to implement laziness. + evaluated. As a particular consequence, pattern compile errors are deferred + to match sites. >>> broken = match "*" + >>> :t broken + broken :: Alternative f => Text -> f Text >>> broken "foo" *** Exception: pcre2_compile: quantifier does not follow a repeatable item * @@ -124,30 +151,23 @@ `Control.Exception.evaluate` comes in handy to force results into the `IO` monad in order to catch errors reliably: - >>> evaluate (broken "foo") `catch` \(_ :: SomePcre2Exception) -> return Nothing + >>> :set -XTypeApplications + >>> handle @SomePcre2Exception (\_ -> return Nothing) $ evaluate $ broken "foo" Nothing Or simply select `IO` as the `Alternative` instance: - >>> :t broken - broken :: Alternative f => Text -> f Text - >>> broken "foo" `catch` \(_ :: SomePcre2Exception) -> return "broken" + >>> handle @SomePcre2Exception (\_ -> return "broken") $ broken "foo" "broken" -} -- ** Basic matching functions match, matchOpt, - matchAll, - matchAllOpt, matches, matchesOpt, captures, capturesOpt, - capturesA, - capturesAOpt, - capturesAll, - capturesAllOpt, -- ** PCRE2-native substitution sub, @@ -260,14 +280,11 @@ AltCircumflex, AltVerbNames, Anchored, - AutoCallout, - BadEscapeIsLiteral, Bsr, Caseless, DepthLimit, DollarEndOnly, DotAll, - DupNames, EndAnchored, EscapedCrIsLf, Extended, @@ -302,23 +319,10 @@ SubUnknownUnset, SubUnsetEmpty, Ucp, - Ungreedy, - UnsafeCallout, - UnsafeCompileRecGuard, - UnsafeSubCallout, - Utf), + Ungreedy), Bsr(..), Newline(..), - - -- ** Callout interface - - CalloutInfo(..), - CalloutIndex(..), - CalloutResult(..), - -- *** Substitution callouts - SubCalloutInfo(..), - SubCalloutResult(..), -- * User errors SomePcre2Exception(), diff --git a/src/hs/Text/Regex/Pcre2/Internal.hs b/src/hs/Text/Regex/Pcre2/Internal.hs --- a/src/hs/Text/Regex/Pcre2/Internal.hs +++ b/src/hs/Text/Regex/Pcre2/Internal.hs @@ -19,9 +19,11 @@ import Control.Exception hiding (TypeError) import Control.Monad.State.Strict import Data.Either (partitionEithers) +import Data.Foldable (toList) import Data.Function ((&)) import Data.Functor ((<&>)) import Data.Functor.Const (Const(..)) +import Data.Functor.Identity (Identity(..)) import Data.IORef import Data.IntMap.Strict (IntMap) import qualified Data.IntMap.Strict as IM @@ -47,6 +49,8 @@ -- * General utilities +type FfiWrapper f = f -> IO (FunPtr f) + -- | There is no @nullForeignPtr@ to pass to `withForeignPtr`, so we have to -- fake it with a `Maybe`. withForeignOrNullPtr :: Maybe (ForeignPtr a) -> (Ptr a -> IO b) -> IO b @@ -93,23 +97,23 @@ -- ** Fast @Text@ slicing -data SliceRange = SliceRange +data Slice = Slice {-# UNPACK #-} !Text.I16 {-# UNPACK #-} !Text.I16 -- | Zero-copy slice a 'Text'. An unset capture is represented by a -- `pcre2_UNSET` range and is interpreted in this library as `Text.empty`. -thinSlice :: Text -> SliceRange -> Text -thinSlice text (SliceRange off offEnd) +thinSlice :: Text -> Slice -> Text +thinSlice text (Slice off offEnd) | off == fromIntegral pcre2_UNSET = Text.empty | otherwise = text & Text.takeWord16 offEnd & Text.dropWord16 off -- | Slice a 'Text', copying if it\'s less than half of the original. -slice :: Text -> SliceRange -> Text -slice text sliceRange = - let substring = thinSlice text sliceRange +smartSlice :: Text -> Slice -> Text +smartSlice text slice = + let substring = thinSlice text slice in if Text.length substring > Text.length text `div` 2 then substring else Text.copy substring @@ -147,13 +151,8 @@ toAlternativeOf :: (Alternative f) => Getting (Alt f a) s a -> s -> f a toAlternativeOf l = let alt = Alt . pure in getAlt . view (l . to alt) --- | See https://github.com/sjshuck/hs-pcre2/issues/17. --- This should go away with https://github.com/sjshuck/hs-pcre2/issues/18. -toAlternativeOf1 :: (Alternative f) => Getting (Alt Maybe a) s a -> s -> f a -toAlternativeOf1 l = maybe empty pure . preview l - -_headNE :: Lens' (NonEmpty a) a -_headNE f (x :| xs) = f x <&> (:| xs) +_Identity :: Lens' (Identity a) a +_Identity f (Identity x) = Identity <$> f x -- ** Streaming support @@ -236,6 +235,9 @@ -- an alternative to specifying them as an `Option`. See `Caseless` for -- example. -- +-- Most options are exported in "Text.Regex.Pcre2". The callout interface is +-- found in "Text.Regex.Pcre2.Unsafe". +-- -- Documentation is scant here. For more complete, accurate information, -- including discussions of corner cases arising from specific combinations of -- options and pattern items, please see the [C API @@ -255,8 +257,6 @@ | AltVerbNames -- ^ Enable backslash escapes in verb names. E.g., -- @(*MARK:L\\(O\\)L)@. | Anchored -- ^ Equivalent to beginning pattern with @^@. - | AutoCallout -- ^ Run callout for every pattern item. Only relevant if a - -- callout is set. | BadEscapeIsLiteral -- ^ Do not throw an error for unrecognized or -- malformed escapes. /\"This is a dangerous option.\"/ | Bsr Bsr -- ^ Override what @\\R@ matches (default given by `defaultBsr`). @@ -268,7 +268,6 @@ -- subject. | DotAll -- ^ A dot also matches a (single-character) newline. Equivalent -- to @(?s)@. - | DupNames -- ^ Allow non-unique capture names. | EndAnchored -- ^ More or less like ending pattern with @$@. | EscapedCrIsLf -- ^ Interpret @\\r@ as @\\n@. | Extended -- ^ In the pattern, ignore whitespace, and enable comments @@ -338,31 +337,26 @@ -- Incompatible with `NeverUcp`. | Ungreedy -- ^ Invert the effect of @?@. Without it, quantifiers are -- non-greedy; with it, they are greedy. Equivalent to @(?U)@. - | UnsafeCallout (CalloutInfo -> IO CalloutResult) -- ^ Run the given callout - -- at every callout point - -- (see [the docs](https://pcre.org/current/doc/html/pcre2callout.html) for - -- more info). Multiples of this option before the rightmost are ignored. - -- - -- /NOTE: The callout is run via `unsafePerformIO` within pure code!/ + | UnsafeCompileRecGuard (Int -> IO Bool) -- ^ Run the given guard on every -- new descent into a level of parentheses, passing the current depth as -- argument. Returning @False@ aborts pattern compilation with an -- exception. Multiples of this option before the rightmost are ignored. -- - -- /NOTE: Currently (PCRE2 version 10\.35) patterns seem to be parsed in/ + -- /NOTE: Currently (PCRE2 version 10\.37) patterns seem to be compiled in/ -- /two passes, both times triggering the recursion guard. Also, it is/ -- /triggered at the beginning of the pattern, passing 0. None of this is/ -- /documented; expect the unexpected in the presence of side effects!/ - -- - -- /NOTE: The guard is run via `unsafePerformIO` within pure code!/ + | UnsafeCallout (CalloutInfo -> IO CalloutResult) -- ^ Run the given callout + -- at every callout point + -- (see [the docs](https://pcre.org/current/doc/html/pcre2callout.html) for + -- more info). Multiples of this option before the rightmost are ignored. + | AutoCallout -- ^ Run callout for every pattern item. Only relevant if a + -- callout is set. | UnsafeSubCallout (SubCalloutInfo -> IO SubCalloutResult) -- ^ Run the -- given callout on every substitution. This is at most once unless -- `SubGlobal` is set. Multiples of this option before the rightmost are -- ignored. - -- - -- /NOTE: The callout is run via `unsafePerformIO` within pure code!/ - | Utf -- ^ Treat both the pattern and subject as UTF rather than fixed-width - -- 16-bit code units. instance Semigroup Option where (<>) = TwoOptions @@ -500,7 +494,6 @@ Caseless -> [CompileOption pcre2_CASELESS] DollarEndOnly -> [CompileOption pcre2_DOLLAR_ENDONLY] DotAll -> [CompileOption pcre2_DOTALL] - DupNames -> [CompileOption pcre2_DUPNAMES] EndAnchored -> [CompileOption pcre2_ENDANCHORED] Extended -> [CompileOption pcre2_EXTENDED] ExtendedMore -> [CompileOption pcre2_EXTENDED_MORE] @@ -516,7 +509,6 @@ NoStartOptimize -> [CompileOption pcre2_NO_START_OPTIMIZE] Ucp -> [CompileOption pcre2_UCP] Ungreedy -> [CompileOption pcre2_UNGREEDY] - Utf -> [CompileOption pcre2_UTF] -- ExtraCompileOption AltBsux -> [CompileExtraOption pcre2_EXTRA_ALT_BSUX] @@ -611,8 +603,8 @@ -- | Use a fake @Prism'@ to extract a category of options. extractOptsOf :: Getting (Alt Maybe a) AppliedOption a -> ExtractOpts [a] -extractOptsOf traversal = state $ partitionEithers . map discrim where - discrim opt = maybe (Right opt) Left $ preview traversal opt +extractOptsOf prism = state $ partitionEithers . map discrim where + discrim opt = maybe (Right opt) Left $ preview prism opt -- | Prepare to compile a `Code`. extractCompileEnv :: ExtractOpts CompileEnv @@ -962,15 +954,11 @@ calloutStateException :: Maybe SomeException, calloutStateSubsLog :: IntMap SubCalloutResult} --- | FFI wrapper. -foreign import ccall "wrapper" mkRecursionGuard - :: (CUInt -> Ptr a -> IO CInt) - -> IO (FunPtr (CUInt -> Ptr a -> IO CInt)) +foreign import ccall "wrapper" mkRecursionGuard :: FfiWrapper + (CUInt -> Ptr a -> IO CInt) --- | FFI wrapper. -foreign import ccall "wrapper" mkCallout - :: (Ptr block -> Ptr a -> IO CInt) - -> IO (FunPtr (Ptr block -> Ptr a -> IO CInt)) +foreign import ccall "wrapper" mkCallout :: FfiWrapper + (Ptr block -> Ptr a -> IO CInt) -- | Within a callout, marshal the original subject and @pcre2_callout_block@ -- data to Haskell and present to the user function. Ensure no pointers are @@ -1000,7 +988,7 @@ [start, end] <- forM [0, 1] $ peekElemOff ovecPtr . (n * 2 +) return $ if start == pcre2_UNSET then Nothing - else Just $ slice calloutSubject $ SliceRange + else Just $ smartSlice calloutSubject $ Slice (fromIntegral start) (fromIntegral end) @@ -1039,7 +1027,7 @@ [start, end] <- forM [0, 1] $ peekElemOff ovecPtr . (n * 2 +) return $ if start == pcre2_UNSET then Nothing - else Just $ slice subCalloutSubject $ SliceRange + else Just $ smartSlice subCalloutSubject $ Slice (fromIntegral start) (fromIntegral end) @@ -1060,25 +1048,24 @@ -- * Packaging @Matcher@s and @Subber@s as public API functions -- | The most general form of a matching function, which can also be used as a --- @Setter'@ to perform substitutions at the Haskell level. Operates globally. --- --- Internal only! Users should not (have to) know about `Matcher`. -_capturesInternal :: Matcher -> FromMatch -> Traversal' Text (NonEmpty Text) +-- @Setter'@ to perform substitutions at the Haskell level. +_capturesInternal :: (Traversable t) => + Matcher -> FromMatch t -> Traversal' Text (t Text) _capturesInternal matcher fromMatch f subject = - traverse f captureLists <&> \captureLists' -> + traverse f captureTs <&> \captureTs' -> -- Swag foldl-as-foldr to create only as many segments as we need to -- stitch back together and no more. let triples = concat $ zipWith3 zip3 - (map NE.toList sliceRangeLists) - (map NE.toList captureLists) - (map NE.toList captureLists') + (map toList sliceTs) + (map toList captureTs) + (map toList captureTs') in Text.concat $ foldr mkSegments termSegments triples 0 where - sliceRangeLists = unsafeLazyStreamToList $ mapMS fromMatch $ matcher subject - captureLists = map (NE.map $ slice subject) sliceRangeLists + sliceTs = unsafeLazyStreamToList $ mapMS fromMatch $ matcher subject + captureTs = map (smartSlice subject <$>) sliceTs - mkSegments (SliceRange off offEnd, c, c') r prevOffEnd + mkSegments (Slice off offEnd, c, c') r prevOffEnd | off == fromIntegral pcre2_UNSET || c == c' = -- This substring is unset or unchanged. Keep going without making -- cuts. @@ -1086,84 +1073,74 @@ | otherwise = -- Emit the subject up until here, and the new substring, and keep -- going, remembering where we are now. - thinSlice subject (SliceRange prevOffEnd off) : c' : r offEnd + thinSlice subject (Slice prevOffEnd off) : c' : r offEnd termSegments off = let offEnd = fromIntegral $ Text.length subject -- If the terminal segment is empty, omit it altogether. That way, - -- `Text.concat` can just return the subject without copying anything in + -- Text.concat can just return the subject without copying anything in -- cases where no substring is changed. - in [thinSlice subject (SliceRange off offEnd) | off /= offEnd] + in [thinSlice subject (Slice off offEnd) | off /= offEnd] --- | A function that takes a C match result and extracts a list of captures. We --- need to pass this effectful callback to `_capturesInternal` because of the --- latter\'s imperative loop that reuses the same @pcre2_match_data@ block. -type FromMatch = Ptr Pcre2_match_data -> IO (NonEmpty SliceRange) +-- | A function that takes a C match result and extracts captures into a +-- container. We need to pass this effectful callback to `_capturesInternal` +-- because of the latter\'s imperative loop that reuses the same +-- @pcre2_match_data@ block. +-- +-- The container type is polymorphic and in practice carries a `Traversable` +-- constraint. Currently the following containers are used: +-- +-- * `NonEmpty` when we want multiple capture groups; +-- +-- * `Identity` when we only want the 0th; +-- +-- * `Proxy` when we are only checking if a match succeeded. +type FromMatch t = Ptr Pcre2_match_data -> IO (t Slice) -- | Read all specifically indexed captures\' offsets from match results. -getWhitelistedSliceRanges :: NonEmpty Int -> FromMatch -getWhitelistedSliceRanges whitelist matchDataPtr = do +getWhitelistedSlices :: (Traversable t) => t Int -> FromMatch t +getWhitelistedSlices whitelist matchDataPtr = do ovecPtr <- pcre2_get_ovector_pointer matchDataPtr let peekOvec :: Int -> IO Text.I16 peekOvec = fmap fromIntegral . peekElemOff ovecPtr - forM whitelist $ \i -> liftM2 SliceRange + forM whitelist $ \i -> liftM2 Slice (peekOvec $ i * 2) (peekOvec $ i * 2 + 1) -- | Read just the 0th capture\'s offsets from match results. -get0thSliceRanges :: FromMatch -get0thSliceRanges = getWhitelistedSliceRanges $ 0 :| [] +get0thSlice :: FromMatch Identity +get0thSlice = getWhitelistedSlices $ Identity 0 -- | Read all captures\' offsets from match results. -getAllSliceRanges :: FromMatch -getAllSliceRanges matchDataPtr = do +getAllSlices :: FromMatch NonEmpty +getAllSlices matchDataPtr = do count <- fromIntegral <$> pcre2_get_ovector_count matchDataPtr let whitelist = 0 :| [1 .. count - 1] - getWhitelistedSliceRanges whitelist matchDataPtr - --- | Placeholder for half-building a `Traversal'` to be passed to `has`. -errorFromMatch :: FromMatch -errorFromMatch _ = return $ error "BUG! Tried to use match results" - --- | Match a pattern to a subject once and return a list of captures, or @[]@ if --- no match. -captures :: Text -> Text -> [Text] -captures = capturesOpt mempty + getWhitelistedSlices whitelist matchDataPtr --- | @capturesOpt mempty = captures@ -capturesOpt :: Option -> Text -> Text -> [Text] -capturesOpt option patt = maybe [] NE.toList . capturesAOpt option patt +-- | Placeholder for building a `Traversal'` to be passed to `has`. +nilFromMatch :: FromMatch Proxy +nilFromMatch _ = return Proxy --- | Match a pattern to a subject once and return a non-empty list of captures +-- | Match a pattern to a subject and return some non-empty list(s) of captures -- in an `Alternative`, or `empty` if no match. The non-empty list constructor -- `:|` serves as a cue to differentiate the 0th capture from the others: -- --- > let parseDate = capturesA "(\\d{4})-(\\d{2})-(\\d{2})" +-- > let parseDate = captures "(\\d{4})-(\\d{2})-(\\d{2})" -- > in case parseDate "submitted 2020-10-20" of -- > Just (date :| [y, m, d]) -> ... -- > Nothing -> putStrLn "didn't match" -capturesA :: (Alternative f) => Text -> Text -> f (NonEmpty Text) -capturesA = capturesAOpt mempty - --- | @capturesAOpt mempty = capturesA@ -- --- @since 1.1.0 -capturesAOpt :: (Alternative f) => Option -> Text -> Text -> f (NonEmpty Text) -capturesAOpt option patt = toAlternativeOf1 $ _capturesOpt option patt - --- | Match a pattern to a subject and lazily produce a list of all --- non-overlapping portions, with all capture groups, that matched. --- --- @since 1.1.0 -capturesAll :: Text -> Text -> [NonEmpty Text] -capturesAll = capturesAllOpt mempty +-- @since 2.0.0 +captures :: (Alternative f) => Text -> Text -> f (NonEmpty Text) +captures = capturesOpt mempty --- | @capturesAllOpt mempty = capturesAll@ +-- | @capturesOpt mempty = captures@ -- --- @since 1.1.0 -capturesAllOpt :: Option -> Text -> Text -> [NonEmpty Text] -capturesAllOpt option patt = toListOf $ _capturesOpt option patt +-- @since 2.0.0 +capturesOpt :: (Alternative f) => Option -> Text -> Text -> f (NonEmpty Text) +capturesOpt option patt = toAlternativeOf $ _capturesOpt option patt -- | Does the pattern match the subject at least once? matches :: Text -> Text -> Bool @@ -1171,30 +1148,19 @@ -- | @matchesOpt mempty = matches@ matchesOpt :: Option -> Text -> Text -> Bool -matchesOpt option patt = has $ _capturesInternal matcher errorFromMatch where +matchesOpt option patt = has $ _capturesInternal matcher nilFromMatch where matcher = unsafePerformIO $ assembleMatcher option patt --- | Match a pattern to a subject once and return the portion that matched in an +-- | Match a pattern to a subject and return the portion(s) that matched in an -- `Alternative`, or `empty` if no match. +-- +-- @since 2.0.0 match :: (Alternative f) => Text -> Text -> f Text match = matchOpt mempty -- | @matchOpt mempty = match@ matchOpt :: (Alternative f) => Option -> Text -> Text -> f Text -matchOpt option patt = toAlternativeOf1 $ _matchOpt option patt - --- | Match a pattern to a subject and lazily return a list of all --- non-overlapping portions that matched. --- --- @since 1.1.0 -matchAll :: Text -> Text -> [Text] -matchAll = matchAllOpt mempty - --- | @matchAllOpt mempty = matchAll@ --- --- @since 1.1.0 -matchAllOpt :: Option -> Text -> Text -> [Text] -matchAllOpt option patt = toListOf $ _matchOpt option patt +matchOpt option patt = toAlternativeOf $ _matchOpt option patt -- | Perform at most one substitution. See -- [the docs](https://pcre.org/current/doc/html/pcre2api.html#SEC36) for the @@ -1225,7 +1191,7 @@ subber = unsafePerformIO $ assembleSubber replacement option patt -- | Given a pattern, produce a traversal (0 or more targets) that focuses from --- a subject to each non-empty list of captures that pattern matches globally. +-- a subject to each non-empty list of captures that pattern matches. -- -- Substitution works in the following way: If a capture is set such that the -- new `Text` is not equal to the old one, a substitution occurs, otherwise it @@ -1266,11 +1232,11 @@ -- | @_capturesOpt mempty = _captures@ _capturesOpt :: Option -> Text -> Traversal' Text (NonEmpty Text) -_capturesOpt option patt = _capturesInternal matcher getAllSliceRanges where +_capturesOpt option patt = _capturesInternal matcher getAllSlices where matcher = unsafePerformIO $ assembleMatcher option patt -- | Given a pattern, produce a traversal (0 or more targets) that focuses from --- a subject to the portions of it that match. +-- a subject to the non-overlapping portions of it that match. -- -- @_match = `_captures` patt . ix 0@ _match :: Text -> Traversal' Text Text @@ -1278,8 +1244,8 @@ -- | @_matchOpt mempty = _match@ _matchOpt :: Option -> Text -> Traversal' Text Text -_matchOpt option patt = _cs . _headNE where - _cs = _capturesInternal matcher get0thSliceRanges +_matchOpt option patt = _cs . _Identity where + _cs = _capturesInternal matcher get0thSlice matcher = unsafePerformIO $ assembleMatcher option patt -- * Support for Template Haskell compile-time regex analysis @@ -1352,7 +1318,11 @@ -- else num - CaptNum (name :: Symbol) '(_, '(name, num) ': _) = num + -- FIXME See Text.Regex.Pcre2.TH, where we end the lookup table with a + -- placeholder entry due to a GHC bug. We must ensure a successful name + -- lookup does not occur on this last entry; hence the extra promoted cons + -- operators in this pattern. + CaptNum (name :: Symbol) '(_, '(name, num) ': _ ': _) = num CaptNum (name :: Symbol) '(hi, _ ': kvs) = CaptNum name '(hi, kvs) CaptNum (name :: Symbol) _ = TypeError (TypeLits.Text "No capture named " :<>: ShowType name) @@ -1374,7 +1344,7 @@ -- > capture @3 -- > capture @"bar" -- --- Specifying a nonexistant number or name will result in a type error. +-- Specifying a nonexistent number or name will result in a type error. capture :: forall i info num. (CaptNum i info ~ num, KnownNat num) => Captures info -> Text capture = view $ _capture @i diff --git a/src/hs/Text/Regex/Pcre2/TH.hs b/src/hs/Text/Regex/Pcre2/TH.hs --- a/src/hs/Text/Regex/Pcre2/TH.hs +++ b/src/hs/Text/Regex/Pcre2/TH.hs @@ -1,14 +1,18 @@ +{-# LANGUAGE DataKinds #-} {-# LANGUAGE LambdaCase #-} -{-# LANGUAGE QuasiQuotes #-} +{-# LANGUAGE RankNTypes #-} {-# LANGUAGE TemplateHaskell #-} module Text.Regex.Pcre2.TH where +import Control.Applicative (Alternative(..)) +import Data.Functor ((<&>)) import Data.IORef -import Data.List.NonEmpty (NonEmpty) +import Data.List.NonEmpty (NonEmpty(..)) import qualified Data.List.NonEmpty as NE import Data.Map.Lazy (Map) import qualified Data.Map.Lazy as Map +import Data.Proxy (Proxy(..)) import Data.Text (Text) import qualified Data.Text as Text import Language.Haskell.TH @@ -23,7 +27,7 @@ globalMatcherCache = unsafePerformIO $ newIORef Map.empty {-# NOINLINE globalMatcherCache #-} --- | Given a `Text`, create or retrieve a `Matcher` from the global cache. +-- | Given a pattern, create or retrieve a `Matcher` from the global cache. memoMatcher :: Text -> Matcher memoMatcher patt = unsafePerformIO $ do cache <- readIORef globalMatcherCache @@ -35,10 +39,6 @@ (Map.insert patt matcher cache, ()) return matcher --- | Generate code to produce \(and memoize\) a `Matcher` from a pattern. -matcherQ :: String -> ExpQ -matcherQ s = [e| memoMatcher $ Text.pack $(stringE s) |] - -- | Predict parenthesized captures \(maybe named\) of a pattern at splice time. predictCaptureNamesQ :: String -> Q [Maybe Text] predictCaptureNamesQ = runIO . predictCaptureNames mempty . Text.pack @@ -55,21 +55,57 @@ [] -> return Nothing -- One or more parenthesized captures. Present - -- [Just "foo", Just "bar", Nothing] + -- [Just "foo", Nothing, Nothing, Just "bar"] -- as - -- '(3, '[ '("foo", 1), '("bar", 2)]). + -- '(4, '[ '("foo", 1), '("bar", 4), '("", 0)]). captureNames -> Just <$> promotedTupleT 2 `appT` hi `appT` kvs where - -- 3 + -- 4 hi = litT $ numTyLit $ fromIntegral $ length captureNames - -- '[ '("foo", 1), '("bar", 2)] - kvs = foldr f promotedNilT $ toKVs captureNames where + -- '[ '("foo", 1), '("bar", 4), '("", 0)] + kvs = foldr f end (toKVs captureNames) where -- '("foo", 1) ': ... - f (number, name) = appT $ appT promotedConsT $ -- ': - promotedTupleT 2 -- '(,) + f (number, name) r = promotedConsT `appT` kv `appT` r where + kv = promotedTupleT 2 -- '(,) `appT` litT (strTyLit $ Text.unpack name) -- "foo" `appT` litT (numTyLit $ fromIntegral number) -- 1 + -- FIXME GHC kind-checks empty '[] as [*] instead of [k], which breaks + -- quasi-quoted regexes with parenthesized captures but no names. + -- Therefore, we avoid ever splicing an empty lookup table by ending it + -- with a placeholder entry (which cannot arise from a pattern since "" + -- is an invalid capture group name). + end = [t| '[ '("", 0)] |] --- | /__As an expression__/ +-- | Helper for `regex` with no parenthesized captures. +matchTH :: (Alternative f) => Text -> Text -> f Text +matchTH patt = toAlternativeOf $ + _capturesInternal (memoMatcher patt) get0thSlice . _Identity + +-- | Helper for `regex` with parenthesized captures. +capturesTH :: (Alternative f) => Text -> Proxy info -> Text -> f (Captures info) +capturesTH patt _ = toAlternativeOf $ + _capturesInternal (memoMatcher patt) getAllSlices . to Captures + +-- | Helper for `regex` as a guard pattern. +matchesTH :: Text -> Text -> Bool +matchesTH patt = has $ _capturesInternal (memoMatcher patt) nilFromMatch + +-- | Helper for `regex` as a pattern that binds local variables. +capturesNumberedTH :: Text -> NonEmpty Int -> Text -> [Text] +capturesNumberedTH patt numbers = concatMap NE.toList . toListOf _cs where + _cs = _capturesInternal (memoMatcher patt) fromMatch + fromMatch = getWhitelistedSlices numbers + +-- | Helper for `_regex` with no parenthesized captures. +_matchTH :: Text -> Traversal' Text Text +_matchTH patt = _capturesInternal (memoMatcher patt) get0thSlice . _Identity + +-- | Helper for `_regex` with parenthesized captures. +_capturesTH :: Text -> Proxy info -> Traversal' Text (Captures info) +_capturesTH patt _ = _cs . wrapped where + _cs = _capturesInternal (memoMatcher patt) getAllSlices + wrapped f cs = f (Captures cs) <&> \(Captures cs') -> cs' + +-- | === As an expression -- -- > regex :: (Alternative f) => String -> Text -> f (Captures info) -- @@ -78,8 +114,8 @@ -- > regex :: (Alternative f) => String -> Text -> f Text -- -- if there are none. In other words, if there is more than the 0th capture, --- this behaves like `capturesA` (except returning an opaque `Captures` instead --- of a list), otherwise it behaves like `match`. +-- this behaves like `captures` (except returning an opaque `Captures` instead +-- of a `NonEmpty` list), otherwise it behaves like `match`. -- -- To retrieve an individual capture from a `Captures`, use `capture`. -- @@ -92,7 +128,7 @@ -- > forM_ ([regex|\s+$|] line :: Maybe Text) $ \spaces -> error $ -- > "line has trailing spaces (" ++ show (Text.length spaces) ++ " characters)" -- --- /__As a pattern__/ +-- === As a pattern -- -- This matches when the regex first matches, whereupon any named captures are -- bound to variables of the same names. @@ -108,14 +144,11 @@ -- If there are no named captures, this simply acts as a guard. regex :: QuasiQuoter regex = QuasiQuoter { - quoteExp = \s -> do - let fromCsQ = capturesInfoQ s >>= maybe [e| _headNE |] toWrapQ - toWrapQ info = [e| - let wrap cs = Captures cs :: Captures $(return info) - in to wrap |] - - [e| toAlternativeOf1 $ - _capturesInternal $(matcherQ s) getAllSliceRanges . $(fromCsQ) |], + quoteExp = \s -> capturesInfoQ s >>= \case + Nothing -> [e| matchTH (Text.pack $(stringE s)) |] + Just info -> [e| capturesTH + (Text.pack $(stringE s)) + (Proxy :: Proxy $(return info)) |], quotePat = \s -> do captureNames <- predictCaptureNamesQ s @@ -124,18 +157,16 @@ -- No named captures. Test whether the string matches without -- creating any new Text values. Nothing -> viewP - [e| has $ _capturesInternal $(matcherQ s) errorFromMatch |] + [e| matchesTH (Text.pack $(stringE s)) |] [p| True |] -- One or more named captures. Attempt to bind only those to local -- variables of the same names. Just numberedNames -> viewP e p where (numbers, names) = NE.unzip numberedNames - e = [e| - let _cs = _capturesInternal - $(matcherQ s) - (getWhitelistedSliceRanges $(liftData numbers)) - in view $ _cs . to NE.toList |] + e = [e| capturesNumberedTH + (Text.pack $(stringE s)) + $(liftData numbers) |] p = foldr f wildP names where f name r = conP '(:) [varP $ mkName $ Text.unpack name, r], @@ -143,7 +174,7 @@ quoteDec = const $ fail "regex: cannot produce declarations"} --- | A global, optical variant of `regex`. Can only be used as an expression. +-- | An optical variant of `regex`. Can only be used as an expression. -- -- > _regex :: String -> Traversal' Text (Captures info) -- > _regex :: String -> Traversal' Text Text @@ -158,17 +189,13 @@ -- > main = putStrLn $ "There are 14 competing standards" & embeddedNumber %~ (+ 1) -- > -- > -- There are 15 competing standards --- _regex :: QuasiQuoter _regex = QuasiQuoter { - quoteExp = \s -> do - let fromCsQ = capturesInfoQ s >>= maybe [e| _headNE |] wrappedQ - wrappedQ info = [e| - let wrapped :: Lens' (NonEmpty Text) (Captures $(return info)) - wrapped f cs = f (Captures cs) <&> \(Captures cs') -> cs' - in wrapped |] - - [e| _capturesInternal $(matcherQ s) getAllSliceRanges . $(fromCsQ) |], + quoteExp = \s -> capturesInfoQ s >>= \case + Nothing -> [e| _matchTH (Text.pack $(stringE s)) |] + Just info -> [e| _capturesTH + (Text.pack $(stringE s)) + (Proxy :: Proxy $(return info)) |], quotePat = const $ fail "_regex: cannot produce a pattern", diff --git a/src/hs/Text/Regex/Pcre2/Unsafe.hs b/src/hs/Text/Regex/Pcre2/Unsafe.hs new file mode 100644 --- /dev/null +++ b/src/hs/Text/Regex/Pcre2/Unsafe.hs @@ -0,0 +1,32 @@ +-- | These items are unsafe for one reason or another, and are sequestered here +-- to require the user to do an extra import to get them. +-- +-- Chief among +-- them is the callout interface: these options and associated datatypes may be +-- used to register effectful callbacks, sometimes referred to as /callouts/ in +-- the PCRE2 API, for regex compilation, matching, and substitution. We include +-- them here for completeness and use them to implement unit tests for this +-- library; for ordinary use, however, seek other means to accomplish whatever +-- is needed (such as accreting effects with optics), since they carry all the +-- problems of `unsafePerformIO`. See +-- the [C API docs](https://pcre.org/current/doc/html/pcre2callout.html) +-- for more information. +module Text.Regex.Pcre2.Unsafe ( + -- ** Options + Option( + AutoCallout, + BadEscapeIsLiteral, + UnsafeCallout, + UnsafeCompileRecGuard, + UnsafeSubCallout), + + -- ** Types + CalloutInfo(..), + CalloutIndex(..), + CalloutResult(..), + SubCalloutInfo(..), + SubCalloutResult(..)) +where + +import System.IO.Unsafe (unsafePerformIO) +import Text.Regex.Pcre2.Internal diff --git a/test/Spec.hs b/test/Spec.hs --- a/test/Spec.hs +++ b/test/Spec.hs @@ -1,22 +1,23 @@ {-# LANGUAGE DataKinds #-} {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE QuasiQuotes #-} -{-# LANGUAGE ScopedTypeVariables #-} {-# LANGUAGE TypeApplications #-} {-# LANGUAGE ViewPatterns #-} module Main where -import Control.Applicative (Alternative) -import Control.Exception (catch, evaluate, handle) -import Control.Monad.RWS.Lazy (ask, evalRWS, forM_, tell, void) +import Control.Applicative (Alternative) +import Control.Exception (catch, evaluate, handle) +import Control.Monad.RWS.Lazy (ask, evalRWS, forM_, tell, void) import Data.IORef -import Data.List.NonEmpty (NonEmpty(..)) -import Data.Text (Text) -import qualified Data.Text as Text +import Data.List.NonEmpty (NonEmpty(..)) +import Data.Proxy (Proxy(..)) +import Data.Text (Text) +import qualified Data.Text as Text import Lens.Micro.Platform import Test.Hspec import Text.Regex.Pcre2 +import Text.Regex.Pcre2.Unsafe main :: IO () main = hspec $ do @@ -39,11 +40,20 @@ it "works using matches" $ do matches "(?i)foo" "FOO" `shouldBe` True - it "works using capturesA" $ do - case capturesA "(\\d{4})-(\\d{2})-(\\d{2})" submitted of + it "works using captures" $ do + case captures "(\\d{4})-(\\d{2})-(\\d{2})" submitted of Just ne -> ne `shouldBe` "2020-10-20" :| ["2020", "10", "20"] Nothing -> expectationFailure "didn't match" + it "is lazy" $ do + counter <- newIORef (0 :: Int) + let callout = UnsafeCallout $ \_ -> do + modifyIORef counter (+ 1) + return CalloutProceed + take 3 (matchOpt callout "(?C1)a" "apples and bananas") + `shouldBe` ["a", "a", "a"] + readIORef counter `shouldReturn` 3 + describe "lens-powered matching" $ do let _nee :: Traversal' Text Text _nee = _match "(?i)\\bnee\\b" @@ -53,14 +63,14 @@ let result = evalRWS (promptNee "We are the knights who say...NEE!") "NOO" - undefined + Proxy result `shouldBe` ("We are the knights who say...NOO!", ["NEE"]) it "signals match failure by not targeting anything" $ do let result = evalRWS (promptNee "Shhhhh") - undefined - undefined + (error "should be unreachable") + Proxy result `shouldBe` ("Shhhhh", []) it "does not substitute when setting equal Text" $ do @@ -113,13 +123,13 @@ `shouldThrow` anyPcre2Exception it "are catchable using Control.Exception.evaluate" $ do - let example = evaluate (broken "foo") `catch` - \(_ :: SomePcre2Exception) -> return Nothing + let example = handle @SomePcre2Exception (\_ -> return Nothing) $ + evaluate $ broken "foo" example `shouldReturn` Nothing it "are catchable using instance Alternative IO" $ do - let example = broken "foo" `catch` - \(_ :: SomePcre2Exception) -> return "broken" + let example = handle @SomePcre2Exception (\_ -> return "broken") $ + broken "foo" example `shouldReturn` "broken" describe "native substitution" $ do @@ -180,30 +190,21 @@ describe "an unset capture" $ do it "is treated as empty" $ do - captures "(a)?" "" `shouldBe` ["", ""] + captures "(a)?" "" `shouldBe` Just ("" :| [""]) it "is unchanged via Traversal'" $ do - ("" & [_regex|(a)?|] . _capture @1 .~ "foo") `shouldBe` "" + set ([_regex|(a)?|] . _capture @1) "foo" "" `shouldBe` "" it "permits other captures to be changed via Traversal'" $ do - ("" & [_regex|(a)?|] . _capture @0 .~ "foo") `shouldBe` "foo" + set ([_regex|(a)?|] . _capture @0) "foo" "" `shouldBe` "foo" describe "Traversal'" $ do it "supports global substitutions" $ do - ("apples and bananas" & _match "a" .~ "o") + set (_match "a") "o" "apples and bananas" `shouldBe` "opples ond bononos" - it "is lazy" $ do - counter <- newIORef (0 :: Int) - let callout = UnsafeCallout $ \_ -> do - modifyIORef counter (+ 1) - return CalloutProceed - take 3 ("apples and bananas" ^.. _matchOpt callout "(?C1)a") - `shouldBe` ["a", "a", "a"] - readIORef counter `shouldReturn` 3 - it "converges in the presence of empty matches" $ do - length (toListOf [_regex||] "12345") `shouldBe` 6 + length (match @[] "" "12345") `shouldBe` 6 describe "bug fixes" bugFixes @@ -214,15 +215,11 @@ f "a" `shouldReturn` "" f "b" `shouldReturn` "b" - issue 12 $ do - captures ".a." "foo bar baz" `shouldBe` ["bar"] - case "foo bar baz" of - [regex|.a(?.)|] -> return () - _ -> - expectationFailure "quasi-quoted pattern didn't match" - - issue 17 $ do - match "\\d+" "123 456" `shouldBe` ["123"] + issue 18 $ do + let result :: (Alternative f) => f Text + result = match "\\d+" "123 456" + result `shouldBe` Just "123" + result `shouldBe` ["123", "456"] where issue :: Int -> Expectation -> Spec @@ -252,7 +249,7 @@ broken :: (Alternative f) => Text -> f Text broken = match "*" --- microlens doesn't have this yet as of 12/5/2020 +-- microlens doesn't have this yet as of 06/26/2021 _Show :: (Read a, Show a) => Traversal' String a _Show f s = case reads s of [(x, "")] -> show <$> f x