diff --git a/README.md b/README.md --- a/README.md +++ b/README.md @@ -63,6 +63,7 @@ the regular expression says it has found a numeric string but the string-to-number conversion function fails. A typed parser will perform both the pattern match and the conversion, so it will never be in that situation. + [Parse, don't validate.](https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/) * Regular expressions are only able to pattern-match [regular](https://en.wikipedia.org/wiki/Chomsky_hierarchy#The_hierarchy) @@ -227,7 +228,7 @@ #!/usr/bin/env stack {- stack script - --resolver nightly-2019-09-13 + --resolver lts-15 --package attoparsec --package text --package text-show @@ -322,27 +323,28 @@ | Program | dense | sparse | | :--- | ---: | ---: | -| Python `re.sub`¹ | 89.23ms | 23.98ms | -| Perl `s///ge`² | 180.65ms | 5.60ms | -| [`Replace.Megaparsec.streamEdit`][m] `String` | 454.95ms | 375.04ms | +| [Python 3.7.4 `re.sub`][sub] *repl* function | 89.23ms | 23.98ms | +| [Perl 5 `s///ge`][s] | 180.65ms | 5.02ms | +| [`Replace.Megaparsec.streamEdit`][m] `String` | 441.94ms | 375.04ms | | [`Replace.Megaparsec.streamEdit`][m] `ByteString` | 529.99ms | 73.76ms | | [`Replace.Megaparsec.streamEdit`][m] `Text` | 547.47ms | 139.21ms | | [`Replace.Attoparsec.ByteString.streamEdit`][ab] | 394.12ms | 41.13ms | | [`Replace.Attoparsec.Text.streamEdit`][at] | 515.26ms | 46.10ms | | [`Text.Regex.Applicative.replace`][ra] `String` | 1083.98ms | 646.40ms | -| [`Text.Regex.PCRE.Heavy.gsub`][ph] `Text` | ⊥³ | 14.76ms | - -¹ Python 3.7.4 - -² This is perl 5, version 28, subversion 2 (v5.28.2) built for x86_64-linux-thread-multi - -³ Does not finish. +| [`Text.Regex.PCRE.Heavy.gsub`][ph] `Text` | > 10min | 14.29ms | +| [`Control.Lens.Regex.ByteString.match`][lb] | > 10min | 4.27ms | +| [`Control.Lens.Regex.Text.match`][lt] | > 10min | 14.74ms | +[sub]: https://docs.python.org/3/library/re.html#re.sub +[s]: https://perldoc.perl.org/functions/s.html [m]: https://hackage.haskell.org/package/replace-megaparsec/docs/Replace-Megaparsec.html#v:streamEdit [ab]: https://hackage.haskell.org/package/replace-attoparsec/docs/Replace-Attoparsec-ByteString.html#v:streamEdit [at]: https://hackage.haskell.org/package/replace-attoparsec/docs/Replace-Attoparsec-Text.html#v:streamEdit [ra]: http://hackage.haskell.org/package/regex-applicative/docs/Text-Regex-Applicative.html#v:replace +[ss]: http://hackage.haskell.org/package/stringsearch/docs/Data-ByteString-Search.html#v:replace [ph]: http://hackage.haskell.org/package/pcre-heavy/docs/Text-Regex-PCRE-Heavy.html#v:gsub +[lb]: https://hackage.haskell.org/package/lens-regex-pcre/docs/Control-Lens-Regex-ByteString.html#v:match +[lt]: https://hackage.haskell.org/package/lens-regex-pcre/docs/Control-Lens-Regex-Text.html#v:match # Hypothetically Asked Questions @@ -356,19 +358,22 @@ 2. *Is this a good idea?* - You may have heard it suggested that monadic parsers are better when + You may have + [heard it suggested](https://stackoverflow.com/questions/57667534/how-can-i-use-a-parser-in-haskell-to-find-the-locations-of-some-substrings-in-a/57712672#comment101804063_57667534) + that monadic parsers are better for pattern-matching when the input stream is mostly signal, and regular expressions are better when the input stream is mostly noise. - The premise of this library is: - that sentiment is outdated; monadic parsers are great for finding - small patterns in a stream of otherwise uninteresting text; and the - reluctance to forego the speedup opportunities afforded by restricting + The premise of this library is that monadic parsers are great for finding + small signal patterns in a stream of otherwise noisy text. + + Our reluctance to forego the speedup opportunities afforded by restricting ourselves to regular grammars is an old superstition about opportunities which [remain mostly unexploited anyway](https://swtch.com/~rsc/regexp/regexp1.html). - The performance compromise of allowing stack memory allocation (a.k.a pushdown - automata, a.k.a context-free grammar) was once considered - [controversial for *general-purpose* programming languages](https://vanemden.wordpress.com/2014/06/18/how-recursion-got-into-programming-a-comedy-of-errors-3/). I think we + The performance compromise of allowing stack memory allocation (a.k.a. pushdown + automata, a.k.a. context-free grammar) was once considered + [controversial for *general-purpose* programming languages](https://vanemden.wordpress.com/2014/06/18/how-recursion-got-into-programming-a-comedy-of-errors-3/). + I think we can now resolve that controversy the same way for pattern matching languages. diff --git a/replace-attoparsec.cabal b/replace-attoparsec.cabal --- a/replace-attoparsec.cabal +++ b/replace-attoparsec.cabal @@ -1,5 +1,5 @@ name: replace-attoparsec -version: 1.2.0.0 +version: 1.2.1.0 cabal-version: 1.18 synopsis: Find, replace, and edit text patterns with Attoparsec parsers homepage: https://github.com/jamesdbrock/replace-attoparsec diff --git a/src/Replace/Attoparsec/ByteString.hs b/src/Replace/Attoparsec/ByteString.hs --- a/src/Replace/Attoparsec/ByteString.hs +++ b/src/Replace/Attoparsec/ByteString.hs @@ -58,34 +58,45 @@ -- == Separate and capture -- -- Parser combinator to find all of the non-overlapping ocurrences --- of the pattern @sep@ in a text stream. Separate the stream into sections: +-- of the pattern @sep@ in a text stream. +-- The 'sepCap' parser will always consume its entire input and can never fail. -- --- * sections which can parsed by the pattern @sep@ will be captured as --- matching sections in 'Right' --- * non-matching sections of the stream will be captured in 'Left'. +-- === Output -- --- This parser will always consume its entire input and can never fail. --- If there are no pattern matches, then the entire input stream will be --- returned as a non-matching 'Left' section. +-- The input stream is separated and output int a list of sections: -- --- The pattern matching parser @sep@ will not be allowed to succeed without --- consuming any input. If we allow the parser to match a zero-width pattern, +-- * Sections which can parsed by the pattern @sep@ will be parsed and captured +-- as 'Right' +-- * Non-matching sections of the stream will be captured in 'Left'. +-- +-- The output list also has these properties: +-- +-- * If the input is @""@ then the output list will be @[]@. +-- * If there are no pattern matches, then +-- the entire input stream will be returned as one non-matching 'Left' section. +-- * The output list will not contain two consecutive 'Left' sections. +-- +-- === Zero-width matches forbidden +-- +-- If the pattern matching parser @sep@ would succeed without consuming any +-- input then 'sepCap' will force it to fail. +-- If we allow @sep@ to match a zero-width pattern, -- then it can match the same zero-width pattern again at the same position -- on the next iteration, which would result in an infinite number of --- overlapping pattern matches. So, for example, the --- pattern @many digit@, which can match zero occurences of a digit, --- will be treated by @sepCap@ as @many1 digit@, and required to match --- at least one digit. +-- overlapping pattern matches. -- +-- === Notes +-- -- This @sepCap@ parser combinator is the basis for all of the other --- features of this module. It is similar to the @sep*@ family of functions --- found in +-- features of this module. +-- +-- It is similar to the @sep*@ family of functions found in -- -- and -- -- but, importantly, it returns the parsed result of the @sep@ parser instead --- of throwing it away. --- +-- of throwing it away, like +-- . sepCap :: Parser a -- ^ The pattern matching parser @sep@ -> Parser [Either B.ByteString a] @@ -228,8 +239,8 @@ -- -- This allows us to write an @editor@ function which can choose to not -- edit the match and just leave it as it is. If the @editor@ function --- always returns the first item in the tuple, then @streamEdit@ changes --- nothing. +-- returns the first item in the tuple, then @streamEdit@ will not change +-- the matched string. -- -- So, for all @sep@: -- diff --git a/src/Replace/Attoparsec/Text.hs b/src/Replace/Attoparsec/Text.hs --- a/src/Replace/Attoparsec/Text.hs +++ b/src/Replace/Attoparsec/Text.hs @@ -59,33 +59,46 @@ -- == Separate and capture -- -- Parser combinator to find all of the non-overlapping ocurrences --- of the pattern @sep@ in a text stream. Separate the stream into sections: +-- of the pattern @sep@ in a text stream. +-- The 'sepCap' parser will always consume its entire input and can never fail. -- --- * sections which can parsed by the pattern @sep@ will be captured as --- matching sections in 'Right' --- * non-matching sections of the stream will be captured in 'Left'. +-- === Output -- --- This parser will always consume its entire input and can never fail. --- If there are no pattern matches, then the entire input stream will be --- returned as a non-matching 'Left' section. +-- The input stream is separated and output int a list of sections: -- --- The pattern matching parser @sep@ will not be allowed to succeed without --- consuming any input. If we allow the parser to match a zero-width pattern, +-- * Sections which can parsed by the pattern @sep@ will be parsed and captured +-- as 'Right' +-- * Non-matching sections of the stream will be captured in 'Left'. +-- +-- The output list also has these properties: +-- +-- * If the input is @""@ then the output list will be @[]@. +-- * If there are no pattern matches, then +-- the entire input stream will be returned as one non-matching 'Left' section. +-- * The output list will not contain two consecutive 'Left' sections. +-- +-- === Zero-width matches forbidden +-- +-- If the pattern matching parser @sep@ would succeed without consuming any +-- input then 'sepCap' will force it to fail. +-- If we allow @sep@ to match a zero-width pattern, -- then it can match the same zero-width pattern again at the same position -- on the next iteration, which would result in an infinite number of --- overlapping pattern matches. So, for example, the --- pattern @many digit@, which can match zero occurences of a digit, --- will be treated by @sepCap@ as @many1 digit@, and required to match --- at least one digit. +-- overlapping pattern matches. -- +-- === Notes +-- -- This @sepCap@ parser combinator is the basis for all of the other --- features of this module. It is similar to the @sep*@ family of functions --- found in +-- features of this module. +-- +-- It is similar to the @sep*@ family of functions found in -- -- and -- -- but, importantly, it returns the parsed result of the @sep@ parser instead --- of throwing it away. +-- of throwing it away, like +-- . + -- sepCap :: Parser a -- ^ The pattern matching parser @sep@ @@ -247,8 +260,8 @@ -- -- This allows us to write an @editor@ function which can choose to not -- edit the match and just leave it as it is. If the @editor@ function --- always returns the first item in the tuple, then @streamEdit@ changes --- nothing. +-- returns the first item in the tuple, then @streamEdit@ will not change +-- the matched string. -- -- So, for all @sep@: -- diff --git a/tests/TestByteString.hs b/tests/TestByteString.hs --- a/tests/TestByteString.hs +++ b/tests/TestByteString.hs @@ -12,7 +12,6 @@ import "parsers" Text.Parser.Token import Replace.Attoparsec.ByteString import Control.Applicative -import Data.Monoid tests :: IO [Test] tests = return @@ -44,10 +43,13 @@ (sepCap (string "aa")) (" a") ("a ") ([Left " ",Right"aa",Left" "]) + , Test $ runParserTest "empty input" (sepCap (fail "" :: Parser ())) "" [] , Test $ streamEditTest "x to o" (string "x") (const "o") "x x x" "o o o" , Test $ streamEditTest "x to o inner" (string "x") (const "o") " x x x " " o o o " , Test $ streamEditTest "ordering" (string "456") (const "ABC") "123456789" "123ABC789" + , Test $ streamEditTest "empty input" (match (fail "")) (fst) "" "" ] + where runParserTest nam p input expected = TestInstance { run = do diff --git a/tests/TestText.hs b/tests/TestText.hs --- a/tests/TestText.hs +++ b/tests/TestText.hs @@ -7,11 +7,9 @@ import Distribution.TestSuite as TestSuite import Replace.Attoparsec.Text import Data.Attoparsec.Text as A --- import Data.Attoparsec.Text.Char import qualified Data.Text as T import Text.Parser.Char (upper) import Control.Applicative -import Data.Monoid tests :: IO [Test] tests = return @@ -51,9 +49,11 @@ (findAll ((A.takeWhile (=='𝅘𝅥𝅯') :: Parser T.Text))) ("𝄞𝅘𝅥𝅘𝅥𝅘𝅥𝅘𝅥𝅘𝅥𝅯𝅘𝅥𝅯") ("𝅘𝅥𝅯𝅘𝅥𝅯𝅘𝅥𝅘𝅥𝅘𝅥𝅘𝅥" :: T.Text) [Left "𝄞𝅘𝅥𝅘𝅥𝅘𝅥𝅘𝅥", Right "𝅘𝅥𝅯𝅘𝅥𝅯𝅘𝅥𝅯𝅘𝅥𝅯", Left "𝅘𝅥𝅘𝅥𝅘𝅥𝅘𝅥"] + , Test $ runParserTest "empty input" (sepCap (fail "" :: Parser ())) "" [] , Test $ streamEditTest "x to o" (string "x") (const "o") "x x x" "o o o" , Test $ streamEditTest "x to o inner" (string "x") (const "o") " x x x " " o o o " , Test $ streamEditTest "ordering" (string "456") (const "ABC") "123456789" "123ABC789" + , Test $ streamEditTest "empty input" (match (fail "")) (fst) "" "" ] where runParserTest nam p input expected = TestInstance