megaparsec 5.2.0 → 5.3.0
raw patch · 19 files changed
+356/−163 lines, 19 filesdep ~basedep ~criteriondep ~megaparsec
Dependency ranges changed: base, criterion, megaparsec
Files
- AUTHORS.md +1/−0
- CHANGELOG.md +17/−0
- README.md +62/−11
- Text/Megaparsec.hs +6/−3
- Text/Megaparsec/ByteString.hs +3/−2
- Text/Megaparsec/ByteString/Lazy.hs +3/−2
- Text/Megaparsec/Char.hs +40/−42
- Text/Megaparsec/Combinator.hs +27/−31
- Text/Megaparsec/Error.hs +7/−3
- Text/Megaparsec/Expr.hs +2/−2
- Text/Megaparsec/Lexer.hs +4/−4
- Text/Megaparsec/Perm.hs +14/−14
- Text/Megaparsec/Prim.hs +84/−19
- Text/Megaparsec/String.hs +2/−2
- Text/Megaparsec/Text.hs +2/−2
- Text/Megaparsec/Text/Lazy.hs +3/−2
- megaparsec.cabal +9/−12
- tests/Spec.hs +0/−10
- tests/Text/Megaparsec/PrimSpec.hs +70/−2
AUTHORS.md view
@@ -50,3 +50,4 @@ * Slava Shklyaev * Tal Walter * Tomáš Janoušek+* Vladislav Zavialov
CHANGELOG.md view
@@ -1,3 +1,20 @@+## Megaparsec 5.3.0++* Added the `match` combinator that allows to get collection of consumed+ tokens along with result of parsing.++* Added the `region` combinator which allows to process parse errors+ happening when its argument parser is run.++* Added the `getNextTokenPosition`, which returns position where the next+ token in the stream begins.++* Defined `Semigroup` and `Monoid` instances of `ParsecT`.++* Dropped support for GHC 7.6.++* Added an `ErrorComponent` instance for `()`.+ ## Megaparsec 5.2.0 * Added `MonadParsec` instance for `RWST`.
README.md view
@@ -19,9 +19,11 @@ * [Tutorials](#tutorials) * [Performance](#performance) * [Comparison with other solutions](#comparison-with-other-solutions)- * [Megaparsec and Attoparsec](#megaparsec-and-attoparsec)- * [Megaparsec and Parsec](#megaparsec-and-parsec)- * [Megaparsec and Parsers](#megaparsec-and-parsers)+ * [Megaparsec vs Attoparsec](#megaparsec-vs-attoparsec)+ * [Megaparsec vs Parsec](#megaparsec-vs-parsec)+ * [Megaparsec vs Trifecta](#megaparsec-vs-trifecta)+ * [Megaparsec vs Earley](#megaparsec-vs-earley)+ * [Megaparsec vs Parsers](#megaparsec-vs-parsers) * [Related packages](#related-packages) * [Links to announcements](#links-to-announcements) * [Authors](#authors)@@ -230,7 +232,7 @@ There are quite a few libraries that can be used for parsing in Haskell, let's compare Megaparsec with some of them. -### Megaparsec and Attoparsec+### Megaparsec vs Attoparsec [Attoparsec](https://github.com/bos/attoparsec) is another prominent Haskell library for parsing. Although the both libraries deal with parsing, it's@@ -248,7 +250,7 @@ usually not huge, just go with Megaparsec, otherwise Attoparsec may be a better choice. -### Megaparsec and Parsec+### Megaparsec vs Parsec Since Megaparsec is a fork of Parsec, it's necessary to list main differences between the two libraries:@@ -284,6 +286,8 @@ * Megaparsec is faster. +* Megaparsec is better supported.+ If you want to see a detailed change log, `CHANGELOG.md` may be helpful. Also see [this original announcement](https://notehub.org/w7037) for another comparison.@@ -297,17 +301,64 @@ project. If you think you still have a reason to use original Parsec, open an issue. -### Megaparsec and Parsers+### Megaparsec vs Trifecta +[Trifecta](https://hackage.haskell.org/package/trifecta) is another Haskell+library featuring good error messages. Like some other projects of Edward+Kmett, it's probably good, but also poorly documented, arcane, and has+unfixed [bugs and flaws](https://github.com/ekmett/trifecta/issues) that+Edward is too busy to fix (simply a fact, no offense intended). Other+reasons one may question choice of Trifecta is his/her parsing library:++* Complicated, doesn't have any tutorials available, and documentation+ doesn't help at all.++* Trifecta can parse `String` and `ByteString` natively, but not `Text`.++* Trifecta's error messages may be different with their own features, but+ certainly not as flexible as Megaparsec's error messages in the latest+ versions.++* Depends on `lens`. This means you'll pull in half of Hackage as transitive+ dependencies. Also if you're not into `lens` and would like to keep your+ code “vanilla”, you may not like the API.++### Megaparsec vs Earley++[Earley](https://hackage.haskell.org/package/Earley) is a newer library that+allows to safely (it your code compiles, then it probably works) parse+context-free grammars (CFG). Megaparsec is a lower-level library compared to+Earley, but there are still enough reasons to choose it over Earley:++* Megaparsec is faster.++* Your grammar may be not context free or you may want introduce some sort+ of state to the parsing process. Almost all non-trivial parsers require+ something of this sort. Even if your grammar is context-free, state may+ allow to add some additional niceties. Earley does not support that.++* Megaparsec's error messages are more flexible allowing to include+ arbitrary data in them, return multiple error messages, mark regions that+ affect any error that happens in those regions, etc.++* The approach Earley uses differs from conventional monadic parsing. If you+ work not alone, chances people you work with, especially beginners will be+ much more productive with libraries taking more traditional path to+ parsing like Megaparsec.++IOW, Megaparsec is less safe but also much more powerful.++### Megaparsec vs Parsers+ There is [Parsers](https://hackage.haskell.org/package/parsers) package, which is great. You can use it with Megaparsec or Parsec, but consider the following: -* It depends on *both* Attoparsec and Parsec, which means you always grab- useless code installing it. This is ridiculous, by the way, because this- package is supposed to be useful for parser builders, so they can write- basic core functionality and get the rest “for free”. But with these- useful functions you get two more parsers as dependencies.+* It depends on Attoparsec, Parsec, and Trifecta, which means you always+ grab half of Hackage as transitive dependencies by using it. This is+ ridiculous, by the way, because this package is supposed to be useful for+ parser builders, so they can write basic core functionality and get the+ rest “for free”. * It currently has a bug in definition of `lookAhead` for various monad transformers like `StateT`, etc. which is visible when you create
Text/Megaparsec.hs view
@@ -32,15 +32,15 @@ -- > -- import Text.Megaparsec.Text -- > -- import Text.Megaparsec.Text.Lazy ----- As you can see the second import depends on data type you want to use as--- input stream. It just defines the useful type-synonym @Parser@.+-- As you can see the second import depends on the data type you want to use+-- as input stream. It just defines the useful type-synonym @Parser@. -- -- Megaparsec 5 uses some type-level machinery to provide flexibility -- without compromising on type safety. Thus type signatures are sometimes -- necessary to avoid ambiguous types. If you're seeing a error message that -- reads like “Ambiguous type variable @e0@ arising from … prevents the -- constraint @(ErrorComponent e0)@ from being resolved”, you need to give--- an explicit signature to your parser to eliminate the ambiguity. It's a+-- an explicit signature to your parser to resolve the ambiguity. It's a -- good idea to provide type signatures for all top-level definitions. -- -- Megaparsec is capable of a lot. Apart from this standard functionality@@ -70,6 +70,8 @@ , A.optional -- $optional , unexpected+ , match+ , region , failure , (<?>) , label@@ -157,6 +159,7 @@ , getInput , setInput , getPosition+ , getNextTokenPosition , setPosition , pushPosition , popPosition
Text/Megaparsec/ByteString.hs view
@@ -16,7 +16,8 @@ import Text.Megaparsec.Prim -- | Modules corresponding to various types of streams define 'Parser'--- accordingly, so user can use it to easily change type of input stream by--- importing different “type modules”. This one is for strict byte-strings.+-- accordingly, so the user can use it to easily change type of input stream+-- by importing different “type modules”. This one is for strict+-- 'ByteString's. type Parser = Parsec Dec ByteString
Text/Megaparsec/ByteString/Lazy.hs view
@@ -16,7 +16,8 @@ import Text.Megaparsec.Prim -- | Modules corresponding to various types of streams define 'Parser'--- accordingly, so user can use it to easily change type of input stream by--- importing different “type modules”. This one is for lazy byte-strings.+-- accordingly, so the user can use it to easily change the type of input+-- stream by importing different “type modules”. This one is for lazy+-- 'ByteString's. type Parser = Parsec Dec ByteString
Text/Megaparsec/Char.hs view
@@ -75,21 +75,21 @@ ---------------------------------------------------------------------------- -- Simple parsers --- | Parses a newline character.+-- | Parse a newline character. newline :: (MonadParsec e s m, Token s ~ Char) => m Char newline = char '\n' {-# INLINE newline #-} --- | Parses a carriage return character followed by a newline character.--- Returns sequence of characters parsed.+-- | Parse a carriage return character followed by a newline character.+-- Return sequence of characters parsed. crlf :: (MonadParsec e s m, Token s ~ Char) => m String crlf = string "\r\n" {-# INLINE crlf #-} --- | Parses a CRLF (see 'crlf') or LF (see 'newline') end of line. Returns--- the sequence of characters parsed.+-- | Parse a CRLF (see 'crlf') or LF (see 'newline') end of line. Return the+-- sequence of characters parsed. -- -- > eol = (pure <$> newline) <|> crlf @@ -97,13 +97,13 @@ eol = (pure <$> newline) <|> crlf <?> "end of line" {-# INLINE eol #-} --- | Parses a tab character.+-- | Parse a tab character. tab :: (MonadParsec e s m, Token s ~ Char) => m Char tab = char '\t' {-# INLINE tab #-} --- | Skips /zero/ or more white space characters.+-- | Skip /zero/ or more white space characters. -- -- See also: 'skipMany' and 'spaceChar'. @@ -114,21 +114,21 @@ ---------------------------------------------------------------------------- -- Categories of characters --- | Parses control characters, which are the non-printing characters of the--- Latin-1 subset of Unicode.+-- | Parse a control character (a non-printing character of the Latin-1+-- subset of Unicode). controlChar :: (MonadParsec e s m, Token s ~ Char) => m Char controlChar = satisfy isControl <?> "control character" {-# INLINE controlChar #-} --- | Parses a Unicode space character, and the control characters: tab,+-- | Parse a Unicode space character, and the control characters: tab, -- newline, carriage return, form feed, and vertical tab. spaceChar :: (MonadParsec e s m, Token s ~ Char) => m Char spaceChar = satisfy isSpace <?> "white space" {-# INLINE spaceChar #-} --- | Parses an upper-case or title-case alphabetic Unicode character. Title+-- | Parse an upper-case or title-case alphabetic Unicode character. Title -- case is used by a small number of letter ligatures like the -- single-character form of Lj. @@ -136,23 +136,22 @@ upperChar = satisfy isUpper <?> "uppercase letter" {-# INLINE upperChar #-} --- | Parses a lower-case alphabetic Unicode character.+-- | Parse a lower-case alphabetic Unicode character. lowerChar :: (MonadParsec e s m, Token s ~ Char) => m Char lowerChar = satisfy isLower <?> "lowercase letter" {-# INLINE lowerChar #-} --- | Parses alphabetic Unicode characters: lower-case, upper-case and--- title-case letters, plus letters of case-less scripts and modifiers--- letters.+-- | Parse an alphabetic Unicode character: lower-case, upper-case, or+-- title-case letter, or a letter of case-less scripts\/modifier letter. letterChar :: (MonadParsec e s m, Token s ~ Char) => m Char letterChar = satisfy isLetter <?> "letter" {-# INLINE letterChar #-} --- | Parses alphabetic or numeric digit Unicode characters.+-- | Parse an alphabetic or numeric digit Unicode characters. ----- Note that numeric digits outside the ASCII range are parsed by this+-- Note that the numeric digits outside the ASCII range are parsed by this -- parser but not by 'digitChar'. Such digits may be part of identifiers but -- are not used by the printer and reader to represent numbers. @@ -160,88 +159,88 @@ alphaNumChar = satisfy isAlphaNum <?> "alphanumeric character" {-# INLINE alphaNumChar #-} --- | Parses printable Unicode characters: letters, numbers, marks,--- punctuation, symbols and spaces.+-- | Parse a printable Unicode character: letter, number, mark, punctuation,+-- symbol or space. printChar :: (MonadParsec e s m, Token s ~ Char) => m Char printChar = satisfy isPrint <?> "printable character" {-# INLINE printChar #-} --- | Parses an ASCII digit, i.e between “0” and “9”.+-- | Parse an ASCII digit, i.e between “0” and “9”. digitChar :: (MonadParsec e s m, Token s ~ Char) => m Char digitChar = satisfy isDigit <?> "digit" {-# INLINE digitChar #-} --- | Parses an octal digit, i.e. between “0” and “7”.+-- | Parse an octal digit, i.e. between “0” and “7”. octDigitChar :: (MonadParsec e s m, Token s ~ Char) => m Char octDigitChar = satisfy isOctDigit <?> "octal digit" {-# INLINE octDigitChar #-} --- | Parses a hexadecimal digit, i.e. between “0” and “9”, or “a” and “f”,--- or “A” and “F”.+-- | Parse a hexadecimal digit, i.e. between “0” and “9”, or “a” and “f”, or+-- “A” and “F”. hexDigitChar :: (MonadParsec e s m, Token s ~ Char) => m Char hexDigitChar = satisfy isHexDigit <?> "hexadecimal digit" {-# INLINE hexDigitChar #-} --- | Parses Unicode mark characters, for example accents and the like, which--- combine with preceding characters.+-- | Parse a Unicode mark character (accents and the like), which combines+-- with preceding characters. markChar :: (MonadParsec e s m, Token s ~ Char) => m Char markChar = satisfy isMark <?> "mark character" {-# INLINE markChar #-} --- | Parses Unicode numeric characters, including digits from various--- scripts, Roman numerals, et cetera.+-- | Parse a Unicode numeric character, including digits from various+-- scripts, Roman numerals, etc. numberChar :: (MonadParsec e s m, Token s ~ Char) => m Char numberChar = satisfy isNumber <?> "numeric character" {-# INLINE numberChar #-} --- | Parses Unicode punctuation characters, including various kinds of+-- | Parse a Unicode punctuation character, including various kinds of -- connectors, brackets and quotes. punctuationChar :: (MonadParsec e s m, Token s ~ Char) => m Char punctuationChar = satisfy isPunctuation <?> "punctuation" {-# INLINE punctuationChar #-} --- | Parses Unicode symbol characters, including mathematical and currency+-- | Parse a Unicode symbol characters, including mathematical and currency -- symbols. symbolChar :: (MonadParsec e s m, Token s ~ Char) => m Char symbolChar = satisfy isSymbol <?> "symbol" {-# INLINE symbolChar #-} --- | Parses Unicode space and separator characters.+-- | Parse a Unicode space and separator characters. separatorChar :: (MonadParsec e s m, Token s ~ Char) => m Char separatorChar = satisfy isSeparator <?> "separator" {-# INLINE separatorChar #-} --- | Parses a character from the first 128 characters of the Unicode character set,--- corresponding to the ASCII character set.+-- | Parse a character from the first 128 characters of the Unicode+-- character set, corresponding to the ASCII character set. asciiChar :: (MonadParsec e s m, Token s ~ Char) => m Char asciiChar = satisfy isAscii <?> "ASCII character" {-# INLINE asciiChar #-} --- | Parses a character from the first 256 characters of the Unicode+-- | Parse a character from the first 256 characters of the Unicode -- character set, corresponding to the ISO 8859-1 (Latin-1) character set. latin1Char :: (MonadParsec e s m, Token s ~ Char) => m Char latin1Char = satisfy isLatin1 <?> "Latin-1 character" {-# INLINE latin1Char #-} --- | @charCategory cat@ Parses character in Unicode General Category @cat@,+-- | @charCategory cat@ parses character in Unicode General Category @cat@, -- see 'Data.Char.GeneralCategory'. charCategory :: (MonadParsec e s m, Token s ~ Char) => GeneralCategory -> m Char charCategory cat = satisfy ((== cat) . generalCategory) <?> categoryName cat {-# INLINE charCategory #-} --- | Returns human-readable name of Unicode General Category.+-- | Return human-readable name of Unicode General Category. categoryName :: GeneralCategory -> String categoryName cat =@@ -320,10 +319,9 @@ {-# INLINE anyChar #-} -- | @oneOf cs@ succeeds if the current character is in the supplied--- list of characters @cs@. Returns the parsed character. Note that this--- parser doesn't automatically generate “expected” component of error--- message, so usually you should label it manually with 'label' or--- ('<?>').+-- collection of characters @cs@. Returns the parsed character. Note that+-- this parser doesn't automatically generate “expected” component of error+-- message, so usually you should label it manually with 'label' or ('<?>'). -- -- See also: 'satisfy'. --@@ -342,9 +340,9 @@ oneOf' cs = satisfy (`elemi` cs) {-# INLINE oneOf' #-} --- | As the dual of 'oneOf', @noneOf cs@ succeeds if the current--- character /not/ in the supplied list of characters @cs@. Returns the--- parsed character.+-- | As the dual of 'oneOf', @noneOf cs@ succeeds if the current character+-- /not/ in the supplied list of characters @cs@. Returns the parsed+-- character. noneOf :: (Foldable f, MonadParsec e s m, Token s ~ Char) => f Char -> m Char noneOf cs = satisfy (`notElem` cs)
Text/Megaparsec/Combinator.hs view
@@ -9,8 +9,8 @@ -- Stability : experimental -- Portability : portable ----- Commonly used generic combinators. Note that all combinators works with--- any 'Alternative' instances.+-- Commonly used generic combinators. Note that all the combinators work+-- with any 'Alternative' instance. {-# LANGUAGE BangPatterns #-} {-# LANGUAGE CPP #-}@@ -52,16 +52,15 @@ between open close p = open *> p <* close {-# INLINE between #-} --- | @choice ps@ tries to apply the parsers in the list @ps@ in order,--- until one of them succeeds. Returns the value of the succeeding parser.+-- | @choice ps@ tries to apply the parsers in the list @ps@ in order, until+-- one of them succeeds. Returns the value of the succeeding parser. choice :: (Foldable f, Alternative m) => f (m a) -> m a choice = asum {-# INLINE choice #-} --- | @count n p@ parses @n@ occurrences of @p@. If @n@ is smaller or--- equal to zero, the parser equals to @return []@. Returns a list of @n@--- values.+-- | @count n p@ parses @n@ occurrences of @p@. If @n@ is smaller or equal+-- to zero, the parser equals to @return []@. Returns a list of @n@ values. count :: Applicative m => Int -> m a -> m [a] count n p = sequenceA (replicate n p)@@ -93,8 +92,8 @@ eitherP a b = (Left <$> a) <|> (Right <$> b) {-# INLINE eitherP #-} --- | @endBy p sep@ parses /zero/ or more occurrences of @p@, separated--- and ended by @sep@. Returns a list of values returned by @p@.+-- | @endBy p sep@ parses /zero/ or more occurrences of @p@, separated and+-- ended by @sep@. Returns a list of values returned by @p@. -- -- > cStatements = cStatement `endBy` semicolon @@ -102,16 +101,16 @@ endBy p sep = many (p <* sep) {-# INLINE endBy #-} --- | @endBy1 p sep@ parses /one/ or more occurrences of @p@, separated--- and ended by @sep@. Returns a list of values returned by @p@.+-- | @endBy1 p sep@ parses /one/ or more occurrences of @p@, separated and+-- ended by @sep@. Returns a list of values returned by @p@. endBy1 :: Alternative m => m a -> m sep -> m [a] endBy1 p sep = some (p <* sep) {-# INLINE endBy1 #-} --- | @manyTill p end@ applies parser @p@ /zero/ or more times until--- parser @end@ succeeds. Returns the list of values returned by @p@. This--- parser can be used to scan comments:+-- | @manyTill p end@ applies parser @p@ /zero/ or more times until parser+-- @end@ succeeds. Returns the list of values returned by @p@. This parser+-- can be used to scan comments: -- -- > simpleComment = string "<!--" >> manyTill anyChar (string "-->") @@ -126,9 +125,8 @@ someTill p end = (:) <$> p <*> manyTill p end {-# INLINE someTill #-} --- | @option x p@ tries to apply parser @p@. If @p@ fails without--- consuming input, it returns the value @x@, otherwise the value returned--- by @p@.+-- | @option x p@ tries to apply parser @p@. If @p@ fails without consuming+-- input, it returns the value @x@, otherwise the value returned by @p@. -- -- > priority = option 0 (digitToInt <$> digitChar) @@ -136,8 +134,8 @@ option x p = p <|> pure x {-# INLINE option #-} --- | @sepBy p sep@ parses /zero/ or more occurrences of @p@, separated--- by @sep@. Returns a list of values returned by @p@.+-- | @sepBy p sep@ parses /zero/ or more occurrences of @p@, separated by+-- @sep@. Returns a list of values returned by @p@. -- -- > commaSep p = p `sepBy` comma @@ -145,30 +143,28 @@ sepBy p sep = sepBy1 p sep <|> pure [] {-# INLINE sepBy #-} --- | @sepBy1 p sep@ parses /one/ or more occurrences of @p@, separated--- by @sep@. Returns a list of values returned by @p@.+-- | @sepBy1 p sep@ parses /one/ or more occurrences of @p@, separated by+-- @sep@. Returns a list of values returned by @p@. sepBy1 :: Alternative m => m a -> m sep -> m [a] sepBy1 p sep = (:) <$> p <*> many (sep *> p) {-# INLINE sepBy1 #-} --- | @sepEndBy p sep@ parses /zero/ or more occurrences of @p@,--- separated and optionally ended by @sep@. Returns a list of values--- returned by @p@.+-- | @sepEndBy p sep@ parses /zero/ or more occurrences of @p@, separated+-- and optionally ended by @sep@. Returns a list of values returned by @p@. sepEndBy :: Alternative m => m a -> m sep -> m [a] sepEndBy p sep = sepEndBy1 p sep <|> pure [] {-# INLINE sepEndBy #-} --- | @sepEndBy1 p sep@ parses /one/ or more occurrences of @p@,--- separated and optionally ended by @sep@. Returns a list of values--- returned by @p@.+-- | @sepEndBy1 p sep@ parses /one/ or more occurrences of @p@, separated+-- and optionally ended by @sep@. Returns a list of values returned by @p@. sepEndBy1 :: Alternative m => m a -> m sep -> m [a] sepEndBy1 p sep = (:) <$> p <*> ((sep *> sepEndBy p sep) <|> pure []) --- | @skipMany p@ applies the parser @p@ /zero/ or more times, skipping--- its result.+-- | @skipMany p@ applies the parser @p@ /zero/ or more times, skipping its+-- result. -- -- > space = skipMany spaceChar @@ -176,8 +172,8 @@ skipMany p = void $ many p {-# INLINE skipMany #-} --- | @skipSome p@ applies the parser @p@ /one/ or more times, skipping--- its result.+-- | @skipSome p@ applies the parser @p@ /one/ or more times, skipping its+-- result. skipSome :: Alternative m => m a -> m () skipSome p = void $ some p
Text/Megaparsec/Error.hs view
@@ -84,7 +84,7 @@ class Ord e => ErrorComponent e where - -- | Represent message passed to 'fail' in parser monad.+ -- | Represent the message passed to 'fail' in parser monad. -- -- @since 5.0.0 @@ -100,7 +100,11 @@ -> Pos -- ^ Actual indentation level -> e --- | “Default error component”. This in our instance of 'ErrorComponent'+instance ErrorComponent () where+ representFail _ = ()+ representIndentation _ _ _ = ()++-- | “Default error component”. This is our instance of 'ErrorComponent' -- provided out-of-box. -- -- @since 5.0.0@@ -133,7 +137,7 @@ -- as set of custom associated data. The data type is parametrized over -- token type @t@ and custom data @e@. ----- Note that stack of source positions contains current position as its+-- Note that the stack of source positions contains current position as its -- head, and the rest of positions allows to track full sequence of include -- files with topmost source file at the end of the list. --
Text/Megaparsec/Expr.hs view
@@ -22,8 +22,8 @@ import Text.Megaparsec.Combinator import Text.Megaparsec.Prim --- | This data type specifies operators that work on values of type @a@.--- An operator is either binary infix or unary prefix or postfix. A binary+-- | This data type specifies operators that work on values of type @a@. An+-- operator is either binary infix or unary prefix or postfix. A binary -- operator has also an associated associativity. data Operator m a
Text/Megaparsec/Lexer.hs view
@@ -353,10 +353,10 @@ ---------------------------------------------------------------------------- -- Character and string literals --- | The lexeme parser parses a single literal character without--- quotes. Purpose of this parser is to help with parsing of conventional--- escape sequences. It's your responsibility to take care of character--- literal syntax in your language (by surrounding it with single quotes or+-- | The lexeme parser parses a single literal character without quotes.+-- Purpose of this parser is to help with parsing of conventional escape+-- sequences. It's your responsibility to take care of character literal+-- syntax in your language (by surrounding it with single quotes or -- similar). -- -- The literal character is parsed according to the grammar rules defined in
Text/Megaparsec/Perm.hs view
@@ -36,8 +36,8 @@ infixl 1 <||>, <|?> infixl 2 <$$>, <$?> --- | The type @PermParser s m a@ denotes a permutation parser that,--- when converted by the 'makePermParser' function, produces instance of+-- | The type @PermParser s m a@ denotes a permutation parser that, when+-- converted by the 'makePermParser' function, produces instance of -- 'MonadParsec' @m@ that parses @s@ stream and returns a value of type @a@ -- on success. --@@ -49,10 +49,10 @@ data Branch s m a = forall b. Branch (PermParser s m (b -> a)) (m b) --- | The parser @makePermParser perm@ parses a permutation of parser described--- by @perm@. For example, suppose we want to parse a permutation of: an--- optional string of @a@'s, the character @b@ and an optional @c@. This can--- be described by:+-- | The parser @makePermParser perm@ parses a permutation of parser+-- described by @perm@. For example, suppose we want to parse a permutation+-- of: an optional string of @a@'s, the character @b@ and an optional @c@.+-- This can be described by: -- -- > test = makePermParser $ -- > (,,) <$?> ("", some (char 'a'))@@ -99,10 +99,10 @@ -> PermParser s m b -- ^ Permutation parser f <$?> xp = newperm f <|?> xp --- | The expression @perm \<||> p@ adds parser @p@ to the permutation--- parser @perm@. The parser @p@ is not allowed to accept empty input — use--- the optional combinator ('<|?>') instead. Returns a new permutation--- parser that includes @p@.+-- | The expression @perm \<||> p@ adds parser @p@ to the permutation parser+-- @perm@. The parser @p@ is not allowed to accept empty input — use the+-- optional combinator ('<|?>') instead. Returns a new permutation parser+-- that includes @p@. (<||>) :: MonadParsec e s m => PermParser s m (a -> b) -- ^ Given permutation parser@@ -110,10 +110,10 @@ -> PermParser s m b -- ^ Resulting parser (<||>) = add --- | The expression @perm \<||> (x, p)@ adds parser @p@ to the--- permutation parser @perm@. The parser @p@ is optional — if it cannot be--- applied, the default value @x@ will be used instead. Returns a new--- permutation parser that includes the optional parser @p@.+-- | The expression @perm \<||> (x, p)@ adds parser @p@ to the permutation+-- parser @perm@. The parser @p@ is optional — if it cannot be applied, the+-- default value @x@ will be used instead. Returns a new permutation parser+-- that includes the optional parser @p@. (<|?>) :: MonadParsec e s m => PermParser s m (a -> b) -- ^ Given permutation parser
Text/Megaparsec/Prim.hs view
@@ -21,6 +21,7 @@ {-# LANGUAGE GeneralizedNewtypeDeriving #-} {-# LANGUAGE MultiParamTypeClasses #-} {-# LANGUAGE RankNTypes #-}+{-# LANGUAGE RecordWildCards #-} {-# LANGUAGE ScopedTypeVariables #-} {-# LANGUAGE TupleSections #-} {-# LANGUAGE TypeFamilies #-}@@ -37,10 +38,13 @@ , MonadParsec (..) , (<?>) , unexpected+ , match+ , region -- * Parser state combinators , getInput , setInput , getPosition+ , getNextTokenPosition , setPosition , pushPosition , popPosition@@ -110,7 +114,7 @@ ---------------------------------------------------------------------------- -- Data types --- | This is Megaparsec's state, it's parametrized over stream type @s@.+-- | This is the Megaparsec's state, it's parametrized over stream type @s@. data State s = State { stateInput :: s@@ -147,8 +151,7 @@ data Reply e s a = Reply (State s) Consumption (Result (Token s) e a) --- | This data structure represents an aspect of result of parser's--- work.+-- | This data structure represents an aspect of result of parser's work. -- -- See also: 'Result', 'Reply'. @@ -156,8 +159,7 @@ = Consumed -- ^ Some part of input stream was consumed | Virgin -- ^ No input was consumed --- | This data structure represents an aspect of result of parser's--- work.+-- | This data structure represents an aspect of result of parser's work. -- -- See also: 'Consumption', 'Reply'. @@ -224,8 +226,8 @@ accHints hs1 c x s hs2 = c x s (hs1 <> hs2) {-# INLINE accHints #-} --- | Replace most recent group of hints (if any) with given 'ErrorItem' (or--- delete it if 'Nothing' is given). This is used in 'label' primitive.+-- | Replace the most recent group of hints (if any) with given 'ErrorItem'+-- (or delete it if 'Nothing' is given). This is used in 'label' primitive. refreshLastHint :: Hints t -> Maybe (ErrorItem t) -> Hints t refreshLastHint (Hints []) _ = Hints []@@ -252,7 +254,11 @@ -- | Update position in stream given tab width, current position, and -- current token. The result is a tuple where the first element will be -- used to report parse errors for current token, while the second element- -- is the incremented position that will be stored in parser's state.+ -- is the incremented position that will be stored in parser's state. The+ -- stored (incremented) position is used whenever position can't+ -- be\/shouldn't be updated by consuming a token. For example, when using+ -- 'failure', we don't grab a new token (we need to fail right were we are+ -- now), so error position will be taken from parser's state. -- -- When you work with streams where elements do not contain information -- about their position in input, result is usually consists of the third@@ -342,8 +348,8 @@ -- You call specific continuation when you want to proceed in that specific -- branch of control flow. --- | @Parsec@ is non-transformer variant of more general 'ParsecT'--- monad transformer.+-- | @Parsec@ is non-transformer variant of more general 'ParsecT' monad+-- transformer. type Parsec e s = ParsecT e s Identity @@ -359,6 +365,18 @@ -> (ParseError (Token s) e -> State s -> m b) -- empty-error -> m b } +instance (ErrorComponent e, Stream s, Semigroup a)+ => Semigroup (ParsecT e s m a) where+ (<>) = A.liftA2 (<>)+ {-# INLINE (<>) #-}++instance (ErrorComponent e, Stream s, Monoid a)+ => Monoid (ParsecT e s m a) where+ mempty = pure mempty+ {-# INLINE mempty #-}+ mappend = A.liftA2 mappend+ {-# INLINE mappend #-}+ instance Functor (ParsecT e s m) where fmap = pMap @@ -555,7 +573,7 @@ -- -- What happens here? First parser consumes “le” and fails (because it -- doesn't see a “t”). The second parser, however, isn't tried, since the- -- first parser has already consumed some input! @try@ fixes this behavior+ -- first parser has already consumed some input! 'try' fixes this behavior -- and allows backtracking to work: -- -- >>> parseTest (try (string "let") <|> string "lexical") "lexical"@@ -569,10 +587,11 @@ -- unexpected "le" -- expecting "let" or "lexical" --- -- Please note that as of Megaparsec 4.4.0, 'string' backtracks+ -- __Please note__ that as of Megaparsec 4.4.0, 'string' backtracks -- automatically (see 'tokens'), so it does not need 'try'. However, the -- examples above demonstrate the idea behind 'try' so well that it was- -- decided to keep them.+ -- decided to keep them. You still need to use 'try' when your+ -- alternatives are complex, composite parsers. try :: m a -> m a @@ -658,10 +677,10 @@ -- > string = tokens (==) -- -- Note that beginning from Megaparsec 4.4.0, this is an auto-backtracking- -- primitive, which means that if it fails, it never consumes any- -- input. This is done to make its consumption model match how error- -- messages for this primitive are reported (which becomes an important- -- thing as user gets more control with primitives like 'withRecovery'):+ -- primitive, which means that if it fails, it never consumes any input.+ -- This is done to make its consumption model match how error messages for+ -- this primitive are reported (which becomes an important thing as user+ -- gets more control with primitives like 'withRecovery'): -- -- >>> parseTest (string "abc") "abd" -- 1:1:@@ -714,7 +733,7 @@ pLabel :: String -> ParsecT e s m a -> ParsecT e s m a pLabel l p = ParsecT $ \s cok cerr eok eerr -> let el = Label <$> NE.nonEmpty l- cl = Label . (NE.fromList "rest of " <>) <$> NE.nonEmpty l+ cl = Label . (NE.fromList "the rest of " <>) <$> NE.nonEmpty l cok' x s' hs = cok x s' (refreshLastHint hs cl) eok' x s' hs = eok x s' (refreshLastHint hs el) eerr' err = eerr err@@ -866,7 +885,7 @@ pUpdateParserState f = ParsecT $ \s _ _ eok _ -> eok () (f s) mempty {-# INLINE pUpdateParserState #-} --- | A synonym for 'label' in form of an operator.+-- | A synonym for 'label' in the form of an operator. infix 0 <?> @@ -880,6 +899,41 @@ unexpected item = failure (E.singleton item) E.empty E.empty {-# INLINE unexpected #-} +-- | Return both the result of a parse and the list of tokens that were+-- consumed during parsing. This relies on the change of the+-- 'stateTokensProcessed' value to evaluate how many tokens were consumed.+--+-- @since 5.3.0++match :: MonadParsec e s m => m a -> m ([Token s], a)+match p = do+ tp <- getTokensProcessed+ s <- getInput+ r <- p+ tp' <- getTokensProcessed+ return (streamTake (tp' - tp) s, r)++-- | Specify how to process 'ParseError's that happen inside of this+-- wrapper. As a side effect of current implementation changing 'errorPos'+-- with this combinator will also change the final 'statePos' in parser+-- state.+--+-- @since 5.3.0++region :: MonadParsec e s m+ => (ParseError (Token s) e -> ParseError (Token s) e)+ -- ^ How to process 'ParseError's+ -> m a -- ^ The “region” that processing applies to+ -> m a+region f m = do+ r <- observing m+ case r of+ Left err -> do+ let ParseError {..} = f err+ updateParserState $ \st -> st { statePos = errorPos }+ failure errorUnexpected errorExpected errorCustom+ Right x -> return x+ -- | Make a singleton non-empty list from a value. nes :: a -> NonEmpty a@@ -906,6 +960,17 @@ getPosition :: MonadParsec e s m => m SourcePos getPosition = NE.head . statePos <$> getParserState++-- | Get position where the next token in the stream begins. If the stream+-- is empty, return 'Nothing'.+--+-- @since 5.3.0++getNextTokenPosition :: forall e s m. MonadParsec e s m => m (Maybe SourcePos)+getNextTokenPosition = do+ State {..} <- getParserState+ let f = fst . updatePos (Proxy :: Proxy s) stateTabWidth (NE.head statePos)+ return (f . fst <$> uncons stateInput) -- | @setPosition pos@ sets the current source position to @pos@. --
Text/Megaparsec/String.hs view
@@ -15,7 +15,7 @@ import Text.Megaparsec.Prim -- | Modules corresponding to various types of streams define 'Parser'--- accordingly, so user can use it to easily change type of input stream by--- importing different “type modules”. This one is for strings.+-- accordingly, so the user can use it to easily change type of input stream+-- by importing different “type modules”. This one is for 'String's. type Parser = Parsec Dec String
Text/Megaparsec/Text.hs view
@@ -16,7 +16,7 @@ import Data.Text -- | Modules corresponding to various types of streams define 'Parser'--- accordingly, so user can use it to easily change type of input stream by--- importing different “type modules”. This one is for strict text.+-- accordingly, so the user can use it to easily change type of input stream+-- by importing different “type modules”. This one is for strict 'Text'. type Parser = Parsec Dec Text
Text/Megaparsec/Text/Lazy.hs view
@@ -16,7 +16,8 @@ import Text.Megaparsec.Prim -- | Modules corresponding to various types of streams define 'Parser'--- accordingly, so user can use it to easily change type of input stream by--- importing different “type modules”. This one is for lazy text.+-- accordingly, so the user can use it to easily change type of the input+-- stream by importing different “type modules”. This one is for lazy+-- 'Text'. type Parser = Parsec Dec Text
megaparsec.cabal view
@@ -27,8 +27,9 @@ -- POSSIBILITY OF SUCH DAMAGE. name: megaparsec-version: 5.2.0+version: 5.3.0 cabal-version: >= 1.10+tested-with: GHC==7.8.4, GHC==7.10.3, GHC==8.0.2 license: BSD2 license-file: LICENSE.md author: Megaparsec contributors,@@ -61,7 +62,7 @@ library build-depends: QuickCheck >= 2.7 && < 3.0- , base >= 4.6 && < 5.0+ , base >= 4.7 && < 5.0 , bytestring >= 0.2 && < 0.11 , containers >= 0.5 && < 0.6 , deepseq >= 1.3 && < 1.5@@ -95,10 +96,6 @@ , Text.Megaparsec.Text.Lazy if flag(dev) ghc-options: -Wall -Werror- if impl(ghc >= 8.0)- ghc-options: -Wcompat- ghc-options: -Wnoncanonical-monadfail-instances- ghc-options: -Wnoncanonical-monoid-instances else ghc-options: -O2 -Wall default-language: Haskell2010@@ -122,13 +119,13 @@ , Text.Megaparsec.PosSpec , Text.Megaparsec.PrimSpec build-depends: QuickCheck >= 2.7 && < 3.0- , base >= 4.6 && < 5.0+ , base >= 4.7 && < 5.0 , bytestring >= 0.2 && < 0.11 , containers >= 0.5 && < 0.6 , exceptions >= 0.6 && < 0.9 , hspec >= 2.0 && < 3.0 , hspec-expectations >= 0.5 && < 0.9- , megaparsec >= 5.2.0+ , megaparsec , mtl >= 2.0 && < 3.0 , scientific >= 0.3.1 && < 0.4 , text >= 0.2 && < 1.3@@ -147,10 +144,10 @@ main-is: Main.hs hs-source-dirs: bench-speed type: exitcode-stdio-1.0- build-depends: base >= 4.6 && < 5.0+ build-depends: base >= 4.7 && < 5.0 , criterion >= 0.6.2.1 && < 1.2 , deepseq >= 1.3 && < 1.5- , megaparsec >= 5.2.0+ , megaparsec if flag(dev) ghc-options: -O2 -Wall -Werror else@@ -161,9 +158,9 @@ main-is: Main.hs hs-source-dirs: bench-memory type: exitcode-stdio-1.0- build-depends: base >= 4.6 && < 5.0+ build-depends: base >= 4.7 && < 5.0 , deepseq >= 1.3 && < 1.5- , megaparsec >= 5.2.0+ , megaparsec , weigh >= 0.0.3 if flag(dev) ghc-options: -O2 -Wall -Werror
tests/Spec.hs view
@@ -1,11 +1,1 @@-{-# LANGUAGE CPP #-}--#if __GLASGOW_HASKELL__ >= 708 {-# OPTIONS_GHC -F -pgmF hspec-discover #-}-#else--module Spec (main) where--main :: IO ()-main = return ()-#endif
tests/Text/Megaparsec/PrimSpec.hs view
@@ -44,6 +44,7 @@ import Control.Monad.Reader import Data.Char (toUpper, chr) import Data.Foldable (asum, concat)+import Data.Function (on) import Data.List (isPrefixOf, foldl') import Data.List.NonEmpty (NonEmpty (..)) import Data.Maybe (fromMaybe, listToMaybe, isJust)@@ -70,6 +71,7 @@ import qualified Data.ByteString.Char8 as B import qualified Data.ByteString.Lazy.Char8 as BL import qualified Data.List.NonEmpty as NE+import qualified Data.Semigroup as G import qualified Data.Set as E import qualified Data.Text as T import qualified Data.Text.Lazy as TL@@ -198,6 +200,37 @@ ( st { statePos = apos } , Left (err apos $ utoks (take (il + 1) stateInput) <> etoks ts) ) + describe "getNextTokenPosition" $ do+ context "when input stream is empty" $+ it "returns Nothing" $+ property $ \st' -> do+ let p :: CustomParser (Maybe SourcePos)+ p = getNextTokenPosition+ st = (st' :: State [Span]) { stateInput = [] }+ runParser' p st `shouldBe` (st, Right Nothing)+ context "when input stream is not empty" $+ it "return the position of start of the next token" $+ property $ \st' h -> do+ let p :: CustomParser (Maybe SourcePos)+ p = getNextTokenPosition+ st = st' { stateInput = h : stateInput st' }+ runParser' p st `shouldBe` (st, (Right . Just . spanStart) h)++ describe "ParsecT Semigroup instance" $+ it "the associative operation works" $+ property $ \a b -> do+ let p = pure [a] G.<> pure [b]+ prs p "" `shouldParse` ([a,b] :: [Int])++ describe "ParsecT Monoid instance" $ do+ it "mempty works" $ do+ let p = mempty+ prs p "" `shouldParse` ([] :: [Int])+ it "mappend works" $+ property $ \a b -> do+ let p = pure [a] `mappend` pure [b]+ prs p "" `shouldParse` ([a,b] :: [Int])+ describe "ParsecT Functor instance" $ do it "obeys identity law" $ property $ \n ->@@ -518,6 +551,41 @@ , errorExpected = E.empty , errorCustom = E.empty }) + describe "match" $+ it "return consumed tokens along with the result" $+ property $ \str -> do+ let p = match (string str)+ prs p str `shouldParse` (str,str)+ prs' p str `succeedsLeaving` ""++ describe "region" $ do+ context "when inner parser succeeds" $+ it "has no effect" $+ property $ \st e n -> do+ let p :: Parser Int+ p = region (const e) (pure n)+ runParser' p st `shouldBe` (st, Right (n :: Int))+ context "when inner parser fails" $+ it "the given function is used on the parse error" $+ property $ \st e0 e1 -> do+ let p :: Parser Int+ p = region f $ failure+ (errorUnexpected e0)+ (errorExpected e0)+ (errorCustom e0)+ f x = ParseError+ { errorPos = ((G.<>) `on` errorPos) x e1+ , errorUnexpected = (E.union `on` errorUnexpected) x e1+ , errorExpected = (E.union `on` errorExpected) x e1+ , errorCustom = (E.union `on` errorCustom) x e1 }+ r = ParseError+ { errorPos = finalPos+ , errorUnexpected = (E.union `on` errorUnexpected) e0 e1+ , errorExpected = (E.union `on` errorExpected) e0 e1+ , errorCustom = (E.union `on` errorCustom) e0 e1 }+ finalPos = statePos st G.<> errorPos e1+ runParser' p st `shouldBe` (st { statePos = finalPos }, Left r)+ describe "failure" $ it "signals correct parse error" $ property $ \us ps xs -> do@@ -540,12 +608,12 @@ grs p s (`shouldFailWith` err (posN (1 :: Int) s) mempty) grs' p s (`failsLeaving` "") context "inner parser produces hints" $- it "replaces the last hint with “rest of <label>”" $+ it "replaces the last hint with “the rest of <label>”" $ property $ \lbl a -> not (null lbl) ==> do let p :: MonadParsec Dec String m => m String p = label lbl (many (char a)) <* empty s = [a]- grs p s (`shouldFailWith` err (posN (1 :: Int) s) (elabel $ "rest of " ++ lbl))+ grs p s (`shouldFailWith` err (posN (1 :: Int) s) (elabel $ "the rest of " ++ lbl)) grs' p s (`failsLeaving` "") context "when inner parser consumes and fails" $ it "reports parse error without modification" $