diff --git a/changelog.md b/changelog.md --- a/changelog.md +++ b/changelog.md @@ -1,5 +1,278 @@ # Revision history for skylighting and skylighting-core +## 0.15 + + * New syntaxes: sparql (#212), sas (#213), mermaid, desktop, + elixir-eex, elixir-heex, abnf, asciidoc, cabal, cobol, context, + csv, haml, idl, jinja, jira, k, logfile, meson, nginx, ninja, + ocamllex, ocamlyacc, q, quarto, rdoc, rmarkdown, rtf, textile, + todo, vue. + + * Update syntax definitions from upstream: bash, cmake, cpp, + crystal, dot, elixir, haskell, markdown, nix, ocaml, orgmode, + perl, php, powershell, python, qml, raku, rust, + spdx-comments, tcsh, typst, yaml, zig, zsh. + + * Allow multiple contexts separated by `!` (#208). + This is an upstream KDE change. + + * Bump min base version to 4.18. + + * Regex: don't hang on bounded repetition with min > max. + `atmost` recursed without a guard for negative counts, so a pattern + like `a{3,1}` made the compiler loop forever building an infinite + regex. Reject the quantifier and fall back to interpreting it as a + literal, consistent with how other invalid quantifiers (`a{}`, `a{3`) + are handled. + + * Regex: don't hang on lazy quantifiers in lookbehinds. + Previously lazy quantifier inside a lookbehind (e.g. `(?<=a+?b)`) hung. + + * Regex: don't hang on subroutine recursion that consumes no input. + Previously a pattern like `x|(?R)` recursed forever. + + * Regex: fix `lastCharOffset` for multibyte UTF-8 characters. + This bug resulted in failure of lookbehinds and word boundary + checks after non-ASCII characters. + + * Regex: don't let `(?i:...)` leak out of its group. + Previously, in `/(?i:a)b/` the trailing `b` was also matched + case-insensitively. + + * Regex: fix `[[:graph:]]`, `[[:word:]]`, and add `[[:digit:]]`. + + * Regex: find nested capturing groups for subroutine calls. + Previously, calls like `(?2)` in `/((a)b)(?2)/` were + silently ignored (matching the empty string). + + * Regex: apply case-insensitivity to classes, escapes, backreferences. + Case-insensitive matching was previously only applied to plain literal + characters, not character classes and escaped literals. Now: + + + `pRegexCharClass` takes the case-sensitivity flag and, when + insensitive, matches a character if any of its case variants is in + the class (negated classes exclude all case variants, as in PCRE). + + Escaped literals like \x61 match case-insensitively when + insensitivity is in effect. + + The matcher compares captured text character by character with + case folding (correct even when the case variants have different + UTF-8 encodings). + + [API change] `MatchCaptured` has a new argument place; + `MatchCaptured !Int !Bool`, where the Bool records the case + sensitivity in effect at its position. + + * Regex: linearize `{m,n}` expansion and cap repeat counts. + Previously `r{m,n}` was expanded to produce an AST quadratic in + n-m, which also made matching quadratic: matching `[ab]{0,800}` + against 800 characters took ~31 ms. We now use + a nested-optional encoding, `r(r(r)?)?`, which is linear; the same + match takes ~0.3 ms. In addition, repeat counts larger than + 65535 (the limit PCRE2 uses) are no longer treated as + quantifiers; like other invalid quantifier syntax, they fall + back to a literal interpretation. Previously `x{100000000}` + would eagerly build a hundred-million-node regex. + + * Regex: treat unmatched `]` outside a character class as a literal. + This the behavior of PCRE2 (used by KSyntaxHighlighting via + QRegularExpression). Previously we treated this as a parse error, + which caused failures in several current KDE syntax definitions. + + * Regex: parse `\0` octal escapes with up to two digits, as in PCRE. + `\0` previously required exactly three following octal digits. + + * Regex: support `\G` (assert position of match start). + Because our matcher is always anchored at the start of + the input it is given, this is exactly AssertBeginning. + + * Regex: support `\gN` and `\g{N}` backreferences. + + * Regex: support inline modifiers like `(?i)` without a colon. + + * Regex: support `\h` and `\H` (horizontal whitespace). + + * Regex: support `\A` (start of subject). + Since matching is always anchored at the start of the input we are + given, `\A` is equivalent to AssertBeginning. + + * Regex: allow an empty first alternative, as in `(?:|a)`. + As in PCRE, an empty alternative matches the empty string. Empty + alternatives other than the first were already supported. + + * Regex: support `\b` (backspace) inside character classes. + + * Regex: support atomic groups `(?>...)`. + + * Regex: leftmost-first (PCRE) match semantics instead of longest-match. + + + Alternation is now leftmost-first, and alternatives are tried in + source order (a fold in the compiler used to reverse them, which + was harmless under longest-match). + + Possessive quantifiers and atomic groups commit to the first match + in backtracking order, fixing the divergence from PCRE. + `(?>a|ab)c` now fails on "abc". + + Lookahead/lookbehind assertions are atomic and keep only the first + match's captures, so `d(?=(a|ab))` captures "a", as in PCRE. + - Lazy quantifiers are compiled to forms that prefer fewer + repetitions instead of being special-cased in the matcher; the old + special case could commit to a short prefix and miss valid longer + continuations. + - Repetition loops terminate via a seen-state set. + + * Regex: fix subroutine calls to groups with multi-digit numbers. + + * Regex: after `(?|...)`, resume group numbering after the max group. + + * Regex: allow a leading literal `]` in a character class to start a range. + As in PCRE, `[]-a]` is the range from `]` to `a`; previously it was parsed + as the three literals. + + * Regex: support `\pL`, `\P{...}`, `\PL`, and `\p{^...}`. + + * Regex: make Eq Match consistent with Ord. + + * Regex: Raise compile errors for stray quantifiers and unsupported flags. + + + a quantifier with nothing to repeat. + + a quantifier after an anchor or word-boundary assertion. + + `{m,n}` with m > n. + + repeat counts over 65535. + + inline flags we do not implement. + + * Regex: Support minimal= (inverted greediness) on RegExpr rules. + + + `compileRegex` takes a new Bool parameter for minimal matching + [API change], and `pSuffix` swaps the greedy and lazy variant + s of a quantifier when it is set. + + The inline flag `(?U)` (and `(?-U)`, `(?U:...))`, which toggles the + same option in PCRE, is now supported. + + RE has a new `reMinimal` field [API change], which + Skylighting.Parser sets from the `minimal` attribute on + RegExpr elements. + + * Support the char attribute on LineContinue. [API change] + The LineContinue constructor of Matcher now carries the character. + + * Fix case-sensitivity defaults to match KDE. + + + Rules (RegExpr, StringDetect, WordDetect, DetectChar, etc.) now + default to case-sensitive matching regardless of the language + element's `casesensitive` attribute. + + The default case sensitivity of keyword lists is given by the + `casesensitive` attribute on the language element, and may be + overridden by the `casesensitive` attribute on general > keywords. + + An `insensitive` attribute on a keyword rule itself now overrides + the keyword list's case sensitivity, but only when the attribute + is actually present. + + Behavior verified against KSyntaxHighlighting's definition.cpp and + highlightingdata.cpp. + + * Honor `additionalDeliminator`/`weakDeliminator` on rules. + + + Keyword rules fold rule-level `additionalDeliminator` and + `weakDeliminator` into their delimiter set, so e.g. Lua's + special variables rule (additionalDeliminator=".") matches + nil in nil.x even though the general keywords element makes + `.` a weak delimiter. + + Rule gains an `rAdditionalDeliminators` field [API change], + and both it and `rWeakDeliminators` now also incorporate + the general > keywords delimiter modifications, applied + before the rule-level ones as in KDE. Both sets are consulted + by WordDetect, Int, Float, HlCHex, and HlCOct: a weak delimiter + counts as a word character and an additional delimiter does + not, with weak delimiters taking precedence. + + WordDetect now uses KDE's boundary conditions: a delimiter (or + line edge) must precede the word or be its first character, and + must follow the word or be its last character. The leading + check also prevents WordDetect from matching in the middle of + a word. + + * Match KDE semantics for Int, Float, HlCOct, HlCHex rules [API change] + + + These rules match only if the *preceding* character is a word + delimiter (or the match is at the start of the line); nothing is + required of the following character. Previously we used a + two-sided `\w`-based word boundary check. + + None of these rules consume a leading sign. Previously Int, + HlCOct, HlCHex consumed an optional '-', and Float an optional + '+' or '-'. + + Int matches decimal digits only (no hex or octal forms). + + HlCOct matches C-style octals: '0' followed by octal digits. + Previously we required a "0o" prefix (and, due to a bug, actually + matched the hex form, so HlCOct never matched real octals). + + Float requires a '.', so "5e2" is not a Float; the exponent is + all-or-nothing (an incomplete exponent like "1.5e+" matches just + "1.5"); and there is no check on what follows the match, so + "1.2.3" matches "1.2". + + Since all four rules (and keyword and WordDetect) now need + only a single effective delimiter set, we replace the + `rWeakDeliminators` and `rAdditionalDeliminators` fields on + Rule with a single `rWordDelimiters` field [API change]. + + * Match KDE line-loop semantics. + + + Remove `cLineBeginContext` from Context [API change] and the + `lineBeginContext` attribute from the parser. KDE no longer + has this feature, no bundled syntax definition uses it. + + Column and `firstNonspaceColumn` now restart on every physical line. + A line continuation only suppresses the previous line's + `lineEndContext`; it no longer carries the previous line's column + state into the next line. + + `lineEmptyContext` defaults to `lineEndContext` when unspecified or + `#stay` (KDE context.cpp, see kde bug 405903). + + An empty line now applies the `lineEmptyContext` switches of + successive top contexts until `#stay`, and does not apply + `lineEndContext` separately. + + `checkLineEnd` likewise applies lineEndContext switches of successive + top contexts until `#stay`. + + Guard against endless loops from broken syntax definitions, as KDE + does: the empty-line and line-end context-switch loops and rule + matching without consuming input are limited to 1024 iterations + without progress, after which highlighting of the line is aborted. + + * Cache capturing groups in RE alongside the compiled regex. + Previously, matchRegex called extractCapturingGroups on every + invocation. Now the RE smart constructor extracts the + capturing groups once at compile time and caches them + alongside the compiled Regex. The tokenizer uses the cached groups + for static rules. For dynamic rules the groups are recomputed + after `subDynamic`, since the substitution rewrites the AST. + + + compileRE now returns Either String (Regex, IntMap Regex) [API change]. + + New function matchRegexWithGroups [API change] takes precomputed groups; + matchRegex keeps its old behavior for external callers. + + * Resolve keywords at load time, not on every tokenize call. + + + `tokenize` no longer resolves keywords; it assumes keyword lists in + the syntax map have already been resolved into word sets. This is + the case for the bundled syntax definitions (which are generated + via loadSyntaxesFromDir) and for definitions loaded with the + functions in Skylighting.Loader. + - `loadValidSyntaxesFromDir` now resolves keywords, fixing an + inconsistency with `loadSyntaxesFromDir` (which already did). + - The skylighting CLI resolves keywords in syntaxes added with + `--definition`. + - Users who build a SyntaxMap by hand from `parseSyntaxDefinition` + output must now apply resolveKeywords themselves; tokenizing with + an unresolved keyword list yields a clear "Keyword with unresolved + list" error. + + * Minor performance improvements in matchers and syntax lookup. + + + `detect2Chars`: compare decoded characters directly (like + detectChar) instead of allocating a two-character Text and + encoding it to a ByteString on every attempt. + + `wordDetect`, `stringDetect`: in the case-sensitive case, compare + bytes with a prefix check instead of walking and decoding a + prefix of the input into Text on every attempt. + + `syntaxByName`: don't rebuild the entire syntax map with lowercased + keys on every lookup; try an exact lookup first and fall back to + a linear scan. + + `syntaxByShortName`: lowercase the query once instead of once per + map entry. + ## 0.14.7 * Update xml syntax definitions: agda, apache, bash, c, clojure, diff --git a/skylighting-core.cabal b/skylighting-core.cabal --- a/skylighting-core.cabal +++ b/skylighting-core.cabal @@ -1,5 +1,5 @@ name: skylighting-core -version: 0.14.7 +version: 0.15 synopsis: syntax highlighting library description: Skylighting is a syntax highlighting library. It derives its tokenizers from XML syntax @@ -113,7 +113,7 @@ Regex.KDE.Compile Regex.KDE.Match other-extensions: CPP - build-depends: base >= 4.8 && < 5.0, + build-depends: base >= 4.18 && < 5, mtl, transformers, text, @@ -147,7 +147,7 @@ type: exitcode-stdio-1.0 main-is: test-skylighting.hs hs-source-dirs: test - build-depends: base >= 4.8 && < 5.0, + build-depends: base >= 4.8 && < 5, tasty, tasty-golden, tasty-hunit, @@ -185,7 +185,7 @@ Default-Language: Haskell2010 executable skylighting-extract - build-depends: base >= 4.8 && < 5.0, + build-depends: base >= 4.8 && < 5, skylighting-core, filepath, text, diff --git a/src/Regex/KDE.hs b/src/Regex/KDE.hs --- a/src/Regex/KDE.hs +++ b/src/Regex/KDE.hs @@ -1,6 +1,7 @@ {-# LANGUAGE OverloadedStrings #-} module Regex.KDE - (Regex(..), compileRegex, matchRegex, testRegex, isWordChar) + (Regex(..), compileRegex, matchRegex, matchRegexWithGroups, + extractCapturingGroups, testRegex, isWordChar) where import Regex.KDE.Regex @@ -15,7 +16,7 @@ testRegex caseSensitive re s = let bs = U.fromString s toSlice (off,len) = U.toString $ B.take len $ B.drop off bs - in case compileRegex caseSensitive (U.fromString re) of + in case compileRegex caseSensitive False (U.fromString re) of Right r -> case matchRegex r bs of Nothing -> Nothing diff --git a/src/Regex/KDE/Compile.hs b/src/Regex/KDE/Compile.hs --- a/src/Regex/KDE/Compile.hs +++ b/src/Regex/KDE/Compile.hs @@ -25,12 +25,18 @@ -- It is described here: https://doc.qt.io/qt-6/qregexp.html -- | Compile a UTF-8 encoded ByteString as a Regex. If the first --- parameter is True, then the Regex will be case sensitive. -compileRegex :: Bool -> ByteString -> Either String Regex -compileRegex caseSensitive bs = +-- parameter is True, then the Regex will be case sensitive. If the +-- second parameter is True, quantifiers are minimal (lazy) rather +-- than greedy by default, and the @?@ modifier makes them greedy +-- instead of lazy -- this corresponds to PCRE's UNGREEDY option +-- (QRegularExpression's InvertedGreedinessOption, set by +-- @minimal="1"@ in KDE syntax definitions). +compileRegex :: Bool -> Bool -> ByteString -> Either String Regex +compileRegex caseSensitive minimal bs = let !res = parseOnly (evalStateT parser RState{ rsCurrentCaptureNumber = 0, - rsCaseSensitive = caseSensitive }) + rsCaseSensitive = caseSensitive, + rsMinimal = minimal }) (decodeUtf8With lenientDecode bs) in res where @@ -44,7 +50,8 @@ data RState = RState { rsCurrentCaptureNumber :: Int - , rsCaseSensitive :: Bool } + , rsCaseSensitive :: Bool + , rsMinimal :: Bool } deriving (Show) type RParser = StateT RState Parser @@ -52,8 +59,12 @@ pRegex :: RParser Regex pRegex = option MatchNull $ - foldr MatchAlt - <$> pAltPart + -- earlier alternatives must be the left operands of MatchAlt, since + -- the matcher prefers them, as in PCRE. The first alternative may + -- be empty, as in (?:|a); as in PCRE, an empty alternative matches + -- the empty string: + (\x xs -> foldr1 MatchAlt (x:xs)) + <$> (pAltPart <|> pure mempty) <*> many (lift (char '|') *> (pAltPart <|> pure mempty)) pAltPart :: RParser Regex @@ -66,30 +77,58 @@ pParenthesized :: RParser Regex pParenthesized = do _ <- lift (char '(') - -- pcrepattern says: A group that starts with (?| resets the capturing - -- parentheses numbers in each alternative. - resetCaptureNumbers <- option False (True <$ lift (string "?|")) - (modifier, stModifier) <- - if resetCaptureNumbers - then return (id, id) - else lift (char '?' *> pGroupModifiers) - <|> do modify (\st -> st{ - rsCurrentCaptureNumber = - rsCurrentCaptureNumber st + 1}) - num <- gets rsCurrentCaptureNumber - pure (MatchCapture num, id) - currentCaptureNumber <- gets rsCurrentCaptureNumber - contents <- option MatchNull $ withStateT stModifier $ - foldr MatchAlt - <$> pAltPart - <*> many (lift (char '|') *> - ((when resetCaptureNumbers - (modify (\st -> - st{ rsCurrentCaptureNumber = currentCaptureNumber })) - >> pAltPart) <|> pure mempty)) - _ <- lift (char ')') - return $ modifier contents + pInlineModifiers <|> do + -- pcrepattern says: A group that starts with (?| resets the capturing + -- parentheses numbers in each alternative. + resetCaptureNumbers <- option False (True <$ lift (string "?|")) + (modifier, stModifier) <- + if resetCaptureNumbers + then return (id, id) + else lift (char '?' *> pGroupModifiers) + <|> do modify (\st -> st{ + rsCurrentCaptureNumber = + rsCurrentCaptureNumber st + 1}) + num <- gets rsCurrentCaptureNumber + pure (MatchCapture num, id) + currentCaptureNumber <- gets rsCurrentCaptureNumber + -- modifiers like (?i: or (?U: are scoped to the group, so save the + -- current flags and restore them after the closing parenthesis: + oldCaseSensitive <- gets rsCaseSensitive + oldMinimal <- gets rsMinimal + modify stModifier + contents <- do + x <- pAltPart <|> pure mempty + n0 <- gets rsCurrentCaptureNumber + let pNextAlt = do + _ <- lift (char '|') + when resetCaptureNumbers $ + modify (\st -> + st{ rsCurrentCaptureNumber = currentCaptureNumber }) + y <- pAltPart <|> pure mempty + n <- gets rsCurrentCaptureNumber + pure (y, n) + rest <- many pNextAlt + -- with (?|, numbering after the group resumes after the highest + -- group number used in any alternative, as in PCRE: + when resetCaptureNumbers $ + modify (\st -> + st{ rsCurrentCaptureNumber = maximum (n0 : map snd rest) }) + pure (foldr1 MatchAlt (x : map fst rest)) + _ <- lift (char ')') + modify $ \st -> st{ rsCaseSensitive = oldCaseSensitive + , rsMinimal = oldMinimal } + return $ modifier contents +-- Inline modifiers like (?i) or (?-i), without a colon, apply from +-- this point to the end of the enclosing group (or pattern). The +-- state change persists after the closing parenthesis; the enclosing +-- group's save/restore of rsCaseSensitive provides the scoping. +pInlineModifiers :: RParser Regex +pInlineModifiers = do + stModifier <- lift $ char '?' *> pRegexModifier <* char ')' + modify stModifier + return MatchNull + pGroupModifiers :: Parser (Regex -> Regex, RState -> RState) pGroupModifiers = (do stmod <- pRegexModifier -- (?i: @@ -100,33 +139,55 @@ ((AssertPositive dir, id) <$ char '=') <|> ((AssertNegative dir, id) <$ char '!') <|> - do c <- digit - return (\_ -> Subroutine (ord c - 48), id) + do ds <- many1 digit + case readMay ds of + Just !n -> return (\_ -> Subroutine n, id) + Nothing -> fail "not a number" <|> do void $ char 'R' return (\_ -> Subroutine 0, id) + <|> -- atomic group (?>...): no backtracking into the group + ((Possessive, id) <$ char '>') pRegexModifier :: Parser (RState -> RState) pRegexModifier = do - -- "adlupimnsx-imnsx" - -- i = 105 - = 45 - ons <- many $ satisfy (inClass "adlupimnsx") + -- Of PCRE's inline flags we implement i and U (ungreedy). We also + -- accept m and s, which are no-ops for us: subjects are single + -- lines, so there are no newlines for (?s) to let . match or for + -- (?m) to change the meaning of ^ and $. Flags that would change + -- semantics we don't implement (x, n, ...) are rejected, causing a + -- compile error, as unknown flags do in PCRE. Turning flags *off* + -- is always safe, since only i and U are ever on. + ons <- many $ satisfy (inClass "imsU") offs <- option [] $ char '-' *> - many (satisfy (inClass "imnsx")) + many (satisfy (inClass "imnsxU")) pure $ \st -> st{ rsCaseSensitive = if 'i' `elem` ons && 'i' `notElem` offs then False else ('i' `elem` offs) || rsCaseSensitive st + , rsMinimal = + if 'U' `elem` ons && 'U' `notElem` offs + then True + else ('U' `notElem` offs) && rsMinimal st } pSuffix :: Regex -> RParser Regex +-- a quantifier after an anchor or word-boundary assertion is a +-- compile error in PCRE ("quantifier does not follow a repeatable +-- item"). We get the same effect by leaving the quantifier +-- unconsumed: *, +, and ? are rejected by pRegexChar as special, and +-- { is rejected there when it begins a valid quantifier. +pSuffix re@AssertBeginning = pure re +pSuffix re@AssertEnd = pure re +pSuffix re@AssertWordBoundary = pure re pSuffix re = option re $ do w <- lift $ satisfy (inClass "*+?{") - (case w of - '*' -> return $ MatchAlt (MatchSome re) MatchNull - '+' -> return $ MatchSome re - '?' -> return $ MatchAlt re MatchNull + case w of + '*' -> withModifier (MatchAlt (MatchSome re) MatchNull) + (MatchAlt MatchNull (Lazy (MatchSome re))) + '+' -> withModifier (MatchSome re) (Lazy (MatchSome re)) + '?' -> withModifier (MatchAlt re MatchNull) (MatchAlt MatchNull re) '{' -> do minn <- lift $ option Nothing $ readMay . T.unpack <$> A.takeWhile isDigit @@ -134,24 +195,64 @@ (readMay . T.unpack <$> A.takeWhile isDigit) _ <- lift $ char '}' case (minn, maxn) of - (Nothing, Nothing) -> mzero - (Just n, Nothing) -> return $! atleast n re - (Nothing, Just n) -> return $! atmost n re - (Just m, Just n) -> return $! between m n re - _ -> fail "pSuffix encountered impossible byte") >>= - lift . pQuantifierModifier + _ | maybe False (> maxRepeat) minn || + maybe False (> maxRepeat) maxn + -> mzero -- the unconsumed {..} then causes a + -- parse error via pRegexChar, as in + -- PCRE ("number too big in {} + -- quantifier") + (Nothing, Nothing) -> mzero -- {} and {,} are literal + (Just n, Nothing) -> withModifier (atleast n re) (atleastLazy n re) + (Nothing, Just n) -> withModifier (atmost n re) (atmostLazy n re) + (Just m, Just n) + | m > n -> mzero -- e.g. a{3,1}: the unconsumed {..} + -- then causes a parse error via + -- pRegexChar, as in PCRE ("numbers + -- out of order in {} quantifier") + | otherwise -> withModifier (between m n re) + (betweenLazy m n re) + _ -> fail "pSuffix encountered impossible byte" where - atmost 0 _ = MatchNull - atmost n r = MatchAlt (mconcat (replicate n r)) (atmost (n-1) r) + -- A lazy quantifier prefers fewer repetitions, which is expressed + -- by putting the empty alternative first; Lazy itself is only ever + -- applied to MatchSome (the matcher relies on this). A possessive + -- quantifier commits to the preferred match of the greedy version. + -- In minimal (ungreedy) mode the roles of the bare quantifier and + -- the ? modifier are swapped, as with PCRE's UNGREEDY option; + -- possessive quantifiers are unaffected. + withModifier :: Regex -> Regex -> RParser Regex + withModifier greedy lazy = do + minimal <- gets rsMinimal + let (bare, questioned) = if minimal + then (lazy, greedy) + else (greedy, lazy) + lift $ (Possessive greedy <$ char '+') <|> (questioned <$ char '?') + <|> pure bare + -- repeat counts larger than this (the limit PCRE2 uses) are not + -- treated as quantifiers: + maxRepeat = 65535 :: Int + + -- nest the optional matches -- r(r(r)?)? -- so that the size of + -- the compiled regex is linear, not quadratic, in n: + atmost n r + | n <= 0 = MatchNull + | otherwise = MatchAlt (r <> atmost (n - 1) r) MatchNull + + atmostLazy n r + | n <= 0 = MatchNull + | otherwise = MatchAlt MatchNull (r <> atmostLazy (n - 1) r) + between 0 n r = atmost n r between m n r = mconcat (replicate m r) <> atmost (n - m) r + betweenLazy 0 n r = atmostLazy n r + betweenLazy m n r = mconcat (replicate m r) <> atmostLazy (n - m) r + atleast n r = mconcat (replicate n r) <> MatchAlt (MatchSome r) MatchNull -pQuantifierModifier :: Regex -> Parser Regex -pQuantifierModifier re = option re $ - (Possessive re <$ char '+') <|> (Lazy re <$ char '?') + atleastLazy n r = mconcat (replicate n r) <> + MatchAlt MatchNull (Lazy (MatchSome r)) pRegexChar :: RParser Regex pRegexChar = do @@ -165,52 +266,98 @@ Just !n -> return $ MatchDynamic n Nothing -> fail "not a number") <|> return (MatchChar (== '%')) - '\\' -> lift pRegexEscapedChar + '\\' -> lift $ pRegexEscapedChar caseSensitive '$' -> return AssertEnd '^' -> return AssertBeginning - '[' -> lift pRegexCharClass + '[' -> lift $ pRegexCharClass caseSensitive + '{' -> do + -- if this { begins a valid quantifier, there is nothing for it + -- to repeat, which is a compile error in PCRE ("quantifier does + -- not follow a repeatable item"); the same happens with a + -- quantifier that pSuffix declined to consume (out-of-order or + -- too-big repeat counts, which are also compile errors in PCRE): + isQuantifier <- lift $ option False (True <$ pQuantifierShape) + if isQuantifier + then fail "quantifier does not follow a repeatable item" + else return $ MatchChar (== '{') _ | isSpecial w -> mzero | otherwise -> return $! MatchChar $ if caseSensitive then (== w) else (\d -> toLower d == toLower w) -pRegexEscapedChar :: Parser Regex -pRegexEscapedChar = do +-- The forms {m}, {m,}, {m,n}, and {,n} are quantifiers (PCRE also +-- recognizes {,n} as of 10.43); anything else beginning with { -- +-- e.g. {}, {,}, {b}, or an unclosed {2 -- is a sequence of literal +-- characters. Assumes the initial { has already been consumed. +pQuantifierShape :: Parser () +pQuantifierShape = do + _ <- (A.takeWhile1 isDigit <* option ',' (char ',' <* A.takeWhile isDigit)) + <|> (char ',' *> A.takeWhile1 isDigit) + void $ char '}' + +pRegexEscapedChar :: Bool -> Parser Regex +pRegexEscapedChar caseSensitive = do c <- A.anyChar (case c of 'b' -> return AssertWordBoundary 'B' -> return $ AssertNegative Forward AssertWordBoundary + -- PCRE's \G asserts the position at which the match attempt + -- started. Since matching is always anchored at the start of + -- the input we are given, that is the same as AssertBeginning: + 'G' -> return AssertBeginning + -- PCRE's \A asserts the start of the subject. Since matching is + -- always anchored at the start of the input we are given, that is + -- also the same as AssertBeginning: + 'A' -> return AssertBeginning '{' -> do -- captured pattern: \1 \2 \{12} ds <- many1 digit _ <- char '}' case readMay ds of - Just !n -> return $ MatchCaptured n + Just !n -> return $ MatchCaptured n caseSensitive Nothing -> fail "not a number" + 'g' -> do -- PCRE backreference syntax: \g1 \g{12} + ds <- (char '{' *> many1 digit <* char '}') <|> many1 digit + case readMay ds of + Just !n -> return $ MatchCaptured n caseSensitive + Nothing -> fail "not a number" 'd' -> return $ MatchChar isDigit 'D' -> return $ MatchChar (not . isDigit) 's' -> return $ MatchChar isSpace 'S' -> return $ MatchChar (not . isSpace) + 'h' -> return $ MatchChar isHorizSpace + 'H' -> return $ MatchChar (not . isHorizSpace) 'w' -> return $ MatchChar isWordChar 'W' -> return $ MatchChar (not . isWordChar) 'p' -> MatchChar <$> pUnicodeCharClass - _ | isDigit c -> - return $! MatchCaptured (ord c - ord '0') - | otherwise -> mzero) <|> (MatchChar . (==) <$> pEscaped c) + 'P' -> MatchChar . (not .) <$> pUnicodeCharClass + _ | isDigit c, c /= '0' -> -- \0 is an octal escape, not a backreference + return $! MatchCaptured (ord c - ord '0') caseSensitive + | otherwise -> mzero) <|> (matchLiteralChar <$> pEscaped c) + where + matchLiteralChar d = MatchChar $ + if caseSensitive + then (== d) + else \x -> toLower x == toLower d pEscaped :: Char -> Parser Char pEscaped c = case c of '\\' -> return c 'a' -> return '\a' + -- \b means backspace inside a character class (outside one, it is + -- a word boundary assertion handled by pRegexEscapedChar): + 'b' -> return '\b' 'f' -> return '\f' 'n' -> return '\n' 'r' -> return '\r' 't' -> return '\t' 'v' -> return '\v' - '0' -> do -- \0ooo matches octal ooo - ds <- A.take 3 - case readMay ("'\\o" ++ T.unpack ds ++ "'") of + '0' -> do -- \0 followed by up to two octal digits (as in PCRE) + ds <- A.scan (0 :: Int) (\s w -> if s < 2 && isOctDigit w + then Just (s + 1) + else Nothing) + case readMay ("'\\o0" ++ T.unpack ds ++ "'") of Just x -> return x Nothing -> fail "invalid octal character escape" _ | c >= '1' && c <= '7' -> do @@ -236,8 +383,8 @@ _ | isPunctuation c || isSymbol c || isSpace c -> return c | otherwise -> fail $ "invalid escape \\" ++ [c] -pRegexCharClass :: Parser Regex -pRegexCharClass = do +pRegexCharClass :: Bool -> Parser Regex +pRegexCharClass caseSensitive = do negated <- option False $ True <$ char '^' let getEscapedClass = do _ <- char '\\' @@ -245,6 +392,8 @@ <|> (not . isDigit <$ char 'D') <|> (isSpace <$ char 's') <|> (not . isSpace <$ char 'S') + <|> (isHorizSpace <$ char 'h') + <|> (not . isHorizSpace <$ char 'H') <|> (isWordChar <$ char 'w') <|> (not . isWordChar <$ char 'W') let getPosixClass = do @@ -256,7 +405,8 @@ <|> ((\c -> isSpace c && c `notElem` ['\n','\r','\f','\v']) <$ string "blank") <|> (isControl <$ string "cntrl") - <|> ((\c -> isPrint c || isSpace c) <$ string "graph:") + <|> (isDigit <$ string "digit") + <|> ((\c -> isPrint c && not (isSpace c)) <$ string "graph") <|> (isLower <$ string "lower") <|> (isUpper <$ string "upper") <|> (isPrint <$ string "print") @@ -264,7 +414,7 @@ <|> (isSpace <$ string "space") <|> ((\c -> isAlphaNum c || generalCategory c == ConnectorPunctuation) - <$ string "word:") + <$ string "word") <|> (isHexDigit <$ string "xdigit") _ <- string ":]" return $! if localNegated then not . res else res @@ -278,20 +428,36 @@ void $ A.string "\\Q" cs <- manyTill anyChar (A.string "\\E") return $! \c -> any (== c) cs - brack <- option [] $ [(==']')] <$ char ']' + -- a ] in first position is a literal; it may also be the start of + -- a range, as in []-a]: + brack <- option [] $ do + _ <- char ']' + (do d <- char '-' *> getC + return [\x -> x >= ']' && x <= d]) + <|> return [(== ']')] fs <- many (getQELiteral <|> getEscapedClass <|> getPosixClass <|> getCRange - <|> (A.string "\\p" *> pUnicodeCharClass)) + <|> (A.string "\\p" *> pUnicodeCharClass) + <|> (A.string "\\P" *> ((not .) <$> pUnicodeCharClass))) void $ char ']' let f c = any ($ c) $ brack ++ fs + -- for case-insensitive matching, a character matches (or, if + -- negated, is excluded) if any of its case variants matches: + let f' c | caseSensitive = f c + | otherwise = f c || f (toLower c) || f (toUpper c) return $! MatchChar $ if negated - then not . f - else f + then not . f' + else f' --- character class \p{Lo}; we assume \p is already parsed +-- character class \p{Lo}, \p{^Lo}, or \pL; we assume \p is already +-- parsed pUnicodeCharClass :: Parser (Char -> Bool) pUnicodeCharClass = do - ds <- char '{' *> A.takeWhile (/= '}') <* char '}' - return $ + (negated, ds) <- + (char '{' *> ((,) <$> option False (True <$ char '^') + <*> (A.takeWhile (/= '}') <* char '}'))) + <|> ((,) False . T.singleton <$> satisfy isAlpha) + let neg = if negated then (not .) else id + return $ neg $ (case ds of "Lu" -> (== UppercaseLetter) "Ll" -> (== LowercaseLetter) @@ -343,6 +509,14 @@ _ -> const False) . generalCategory +-- PCRE's \h matches this fixed list of horizontal whitespace +-- characters (which is not the same as Unicode category Zs): +isHorizSpace :: Char -> Bool +isHorizSpace c = + c == '\t' || c == ' ' || c == '\xA0' || c == '\x1680' || c == '\x180E' || + (c >= '\x2000' && c <= '\x200A') || c == '\x202F' || c == '\x205F' || + c == '\x3000' + isSpecial :: Char -> Bool isSpecial '\\' = True isSpecial '?' = True @@ -350,7 +524,9 @@ isSpecial '+' = True -- isSpecial '{' = True -- this is okay except in suffixes isSpecial '[' = True -isSpecial ']' = True +-- an unmatched ] is treated as a literal (as in PCRE), so it is +-- not included here; the ] terminating a character class is consumed +-- by pRegexCharClass: isSpecial '%' = True isSpecial '(' = True isSpecial ')' = True diff --git a/src/Regex/KDE/Match.hs b/src/Regex/KDE/Match.hs --- a/src/Regex/KDE/Match.hs +++ b/src/Regex/KDE/Match.hs @@ -5,13 +5,18 @@ {-# LANGUAGE BinaryLiterals #-} module Regex.KDE.Match ( matchRegex + , matchRegexWithGroups + , extractCapturingGroups ) where import qualified Data.ByteString as B import Data.ByteString (ByteString) import qualified Data.ByteString.UTF8 as U +import Data.Char (toLower) import qualified Data.Set as Set import Data.Set (Set) +import Data.Bits (shiftL, (.|.)) +import Data.Word (Word8) import Regex.KDE.Regex import qualified Data.IntMap.Strict as M #if !MIN_VERSION_base(4,11,0) @@ -21,20 +26,72 @@ -- Note that all matches are from the beginning of the string. -- The ^ anchor is implicit at the beginning of the regex. +-- To reproduce PCRE's leftmost-first (backtracking) semantics in a +-- set-based matcher, every match carries a path recording the choices +-- made to reach it: 0 for taking the left branch of an alternation or +-- continuing a greedy repetition, 1 for the right branch or ending +-- the repetition (for lazy repetitions the loop codes are reversed). +-- Comparing paths lexicographically gives the order in which a +-- backtracking matcher like PCRE would find the matches, so the +-- preferred match is always the one with the smallest path. + +-- A sequence of binary choices, packed into an Integer (most recent +-- choice in the least significant bit) together with its length. +-- Compared lexicographically as a bit sequence, with a proper prefix +-- ordered before its extensions. +data Path = Path !Integer !Int + deriving (Show, Eq) + +instance Ord Path where + compare (Path i1 l1) (Path i2 l2) = + compare (i1 `shiftL` max 0 (l2 - l1)) (i2 `shiftL` max 0 (l1 - l2)) + <> compare l1 l2 + +emptyPath :: Path +emptyPath = Path 0 0 + +pathSnoc :: Path -> Word8 -> Path +pathSnoc (Path i l) b = Path ((i `shiftL` 1) .|. fromIntegral b) (l + 1) + data Match = Match { matchBytes :: !ByteString , matchOffset :: !Int , matchCaptures :: !(M.IntMap (Int, Int)) -- starting offset, length in bytes - } deriving (Show, Eq) + , matchPath :: !Path + } deriving (Show) --- preferred matches are <= +-- consistent with Ord (which ignores matchBytes, since all matches +-- in a given run share it): +instance Eq Match where + m1 == m2 = compare m1 m2 == EQ + +-- preferred matches are <=; the path (priority) is decisive, and the +-- other comparisons only make the order total: instance Ord Match where - m1 <= m2 - | matchOffset m1 > matchOffset m2 = True - | matchOffset m1 < matchOffset m2 = False - | otherwise = matchCaptures m1 >= matchCaptures m2 + compare m1 m2 = + compare (matchPath m1) (matchPath m2) <> + compare (matchOffset m1) (matchOffset m2) <> + compare (matchCaptures m1) (matchCaptures m2) +-- the state of a match, disregarding its priority +stateKey :: Match -> (Int, M.IntMap (Int, Int)) +stateKey m = (matchOffset m, matchCaptures m) + +-- append a choice to the path of every match +addChoice :: Word8 -> Set Match -> Set Match +addChoice !b = Set.map (\m -> m{ matchPath = pathSnoc (matchPath m) b }) + +-- Discard any match whose state coincides with that of a preferred +-- (smaller-path) match: their futures are identical, and a +-- backtracking matcher would explore the preferred one first. +dedup :: Set Match -> Set Match +dedup = snd . Set.foldl' step (Set.empty, Set.empty) + where + step (!seen, !out) m + | stateKey m `Set.member` seen = (seen, out) + | otherwise = (Set.insert (stateKey m) seen, Set.insert m out) + mapMatching :: (Match -> Match) -> Set Match -> Set Match mapMatching f = Set.filter ((>= 0) . matchOffset) . Set.map f @@ -42,22 +99,28 @@ sizeLimit :: Int sizeLimit = 2000 --- prune matches if it gets out of hand +-- prune matches if it gets out of hand, keeping preferred matches prune :: Set Match -> Set Match prune ms = if Set.size ms > sizeLimit then Set.take sizeLimit ms else ms --- first argument is a map of capturing groups, needed for Subroutine. -exec :: M.IntMap Regex -> Direction -> Regex -> Set Match -> Set Match +-- first argument: the set of subroutine calls (group number, offset) +-- currently being evaluated -- used to prevent infinite recursion -- +-- and a map of capturing groups, needed for Subroutine. +exec :: (Set (Int, Int), M.IntMap Regex) + -> Direction -> Regex -> Set Match -> Set Match exec _ _ MatchNull = id -exec cgs dir (Lazy re) = -- note: the action is below under Concat - exec cgs dir (MatchConcat (Lazy re) MatchNull) +exec cgs dir (Lazy (MatchSome re)) = someLoop cgs dir re 1 0 +exec cgs dir (Lazy re) = -- Lazy is only applied to MatchSome (see Compile) + exec cgs dir re exec cgs dir (Possessive re) = - foldr - (\elt s -> case Set.lookupMin (exec cgs dir re (Set.singleton elt)) of - Nothing -> s - Just m -> Set.insert m s) + -- commit to the first match (in backtracking order) of re; its + -- internal choices are forgotten, so the path is reset: + Set.foldl' + (\s m -> case Set.lookupMin (exec cgs dir re (Set.singleton m)) of + Nothing -> s + Just m' -> Set.insert m'{ matchPath = matchPath m } s) mempty exec cgs dir (MatchDynamic n) = -- if this hasn't been replaced, match literal exec cgs dir (MatchChar (== '%') <> @@ -65,11 +128,16 @@ exec _ _ AssertEnd = Set.filter (\m -> matchOffset m == B.length (matchBytes m)) exec _ _ AssertBeginning = Set.filter (\m -> matchOffset m == 0) exec cgs _ (AssertPositive dir regex) = - Set.unions . Set.map - (\m -> Set.map (\m' -> -- we keep captures but not matches - m'{ matchBytes = matchBytes m, - matchOffset = matchOffset m }) - $ exec cgs dir regex (Set.singleton m)) + -- assertions are atomic: only the captures of the first match (in + -- backtracking order) of the assertion are kept, as in PCRE: + Set.foldl' + (\s m -> case Set.lookupMin (exec cgs dir regex (Set.singleton m)) of + Nothing -> s + Just m' -> Set.insert + m'{ matchBytes = matchBytes m + , matchOffset = matchOffset m + , matchPath = matchPath m } s) + mempty exec cgs _ (AssertNegative dir regex) = Set.filter (\m -> null (exec cgs dir regex (Set.singleton m))) exec _ _ AssertWordBoundary = Set.filter atWordBoundary @@ -94,22 +162,7 @@ _ -> m{ matchOffset = -1 } exec cgs dir (MatchConcat (MatchConcat r1 r2) r3) = exec cgs dir (MatchConcat r1 (MatchConcat r2 r3)) -exec cgs Forward (MatchConcat (Lazy r1) r2) = - Set.foldl Set.union mempty . Set.map - (\m -> - let ms1 = exec cgs Forward r1 (Set.singleton m) - in if Set.null ms1 - then ms1 - else go ms1) - where - go ms = case Set.lookupMax ms of -- find shortest match - Nothing -> Set.empty - Just m' -> - let s' = exec cgs Forward r2 (Set.singleton m') - in if Set.null s' - then go (Set.delete m' ms) - else s' -exec cgs Forward (MatchConcat r1 r2) = -- TODO longest match first +exec cgs Forward (MatchConcat r1 r2) = \ms -> let ms1 = exec cgs Forward r1 ms in if Set.null ms1 @@ -117,14 +170,9 @@ else exec cgs Forward r2 (prune ms1) exec cgs Backward (MatchConcat r1 r2) = exec cgs Backward r1 . exec cgs Backward r2 -exec cgs dir (MatchAlt r1 r2) = \ms -> exec cgs dir r1 ms <> exec cgs dir r2 ms -exec cgs dir (MatchSome re) = go - where - go ms = case exec cgs dir re ms of - ms' | Set.null ms' -> Set.empty - | ms' == ms -> ms - | otherwise -> let ms'' = prune ms' - in ms'' <> go ms'' +exec cgs dir (MatchAlt r1 r2) = \ms -> + dedup $ exec cgs dir r1 (addChoice 0 ms) <> exec cgs dir r2 (addChoice 1 ms) +exec cgs dir (MatchSome re) = someLoop cgs dir re 0 1 exec cgs dir (MatchCapture i re) = Set.foldr Set.union Set.empty . Set.map (\m -> @@ -134,26 +182,69 @@ let len = matchOffset m' - matchOffset m in m'{ matchCaptures = M.insert i (matchOffset m, len) (matchCaptures m') } -exec _ dir (MatchCaptured n) = mapMatching matchCaptured +exec _ dir (MatchCaptured n caseSensitive) = mapMatching matchCaptured where matchCaptured m = case M.lookup n (matchCaptures m) of Just (offset, len) -> let capture = B.take len $ B.drop offset $ matchBytes m in case dir of - Forward | B.isPrefixOf capture - (B.drop (matchOffset m) (matchBytes m)) + Forward + | caseSensitive + , B.isPrefixOf capture + (B.drop (matchOffset m) (matchBytes m)) -> m{ matchOffset = matchOffset m + B.length capture } - Backward | B.isSuffixOf capture - (B.take (matchOffset m) (matchBytes m)) + | not caseSensitive + , Just len' <- ciPrefixLength (U.toString capture) + (B.drop (matchOffset m) (matchBytes m)) + -> m{ matchOffset = matchOffset m + len' } + Backward + | caseSensitive + , B.isSuffixOf capture + (B.take (matchOffset m) (matchBytes m)) -> m{ matchOffset = matchOffset m - B.length capture } + | not caseSensitive + , Just off' <- ciSuffixOffset + (reverse (U.toString capture)) + (matchBytes m) (matchOffset m) + -> m{ matchOffset = off' } _ -> m{ matchOffset = -1 } Nothing -> m{ matchOffset = -1 } -exec cgs dir (Subroutine i) = +exec (active, cgs) dir (Subroutine i) = case M.lookup i cgs of Nothing -> id -- ignore references to nonexistent groups - Just re' -> exec cgs dir re' + Just re' -> \ms -> + -- A subroutine that calls itself again without having consumed + -- any input can never make progress: block re-entry at the same + -- offset so that zero-progress recursion (e.g. `x|(?R)`) fails + -- instead of looping forever. + dedup $ Set.unions + [ exec (Set.insert (i, matchOffset m) active, cgs) dir re' + (Set.singleton m) + | m <- Set.toList ms + , (i, matchOffset m) `Set.notMember` active ] +-- Match one or more repetitions of a regex. contB is the path code +-- appended when continuing with another repetition, stopB the one +-- appended when stopping: 0/1 for greedy, 1/0 for lazy repetitions, +-- so that the paths order the results the way a backtracking matcher +-- would find them. +someLoop :: (Set (Int, Int), M.IntMap Regex) -> Direction -> Regex + -> Word8 -> Word8 -> Set Match -> Set Match +someLoop cgs dir re !contB !stopB = \ms0 -> + let ms1 = dedup $ exec cgs dir re ms0 -- first, obligatory repetition + in go (Set.map stateKey ms1) ms1 + where + go !seen ms + | Set.null ms = Set.empty + | otherwise = + let ms' = dedup $ prune $ exec cgs dir re (addChoice contB ms) + -- Drop matches that revisit an already-seen state: they + -- have no new futures, and this guarantees termination + -- when an iteration can match the empty string. + new = Set.filter (\m -> stateKey m `Set.notMember` seen) ms' + in addChoice stopB ms <> go (seen <> Set.map stateKey new) new + atWordBoundary :: Match -> Bool atWordBoundary m = case lastCharOffset (matchBytes m) (matchOffset m) of @@ -163,46 +254,86 @@ (cur:next:_) -> isWordChar cur /= isWordChar next _ -> True +-- If the characters of the first argument match the beginning of the +-- bytestring case-insensitively, return the length in bytes of the +-- matching prefix. +ciPrefixLength :: String -> ByteString -> Maybe Int +ciPrefixLength [] _ = Just 0 +ciPrefixLength (c:cs) bs = + case U.decode bs of + Just (d, n) | toLower d == toLower c -> + (n +) <$> ciPrefixLength cs (B.drop n bs) + _ -> Nothing + +-- If the characters of the first argument (reversed) match the +-- characters just before the given offset case-insensitively, return +-- the offset at which the match begins. +ciSuffixOffset :: String -> ByteString -> Int -> Maybe Int +ciSuffixOffset [] _ off = Just off +ciSuffixOffset (c:cs) bs off = + case lastCharOffset bs off of + Just off' | Just (d, _) <- U.decode (B.drop off' bs) + , toLower d == toLower c -> ciSuffixOffset cs bs off' + _ -> Nothing + +-- Return the offset of the start of the (UTF-8 encoded) character +-- that ends at (i.e., whose last byte is just before) offset n. lastCharOffset :: ByteString -> Int -> Maybe Int lastCharOffset _ 0 = Nothing -lastCharOffset _ 1 = Just 0 -lastCharOffset bs n = - case B.index bs (n - 2) of - w | w < 0b10000000 -> Just (n - 1) - | w >= 0b11000000 -> Just (n - 1) - | otherwise -> lastCharOffset bs (n - 1) +lastCharOffset bs n = go (n - 1) + where + go !k + | k <= 0 = Just 0 + | isContinuationByte (B.index bs k) = go (k - 1) + | otherwise = Just k + isContinuationByte w = w >= 0b10000000 && w < 0b11000000 -- | Match a Regex against a (presumed UTF-8 encoded) ByteString, -- returning the matched text and a map of (offset, size) -- pairs for captures. Note that all matches are from the --- beginning of the string (a @^@ anchor is implicit). Note --- also that to avoid pathological performance in certain cases, --- the matcher is limited to considering 2000 possible matches --- at a time; when that threshold is reached, it discards --- smaller matches. Hence certain regexes may incorrectly fail to --- match: e.g. @a*a{3000}$@ on a string of 3000 @a@s. +-- beginning of the string (a @^@ anchor is implicit). As in +-- PCRE, the match returned is the first one that a backtracking +-- matcher would find (leftmost alternatives are preferred), which +-- is not necessarily the longest. Note also that to avoid +-- pathological performance in certain cases, the matcher is limited +-- to considering 2000 possible matches at a time; when that +-- threshold is reached, it discards lower-priority matches. Hence +-- certain regexes may incorrectly fail to match: e.g. @a*a{3000}$@ +-- on a string of 3000 @a@s. matchRegex :: Regex -> ByteString -> Maybe (ByteString, M.IntMap (Int, Int)) -matchRegex re bs = - let capturingGroups = extractCapturingGroups re - in toResult <$> Set.lookupMin - (exec capturingGroups Forward re - (Set.singleton (Match bs 0 M.empty))) +matchRegex re = matchRegexWithGroups (extractCapturingGroups re) re + +-- | Like 'matchRegex', but takes the map of capturing groups (as +-- computed by 'extractCapturingGroups') as an argument, so that it +-- can be computed once per regex instead of once per match. (The +-- map is only consulted for regexes containing subroutine calls +-- like @(?1)@ or @(?R)@.) +matchRegexWithGroups :: M.IntMap Regex + -> Regex + -> ByteString + -> Maybe (ByteString, M.IntMap (Int, Int)) +matchRegexWithGroups capturingGroups re bs = + toResult <$> Set.lookupMin + (exec (Set.empty, capturingGroups) Forward re + (Set.singleton (Match bs 0 M.empty emptyPath))) where toResult m = (B.take (matchOffset m) (matchBytes m), (matchCaptures m)) +-- | Extract the capturing groups of a regex, numbered as in the +-- regex, with the whole regex at key 0. extractCapturingGroups :: Regex -> M.IntMap Regex -extractCapturingGroups regex = M.singleton 0 regex <> - case regex of - MatchSome re -> extractCapturingGroups re - MatchAlt re1 re2 -> - extractCapturingGroups re1 <> extractCapturingGroups re2 - MatchConcat re1 re2 -> - extractCapturingGroups re1 <> extractCapturingGroups re2 - MatchCapture i re -> M.singleton i re - AssertPositive _ re -> extractCapturingGroups re - AssertNegative _ re -> extractCapturingGroups re - Possessive re -> extractCapturingGroups re - Lazy re -> extractCapturingGroups re - _ -> mempty +extractCapturingGroups regex = M.insert 0 regex (go regex) + where + -- Note: left-biased union means that with (?|...), which reuses + -- group numbers, the first alternative's group wins. + go (MatchSome re) = go re + go (MatchAlt re1 re2) = go re1 <> go re2 + go (MatchConcat re1 re2) = go re1 <> go re2 + go (MatchCapture i re) = M.insert i re (go re) + go (AssertPositive _ re) = go re + go (AssertNegative _ re) = go re + go (Possessive re) = go re + go (Lazy re) = go re + go _ = mempty diff --git a/src/Regex/KDE/Regex.hs b/src/Regex/KDE/Regex.hs --- a/src/Regex/KDE/Regex.hs +++ b/src/Regex/KDE/Regex.hs @@ -22,7 +22,7 @@ MatchAlt !Regex !Regex | MatchConcat !Regex !Regex | MatchCapture !Int !Regex | - MatchCaptured !Int | + MatchCaptured !Int !Bool | -- group number, case sensitivity AssertWordBoundary | AssertBeginning | AssertEnd | @@ -43,7 +43,8 @@ ")" show (MatchCapture i re) = "(MatchCapture " <> show i <> " " <> show re <> ")" - show (MatchCaptured n) = "(MatchCaptured " <> show n <> ")" + show (MatchCaptured n cs) = "(MatchCaptured " <> show n <> " " <> + show cs <> ")" show AssertWordBoundary = "AssertWordBoundary" show AssertBeginning = "AssertBeginning" show AssertEnd = "AssertEnd" diff --git a/src/Skylighting/Core.hs b/src/Skylighting/Core.hs --- a/src/Skylighting/Core.hs +++ b/src/Skylighting/Core.hs @@ -39,13 +39,17 @@ -- | Lookup a syntax by full name (case insensitive). syntaxByName :: SyntaxMap -> Text -> Maybe Syntax syntaxByName syntaxmap name = - Map.lookup (Text.toLower name) (Map.mapKeys Text.toLower syntaxmap) + Map.lookup name syntaxmap `mplus` -- fast path for exact match + listToMaybe [s | (k, s) <- Map.toList syntaxmap + , Text.toLower k == lcname ] + where lcname = Text.toLower name -- | Lookup a syntax by short name (case insensitive). syntaxByShortName :: SyntaxMap -> Text -> Maybe Syntax syntaxByShortName syntaxmap name = listToMaybe [s | s <- Map.elems syntaxmap - , Text.toLower (sShortname s) == Text.toLower name ] + , Text.toLower (sShortname s) == lcname ] + where lcname = Text.toLower name -- | Lookup syntax by (in order) full name (case insensitive), -- short name (case insensitive), extension. diff --git a/src/Skylighting/Loader.hs b/src/Skylighting/Loader.hs --- a/src/Skylighting/Loader.hs +++ b/src/Skylighting/Loader.hs @@ -64,7 +64,9 @@ -- SyntaxMap will be made up of only the files that could successfully be loaded -- and parsed. loadValidSyntaxesFromDir :: FilePath -> IO (LoadErrMap, SyntaxMap) -loadValidSyntaxesFromDir path = foldM go (mempty, mempty) =<< syntaxFiles path +loadValidSyntaxesFromDir path = do + (errMap, sm) <- foldM go (mempty, mempty) =<< syntaxFiles path + return (errMap, M.map (resolveKeywords sm) sm) where go (errMap, syntaxMap) file = loadSyntaxFromFile file >>= \case diff --git a/src/Skylighting/Parser.hs b/src/Skylighting/Parser.hs --- a/src/Skylighting/Parser.hs +++ b/src/Skylighting/Parser.hs @@ -80,7 +80,10 @@ _ -> defaultVal -- | Parses a file containing a Kate XML syntax definition --- into a 'Syntax' description. +-- into a 'Syntax' description. Note that the resulting 'Syntax' +-- must be processed with 'resolveKeywords' (once the full syntax +-- map is assembled) before it can be used for tokenizing; the +-- functions in Skylighting.Loader do this automatically. parseSyntaxDefinition :: FilePath -> IO (Either String Syntax) parseSyntaxDefinition fp = do bs <- BL.readFile fp @@ -146,7 +149,7 @@ let itemDatas = getItemData hlEl - let defKeywordAttr = getKeywordAttrs rootEl + let defKeywordAttr = getKeywordAttrs casesensitive rootEl let contextEls = getElementsNamed "contexts" hlEl >>= getElementsNamed "context" @@ -154,7 +157,7 @@ let syntaxname = getAttrValue "name" rootEl contexts <- mapM - (getContext casesensitive syntaxname itemDatas lists defKeywordAttr) + (getContext syntaxname itemDatas lists defKeywordAttr) contextEls startingContext <- case contexts of @@ -216,24 +219,35 @@ | otherwise -> (T.drop 2 y, x) getParser :: Monad m - => Bool -> Text -> ItemData -> M.Map Text [ListItem] -> KeywordAttr + => Text -> ItemData -> M.Map Text [ListItem] -> KeywordAttr -> Text -> Element -> ExceptT String m Rule -getParser casesensitive syntaxname itemdatas lists kwattr cattr el = do +getParser syntaxname itemdatas lists kwattr cattr el = do let name = nameLocalName . elementName $ el let attribute = getAttrValue "attribute" el let context = getAttrValue "context" el let char0 = readChar $ getAttrValue "char" el let char1 = readChar $ getAttrValue "char1" el let str' = getAttrValue "String" el - let insensitive = vBool (not casesensitive) $ getAttrValue "insensitive" el + -- rules default to case-sensitive matching; the language's + -- casesensitive attribute applies only to keyword lists (as in KDE): + let insensitive = vBool False $ getAttrValue "insensitive" el let includeAttrib = vBool False $ getAttrValue "includeAttrib" el let weakDelim = Set.fromList $ T.unpack $ getAttrValue "weakDeliminator" el + let additionalDelim = Set.fromList $ T.unpack $ + getAttrValue "additionalDeliminator" el + -- In KDE the word delimiters of a rule are the language's word + -- delimiters (standard delimiters as modified by additionalDeliminator + -- and weakDeliminator on general > keywords, which we get from + -- kwattr), further modified by the additionalDeliminator and + -- weakDeliminator attributes on the rule itself: + let ruleDelims = Set.union (keywordDelims kwattr) additionalDelim + Set.\\ weakDelim let lookahead = vBool False $ getAttrValue "lookAhead" el let firstNonSpace = vBool False $ getAttrValue "firstNonSpace" el let column' = getAttrValue "column" el let dynamic = vBool False $ getAttrValue "dynamic" el - children <- mapM (getParser casesensitive - syntaxname itemdatas lists kwattr attribute) + let minimal = vBool False $ getAttrValue "minimal" el + children <- mapM (getParser syntaxname itemdatas lists kwattr attribute) [e | NodeElement e <- elementNodes el ] let tildeRegex = name == "RegExpr" && T.take 1 str' == "^" let str = if tildeRegex then T.drop 1 str' else str' @@ -242,11 +256,13 @@ else either (\_ -> Nothing) (Just . fst) $ TR.decimal column' let re = RegExpr RE{ reString = TE.encodeUtf8 str - , reCaseSensitive = not insensitive } + , reCaseSensitive = not insensitive + , reMinimal = minimal } + let contextSwitches = parseContextSwitches syntaxname context let (incsyntax, inccontext) = - case T.breakOn "##" context of - (_,x) | T.null x -> (syntaxname, context) - (cont, lang) -> (T.drop 2 lang, cont) + case contextSwitches of + (Push (s,c) : _) -> (s,c) + _ -> error "IncludeRules doesn't specify a syntax and context" matcher <- case name of "DetectChar" -> return $ DetectChar char0 "Detect2Chars" -> return $ Detect2Chars char0 char1 @@ -255,34 +271,50 @@ "StringDetect" -> return $ StringDetect str "WordDetect" -> return $ WordDetect str "RegExpr" -> return $ re - "keyword" -> return $ Keyword kwattr (Left str) + -- an insensitive attribute on the keyword rule itself + -- (if present) overrides the case sensitivity of the + -- keyword list, and additionalDeliminator and + -- weakDeliminator attributes adjust its delimiters + -- (as in KDE): + "keyword" -> return $ + let kwattr' = if M.member (String.fromString "insensitive") + (elementAttributes el) + then kwattr{ keywordCaseSensitive = + not insensitive } + else kwattr + in Keyword kwattr'{ keywordDelims = ruleDelims } + (Left str) "Int" -> return $ Int "Float" -> return $ Float "HlCOct" -> return $ HlCOct "HlCHex" -> return $ HlCHex "HlCStringChar" -> return $ HlCStringChar "HlCChar" -> return $ HlCChar - "LineContinue" -> return $ LineContinue + -- KDE uses the first character of the char attribute, + -- or backslash if it is absent or empty: + "LineContinue" -> return $ LineContinue $ + case T.uncons (getAttrValue "char" el) of + Just (c, _) -> c + Nothing -> '\\' "IncludeRules" -> return $ IncludeRules (incsyntax, inccontext) "DetectSpaces" -> return $ DetectSpaces "DetectIdentifier" -> return $ DetectIdentifier _ -> throwError $ "Unknown element " ++ T.unpack name - let contextSwitch = if name == "IncludeRules" - then [] -- is this right? - else parseContextSwitch incsyntax inccontext return $ Rule{ rMatcher = matcher , rAttribute = fromMaybe NormalTok $ if T.null attribute then M.lookup cattr itemdatas else M.lookup attribute itemdatas , rIncludeAttribute = includeAttrib - , rWeakDeliminators = weakDelim + , rWordDelimiters = ruleDelims , rDynamic = dynamic , rCaseSensitive = not insensitive , rChildren = children - , rContextSwitch = contextSwitch + , rContextSwitch = if name == "IncludeRules" + then [] + else contextSwitches , rLookahead = lookahead , rFirstNonspace = firstNonSpace , rColumn = column @@ -290,25 +322,22 @@ getContext :: Monad m - => Bool - -> Text + => Text -> ItemData -> M.Map Text [ListItem] -> KeywordAttr -> Element -> ExceptT String m Context -getContext casesensitive syntaxname itemDatas lists kwattr el = do +getContext syntaxname itemDatas lists kwattr el = do let name = getAttrValue "name" el let attribute = getAttrValue "attribute" el let lineEmptyContext = getAttrValue "lineEmptyContext" el let lineEndContext = getAttrValue "lineEndContext" el - let lineBeginContext = getAttrValue "lineBeginContext" el let fallthrough = vBool False $ getAttrValue "fallthrough" el let fallthroughContext = getAttrValue "fallthroughContext" el let dynamic = vBool False $ getAttrValue "dynamic" el - parsers <- mapM (getParser casesensitive - syntaxname itemDatas lists kwattr attribute) + parsers <- mapM (getParser syntaxname itemDatas lists kwattr attribute) [e | NodeElement e <- elementNodes el ] return $ Context { @@ -316,15 +345,19 @@ , cSyntax = syntaxname , cRules = parsers , cAttribute = fromMaybe NormalTok $ M.lookup attribute itemDatas + -- lineEmptyContext defaults to lineEndContext when it is + -- unspecified or #stay; this avoids skipping empty lines + -- after a line continuation character (KDE context.cpp, + -- Context::resolveContexts, see kde bug 405903): , cLineEmptyContext = - parseContextSwitch syntaxname lineEmptyContext + case parseContextSwitches syntaxname lineEmptyContext of + [] -> parseContextSwitches syntaxname lineEndContext + cs -> cs , cLineEndContext = - parseContextSwitch syntaxname lineEndContext - , cLineBeginContext = - parseContextSwitch syntaxname lineBeginContext + parseContextSwitches syntaxname lineEndContext , cFallthrough = fallthrough , cFallthroughContext = - parseContextSwitch syntaxname fallthroughContext + parseContextSwitches syntaxname fallthroughContext , cDynamic = dynamic } @@ -334,33 +367,41 @@ | e <- (getElementsNamed "itemDatas" el >>= getElementsNamed "itemData") ] -getKeywordAttrs :: Element -> KeywordAttr -getKeywordAttrs el = +-- The default case sensitivity of keyword lists is given by the +-- casesensitive attribute on the language element, and may be +-- overridden by the casesensitive attribute on general > keywords: +getKeywordAttrs :: Bool -> Element -> KeywordAttr +getKeywordAttrs casesensitive el = case (getElementsNamed "general" el >>= getElementsNamed "keywords") of - [] -> defaultKeywordAttr + [] -> defaultKeywordAttr{ keywordCaseSensitive = casesensitive } (x:_) -> let weakDelim = T.unpack $ getAttrValue "weakDeliminator" x additionalDelim = T.unpack $ getAttrValue "additionalDeliminator" x in KeywordAttr { keywordCaseSensitive = - vBool True $ getAttrValue "casesensitive" x + vBool casesensitive $ getAttrValue "casesensitive" x , keywordDelims = Set.union standardDelims (Set.fromList additionalDelim) Set.\\ Set.fromList weakDelim } -parseContextSwitch :: Text -> Text -> [ContextSwitch] -parseContextSwitch syntaxname t = +parseContextSwitches :: Text -> Text -> [ContextSwitch] +parseContextSwitches syntaxname t = if T.null t || t == "#stay" then [] else case T.stripPrefix "#pop" t of - Just rest -> Pop : parseContextSwitch syntaxname rest - Nothing -> - let (othersyntax, contextname) = - splitContext (T.dropWhile (=='!') t) - syntaxname' = if T.null othersyntax - then syntaxname - else othersyntax - in [Push (syntaxname', contextname)] + Just rest -> + Pop : parseContextSwitches syntaxname rest + Nothing -> + -- a sequence of contexts separated by !; in + -- A!B!C, C is the top of the stack. + let cs = filter (not . T.null) $ T.split (== '!') t + toContext x = + let (othersyntax, contextname) = splitContext x + syntaxname' = if T.null othersyntax + then syntaxname + else othersyntax + in (syntaxname', contextname) + in map (Push . toContext) cs type ItemData = M.Map Text TokenType diff --git a/src/Skylighting/Regex.hs b/src/Skylighting/Regex.hs --- a/src/Skylighting/Regex.hs +++ b/src/Skylighting/Regex.hs @@ -8,10 +8,11 @@ module Skylighting.Regex ( Regex(..) , RE - , pattern RE, reCaseSensitive, reString + , pattern RE, reCaseSensitive, reMinimal, reString , compileRE , compileRegex , matchRegex + , matchRegexWithGroups , testRegex , isWordChar ) where @@ -21,6 +22,7 @@ import qualified Data.ByteString.Base64 as Base64 import qualified Data.ByteString.Char8 as BS import Data.Data +import qualified Data.IntMap.Strict as M import qualified Data.Text as Text import qualified Data.Text.Encoding as TE #if !MIN_VERSION_base(4,13,0) @@ -34,25 +36,33 @@ data RE = RE'{ _reString :: BS.ByteString , _reCaseSensitive :: Bool - , _reCompiled :: Either String Regex + , _reMinimal :: Bool + , _reCompiled :: Either String (Regex, M.IntMap Regex) } deriving Typeable --- We define a smart constructor which also holds the compiled regex, to avoid --- recompiling each time we tokenize. +-- We define a smart constructor which also holds the compiled regex +-- and its capturing groups, to avoid recomputing them each time we +-- tokenize. {-# COMPLETE RE #-} -pattern RE :: BS.ByteString -> Bool -> RE -pattern RE {reString, reCaseSensitive} <- RE' reString reCaseSensitive _ where - RE str caseSensitive = RE' str caseSensitive (compileRegex caseSensitive str) +pattern RE :: BS.ByteString -> Bool -> Bool -> RE +pattern RE {reString, reCaseSensitive, reMinimal} <- + RE' reString reCaseSensitive reMinimal _ where + RE str caseSensitive minimal = + RE' str caseSensitive minimal + (fmap (\r -> (r, extractCapturingGroups r)) + (compileRegex caseSensitive minimal str)) -- Unfortunately this means we need to derive all the instances ourselves. instance Show RE where - showsPrec d (RE str caseSensitive) = showParen (d > 10) - $ showString "RE {reString = " + showsPrec d (RE str caseSensitive minimal) = showParen (d > 10) + $ showString "RE {reString = " . showsPrec 11 str . showString ", reCaseSensitive = " . showsPrec 11 caseSensitive + . showString ", reMinimal = " + . showsPrec 11 minimal . showString "}" instance Read RE where @@ -66,11 +76,15 @@ Ident "reCaseSensitive" <- lexP Punc "=" <- lexP caseSensitive <- readPrec + Punc "," <- lexP + Ident "reMinimal" <- lexP + Punc "=" <- lexP + minimal <- readPrec Punc "}" <- lexP - pure (RE str caseSensitive) + pure (RE str caseSensitive minimal) -toComparisonKey :: RE -> (BS.ByteString, Bool) -toComparisonKey (RE x y) = (x, y) +toComparisonKey :: RE -> (BS.ByteString, Bool, Bool) +toComparisonKey (RE x y z) = (x, y, z) instance Eq RE where x == y = toComparisonKey x == toComparisonKey y @@ -84,22 +98,24 @@ tyRE = mkDataType "Skylighting.Regex.RE" [conRE] instance Data RE where - gfoldl k z (RE s c) = z RE `k` s `k` c - gunfold k z _ = k (k (z RE)) + gfoldl k z (RE s c m) = z RE `k` s `k` c `k` m + gunfold k z _ = k (k (k (z RE))) toConstr _ = conRE dataTypeOf _ = tyRE instance Binary RE where - put (RE x y) = put x >> put y - get = RE <$> get <*> get + put (RE x y z) = put x >> put y >> put z + get = RE <$> get <*> get <*> get instance ToJSON RE where toJSON re = object [ "reString" .= encodeToText (reString re) - , "reCaseSensitive" .= reCaseSensitive re ] + , "reCaseSensitive" .= reCaseSensitive re + , "reMinimal" .= reMinimal re ] instance FromJSON RE where parseJSON = withObject "RE" $ \v -> RE <$> ((v .: "reString") >>= decodeFromText) <*> v .: "reCaseSensitive" + <*> v .:? "reMinimal" .!= False -- functions to marshall bytestrings to text @@ -109,5 +125,7 @@ decodeFromText :: (Monad m, MonadFail m) => Text.Text -> m BS.ByteString decodeFromText = either fail return . Base64.decode . TE.encodeUtf8 -compileRE :: RE -> Either String Regex +-- | The compiled regex and its capturing groups (cached in the +-- 'RE' by the smart constructor). +compileRE :: RE -> Either String (Regex, M.IntMap Regex) compileRE = _reCompiled diff --git a/src/Skylighting/Tokenizer.hs b/src/Skylighting/Tokenizer.hs --- a/src/Skylighting/Tokenizer.hs +++ b/src/Skylighting/Tokenizer.hs @@ -34,7 +34,6 @@ import Debug.Trace import Skylighting.Regex import Skylighting.Types -import Skylighting.Parser (resolveKeywords) import Data.List.NonEmpty (NonEmpty((:|)), (<|), toList) #if !MIN_VERSION_base(4,11,0) import Data.Semigroup @@ -56,6 +55,9 @@ , column :: Int , lineContinuation :: Bool , firstNonspaceColumn :: Maybe Int + , loopCounter :: Int + -- ^ number of consecutive rule-matching iterations without + -- consuming any input; used to guard against endless loops } -- | Configuration options for 'tokenize'. @@ -137,20 +139,23 @@ z -> z) -- | Tokenize some text using 'Syntax'. +-- Note that the syntax definitions are assumed to have their +-- keyword lists already resolved into word sets (as is the case +-- for the bundled syntax definitions and for definitions loaded +-- with the functions in Skylighting.Loader). If you construct a +-- syntax map yourself from syntaxes parsed with +-- 'Skylighting.Parser.parseSyntaxDefinition', apply +-- 'Skylighting.Parser.resolveKeywords' to each syntax first. tokenize :: TokenizerConfig -> Syntax -> Text -> Either String [SourceLine] tokenize config syntax inp = eitherStack >>= \(!stack) -> - case runTokenizerM action - config{ syntaxMap = Map.map (resolveKeywords (syntaxMap config)) - (syntaxMap config) } - (startingState stack) of + case runTokenizerM action config (startingState stack) of (_, Success ls) -> Right ls (_, Error e) -> Left e (_, Failure) -> Left "Could not tokenize code" where action = mapM tokenizeLine (zip (BS.lines (encodeUtf8 inp)) [1..]) - eitherStack = case lookupContext (sStartingContext syntax) - (resolveKeywords (syntaxMap config) syntax) of + eitherStack = case lookupContext (sStartingContext syntax) syntax of Just c -> Right $ ContextStack ((c, Captures mempty) :| []) Nothing -> Left "No starting context specified" startingState stack = @@ -162,6 +167,7 @@ , column = 0 , lineContinuation = False , firstNonspaceColumn = Nothing + , loopCounter = 0 } info :: String -> TokenizerM () @@ -240,35 +246,86 @@ tokenizeLine :: (ByteString, Int) -> TokenizerM [Token] tokenizeLine (!ln, !linenum) = do - modify $ \st -> st{ input = ln, endline = BS.null ln, prevChar = '\n' } - cur <- currentContext - lineCont <- gets lineContinuation - if lineCont - then modify $ \st -> st{ lineContinuation = False } - else do - let !mbFirstNonspace = BS.findIndex (not . isSpace) $! ln - modify $ \st -> st{ column = 0 - , firstNonspaceColumn = mbFirstNonspace } - doContextSwitches (cLineBeginContext cur) + -- column and firstNonspaceColumn restart on every physical line; + -- a line continuation only suppresses the previous line's + -- lineEndContext (KDE abstracthighlighter.cpp, highlightLine). + let !mbFirstNonspace = BS.findIndex (not . isSpace) $! ln + modify $ \st -> st{ input = ln + , endline = BS.null ln + , prevChar = '\n' + , lineContinuation = False + , column = 0 + , firstNonspaceColumn = mbFirstNonspace + , loopCounter = 0 } if BS.null ln - then doContextSwitches (cLineEmptyContext cur) - else doContextSwitches (cLineBeginContext cur) - ts <- normalizeHighlighting . catMaybes <$> many getToken - eol <- gets endline - if eol then do - currentContext >>= checkLineEnd - return ts - else do -- fail if we haven't consumed whole line - col <- gets column - throwError $ "Could not match anything at line " ++ - show linenum ++ " column " ++ show col + -- Empty lines get only the lineEmptyContext switches (which + -- default to the lineEndContext switches), applied for + -- successive top contexts until #stay; the lineEndContext is + -- not applied separately (KDE abstracthighlighter.cpp). + handleEmptyLine loopLimit + return [] + else do + ts <- normalizeHighlighting . catMaybes <$> many getToken + eol <- gets endline + if eol + then do + checkLineEnd + return ts + else do -- fail if we haven't consumed whole line + col <- gets column + throwError $ "Could not match anything at line " ++ + show linenum ++ " column " ++ show col +-- | Limit on iterations that make no progress, to avoid endless +-- loops with broken syntax definitions, as in KDE's +-- abstracthighlighter.cpp. +loopLimit :: Int +loopLimit = 1024 + +-- | Apply line-empty context switches for successive top contexts +-- until a context with no switches (#stay) is on top, guarding +-- against endless loops (KDE abstracthighlighter.cpp, highlightLine). +handleEmptyLine :: Int -> TokenizerM () +handleEmptyLine counter = do + cur <- currentContext + case cLineEmptyContext cur of + [] -> return () -- #stay + switches + | counter <= 0 -> info $ "Endless switch context transitions " ++ + "for line empty context, aborting highlighting of line." + | otherwise -> do + before <- gets (fmap fst . unContextStack . contextStack) + doContextSwitches switches + after <- gets (fmap fst . unContextStack . contextStack) + -- if the stack is unchanged (e.g. #pop of the initial + -- context), stop: + when (before /= after) $ handleEmptyLine (counter - 1) + getToken :: TokenizerM (Maybe Token) getToken = do inp <- gets input gets endline >>= guard . not !context <- currentContext + counter <- gets loopCounter + if counter > loopLimit + -- too many iterations without consuming input (e.g. a cycle of + -- context switches): abort highlighting of this line, giving + -- the rest of it the context's attribute, as in KDE's + -- abstracthighlighter.cpp. + then do + info "Endless state transitions, aborting highlighting of line." + t <- decodeBS inp + modify $ \st -> st{ input = BS.empty + , endline = True + , prevChar = Text.last t + , column = column st + Text.length t } + return $ Just (cAttribute context, t) + else getToken' context inp counter + +getToken' :: Context -> ByteString -> Int -> TokenizerM (Maybe Token) +getToken' context inp counter = do + modify $ \st -> st{ loopCounter = counter + 1 } msum (map (\r -> tryRule r inp) (cRules context)) <|> case cFallthroughContext context of [] | cFallthrough context -> Nothing <$ doContextSwitches [Pop] @@ -289,7 +346,8 @@ modify $ \st -> st{ input = rest, endline = BS.null rest, prevChar = Text.last t, - column = column st + numchars } + column = column st + numchars, + loopCounter = 0 } -- input was consumed: progress return t tryRule :: Rule -> ByteString -> TokenizerM (Maybe Token) @@ -313,6 +371,7 @@ modify $ \st -> st{ captures = Captures mempty } let attr = rAttribute rule + let delims = rWordDelimiters rule mbtok <- case rMatcher rule of DetectChar c -> withAttr attr $ detectChar (rDynamic rule) c inp Detect2Chars c d -> withAttr attr $ @@ -320,12 +379,12 @@ AnyChar cs -> withAttr attr $ anyChar cs inp RangeDetect c d -> withAttr attr $ rangeDetect c d inp RegExpr re -> withAttr attr $ regExpr (rDynamic rule) re inp - Int -> withAttr attr $ parseInt inp - HlCOct -> withAttr attr $ parseOct inp - HlCHex -> withAttr attr $ parseHex inp + Int -> withAttr attr $ parseInt delims inp + HlCOct -> withAttr attr $ parseOct delims inp + HlCHex -> withAttr attr $ parseHex delims inp HlCStringChar -> withAttr attr $ parseCStringChar inp HlCChar -> withAttr attr $ parseCChar inp - Float -> withAttr attr $ parseFloat inp + Float -> withAttr attr $ parseFloat delims inp Keyword _kwattr (Left listname) -> throwError $ "Keyword with unresolved list " <> show listname Keyword kwattr (Right kws) -> @@ -335,8 +394,8 @@ s inp WordDetect s -> withAttr attr $ wordDetect (rCaseSensitive rule) - (rWeakDeliminators rule) s inp - LineContinue -> withAttr attr $ lineContinue inp + delims s inp + LineContinue c -> withAttr attr $ lineContinue c inp DetectSpaces -> withAttr attr $ detectSpaces inp DetectIdentifier -> withAttr attr $ detectIdentifier inp IncludeRules cname -> includeRules @@ -350,17 +409,23 @@ Nothing -> return Nothing Just (tt, s) | rLookahead rule -> do - (oldinput, oldendline, oldprevChar, oldColumn) <- + (oldinput, oldendline, oldprevChar, oldColumn, + oldLoopCounter) <- case oldstate of Nothing -> throwError "oldstate not saved with lookahead rule" Just st -> return (input st, endline st, - prevChar st, column st) + prevChar st, column st, + loopCounter st) + -- restore loopCounter too: a lookahead match makes + -- no progress, so it must not reset the + -- endless-loop guard (takeChars reset it): modify $ \st -> st{ input = oldinput , endline = oldendline , prevChar = oldprevChar - , column = oldColumn } + , column = oldColumn + , loopCounter = oldLoopCounter } return Nothing | otherwise -> do case mbchildren of @@ -381,24 +446,34 @@ then return Nothing else return $ Just (tt, res) -wordDetect :: Bool -> Set.Set Char -> Text -> ByteString -> TokenizerM Text -wordDetect caseSensitive weakDelims s inp = do - -- Removed the next line because KDE seems to allow - -- \n Set.Set Char -> Text -> ByteString + -> TokenizerM Text +wordDetect caseSensitive delims s inp = do + t <- if caseSensitive + then do -- fast path: compare bytes without decoding + guard $ encodeUtf8 s `BS.isPrefixOf` inp + return s + else do + t <- decodeBS $ UTF8.take (Text.length s) inp + -- we assume here that the case fold will not change length, + -- which is safe for ASCII keywords and the like... + guard $ mk s == mk t + return t guard $ not (Text.null t) + let isDelim = (`Set.member` delims) + -- KDE requires a word delimiter (or start of line) before the + -- word, or as its first character (this is why \n '\n' Just (x,_) -> x - guard $ isWordBoundary weakDelims c d + -- ... and a word delimiter (or end of line) after the word, or + -- as its last character: + guard $ isDelim d || isDelim c takeChars (Text.length t) stringDetect :: Bool -> Bool -> Text -> ByteString -> TokenizerM Text @@ -409,12 +484,14 @@ info $ "Dynamic string: " ++ show dynStr return dynStr else return s - t <- decodeBS $ UTF8.take (Text.length s') inp - -- we assume here that the case fold will not change length, - -- which is safe for ASCII keywords and the like... - guard $ if caseSensitive - then s' == t - else mk s' == mk t + if caseSensitive + then -- fast path: compare bytes without decoding + guard $ encodeUtf8 s' `BS.isPrefixOf` inp + else do + t <- decodeBS $ UTF8.take (Text.length s') inp + -- we assume here that the case fold will not change length, + -- which is safe for ASCII keywords and the like... + guard $ mk s' == mk t takeChars (Text.length s') subDynamicText :: Text -> TokenizerM Text @@ -463,17 +540,29 @@ (Just (NormalTok, xs), Just attr) -> Just (attr, xs) _ -> mbtok -checkLineEnd :: Context -> TokenizerM () -checkLineEnd c = do - unless (null (cLineEndContext c)) $ do - eol <- gets endline - info $ "checkLineEnd for " ++ show (cName c) ++ " eol = " ++ show eol ++ " cLineEndContext = " ++ show (cLineEndContext c) - when eol $ do - lineCont' <- gets lineContinuation - unless lineCont' $ do - doContextSwitches (cLineEndContext c) - c' <- currentContext - unless (c == c') $ checkLineEnd c' +-- | Apply line-end context switches for successive top contexts +-- until a context with no switches (#stay) is on top, guarding +-- against endless loops (KDE abstracthighlighter.cpp, highlightLine). +checkLineEnd :: TokenizerM () +checkLineEnd = do + lineCont' <- gets lineContinuation + unless lineCont' $ go loopLimit + where + go counter = do + c <- currentContext + unless (null (cLineEndContext c)) $ + if counter <= 0 + then info $ "Endless switch context transitions " ++ + "for line end context, aborting highlighting of line." + else do + info $ "checkLineEnd for " ++ show (cName c) ++ + " cLineEndContext = " ++ show (cLineEndContext c) + before <- gets (fmap fst . unContextStack . contextStack) + doContextSwitches (cLineEndContext c) + after <- gets (fmap fst . unContextStack . contextStack) + -- if the stack is unchanged (e.g. #pop of the initial + -- context), stop: + when (before /= after) $ go (counter - 1) detectChar :: Bool -> Char -> ByteString -> TokenizerM Text detectChar dynamic c inp = do @@ -500,9 +589,12 @@ d' <- if dynamic && d >= '0' && d <= '9' then getDynamicChar d else return d - if (encodeUtf8 (Text.pack [c',d'])) `BS.isPrefixOf` inp - then takeChars 2 - else mzero + case UTF8.uncons inp of + Just (x, rest) | x == c' -> + case UTF8.uncons rest of + Just (y, _) | y == d' -> takeChars 2 + _ -> mzero + _ -> mzero rangeDetect :: Char -> Char -> ByteString -> TokenizerM Text rangeDetect c d inp = do @@ -534,9 +626,9 @@ not (isAlphaNum d || d == '_')) t) _ -> mzero -lineContinue :: ByteString -> TokenizerM Text -lineContinue inp = do - if inp == "\\" +lineContinue :: Char -> ByteString -> TokenizerM Text +lineContinue c inp = do + if inp == UTF8.fromString [c] then do modify $ \st -> st{ lineContinuation = True } takeChars 1 @@ -552,16 +644,20 @@ regExpr dynamic re inp = do -- return $! traceShowId $! (reStr, inp) let reStr = reString re - when (BS.take 2 reStr == "\\b") $ wordBoundary mempty inp - regex <- case compileRE re of + when (BS.take 2 reStr == "\\b") $ wordBoundary inp + (regex, groups) <- case compileRE re of Right r -> return r Left e -> throwError $ "Error compiling regex " ++ UTF8.toString reStr ++ ": " ++ e - regex' <- if dynamic - then subDynamic regex - else return regex - case matchRegex regex' inp of + mbmatch <- if dynamic + then do + regex' <- subDynamic regex + -- the capturing groups have to be recomputed after + -- dynamic substitution (matchRegex does this): + return $ matchRegex regex' inp + else return $ matchRegexWithGroups groups regex inp + case mbmatch of Just (matchedBytes, capts) -> do unless (null capts) $ modify $ \st -> st{ captures = Captures $ @@ -572,19 +668,27 @@ toSlice :: ByteString -> (Int, Int) -> ByteString toSlice bs (off, len) = BS.take len $ BS.drop off bs -wordBoundary :: Set.Set Char -> ByteString -> TokenizerM () -wordBoundary weakDelims inp = do +wordBoundary :: ByteString -> TokenizerM () +wordBoundary inp = do case UTF8.uncons inp of Nothing -> return () Just (d, _) -> do c <- gets prevChar - guard $ isWordBoundary weakDelims c d + guard $ isWordBoundary c d -isWordBoundary :: Set.Set Char -> Char -> Char -> Bool -isWordBoundary weakDelims c d = - (isWordChar c || c `Set.member` weakDelims) /= - (isWordChar d || d `Set.member` weakDelims) +isWordBoundary :: Char -> Char -> Bool +isWordBoundary c d = isWordChar c /= isWordChar d +-- In KDE, Int, Float, HlCOct, and HlCHex rules match only if the +-- preceding character is a word delimiter (or we are at the start +-- of the line, which we detect via prevChar == '\n', since '\n' +-- is always a delimiter). Nothing is required of the character +-- following the match. +precededByWordDelim :: Set.Set Char -> TokenizerM () +precededByWordDelim delims = do + c <- gets prevChar + guard $ c `Set.member` delims + decodeBS :: ByteString -> TokenizerM Text decodeBS bs = case decodeUtf8' bs of Left _ -> throwError ("ByteString " ++ @@ -661,74 +765,65 @@ pCStringChar <|> () <$ A.satisfy (\c -> c /= '\'' && c /= '\\') () <$ A.char '\'' -parseInt :: ByteString -> TokenizerM Text -parseInt inp = do - wordBoundary mempty inp - case A.parseOnly (A.match (pHex <|> pOct <|> pDec)) inp of +-- Like KDE's Int rule: a sequence of one or more decimal digits. +-- No sign, and no hex or octal forms. +parseInt :: Set.Set Char -> ByteString -> TokenizerM Text +parseInt delims inp = do + precededByWordDelim delims + case A.parseOnly (A.match (void $ A.takeWhile1 (A.inClass "0-9"))) inp of Left _ -> mzero Right (r,_) -> takeChars (BS.length r) -- assumes ascii -pDec :: A.Parser () -pDec = do - mbMinus - void $ A.takeWhile1 (A.inClass "0-9") - -parseOct :: ByteString -> TokenizerM Text -parseOct inp = do - wordBoundary mempty inp - case A.parseOnly (A.match pHex) inp of +-- Like KDE's HlCOct rule: a C-style octal, 0 followed by one or +-- more octal digits. No sign, and no "0o" prefix. +parseOct :: Set.Set Char -> ByteString -> TokenizerM Text +parseOct delims inp = do + precededByWordDelim delims + case A.parseOnly (A.match pOct) inp of Left _ -> mzero Right (r,_) -> takeChars (BS.length r) -- assumes ascii pOct :: A.Parser () pOct = do - mbMinus _ <- A.char '0' - _ <- A.satisfy (A.inClass "Oo") _ <- A.takeWhile1 (A.inClass "0-7") return () -parseHex :: ByteString -> TokenizerM Text -parseHex inp = do - wordBoundary mempty inp +-- Like KDE's HlCHex rule: 0x or 0X followed by one or more hex +-- digits. No sign. +parseHex :: Set.Set Char -> ByteString -> TokenizerM Text +parseHex delims inp = do + precededByWordDelim delims case A.parseOnly (A.match pHex) inp of Left _ -> mzero Right (r,_) -> takeChars (BS.length r) -- assumes ascii pHex :: A.Parser () pHex = do - mbMinus _ <- A.char '0' _ <- A.satisfy (A.inClass "Xx") _ <- A.takeWhile1 (A.inClass "0-9a-fA-F") return () -mbMinus :: A.Parser () -mbMinus = (() <$ A.char '-') <|> return () - -mbPlusMinus :: A.Parser () -mbPlusMinus = () <$ A.satisfy (A.inClass "+-") <|> return () - -parseFloat :: ByteString -> TokenizerM Text -parseFloat inp = do - wordBoundary mempty inp +-- Like KDE's Float rule: optional digits, a mandatory '.', and +-- optional digits (at least one digit is required on one side of +-- the dot), followed by an optional exponent (e or E, an optional +-- sign, and digits); if the exponent is not complete, the match +-- ends before the e/E. No leading sign, and "5e2" is not a Float. +parseFloat :: Set.Set Char -> ByteString -> TokenizerM Text +parseFloat delims inp = do + precededByWordDelim delims case A.parseOnly (A.match pFloat) inp of Left _ -> mzero Right (r,_) -> takeChars (BS.length r) -- assumes all ascii where pFloat :: A.Parser () pFloat = do - let digits = A.takeWhile1 (A.inClass "0-9") - mbPlusMinus - before <- A.option False $ True <$ digits - dot <- A.option False $ True <$ A.satisfy (A.inClass ".") - after <- A.option False $ True <$ digits - e <- A.option False $ True <$ (A.satisfy (A.inClass "Ee") >> - mbPlusMinus >> digits) - mbnext <- A.peekChar - case mbnext of - Nothing -> return () - Just c -> guard (not $ A.inClass "." c) - guard $ (before && not dot && e) -- 5e2 - || (before && dot && (after || not e)) -- 5.2e2 or 5.2 or 5. - || (not before && dot && after) -- .23 or .23e2 + before <- A.takeWhile (A.inClass "0-9") + _ <- A.char '.' + after <- A.takeWhile (A.inClass "0-9") + guard $ not (BS.null before && BS.null after) + A.option () $ do + _ <- A.satisfy (A.inClass "Ee") + _ <- A.option '+' (A.satisfy (A.inClass "+-")) + void $ A.takeWhile1 (A.inClass "0-9") diff --git a/src/Skylighting/Types.hs b/src/Skylighting/Types.hs --- a/src/Skylighting/Types.hs +++ b/src/Skylighting/Types.hs @@ -109,7 +109,7 @@ | HlCHex | HlCStringChar | HlCChar - | LineContinue + | LineContinue !Char | IncludeRules !ContextName | DetectSpaces | DetectIdentifier @@ -130,7 +130,7 @@ rMatcher :: !Matcher , rAttribute :: !TokenType , rIncludeAttribute :: !Bool - , rWeakDeliminators :: Set.Set Char + , rWordDelimiters :: Set.Set Char , rDynamic :: !Bool , rCaseSensitive :: !Bool , rChildren :: ![Rule] @@ -178,7 +178,6 @@ , cAttribute :: !TokenType , cLineEmptyContext :: ![ContextSwitch] , cLineEndContext :: ![ContextSwitch] - , cLineBeginContext :: ![ContextSwitch] , cFallthrough :: !Bool , cFallthroughContext :: ![ContextSwitch] , cDynamic :: !Bool diff --git a/test/expected/abc.haskell.native b/test/expected/abc.haskell.native --- a/test/expected/abc.haskell.native +++ b/test/expected/abc.haskell.native @@ -103,7 +103,9 @@ , ( OtherTok , "=" ) , ( NormalTok , " " ) , ( FunctionTok , "mapM_" ) - , ( NormalTok , " (\\w " ) + , ( NormalTok , " (" ) + , ( OperatorTok , "\\" ) + , ( NormalTok , "w " ) , ( OtherTok , "->" ) , ( NormalTok , " " ) , ( FunctionTok , "print" ) diff --git a/test/test-skylighting.hs b/test/test-skylighting.hs --- a/test/test-skylighting.hs +++ b/test/test-skylighting.hs @@ -103,12 +103,14 @@ testCase ("regex " <> (Text.unpack $ TE.decodeUtf8 regex) <> " in " <> sFilename syn) - $ case compileRegex True regex of + $ case compileRegex True False regex of Right _ -> assertBool "regex does not compile" True Left e -> assertFailure ("regex does not compile: " <> show e)) $ getRegexesFromSyntax syn)) syntaxes , testGroup "Regex module" $ map regexTest regexTests + , testGroup "Regex module compile errors" $ + map regexErrorTest regexErrorTests , testGroup "Regression tests" $ let perl = maybe (error "could not find Perl syntax") id (lookupSyntax "Perl" sMap) @@ -119,7 +121,21 @@ bash = maybe (error "could not find bash syntax") id (lookupSyntax "bash" sMap) c = maybe (error "could not find C syntax") id - (lookupSyntax "c" sMap) in + (lookupSyntax "c" sMap) + dosbat = maybe (error "could not find MS-DOS Batch syntax") id + (lookupSyntax "MS-DOS Batch" sMap) + cmake = maybe (error "could not find CMake syntax") id + (lookupSyntax "CMake" sMap) + lua = maybe (error "could not find Lua syntax") id + (lookupSyntax "Lua" sMap) + awk = maybe (error "could not find AWK syntax") id + (lookupSyntax "AWK" sMap) + glsl = maybe (error "could not find GLSL syntax") id + (lookupSyntax "GLSL" sMap) + makefile = maybe (error "could not find Makefile syntax") id + (lookupSyntax "Makefile" sMap) + markdown = maybe (error "could not find Markdown syntax") id + (lookupSyntax "Markdown" sMap) in [ testCase "perl NUL case" $ Right [[(OtherTok,"s\NULb\NUL") ,(StringTok,"c") @@ -195,6 +211,40 @@ @=? tokenize defConfig bash "f() {\n echo > f\n}\n" + , testCase "LineContinue with char attribute (dosbat ^)" $ Right + [ [ ( BuiltInTok , "echo" ) + , ( NormalTok , " foo " ) + , ( SpecialCharTok , "^" ) ] + , [ ( NormalTok , "bar" ) ] ] + @=? tokenize defConfig dosbat "echo foo ^\nbar" + + , testCase "keyword rule insensitive attribute (cmake)" $ Right + [ [ ( ControlFlowTok , "if" ) + , ( NormalTok , "(" ) + , ( OtherTok , "YES" ) + , ( NormalTok , ")" ) ] + , [ ( ControlFlowTok , "if" ) + , ( NormalTok , "(" ) + , ( OtherTok , "yes" ) + , ( NormalTok , ")" ) ] ] + @=? tokenize defConfig cmake "if(YES)\nif(yes)" + + , testCase "keyword rule additionalDeliminator attribute (lua)" $ Right + [ [ ( VariableTok , "a" ) + , ( NormalTok , " " ) + , ( OperatorTok , "=" ) + , ( NormalTok , " " ) + , ( KeywordTok , "nil" ) + , ( OperatorTok , "." ) + , ( VariableTok , "x" ) ] ] + @=? tokenize defConfig lua "a = nil.x" + + , testCase "Int respects general weakDeliminator (awk)" $ Right + [ [ ( NormalTok , "x" ) + , ( OperatorTok , "@" ) + , ( NormalTok , "5" ) ] ] + @=? tokenize defConfig awk "x@5" + , testCase "C floating-point literal (#174)" $ Right [ [ ( DataTypeTok , "double") , ( NormalTok , " x " ) @@ -205,9 +255,104 @@ @=? tokenize defConfig c "double x = 0.5;\n" + -- HlCOct matches C-style octals (0 followed by octal digits); + -- HlCHex matches 0x followed by hex digits: + , testCase "HlCOct and HlCHex rules (glsl)" $ Right + [ [ ( NormalTok , "x " ) + , ( OperatorTok , "=" ) + , ( NormalTok , " " ) + , ( BaseNTok , "0755" ) + , ( OperatorTok , ";" ) ] + , [ ( NormalTok , "y " ) + , ( OperatorTok , "=" ) + , ( NormalTok , " " ) + , ( BaseNTok , "0x1F" ) + , ( OperatorTok , ";" ) ] ] + @=? tokenize defConfig glsl "x = 0755;\ny = 0x1F;" + + -- As in KDE, the Float rule requires a '.', so 5e2 is matched + -- by Int (leaving e2 unmatched), and an incomplete exponent is + -- excluded from the match: + , testCase "Float rule requires dot; exponent all-or-nothing (glsl)" $ + Right + [ [ ( NormalTok , "x " ) + , ( OperatorTok , "=" ) + , ( NormalTok , " " ) + , ( DecValTok , "5" ) + , ( NormalTok , "e2" ) + , ( OperatorTok , ";" ) ] + , [ ( NormalTok , "y " ) + , ( OperatorTok , "=" ) + , ( NormalTok , " " ) + , ( FloatTok , "1.5" ) + , ( NormalTok , "e" ) + , ( OperatorTok , "+;" ) ] ] + @=? tokenize defConfig glsl "x = 5e2;\ny = 1.5e+;" + + -- As in KDE, Int and Float rules do not consume a leading sign: + , testCase "Int does not include leading minus (glsl)" $ Right + [ [ ( NormalTok , "x " ) + , ( OperatorTok , "=" ) + , ( NormalTok , " " ) + , ( OperatorTok , "-" ) + , ( DecValTok , "15" ) + , ( OperatorTok , ";" ) ] ] + @=? tokenize defConfig glsl "x = -15;" + + -- As in KDE, column and firstNonspaceColumn restart on every + -- physical line; a line continuation only suppresses the + -- previous line's lineEndContext. So the column="0" Target + -- rule applies to "bar" on the continuation line: + , testCase "column restarts after LineContinue (makefile)" $ Right + [ [ ( DecValTok , "foo " ) + , ( CharTok , "\\" ) ] + , [ ( DecValTok , "bar:" ) + , ( DataTypeTok , " baz" ) ] ] + @=? tokenize defConfig makefile "foo \\\nbar: baz" + + -- As in KDE, an empty line applies the lineEmptyContext + -- switches of successive top contexts until #stay (and + -- lineEmptyContext defaults to lineEndContext). Here the empty + -- line pops blockquote and then enters Normal Text's + -- lineEmptyContext (find-code-block), so the indented line + -- becomes a code block: + , testCase "empty line applies successive lineEmptyContexts (markdown)" $ + Right + [ [ ( AttributeTok , "> quote" ) ] + , [] + , [ ( InformationTok , " code" ) ] ] + @=? tokenize defConfig markdown "> quote\n\n code" + + -- A zero-progress cycle of context switches driven by a + -- lookahead rule must trigger the endless-loop guard (as in + -- KDE), aborting the line instead of hanging; in particular a + -- lookahead match must not reset the loop counter: + , testCase "zero-progress lookahead loop aborts line" $ + Right [ [ ( NormalTok, "xy" ) ] ] + @=? tokenize defConfig{ syntaxMap = + addSyntaxDefinition loopSyntax (syntaxMap defConfig) } + loopSyntax "xy" ] ] +-- | A syntax definition with a zero-progress loop: a lookahead rule +-- pushes a context that immediately pops back via fallthrough. +loopSyntax :: Syntax +loopSyntax = either error id $ parseSyntaxDefinitionFromText "loop.xml" $ + " " section=\"Other\" extensions=\"\">" + <> "" + <> " " lineEndContext=\"#stay\">" + <> "" + <> "" + <> " " lineEndContext=\"#stay\" fallthroughContext=\"#pop\">" + <> "" + <> "" + <> "" + <> "" + compareValues :: FilePath -> Text -> Text -> IO (Maybe String) compareValues referenceFile expected actual = if expected == actual @@ -301,7 +446,8 @@ , ("abc|ab$", "abd", Nothing) , ("[\\x50-\\x51]*", "PQR", Just ("PQ", [])) , ("[\\x{2019}]*", "\x2019PQR", Just ("\x2019", [])) - , ("(?:ab)*|a.*", "abababa", Just ("abababa", [])) + , ("(?:ab)*|a.*", "abababa", Just ("ababab", [])) + -- leftmost-first: first alternative matches, so second is never tried , ("a[b-e]*", "abcdefg", Just ("abcde", [])) , ("a[b-e\\n-]*", "abcde\nb-bcfg", Just ("abcde\nb-bc", [])) , ("^\\s+\\S+\\s+$", " abc ", Just (" abc ", [])) @@ -344,11 +490,226 @@ , ("([abc](?1)*)", "abcd", Just ("abc", [(1,"abc")])) , ("(x(?1)*)", "xxxxy", Just ("xxxx", [(1,"xxxx")])) , ("a|\\((?0)\\)", "(((a)))", Just ("(((a)))", [])) - , ("([abc](x(?1))*)", "axbxcc", Just ("axbxc", [(1,"axbxc"),(2,"xc")])) - -- note: pcre gives insetad (2, "xbxc") -- I don't understand why + , ("([abc](x(?1))*)", "axbxcc", Just ("axbxc", [(1,"axbxc"),(2,"xbxc")])) + -- group 2's last iteration is "xbxc": the recursion (?1) inside it + -- matches "bxc", and inner iterations' captures are overwritten , ("[\\p{Nd}]", "33", Just ("3", [])) , ("\\p{N}", "33", Just ("3", [])) + -- {m,n} with m > n is a compile error (see regexErrorTests; it + -- used to send the compiler into an infinite loop, and later was + -- treated as a literal) + -- lazy quantifiers in lookbehinds used to hang the matcher: + , ("ab(?<=a+?b)c", "abc", Just ("abc", [])) + , ("ab(?<=a+?)c", "abc", Nothing) + -- recursive subroutine calls that consume no input used to hang; + -- now re-entering a subroutine at the same offset just fails: + , ("x|(?R)", "x", Just ("x", [])) + , ("a|(?R)(?R)", "aa", Just ("a", [])) + -- leftmost-first: the first alternative succeeds on "a" + -- backward matching (lookbehind, \b) after multibyte characters + -- used to land inside a UTF-8 sequence: + , ("\x00e9(?<=\x00e9)x", "\x00e9x", Just ("\x00e9x", [])) + , ("\x00e9\\bx", "\x00e9x", Nothing) + , ("\x2019(?<=\x2019)x", "\x2019x", Just ("\x2019x", [])) + , ("\x00e9(?...): + , ("(?>ab|a)c", "abc", Just ("abc", [])) + , ("(?>ab|a)c", "ac", Just ("ac", [])) + , ("(?>a+)ab", "aaab", Nothing) + , ("x(?>)y", "xy", Just ("xy", [])) + , ("(?>a|ab)c", "abc", Nothing) + , ("(?>(a|ab)c)", "abc", Just ("abc", [(1,"ab")])) + -- alternation is leftmost-first, not longest-match (as in PCRE): + , ("a|ab", "ab", Just ("a", [])) + , ("(a|ab)c?", "abc", Just ("a", [(1,"a")])) + , ("(?=(a|ab))", "ab", Just ("", [(1,"a")])) + -- lazy quantifiers match as little as possible: + , ("a+?b", "aaab", Just ("aaab", [])) + , ("a*?b", "aaab", Just ("aaab", [])) + , ("(a+?)ab", "aaab", Just ("aaab", [(1,"aa")])) + -- subroutine calls to groups with multi-digit numbers: + , ("(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)(k)(l)x(?12)", "abcdefghijklxl", + Just ("abcdefghijklxl", + [(1,"a"),(2,"b"),(3,"c"),(4,"d"),(5,"e"),(6,"f"),(7,"g"), + (8,"h"),(9,"i"),(10,"j"),(11,"k"),(12,"l")])) + , ("(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)(k)(l)x(?12)", "abcdefghijklxa", Nothing) + -- numbering after (?|...) resumes after the highest group number + -- used in any alternative: + , ("(?|(a)(b)|(c))(d)\\2", "abdb", + Just ("abdb", [(1,"a"),(2,"b"),(3,"d")])) + , ("(?|(a)(b)|(c))(d)\\2", "cdd", Nothing) + -- a ] in first position in a character class is a literal, and + -- may be the start of a range: + , ("[]-a]+", "^_`", Just ("^_`", [])) + , ("[]-a]+", "b", Nothing) + , ("[]-]+", "]-]", Just ("]-]", [])) + , ("[]-]+", "^", Nothing) + , ("[^]-a]+", "bz!", Just ("bz!", [])) + , ("[^]-a]+", "^", Nothing) + , ("[]a-]+", "a]-", Just ("a]-", [])) + -- \pL is short for \p{L}; \P is the complement of \p: + , ("\\pL+", "ab\x3a3\&9", Just ("ab\x3a3", [])) + , ("\\pN", "9", Just ("9", [])) + , ("\\pN", "a", Nothing) + , ("\\PL+", "9!", Just ("9!", [])) + , ("\\P{L}+", "9!a", Just ("9!", [])) + , ("\\p{^L}+", "9!a", Just ("9!", [])) + , ("[\\pL]+", "ab9", Just ("ab", [])) + , ("[\\PL]+", "9!a", Just ("9!", [])) + , ("[\\P{N}]+", "a!9", Just ("a!", [])) + , ("[^\\PL]+", "ab9", Just ("ab", [])) + -- {,n} is a quantifier (as in PCRE 10.43+), but {,} and {b} are + -- literal: + , ("a{,2}", "aaa", Just ("aa", [])) + , ("a{,}", "a{,}", Just ("a{,}", [])) + , ("a{b}", "a{b}", Just ("a{b}", [])) + -- (?s) and (?m) are accepted (and are no-ops on our single-line + -- subjects): + , ("(?s)a.b", "axb", Just ("axb", [])) + , ("(?m)^ab", "ab", Just ("ab", [])) + , ("(?ims)ab", "AB", Just ("AB", [])) + -- (?U) makes quantifiers minimal by default and ? makes them + -- greedy (PCRE's UNGREEDY option, QRegularExpression's + -- InvertedGreedinessOption, minimal="1" in KDE syntax files): + , ("(?U)a+", "aaa", Just ("a", [])) + , ("(?U)a+?", "aaa", Just ("aaa", [])) + , ("(?U)a*b", "aabb", Just ("aab", [])) + , ("(?U)a?", "a", Just ("", [])) + , ("(?U)a{2,4}", "aaaaa", Just ("aa", [])) + , ("(?U)a{2,4}?", "aaaaa", Just ("aaaa", [])) + , ("(?U)a{2}", "aaa", Just ("aa", [])) + , ("(?U)a*+b", "aab", Just ("aab", [])) -- possessive is unaffected + , ("(?U)<(.*)>", "", Just ("", [(1, "x")])) + , ("(?U:a+)a", "aaa", Just ("aa", [])) + , ("((?U)a+)(a+)", "aaaa", Just ("aaaa", [(1, "a"), (2, "aaa")])) + , ("(?iU)ab+", "ABBB", Just ("AB", [])) + , ("(?U)(?-U)a+", "aaa", Just ("aaa", [])) ] + +-- these should fail to compile, as they do in PCRE ("quantifier does +-- not follow a repeatable item", "numbers out of order in {} +-- quantifier", "number too big in {} quantifier", or an unsupported +-- inline flag): +regexErrorTests :: [String] +regexErrorTests = + [ "{2}" + , "({2})" + , "a|{2}" + , "a{2}{3}" + , "a+{2}" + , "{,2}" + , "a{3,1}" + , "x{70000}" + , "a{2,70000}" + , "^{2}" + , "^*a" + , "a$*" + , "\\b+a" + , "(?x)a b" + , "(?n)(a)b" + , "(?u)a" + ] + +regexErrorTest :: String -> TestTree +regexErrorTest re = + testCase ("/" ++ re ++ "/") $ + case compileRegex True False (TE.encodeUtf8 (Text.pack re)) of + Left _ -> return () + Right _ -> assertFailure "regex compiled, but an error was expected" vividize :: Diff Text -> Text diff --git a/xml/abnf.xml b/xml/abnf.xml new file mode 100644 --- /dev/null +++ b/xml/abnf.xml @@ -0,0 +1,175 @@ + + +]> + + + + + + ALPHA + BIT + CHAR + CR + CRLF + CTL + DIGIT + DQUOTE + HEXDIG + HTAB + LF + LWSP + OCTET + SP + VCHAR + WSP + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/xml/asciidoc.xml b/xml/asciidoc.xml new file mode 100644 --- /dev/null +++ b/xml/asciidoc.xml @@ -0,0 +1,640 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +]> + + + + anchor + btn + footnote + footnoteref + icon + image + indexterm + indexterm2 + kbd + menu + pass + toc + xref + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/xml/bash.xml b/xml/bash.xml --- a/xml/bash.xml +++ b/xml/bash.xml @@ -71,11 +71,11 @@ + - + @@ -1227,7 +1229,7 @@ - + @@ -1235,9 +1237,19 @@ - + + + + + + + + + + + @@ -1276,15 +1288,15 @@ - + - + - + @@ -1934,7 +1946,7 @@ - + diff --git a/xml/cabal.xml b/xml/cabal.xml new file mode 100644 --- /dev/null +++ b/xml/cabal.xml @@ -0,0 +1,398 @@ + + + +]> + + + + + if + elif + else + + + + os + arche + impl + flag + + + + executable + library + benchmark + test-suite + source-repository + flag + foreign-library + custom-setup + common + + + + True + False + + + + asm-options + asm-sources + author + autogen-includes + autogen-modules + branch + bug-reports + build-depends + build-tool-depends + build-tools + build-type + buildable + c-sources + cabal-version + category + cc-options + cmm-options + cmm-sources + copyright + cpp-options + cxx-options + cxx-sources + data-dir + data-files + default-extensions + default-language + default + description + executable + exposed-modules + exposed + extensions + extra-bundled-libraries + extra-doc-files + extra-dynamic-library-flavours + extra-framework-dirs + extra-ghci-libraries + extra-lib-dirs-static + extra-lib-dirs + extra-libraries-static + extra-libraries + extra-library-flavours + extra-source-files + extra-tmp-files + frameworks + ghc-options + ghc-prof-options + ghc-shared-options + ghcjs-options + ghcjs-prof-options + ghcjs-shared-options + homepage + hs-source-dir + hs-source-dirs + hugs-options + import + include-dirs + includes + install-includes + js-sources + ld-options + lib-version-info + lib-version-linux + license-file + license + location + main-is + maintainer + manual + mixins + mod-def-file + module + name + nhc98-options + options + other-extensions + other-language + other-languages + other-modules + package-url + pkgconfig-depends + reexported-modules + scope + setup-depends + signatures + stability + subdir + synopsis + tag + test-module + tested-with + type + version + virtual-modules + + + + Haskell98 + Haskell2010 + + + + ghc + nhc + yhc + hugs + hbc + helium + jhc + lhc + + + + simple + custom + configure + + + + language_pragmas##Haskell + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/xml/cmake.xml b/xml/cmake.xml --- a/xml/cmake.xml +++ b/xml/cmake.xml @@ -7,15 +7,15 @@ ]> + SPDX-License-Identifier: LGPL-2.0-or-later +--> diff --git a/xml/context.xml b/xml/context.xml new file mode 100644 --- /dev/null +++ b/xml/context.xml @@ -0,0 +1,214 @@ + + + +]> + + + + \part + \chapter + \section + \subsection + \subsubsection + \subsubsubsection + \subsubsubsubsection + \title + \subject + \subsubject + \subsubsubject + \subsubsubsubject + \subsubsubsubsubject + + + \m + \math + \mathematics + \formula + + + \bTABLEhead + \bTABLEnext + \bTABLEbody + \bTABLEfoot + \bTABLE + \bTR + \bTD + + + \eTABLEhead + \eTABLEnext + \eTABLEbody + \eTABLEfoot + \eTABLE + \eTR + \eTD + + + \startMPinclusions + \startuseMPgraphic + \startreusableMPgraphic + \startstaticMPfigure + \startuniqueMPgraphic + \startMPpage + \startMPcode + \startMP + + + \stopMPinclusions + \stopuseMPgraphic + \stopreusableMPgraphic + \stopstaticMPfigure + \stopuniqueMPgraphic + \stopMPpage + \stopMPcode + \stopMP + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/xml/cpp.xml b/xml/cpp.xml --- a/xml/cpp.xml +++ b/xml/cpp.xml @@ -12,7 +12,7 @@ name="C++" alternativeNames="CPP" section="Sources" - version="19" + version="21" kateversion="5.79" indenter="cstyle" style="C++" @@ -462,9 +462,11 @@ Q_RELOCATABLE_TYPE + QML_ADDED_IN_MINOR_VERSION QML_ANONYMOUS QML_ATTACHED + QML_CONSTRUCTIBLE_VALUE QML_DECLARE_TYPE QML_DECLARE_TYPEINFO QML_ELEMENT @@ -479,6 +481,7 @@ QML_REMOVED_IN_MINOR_VERSION QML_SEQUENTIAL_CONTAINER QML_SINGLETON + QML_STRUCTURED_VALUE QML_UNAVAILABLE QML_UNCREATABLE QML_VALUE_TYPE @@ -658,13 +661,17 @@ Q3DBars + Q3DBarsWidgetItem Q3DCamera + Q3DGraphsWidgetItem Q3DInputHandler Q3DLight Q3DObject Q3DScatter + Q3DScatterWidgetItem Q3DScene Q3DSurface + Q3DSurfaceWidgetItem Q3DTheme QAbstract3DAxis QAbstract3DGraph @@ -686,6 +693,7 @@ QAbstractClipBlendNode QAbstractDataProxy QAbstractEventDispatcher + QAbstractEventDispatcherV2 QAbstractExtensionFactory QAbstractExtensionManager QAbstractFileIconProvider @@ -737,8 +745,11 @@ QAccelerometer QAccelerometerFilter QAccelerometerReading + QAccessibilityHints QAccessible QAccessibleActionInterface + QAccessibleAnnouncementEvent + QAccessibleAttributesInterface QAccessibleEditableTextInterface QAccessibleEvent QAccessibleInterface @@ -810,6 +821,8 @@ QAtomicScopedValueRollback QAttribute QAudioBuffer + QAudioBufferInput + QAudioBufferOutput QAudioDecoder QAudioDecoderControl QAudioDevice @@ -855,6 +868,7 @@ QBarDataItem QBarDataProxy QBarLegendMarker + QBarModelMapper QBarSeries QBarSet QBaseIterator @@ -963,6 +977,7 @@ QCheckBox QChildEvent QChildWindowEvent + QChronoTimer QClearBuffers QClipAnimator QClipBlendNodeCreatedChangeBase @@ -1089,6 +1104,7 @@ QDir QDirectionalLight QDirIterator + QDirListing QDirModel QDispatchCompute QDistanceFilter @@ -1101,6 +1117,7 @@ QDnsMailExchangeRecord QDnsServiceRecord QDnsTextRecord + QDnsTlsAssociationRecord QDockWidget QDomAttr QDomCDATASection @@ -1177,7 +1194,10 @@ QFontInfo QFontMetrics QFontMetricsF + QFontVariableAxis QFormBuilder + QFormDataBuilder + QFormDataPartBuilder QFormLayout QForwardRenderer QFrame @@ -1295,21 +1315,26 @@ QGraphicsView QGraphicsWidget QGraphTheme + QGraphsTheme QGregorianCalendar QGridLayout QGroupBox + QGrpcBidiStream QGrpcBidirStream QGrpcCallOptions QGrpcCallReply QGrpcChannel QGrpcChannelOperation QGrpcChannelOptions + QGrpcClientBase QGrpcClientInterceptor QGrpcClientInterceptorManager QGrpcClientStream QGrpcHttp2Channel QGrpcInterceptorContinuation QGrpcOperation + QGrpcOperationContext + QGrpcSerializationFormat QGrpcServerStream QGrpcStatus QGuiApplication @@ -1338,6 +1363,7 @@ QHelpIndexWidget QHelpLink QHelpSearchEngine + QHelpSearchEngineCore QHelpSearchQuery QHelpSearchQueryWidget QHelpSearchResult @@ -1361,11 +1387,13 @@ QHttpMultiPart QHttpPart QHttpServer + QHttpServerConfiguration QHttpServerRequest QHttpServerResponder QHttpServerResponse QHttpServerRouter QHttpServerRouterRule + QHttpServerWebSocketUpgradeResponse QHumidityFilter QHumidityReading QHumiditySensor @@ -1416,6 +1444,8 @@ QIterable QIterator QJalaliCalendar + QJniArray + QJniArrayBase QJniEnvironment QJniObject QJoint @@ -1453,6 +1483,7 @@ QLayoutItem QLCDNumber QLegend + QLegendData QLegendMarker QLEInteger QLerpClipBlend @@ -1559,6 +1590,7 @@ QMessageLogContext QMessageLogger QMetaClassInfo + QMetaContainer QMetaDataReaderControl QMetaDataWriterControl QMetaEnum @@ -1679,6 +1711,9 @@ QOAuth1 QOAuth1Signature QOAuth2AuthorizationCodeFlow + QOAuth2DeviceAuthorizationFlow + QOAuthHttpServerReplyHandler + QOAuthUriSchemeReplyHandler QObject QObjectBindableProperty QObjectCleanupHandler @@ -1822,6 +1857,7 @@ QPainter QPainterPath QPainterPathStroker + QPainterStateGuard QPaintEvent QPair QPalette @@ -1836,6 +1872,7 @@ QPdfDocumentRenderOptions QPdfLink QPdfLinkModel + QPdfOutputIntent QPdfPageNavigation QPdfPageNavigator QPdfPageRenderer @@ -1862,6 +1899,7 @@ QPictureFormatPlugin QPictureIO QPieLegendMarker + QPieModelMapper QPieSeries QPieSlice QPinchGesture @@ -1901,6 +1939,7 @@ QPlaneGeometryView QPlaneMesh QPlatformSurfaceEvent + QPlaybackOptions QPluginLoader QPoint QPointer @@ -1946,6 +1985,7 @@ QProtobufJsonSerializer QProtobufMessage QProtobufMessageDeleter + QProtobufRepeatedIterator QProtobufSerializer QProximityFilter QProximityReading @@ -2016,6 +2056,7 @@ QRadioTunerControl QRandomGenerator QRandomGenerator64 + QRangeModel QRasterMode QRasterPaintEngine QRasterWindow @@ -2065,6 +2106,7 @@ QRgba64 QRgbaFloat QRhi + QRhiAdapter QRhiBuffer QRhiColorAttachment QRhiCommandBuffer @@ -2098,6 +2140,7 @@ QRhiShaderResourceBinding QRhiShaderResourceBindings QRhiShaderStage + QRhiShadingRateMap QRhiStats QRhiSwapChain QRhiSwapChainHdrInfo @@ -2117,6 +2160,7 @@ QRhiVulkanCommandBufferNativeHandles QRhiVulkanInitParams QRhiVulkanNativeHandles + QRhiVulkanQueueSubmitParams QRhiVulkanRenderPassNativeHandles QRhiWidget QRomanCalendar @@ -2282,6 +2326,7 @@ QSphereMesh QSpinBox QSplashScreen + QSpline3DSeries QSplineSeries QSplitter QSplitterHandle @@ -2512,6 +2557,7 @@ QTouchDevice QTouchEvent QTouchEventSequence + QTouchEventWidgetSequence QTransform QTranslator QTransposeProxyModel @@ -2565,6 +2611,7 @@ QVideoFilterRunnable QVideoFrame QVideoFrameFormat + QVideoFrameInput QVideoProbe QVideoRendererControl QVideoSink @@ -2648,12 +2695,16 @@ QWebEngineCertificateError QWebEngineClientCertificateSelection QWebEngineClientCertificateStore + QWebEngineClientHints QWebEngineContextMenuRequest QWebEngineCookieStore QWebEngineDesktopMediaRequest QWebEngineDownloadRequest + QWebEngineExtensionInfo + QWebEngineExtensionManager QWebEngineFileSystemAccessRequest QWebEngineFindTextResult + QWebEngineFrame QWebEngineFullScreenRequest QWebEngineHistory QWebEngineHistoryItem @@ -2664,7 +2715,9 @@ QWebEngineNewWindowRequest QWebEngineNotification QWebEnginePage + QWebEnginePermission QWebEngineProfile + QWebEngineProfileBuilder QWebEngineQuotaRequest QWebEngineRegisterProtocolHandlerRequest QWebEngineScript @@ -2744,6 +2797,7 @@ QXmlStreamReader QXmlStreamWriter QXYLegendMarker + QXYModelMapper QXYSeries QByteArrayLiteral diff --git a/xml/crystal.xml b/xml/crystal.xml --- a/xml/crystal.xml +++ b/xml/crystal.xml @@ -33,7 +33,7 @@ + alias begin break case @@ -50,13 +51,18 @@ elsif end ensure + enum for + fun if in + lib next rescue return then + type + union unless until when diff --git a/xml/csv.xml b/xml/csv.xml new file mode 100644 --- /dev/null +++ b/xml/csv.xml @@ -0,0 +1,214 @@ + + +]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/xml/desktop.xml b/xml/desktop.xml new file mode 100644 --- /dev/null +++ b/xml/desktop.xml @@ -0,0 +1,34 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/xml/dot.xml b/xml/dot.xml --- a/xml/dot.xml +++ b/xml/dot.xml @@ -1,7 +1,7 @@ - + diff --git a/xml/elixir-eex.xml b/xml/elixir-eex.xml new file mode 100644 --- /dev/null +++ b/xml/elixir-eex.xml @@ -0,0 +1,60 @@ + + +]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/xml/elixir-heex.xml b/xml/elixir-heex.xml new file mode 100644 --- /dev/null +++ b/xml/elixir-heex.xml @@ -0,0 +1,248 @@ + + + + + +]> + + + + let + for + if + + + assigns + + + phx-click + phx-click-away + phx-change + phx-submit + phx-disable-with + phx-trigger-action + phx-auto-recover + phx-blur + phx-focus + phx-window-blur + phx-window-focus + phx-keydown + phx-keyup + phx-window-keydown + phx-window-keyup + phx-key + phx-viewport-top + phx-viewport-bottom + phx-update + phx-mounted + phx-remove + phx-hook + phx-connected + phx-disconnected + phx-debounce + phx-throttle + phx-track-static + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/xml/elixir.xml b/xml/elixir.xml --- a/xml/elixir.xml +++ b/xml/elixir.xml @@ -1,12 +1,14 @@ + + ]> - @@ -40,6 +42,7 @@ else if raise + reraise rescue throw try @@ -47,44 +50,42 @@ case - bc - lc + with for receive - exit after quote unquote + unquote_splicing super and not or when - xor in - inlist - inbits - + nil true false defmodule - def - defp defprotocol defimpl defrecord defstruct + defexception + defoverridable + + + def + defp defmacro defmacrop + defguard + defguardp defdelegate - defcallback - defmacrocallback - defexception - defoverridable import @@ -92,148 +93,435 @@ alias use + + __MODULE__ + __ENV__ + __DIR__ + __STACKTRACE__ + __CALLER__ + + + after_compile + after_verify + before_compile + behaviour + callback + compile + deprecated + dialyzer + external_resource + file + impl + nifs + on_definition + on_load + vsn + + + derive + enforce_keys + + + type + typep + opaque + spec + callback + macrocallback + optional_callbacks + - - - - - - - - - - - + + - - - + + + + - - - + + - + + + - - + + + + + + + + + + + + + + + + + + - + + + + - - - - - - - - - - - - - - - - - - - - - - - - + + + + - - - - - - - - + + + + + + + + + + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - + + + + - - - - + + + - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - + + - - + + + - - - + - - - - + + + + - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - + - + diff --git a/xml/haml.xml b/xml/haml.xml new file mode 100644 --- /dev/null +++ b/xml/haml.xml @@ -0,0 +1,1424 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + ._*$?:"=,;]|[0-9]+|-\w|[[:alpha:]]\w+)"> + + + + ?|(?!<))|[!>]=?|\.\.\.?|(?<! )[|]|[|](?![\t ]*$))"> + + + + + + + + + + + ?)?|>=?|=~|===?|\*\*?"> + +]> + + + + + + + and + begin + do + else + elsif + ensure + for + if + not + or + rescue + then + unless + until + when + while + + + + break + case + defined? + end + in + module + next + redo + retry + return + yield + + + + ruby-keyword1 + ruby-keyword2 + class + def + + + + access-control##Ruby + + + + attribute-definitions##Ruby + + + + definitions##Ruby + + + + pseudo-variables##Ruby + + + + kernel-methods##Ruby + + + + mixin-methods##Ruby + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/xml/haskell.xml b/xml/haskell.xml --- a/xml/haskell.xml +++ b/xml/haskell.xml @@ -1,8 +1,8 @@ + ]> - + case @@ -473,7 +473,8 @@ - + + @@ -541,11 +542,12 @@ + - - + + diff --git a/xml/idl.xml b/xml/idl.xml new file mode 100644 --- /dev/null +++ b/xml/idl.xml @@ -0,0 +1,120 @@ + + + + + + any + attribute + case + const + context + default + enum + exception + FALSE + fixed + public + in + inout + interface + module + Object + oneway + out + raises + readonly + sequence + struct + switch + TRUE + typedef + unsigned + union + + + boolean + char + double + float + long + octet + short + string + void + wchar + wstring + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/xml/j.xml b/xml/j.xml --- a/xml/j.xml +++ b/xml/j.xml @@ -1,121 +1,129 @@ - - - - + + + + - - - - - + + + + + ]> + + {{ + }} + assert. + break. + case. + catch. + catchd. + catcht. + continue. + do. + else. + elseif. + end. + for. + fcase. + if. + return. + select. + throw. + try. + while. + whilst. + - - - - + + + + + + + + - - + + - - - - - - - + + + + + + + + + + + + + + + - + + - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + - - + + diff --git a/xml/jinja.xml b/xml/jinja.xml new file mode 100644 --- /dev/null +++ b/xml/jinja.xml @@ -0,0 +1,298 @@ + + + + + + + + and + in + not + or + + + break + continue + else + elif + endfor + endif + for + if + recursive + + + autoescape + block + call + debug + do + endautoescape + endblock + endcall + endfilter + endmacro + endraw + endset + endtrans + endwith + extends + filter + macro + notrimmed + pluralize + raw + required + scoped + set + trans + trimmed + with + + + as + context + from + include + ignore + import + missing + with + without + + + false + none + true + False + None + True + + + caller + cycler + dict + joiner + kwargs + lipsum + loop + namespace + range + self + super + varargs + + + abs + attr + batch + capitalize + center + default + dictsort + escape + filesizeformat + first + float + forceescape + format + groupby + indent + int + items + join + last + length + list + lower + map + max + min + pprint + random + reject + rejectattr + replace + reverse + round + safe + select + selectattr + slice + sort + string + striptags + sum + title + tojson + trim + truncate + unique + upper + urlencode + urlize + wordcount + wordwrap + xmlattr + + + boolean + callable + defined + divisibleby + eq + escaped + even + false + filter + float + ge + gt + in + integer + iterable + le + lower + lt + mapping + ne + none + number + odd + sameas + sequence + string + test + true + undefined + upper + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/xml/jira.xml b/xml/jira.xml new file mode 100644 --- /dev/null +++ b/xml/jira.xml @@ -0,0 +1,779 @@ + + + + + + + + + + + + + + + + + + + + + + + +]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/xml/k.xml b/xml/k.xml new file mode 100644 --- /dev/null +++ b/xml/k.xml @@ -0,0 +1,424 @@ + + + + + + + +]> + + + + while + if + do + + + abs + acos + asin + atan + avg + bin + by + cos + delete + div + exec + exit + exp + from + getenv + i + in + insert + last + like + log + max + min + prd + select + setenv + sin + sqrt + ss + sum + tan + update + wavg + within + wsum + xexp + + + .Q.addmonths + .Q.addr + .Q.host + .Q.chk + .Q.cn + .Q.dd + .Q.dpft + .Q.dsftg + .Q.def + .Q.en + .Q.fc + .Q.fk + .Q.fmt + .Q.foo + .Q.fs + .Q.ft + .Q.fu + .Q.gc + .Q.hdpf + .Q.ind + .Q.j10 + .Q.x10 + .Q.j12 + .Q.x12 + .Q.k + .Q.l + .Q.opt + .Q.par + .Q.qp + .Q.qt + .Q.s + .Q.s1 + .Q.ty + .Q.v + .Q.V + .Q.view + .Q.w + .Q.M + .Q.pf + .Q.pt + .Q.PD + .Q.PV + .Q.pd + .Q.pv + .Q.pn + .Q.bv + .Q.vp + .Q.P + .Q.D + .Q.u + + + .h.br + .h.c0 + .h.c1 + .h.cd + .h.code + .h.data + .h.eb + .h.ec + .h.ed + .h.edsn + .h.es + .h.ex + .h.fram + .h.ha + .h.hb + .h.hc + .h.he + .h.hn + .h.hp + .h.hr + .h.ht + .h.hta + .h.htac + .h.htc + .h.html + .h.http + .h.hu + .h.hug + .h.hy + .h.iso8601 + .h.jx + .h.logo + .h.nbr + .h.pre + .h.text + .h.tx + .h.ty + .h.uh + .h.xd + .h.xmp + .h.xs + .h.xt + + + .o.B0 + .o.C0 + .o.Cols + .o.Columns + .o.FG + .o.Fkey + .o.Gkey + .o.Key + .o.PS + .o.Special + .o.Stats + .o.T + .o.T0 + .o.TI + .o.Tables + .o.Ts + .o.TypeInfo + .o.ex + .o.o + .o.t + + + .z.a + .z.ac + .z.b + .z.bm + .z.c + .z.exit + .z.f + .z.h + .z.i + .z.k + .z.K + .z.l + .z.n + .z.N + .z.o + .z.p + .z.P + .z.pc + .z.pg + .z.ph + .z.pi + .z.po + .z.pp + .z.ps + .z.pw + .z.q + .z.s + .z.ts + .z.u + .z.vs + .z.w + .z.W + .z.ws + .z.x + .z.z + .z.Z + .z.t + .z.T + .z.d + .z.D + .z.zd + + + .q.aj + .q.aj0 + .q.all + .q.and + .q.any + .q.asc + .q.asof + .q.attr + .q.avgs + .q.ceiling + .q.cols + .q.cor + .q.count + .q.cov + .q.cross + .q.csv + .q.cut + .q.deltas + .q.desc + .q.dev + .q.differ + .q.distinct + .q.each + .q.ej + .q.enlist + .q.eval + .q.except + .q.fby + .q.fills + .q.first + .q.fkeys + .q.flip + .q.floor + .q.get + .q.group + .q.gtime + .q.hclose + .q.hcount + .q.hdel + .q.hopen + .q.hsym + .q.iasc + .q.idesc + .q.ij + .q.inter + .q.inv + .q.key + .q.keys + .q.lj + .q.load + .q.lower + .q.lsq + .q.ltime + .q.ltrim + .q.mavg + .q.maxs + .q.mcount + .q.md5 + .q.mdev + .q.med + .q.meta + .q.mins + .q.mmax + .q.mmin + .q.mmu + .q.mod + .q.msum + .q.neg + .q.next + .q.not + .q.null + .q.or + .q.over + .q.parse + .q.peach + .q.pj + .q.plist + .q.prds + .q.prev + .q.prior + .q.rand + .q.rank + .q.ratios + .q.raze + .q.read0 + .q.read1 + .q.reciprocal + .q.reverse + .q.rload + .q.rotate + .q.rsave + .q.rtrim + .q.save + .q.scan + .q.set + .q.show + .q.signum + .q.ssr + .q.string + .q.sublist + .q.sums + .q.sv + .q.system + .q.tables + .q.til + .q.trim + .q.txf + .q.type + .q.uj + .q.ungroup + .q.union + .q.upper + .q.upsert + .q.value + .q.var + .q.view + .q.views + .q.vs + .q.where + .q.wj + .q.wj1 + .q.xasc + .q.xbar + .q.xcol + .q.xcols + .q.xdesc + .q.xgroup + .q.xkey + .q.xlog + .q.xprev + .q.xrank + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/xml/logfile.xml b/xml/logfile.xml new file mode 100644 --- /dev/null +++ b/xml/logfile.xml @@ -0,0 +1,36 @@ + + + + + + + + + +]> + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/xml/markdown.xml b/xml/markdown.xml --- a/xml/markdown.xml +++ b/xml/markdown.xml @@ -94,7 +94,7 @@ ]> - + @@ -367,6 +367,7 @@ + @@ -375,6 +376,9 @@ + + + @@ -448,7 +452,7 @@ - + @@ -519,6 +523,18 @@ + + + + + + + + + + + + diff --git a/xml/markdown.xml.patch b/xml/markdown.xml.patch deleted file mode 100644 --- a/xml/markdown.xml.patch +++ /dev/null @@ -1,23 +0,0 @@ -diff --git a/skylighting-core/xml/markdown.xml b/skylighting-core/xml/markdown.xml -index 92831ed..d2975fe 100644 ---- a/skylighting-core/xml/markdown.xml -+++ b/skylighting-core/xml/markdown.xml -@@ -339,7 +339,6 @@ - - - -- - - - -@@ -456,10 +455,6 @@ - - - -- -- -- -- - - - diff --git a/xml/mermaid.xml b/xml/mermaid.xml new file mode 100644 --- /dev/null +++ b/xml/mermaid.xml @@ -0,0 +1,4811 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + x)]|[|][\\/]|//|\\\\)|\((?!\))|\\(?![\\|]-)|/(?![/|]-))++"> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ~:"|^@"> + )|-(?!->)|(?!\b&quadrant_kw;)\w++)++"> + + + + + + {&md_syms;]++|&md_sym_no_md_no_lt;)++"> + {&md_syms;]++|&md_sym_no_md_no_lt;)++"> + + + + + + + ])|\s+(?!as($|\s)))+"> + + + + + + + + + + + + + + : #&]+|&prefix_ent_no_ent;)"> + : #&"]|&prefix_ent_no_ent;)"> + + + + + + + + + + + + + + +]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/xml/meson.xml b/xml/meson.xml new file mode 100644 --- /dev/null +++ b/xml/meson.xml @@ -0,0 +1,277 @@ + + + + + + + if + else + endif + foreach + endforeach + + + add_global_arguments + add_global_link_arguments + add_languages + add_project_arguments + add_project_link_arguments + add_test_setup + alias_target + assert + benchmark + both_libraries + build_target + configuration_data + configure_file + custom_target + declare_dependency + dependency + disabler + environment + error + executable + files + find_library + find_program + generator + get_option + get_variable + import + include_directories + install_data + install_headers + install_man + install_subdir + is_disabler + is_variable + jar + join_paths + library + message + project + run_command + run_target + set_variable + shared_library + shared_module + static_library + subdir + subdir_done + subproject + test + vcs_tag + warning + + + and + or + not + + + true + false + + + meson + + + build_machine + host_machine + target_machine + + + add_dist_script + add_install_script + add_postconf_script + backend + build_root + source_root + current_build_dir + current_source_dir + get_cross_property + get_compiler + has_exe_wrapper + install_dependency_manifest + is_cross_build + is_subproject + is_unity + override_find_program + project_version + project_license + project_name + version + + + cpu_family + cpu + system + endian + + + alignment + append + as_system + cmd_array + compiled + compiles + compute_int + contains + endswith + extract_all_objects + extract_objects + find_library + first_supported_argument + first_supported_link_argument + format + found + full_path + get_argument_syntax + get_configtool_variable + get_define + get_id + get + get_pkgconfig_variable + get_supported_arguments + get_supported_function_attributes + get_supported_link_arguments + get_unquoted + get_variable + gettext + has_argument + has_function_attribute + has_function + has_header_symbol + has + has_key + has_link_argument + has_member + has_members + has_multi_arguments + has_multi_link_arguments + has_type + include_type + is_even + is_odd + join + length + links + merge_from + name + partial_dependency + path + pkgconfig_gen + prepend + private_dir_include + process + returncode + run + set10 + set + set_quoted + sizeof + split + startswith + stderr + stdout + strip + symbols_have_underscore_prefix + to_lower + to_string + to_upper + type_name + underscorify + version_compare + version + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/xml/nginx.xml b/xml/nginx.xml new file mode 100644 --- /dev/null +++ b/xml/nginx.xml @@ -0,0 +1,934 @@ + + + +] +> + + + + + absolute_redirect + accept_mutex + accept_mutex_delay + access_log + add_after_body + add_before_body + add_header + add_trailer + addition_types + aio + aio_write + alias + allow + ancient_browser + ancient_browser_value + api + auth_basic + auth_basic_user_file + auth_delay + auth_http + auth_http_header + auth_http_pass_client_cert + auth_http_timeout + auth_jwt + auth_jwt_claim_set + auth_jwt_header_set + auth_jwt_key_cache + auth_jwt_key_file + auth_jwt_key_request + auth_jwt_leeway + auth_jwt_require + auth_jwt_type + auth_request + auth_request_set + autoindex + autoindex_exact_size + autoindex_format + autoindex_localtime + break + charset + charset_map + charset_types + chunked_transfer_encoding + client_body_buffer_size + client_body_in_file_only + client_body_in_single_buffer + client_body_temp_path + client_body_timeout + client_header_buffer_size + client_header_timeout + client_max_body_size + connect_timeout + connection_pool_size + create_full_put_path + daemon + dav_access + dav_methods + debug_connection + debug_points + default_type + deny + directio + directio_alignment + disable_symlinks + empty_gif + env + error_log + error_page + etag + events + expires + f4f + f4f_buffer_size + fastcgi_bind + fastcgi_buffer_size + fastcgi_buffering + fastcgi_buffers + fastcgi_busy_buffers_size + fastcgi_cache + fastcgi_cache_background_update + fastcgi_cache_bypass + fastcgi_cache_key + fastcgi_cache_lock + fastcgi_cache_lock_age + fastcgi_cache_lock_timeout + fastcgi_cache_max_range_offset + fastcgi_cache_methods + fastcgi_cache_min_uses + fastcgi_cache_path + fastcgi_cache_purge + fastcgi_cache_revalidate + fastcgi_cache_use_stale + fastcgi_cache_valid + fastcgi_catch_stderr + fastcgi_connect_timeout + fastcgi_force_ranges + fastcgi_hide_header + fastcgi_ignore_client_abort + fastcgi_ignore_headers + fastcgi_index + fastcgi_intercept_errors + fastcgi_keep_conn + fastcgi_limit_rate + fastcgi_max_temp_file_size + fastcgi_next_upstream + fastcgi_next_upstream_timeout + fastcgi_next_upstream_tries + fastcgi_no_cache + fastcgi_param + fastcgi_pass + fastcgi_pass_header + fastcgi_pass_request_body + fastcgi_pass_request_headers + fastcgi_read_timeout + fastcgi_request_buffering + fastcgi_send_lowat + fastcgi_send_timeout + fastcgi_socket_keepalive + fastcgi_split_path_info + fastcgi_store + fastcgi_store_access + fastcgi_temp_file_write_size + fastcgi_temp_path + flv + geo + geoip_city + geoip_country + geoip_org + geoip_proxy + geoip_proxy_recursive + google_perftools_profiles + grpc_bind + grpc_buffer_size + grpc_connect_timeout + grpc_hide_header + grpc_ignore_headers + grpc_intercept_errors + grpc_next_upstream + grpc_next_upstream_timeout + grpc_next_upstream_tries + grpc_pass + grpc_pass_header + grpc_read_timeout + grpc_send_timeout + grpc_set_header + grpc_socket_keepalive + grpc_ssl_certificate + grpc_ssl_certificate_key + grpc_ssl_ciphers + grpc_ssl_conf_command + grpc_ssl_crl + grpc_ssl_name + grpc_ssl_password_file + grpc_ssl_protocols + grpc_ssl_server_name + grpc_ssl_session_reuse + grpc_ssl_trusted_certificate + grpc_ssl_verify + grpc_ssl_verify_depth + gunzip + gunzip_buffers + gzip + gzip_buffers + gzip_comp_level + gzip_disable + gzip_http_version + gzip_min_length + gzip_proxied + gzip_static + gzip_types + gzip_vary + hash + health_check + health_check_timeout + hls + hls_buffers + hls_forward_args + hls_fragment + hls_mp4_buffer_size + hls_mp4_max_buffer_size + http + http2 + http2_body_preread_size + http2_chunk_size + http2_idle_timeout + http2_max_concurrent_pushes + http2_max_concurrent_streams + http2_max_field_size + http2_max_header_size + http2_max_requests + http2_push + http2_push_preload + http2_recv_buffer_size + http2_recv_timeout + http3 + http3_hq + http3_max_concurrent_streams + http3_stream_buffer_size + if + if_modified_since + ignore_invalid_headers + image_filter + image_filter_buffer + image_filter_interlace + image_filter_jpeg_quality + image_filter_sharpen + image_filter_transparency + image_filter_webp_quality + imap_auth + imap_capabilities + imap_client_buffer + include + index + internal + internal_redirect + ip_hash + js_access + js_body_filter + js_content + js_fetch_buffer_size + js_fetch_ciphers + js_fetch_max_response_buffer_size + js_fetch_protocols + js_fetch_timeout + js_fetch_trusted_certificate + js_fetch_verify + js_fetch_verify_depth + js_filter + js_header_filter + js_import + js_include + js_path + js_periodic + js_preload_object + js_preread + js_set + js_shared_dict_zone + js_var + keepalive + keepalive_disable + keepalive_requests + keepalive_time + keepalive_timeout + keyval + keyval_zone + large_client_header_buffers + least_conn + least_time + limit_conn + limit_conn_dry_run + limit_conn_log_level + limit_conn_status + limit_conn_zone + limit_except + limit_rate + limit_rate_after + limit_req + limit_req_dry_run + limit_req_log_level + limit_req_status + limit_req_zone + limit_zone + lingering_close + lingering_time + lingering_timeout + listen + load_module + location + lock_file + log_format + log_not_found + log_subrequest + mail + map + map_hash_bucket_size + map_hash_max_size + master_process + match + max_errors + max_ranges + memcached_bind + memcached_buffer_size + memcached_connect_timeout + memcached_gzip_flag + memcached_next_upstream + memcached_next_upstream_timeout + memcached_next_upstream_tries + memcached_pass + memcached_read_timeout + memcached_send_timeout + memcached_socket_keepalive + merge_slashes + mgmt + min_delete_depth + mirror + mirror_request_body + modern_browser + modern_browser_value + mp4 + mp4_buffer_size + mp4_limit_rate + mp4_limit_rate_after + mp4_max_buffer_size + mp4_start_key_frame + mqtt + mqtt_buffers + mqtt_preread + mqtt_rewrite_buffer_size + mqtt_set_connect + msie_padding + msie_refresh + multi_accept + ntlm + open_file_cache + open_file_cache_errors + open_file_cache_min_uses + open_file_cache_valid + open_log_file_cache + otel_exporter + otel_service_name + otel_span_attr + otel_span_name + otel_trace + otel_trace_context + output_buffers + override_charset + pass + pcre_jit + perl + perl_modules + perl_require + perl_set + pid + pop3_auth + pop3_capabilities + port_in_redirect + postpone_output + preread_buffer_size + preread_timeout + protocol + proxy_bind + proxy_buffer + proxy_buffer_size + proxy_buffering + proxy_buffers + proxy_busy_buffers_size + proxy_cache + proxy_cache_background_update + proxy_cache_bypass + proxy_cache_convert_head + proxy_cache_key + proxy_cache_lock + proxy_cache_lock_age + proxy_cache_lock_timeout + proxy_cache_max_range_offset + proxy_cache_methods + proxy_cache_min_uses + proxy_cache_path + proxy_cache_purge + proxy_cache_revalidate + proxy_cache_use_stale + proxy_cache_valid + proxy_connect_timeout + proxy_cookie_domain + proxy_cookie_flags + proxy_cookie_path + proxy_download_rate + proxy_force_ranges + proxy_half_close + proxy_headers_hash_bucket_size + proxy_headers_hash_max_size + proxy_hide_header + proxy_http_version + proxy_ignore_client_abort + proxy_ignore_headers + proxy_intercept_errors + proxy_limit_rate + proxy_max_temp_file_size + proxy_method + proxy_next_upstream + proxy_next_upstream_timeout + proxy_next_upstream_tries + proxy_no_cache + proxy_pass + proxy_pass_error_message + proxy_pass_header + proxy_pass_request_body + proxy_pass_request_headers + proxy_protocol + proxy_protocol_timeout + proxy_read_timeout + proxy_redirect + proxy_request_buffering + proxy_requests + proxy_responses + proxy_send_lowat + proxy_send_timeout + proxy_session_drop + proxy_set_body + proxy_set_header + proxy_smtp_auth + proxy_socket_keepalive + proxy_ssl + proxy_ssl_certificate + proxy_ssl_certificate_key + proxy_ssl_ciphers + proxy_ssl_conf_command + proxy_ssl_crl + proxy_ssl_name + proxy_ssl_password_file + proxy_ssl_protocols + proxy_ssl_server_name + proxy_ssl_session_reuse + proxy_ssl_trusted_certificate + proxy_ssl_verify + proxy_ssl_verify_depth + proxy_store + proxy_store_access + proxy_temp_file_write_size + proxy_temp_path + proxy_timeout + proxy_upload_rate + queue + quic_active_connection_id_limit + quic_bpf + quic_gso + quic_host_key + quic_retry + random + random_index + read_ahead + read_timeout + real_ip_header + real_ip_recursive + recursive_error_pages + referer_hash_bucket_size + referer_hash_max_size + request_pool_size + reset_timedout_connection + resolver + resolver_timeout + return + rewrite + rewrite_log + root + satisfy + scgi_bind + scgi_buffer_size + scgi_buffering + scgi_buffers + scgi_busy_buffers_size + scgi_cache + scgi_cache_background_update + scgi_cache_bypass + scgi_cache_key + scgi_cache_lock + scgi_cache_lock_age + scgi_cache_lock_timeout + scgi_cache_max_range_offset + scgi_cache_methods + scgi_cache_min_uses + scgi_cache_path + scgi_cache_purge + scgi_cache_revalidate + scgi_cache_use_stale + scgi_cache_valid + scgi_connect_timeout + scgi_force_ranges + scgi_hide_header + scgi_ignore_client_abort + scgi_ignore_headers + scgi_intercept_errors + scgi_limit_rate + scgi_max_temp_file_size + scgi_next_upstream + scgi_next_upstream_timeout + scgi_next_upstream_tries + scgi_no_cache + scgi_param + scgi_pass + scgi_pass_header + scgi_pass_request_body + scgi_pass_request_headers + scgi_read_timeout + scgi_request_buffering + scgi_send_timeout + scgi_socket_keepalive + scgi_store + scgi_store_access + scgi_temp_file_write_size + scgi_temp_path + secure_link + secure_link_md5 + secure_link_secret + send_lowat + send_timeout + sendfile + sendfile_max_chunk + server + server_name + server_name_in_redirect + server_names_hash_bucket_size + server_names_hash_max_size + server_tokens + session_log + session_log_format + session_log_zone + set + set_real_ip_from + slice + smtp_auth + smtp_capabilities + smtp_client_buffer + smtp_greeting_delay + source_charset + split_clients + ssi + ssi_last_modified + ssi_min_file_chunk + ssi_silent_errors + ssi_types + ssi_value_length + ssl + ssl_alpn + ssl_buffer_size + ssl_certificate + ssl_certificate_key + ssl_ciphers + ssl_client_certificate + ssl_conf_command + ssl_crl + ssl_dhparam + ssl_early_data + ssl_ecdh_curve + ssl_engine + ssl_handshake_timeout + ssl_name + ssl_ocsp + ssl_ocsp_cache + ssl_ocsp_responder + ssl_password_file + ssl_prefer_server_ciphers + ssl_preread + ssl_protocols + ssl_reject_handshake + ssl_server_name + ssl_session_cache + ssl_session_ticket_key + ssl_session_tickets + ssl_session_timeout + ssl_stapling + ssl_stapling_file + ssl_stapling_responder + ssl_stapling_verify + ssl_trusted_certificate + ssl_verify + ssl_verify_client + ssl_verify_depth + starttls + state + status + status_format + status_zone + sticky + sticky_cookie_insert + stream + stub_status + sub_filter + sub_filter_last_modified + sub_filter_once + sub_filter_types + subrequest_output_buffer_size + tcp_nodelay + tcp_nopush + thread_pool + timeout + timer_resolution + try_files + types + types_hash_bucket_size + types_hash_max_size + underscores_in_headers + uninitialized_variable_warn + upstream + upstream_conf + usage_report + use + user + userid + userid_domain + userid_expires + userid_flags + userid_mark + userid_name + userid_p3p + userid_path + userid_service + uuid_file + uwsgi_bind + uwsgi_buffer_size + uwsgi_buffering + uwsgi_buffers + uwsgi_busy_buffers_size + uwsgi_cache + uwsgi_cache_background_update + uwsgi_cache_bypass + uwsgi_cache_key + uwsgi_cache_lock + uwsgi_cache_lock_age + uwsgi_cache_lock_timeout + uwsgi_cache_max_range_offset + uwsgi_cache_methods + uwsgi_cache_min_uses + uwsgi_cache_path + uwsgi_cache_purge + uwsgi_cache_revalidate + uwsgi_cache_use_stale + uwsgi_cache_valid + uwsgi_connect_timeout + uwsgi_force_ranges + uwsgi_hide_header + uwsgi_ignore_client_abort + uwsgi_ignore_headers + uwsgi_intercept_errors + uwsgi_limit_rate + uwsgi_max_temp_file_size + uwsgi_modifier1 + uwsgi_modifier2 + uwsgi_next_upstream + uwsgi_next_upstream_timeout + uwsgi_next_upstream_tries + uwsgi_no_cache + uwsgi_param + uwsgi_pass + uwsgi_pass_header + uwsgi_pass_request_body + uwsgi_pass_request_headers + uwsgi_read_timeout + uwsgi_request_buffering + uwsgi_send_timeout + uwsgi_socket_keepalive + uwsgi_ssl_certificate + uwsgi_ssl_certificate_key + uwsgi_ssl_ciphers + uwsgi_ssl_conf_command + uwsgi_ssl_crl + uwsgi_ssl_name + uwsgi_ssl_password_file + uwsgi_ssl_protocols + uwsgi_ssl_server_name + uwsgi_ssl_session_reuse + uwsgi_ssl_trusted_certificate + uwsgi_ssl_verify + uwsgi_ssl_verify_depth + uwsgi_store + uwsgi_store_access + uwsgi_temp_file_write_size + uwsgi_temp_path + valid_referers + variables_hash_bucket_size + variables_hash_max_size + worker_aio_requests + worker_connections + worker_cpu_affinity + worker_priority + worker_processes + worker_rlimit_core + worker_rlimit_nofile + worker_shutdown_timeout + working_directory + xclient + xml_entities + xslt_last_modified + xslt_param + xslt_string_param + xslt_stylesheet + xslt_types + zone + zone_sync + zone_sync_buffers + zone_sync_connect_retry_interval + zone_sync_connect_timeout + zone_sync_interval + zone_sync_recv_buffer_size + zone_sync_server + zone_sync_ssl + zone_sync_ssl_certificate + zone_sync_ssl_certificate_key + zone_sync_ssl_ciphers + zone_sync_ssl_conf_command + zone_sync_ssl_crl + zone_sync_ssl_name + zone_sync_ssl_password_file + zone_sync_ssl_protocols + zone_sync_ssl_server_name + zone_sync_ssl_trusted_certificate + zone_sync_ssl_verify + zone_sync_ssl_verify_depth + zone_sync_timeout + + + + + $ancient_browser + $args + $binary_remote_addr + $body_bytes_sent + $bytes_received + $bytes_sent + $connection + $connection_requests + $connection_time + $connections_active + $connections_reading + $connections_waiting + $connections_writing + $content_length + $content_type + $date_gmt + $date_local + $document_root + $document_uri + $fastcgi_path_info + $fastcgi_script_name + $geoip_area_code + $geoip_city + $geoip_city_continent_code + $geoip_city_country_code + $geoip_city_country_code3 + $geoip_city_country_name + $geoip_country_code + $geoip_country_code3 + $geoip_country_name + $geoip_dma_code + $geoip_latitude + $geoip_longitude + $geoip_org + $geoip_postal_code + $geoip_region + $geoip_region_name + $gzip_ratio + $host + $hostname + $http2 + $http3 + $https + $invalid_referer + $is_args + $jwt_payload + $limit_conn_status + $limit_rate + $limit_req_status + $memcached_key + $modern_browser + $mqtt_preread_clientid + $mqtt_preread_username + $msec + $msie + $nginx_version + $otel_parent_id + $otel_parent_sampled + $otel_span_id + $otel_trace_id + $pid + $pipe + $protocol + $proxy_add_x_forwarded_for + $proxy_host + $proxy_port + $proxy_protocol_addr + $proxy_protocol_port + $proxy_protocol_server_addr + $proxy_protocol_server_port + $proxy_protocol_tlv_aws_vpce_id + $proxy_protocol_tlv_azure_pel_id + $proxy_protocol_tlv_gcp_conn_id + $query_string + $realip_remote_addr + $realip_remote_port + $realpath_root + $remote_addr + $remote_port + $remote_user + $request + $request_body + $request_body_file + $request_completion + $request_filename + $request_id + $request_length + $request_method + $request_time + $request_uri + $scheme + $secure_link + $secure_link_expires + $server_addr + $server_name + $server_port + $server_protocol + $session_log_binary_id + $session_log_id + $session_time + $slice_range + $ssl_alpn_protocol + $ssl_cipher + $ssl_ciphers + $ssl_client_cert + $ssl_client_escaped_cert + $ssl_client_fingerprint + $ssl_client_i_dn + $ssl_client_i_dn_legacy + $ssl_client_raw_cert + $ssl_client_s_dn + $ssl_client_s_dn_legacy + $ssl_client_serial + $ssl_client_v_end + $ssl_client_v_remain + $ssl_client_v_start + $ssl_client_verify + $ssl_curve + $ssl_curves + $ssl_early_data + $ssl_preread_alpn_protocols + $ssl_preread_protocol + $ssl_preread_server_name + $ssl_protocol + $ssl_server_name + $ssl_session_id + $ssl_session_reused + $status + $tcpinfo_rcv_space + $tcpinfo_rtt + $tcpinfo_rttvar + $tcpinfo_snd_cwnd + $time_iso8601 + $time_local + $uid_got + $uid_reset + $uid_set + $upstream_addr + $upstream_bytes_received + $upstream_bytes_sent + $upstream_cache_status + $upstream_connect_time + $upstream_first_byte_time + $upstream_header_time + $upstream_last_server_name + $upstream_queue_time + $upstream_response_length + $upstream_response_time + $upstream_session_time + $upstream_status + $uri + + + + on + off + ssl + all + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/xml/ninja.xml b/xml/ninja.xml new file mode 100644 --- /dev/null +++ b/xml/ninja.xml @@ -0,0 +1,284 @@ + + + +]> + + + + rule + build + pool + default + include + subninja + + + + console + phony + depth + + + + builddir + ninja_required_version + + + + command + console + depfile + deps + msvc_deps_prefix + description + dyndep + generator + in + in_newline + out + pool + restat + rspfile + rspfile_content + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/xml/nix.xml b/xml/nix.xml --- a/xml/nix.xml +++ b/xml/nix.xml @@ -4,7 +4,7 @@ ]> + + + + +]> + + + + + + and + as + eof + let + parse + rule + shortest + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/xml/ocamlyacc.xml b/xml/ocamlyacc.xml new file mode 100644 --- /dev/null +++ b/xml/ocamlyacc.xml @@ -0,0 +1,162 @@ + + + + + + + + + +]> + + + + + %token + %type + + + + %left + %right + %nonassoc + %start + + + + %prec + error + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/xml/orgmode.xml b/xml/orgmode.xml --- a/xml/orgmode.xml +++ b/xml/orgmode.xml @@ -33,7 +33,7 @@ - + TODO @@ -243,7 +243,7 @@ - + diff --git a/xml/perl.xml b/xml/perl.xml --- a/xml/perl.xml +++ b/xml/perl.xml @@ -42,7 +42,7 @@ Enhance tr/// and y/// support. --> - + if @@ -860,7 +860,7 @@ - + diff --git a/xml/php.xml b/xml/php.xml --- a/xml/php.xml +++ b/xml/php.xml @@ -77,7 +77,7 @@ ]> - - + @@ -11349,7 +11349,7 @@ - + diff --git a/xml/powershell.xml b/xml/powershell.xml --- a/xml/powershell.xml +++ b/xml/powershell.xml @@ -4,7 +4,7 @@ ]> - + + + + + + diff --git a/xml/python.xml b/xml/python.xml --- a/xml/python.xml +++ b/xml/python.xml @@ -78,12 +78,19 @@ - + import from as + + lazy from + lazy import class @@ -123,7 +130,7 @@ yield yield from @@ -167,6 +174,7 @@ filter float format + frozendict frozenset getattr globals @@ -235,6 +243,7 @@ __qualname__ __slots__ + SIGNAL SLOT @@ -452,13 +461,15 @@ - + + + @@ -502,9 +513,24 @@ - + + + + + + + + diff --git a/xml/q.xml b/xml/q.xml new file mode 100644 --- /dev/null +++ b/xml/q.xml @@ -0,0 +1,210 @@ + + + + + + .Q.addmonths + .Q.addr + .Q.host + .Q.chk + .Q.cn + .Q.dd + .Q.dpft + .Q.dsftg + .Q.def + .Q.en + .Q.fc + .Q.fk + .Q.fmt + .Q.fs + .Q.ft + .Q.fu + .Q.gc + .Q.hdpf + .Q.ind + .Q.j10 + .Q.x10 + .Q.j12 + .Q.x12 + .Q.k + .Q.l + .Q.opt + .Q.par + .Q.qp + .Q.qt + .Q.s + .Q.ty + .Q.v + .Q.V + .Q.view + .Q.w + .Q.M + .Q.pf + .Q.pt + .Q.PD + .Q.PV + .Q.pd + .Q.pv + .Q.pn + .Q.bv + .Q.vp + .Q.P + .Q.D + .Q.u + + + aj + aj0 + all + and + any + asc + asof + attr + avgs + ceiling + cols + cor + count + cov + cross + csv + cut + deltas + desc + dev + differ + distinct + each + ej + enlist + eval + except + fby + fills + first + fkeys + flip + floor + from + get + group + gtime + hclose + hcount + hdel + hopen + hsym + iasc + idesc + ij + inter + inv + key + keys + lj + load + lower + lsq + ltime + ltrim + mavg + maxs + mcount + md5 + mdev + med + meta + mins + mmax + mmin + mmu + mod + msum + neg + next + not + null + or + over + parse + peach + pj + plist + prds + prev + prior + rand + rank + ratios + raze + read0 + read1 + reciprocal + reverse + rload + rotate + rsave + rtrim + save + scan + set + show + signum + ssr + string + sublist + sums + sv + system + tables + til + trim + txf + type + uj + ungroup + union + update + upper + upsert + value + var + view + views + vs + where + wj + wj1 + xasc + xbar + xcol + xcols + xdesc + xgroup + xkey + xlog + xprev + xrank + + + + + + + + + + + + + + + + + + + + + + diff --git a/xml/qml.xml b/xml/qml.xml --- a/xml/qml.xml +++ b/xml/qml.xml @@ -4,7 +4,7 @@ ]> - @@ -91,6 +91,9 @@ pragma readonly required + virtual + final + override diff --git a/xml/quarto.xml b/xml/quarto.xml new file mode 100644 --- /dev/null +++ b/xml/quarto.xml @@ -0,0 +1,233 @@ + + + + + + + +]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/xml/raku.xml b/xml/raku.xml --- a/xml/raku.xml +++ b/xml/raku.xml @@ -73,7 +73,7 @@ ]> - + @@ -647,6 +647,7 @@ + @@ -847,7 +848,7 @@ - + @@ -872,7 +873,7 @@ - + @@ -895,7 +896,7 @@ - + @@ -1749,6 +1750,7 @@ + diff --git a/xml/rdoc.xml b/xml/rdoc.xml new file mode 100644 --- /dev/null +++ b/xml/rdoc.xml @@ -0,0 +1,390 @@ + + + + + + unix + windows + + + example + html + latex + text + TRUE + FALSE + + + + \docType + \encoding + \keyword + \name + + + \author + \concept + \description + \details + \format + \note + \references + \seealso + \source + \title + + + \section + + + \newcommand + \renewcommand + + + \examples + \usage + + + \alias + \Rdversion + \synopsis + \RdOpts + + + \arguments + \value + + + + + \acronym + \bold + \cite + \command + \dfn + \dQuote + \emph + \file + \linkS4class + \pkg + \sQuote + \strong + \var + + + \enc + \method + \S3method + \S4method + + + \subsection + + + \newcommand + \renewcommand + + + \enumerate + \itemize + + + \item + + + \describe + + + \if + \ifelse + + + \code + + + \email + \env + \kbd + \option + \out + \preformatted + \samp + \url + \verb + \deqn + \eqn + + + \tabular + \href + \figure + + + \link + + + \Sexpr + + + + \dontrun + \special + \v + \var + + + \method + \S3method + \S4method + + + \l + \link + + + \dontshow + \donttest + \testonly + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/xml/rmarkdown.xml b/xml/rmarkdown.xml new file mode 100644 --- /dev/null +++ b/xml/rmarkdown.xml @@ -0,0 +1,175 @@ + + + + + + + +]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/xml/rtf.xml b/xml/rtf.xml new file mode 100644 --- /dev/null +++ b/xml/rtf.xml @@ -0,0 +1,48 @@ + + ] > + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/xml/rust.xml b/xml/rust.xml --- a/xml/rust.xml +++ b/xml/rust.xml @@ -40,7 +40,7 @@ ]> - + fn @@ -373,6 +373,7 @@ + @@ -386,7 +387,6 @@ - diff --git a/xml/sas.xml b/xml/sas.xml new file mode 100644 --- /dev/null +++ b/xml/sas.xml @@ -0,0 +1,1370 @@ + + + + + + + + + + +]> + + + + + + + + BY + DO + ELSE + END + GOTO + IF + LINK + OTHERWISE + SELECT + THEN + TO + UNTIL + WHEN + WHILE + + + + AND + BETWEEN + CONTAINS + EQ + EQT + GE + GET + GT + IN + IS + LE + LET + LIKE + LT + NE + NET + NOT + OR + SAME + ALSO + + + + _ALL_ + _BLANKPAGE_ + _CHARACTER_ + _ERROR_ + _FILE_ + _INFILE_ + _IORC_ + _MRG_ + _NULL_ + _NUMERIC_ + _N_ + _PAGE_ + _TEMPORARY_ + + + + AXIS + CAS + CASLIB + CATNAME + DECLARE + ENDRSUBMIT + ENDSAS + FILENAME + FOOTNOTE + GOPTIONS + KILLTASK + LEGEND + LIBNAME + LISTTASK + LOCK + MISSING + ODS + PAGE + PATTERN + PROCEDURE + RESETLINE + RGET + RSPT + RSUBMIT + SASFILE + SIGNOFF + SIGNON + SKIP + SYMBOL + SYSECHO + SYSTASK + TITLE + TRANTAB + WAITFOR + + + + ABORT + ARRAY + ATTRIB + BY + CALL + CONTINUE + DELETE + DESCRIBE + DROP + ERROR + EXECUTE + FILE + FORMAT + INFILE + INFORMAT + INPUT + KEEP + LABEL + LEAVE + LENGTH + LIST + LOSTCARD + MERGE + MODIFY + OUTPUT + PUT + PUTLOG + REDIRECT + REMOVE + RENAME + REPLACE + RETAIN + RETURN + SET + STOP + UPDATE + WHERE + + + + %ABORT + %COMPSTOR + %COPY + %DISPLAY + %DQLOAD + %DQPUTLOC + %DQUNLOAD + %ELSE + %GLOBAL + %GOTO + %IF + %INC + %INCLUDE + %INPUT + %LABEL + %LET + %LIST + %LOCAL + %PUT + %RETURN + %RUN + %SYMDEL + %SYSMACDELETE + %SYSMACEXEC + %SYSEXEC + %THEN + %TO + %TPLOT + %UNTIL + %WHILE + %WINDOW + + + + ALTER + APPEND + BUFNO + BUFSIZE + CASLIB + CNTLLEV + COMPRESS + COPIES + DATALIMIT + DLDMGACTION + DROP + DUPLICATE + ENCODING + ENCRYPT + ENCRYPTKEY + EOC + EXTENDOBSCOUNTER + FILECLOSE + FIRSTOBS + GENMAX + GENNUM + IDXNAME + IDXWHERE + IN + INDEX + KEEP + LABEL + MEMTYPE + OBS + OBSBUF + ONDEMAND + ORDERBY + OUTREP + PARTITION + POINTOBS + PROMOTE + PW + PWREQ + READ + READTRANSFERSIZE + RENAME + REPEMPTY + REPLACE + REUSE + ROLE + RTS + SCRIPT + SGIO + SORTEDBY + SPILL + TAG + TEMPEXPRESS + TEMPNAMES + TOBSNO + TRANSCODE_FAIL + TRANTAB + TYPE + WHERE + WHEREUP + WRITE + WRITETRANSFERSIZE + + + + ABS + ADDMATRIX + ADDR + ADDRLONG + ADDROW + ADD_TABLE_ATTR + AIRY + ALLCOMB + ALLCOMBI + ALLPERM + ANYALNUM + ANYALPHA + ANYCNTRL + ANYDIGIT + ANYFIRST + ANYGRAPH + ANYLOWER + ANYNAME + ANYPRINT + ANYPUNCT + ANYSPACE + ANYUPPER + ANYXDIGIT + ARCOS + ARCOSH + ARMCONV + ARMEND + ARMGTID + ARMINIT + ARMJOIN + ARMPROC + ARMSTOP + ARMSTRT + ARMUPDT + ARSIN + ARSINH + ARTANH + ASCEBC + ATAN + ATAN2 + ATTRC + ATTRN + BAND + BETA + BETAINV + BLACKCLPRC + BLACKPTPRC + BLKSHCLPRC + BLKSHPTPRC + BLSHIFT + BNOT + BOR + BQUOTE + BRSHIFT + BXOR + BYTE + CAT + CATQ + CATS + CATT + CATX + CDF + CEIL + CEILZ + CEXIST + CHAR + CHOL + CHOOSEC + CHOOSEN + CINV + CLIBEXIST + CLOSE + CMISS + CMPRES + CNONCT + COALESCE + COALESCEC + COLLATE + COMB + COMPANION_NEXT + COMPARE + COMPBL + COMPCOST + COMPFUZZ + COMPGED + COMPLEV + COMPOUND + COMPRESS + COMPSTOR + CONSTANT + CONVX + CONVXP + COS + COSH + COT + COUNT + COUNTC + COUNTW + CSC + CSS + CUMIPMT + CUMPRINC + CUROBS + CV + DACCDB + DACCDBSL + DACCSL + DACCSYD + DACCTAB + DAIRY + DATATYP + DATDIF + DATE + DATEJUL + DATEPART + DATETIME + DAY + DCLOSE + DCREATE + DEFINE + DEPDB + DEPDBSL + DEPSL + DEPSYD + DEPTAB + DEQUOTE + DET + DEVIANCE + DHMS + DICTIONARY + DIF + DIGAMMA + DIM + DINFO + DISCARD + DIVIDE + DNUM + DOPEN + DOPTNAME + DOPTNUM + DOSUBL + DQCASE + DQGENDER + DQGENDERINFOGET + DQGENDERPARSED + DQIDENTIFY + DQLOCALEGUESS + DQLOCALEINFOGET + DQLOCALEINFOLIST + DQMATCH + DQMATCHINFOGET + DQMATCHPARSED + DQPARSE + DQPARSEINFOGET + DQPARSETOKENGET + DQPARSETOKENPUT + DQPATTERN + DQSCHEMEAPPLY + DQSRVARCHJOB + DQSRVCOPYLOG + DQSRVDELETELOG + DQSRVJOBSTATUS + DQSRVKILLJOB + DQSRVPROFJOBFILE + DQSRVPROFJOBREP + DQSRVUSER + DQSTANDARDIZE + DQTOKEN + DREAD + DROPNOTE + DSNAME + DSNCATLGD + DUR + DURP + DYNAMIC_ARRAY + EBCASC + EFFRATE + ELEMMULT + ENTRY_FIRST + ENTRY_NEXT + ENVLEN + ERF + ERFC + EUCLID + EVAL + EXECUTE + EXIST + EXISTS + EXP + EXPMATRIX + FACT + FAPPEND + FCLOSE + FCOL + FCOPY + FDELETE + FETCH + FETCHOBS + FEXIST + FGET + FILEATTR + FILEEXIST + FILENAME + FILEREF + FILLMATRIX + FINANCE + FIND + FINDC + FINDFILE + FINDTABLE + FINDW + FINFO + FINV + FIPNAME + FIPNAMEL + FIPSTATE + FIRST + FLOOR + FLOORZ + FMTINFO + FNONCT + FNOTE + FOPEN + FOPTNAME + FOPTNUM + FPOINT + FPOS + FPUT + FREAD + FREWIND + FRLEN + FSEP + FUZZ + FWRITE + GAMINV + GAMMA + GARKHCLPRC + GARKHPTPRC + GCD + GEODIST + GEOMEAN + GEOMEANZ + GETCASURL + GETDVI + GETJPI + GETLCASLIB + GETLOG + GETLSESSREF + GETLTAG + GETMSG + GETOPTION + GETQUOTA + GETSESSOPT + GETSYM + GETTERM + GETVARC + GETVARN + GITFN_CLONE + GITFN_COMMIT + GITFN_COMMITFREE + GITFN_COMMIT_GET + GITFN_COMMIT_LOG + GITFN_CO_BRANCH + GITFN_DEL_REPO + GITFN_DIFF + GITFN_DIFF_FREE + GITFN_DIFF_GET + GITFN_DIFF_IDX_F + GITFN_IDX_ADD + GITFN_IDX_REMOVE + GITFN_MRG_BRANCH + GITFN_NEW_BRANCH + GITFN_PULL + GITFN_PUSH + GITFN_RESET + GITFN_RESET_FILE + GITFN_STATUS + GITFN_STATUSFREE + GITFN_STATUS_GET + GITFN_VERSION + GRAYCODE + GRDSVC_ENABLE + GRDSVC_GETADDR + GRDSVC_GETINFO + GRDSVC_GETNAME + GRDSVC_NNODES + HARMEAN + HARMEANZ + HASHING + HASHING_FILE + HASHING_HMAC + HASHING_HMAC_FILE + HASHING_HMAC_INIT + HASHING_INIT + HASHING_PART + HASHING_TERM + HBOUND + HMS + HOLIDAY + HOLIDAYCK + HOLIDAYCOUNT + HOLIDAYNAME + HOLIDAYNX + HOLIDAYNY + HOLIDAYTEST + HOUR + HTMLDECODE + HTMLENCODE + IBESSEL + IDENTITY + IFC + IFN + INDEX + INDEXC + INDEXW + INPUT + INPUTC + INPUTN + INSERT_CATALOG + INSERT_DATASET + INSERT_FDB + INSERT_FILE + INSERT_HTML + INSERT_MDDB + INSERT_PACKAGE + INSERT_REF + INSERT_SQLVIEW + INSERT_VIEWER + INT + INTCINDEX + INTCK + INTCYCLE + INTFIT + INTFMT + INTGET + INTINDEX + INTNEST + INTNX + INTRR + INTSEAS + INTSHIFT + INTTEST + INTZ + INV + INVCDF + INVERSE + IORCMSG + IPMT + IQR + IRR + IS8601_CONVERT + ISARRAY + ISBLOB + ISDICTIONARY + ISDOUBLE + ISINTEGER + ISLIST + ISNULL + ISSTRING + ISTABLE + ISTYPE + JBESSEL + JSONPP + JULDATE + JULDATE7 + KCOMPARE + KCOMPRESS + KCOUNT + KINDEX + KINDEXC + KLEFT + KLENGTH + KLOWCASE + KREVERSE + KRIGHT + KSCAN + KSUBSTR + KSUBSTRB + KTRANSLATE + KTRIM + KTRUNCATE + KUPCASE + KUPDATE + KUPDATEB + KURTOSIS + KVERIFY + LABEL + LAG + LARGEST + LBOUND + LCM + LCOMB + LEFT + LENGTH + LENGTHC + LENGTHM + LENGTHN + LEXCOMB + LEXCOMBI + LEXPERK + LEXPERM + LFACT + LGAMMA + LIBNAME + LIBREF + LIMMOMENT + LOG + LOG10 + LOG1PX + LOG2 + LOGBETA + LOGCDF + LOGISTIC + LOGPDF + LOGSDF + LOWCASE + LPERM + LPNORM + MAD + MARGRCLPRC + MARGRPTPRC + MAX + MD5 + MDY + MEAN + MEDIAN + MIN + MINUTE + MISSING + MOD + MODEXIST + MODULE + MODULEC + MODULEN + MODZ + MONTH + MOPEN + MORT + MSPLINT + MULT + MVALID + N + NETPV + NEWTABLE + NLITERAL + NMISS + NODENAME + NOMRATE + NORMAL + NOTALNUM + NOTALPHA + NOTCNTRL + NOTDIGIT + NOTE + NOTFIRST + NOTGRAPH + NOTLOWER + NOTNAME + NOTPRINT + NOTPUNCT + NOTSPACE + NOTUPPER + NOTXDIGIT + NPV + NRBQUOTE + NRQUOTE + NRSTR + NVALID + NWKDOM + OPEN + ORDINAL + PACKAGE_BEGIN + PACKAGE_DESTROY + PACKAGE_END + PACKAGE_FIRST + PACKAGE_NEXT + PACKAGE_PUBLISH + PACKAGE_TERM + PATHNAME + PCTL + PDF + PEEK + PEEKC + PEEKCLONG + PEEKLONG + PERM + PMF + PMT + POINT + POISSON + POKE + POKELONG + POWER + PPMT + PRINTTABLE + PROBBETA + PROBBNML + PROBBNRM + PROBCHI + PROBF + PROBGAM + PROBHYPR + PROBIT + PROBMC + PROBMED + PROBNEGB + PROBNORM + PROBT + PROPCASE + PRXCHANGE + PRXDEBUG + PRXFREE + PRXMATCH + PRXNEXT + PRXPAREN + PRXPARSE + PRXPOSN + PRXSUBSTR + PTRLONGADD + PUT + PUTC + PUTLOG + PUTN + PUTSYM + PVP + QCMPRES + QLEFT + QLOWCASE + QSCAN + QSUBSTR + QSYSFUNC + QTR + QTRIM + QUANTILE + QUOTE + QUPCASE + RANBIN + RANCAU + RANCOMB + RAND + RANEXP + RANGAM + RANGE + RANK + RANNOR + RANPERK + RANPERM + RANPOI + RANTBL + RANTRI + RANUNI + READPATH + READ_ARRAY + RENAME + REPEAT + RESOLVE + RETRIEVE_CATALOG + RETRIEVE_DATASET + RETRIEVE_FDB + RETRIEVE_FILE + RETRIEVE_HTML + RETRIEVE_MDDB + RETRIEVE_NESTED + RETRIEVE_PACKAGE + RETRIEVE_REF + RETRIEVE_SQLVIEW + RETRIEVE_VIEWER + REVERSE + REWIND + RIGHT + RMS + ROUND + ROUNDE + ROUNDZ + RUN_MACRO + RUN_SASFILE + RXCHANGE + RXFREE + RXSUBSTR + SAVING + SAVINGS + SCAN + SCANQ + SDF + SEC + SECOND + SESSFOUND + SESSIONS + SET + SETNULL + SETTERM + SHA256 + SHA256HEX + SHA256HMACHEX + SIGN + SIN + SINH + SKEWNESS + SLEEP + SMALLEST + SOAPWEB + SOAPWEBMETA + SOAPWIPSERVICE + SOAPWIPSRS + SOAPWS + SOAPWSMETA + SOFTMAX + SOLVE + SORT + SORTC + SORTN + SORT_REV + SOUNDEX + SPEDIS + SQRT + SQUANTILE + STD + STDERR + STDIZE + STFIPS + STNAME + STNAMEL + STR + STREAMINIT + STRIP + STRUCTINDEX + SUBPAD + SUBSTR + SUBSTRN + SUBTRACTMATRIX + SUM + SUMABS + SUPERQ + SYMEXIST + SYMGET + SYMGLOBL + SYMLOCAL + SYMPUT + SYMPUTX + SYSEVALF + SYSEXIST + SYSFUNC + SYSGET + SYSMSG + SYSPARM + SYSPROCESSID + SYSPROCESSNAME + SYSPROD + SYSRC + SYSTEM + TABCOLUMNS + TABTYPES + TAN + TANH + TERMIN + TERMOUT + TIME + TIMEPART + TIMEVALUE + TINV + TNONCT + TODAY + TRACEBACK + TRANSLATE + TRANSPOSE + TRANSTRN + TRANWRD + TRIGAMMA + TRIM + TRIMN + TRUNC + TSO + TTCLOSE + TTCONTRL + TTOPEN + TTREAD + TTWRITE + TYPEOF + TZID + TZONEID + TZONENAME + TZONEOFF + TZONES2U + TZONEU2S + UNIFORM + UNQUOTE + UPCASE + URLDECODE + URLENCODE + USS + UUIDGEN + VAR + VARFMT + VARINFMT + VARLABEL + VARLEN + VARNAME + VARNUM + VARRAY + VARRAYX + VARTYPE + VERIFY + VFORMAT + VFORMATD + VFORMATDX + VFORMATN + VFORMATNX + VFORMATW + VFORMATWX + VFORMATX + VINARRAY + VINARRAYX + VINFORMAT + VINFORMATD + VINFORMATDX + VINFORMATN + VINFORMATNX + VINFORMATW + VINFORMATWX + VINFORMATX + VLABEL + VLABELX + VLENGTH + VLENGTHX + VMS + VNAME + VNAMEX + VNEXT + VTYPE + VTYPEX + VVALUE + VVALUEX + WEEK + WEEKDAY + WHICHC + WHICHN + WRITE_ARRAY + WTO + YEAR + YIELDP + YRDIF + YYQ + ZEROMATRIX + ZIPCITY + ZIPCITYDISTANCE + ZIPFIPS + ZIPNAME + ZIPNAMEL + ZIPSTATE + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/xml/sparql.xml b/xml/sparql.xml new file mode 100644 --- /dev/null +++ b/xml/sparql.xml @@ -0,0 +1,193 @@ + + + + + + ADD + AS + ASC + ASK + BASE + BIND + BY + CLEAR + CONSTRUCT + COPY + CREATE + DATA + DEFAULT + DELETE + DESC + DESCRIBE + DISTINCT + DROP + EXISTS + FILTER + FROM + GRAPH + GROUP + HAVING + IN + INSERT + INTO + LIMIT + LOAD + MINUS + MOVE + NAMED + NOT + OFFSET + OPTIONAL + ORDER + PREFIX + REDUCED + SELECT + SERVICE + SILENT + TO + UNDEF + UNION + USING + VALUES + WHERE + WITH + + + ABS + BNODE + BOUND + CEIL + COALESCE + CONCAT + CONTAINS + DATATYPE + DAY + ENCODE_FOR_URI + FLOOR + HOURS + IF + IRI + ISBLANK + ISIRI + ISLITERAL + ISNUMERIC + ISURI + LANG + LANGMATCHES + LCASE + MD5 + MINUTES + MONTH + NOW + RAND + REGEX + REPLACE + ROUND + SAMETERM + SECONDS + SHA1 + SHA256 + SHA384 + SHA512 + STR + STRAFTER + STRBEFORE + STRDT + STRENDS + STRLANG + STRLEN + STRSTARTS + STRUUID + SUBSTR + TIMEZONE + TZ + UCASE + URI + UUID + YEAR + + + AVG + COUNT + GROUP_CONCAT + MAX + MIN + SAMPLE + SUM + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/xml/spdx-comments.xml b/xml/spdx-comments.xml --- a/xml/spdx-comments.xml +++ b/xml/spdx-comments.xml @@ -6,545 +6,808 @@ ./generate-spdx-syntax.py > ../syntax/spdx-comments.xml --> diff --git a/xml/zsh.xml b/xml/zsh.xml --- a/xml/zsh.xml +++ b/xml/zsh.xml @@ -1,2172 +1,2987 @@ - - - - |&;)])"> - - - |&;()"> - - |&;)`"> - - - - - - - - "> - - - - - - - - - - - - - - '"`\\/]"> - - |&;)])"> - - - - |&;{}\\`'"$"> - - - - - - - - - - - - - - - - - - - - - '`"\\|()]*+(?:&_ispattern_q;)?)*(?:>|(?=[|()]))"> - - - |&;)]|&nogroupend;)|[ &tab;|(]|$)"> - - - - - - - - - - - - - -]> - - - - - - - - continue - break - case - do - done - elif - else - end - esac - fi - for - foreach - function - if - in - repeat - return - select - then - until - while - - - - - - . - : - alias - autoload - bg - bindkey - builtin - bye - cap - cd - chdir - clone - command - comparguments - compcall - compctl - compdescribe - compfiles - compgroups - compquote - comptags - comptry - compvalues - coproc - dirs - disable - disown - echo - echotc - echoti - emulate - enable - eval - exec - exit - false - fc - fg - functions - getcap - hash - history - jobs - kill - limit - log - logout - nocorrect - noglob - popd - print - printf - pushd - pushln - pwd - r - rehash - sched - set - setcap - setopt - shift - source - stat - suspend - test - times - trap - true - ttyctl - type - ulimit - umask - unalias - unfunction - unhash - unlimit - unset - unsetopt - vared - wait - whence - where - which - zcompile - zformat - zftp - zle - zmodload - zparseopts - zprof - zpty - zregexparse - zsocket - zstyle - ztcp - - - - declare - export - float - getln - getopts - integer - let - local - read - readonly - typeset - unset - - - - unixcommands##Bash - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + |&;)])"> + + + |&;()"> + + |&;)`"> + |&;"> + + + + + + + + + "> + + + + + + + + + + + + + + '"`\\/]"> + + |&;)])"> + + + + |&;{}\\`'"$"> + + + + + + + + + + + + + + + + + + + + + '`"\\|()]*+(?:&_ispattern_q;)?)*(?:>|(?=[|()]))"> + + + |&;)]|&nogroupend;)|[&ws;|(]|$)"> + + + + + + + + + + + + + + + + + + + + + + + + + =&^|?:"> +]> + + + + + + + + continue + break + case + do + done + elif + else + end + esac + fi + for + foreach + function + if + in + repeat + return + select + then + until + while + + + + - + . + : + alias + autoload + bg + bindkey + builtin + bye + cap + cd + chdir + clone + command + comparguments + compcall + compctl + compdescribe + compfiles + compgroups + compquote + comptags + comptry + compvalues + coproc + dirs + disable + disown + echo + echotc + echoti + emulate + enable + eval + exec + exit + false + fc + fg + functions + getcap + hash + history + jobs + kill + limit + log + logout + nocorrect + noglob + popd + print + printf + pushd + pushln + pwd + r + rehash + sched + set + setcap + setopt + shift + source + stat + suspend + test + times + trap + true + ttyctl + type + ulimit + umask + unalias + unfunction + unhash + unlimit + unset + unsetopt + vared + wait + whence + where + which + zcompile + zformat + zftp + zle + zmodload + zparseopts + zprof + zpty + zregexparse + zsocket + zstyle + ztcp + + + + declare + export + float + getln + getopts + integer + let + local + read + readonly + typeset + unset + + + + unixcommands##Bash + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +