cgrep 8.1.0 → 9.0.0
raw patch · 38 files changed
+10485/−7746 lines, 38 filesdep +atomic-primopsdep +clockdep +concurrencydep −cmdargsdep −mono-traversabledep −posix-pathsdep ~base
Dependencies added: atomic-primops, clock, concurrency, optparse-applicative, os-string, regex-pcre-text, regex-tdfa, template-haskell
Dependencies removed: cmdargs, mono-traversable, posix-paths, rawfilepath, regex-pcre, regex-posix, unagi-chan
Dependency ranges changed: base
Files
- README.md +497/−68
- cgrep.cabal +166/−131
- src/CGrep/Boundary.hs +20/−13
- src/CGrep/Common.hs +56/−58
- src/CGrep/ContextFilter.hs +0/−309
- src/CGrep/Distance.hs +38/−27
- src/CGrep/FileKind.hs +18/−16
- src/CGrep/FileType.hs +103/−31
- src/CGrep/FileTypeMap.hs +237/−4906
- src/CGrep/FileTypeMapTH.hs +4856/−0
- src/CGrep/Line.hs +135/−0
- src/CGrep/Match.hs +297/−0
- src/CGrep/Output.hs +0/−273
- src/CGrep/Parser/Atom.hs +139/−180
- src/CGrep/Parser/Char.hs +232/−57
- src/CGrep/Parser/Chunk.hs +138/−132
- src/CGrep/Parser/Line.hs +0/−91
- src/CGrep/Parser/Token.hs +270/−260
- src/CGrep/Search.hs +387/−47
- src/CGrep/Semantic/ContextFilter.hs +386/−0
- src/CGrep/Semantic/Tests.hs +1374/−0
- src/CGrep/Strategy/BoyerMoore.hs +75/−69
- src/CGrep/Strategy/Levenshtein.hs +38/−46
- src/CGrep/Strategy/Regex.hs +51/−48
- src/CGrep/Strategy/Semantic.hs +87/−94
- src/CGrep/Strategy/Tokenizer.hs +80/−80
- src/CGrep/Text.hs +159/−0
- src/CGrep/Types.hs +0/−31
- src/CmdOptions.hs +235/−67
- src/Config.hs +90/−111
- src/Main.hs +65/−84
- src/Options.hs +65/−61
- src/OsPath.hs +52/−0
- src/PutMessage.hs +79/−0
- src/Reader.hs +12/−10
- src/Search.hs +0/−331
- src/Util.hs +48/−49
- src/Verbose.hs +0/−66
README.md view
@@ -1,84 +1,513 @@-CGrep: a context-aware grep for source codes-============================================+# CGrep: a context-aware grep for source codes [](https://hackage.haskell.org/package/cgrep) [](https://gitter.im/awgn/cgrep?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) -Usage------+**Version 9.0.0** - A powerful, context-aware search tool designed specifically for source code. +CGrep extends the capabilities of traditional grep by understanding the structure and semantics of source code across multiple programming languages. It allows developers to search within specific contexts like code, comments, or string literals, and provides advanced pattern matching with semantic awareness.++---++## What's New in Version 9.0++### 🚀 Major Performance Improvements++- **75% faster plain token search** - Dramatically improved performance for standard token-based searches+- **39% faster semantic search** - Significant speed boost for advanced semantic pattern matching+- **Full UTF-8 support** - Switched from bytestream to text processing, enabling proper handling of UTF-8 character sets and international characters++### ✨ Enhanced Features++- **Semantic Test Filtering** - New capability to filter out test code from search results across 27+ programming languages and their respective testing frameworks (see [Test Framework Support](#test-framework-support) below)+- **Improved Text Processing** - Native support for Unicode and multi-byte character encodings with accurate column positioning++---++## Installation++### From Hackage++```bash+cabal update+cabal install cgrep ```-Cgrep 8.0.0. Usage: cgrep [OPTION] [PATTERN] files... -cgrep [OPTIONS] [ITEM]+### From Source -Pattern:- -f --file=FILE Read PATTERNs from file (one per line)- -w --word Force word matching- -p --prefix Force prefix matching- -s --suffix Force suffix matching- -e --edit Use edit distance- -G --regex Use regex matching (posix)- -P --pcre Use regex matching (pcre)- -i --ignore-case Ignore case distinctions+```bash+git clone https://github.com/awgn/cgrep.git+cd cgrep+cabal install+``` -Context filters:- -c --code Enable search in source code- -m --comment Enable search in comments- -l --literal Enable search in string literals+or using stack: -Token filters:- --name --identifier Identifiers- --type --native Native Types- --keyword Keywords- --number Literal numbers- --string Literal strings- --op Operators+```bash+stack build+stack install+``` -Semantic:- -S --semantic "code" pattern: _, _1, _2... (identifiers), $, $1,- $2... (optionals), ANY, KEY, STR, LIT, NUM, HEX, OCT,- OR+--- -Output control:- --max-count=INT Stop search in files after INT matches- -t --type-filter=ITEM Specify file types. ie: Cpp, +Haskell, -Makefile- -k --kind-filter=ITEM Specify file kinds. Text, Config, Language, Data,- Markup or Script- --force-type=ITEM Force the type of file- --type-list List the supported file types- -v --invert-match Select non-matching lines- --multiline=INT Enable multi-line matching- -r --recursive Enable recursive search (don't follow symlinks)- -T --skip-test Skip files that have 'test' in the name- --prune-dir=ITEM Do not descend into dir- -L --follow Follow symlinks+## Usage -Output format:- --show-match Show list of matching tokens- --color Use colors to highlight the match strings- --no-color Do not use colors (override config file)- -h --no-filename Suppress the file name prefix on output- --no-numbers Suppress both line and column numbers on output- --no-column Suppress the column number on output- --count Print only a count of matching lines per file- --filename-only Print only the name of files containing matches- --json Format output as json object- --vim Run vim editor passing the files that match- --editor Run the editor specified by EDITOR var., passing- the files that match- --fileline When edit option is specified, pass the list of- matching files in file:line format (e.g. vim- 'file-line' plugin)+```+cgrep 9.0.0 - Usage: cgrep [OPTION] [PATTERN] files... -Concurrency:- -j --threads=INT Number threads to run in parallel+Usage: cgrep [--file FILE] [-w|--word] [-p|--prefix] [-s|--suffix] [-e|--edit]+ [-G|--regex] [-i|--ignore-case] [-c|--code] [-m|--comment]+ [-l|--literal] [--identifier|--name] [--native|--type] [--keyword]+ [--number] [--string] [--op] [--type TYPE] [--kind KIND]+ [--code-only] [--hdr-only] [-T|--tests ARG] [--prune-dir DIR]+ [-r|--recursive] [-L|--follow] [-S|--semantic] [--strict]+ [--max-count INT] [--force-type TYPE] [--type-list]+ [-v|--invert-match] [-j|--threads INT] [--show-match] [--color]+ [--no-color] [-h|--no-filename] [--no-numbers] [--no-column]+ [--count] [--filename-only] [--json] [--vim] [--editor]+ [--fileline] [--verbose] [--stats] [--null-output] [--palette]+ [PATTERN FILES...] [--version] -Miscellaneous:- --verbose=INT Verbose level: 1, 2 or 3- --no-shallow Disable shallow-search- --palette Show color palette- -? --help Display help message- -V --version Print version information- --numeric-version Print just the version number+ Context-aware grep for source codes++Available options:+ --file FILE Read PATTERNs from file (one per line)+ -w,--word Force word matching+ -p,--prefix Force prefix matching+ -s,--suffix Force suffix matching+ -e,--edit Use edit distance+ -G,--regex Use regex matching (posix)+ -i,--ignore-case Ignore case distinctions+ -c,--code Enable search in source code+ -m,--comment Enable search in comments+ -l,--literal Enable search in string literals+ --identifier,--name Identifiers+ --native,--type Native Types+ --keyword Keywords+ --number Literal numbers+ --string Literal strings+ --op Operators+ --type TYPE Specify file types. ie: Cpp, +Haskell, -Makefile+ --kind KIND Specify file kinds. Text, Config, Language, Data,+ Markup or Script+ --code-only Parse code modules only (skip headers/interfaces)+ --hdr-only Parse headers/interfaces only (skip modules)+ -T,--tests ARG Filter tests: 'True' tests only, 'False' code only,+ omitted (search all)+ --prune-dir DIR Do not descend into dir+ -r,--recursive Enable recursive search (don't follow symlinks)+ -L,--follow Follow symlinks+ -S,--semantic "code" pattern: _, _1, _2... (identifiers), ANY, KEY,+ STR, LIT, NUM, HEX, OCT+ --strict Enable strict semantic for operators+ --max-count INT Stop search in files after INT matches+ --force-type TYPE Force the type of file+ --type-list List the supported file types+ -v,--invert-match Select non-matching lines+ -j,--threads INT Approximate number of threads to run search+ --show-match Show list of matching tokens+ --color Use colors to highlight the match strings+ --no-color Do not use colors (override config file)+ -h,--no-filename Suppress the file name prefix on output+ --no-numbers Suppress both line and column numbers on output+ --no-column Suppress the column number on output+ --count Print only a count of matching lines per file+ --filename-only Print only the name of files containing matches+ --json Format output as json object+ --vim Run vim editor passing the files that match+ --editor Run the editor specified by EDITOR var., passing the+ files that match+ --fileline When edit option is specified, pass the list of+ matching files in file:line format (e.g. vim+ 'file-line' plugin)+ --verbose Enable verbose mode+ --stats Print statistics about the search+ --null-output Disable output for performance evaluation+ --palette Show color palette+ -h,--help Show this help text+ --version Show version information and exit ```++---++## Examples++### Basic Searches++Search for a simple pattern in source files:++```bash+cgrep "main" *.c+```++Search recursively in a directory:++```bash+cgrep -r "TODO" src/+```++Case-insensitive search:++```bash+cgrep -i "buffer" *.cpp+```++### Context-Aware Searching++Search only in code (exclude comments and strings):++```bash+cgrep -c "malloc" *.c+```++Search only in comments:++```bash+cgrep -m "TODO" -r src/+```++Search only in string literals:++```bash+cgrep -l "hello" *.cpp+```++Search in both code and comments, but not in strings:++```bash+cgrep -c -m "config" *.js+```++### Token Filters++Search for identifiers only:++```bash+cgrep --identifier "main" *.c+```++Search for native types:++```bash+cgrep --type "int" *.c+```++Search for string literals containing specific text:++```bash+cgrep --string "hello" *.cpp+```++### File Type and Kind Filters++Search only in C++ files (recursively):++```bash+cgrep --type=Cpp -r "char" test/+```++Search in Haskell files, but exclude test code:++```bash+cgrep --type=Haskell -T False "function" -r .+```++Search by file kind (configuration files):++```bash+cgrep --kind=Config "database" -r /etc/+```++### Test Filtering (New in v9)++Search only in production code, excluding all tests:++```bash+cgrep -T False "function" -r src/+```++Search only in test code:++```bash+cgrep -T True "mock" -r tests/+```++This feature automatically detects and filters test code based on language-specific conventions (see [Test Framework Support](#test-framework-support)).++### Semantic Search++Semantic search allows you to match code patterns using wildcards:++- `_`, `_1`, `_2`, ... : Match any identifier+- `ANY` : Match any token+- `KEY` : Match any keyword+- `STR` : Match any string literal+- `LIT` : Match any literal+- `NUM` : Match any number+- `HEX` : Match any hexadecimal number+- `OCT` : Match any octal number++Find variable assignments with numeric literals:++```bash+cgrep -S "_ = NUM" *.c+```++### Advanced Pattern Matching++Use regular expressions (POSIX):++```bash+cgrep -G "main|return" *.c+```++Use word boundaries:++```bash+cgrep -w "read" *.c # Matches "read" but not "thread" or "reader"+```++Prefix matching:++```bash+cgrep -p "ma" *.c # Matches "main", etc.+```++Suffix matching:++```bash+cgrep -s "rn" *.c # Matches "return", etc.+```++Case-insensitive search:++```bash+cgrep -i "SED" test.cpp # Matches "Sed", "sed", etc.+```++### Output Formatting++Show only filenames of files containing matches:++```bash+cgrep --filename-only "char" *.cpp+```++Count matches per file:++```bash+cgrep --count "char" test.cpp+```++JSON output (useful for scripting):++```bash+cgrep --json "error" *.log+```++Suppress filename prefix:++```bash+cgrep -h "pattern" file.c+```++Show matching tokens:++```bash+cgrep --show-match "std::" *.cpp+```++### Editor Integration++Open matching files in vim:++```bash+cgrep --vim "FIXME" -r src/+```++Open with your default editor (set via EDITOR environment variable):++```bash+cgrep --editor "bug" -r src/+```++Use vim with file:line format (works with vim-file-line plugin):++```bash+cgrep --vim --fileline "error" -r src/+```++### Performance and Control++Limit number of threads:++```bash+cgrep -j 4 "pattern" -r large_codebase/+```++Stop after finding N matches in each file:++```bash+cgrep --max-count=5 "TODO" -r src/+```++Show search statistics:++```bash+cgrep --stats "function" -r src/+```++### UTF-8 and International Characters++Version 9's improved UTF-8 support allows searching in files with international characters:++```bash+cgrep "Hello" test.utf8+```++---++## Test Framework Support++Version 9 introduces intelligent test code filtering across 27+ programming languages. When using the `-T` flag, cgrep can automatically detect and filter test code based on language-specific conventions and testing frameworks.++| Language | Testing Frameworks Detected | Detection Patterns |+|----------|----------------------------|-------------------|+| **Rust** | Built-in, cargo test | `#[test]`, `#[cfg(test)]` modules |+| **Go** | Built-in testing | `func Test*`, `func Benchmark*` |+| **Java** | JUnit | `@Test` annotations |+| **Kotlin** | JUnit | `@Test` annotations |+| **C** | Google Test, Catch2 | `TEST()`, `TEST_F()`, `TEST_CASE()`, `test_*` functions |+| **C++** | Google Test, Catch2, Boost.Test | `TEST()`, `TEST_F()`, `TEST_CASE()`, `BOOST_AUTO_TEST`, `Test*` functions |+| **Python** | pytest, unittest | `test_*` functions, `Test*` classes, `@pytest`, `@unittest` decorators |+| **Zig** | Built-in | `test "..."` blocks |+| **JavaScript** | Mocha, Jasmine, Jest | `describe()`, `it()`, `test()`, `context()` |+| **TypeScript** | Mocha, Jasmine, Jest | `describe()`, `it()`, `test()`, `context()` |+| **Scala** | ScalaTest, MUnit | `test()`, `it()`, `describe()`, `scenario()`, `feature()` |+| **Haskell** | HSpec, Tasty, QuickCheck, HUnit | `describe`, `it`, `context`, `testCase`, `testGroup`, `testProperty`, `prop_*` functions |+| **C#** | NUnit, xUnit, MSTest | `[Test]`, `[Fact]`, `[Theory]`, `[TestMethod]`, `[TestFixture]`, `[TestClass]` |+| **F#** | NUnit, xUnit, Expecto | `[<Test>]`, `[<Fact>]`, `[<Theory>]`, `testCase`, `testList`, `test` |+| **Dart** | Built-in test package, Flutter | `test()`, `group()`, `testWidgets()` |+| **Elixir** | ExUnit | `test "..."`, `describe "..."`, `defmodule *Test` |+| **Ruby** | RSpec, Minitest | `describe`, `context`, `it`, `test_*`, `def test_*` |+| **PHP** | PHPUnit | `@test` annotations, `test*` methods, `*Test` classes |+| **Swift** | XCTest | `XCTestCase` classes, `func test*()` |+| **Objective-C** | XCTest | `XCTestCase` classes, test methods |+| **R** | testthat | `test_that()`, `describe()`, `context()` |+| **Julia** | Built-in Test | `@testset`, `@test` |+| **Perl** | Test::More, Test::Simple | `subtest`, `*.t` files |+| **OCaml** | OUnit, Alcotest | `let test_*`, `test_case` |+| **Erlang** | EUnit | `*_test()`, `*_test_()` functions |+| **Nim** | unittest | `unittest` module, `test "..."` |+| **Clojure** | clojure.test | `(deftest ...)`, `(testing ...)` |+| **D** | Built-in | `unittest { ... }` blocks |++### Example: Finding Production Code Only++```bash+# Search for "config" only in production code, skipping all tests+cgrep -T False "config" -r src/++# Search for mock usage only in test files+cgrep -T True "mock" -r .+```++---++## Supported Languages++CGrep supports a wide range of programming languages and file formats:++**Programming Languages:** C, C++, C#, Java, Kotlin, Scala, JavaScript, TypeScript, CoffeeScript, Python, Ruby, Perl, PHP, Go, Rust, Haskell, OCaml, F#, Erlang, Elixir, Clojure, Lisp, Scheme, Lua, R, Julia, Dart, Nim, Zig, D, Swift, Objective-C, Chapel, Awk, Shell scripts (Bash, Fish)++**Markup & Config:** HTML, XML, LaTeX, Markdown, YAML, JSON, TOML, INI, Dhall, CMake, Makefile, Cabal++To see the complete list of supported file types:++```bash+cgrep --type-list+```++---++## Configuration++CGrep can be configured using a configuration file located at `~/.cgreprc` (or `$XDG_CONFIG_HOME/cgrep/cgreprc`).++Example configuration:++```yaml+colors: true+file-types:+ - +Cpp+ - +Haskell+ - -Test+jobs: 8+```++---++## Performance Tips++1. **Use file type filters** to reduce the number of files processed:+ ```bash+ cgrep --type=Cpp "pattern" -r .+ ```++2. **Limit the search scope** with `--prune-dir` to exclude directories:+ ```bash+ cgrep --prune-dir=node_modules --prune-dir=.git "pattern" -r .+ ```++3. **Use context filters** when you know where to search:+ ```bash+ cgrep -c "pattern" -r . # Search only in code+ ```++4. **Use test filtering** to focus on production code:+ ```bash+ cgrep -T False "pattern" -r . # Exclude all test code+ ```++5. **Adjust thread count** for optimal performance on your system:+ ```bash+ cgrep -j 16 "pattern" -r .+ ```++---++## Benchmarks (v9 vs v8)++| Search Type | v8 | v9 | Improvement |+|-------------|----|----|-------------|+| Plain token search | 1.0x | **1.75x** | **+75%** |+| Semantic search | 1.0x | **1.39x** | **+39%** |++*Benchmarks performed on a typical codebase with ~100k lines of code.*++---++## Contributing++Contributions are welcome! Please feel free to submit issues or pull requests on [GitHub](https://github.com/awgn/cgrep).++---++## License++CGrep is released under the GPL-2.0-or-later license. See the LICENSE file for details.++---++## Author++Nicola Bonelli <nicola@larthia.com>++---++## Links++- **Homepage:** [http://awgn.github.io/cgrep/](http://awgn.github.io/cgrep/)+- **Hackage:** [https://hackage.haskell.org/package/cgrep](https://hackage.haskell.org/package/cgrep)+- **GitHub:** [https://github.com/awgn/cgrep](https://github.com/awgn/cgrep)+- **Gitter Chat:** [https://gitter.im/awgn/cgrep](https://gitter.im/awgn/cgrep)
cgrep.cabal view
@@ -1,139 +1,174 @@-Cabal-version: 2.2-Name: cgrep-Description: Cgrep: a context-aware grep for source codes-Version: 8.1.0-Synopsis: Command line tool-Homepage: http://awgn.github.io/cgrep/-License: GPL-2.0-or-later-License-file: LICENSE-Author: Nicola Bonelli-Maintainer: Nicola Bonelli <nicola@larthia.com>-Category: Utils-Build-type: Simple-Stability: Experimental-Extra-source-files: README.md+cabal-version: 2.2+name: cgrep+description: Cgrep: a context-aware grep for source codes+version: 9.0.0+synopsis: Command line tool+homepage: http://awgn.github.io/cgrep/+license: GPL-2.0-or-later+license-file: LICENSE+author: Nicola Bonelli+maintainer: Nicola Bonelli <nicola@larthia.com>+category: Utils+build-type: Simple+stability: Experimental+extra-source-files: README.md -Common common-options- build-depends: base ^>= 4.15.0.0+flag enable_pcre {+ Description: "Use PCRE regex engine (default: disabled)"+ Default: False+} - ghc-options: -Wall- -Wcompat- -Widentities- -Wincomplete-uni-patterns- -Wincomplete-record-updates- if impl(ghc >= 8.0)- ghc-options: -Wredundant-constraints- if impl(ghc >= 8.2)- ghc-options: -fhide-source-paths- if impl(ghc >= 8.4)- ghc-options: -Wmissing-export-lists- -Wpartial-fields- if impl(ghc >= 8.8)- ghc-options: -Wmissing-deriving-strategies+common common-options+ build-depends: base ^>=4.15.0.0+ ghc-options:+ -Wall+ -Wcompat+ -Widentities+ -Wincomplete-uni-patterns+ -Wincomplete-record-updates+ -Wunused-binds+ -Wunused-do-bind+ -Wunused-foralls+ -Wunused-imports+ -Wunused-local-binds+ -Wunused-matches+ -Wunused-pattern-binds+ -Wunused-top-binds+ -Wunused-type-patterns - default-language: Haskell2010+ if impl(ghc >=8.0)+ ghc-options: -Wredundant-constraints -Executable cgrep- Main-Is: Main.hs- Hs-Source-Dirs: src- Default-Extensions: FlexibleContexts- FlexibleInstances- GeneralisedNewtypeDeriving- DerivingStrategies- MultiWayIf- LambdaCase- OverloadedLists- OverloadedRecordDot- OverloadedStrings- PatternSynonyms- RecordWildCards- ScopedTypeVariables- TupleSections- TypeApplications- UnboxedSums- UnboxedTuples- ViewPatterns- BangPatterns- MagicHash+ if impl(ghc >=8.2)+ ghc-options: -fhide-source-paths - Other-Modules: Options- Verbose- CmdOptions- Util- Config- Reader- Search- CGrep.FileType- CGrep.FileKind- CGrep.FileTypeMap- CGrep.ContextFilter- CGrep.Types- CGrep.Output- CGrep.Distance- CGrep.Search- CGrep.Common- CGrep.Boundary- CGrep.Parser.Char- CGrep.Parser.Chunk- CGrep.Parser.Token- CGrep.Parser.Atom- CGrep.Parser.Line- CGrep.Strategy.Semantic- CGrep.Strategy.Tokenizer- CGrep.Strategy.Levenshtein- CGrep.Strategy.BoyerMoore- CGrep.Strategy.Regex- Paths_cgrep- Autogen-modules: Paths_cgrep+ if impl(ghc >=8.4)+ ghc-options:+ -Wmissing-export-lists+ -Wpartial-fields - Build-Depends: base < 5.0,- cmdargs,- bytestring,- directory,- filepath,- stm,- containers,- vector,- array,- ghc-prim,- dlist,- ansi-terminal,- split,- safe,- stringsearch,- unordered-containers,- regex-base,- regex-posix,- regex-pcre,- either,- mtl,- unix-compat,- async,- utf8-string,- unicode-show,- transformers,- process,- aeson,- yaml,- exceptions,- mono-traversable,- bytestring-strict-builder,- bitwise,- mmap,- unagi-chan,- posix-paths,- rawfilepath,- monad-loops,- deepseq,- bitarray,- text,- extra+ if impl(ghc >=8.8)+ ghc-options: -Wmissing-deriving-strategies+ default-language: Haskell2010 - Ghc-options: -O2 -optc-O3 - -funbox-strict-fields- -fwrite-ide-info- -hiedir=.hie- -threaded- -rtsopts "-with-rtsopts=-N -H1g -qn2"+executable cgrep+ import: common-options+ main-is: Main.hs+ hs-source-dirs: src+ default-extensions:+ CPP+ BangPatterns+ DerivingStrategies+ DeriveGeneric+ ExistentialQuantification+ FlexibleContexts+ FlexibleInstances+ GeneralisedNewtypeDeriving+ LambdaCase+ MagicHash+ MultiWayIf+ OverloadedLists+ OverloadedRecordDot+ OverloadedStrings+ PatternSynonyms+ RecordWildCards+ ScopedTypeVariables+ TupleSections+ TypeApplications+ UnboxedSums+ UnboxedTuples+ ViewPatterns - Default-language: Haskell2010+ other-modules:+ CGrep.Boundary+ CGrep.Common+ CGrep.Text+ CGrep.Line+ CGrep.Distance+ CGrep.FileKind+ CGrep.FileType+ CGrep.FileTypeMap+ CGrep.FileTypeMapTH+ CGrep.Match+ CGrep.Search+ CGrep.Semantic.ContextFilter+ CGrep.Semantic.Tests+ CGrep.Parser.Atom+ CGrep.Parser.Char+ CGrep.Parser.Chunk+ CGrep.Parser.Token+ CGrep.Strategy.BoyerMoore+ CGrep.Strategy.Levenshtein+ CGrep.Strategy.Regex+ CGrep.Strategy.Semantic+ CGrep.Strategy.Tokenizer+ CmdOptions+ Config+ Options+ OsPath+ Paths_cgrep+ PutMessage+ Reader+ Util++ autogen-modules: Paths_cgrep+ build-depends:+ aeson,+ ansi-terminal,+ atomic-primops,+ array,+ async,+ concurrency,+ clock,+ base <5.0,+ bitarray,+ bitwise,+ bytestring,+ bytestring-strict-builder,+ containers,+ deepseq,+ directory,+ dlist,+ either,+ exceptions,+ extra,+ filepath,+ ghc-prim,+ template-haskell,+ mmap,+ monad-loops,+ mtl,+ optparse-applicative,+ os-string,+ process,+ regex-base,+ regex-tdfa,+ safe,+ split,+ stm,+ stringsearch,+ text,+ transformers,+ unicode-show,+ unix-compat,+ unordered-containers,+ utf8-string,+ vector,+ yaml,++ ghc-options:+ -O2+ -optc-O3+ -funbox-strict-fields+ -fwrite-ide-info+ -hiedir=.hie+ -threaded+ -rtsopts+ "-with-rtsopts=-N -H16m -A64m -qa"++ if flag (enable_pcre) {+ build-depends: regex-pcre-text+ cpp-options: -DENABLE_PCRE+ }++ default-language: Haskell2010
src/CGrep/Boundary.hs view
@@ -1,5 +1,5 @@ ----- Copyright (c) 2013-2023 Nicola Bonelli <nicola@larthia.com>+-- Copyright (c) 2013-2025 Nicola Bonelli <nicola@larthia.com> -- -- This program is free software; you can redistribute it and/or modify -- it under the terms of the GNU General Public License as published by@@ -15,23 +15,29 @@ -- along with this program; if not, write to the Free Software -- Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. --+{-# LANGUAGE DeriveLift #-} module CGrep.Boundary (- Boundary(..)- , BoundaryType(..)- , pattern Begin- , pattern End) where--import qualified Data.ByteString.Char8 as C-import Data.Word ( Word8 )+ Boundary (..),+ BoundaryType (..),+ pattern Begin,+ pattern End,+)+where -data Boundary = Boundary {- bBegin :: C.ByteString- , bEnd :: C.ByteString-} deriving stock (Show, Eq)+import qualified Data.Text as T+import Data.Word (Word8)+import Language.Haskell.TH.Syntax (Lift) +data Boundary = Boundary+ { bBegin :: T.Text+ , bBeginLen :: Int+ , bEnd :: T.Text+ , bEndLen :: Int+ }+ deriving stock (Show, Eq, Lift) -newtype BoundaryType = BoundaryType { unpackBoundaryType :: Word8 }+newtype BoundaryType = BoundaryType {unpackBoundaryType :: Word8} deriving newtype (Eq, Ord) instance Show BoundaryType where@@ -40,6 +46,7 @@ pattern Begin :: BoundaryType pattern Begin = BoundaryType 0+ pattern End :: BoundaryType pattern End = BoundaryType 1
src/CGrep/Common.hs view
@@ -1,5 +1,5 @@ ----- Copyright (c) 2013-2023 Nicola Bonelli <nicola@larthia.com>+-- Copyright (c) 2013-2025 Nicola Bonelli <nicola@larthia.com> -- -- This program is free software; you can redistribute it and/or modify -- it under the terms of the GNU General Public License as published by@@ -16,81 +16,79 @@ -- Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. -- -module CGrep.Common ( Text8- , getTargetName- , getTargetContents- , expandMultiline- , ignoreCase- , subText- , trim- , trim8- , takeN) where+module CGrep.Common (+ runSearch,+ getTargetName,+ getTargetContents,+ ignoreCase,+ sliceToMaxIndex,+ trim,+ trimT,+ takeN,+)+where -import Data.Char (toLower)+import CGrep.Line (LineIndex)+import CGrep.Match (Match, mkMatches) import CGrep.Parser.Char (isSpace)-import CGrep.Types ( Offset, Text8 )--import Options- ( Options(Options, no_shallow, multiline, ignore_case) )--import Util ( spanGroup )-import Data.Int (Int64)-import System.IO.MMap ( mmapFileByteString )--import System.Posix.FilePath ( RawFilePath )-import qualified Data.Vector.Unboxed as UV-import Data.List (groupBy, group, sortOn, sort)--import GHC.Exts ( groupWith )--import qualified Data.ByteString.Char8 as C+import CGrep.Parser.Chunk (Chunk)+import qualified Data.Text as T+import qualified Data.Text.IO.Utf8 as TIO+import qualified Data.Text.Unsafe as TU+import Reader (ReaderIO)+import System.OsPath+import qualified System.OsString as OS+import Options (Options (..)) takeN :: Int -> String -> String-takeN n xs | length xs > n = take n xs <> "..."- | otherwise = xs+takeN n xs+ | length xs > n = take n xs <> "..."+ | otherwise = xs {-# INLINE takeN #-} - trim :: String -> String trim = (dropWhile isSpace . reverse) . dropWhile isSpace . reverse {-# INLINE trim #-} -trim8 :: Text8 -> Text8-trim8 = (C.dropWhile isSpace . C.reverse) . C.dropWhile isSpace . C.reverse-{-# INLINE trim8 #-}-+trimT :: T.Text -> T.Text+trimT = T.stripEnd . T.stripStart -getTargetName :: RawFilePath -> RawFilePath-getTargetName (C.null -> True )= "<STDIN>"+getTargetName :: OsPath -> OsPath+getTargetName (OS.null -> True) = unsafeEncodeUtf "<STDIN>" getTargetName name = name {-# INLINE getTargetName #-} --getTargetContents :: RawFilePath -> IO Text8-getTargetContents (C.null -> True) = C.getContents-getTargetContents xs = mmapFileByteString (C.unpack xs) Nothing+getTargetContents :: OsPath -> IO T.Text+getTargetContents (OS.null -> True) = TIO.getContents+getTargetContents xs = decodeUtf xs >>= TIO.readFile {-# INLINE getTargetContents #-} --expandMultiline :: Options -> Text8 -> Text8-expandMultiline Options { multiline = n } xs- | n == 1 = xs- | otherwise = C.unlines $ map C.unwords $ spanGroup n (C.lines xs)-{-# INLINE expandMultiline #-}---ignoreCase :: Options -> Text8 -> Text8+ignoreCase :: Options -> T.Text -> T.Text ignoreCase opt- | ignore_case opt = C.map toLower+ | ignore_case opt = T.toLower | otherwise = id {-# INLINE ignoreCase #-} +sliceToMaxIndex+ :: [[Int]] -> T.Text -> T.Text+sliceToMaxIndex+ [] txt = txt+sliceToMaxIndex+ indices txt = case T.findIndex (== '\n') (TU.dropWord8 maxOff txt) of+ Nothing -> txt+ (Just n) -> TU.takeWord8 (maxOff + n) txt+ where+ maxOff = maximum (lastDef 0 <$> indices)+ lastDef def xs = if null xs then def else last xs+{-# INLINE sliceToMaxIndex #-} -subText :: [[Offset]] -> Text8 -> Text8-subText [] txt = txt-subText indices txt = case C.elemIndex '\n' (C.drop maxOff txt) of- Nothing -> txt- (Just n) -> C.take (maxOff + n) txt- where maxOff = fromIntegral $ maximum (lastDef 0 <$> indices)- lastDef def xs = if null xs then def else last xs-{-# INLINE subText #-}+runSearch ::+ LineIndex ->+ OsPath ->+ Bool ->+ ReaderIO [Match] ->+ ReaderIO [Match]+runSearch lindex filename eligible doSearch =+ if eligible+ then doSearch+ else mkMatches lindex filename T.empty ([] :: [Chunk])
− src/CGrep/ContextFilter.hs
@@ -1,309 +0,0 @@------ Copyright (c) 2013-2023 Nicola Bonelli <nicola@larthia.com>------ This program is free software; you can redistribute it and/or modify--- it under the terms of the GNU General Public License as published by--- the Free Software Foundation; either version 2 of the License, or--- (at your option) any later version.------ This program is distributed in the hope that it will be useful,--- but WITHOUT ANY WARRANTY; without even the implied warranty of--- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the--- GNU General Public License for more details.------ You should have received a copy of the GNU General Public License--- along with this program; if not, write to the Free Software--- Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.-----module CGrep.ContextFilter where--import CGrep.Types ( Text8 )-import CGrep.Parser.Char ( chr, isSpace, ord )--import qualified Data.ByteString.Char8 as C-import qualified Data.ByteString.Short as B--import qualified Data.Map as Map--import Options ( Options(..) )-import Data.List (findIndex, nub, find)-import Data.Maybe (fromMaybe, isJust)-import qualified Data.Aeson.KeyMap as B-import CGrep.Boundary ( Boundary(..) )--import Data.Int ( Int32, Int64 )-import Data.Bits ( Bits((.|.), complement, (.&.), xor, shiftL, shiftR) )--import qualified Data.Vector as V-import qualified Data.Vector.Unboxed as UV--import Data.HashMap.Internal.Strict (alter)-import Data.Word (Word64)-import qualified Data.ByteString.Unsafe as U--import Util ( findWithIndex )--type FilterFunction = ContextFilter -> Text8 -> Text8--data Context = Code | Comment | Literal- deriving stock (Eq, Show)--newtype ContextBit = ContextBit Int32- deriving stock (Show)- deriving newtype (Eq, Bits)---contextBitEmpty :: ContextBit-contextBitEmpty = ContextBit 0-contextBitCode :: ContextBit-contextBitCode = ContextBit 0x1-contextBitComment :: ContextBit-contextBitComment = ContextBit 0x2-contextBitLiteral :: ContextBit-contextBitLiteral = ContextBit 0x4---(~=) :: ContextBit -> Bool -> ContextBit-b ~= True = b-_ ~= False = contextBitEmpty-{-# INLINE (~=) #-}---(~?) :: ContextFilter -> ContextBit -> Bool-f ~? b = (unFilter f .&. b) /= contextBitEmpty-{-# INLINE (~?) #-}--(~!) :: ContextFilter -> ContextBit -> ContextFilter-a ~! b = ContextFilter $ unFilter a .&. complement b-{-# INLINE (~!) #-}---newtype ContextFilter = ContextFilter { unFilter :: ContextBit }- deriving stock (Show)- deriving newtype (Eq, Bits)--contextFilterAll :: ContextFilter-contextFilterAll = ContextFilter (contextBitCode .|. contextBitComment .|. contextBitLiteral)-{-# NOINLINE contextFilterAll #-}--isContextFilterAll :: ContextFilter -> Bool-isContextFilterAll f = f == contextFilterAll-{-# INLINE isContextFilterAll #-}--codeFilter :: ContextFilter -> Bool-codeFilter f = (unFilter f .&. contextBitCode) /= contextBitEmpty-{-# INLINE codeFilter #-}--commentFilter :: ContextFilter -> Bool-commentFilter f = (unFilter f .&. contextBitComment) /= contextBitEmpty-{-# INLINE commentFilter #-}--literalFilter :: ContextFilter -> Bool-literalFilter f = (unFilter f .&. contextBitLiteral) /= contextBitEmpty-{-# INLINE literalFilter #-}---data ParConfig = ParConfig- { commBound :: [Boundary]- , litrBound :: [Boundary]- , rawBound :: [Boundary]- , chrBound :: [Boundary]- , inits :: B.ShortByteString- , alterBoundary :: Bool- }---mkParConfig :: [Boundary] -> [Boundary] -> [Boundary] -> [Boundary] -> Bool -> ParConfig-mkParConfig cs ls rs chs ab =- ParConfig {- commBound = cs- , litrBound = ls- , rawBound = rs- , chrBound = chs- , inits = (B.pack . nub) ((fromIntegral . ord . C.head . bBegin <$> cs) <>- (fromIntegral . ord . C.head . bBegin <$> ls) <>- (fromIntegral . ord . C.head . bBegin <$> rs) <>- (fromIntegral . ord . C.head . bBegin <$> chs))- , alterBoundary = ab }--data ParState = ParState- { ctxState :: !ContextState- , nextState :: !ContextState- , display :: !Bool- , skip :: {-# UNPACK #-} !Int- } deriving stock (Show)---data ContextState =- CodeState1 | CodeStateN |- CommState1 {-# UNPACK #-} !Int |- CommStateN {-# UNPACK #-} !Int |- ChrState {-# UNPACK #-} !Int |- LitrState1 {-# UNPACK #-} !Int |- LitrStateN {-# UNPACK #-} !Int |- RawState {-# UNPACK #-} !Int- deriving stock (Show, Eq, Ord)---mkContextFilter :: Options -> ContextFilter-mkContextFilter Options{..} =- if not (code || comment || literal)- then contextFilterAll- else ContextFilter $ contextBitCode ~= code .|. contextBitComment ~= comment .|. contextBitLiteral ~= literal---unpackBoundary :: Boundary -> (String, String)-unpackBoundary (Boundary a b) = (C.unpack a, C.unpack b)-{-# INLINE unpackBoundary #-}---getContext :: ContextState -> Context-getContext CodeState1 = Code-getContext CodeStateN = Code-getContext (CommState1 _ ) = Comment-getContext (CommStateN _ ) = Comment-getContext (LitrState1 _) = Literal-getContext (LitrStateN _) = Literal-getContext (RawState _) = Literal-getContext (ChrState _) = Literal-{-# INLINE getContext #-}---- contextFilterFun:-----data ParData = ParData {- pdText :: {-# UNPACK #-}!Text8,- pdState :: !ParState-}--runContextFilter :: ParConfig -> ContextFilter -> Text8 -> Text8-runContextFilter conf@ParConfig{..} f txt | alterBoundary = fst $ C.unfoldrN (C.length txt) (contextFilter' conf) (ParData txt (ParState CodeState1 CodeState1 (codeFilter f) 0))- | otherwise = fst $ C.unfoldrN (C.length txt) (contextFilter'' conf) (ParData txt (ParState CodeState1 CodeState1 (codeFilter f) 0))- where contextFilter' :: ParConfig -> ParData -> Maybe (Char, ParData)- contextFilter' c (ParData txt@(C.uncons -> Just (x,xs)) s) =- let !s' = nextContextState c s txt f- in if display s'- then case (# getContext (ctxState s), getContext (ctxState s') #) of- (# Code, Literal #) -> Just (chr 2, ParData xs s')- (# Literal, Code #) -> Just (chr 3, ParData xs s')- _ -> Just (x, ParData xs s')- else if isSpace x- then Just (x, ParData xs s')- else Just (' ', ParData xs s')-- contextFilter' _ (ParData (C.uncons -> Nothing) _) = Nothing-- contextFilter'' :: ParConfig -> ParData -> Maybe (Char, ParData)- contextFilter'' c (ParData txt@(C.uncons -> Just (x,xs)) s) =- let !s' = nextContextState c s txt f- in if display s' || isSpace x- then Just (x, ParData xs s')- else Just (' ', ParData xs s')---{-# INLINE nextContextState #-}-nextContextState :: ParConfig -> ParState -> Text8 -> ContextFilter -> ParState-nextContextState c s@ParState{..} txt f- | skip > 0 = {-# SCC skip #-} transState s{ skip = skip - 1 }-- | CodeState1 <- ctxState = {-# SCC next_code1 #-} if U.unsafeHead txt `B.elem` inits c- then case findPrefixBoundary txt (commBound c) of- (# i, Just b #) -> {-# SCC next_code1_1 #-} transState s{ nextState = CommState1 i, display = commentFilter f, skip = C.length (bBegin b) - 1 }- _ -> case findPrefixBoundary txt (litrBound c) of- (# i, Just b #) -> {-# SCC next_code1_2 #-} transState s{ nextState = LitrState1 i, display = codeFilter f, skip = C.length (bBegin b) - 1 }- _ -> case findPrefixBoundary txt (rawBound c) of- (# i, Just b #) -> {-# SCC next_code1_3 #-} transState s{ nextState = RawState i, display = codeFilter f, skip = C.length (bBegin b) - 1 }- _ -> case findPrefixBoundary' txt (chrBound c) of- (# i, Just b #) -> transState s{ nextState = ChrState i, display = codeFilter f, skip = C.length (bBegin b) - 1 }- _ -> {-# SCC next_code1_5 #-} s{ ctxState = CodeStateN, nextState = CodeStateN, display = codeFilter f, skip = 0 }- else {-# SCC next_code1_0 #-} s{ ctxState = CodeStateN, nextState = CodeStateN, display = codeFilter f, skip = 0 }-- | CodeStateN <- ctxState = {-# SCC next_code #-} if {-# SCC next_code_if #-} U.unsafeHead txt `B.elem` inits c- then {-# SCC next_code_then #-} case findPrefixBoundary txt (commBound c) of- (#i , Just b #) -> {-# SCC next_code1_1 #-} transState s{ nextState = CommState1 i, display = commentFilter f, skip = C.length (bBegin b) - 1 }- _ -> case findPrefixBoundary txt (litrBound c) of- (#i, Just b #) -> {-# SCC next_code1_2 #-} transState s{ nextState = LitrState1 i, display = codeFilter f, skip = C.length (bBegin b) - 1 }- _ -> case findPrefixBoundary txt (rawBound c) of- (# i, Just b #) -> {-# SCC next_code1_3 #-} transState s{ nextState = RawState i, display = codeFilter f, skip = C.length (bBegin b) - 1 }- _ -> case findPrefixBoundary' txt (chrBound c) of- (# i, Just b #) -> transState s{ nextState = ChrState i, display = codeFilter f, skip = C.length (bBegin b) - 1 }- _ -> {-# SCC next_code_5 #-} s- else {-# SCC next_code_else #-} s-- | CommState1 n <- ctxState =- let Boundary _ e = commBound c !! n- in {-# SCC next_comm1 #-} if e `C.isPrefixOf` txt- then transState $ s{ nextState = CodeState1, display = commentFilter f, skip = C.length e - 1}- else s{ ctxState = CommStateN n, nextState = CommStateN n, display = commentFilter f, skip = 0 }-- | CommStateN n <- ctxState =- let Boundary _ e = commBound c !! n- in {-# SCC next_comm #-} if e `C.isPrefixOf` txt- then transState $ s{ nextState = CodeState1, display = commentFilter f, skip = C.length e - 1}- else s-- | LitrState1 n <- ctxState =- if C.head txt == '\\'- then s { display = displayContext ctxState f, skip = 1 }- else let Boundary _ e = litrBound c !! n- in {-# SCC next_liter #-} if e `C.isPrefixOf` txt- then s{ ctxState = CodeState1, nextState = CodeState1, display = codeFilter f, skip = C.length e - 1}- else s{ ctxState = LitrStateN n, nextState = LitrStateN n, display = literalFilter f, skip = 0 }-- | LitrStateN n <- ctxState =- if C.head txt == '\\'- then s { display = displayContext ctxState f, skip = 1 }- else let Boundary _ e = litrBound c !! n- in {-# SCC next_liter #-} if e `C.isPrefixOf` txt- then s{ ctxState = CodeState1, nextState = CodeState1, display = codeFilter f, skip = C.length e - 1}- else s-- | ChrState n <- ctxState =- if C.head txt == '\\'- then s { display = displayContext ctxState f, skip = 1 }- else let Boundary _ e = chrBound c !! n- in {-# SCC next_chr #-} if e `C.isPrefixOf` txt- then s{ ctxState = CodeState1, nextState = CodeState1, display = codeFilter f, skip = C.length e - 1}- else s{ display = literalFilter f , skip = 0}-- | RawState n <- ctxState =- let Boundary _ e = rawBound c !! n- in {-# SCC next_raw #-} if e `C.isPrefixOf` txt- then s{ ctxState = CodeState1, nextState = CodeState1, display = codeFilter f, skip = C.length e - 1}- else s{ display = literalFilter f , skip = 0}---displayContext :: ContextState -> ContextFilter -> Bool-displayContext CodeState1 cf = cf ~? contextBitCode-displayContext CodeStateN cf = cf ~? contextBitCode-displayContext (CommState1 _) cf = cf ~? contextBitComment-displayContext (CommStateN _) cf = cf ~? contextBitComment-displayContext (LitrState1 _) cf = cf ~? contextBitLiteral-displayContext (LitrStateN _) cf = cf ~? contextBitLiteral-displayContext (RawState _) cf = cf ~? contextBitLiteral-displayContext (ChrState _) cf = cf ~? contextBitLiteral-{-# INLINE displayContext #-}---transState :: ParState -> ParState-transState s@ParState {..} | skip == 0 = s{ ctxState = nextState }- | otherwise = s-{-# INLINE transState #-}---findPrefixBoundary :: Text8 -> [Boundary] -> (# Int, Maybe Boundary #)-findPrefixBoundary xs vb = {-# SCC findPrefixBoundary #-}- findWithIndex (\(Boundary b _ ) -> b `C.isPrefixOf` xs) vb-{-# INLINE findPrefixBoundary #-}---findPrefixBoundary' :: Text8 -> [Boundary] -> (# Int, Maybe Boundary #)-findPrefixBoundary' txt bs =- case findWithIndex (\(Boundary beg _ ) -> beg `C.isPrefixOf` txt) bs of- elm@(# idx, Just b@(Boundary _ end) #) -> case C.tail txt of- (C.uncons -> Just (y , ys)) ->- let skip = if y == '\\' then 1 else 0- in if end `C.isPrefixOf` C.drop skip ys then elm else (#0, Nothing #)- _ -> (# 0, Nothing #)- _ -> (# 0, Nothing #)
src/CGrep/Distance.hs view
@@ -1,5 +1,5 @@ ----- Copyright (c) 2013-2023 Nicola Bonelli <nicola@larthia.com>+-- Copyright (c) 2013-2025 Nicola Bonelli <nicola@larthia.com> -- -- This program is free software; you can redistribute it and/or modify -- it under the terms of the GNU General Public License as published by@@ -15,38 +15,49 @@ -- along with this program; if not, write to the Free Software -- Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. --+{-# OPTIONS_GHC -Wno-x-partial #-} module CGrep.Distance (distance, (~==)) where -- from http://www.haskell.org/haskellwiki/Edit_distance -- -distance :: Eq a => [a] -> [a] -> Int-distance a b- = last (if lab == 0 then mainDiag- else if lab > 0 then lowers !! (lab - 1)- else {- < 0 -} uppers !! (-1 - lab))- where mainDiag = oneDiag a b (head uppers) (-1 : head lowers)- uppers = eachDiag a b (mainDiag : uppers) -- upper diagonals- lowers = eachDiag b a (mainDiag : lowers) -- lower diagonals- eachDiag _a [] _diags = []- eachDiag a' (_bch:bs) (lastDiag:diags) = oneDiag a' bs nextDiag lastDiag : eachDiag a' bs diags- where nextDiag = head (tail diags)- eachDiag _ _ [] = undefined -- the original implementation does not cover this case...- oneDiag a' b' diagAbove diagBelow = thisdiag- where doDiag [] _b _nw _n _w = []- doDiag _a [] _nw _n _w = []- doDiag (ach:as) (bch:bs) nw n w = me : doDiag as bs me (tail n) (tail w)- where me = if ach == bch then nw else 1 + min3 (head w) nw (head n)- firstelt = 1 + head diagBelow- thisdiag = firstelt : doDiag a' b' firstelt diagAbove (tail diagBelow)- lab = length a - length b- min3 x y z = if x < y then x else min y z-+distance :: (Eq a) => [a] -> [a] -> Int+distance a b =+ last+ ( if lab == 0+ then mainDiag+ else+ if lab > 0+ then lowers !! (lab - 1)+ else {- < 0 -} uppers !! (-1 - lab)+ )+ where+ mainDiag = oneDiag a b (head uppers) (-1 : head lowers)+ uppers = eachDiag a b (mainDiag : uppers) -- upper diagonals+ lowers = eachDiag b a (mainDiag : lowers) -- lower diagonals+ eachDiag _a [] _diags = []+ eachDiag a' (_bch : bs) (lastDiag : diags) = oneDiag a' bs nextDiag lastDiag : eachDiag a' bs diags+ where+ nextDiag = head (tail diags)+ eachDiag _ _ [] = undefined -- the original implementation does not cover this case...+ oneDiag a' b' diagAbove diagBelow = thisdiag+ where+ doDiag [] _b _nw _n _w = []+ doDiag _a [] _nw _n _w = []+ doDiag (ach : as) (bch : bs) nw n w = me : doDiag as bs me (tail n) (tail w)+ where+ me = if ach == bch then nw else 1 + min3 (head w) nw (head n)+ firstelt = 1 + head diagBelow+ thisdiag = firstelt : doDiag a' b' firstelt diagAbove (tail diagBelow)+ lab = length a - length b+ min3 x y z = if x < y then x else min y z (~==) :: String -> String -> Bool-a ~== b | len < 5 = dist < 3- | otherwise = dist < (len * 40 `div` 100)- where len = fromIntegral (length a `min` length b)- dist = distance a b+a ~== b+ | len < 5 = dist < 3+ | otherwise = dist < (len * 40 `div` 100)+ where+ len = length a `min` length b+ dist = distance a b {-# INLINE (~==) #-}
src/CGrep/FileKind.hs view
@@ -1,5 +1,5 @@ ----- Copyright (c) 2013-2023 Nicola Bonelli <nicola@larthia.com>+-- Copyright (c) 2013-2025 Nicola Bonelli <nicola@larthia.com> -- -- This program is free software; you can redistribute it and/or modify -- it under the terms of the GNU General Public License as published by@@ -15,29 +15,31 @@ -- along with this program; if not, write to the Free Software -- Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. --+{-# LANGUAGE DeriveLift #-} module CGrep.FileKind (- FileKind(..)-) where+ FileKind (..),+)+where +import Language.Haskell.TH.Syntax (Lift) data FileKind = KindText | KindConfig | KindLanguage | KindData | KindMarkup | KindScript- deriving stock (Eq, Ord, Enum, Bounded)-+ deriving stock (Eq, Ord, Enum, Bounded, Lift) instance Show FileKind where- show KindText = "Text"- show KindConfig = "Config"+ show KindText = "Text"+ show KindConfig = "Config" show KindLanguage = "Language"- show KindData = "Data"- show KindMarkup = "Markup"- show KindScript = "Script"+ show KindData = "Data"+ show KindMarkup = "Markup"+ show KindScript = "Script" instance Read FileKind where- readsPrec _ "Text" = [(KindText, "")]- readsPrec _ "Config" = [(KindConfig, "")]+ readsPrec _ "Text" = [(KindText, "")]+ readsPrec _ "Config" = [(KindConfig, "")] readsPrec _ "Language" = [(KindLanguage, "")]- readsPrec _ "Data" = [(KindData, "")]- readsPrec _ "Markup" = [(KindMarkup, "")]- readsPrec _ "Script" = [(KindScript, "")]- readsPrec _ _ = []+ readsPrec _ "Data" = [(KindData, "")]+ readsPrec _ "Markup" = [(KindMarkup, "")]+ readsPrec _ "Script" = [(KindScript, "")]+ readsPrec _ _ = []
src/CGrep/FileType.hs view
@@ -1,5 +1,5 @@ ------ Copyright (c) 2013-2023 Nicola Bonelli <nicola@larthia.com>+-- Copyright (c) 2013-2025 Nicola Bonelli <nicola@larthia.com> -- -- This program is free software; you can redistribute it and/or modify -- it under the terms of the GNU General Public License as published by@@ -15,50 +15,122 @@ -- along with this program; if not, write to the Free Software -- Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. --+{-# LANGUAGE DeriveLift #-} module CGrep.FileType (- FileType(..)- , FileSelector(..)- , readTypeList- , readKindList )- where+ FileType (..),+ FileSelector (..),+ readTypeList,+ readKindList,+ ext,+ hdr,+ name,+)+where -import qualified Data.Map as Map-import Control.Monad ( forM_ )-import Control.Applicative ( Alternative((<|>)) )-import Data.Maybe ( fromJust )+import CGrep.FileKind (FileKind)+import Language.Haskell.TH.Syntax (Lift)+import System.OsPath (OsPath)+import qualified System.OsPath as OS+import Util (prettyRead) -import qualified Data.ByteString.Char8 as C-import Options ( Options(Options, type_force) )-import Util ( prettyRead )-import System.Posix.FilePath (RawFilePath)-import CGrep.FileKind ( FileKind )+data FileType+ = Agda+ | Assembly+ | Awk+ | Bash+ | C+ | CMake+ | Cabal+ | Chapel+ | Clojure+ | Coffee+ | Conf+ | Cpp+ | Csh+ | Csharp+ | Css+ | Cql+ | D+ | Dart+ | Dhall+ | Elm+ | Elixir+ | Erlang+ | Fish+ | Fortran+ | Fsharp+ | Go+ | GoMod+ | Haskell+ | Html+ | Idris+ | Java+ | Javascript+ | Json+ | Julia+ | Kotlin+ | Ksh+ | Latex+ | Lisp+ | Lua+ | Make+ | Nim+ | Nmap+ | OCaml+ | ObjectiveC+ | PHP+ | Perl+ | Python+ | R+ | Ruby+ | Rust+ | Scala+ | SmallTalk+ | Swift+ | Sql+ | Tcl+ | Text+ | Unison+ | VHDL+ | Verilog+ | Yaml+ | Toml+ | Ini+ | Zig+ | Zsh+ deriving stock (Read, Show, Eq, Ord, Bounded, Lift) +data FileSelector = Name OsPath | Ext OsPath | Hdr OsPath+ deriving stock (Eq, Ord, Lift) -data FileType = Agda | Assembly | Awk | Bash | C | CMake | Cabal | Chapel | Clojure | Coffee | Conf | Cpp | Csh | Csharp | Css | Cql |- D | Dart | Dhall | Elm | Elixir | Erlang | Fish | Fortran | Fsharp | Go | GoMod | Haskell | Html | Idris | Java | Javascript | Json | Julia | Kotlin |- Ksh | Latex | Lisp | Lua | Make | Nim | Nmap | OCaml | ObjectiveC | PHP | Perl | Python | R | Ruby | Rust | Scala | SmallTalk | Swift | Sql | Tcl |- Text | Unison | VHDL | Verilog | Yaml | Toml | Ini | Zig | Zsh- deriving stock (Read, Show, Eq, Ord, Bounded)+ext :: String -> FileSelector+ext = Ext . OS.unsafeEncodeUtf+{-# INLINE ext #-} -data FileSelector = Name RawFilePath| Ext C.ByteString- deriving stock (Eq, Ord)+hdr :: String -> FileSelector+hdr = Hdr . OS.unsafeEncodeUtf+{-# INLINE hdr #-} +name :: String -> FileSelector+name = Name . OS.unsafeEncodeUtf+{-# INLINE name #-} instance Show FileSelector where- show (Name x) = C.unpack x- show (Ext e) = "*." <> C.unpack e-+ show (Name x) = show x+ show (Ext e) = "*." <> show e+ show (Hdr e) = "*." <> show e -- utility functions readTypeList :: [String] -> ([FileType], [FileType], [FileType])-readTypeList = foldl run ([],[],[])- where run :: ([FileType], [FileType], [FileType]) -> String -> ([FileType], [FileType], [FileType])- run (l1, l2, l3) l- | '+':xs <- l = (l1, prettyRead xs "Type" : l2, l3)- | '-':xs <- l = (l1, l2, prettyRead xs "Type" : l3)- | otherwise = (prettyRead l "Type" : l1, l2, l3)+readTypeList = foldl run ([], [], [])+ where+ run :: ([FileType], [FileType], [FileType]) -> String -> ([FileType], [FileType], [FileType])+ run (l1, l2, l3) l+ | '+' : xs <- l = (l1, prettyRead xs "Type" : l2, l3)+ | '-' : xs <- l = (l1, l2, prettyRead xs "Type" : l3)+ | otherwise = (prettyRead l "Type" : l1, l2, l3) readKindList :: [String] -> [FileKind] readKindList = map (`prettyRead` "Kind")
src/CGrep/FileTypeMap.hs view
@@ -1,4906 +1,237 @@------ Copyright (c) 2013-2023 Nicola Bonelli <nicola@larthia.com>------ This program is free software; you can redistribute it and/or modify--- it under the terms of the GNU General Public License as published by--- the Free Software Foundation; either version 2 of the License, or--- (at your option) any later version.------ This program is distributed in the hope that it will be useful,--- but WITHOUT ANY WARRANTY; without even the implied warranty of--- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the--- GNU General Public License for more details.------ You should have received a copy of the GNU General Public License--- along with this program; if not, write to the Free Software--- Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.-----{-# LANGUAGE TupleSections #-}--module CGrep.FileTypeMap (- FileTypeInfo (..),- FileTypeInfoMap (..),- WordType (..),- CharIdentifierF,- fileTypeLookup,- fileTypeInfoLookup,- fileTypeInfoMap,- dumpFileTypeInfoMap,- contextFilter,-) where--import CGrep.ContextFilter (- ContextFilter,- FilterFunction,- isContextFilterAll,- mkParConfig,- runContextFilter,- )-import CGrep.FileType (FileSelector (..), FileType (..))--import CGrep.Types (Text8)-import Control.Applicative (Alternative ((<|>)))-import Control.Monad (forM_)-import qualified Data.ByteString.Char8 as C-import qualified Data.Map as Map-import Data.Maybe (fromJust, isJust)-import Options (Options (Options, keyword, type_force))--import qualified Data.Array.BitArray as BA--import Data.List (findIndex)-import qualified Data.Set as S--import CGrep.Boundary (Boundary (Boundary))-import CGrep.Parser.Char (- isAlphaNum_,- isAlphaNum_',- isAlphaNum_and,- isAlpha_,- isAlpha_',- isAlpha_and,- )-import CGrep.Parser.Char (isAlpha, isAlphaNum)--import System.Posix.FilePath (RawFilePath, takeBaseName, takeExtension, takeFileName)--import Data.Aeson (Value (Bool, String))-import Data.Bits (Bits (isSigned))-import qualified Data.HashMap.Strict as HM-import Data.MonoTraversable (WrappedPoly)-import GHC.Conc (BlockReason (BlockedOnBlackHole))-import CGrep.FileKind--newtype FileTypeInfoMap = FileTypeInfoMap {- unMapInfo :: Map.Map FileType FileTypeInfo-}--newtype FileTypeMap = FileTypeMap {- unMap :: Map.Map FileSelector (FileType, FileKind)-}--type CharIdentifierF = (Char -> Bool)--data WordType = Keyword | NativeType- deriving stock (Eq, Ord, Show)--data FileTypeInfo = FileTypeInfo- { ftSelector :: [FileSelector]- , ftKind :: FileKind- , ftChar :: [Boundary]- , ftString :: [Boundary]- , ftRawString :: [Boundary]- , ftComment :: [Boundary]- , ftIdentifierChars :: Maybe (CharIdentifierF, CharIdentifierF)- , ftKeywords :: HM.HashMap C.ByteString WordType- }--fileTypeInfoMap :: FileTypeInfoMap-fileTypeInfoMap = FileTypeInfoMap $- Map.fromList- [- ( Agda- , FileTypeInfo- { ftSelector = [Ext "agda", Ext "lagda"]- , ftKind = KindLanguage- , ftComment = ["{-" ~~ "-}", "--" ~~ "\n"]- , ftChar = ["'" ~~ "'"]- , ftString = ["\"" ~~ "\""]- , ftRawString = []- , ftIdentifierChars = Nothing- , ftKeywords =- reserved- [ "abstract"- , "codata"- , "constructor"- , "data"- , "eta-equality"- , "field"- , "forall"- , "hiding"- , "import"- , "in"- , "inductive"- , "infix"- , "infixl"- , "infixr"- , "instance"- , "let"- , "module"- , "mutual"- , "no-eta-equality"- , "open"- , "pattern"- , "postulate"- , "primitive"- , "private"- , "public"- , "quoteContext"- , "quoteGoal"- , "record"- , "renaming"- , "rewrite"- , "Set"- , "syntax"- , "tactic"- , "using"- , "where"- , "with"- ]- }- )- ,- ( Assembly- , FileTypeInfo- { ftSelector = [Ext "s", Ext "S"]- , ftKind = KindLanguage- , ftComment = ["#" ~~ "\n", ";" ~~ "\n", "|" ~~ "\n", "!" ~~ "\n", "/*" ~~ "*/"]- , ftChar = []- , ftString = ["\"" ~~ "\""]- , ftRawString = []- , ftIdentifierChars = Nothing- , ftKeywords = HM.empty- }- )- ,- ( Awk- , FileTypeInfo- { ftSelector = [Ext "awk", Ext "mawk", Ext "gawk"]- , ftKind = KindScript- , ftComment = ["{-" ~~ "-}", "--" ~~ "\n"]- , ftChar = []- , ftString = ["\"" ~~ "\""]- , ftRawString = []- , ftIdentifierChars = Nothing- , ftKeywords =- reserved- [ "BEGIN"- , "END"- , "if"- , "else"- , "while"- , "do"- , "for"- , "in"- , "break"- , "continue"- , "delete"- , "next"- , "nextfile"- , "function"- , "func"- , "exit"- ]- }- )- ,- ( Bash- , FileTypeInfo- { ftSelector = [Ext "sh", Ext "bash"]- , ftKind = KindScript- , ftComment = ["#" ~~ "\n"]- , ftChar = []- , ftString = ["'" ~~ "'", "\"" ~~ "\""]- , ftRawString = []- , ftIdentifierChars = Nothing- , ftKeywords =- reserved- [ "if"- , "then"- , "elif"- , "else"- , "fi"- , "time"- , "for"- , "in"- , "until"- , "while"- , "do"- , "done"- , "case"- , "esac"- , "coproc"- , "select"- , "function"- ]- }- )- ,- ( C- , FileTypeInfo- { ftSelector = [Ext "c", Ext "C", Ext "inc"]- , ftKind = KindLanguage- , ftComment = ["/*" ~~ "*/", "//" ~~ "\n"]- , ftChar = ["'" ~~ "'"]- , ftString = ["\"" ~~ "\""]- , ftRawString = ["R\"" ~~ "\""]- , ftIdentifierChars = Nothing- , ftKeywords =- types- [ "int"- , "short"- , "long"- , "float"- , "double"- , "char"- , "bool"- , "int8_t"- , "int16_t"- , "int32_t"- , "int64_t"- , "int_fast8_t"- , "int_fast16_t"- , "int_fast32_t"- , "int_fast64_t"- , "int_least8_t"- , "int_least16_t"- , "int_least32_t"- , "int_least64_t"- , "intmax_t"- , "intptr_t"- , "uint8_t"- , "uint16_t"- , "uint32_t"- , "uint64_t"- , "uint_fast8_t"- , "uint_fast16_t"- , "uint_fast32_t"- , "uint_fast64_t"- , "uint_least8_t"- , "uint_least16_t"- , "uint_least32_t"- , "uint_least64_t"- , "uintmax_t"- , "uintptr_t"- ]- <> reserved- [ "auto"- , "break"- , "case"- , "const"- , "continue"- , "default"- , "do"- , "else"- , "enum"- , "extern"- , "for"- , "goto"- , "if"- , "inline"- , "register"- , "restrict"- , "return"- , "signed"- , "sizeof"- , "static"- , "struct"- , "switch"- , "typedef"- , "union"- , "unsigned"- , "void"- , "volatile"- , "while"- , "_Alignas"- , "_Alignof"- , "_Atomic"- , "_Bool"- , "_Complex"- , "_Decimal128"- , "_Decimal32"- , "_Decimal64"- , "_Generic"- , "_Imaginary"- , "_Noreturn"- , "_Static_assert"- , "_Thread_local"- , "if"- , "elif"- , "else"- , "endif"- , "ifdef"- , "ifndef"- , "define"- , "undef"- , "include"- , "line"- , "error"- , "pragma"- , "defined"- , "__has_c_attribute"- , "_Pragma"- , "asm"- , "fortran"- ]- }- )- ,- ( CMake- , FileTypeInfo- { ftSelector = [Name "CMakeLists.txt", Ext "cmake"]- , ftKind = KindScript- , ftComment = ["#" ~~ "\n"]- , ftChar = ["'" ~~ "'"]- , ftString = ["\"" ~~ "\""]- , ftRawString = []- , ftIdentifierChars = Nothing- , ftKeywords = HM.empty- }- )- ,- ( Cabal- , FileTypeInfo- { ftSelector = [Ext "cabal"]- , ftKind = KindConfig- , ftComment = ["--" ~~ "\n"]- , ftChar = []- , ftString = ["\"" ~~ "\""]- , ftRawString = []- , ftIdentifierChars = Just (const False, const False)- , ftKeywords = HM.empty- }- )- ,- ( Chapel- , FileTypeInfo- { ftSelector = [Ext "chpl"]- , ftKind = KindLanguage- , ftComment = ["/*" ~~ "*/", "//" ~~ "\n"]- , ftChar = []- , ftString = ["\"" ~~ "\"", "'" ~~ "'"]- , ftRawString = ["\"\"\"" ~~ "\"\"\"", "'''" ~~ "'''"]- , ftIdentifierChars = Just (isAlpha_, isAlphaNum_and "$")- , ftKeywords =- types- [ "void"- , "nothing"- , "bool"- , "int"- , "uint"- , "real"- , "imag"- , "complex"- , "string"- , "bytes"- ]- <> reserved- [ "align"- , "as"- , "atomic"- , "begin"- , "bool"- , "borrowed"- , "break"- , "by"- , "catch"- , "class"- , "cobegin"- , "coforall"- , "config"- , "const"- , "continue"- , "defer"- , "delete"- , "dmapped"- , "do"- , "domain"- , "else"- , "enum"- , "except"- , "export"- , "extern"- , "false"- , "for"- , "forall"- , "forwarding"- , "if"- , "in"- , "index"- , "inline"- , "inout"- , "iter"- , "label"- , "let"- , "lifetime"- , "local"- , "locale"- , "module"- , "new"- , "nil"- , "noinit"- , "on"- , "only"- , "otherwise"- , "out"- , "override"- , "owned"- , "param"- , "private"- , "prototype"- , "proc"- , "public"- , "record"- , "reduce"- , "ref"- , "require"- , "return"- , "scan"- , "select"- , "serial"- , "shared"- , "single"- , "sparse"- , "subdomain"- , "sync"- , "then"- , "this"- , "throw"- , "throws"- , "true"- , "try"- , "type"- , "union"- , "unmanaged"- , "use"- , "var"- , "when"- , "where"- , "while"- , "with"- , "yield"- , "zip"- ]- }- )- ,- ( Clojure- , FileTypeInfo- { ftSelector = [Ext "clj", Ext "cljs", Ext "cljc", Ext "edn"]- , ftKind = KindLanguage- , ftComment = [";" ~~ "\n"]- , ftChar = []- , ftString = ["\"" ~~ "\""]- , ftRawString = []- , ftIdentifierChars = Just (isAlpha, isAlphaNum_and "*+!-_?")- , ftKeywords =- types- [ "nil"- , "boolean"- , "char"- , "byte"- , "short"- , "int"- , "long"- , "float"- , "double"- ]- <> reserved- [ "and"- , "let"- , "def"- , "defn"- , "if"- , "else"- , "do"- , "quote"- , "var"- , "fn"- , "loop"- , "recur"- , "throw"- , "try"- , "monitor-enter"- , "monitor-exit"- ]- }- )- ,- ( Coffee- , FileTypeInfo- { ftSelector = [Ext "coffee"]- , ftKind = KindScript- , ftComment = ["#" ~~ "\n", "###" ~~ "###"]- , ftChar = []- , ftString = ["'" ~~ "'", "\"" ~~ "\""]- , ftRawString = ["'''" ~~ "'''", "\"\"\"" ~~ "\"\"\""]- , ftIdentifierChars = Just (isAlpha_and "$", isAlphaNum_and "$")- , ftKeywords =- reserved- [ "case"- , "default"- , "function"- , "var"- , "void"- , "with"- , "const"- , "let"- , "enum"- , "export"- , "import"- , "native"- , "__hasProp"- , "__extends"- , "__slice"- , "__bind"- , "__indexOf"- , "implements"- , "interface"- , "package"- , "private"- , "protected"- , "public"- , "static"- , "yield"- , "true"- , "false"- , "null"- , "this"- , "new"- , "delete"- , "typeof"- , "in"- , "arguments"- , "eval"- , "instanceof"- , "return"- , "throw"- , "break"- , "continue"- , "debugger"- , "if"- , "else"- , "switch"- , "for"- , "while"- , "do"- , "try"- , "catch"- , "finally"- , "class"- , "extends"- , "super"- , "undefined"- , "then"- , "unless"- , "until"- , "loop"- , "of"- , "by"- , "when"- , "and"- , "or"- , "is"- , "isnt"- , "not"- , "yes"- , "no"- , "on"- , "off"- ]- }- )- ,- ( Conf- , FileTypeInfo- { ftSelector = [Ext "config", Ext "conf", Ext "cfg", Ext "doxy"]- , ftKind = KindConfig- , ftComment = ["#" ~~ "\n"]- , ftChar = []- , ftString = ["'" ~~ "'", "\"" ~~ "\""]- , ftRawString = []- , ftIdentifierChars = Nothing- , ftKeywords = HM.empty- }- )- ,- ( Cpp- , FileTypeInfo- { ftSelector =- [ Ext "cpp"- , Ext "CPP"- , Ext "cxx"- , Ext "cc"- , Ext "cp"- , Ext "c++"- , Ext "tcc"- , Ext "h"- , Ext "H"- , Ext "hpp"- , Ext "ipp"- , Ext "HPP"- , Ext "hxx"- , Ext "hh"- , Ext "hp"- , Ext "h++"- , Ext "cu"- , Ext "cuh"- ]- , ftKind = KindLanguage- , ftComment = ["/*" ~~ "*/", "//" ~~ "\n"]- , ftChar = ["'" ~~ "'"]- , ftString = ["\"" ~~ "\""]- , ftRawString = ["R\"(" ~~ ")\"", "R\"-(" ~~ ")-\"", "R\"--(" ~~ ")--\""]- , ftIdentifierChars = Nothing- , ftKeywords =- types- [ "int"- , "short"- , "long"- , "float"- , "double"- , "char"- , "bool"- , "int8_t"- , "int16_t"- , "int32_t"- , "int64_t"- , "int_fast8_t"- , "int_fast16_t"- , "int_fast32_t"- , "int_fast64_t"- , "int_least8_t"- , "int_least16_t"- , "int_least32_t"- , "int_least64_t"- , "intmax_t"- , "intptr_t"- , "uint8_t"- , "uint16_t"- , "uint32_t"- , "uint64_t"- , "uint_fast8_t"- , "uint_fast16_t"- , "uint_fast32_t"- , "uint_fast64_t"- , "uint_least8_t"- , "uint_least16_t"- , "uint_least32_t"- , "uint_least64_t"- , "uintmax_t"- , "uintptr_t"- , "nullptr_t"- ]- <> reserved- [ "alignas"- , "alignof"- , "and"- , "and_eq"- , "asm"- , "atomic_cancel"- , "atomic_commit"- , "atomic_noexcept"- , "auto"- , "bitand"- , "bitor"- , "break"- , "case"- , "catch"- , "class"- , "compl"- , "concept"- , "const"- , "consteval"- , "constexpr"- , "constinit"- , "const_cast"- , "continue"- , "co_await"- , "co_return"- , "co_yield"- , "decltype"- , "default"- , "delete"- , "do"- , "dynamic_cast"- , "else"- , "enum"- , "explicit"- , "export"- , "extern"- , "false"- , "for"- , "friend"- , "goto"- , "if"- , "inline"- , "mutable"- , "namespace"- , "new"- , "noexcept"- , "not"- , "not_eq"- , "nullptr"- , "operator"- , "or"- , "or_eq"- , "private"- , "protected"- , "public"- , "reflexpr"- , "register"- , "reinterpret_cast"- , "requires"- , "return"- , "signed"- , "sizeof"- , "static"- , "static_assert"- , "static_cast"- , "struct"- , "switch"- , "synchronized"- , "template"- , "this"- , "thread_local"- , "throw"- , "true"- , "try"- , "typedef"- , "typeid"- , "typename"- , "union"- , "unsigned"- , "using"- , "virtual"- , "void"- , "volatile"- , "wchar_t"- , "while"- , "xor"- , "xor_eq"- , "final"- , "override"- , "transaction_safe"- , "transaction_safe_dynamic"- , "import"- , "module"- , "elif"- , "endif"- , "ifdef"- , "ifndef"- , "define"- , "undef"- , "include"- , "line"- , "error"- , "pragma"- , "defined"- , "__has_include"- , "__has_cpp_attribute"- , "export"- , "import"- , "module"- ]- }- )- ,- ( Csharp- , FileTypeInfo- { ftSelector = [Ext "cs", Ext "CS"]- , ftKind = KindLanguage- , ftComment = ["/*" ~~ "*/", "//" ~~ "\n"]- , ftChar = ["'" ~~ "'"]- , ftString = ["\"" ~~ "\""]- , ftRawString = ["\"\"\"" ~~ "\"\"\""]- , ftIdentifierChars = Nothing- , ftKeywords =- types- [ "bool"- , "byte"- , "sbyte"- , "char"- , "decimal"- , "double"- , "float"- , "int"- , "uint"- , "nint"- , "nuint"- , "long"- , "ulong"- , "short"- , "ushort"- ]- <> reserved- [ "abstract"- , "as"- , "base"- , "break"- , "case"- , "catch"- , "checked"- , "class"- , "const"- , "continue"- , "default"- , "delegate"- , "do"- , "else"- , "enum"- , "event"- , "explicit"- , "extern"- , "false"- , "finally"- , "fixed"- , "for"- , "foreach"- , "goto"- , "if"- , "implicit"- , "in"- , "interface"- , "internal"- , "is"- , "lock"- , "namespace"- , "new"- , "null"- , "object"- , "operator"- , "out"- , "override"- , "params"- , "private"- , "protected"- , "public"- , "readonly"- , "ref"- , "return"- , "sealed"- , "sizeof"- , "stackalloc"- , "static"- , "string"- , "struct"- , "switch"- , "this"- , "throw"- , "true"- , "try"- , "typeof"- , "unchecked"- , "unsafe"- , "using"- , "virtual"- , "void"- , "volatile"- , "while"- ]- }- )- ,- ( Csh- , FileTypeInfo- { ftSelector = [Ext "csh", Ext "tcsh"]- , ftKind = KindScript- , ftComment = ["#" ~~ "\n"]- , ftChar = []- , ftString = ["'" ~~ "'", "\"" ~~ "\""]- , ftRawString = []- , ftIdentifierChars = Nothing- , ftKeywords =- reserved- [ "alias"- , "alloc"- , "bg"- , "bindkey"- , "break"- , "breaksw"- , "built-ins"- , "bye"- , "case"- , "cd"- , "chdir"- , "complete"- , "continue"- , "default"- , "dirs"- , "echo"- , "echotc"- , "else"- , "end"- , "endif"- , "endsw"- , "eval"- , "exec"- , "exit"- , "fg"- , "filetest"- , "foreach"- , "glob"- , "goto"- , "hashstat"- , "history"- , "hup"- , "if"- , "jobs"- , "kill"- , "limit"- , "log"- , "login"- , "logout"- , "ls-F"- , "newgrp"- , "nice"- , "nohup"- , "notify"- , "onintr"- , "popd"- , "printenv"- , "pushd"- , "rehash"- , "repeat"- , "sched"- , "set"- , "setenv"- , "settc"- , "setty"- , "shift"- , "source"- , "stop"- , "suspend"- , "switch"- , "telltc"- , "time"- , "umask"- , "unalias"- , "uncomplete"- , "unhash"- , "unlimit"- , "unset"- , "unsetenv"- , "wait"- , "watchlog"- , "where"- , "which"- , "while"- ]- }- )- ,- ( Css- , FileTypeInfo- { ftSelector = [Ext "css"]- , ftKind = KindMarkup- , ftComment = ["/*" ~~ "*/"]- , ftChar = []- , ftString = ["\"" ~~ "\""]- , ftRawString = []- , ftIdentifierChars = Just (isAlpha_and "-", isAlphaNum_and "-")- , ftKeywords = HM.empty- }- )- ,- ( Cql- , FileTypeInfo- { ftSelector = [Ext "cql"]- , ftKind = KindLanguage- , ftComment = ["--" ~~ "\n"]- , ftChar = []- , ftString = ["'" ~~ "'"]- , ftRawString = []- , ftIdentifierChars = Nothing- , ftKeywords =- reserved- [ "ASCII"- , "ADD"- , "AGGREGATE"- , "ALL"- , "ALLOW"- , "ALTER"- , "AND"- , "ANY"- , "APPLY"- , "AS"- , "ASC"- , "AUTHORIZE"- , "BATCH"- , "BEGIN"- , "BIGINT"- , "BY"- , "CLUSTERING"- , "COLUMNFAMILY"- , "COMPACT"- , "CONSISTENCY"- , "COUNT"- , "COUNTER"- , "CREATE"- , "CUSTOM"- , "DATE"- , "DECIMAL"- , "DELETE"- , "DESC"- , "DISTINCT"- , "DOUBLE"- , "DROP"- , "EACH_QUORUM"- , "ENTRIES"- , "EXISTS"- , "FILTERING"- , "FLOAT"- , "FROM"- , "FROZEN"- , "FULL"- , "GRANT"- , "IF"- , "IN"- , "INDEX"- , "INET"- , "INFINITY"- , "INSERT"- , "INT"- , "INTO"- , "KEY"- , "KEYSPACE"- , "KEYSPACES"- , "LEVEL"- , "LIMIT"- , "LIST"- , "LOCAL_ONE"- , "LOCAL_QUORUM"- , "MAP"- , "MATERIALIZED"- , "MODIFY"- , "NAN"- , "OF"- , "ON"- , "ONE"- , "ORDER"- , "PARTITION"- , "PASSWORD"- , "PER"- , "PERMISSION"- , "PERMISSIONS"- , "PRIMARY"- , "QUORUM"- , "RECURSIVE"- , "RENAME"- , "REVOKE"- , "SCHEMA"- , "SELECT"- , "SET"- , "SMALLINT"- , "STATIC"- , "STORAGE"- , "SUPERUSER"- , "T"- , "TABLE"- , "TEXT"- , "THREE"- , "TIME"- , "TIMESTAMP"- , "TIMEUUID"- , "TINYINT"- , "TO"- , "TOKEN"- , "TRUNCATE"- , "TTL"- , "TUPLE"- , "TWO"- , "TYPE"- , "UNLOGGED"- , "UPDATE"- , "USE"- , "USER"- , "USERS"- , "USING"- , "UUID"- , "VALUES"- , "VARCHAR"- , "VARINT"- , "VIEW"- , "WHERE"- , "WITH"- , "WRITETIME"- ]- }- )- ,- ( D- , FileTypeInfo- { ftSelector = [Ext "d", Ext "D"]- , ftKind = KindLanguage- , ftComment = ["/*" ~~ "*/", "//" ~~ "\n"]- , ftChar = ["'" ~~ "'"]- , ftString = ["\"" ~~ "\""]- , ftRawString = ["r\"" ~~ "\"", "`" ~~ "`"]- , ftIdentifierChars = Nothing- , ftKeywords =- types- [ "bool"- , "byte"- , "ubyte"- , "short"- , "ushort"- , "int"- , "uint"- , "long"- , "ulong"- , "cent"- , "ucent"- , "char"- , "wchar"- , "dchar"- , "float"- , "double"- , "real"- , "ifloat"- , "idouble"- , "ireal"- , "cfloat"- , "cdouble"- , "creal"- , "void"- ]- <> reserved- [ "abstract"- , "alias"- , "align"- , "asm"- , "assert"- , "auto"- , "body"- , "break"- , "case"- , "cast"- , "catch"- , "class"- , "const"- , "continue"- , "debug"- , "default"- , "delegate"- , "delete"- , "deprecated"- , "do"- , "else"- , "enum"- , "export"- , "extern"- , "false"- , "final"- , "finally"- , "for"- , "foreach"- , "foreach_reverse"- , "function"- , "goto"- , "if"- , "immutable"- , "import"- , "in"- , "inout"- , "interface"- , "invariant"- , "is"- , "lazy"- , "macro"- , "mixin"- , "module"- , "new"- , "nothrow"- , "null"- , "out"- , "override"- , "package"- , "pragma"- , "private"- , "protected"- , "public"- , "pure"- , "ref"- , "return"- , "scope"- , "shared"- , "static"- , "struct"- , "super"- , "switch"- , "synchronized"- , "template"- , "this"- , "throw"- , "true"- , "try"- , "typeid"- , "typeof"- , "union"- , "unittest"- , "version"- , "while"- , "with"- , "__FILE__"- , "__FILE_FULL_PATH__"- , "__MODULE__"- , "__LINE__"- , "__FUNCTION__"- , "__PRETTY_FUNCTION__"- , "__gshared"- , "__traits"- , "__vector"- , "__parameters"- ]- }- )- ,- ( Dart- , FileTypeInfo- { ftSelector = [Ext "dart"]- , ftKind = KindLanguage- , ftComment = ["/*" ~~ "*/", "//" ~~ "\n"]- , ftChar = []- , ftString = ["\"" ~~ "\"", "'" ~~ "'"]- , ftRawString = ["'''" ~~ "'''", "\"\"\"" ~~ "\"\"\""]- , ftIdentifierChars = Nothing- , ftKeywords =- types ["int", "double", "num", "bool", "null"]- <> reserved- [ "assert"- , "break"- , "case"- , "catch"- , "class"- , "const"- , "continue"- , "default"- , "do"- , "else"- , "enum"- , "extends"- , "false"- , "final"- , "finally"- , "for"- , "if"- , "in"- , "is"- , "new"- , "rethrow"- , "return"- , "super"- , "switch"- , "this"- , "throw"- , "true"- , "try"- , "var"- , "void"- , "while"- , "with"- , "async"- , "hide"- , "on"- , "show"- , "sync"- , "abstract"- , "as"- , "covariant"- , "deferred"- , "dynamic"- , "export"- , "extension"- , "external"- , "factory"- , "function"- , "get"- , "implements"- , "import"- , "interface"- , "library"- , "mixin"- , "operator"- , "part"- , "set"- , "static"- , "typedef"- , "await"- , "yield"- ]- }- )- ,- ( Dhall- , FileTypeInfo- { ftSelector = [Ext "dhall"]- , ftKind = KindConfig- , ftComment = ["{-" ~~ "-}", "--" ~~ "\n"]- , ftChar = []- , ftString = ["\"" ~~ "\""]- , ftRawString = []- , ftIdentifierChars = Just (isAlpha_', isAlphaNum_')- , ftKeywords =- types ["Bool", "Natural", "Integer", "Double", "Text", "Date", "Time", "List", "Optional"]- <> reserved- [ "if"- , "then"- , "else"- , "toMap"- , "with"- , "merge"- , "showConstructor"- , "missing"- , "as"- , "using"- , "let"- , "assert"- ]- }- )- ,- ( Elixir- , FileTypeInfo- { ftSelector = [Ext "ex", Ext "exs"]- , ftKind = KindLanguage- , ftComment = ["#" ~~ "\n"]- , ftChar = []- , ftString = ["\"" ~~ "\"", "'" ~~ "'"]- , ftRawString = []- , ftIdentifierChars = Nothing- , ftKeywords =- types- [ "integer"- , "float"- , "boolean"- , "atom"- , "binary"- , "string"- , "list"- , "tuple"- , "map"- ]- <> reserved- [ "true"- , "false"- , "nil"- , "when"- , "and"- , "or"- , "not"- , "in"- , "fn"- , "do"- , "end"- , "catch"- , "rescue"- , "after"- , "else"- ]- }- )- ,- ( Elm- , FileTypeInfo- { ftSelector = [Ext "elm"]- , ftKind = KindLanguage- , ftComment = ["{-" ~~ "-}", "--" ~~ "\n"]- , ftChar = ["'" ~~ "'"]- , ftString = ["\"" ~~ "\""]- , ftRawString = ["\"\"\"" ~~ "\"\"\""]- , ftIdentifierChars = Just (isAlpha, isAlphaNum_)- , ftKeywords =- types- [ "Bool"- , "Int"- , "Float"- , "String"- , "Char"- , "List"- , "Maybe"- , "Result"- , "Order"- , "Never"- , "Html"- , "msg"- , "Cmd"- , "Sub"- ]- <> reserved- [ "type"- , "alias"- , "port"- , "if"- , "then"- , "else"- , "case"- , "of"- , "let"- , "in"- , "infix"- , "left"- , "right"- , "non"- , "module"- , "import"- , "exposing"- , "as"- , "where"- , "effect"- , "command"- , "subscription"- , "true"- , "false"- , "null"- ]- }- )- ,- ( Erlang- , FileTypeInfo- { ftSelector = [Ext "erl", Ext "ERL", Ext "hrl", Ext "HRL"]- , ftKind = KindLanguage- , ftComment = ["%" ~~ "\n"]- , ftChar = []- , ftString = ["\"" ~~ "\""]- , ftRawString = []- , ftIdentifierChars = Nothing- , ftKeywords =- types- [ "Atom"- , "Integer"- , "Float"- , "Boolean"- , "String"- , "Tuple"- , "List"- , "Function"- , "Binary"- , "PID"- , "Port"- , "Reference"- , "Map"- ]- <> reserved- [ "after"- , "and"- , "andalso"- , "band"- , "begin"- , "bnot"- , "bor"- , "bsl"- , "bsr"- , "bxor"- , "case"- , "catch"- , "cond"- , "div"- , "end"- , "fun"- , "if"- , "let"- , "not"- , "of"- , "or"- , "orelse"- , "receive"- , "rem"- , "try"- , "when"- , "xor"- ]- }- )- ,- ( Fish- , FileTypeInfo- { ftSelector = [Ext "fish"]- , ftKind = KindScript- , ftComment = ["#" ~~ "\n"]- , ftChar = []- , ftString = ["'" ~~ "'", "\"" ~~ "\""]- , ftRawString = []- , ftIdentifierChars = Nothing- , ftKeywords =- reserved- [ "and"- , "argparse"- , "begin"- , "break"- , "builtin"- , "case"- , "command"- , "continue"- , "else"- , "end"- , "eval"- , "exec"- , "for"- , "function"- , "if"- , "not"- , "or"- , "read"- , "return"- , "set"- , "status"- , "string"- , "switch"- , "test"- , "time"- , "while"- ]- }- )- ,- ( Fortran- , FileTypeInfo- { ftSelector =- [ Ext "f"- , Ext "for"- , Ext "ftn"- , Ext "F"- , Ext "FOR"- , Ext "FTN"- , Ext "fpp"- , Ext "FPP"- , Ext "f90"- , Ext "f95"- , Ext "f03"- , Ext "f08"- , Ext "F90"- , Ext "F95"- , Ext "F03"- , Ext "F08"- ]- , ftKind = KindLanguage- , ftComment = ["!" ~~ "\n"]- , ftChar = []- , ftString = ["\"" ~~ "\"", "'" ~~ "'"]- , ftRawString = []- , ftIdentifierChars = Nothing- , ftKeywords =- types ["integer", "real", "complex", "logical", "character"]- <> reserved- [ -- fortran77- "assign"- , "backspace"- , "block"- , "data"- , "call"- , "close"- , "common"- , "continue"- , "data"- , "dimension"- , "do"- , "else"- , "else"- , "if"- , "end"- , "endfile"- , "endif"- , "entry"- , "equivalence"- , "external"- , "format"- , "function"- , "goto"- , "if"- , "implicit"- , "inquire"- , "intrinsic"- , "open"- , "parameter"- , "pause"- , "print"- , "program"- , "read"- , "return"- , "rewind"- , "rewrite"- , "save"- , "stop"- , "subroutine"- , "then"- , "write"- , -- fortran 90- "allocatable"- , "allocate"- , "case"- , "contains"- , "cycle"- , "deallocate"- , "elsewhere"- , "exit?"- , "include"- , "interface"- , "intent"- , "module"- , "namelist"- , "nullify"- , "only"- , "operator"- , "optional"- , "pointer"- , "private"- , "procedure"- , "public"- , "recursive"- , "result"- , "select"- , "sequence"- , "target"- , "use"- , "while"- , "where"- , -- fortran 95- "elemental"- , "forall"- , "pure"- , -- fortran 03- "abstract"- , "associate"- , "asynchronous"- , "bind"- , "class"- , "deferred"- , "enum"- , "enumerator"- , "extends"- , "final"- , "flush"- , "generic"- , "import"- , "non_overridable"- , "nopass"- , "pass"- , "protected"- , "value"- , "volatile"- , "wait"- , -- fortran 08- "block"- , "codimension"- , "do"- , "concurrent"- , "contiguous"- , "critical"- , "error"- , "stop"- , "submodule"- , "sync"- , "all"- , "sync"- , "images"- , "sync"- , "memory"- , "lock"- , "unlock"- ]- }- )- ,- ( Fsharp- , FileTypeInfo- { ftSelector = [Ext "fs", Ext "fsx", Ext "fsi"]- , ftKind = KindLanguage- , ftComment = ["(*" ~~ "*)", "//" ~~ "\n"]- , ftChar = ["'" ~~ "'"]- , ftString = ["\"" ~~ "\""]- , ftRawString = ["\"\"\"" ~~ "\"\"\""]- , ftIdentifierChars = Just (isAlpha_and "$@`?", isAlphaNum_and "$@`?")- , ftKeywords =- types- [ "bool"- , "byte"- , "sbyte"- , "int16"- , "uint16"- , "int"- , "uint"- , "int64"- , "uint64"- , "nativeint"- , "unativeint"- , "decimal"- , "float"- , "double"- , "float32"- , "single"- , "char"- , "string"- ]- <> reserved- [ "abstract"- , "and"- , "as"- , "assert"- , "base"- , "begin"- , "class"- , "default"- , "delegate"- , "do"- , "done"- , "downcast"- , "downto"- , "elif"- , "else"- , "end"- , "exception"- , "extern"- , "FALSE"- , "finally"- , "fixed"- , "for"- , "fun"- , "function"- , "global"- , "if"- , "in"- , "inherit"- , "inline"- , "interface"- , "internal"- , "lazy"- , "let"- , "let!"- , "match"- , "match!"- , "member"- , "module"- , "mutable"- , "namespace"- , "new"- , "not"- , "null"- , "of"- , "open"- , "or"- , "override"- , "private"- , "public"- , "rec"- , "return"- , "return!"- , "select"- , "static"- , "struct"- , "then"- , "to"- , "TRUE"- , "try"- , "type"- , "upcast"- , "use"- , "use!"- , "val"- , "void"- , "when"- , "while"- , "with"- , "yield"- , "yield!"- , "const"- , "asr"- , "land"- , "lor"- , "lsl"- , "lsr"- , "lxor"- , "mod"- , "sig"- , "break"- , "checked"- , "component"- , "const"- , "constraint"- , "continue"- , "event"- , "external"- , "include"- , "mixin"- , "parallel"- , "process"- , "protected"- , "pure"- , "sealed"- , "tailcall"- , "trait"- , "virtual"- ]- }- )- ,- ( Go- , FileTypeInfo- { ftSelector = [Ext "go"]- , ftKind = KindLanguage- , ftComment = ["/*" ~~ "*/", "//" ~~ "\n"]- , ftChar = ["'" ~~ "'"]- , ftString = ["\"" ~~ "\""]- , ftRawString = ["`" ~~ "`"]- , ftIdentifierChars = Nothing- , ftKeywords =- types- [ "int8"- , "int16"- , "int32"- , "int64"- , "uint8"- , "uint16"- , "uint32"- , "uint64"- , "int"- , "uint"- , "rune"- , "byte"- , "uintptr"- , "float32"- , "float64"- , "complex64"- , "complex128"- , "string"- , "error"- , "bool"- ]- <> reserved- [ "break"- , "default"- , "func"- , "interface"- , "select"- , "case"- , "defer"- , "go"- , "map"- , "struct"- , "chan"- , "else"- , "goto"- , "package"- , "switch"- , "const"- , "fallthrough"- , "if"- , "range"- , "type"- , "continue"- , "for"- , "import"- , "return"- , "var"- , "append"- , "cap"- , "close"- , "copy"- , "false"- , "float32"- , "float64"- , "imag"- , "iota"- , "len"- , "make"- , "new"- , "nil"- , "panic"- , "print"- , "println"- , "real"- , "recover"- , "true"- ]- }- )- ,- ( GoMod- , FileTypeInfo- { ftSelector = [Name "go.mod"]- , ftKind = KindConfig- , ftComment = ["//" ~~ "\n"]- , ftChar = []- , ftString = ["\"" ~~ "\""]- , ftRawString = []- , ftIdentifierChars = Just (const False, const False)- , ftKeywords =- reserved- [ "module"- , "go"- , "require"- , "exclude"- , "replace"- , "retract"- ]- }- )- ,- ( Haskell- , FileTypeInfo- { ftSelector = [Ext "hs", Ext "lhs", Ext "hsc"]- , ftKind = KindLanguage- , ftComment = ["{-" ~~ "-}", "--" ~~ "\n"]- , ftChar = ["'" ~~ "'"]- , ftString = ["\"" ~~ "\""]- , ftRawString = ["[r|" ~~ "|]", "[q|" ~~ "|]", "[s|" ~~ "|]", "[here|" ~~ "|]", "[i|" ~~ "|]"]- , ftIdentifierChars = Just (isAlpha_', isAlphaNum_')- , ftKeywords =- types- [ "Bool"- , "Char"- , "String"- , "ByteString"- , "Text"- , "Int"- , "Int8"- , "Int16"- , "Int32"- , "Int64"- , "Word8"- , "Word16"- , "Word32"- , "Word64"- , "Integer"- , "Float"- , "Double"- , "Complex"- ]- <> reserved- [ "as"- , "case"- , "class"- , "data"- , "default"- , "deriving"- , "do"- , "else"- , "hiding"- , "if"- , "import"- , "in"- , "infix"- , "infixl"- , "infixr"- , "instance"- , "let"- , "module"- , "newtype"- , "of"- , "qualified"- , "then"- , "type"- , "where"- , "forall"- , "mdo"- , "family"- , "role"- , "pattern"- , "static"- , "group"- , "by"- , "using"- , "foreign"- , "export"- , "label"- , "dynamic"- , "safe"- , "interruptible"- , "unsafe"- , "stdcall"- , "ccall"- , "capi"- , "prim"- , "javascript"- , "rec"- , "proc"- ]- }- )- ,- ( Html- , FileTypeInfo- { ftSelector = [Ext "htm", Ext "html"]- , ftKind = KindMarkup- , ftComment = ["<!--" ~~ "-->"]- , ftChar = []- , ftString = ["\"" ~~ "\""]- , ftRawString = []- , ftIdentifierChars = Just (isAlpha_, isAlpha_and "-:.")- , ftKeywords = HM.empty- }- )- ,- ( Idris- , FileTypeInfo- { ftSelector = [Ext "idr", Ext "lidr"]- , ftKind = KindLanguage- , ftComment = ["{-" ~~ "-}", "--" ~~ "\n", "|||" ~~ "\n"]- , ftChar = ["'" ~~ "'"]- , ftString = ["\"" ~~ "\""]- , ftRawString = []- , ftIdentifierChars = Just (isAlpha_', isAlphaNum_')- , ftKeywords =- types ["Int", "Integer", "Double", "Char", "String", "Ptr", "Bool"]- <> reserved- [ "abstract"- , "auto"- , "codata"- , "data"- , "if"- , "parameters"- , "public"- , "then"- , "total"- , "using"- , "case"- , "class"- , "concrete"- , "covering"- , "default"- , "do"- , "else"- , "export"- , "failing"- , "forall"- , "implementation"- , "implicit"- , "import"- , "impossible"- , "in"- , "incomplete"- , "instance"- , "interface"- , "let"- , "module"- , "mutual"- , "namespace"- , "of"- , "open"- , "params"- , "partial"- , "postulate"- , "private"- , "proof"- , "public"- , "record"- , "return"- , "rewrite"- , "syntax"- , "then"- , "total"- , "unreachable"- , "where"- , "with"- , "using"- ]- }- )- ,- ( Java- , FileTypeInfo- { ftSelector = [Ext "java"]- , ftKind = KindLanguage- , ftComment = ["/*" ~~ "*/", "//" ~~ "\n"]- , ftChar = ["'" ~~ "'"]- , ftString = ["\"" ~~ "\""]- , ftRawString = []- , ftIdentifierChars = Just (isAlpha_and "$", isAlphaNum_and "$")- , ftKeywords =- types- [ "bytes"- , "char"- , "short"- , "int"- , "long"- , "float"- , "double"- , "boolean"- , "void"- ]- <> reserved- [ "abstract"- , "assert"- , "break"- , "case"- , "catch"- , "class"- , "continue"- , "const"- , "default"- , "do"- , "else"- , "enum"- , "exports"- , "extends"- , "final"- , "finally"- , "for"- , "goto"- , "if"- , "implements"- , "import"- , "instanceof"- , "int"- , "interface"- , "module"- , "native"- , "new"- , "package"- , "private"- , "protected"- , "public"- , "requires"- , "return"- , "static"- , "strictfp"- , "super"- , "switch"- , "synchronized"- , "this"- , "throw"- , "throws"- , "transient"- , "try"- , "var"- , "volatile"- , "while"- , "true"- , "false"- , "null"- ]- }- )- ,- ( Javascript- , FileTypeInfo- { ftSelector = [Ext "js"]- , ftKind = KindLanguage- , ftComment = ["/*" ~~ "*/", "//" ~~ "\n"]- , ftChar = ["'" ~~ "'"]- , ftString = ["\"" ~~ "\""]- , ftRawString = []- , ftIdentifierChars = Just (isAlpha_and "$", isAlphaNum_and "$")- , ftKeywords =- reserved- [ "abstract"- , "arguments"- , "await"- , "break"- , "byte"- , "case"- , "catch"- , "char"- , "class"- , "const"- , "continue"- , "debugger"- , "default"- , "delete"- , "do"- , "double"- , "else"- , "enum"- , "eval"- , "export"- , "extends"- , "false"- , "final"- , "finally"- , "float"- , "for"- , "function"- , "goto"- , "if"- , "implements"- , "import"- , "in"- , "instanceof"- , "int"- , "interface"- , "let"- , "long"- , "native"- , "new"- , "null"- , "package"- , "private"- , "protected"- , "public"- , "return"- , "short"- , "static"- , "super"- , "switch"- , "synchronized"- , "this"- , "throw"- , "throws"- , "transient"- , "true"- , "try"- , "typeof"- , "var"- , "volatile"- , "while"- , "with"- , "yield"- ]- }- )- ,- ( Json- , FileTypeInfo- { ftSelector = [Ext "json", Ext "ndjson"]- , ftKind = KindData- , ftComment = []- , ftChar = []- , ftString = ["\"" ~~ "\""]- , ftRawString = []- , ftIdentifierChars = Just (const False, const False)- , ftKeywords = HM.empty- }- )- ,- ( Julia- , FileTypeInfo- { ftSelector = [Ext "jl"]- , ftKind = KindLanguage- , ftComment = ["#" ~~ "\n", "#-" ~~ "-#"]- , ftChar = ["'" ~~ "'"]- , ftString = ["\"" ~~ "\""]- , ftRawString = ["\"\"\"" ~~ "\"\"\""]- , ftIdentifierChars = Nothing- , ftKeywords =- types- [ "Float16"- , "Float32"- , "Float64"- , "Bool"- , "Char"- , "Int8"- , "UInt8"- , "Int16"- , "UInt16"- , "Int32"- , "UInt32"- , "Int64"- , "UInt64"- , "Int128"- , "UInt128"- , -- super types- "Integer"- , "AbstractChar"- , "AbstractFloat"- , "Signed"- , "Unsigned"- ]- <> reserved- [ "baremodule"- , "begin"- , "break"- , "catch"- , "const"- , "continue"- , "do"- , "else"- , "elseif"- , "end"- , "export"- , "false"- , "finally"- , "for"- , "function"- , "global"- , "if"- , "import"- , "let"- , "local"- , "macro"- , "module"- , "quote"- , "return"- , "struct"- , "true"- , "try"- , "using"- , "while"- ]- }- )- ,- ( Kotlin- , FileTypeInfo- { ftSelector = [Ext "kt", Ext "kts", Ext "ktm"]- , ftKind = KindLanguage- , ftComment = ["/*" ~~ "*/", "//" ~~ "\n"]- , ftChar = ["'" ~~ "'"]- , ftString = ["\"" ~~ "\""]- , ftRawString = ["\"\"\"" ~~ "\"\"\""]- , ftIdentifierChars = Nothing- , ftKeywords =- types- [ "Byte"- , "Short"- , "Int"- , "Long"- , "Float"- , "Double"- , "UByte"- , "UShort"- , "UInt"- , "ULong"- , "Boolean"- , "Char"- ]- <> reserved- [ "as"- , "break"- , "class"- , "continue"- , "do"- , "else"- , "false"- , "for"- , "fun"- , "if"- , "in"- , "interface"- , "is"- , "null"- , "object"- , "package"- , "return"- , "super"- , "this"- , "throw"- , "true"- , "try"- , "typealias"- , "typeof"- , "val"- , "var"- , "when"- , "while"- , "by"- , "catch"- , "constructor"- , "delegate"- , "dynamic"- , "field"- , "file"- , "finally"- , "get"- , "import"- , "init"- , "param"- , "property"- , "receiver"- , "set"- , "setparam"- , "value"- , "where"- , "abstract"- , "actual"- , "annotation"- , "companion"- , "const"- , "crossinline"- , "data"- , "enum"- , "expect"- , "external"- , "final"- , "infix"- , "inline"- , "inner"- , "internal"- , "lateinit"- , "noinline"- , "open"- , "operator"- , "out"- , "override"- , "private"- , "protected"- , "public"- , "reified"- , "sealed"- , "suspend"- , "tailrec"- , "vararg"- , "field"- , "it"- ]- }- )- ,- ( Ksh- , FileTypeInfo- { ftSelector = [Ext "ksh"]- , ftKind = KindScript- , ftComment = ["#" ~~ "\n"]- , ftChar = []- , ftString = ["'" ~~ "'", "\"" ~~ "\""]- , ftRawString = []- , ftIdentifierChars = Nothing- , ftKeywords =- reserved- [ "case"- , "do"- , "done"- , "elif"- , "else"- , "esac"- , "fi"- , "for"- , "function"- , "if"- , "in"- , "select"- , "then"- , "time"- , "until"- , "while"- ]- }- )- ,- ( Latex- , FileTypeInfo- { ftSelector = [Ext "latex", Ext "tex"]- , ftKind = KindMarkup- , ftComment = ["%" ~~ "\n"]- , ftChar = []- , ftString = ["\"" ~~ "\""]- , ftRawString = []- , ftIdentifierChars = Just (const False, const False)- , ftKeywords = HM.empty- }- )- ,- ( Lisp- , FileTypeInfo- { ftSelector = [Ext "lisp", Ext "cl"]- , ftKind = KindLanguage- , ftComment = [";" ~~ "\n", "#|" ~~ "|#"]- , ftChar = []- , ftString = ["\"" ~~ "\""]- , ftRawString = []- , ftIdentifierChars = Just (isAlpha_, isAlphaNum_and "!$%&*+-./:<=>?@^~")- , ftKeywords =- types- [ "array"- , "atom"- , "bignum"- , "bit"- , "bit-vector"- , "character"- , "compiled-function"- , "complex"- , "cons"- , "double-float"- , "fixnum"- , "float"- , "function"- , "hash-table"- , "integer"- , "keyword"- , "list"- , "long-float"- , "nil"- , "null"- , "number"- , "package"- , "pathname"- , "random-state"- , "ratio"- , "rational"- , "readtable"- , "sequence"- , "short-float"- , "signed-byte"- , "simple-array"- , "simple-bit-vector"- , "simple-string"- , "simple-vector"- , "single-float"- , "standard-char"- , "stream"- , "string"- , "symbol"- , "t"- , "unsigned-byte"- , "vector"- ]- }- )- ,- ( Lua- , FileTypeInfo- { ftSelector = [Ext "lua"]- , ftKind = KindLanguage- , ftComment = ["--[[" ~~ "--]]", "--" ~~ "\n"]- , ftChar = []- , ftString = ["'" ~~ "'", "\"" ~~ "\""]- , ftRawString = ["[===[" ~~ "]===]", "[==[" ~~ "]==]", "[=[" ~~ "]=]", "[[" ~~ "]]"]- , ftIdentifierChars = Nothing- , ftKeywords =- reserved- [ "and"- , "break"- , "do"- , "else"- , "elseif"- , "end"- , "false"- , "for"- , "function"- , "if"- , "in"- , "local"- , "nil"- , "not"- , "or"- , "repeat"- , "return"- , "then"- , "true"- , "until"- , "while"- ]- }- )- ,- ( Make- , FileTypeInfo- { ftSelector = [Name "Makefile", Name "makefile", Name "GNUmakefile", Ext "mk", Ext "mak", Ext "make"]- , ftKind = KindScript- , ftComment = ["#" ~~ "\n"]- , ftChar = []- , ftString = ["'" ~~ "'", "\"" ~~ "\""]- , ftRawString = []- , ftIdentifierChars = Just (isAlpha_and "-", isAlpha_and "-")- , ftKeywords = HM.empty- }- )- ,- ( Nmap- , FileTypeInfo- { ftSelector = [Ext "nse"]- , ftKind = KindScript- , ftComment = ["--" ~~ "\n", "[[" ~~ "]]"]- , ftChar = []- , ftString = ["'" ~~ "'", "\"" ~~ "\""]- , ftRawString = []- , ftIdentifierChars = Just (const False, const False)- , ftKeywords = HM.empty- }- )- ,- ( Nim- , FileTypeInfo- { ftSelector = [Ext "nim"]- , ftKind = KindLanguage- , ftComment = ["#[" ~~ "#]", "#" ~~ "\n"]- , ftChar = ["'" ~~ "'"]- , ftString = ["\"" ~~ "\""]- , ftRawString = ["\"\"\"" ~~ "\"\"\""]- , ftIdentifierChars = Nothing- , ftKeywords =- types- [ "int"- , "int8"- , "int16"- , "int32"- , "int64"- , "uint"- , "uint8"- , "uint16"- , "uint32"- , "uint64"- , "float"- , "float32"- , "float64"- , "bool"- , "string"- , "cstring"- ]- <> reserved- [ "addr"- , "and"- , "as"- , "asm"- , "bind"- , "block"- , "break"- , "case"- , "cast"- , "concept"- , "const"- , "continue"- , "converter"- , "defer"- , "discard"- , "distinct"- , "div"- , "do"- , "elif"- , "else"- , "end"- , "enum"- , "except"- , "export"- , "finally"- , "for"- , "from"- , "func"- , "if"- , "import"- , "in"- , "include"- , "interface"- , "is"- , "isnot"- , "iterator"- , "let"- , "macro"- , "method"- , "mixin"- , "mod"- , "nil"- , "not"- , "notin"- , "object"- , "of"- , "or"- , "out"- , "proc"- , "ptr"- , "raise"- , "ref"- , "return"- , "shl"- , "shr"- , "static"- , "template"- , "try"- , "tuple"- , "type"- , "using"- , "var"- , "when"- , "while"- , "xor"- , "yield"- ]- }- )- ,- ( OCaml- , FileTypeInfo- { ftSelector = [Ext "ml", Ext "mli"]- , ftKind = KindLanguage- , ftComment = ["(*" ~~ "*)"]- , ftChar = ["'" ~~ "'"]- , ftString = ["\"" ~~ "\""]- , ftRawString = ["{id|" ~~ "|id}"]- , ftIdentifierChars = Just (isAlpha_, isAlphaNum_')- , ftKeywords =- types- [ "int"- , "float"- , "char"- , "string"- , "bool"- , "unit"- , "list"- , "array"- , "exn"- , "option"- , "ref"- ]- <> reserved- [ "and"- , "as"- , "assert"- , "asr"- , "begin"- , "class"- , "constraint"- , "do"- , "done"- , "downto"- , "else"- , "end"- , "exception"- , "external"- , "false"- , "for"- , "fun"- , "function"- , "functor"- , "if"- , "in"- , "include"- , "inherit"- , "initializer"- , "land"- , "lazy"- , "let"- , "lor"- , "lsl"- , "lsr"- , "lxor"- , "match"- , "method"- , "mod"- , "module"- , "mutable"- , "new"- , "nonrec"- , "object"- , "of"- , "open"- , "or"- , "private"- , "rec"- , "sig"- , "struct"- , "then"- , "to"- , "true"- , "try"- , "type"- , "val"- , "virtual"- , "when"- , "while"- , "with"- ]- }- )- ,- ( ObjectiveC- , FileTypeInfo- { ftSelector = [Ext "m", Ext "mi"]- , ftKind = KindLanguage- , ftComment = ["/*" ~~ "*/", "//" ~~ "\n"]- , ftChar = ["'" ~~ "'"]- , ftString = ["\"" ~~ "\""]- , ftRawString = []- , ftIdentifierChars = Nothing- , ftKeywords =- types- [ "char"- , "int"- , "short"- , "long"- , "float"- , "double"- , "signed"- , "unsigned"- ]- <> reserved- [ "void"- , "id"- , "const"- , "volatile"- , "in"- , "out"- , "inout"- , "bycopy"- , "byref"- , "oneway"- , "self"- , "super"- , "interface"- , "end"- , "@implementation"- , "@end"- , "@interface"- , "@end"- , "@implementation"- , "@end"- , "@protoco"- , "@end"- , "@class"- ]- }- )- ,- ( PHP- , FileTypeInfo- { ftSelector = [Ext "php", Ext "php3", Ext "php4", Ext "php5", Ext "phtml"]- , ftKind = KindScript- , ftComment = ["/*" ~~ "*/", "//" ~~ "\n", "#" ~~ "\n"]- , ftChar = ["'" ~~ "'"]- , ftString = ["'" ~~ "'", "\"" ~~ "\""]- , ftRawString = ["<<END" ~~ "END;", "<<'END'" ~~ "END;"]- , ftIdentifierChars = Nothing- , ftKeywords =- reserved- [ "__halt_compiler"- , "abstract"- , "and"- , "array"- , "as"- , "break"- , "callable"- , "case"- , "catch"- , "class"- , "clone"- , "const"- , "continue"- , "declare"- , "default"- , "die"- , "do"- , "echo"- , "else"- , "elseif"- , "empty"- , "enddeclare"- , "endfor"- , "endforeach"- , "endif"- , "endswitch"- , "endwhile"- , "eval"- , "exit"- , "extends"- , "final"- , "finally"- , "fn"- , "for"- , "foreach"- , "function"- , "global"- , "goto"- , "if"- , "implements"- , "include"- , "include_once"- , "instanceof"- , "insteadof"- , "interface"- , "isset"- , "list"- , "match"- , "namespace"- , "new"- , "or"- , "print"- , "private"- , "protected"- , "public"- , "readonly"- , "require"- , "require_once"- , "return"- , "static"- , "switch"- , "throw"- , "trait"- , "try"- , "unset"- , "use"- , "var"- , "while"- , "xor"- , "yield"- , "yield"- , "from"- ]- }- )- ,- ( Perl- , FileTypeInfo- { ftSelector = [Ext "pl", Ext "pm", Ext "pm6", Ext "plx", Ext "perl"]- , ftKind = KindScript- , ftComment = ["=pod" ~~ "=cut", "#" ~~ "\n"]- , ftChar = []- , ftString = ["'" ~~ "'", "\"" ~~ "\""]- , ftRawString = ["<<\"END\";" ~~ "END", "<<'END'" ~~ "END", "<<'EOT';" ~~ "EOT", "<<\"EOT\";" ~~ "EOT"]- , ftIdentifierChars = Nothing- , ftKeywords = reserved []- }- )- ,- ( Python- , FileTypeInfo- { ftSelector = [Ext "py", Ext "pyx", Ext "pxd", Ext "pxi", Ext "scons"]- , ftKind = KindLanguage- , ftComment = ["#" ~~ "\n"]- , ftChar = []- , ftString = ["'" ~~ "'", "\"" ~~ "\""]- , ftRawString = ["\"\"\"" ~~ "\"\"\"", "'''" ~~ "'''", "r'" ~~ "'"]- , ftIdentifierChars = Nothing- , ftKeywords =- reserved- [ "False"- , "await"- , "else"- , "import"- , "pass"- , "None"- , "break"- , "except"- , "in"- , "raise"- , "True"- , "class"- , "finally"- , "is"- , "return"- , "and"- , "continue"- , "for"- , "lambda"- , "try"- , "as"- , "def"- , "from"- , "nonlocal"- , "while"- , "assert"- , "del"- , "global"- , "not"- , "with"- , "async"- , "elif"- , "if"- , "or"- , "yield"- ]- }- )- ,- ( R- , FileTypeInfo- { ftSelector = [Ext "r", Ext "rdata", Ext "rds", Ext "rda"]- , ftKind = KindLanguage- , ftComment = ["#" ~~ "\n"]- , ftChar = []- , ftString = ["\"" ~~ "\"", "'" ~~ "'"]- , ftRawString = []- , ftIdentifierChars = Just (isAlpha_and ".", isAlphaNum_and ".")- , ftKeywords =- reserved- [ "if"- , "else"- , "repeat"- , "while"- , "function"- , "for"- , "in"- , "next"- , "break"- , "TRUE"- , "FALSE"- , "NULL"- , "Inf"- , "NaN"- , "NA"- , "NA_integer_"- , "NA_real_"- , "NA_complex_"- , "NA_character_"- , "…"- ]- }- )- ,- ( Ruby- , FileTypeInfo- { ftSelector = [Ext "rb", Ext "ruby"]- , ftKind = KindLanguage- , ftComment = ["=begin" ~~ "=end", "#" ~~ "\n"]- , ftChar = ["'" ~~ "'"]- , ftString = ["'" ~~ "'", "\"" ~~ "\"", "%|" ~~ "|", "%q(" ~~ ")", "%Q(" ~~ ")"]- , ftRawString = []- , ftIdentifierChars = Nothing- , ftKeywords =- reserved- [ "BEGIN"- , "END"- , "alias"- , "and"- , "begin"- , "break"- , "case"- , "class"- , "def"- , "module"- , "next"- , "nil"- , "not"- , "or"- , "redo"- , "rescue"- , "retry"- , "return"- , "elsif"- , "end"- , "false"- , "ensure"- , "for"- , "if"- , "true"- , "undef"- , "unless"- , "do"- , "else"- , "super"- , "then"- , "until"- , "when"- , "while"- , "defined?"- , "self"- ]- }- )- ,- ( Rust- , FileTypeInfo- { ftSelector = [Ext "rs", Ext "rlib"]- , ftKind = KindLanguage- , ftComment = ["/*" ~~ "*/", "//" ~~ "\n"]- , ftChar = ["'" ~~ "'"]- , ftString = ["\"" ~~ "\""]- , ftRawString = ["r##\"" ~~ "\"##", "r#\"" ~~ "\"#", "r\"" ~~ "\""]- , ftIdentifierChars = Just (isAlpha_, isAlphaNum_and "#")- , ftKeywords =- types- [ "i8"- , "u8"- , "i16"- , "u16"- , "i32"- , "u32"- , "i64"- , "u64"- , "i128"- , "u128"- , "isize"- , "usize"- , "bool"- , "char"- , "str"- , "String"- ]- <> reserved- [ "as"- , "use"- , "extern"- , "crate"- , "break"- , "const"- , "continue"- , "crate"- , "else"- , "if"- , "let"- , "enum"- , "extern"- , "false"- , "fn"- , "for"- , "if"- , "impl"- , "in"- , "for"- , "let"- , "loop"- , "match"- , "mod"- , "move"- , "mut"- , "pub"- , "impl"- , "ref"- , "return"- , "Self"- , "self"- , "static"- , "struct"- , "super"- , "trait"- , "true"- , "type"- , "unsafe"- , "use"- , "where"- , "while"- , "abstract"- , "alignof"- , "become"- , "box"- , "do"- , "final"- , "macro"- , "offsetof"- , "override"- , "priv"- , "proc"- , "pure"- , "sizeof"- , "typeof"- , "unsized"- , "virtual"- , "yield"- , "async"- , "await"- , "dyn"- , -- weak keywords- "macro_rules"- , "union"- , "'static"- , -- reserved for future use- "abstract"- , "become"- , "box"- , "do"- , "final"- , "macro"- , "override"- , "priv"- , "typeof"- , "unsized"- , "virtual"- , "yield"- , "try"- ]- }- )- ,- ( Scala- , FileTypeInfo- { ftSelector = [Ext "scala"]- , ftKind = KindLanguage- , ftComment = ["/*" ~~ "*/", "//" ~~ "\n"]- , ftChar = ["'" ~~ "'"]- , ftString = ["\"" ~~ "\"", "'" ~~ "'"]- , ftRawString = []- , ftIdentifierChars = Just (isAlpha_and "$", isAlphaNum_and "$")- , ftKeywords =- types- [ "Byte"- , "Short"- , "Int"- , "Long"- , "Float"- , "Double"- , "Char"- , "String"- , "Boolean"- , "Unit"- , "Null"- , "Nothing"- , "Any"- , "AnyRef"- ]- <> reserved- [ "abstract"- , "case"- , "catch"- , "class"- , "def"- , "do"- , "else"- , "extends"- , "false"- , "final"- , "finally"- , "for"- , "forSome"- , "if"- , "implicit"- , "import"- , "lazy"- , "match"- , "new"- , "null"- , "object"- , "override"- , "package"- , "private"- , "protected"- , "return"- , "sealed"- , "super"- , "this"- , "throw"- , "trait"- , "true"- , "try"- , "type"- , "val"- , "var"- , "while"- , "with"- , "yield"- ]- }- )- ,- ( SmallTalk- , FileTypeInfo- { ftSelector = [Ext "st", Ext "gst"]- , ftKind = KindLanguage- , ftComment = ["\"" ~~ "\""]- , ftChar = ["$" ~~ ""]- , ftString = ["'" ~~ "'"]- , ftRawString = []- , ftIdentifierChars = Just (isAlpha, isAlphaNum)- , ftKeywords =- reserved- [ "true"- , "false"- , "nil"- , "self"- , "super"- , "thisContext"- ]- }- )- ,- ( Swift- , FileTypeInfo- { ftSelector = [Ext "swift"]- , ftKind = KindLanguage- , ftComment = ["/*" ~~ "*/", "//" ~~ "\n"]- , ftChar = []- , ftString = ["\"" ~~ "\""]- , ftRawString = ["\"\"\"" ~~ "\"\"\""]- , ftIdentifierChars = Nothing- , ftKeywords =- types- [ "Int"- , "Int8"- , "Int16"- , "Int32"- , "Int64"- , "UInt"- , "UInt8"- , "UInt16"- , "UInt32"- , "UInt64"- , "Float"- , "Double"- , "Float80"- , "Float16"- , "Bool"- , "String"- , "Character"- , "Array"- , "Dictionary"- , "Set"- , "Optional"- , "Any"- , "AnyObject"- ]- <> reserved- [ "associatedtype"- , "class"- , "deinit"- , "enum"- , "extension"- , "fileprivate"- , "func"- , "import"- , "init"- , "inout"- , "internal"- , "let"- , "open"- , "operator"- , "private"- , "precedencegroup"- , "protocol"- , "public"- , "rethrows"- , "static"- , "struct"- , "subscript"- , "typealias"- , "var"- , "break"- , "case"- , "catch"- , "continue"- , "default"- , "defer"- , "do"- , "else"- , "fallthrough"- , "for"- , "guard"- , "if"- , "in"- , "repeat"- , "return"- , "throw"- , "switch"- , "where"- , "while"- , "Any"- , "as"- , "catch"- , "false"- , "is"- , "nil"- , "rethrows"- , "self"- , "Self"- , "super"- , "throw"- , "throws"- , "true"- , "try"- , "#available"- , "#colorLiteral"- , "#column"- , "#dsohandle"- , "#elseif"- , "#else"- , "#endif"- , "#error"- , "#fileID"- , "#fileLiteral"- , "#filePath"- , "#file"- , "#function"- , "#if"- , "#imageLiteral"- , "#keyPath"- , "#line"- , "#selector"- , "#sourceLocation"- , "#warning"- , "associativity"- , "convenience"- , "didSet"- , "dynamic"- , "final"- , "get"- , "indirect"- , "infix"- , "lazy"- , "left"- , "mutating"- , "none"- , "nonmutating"- , "optional"- , "override"- , "postfix"- , "precedence"- , "prefix"- , "Protocol"- , "required"- , "right"- , "set"- , "some"- , "Type"- , "unowned"- , "weak"- , "willSet"- ]- }- )- ,- ( Sql- , FileTypeInfo- { ftSelector = [Ext "sql"]- , ftKind = KindLanguage- , ftComment = ["--" ~~ "\n"]- , ftChar = []- , ftString = ["'" ~~ "'"]- , ftRawString = []- , ftIdentifierChars = Nothing- , ftKeywords =- reserved- [ "ABORT"- , "ABORTSESSION"- , "ABS"- , "ABSOLUTE"- , "ACCESS"- , "ACCESSIBLE"- , "ACCESS_LOCK"- , "ACCOUNT"- , "ACOS"- , "ACOSH"- , "ACTION"- , "ADD"- , "ADD_MONTHS"- , "ADMIN"- , "AFTER"- , "AGGREGATE"- , "ALIAS"- , "ALL"- , "ALLOCATE"- , "ALLOW"- , "ALTER"- , "ALTERAND"- , "AMP"- , "ANALYSE"- , "ANALYZE"- , "AND"- , "ANSIDATE"- , "ANY"- , "ARE"- , "ARRAY"- , "ARRAY_AGG"- , "ARRAY_EXISTS"- , "ARRAY_MAX_CARDINALITY"- , "AS"- , "ASC"- , "ASENSITIVE"- , "ASIN"- , "ASINH"- , "ASSERTION"- , "ASSOCIATE"- , "ASUTIME"- , "ASYMMETRIC"- , "AT"- , "ATAN"- , "ATAN2"- , "ATANH"- , "ATOMIC"- , "AUDIT"- , "AUTHORIZATION"- , "AUX"- , "AUXILIARY"- , "AVE"- , "AVERAGE"- , "AVG"- , "BACKUP"- , "BEFORE"- , "BEGIN"- , "BEGIN_FRAME"- , "BEGIN_PARTITION"- , "BETWEEN"- , "BIGINT"- , "BINARY"- , "BIT"- , "BLOB"- , "BOOLEAN"- , "BOTH"- , "BREADTH"- , "BREAK"- , "BROWSE"- , "BT"- , "BUFFERPOOL"- , "BULK"- , "BUT"- , "BY"- , "BYTE"- , "BYTEINT"- , "BYTES"- , "CALL"- , "CALLED"- , "CAPTURE"- , "CARDINALITY"- , "CASCADE"- , "CASCADED"- , "CASE"- , "CASESPECIFIC"- , "CASE_N"- , "CAST"- , "CATALOG"- , "CCSID"- , "CD"- , "CEIL"- , "CEILING"- , "CHANGE"- , "CHAR"- , "CHAR2HEXINT"- , "CHARACTER"- , "CHARACTERS"- , "CHARACTER_LENGTH"- , "CHARS"- , "CHAR_LENGTH"- , "CHECK"- , "CHECKPOINT"- , "CLASS"- , "CLASSIFIER"- , "CLOB"- , "CLONE"- , "CLOSE"- , "CLUSTER"- , "CLUSTERED"- , "CM"- , "COALESCE"- , "COLLATE"- , "COLLATION"- , "COLLECT"- , "COLLECTION"- , "COLLID"- , "COLUMN"- , "COLUMN_VALUE"- , "COMMENT"- , "COMMIT"- , "COMPLETION"- , "COMPRESS"- , "COMPUTE"- , "CONCAT"- , "CONCURRENTLY"- , "CONDITION"- , "CONNECT"- , "CONNECTION"- , "CONSTRAINT"- , "CONSTRAINTS"- , "CONSTRUCTOR"- , "CONTAINS"- , "CONTAINSTABLE"- , "CONTENT"- , "CONTINUE"- , "CONVERT"- , "CONVERT_TABLE_HEADER"- , "COPY"- , "CORR"- , "CORRESPONDING"- , "COS"- , "COSH"- , "COUNT"- , "COVAR_POP"- , "COVAR_SAMP"- , "CREATE"- , "CROSS"- , "CS"- , "CSUM"- , "CT"- , "CUBE"- , "CUME_DIST"- , "CURRENT"- , "CURRENT_CATALOG"- , "CURRENT_DATE"- , "CURRENT_DEFAULT_TRANSFORM_GROUP"- , "CURRENT_LC_CTYPE"- , "CURRENT_PATH"- , "CURRENT_ROLE"- , "CURRENT_ROW"- , "CURRENT_SCHEMA"- , "CURRENT_SERVER"- , "CURRENT_TIME"- , "CURRENT_TIMESTAMP"- , "CURRENT_TIMEZONE"- , "CURRENT_TRANSFORM_GROUP_FOR_TYPE"- , "CURRENT_USER"- , "CURRVAL"- , "CURSOR"- , "CV"- , "CYCLE"- , "DATA"- , "DATABASE"- , "DATABASES"- , "DATABLOCKSIZE"- , "DATE"- , "DATEFORM"- , "DAY"- , "DAYS"- , "DAY_HOUR"- , "DAY_MICROSECOND"- , "DAY_MINUTE"- , "DAY_SECOND"- , "DBCC"- , "DBINFO"- , "DEALLOCATE"- , "DEC"- , "DECFLOAT"- , "DECIMAL"- , "DECLARE"- , "DEFAULT"- , "DEFERRABLE"- , "DEFERRED"- , "DEFINE"- , "DEGREES"- , "DEL"- , "DELAYED"- , "DELETE"- , "DENSE_RANK"- , "DENY"- , "DEPTH"- , "DEREF"- , "DESC"- , "DESCRIBE"- , "DESCRIPTOR"- , "DESTROY"- , "DESTRUCTOR"- , "DETERMINISTIC"- , "DIAGNOSTIC"- , "DIAGNOSTICS"- , "DICTIONARY"- , "DISABLE"- , "DISABLED"- , "DISALLOW"- , "DISCONNECT"- , "DISK"- , "DISTINCT"- , "DISTINCTROW"- , "DISTRIBUTED"- , "DIV"- , "DO"- , "DOCUMENT"- , "DOMAIN"- , "DOUBLE"- , "DROP"- , "DSSIZE"- , "DUAL"- , "DUMP"- , "DYNAMIC"- , "EACH"- , "ECHO"- , "EDITPROC"- , "ELEMENT"- , "ELSE"- , "ELSEIF"- , "EMPTY"- , "ENABLED"- , "ENCLOSED"- , "ENCODING"- , "ENCRYPTION"- , "END"- , "END"- , "EXEC"- , "ENDING"- , "END_FRAME"- , "END_PARTITION"- , "EQ"- , "EQUALS"- , "ERASE"- , "ERRLVL"- , "ERROR"- , "ERRORFILES"- , "ERRORTABLES"- , "ESCAPE"- , "ESCAPED"- , "ET"- , "EVERY"- , "EXCEPT"- , "EXCEPTION"- , "EXCLUSIVE"- , "EXEC"- , "EXECUTE"- , "EXISTS"- , "EXIT"- , "EXP"- , "EXPLAIN"- , "EXTERNAL"- , "EXTRACT"- , "FALLBACK"- , "FALSE"- , "FASTEXPORT"- , "FENCED"- , "FETCH"- , "FIELDPROC"- , "FILE"- , "FILLFACTOR"- , "FILTER"- , "FINAL"- , "FIRST"- , "FIRST_VALUE"- , "FLOAT"- , "FLOAT4"- , "FLOAT8"- , "FLOOR"- , "FOR"- , "FORCE"- , "FOREIGN"- , "FORMAT"- , "FOUND"- , "FRAME_ROW"- , "FREE"- , "FREESPACE"- , "FREETEXT"- , "FREETEXTTABLE"- , "FREEZE"- , "FROM"- , "FULL"- , "FULLTEXT"- , "FUNCTION"- , "FUSION"- , "GE"- , "GENERAL"- , "GENERATED"- , "GET"- , "GIVE"- , "GLOBAL"- , "GO"- , "GOTO"- , "GRANT"- , "GRAPHIC"- , "GROUP"- , "GROUPING"- , "GROUPS"- , "GT"- , "HANDLER"- , "HASH"- , "HASHAMP"- , "HASHBAKAMP"- , "HASHBUCKET"- , "HASHROW"- , "HAVING"- , "HELP"- , "HIGH_PRIORITY"- , "HOLD"- , "HOLDLOCK"- , "HOST"- , "HOUR"- , "HOURS"- , "HOUR_MICROSECOND"- , "HOUR_MINUTE"- , "HOUR_SECOND"- , "IDENTIFIED"- , "IDENTITY"- , "IDENTITYCOL"- , "IDENTITY_INSERT"- , "IF"- , "IGNORE"- , "ILIKE"- , "IMMEDIATE"- , "IN"- , "INCLUSIVE"- , "INCONSISTENT"- , "INCREMENT"- , "INDEX"- , "INDICATOR"- , "INFILE"- , "INHERIT"- , "INITIAL"- , "INITIALIZE"- , "INITIALLY"- , "INITIATE"- , "INNER"- , "INOUT"- , "INPUT"- , "INS"- , "INSENSITIVE"- , "INSERT"- , "INSTEAD"- , "INT"- , "INT1"- , "INT2"- , "INT3"- , "INT4"- , "INT8"- , "INTEGER"- , "INTEGERDATE"- , "INTERSECT"- , "INTERSECTION"- , "INTERVAL"- , "INTO"- , "IO_AFTER_GTIDS"- , "IO_BEFORE_GTIDS"- , "IS"- , "ISNULL"- , "ISOBID"- , "ISOLATION"- , "ITERATE"- , "JAR"- , "JOIN"- , "JOURNAL"- , "JSON_ARRAY"- , "JSON_ARRAYAGG"- , "JSON_EXISTS"- , "JSON_OBJECT"- , "JSON_OBJECTAGG"- , "JSON_QUERY"- , "JSON_TABLE"- , "JSON_TABLE_PRIMITIVE"- , "JSON_VALUE"- , "KEEP"- , "KEY"- , "KEYS"- , "KILL"- , "KURTOSIS"- , "LABEL"- , "LAG"- , "FTUAGE"- , "LARGE"- , "LAST"- , "LAST_VALUE"- , "LATERAL"- , "LC_CTYPE"- , "LE"- , "LEAD"- , "LEADING"- , "LEAVE"- , "LEFT"- , "LESS"- , "LEVEL"- , "LIKE"- , "LIKE_REGEX"- , "LIMIT"- , "LINEAR"- , "LINENO"- , "LINES"- , "LISTAGG"- , "LN"- , "LOAD"- , "LOADING"- , "LOCAL"- , "LOCALE"- , "LOCALTIME"- , "LOCALTIMESTAMP"- , "LOCATOR"- , "LOCATORS"- , "LOCK"- , "LOCKING"- , "LOCKMAX"- , "LOCKSIZE"- , "LOG"- , "LOG10"- , "LOGGING"- , "LOGON"- , "LONG"- , "LONGBLOB"- , "LONGTEXT"- , "LOOP"- , "LOWER"- , "LOW_PRIORITY"- , "LT"- , "MACRO"- , "MAINTAINED"- , "MAP"- , "MASTER_BIND"- , "MASTER_SSL_VERIFY_SERVER_CERT"- , "MATCH"- , "MATCHES"- , "MATCH_NUMBER"- , "MATCH_RECOGNIZE"- , "MATERIALIZED"- , "MAVG"- , "MAX"- , "MAXEXTENTS"- , "MAXIMUM"- , "MAXVALUE"- , "MCHARACTERS"- , "MDIFF"- , "MEDIUMBLOB"- , "MEDIUMINT"- , "MEDIUMTEXT"- , "MEMBER"- , "MERGE"- , "METHOD"- , "MICROSECOND"- , "MICROSECONDS"- , "MIDDLEINT"- , "MIN"- , "MINDEX"- , "MINIMUM"- , "MINUS"- , "MINUTE"- , "MINUTES"- , "MINUTE_MICROSECOND"- , "MINUTE_SECOND"- , "MLINREG"- , "MLOAD"- , "MLSLABEL"- , "MOD"- , "MODE"- , "MODIFIES"- , "MODIFY"- , "MODULE"- , "MONITOR"- , "MONRESOURCE"- , "MONSESSION"- , "MONTH"- , "MONTHS"- , "MSUBSTR"- , "MSUM"- , "MULTISET"- , "NAMED"- , "NAMES"- , "NATIONAL"- , "NATURAL"- , "NCHAR"- , "NCLOB"- , "NE"- , "NESTED_TABLE_ID"- , "NEW"- , "NEW_TABLE"- , "NEXT"- , "NEXTVAL"- , "NO"- , "NOAUDIT"- , "NOCHECK"- , "NOCOMPRESS"- , "NONCLUSTERED"- , "NONE"- , "NORMALIZE"- , "NOT"- , "NOTNULL"- , "NOWAIT"- , "NO_WRITE_TO_BINLOG"- , "NTH_VALUE"- , "NTILE"- , "NULL"- , "NULLIF"- , "NULLIFZERO"- , "NULLS"- , "NUMBER"- , "NUMERIC"- , "NUMPARTS"- , "OBID"- , "OBJECT"- , "OBJECTS"- , "OCCURRENCES_REGEX"- , "OCTET_LENGTH"- , "OF"- , "OFF"- , "OFFLINE"- , "OFFSET"- , "OFFSETS"- , "OLD"- , "OLD_TABLE"- , "OMIT"- , "ON"- , "ONE"- , "ONLINE"- , "ONLY"- , "OPEN"- , "OPENDATASOURCE"- , "OPENQUERY"- , "OPENROWSET"- , "OPENXML"- , "OPERATION"- , "OPTIMIZATION"- , "OPTIMIZE"- , "OPTIMIZER_COSTS"- , "OPTION"- , "OPTIONALLY"- , "OR"- , "ORDER"- , "ORDINALITY"- , "ORGANIZATION"- , "OUT"- , "OUTER"- , "OUTFILE"- , "OUTPUT"- , "OVER"- , "OVERLAPS"- , "OVERLAY"- , "OVERRIDE"- , "PACKAGE"- , "PAD"- , "PADDED"- , "PARAMETER"- , "PARAMETERS"- , "PART"- , "PARTIAL"- , "PARTITION"- , "PARTITIONED"- , "PARTITIONING"- , "PASSWORD"- , "PATH"- , "PATTERN"- , "PCTFREE"- , "PER"- , "PERCENT"- , "PERCENTILE_CONT"- , "PERCENTILE_DISC"- , "PERCENT_RANK"- , "PERIOD"- , "PERM"- , "PERMANENT"- , "PIECESIZE"- , "PIVOT"- , "PLACING"- , "PLAN"- , "PORTION"- , "POSITION"- , "POSITION_REGEX"- , "POSTFIX"- , "POWER"- , "PRECEDES"- , "PRECISION"- , "PREFIX"- , "PREORDER"- , "PREPARE"- , "PRESERVE"- , "PREVVAL"- , "PRIMARY"- , "PRINT"- , "PRIOR"- , "PRIQTY"- , "PRIVATE"- , "PRIVILEGES"- , "PROC"- , "PROCEDURE"- , "PROFILE"- , "PROGRAM"- , "PROPORTIONAL"- , "PROTECTION"- , "PSID"- , "PTF"- , "PUBLIC"- , "PURGE"- , "QUALIFIED"- , "QUALIFY"- , "QUANTILE"- , "QUERY"- , "QUERYNO"- , "RADIANS"- , "RAISERROR"- , "RANDOM"- , "RANGE"- , "RANGE_N"- , "RANK"- , "RAW"- , "READ"- , "READS"- , "READTEXT"- , "READ_WRITE"- , "REAL"- , "RECONFIGURE"- , "RECURSIVE"- , "REF"- , "REFERENCES"- , "REFERENCING"- , "REFRESH"- , "REGEXP"- , "REGR_AVGX"- , "REGR_AVGY"- , "REGR_COUNT"- , "REGR_INTERCEPT"- , "REGR_R2"- , "REGR_SLOPE"- , "REGR_SXX"- , "REGR_SXY"- , "REGR_SYY"- , "RELATIVE"- , "RELEASE"- , "RENAME"- , "REPEAT"- , "REPLACE"- , "REPLICATION"- , "REPOVERRIDE"- , "REQUEST"- , "REQUIRE"- , "RESIGNAL"- , "RESOURCE"- , "RESTART"- , "RESTORE"- , "RESTRICT"- , "RESULT"- , "RESULT_SET_LOCATOR"- , "RESUME"- , "RET"- , "RETRIEVE"- , "RETURN"- , "RETURNING"- , "RETURNS"- , "REVALIDATE"- , "REVERT"- , "REVOKE"- , "RIGHT"- , "RIGHTS"- , "RLIKE"- , "ROLE"- , "ROLLBACK"- , "ROLLFORWARD"- , "ROLLUP"- , "ROUND_CEILING"- , "ROUND_DOWN"- , "ROUND_FLOOR"- , "ROUND_HALF_DOWN"- , "ROUND_HALF_EVEN"- , "ROUND_HALF_UP"- , "ROUND_UP"- , "ROUTINE"- , "ROW"- , "ROWCOUNT"- , "ROWGUIDCOL"- , "ROWID"- , "ROWNUM"- , "ROWS"- , "ROWSET"- , "ROW_NUMBER"- , "RULE"- , "RUN"- , "RUNNING"- , "SAMPLE"- , "SAMPLEID"- , "SAVE"- , "SAVEPOINT"- , "SCHEMA"- , "SCHEMAS"- , "SCOPE"- , "SCRATCHPAD"- , "SCROLL"- , "SEARCH"- , "SECOND"- , "SECONDS"- , "SECOND_MICROSECOND"- , "SECQTY"- , "SECTION"- , "SECURITY"- , "SECURITYAUDIT"- , "SEEK"- , "SEL"- , "SELECT"- , "SEMANTICKEYPHRASETABLE"- , "SEMANTICSIMILARITYDETAILSTABLE"- , "SEMANTICSIMILARITYTABLE"- , "SENSITIVE"- , "SEPARATOR"- , "SEQUENCE"- , "SESSION"- , "SESSION_USER"- , "SET"- , "SETRESRATE"- , "SETS"- , "SETSESSRATE"- , "SETUSER"- , "SHARE"- , "SHOW"- , "SHUTDOWN"- , "SIGNAL"- , "SIMILAR"- , "SIMPLE"- , "SIN"- , "SINH"- , "SIZE"- , "SKEW"- , "SKIP"- , "SMALLINT"- , "SOME"- , "SOUNDEX"- , "SOURCE"- , "SPACE"- , "SPATIAL"- , "SPECIFIC"- , "SPECIFICTYPE"- , "SPOOL"- , "SQL"- , "SQLEXCEPTION"- , "SQLSTATE"- , "SQLTEXT"- , "SQLWARNING"- , "SQL_BIG_RESULT"- , "SQL_CALC_FOUND_ROWS"- , "SQL_SMALL_RESULT"- , "SQRT"- , "SS"- , "SSL"- , "STANDARD"- , "START"- , "STARTING"- , "STARTUP"- , "STATE"- , "STATEMENT"- , "STATIC"- , "STATISTICS"- , "STAY"- , "STDDEV_POP"- , "STDDEV_SAMP"- , "STEPINFO"- , "STOGROUP"- , "STORED"- , "STORES"- , "STRAIGHT_JOIN"- , "STRING_CS"- , "STRUCTURE"- , "STYLE"- , "SUBMULTISET"- , "SUBSCRIBER"- , "SUBSET"- , "SUBSTR"- , "SUBSTRING"- , "SUBSTRING_REGEX"- , "SUCCEEDS"- , "SUCCESSFUL"- , "SUM"- , "SUMMARY"- , "SUSPEND"- , "SYMMETRIC"- , "SYNONYM"- , "SYSDATE"- , "SYSTEM"- , "SYSTEM_TIME"- , "SYSTEM_USER"- , "SYSTIMESTAMP"- , "TABLE"- , "TABLESAMPLE"- , "TABLESPACE"- , "TAN"- , "TANH"- , "TBL_CS"- , "TEMPORARY"- , "TERMINATE"- , "TERMINATED"- , "TEXTSIZE"- , "THAN"- , "THEN"- , "THRESHOLD"- , "TIME"- , "TIMESTAMP"- , "TIMEZONE_HOUR"- , "TIMEZONE_MINUTE"- , "TINYBLOB"- , "TINYINT"- , "TINYTEXT"- , "TITLE"- , "TO"- , "TOP"- , "TRACE"- , "TRAILING"- , "TRAN"- , "TRANSACTION"- , "TRANSLATE"- , "TRANSLATE_CHK"- , "TRANSLATE_REGEX"- , "TRANSLATION"- , "TREAT"- , "TRIGGER"- , "TRIM"- , "TRIM_ARRAY"- , "TRUE"- , "TRUNCATE"- , "TRY_CONVERT"- , "TSEQUAL"- , "TYPE"- , "UC"- , "UESCAPE"- , "UID"- , "UNDEFINED"- , "UNDER"- , "UNDO"- , "UNION"- , "UNIQUE"- , "UNKNOWN"- , "UNLOCK"- , "UNNEST"- , "UNPIVOT"- , "UNSIGNED"- , "UNTIL"- , "UPD"- , "UPDATE"- , "UPDATETEXT"- , "UPPER"- , "UPPERCASE"- , "USAGE"- , "USE"- , "USER"- , "USING"- , "UTC_DATE"- , "UTC_TIME"- , "UTC_TIMESTAMP"- , "VALIDATE"- , "VALIDPROC"- , "VALUE"- , "VALUES"- , "VALUE_OF"- , "VARBINARY"- , "VARBYTE"- , "VARCHAR"- , "VARCHAR2"- , "VARCHARACTER"- , "VARGRAPHIC"- , "VARIABLE"- , "VARIADIC"- , "VARIANT"- , "VARYING"- , "VAR_POP"- , "VAR_SAMP"- , "VCAT"- , "VERBOSE"- , "VERSIONING"- , "VIEW"- , "VIRTUAL"- , "VOLATILE"- , "VOLUMES"- , "WAIT"- , "WAITFOR"- , "WHEN"- , "WHENEVER"- , "WHERE"- , "WHILE"- , "WIDTH_BUCKET"- , "WINDOW"- , "WITH"- , "WITHIN"- , "WITHIN_GROUP"- , "WITHOUT"- , "WLM"- , "WORK"- , "WRITE"- , "WRITETEXT"- , "XMLCAST"- , "XMLEXISTS"- , "XMLNAMESPACES"- , "XOR"- , "YEAR"- , "YEARS"- , "YEAR_MONTH"- , "ZEROFILL"- , "ZEROIFNULL"- , "ZONE"- ]- }- )- ,- ( Tcl- , FileTypeInfo- { ftSelector = [Ext "tcl", Ext "tk"]- , ftKind = KindScript- , ftComment = ["#" ~~ "\n"]- , ftChar = []- , ftString = ["\"" ~~ "\""]- , ftRawString = []- , ftKeywords = HM.empty- , ftIdentifierChars = Just (isAlpha, isAlphaNum)- }- )- ,- ( Text- , FileTypeInfo- { ftSelector =- [ Ext "txt"- , Ext "md"- , Ext "markdown"- , Ext "mdown"- , Ext "mkdn"- , Ext "mkd"- , Ext "mdwn"- , Ext "mdtxt"- , Ext "mdtext"- , Ext "text"- , Name "README"- , Name "INSTALL"- , Name "VERSION"- , Name "LICENSE"- , Name "AUTHORS"- , Name "CHANGELOG"- , Name "go.sum"- ]- , ftKind = KindText- , ftComment = []- , ftChar = []- , ftString = []- , ftRawString = []- , ftIdentifierChars = Just (const False, const False)- , ftKeywords = HM.empty- }- )- ,- ( Unison- , FileTypeInfo- { ftSelector = [Ext "u"]- , ftKind = KindLanguage- , ftComment = ["{-" ~~ "-}", "--" ~~ "\n"]- , ftChar = ["'" ~~ "'"]- , ftString = ["\"" ~~ "\"", "\"\"\"" ~~ "\"\"\""]- , ftRawString = []- , ftIdentifierChars = Just (isAlpha_', isAlphaNum_and ['!', '\''])- , ftKeywords =- types- [ "Text"- , "Number"- , "Boolean"- , "List"- , "Optional"- , "Maybe"- , "Either"- , "Tuple"- , "Function"- ]- <> reserved- [ "type"- , "ability"- , "structural"- , "unique"- , "if"- , "then"- , "else"- , "forall"- , "handle"- , "with"- , "where"- , "use"- , "true"- , "false"- , "alias"- , "typeLink"- , "termLink"- , "let"- , "namespace"- , "match"- , "cases"- ]- }- )- ,- ( VHDL- , FileTypeInfo- { ftSelector = [Ext "vhd", Ext "vhdl"]- , ftKind = KindLanguage- , ftComment = ["--" ~~ "\n"]- , ftChar = ["'" ~~ "'"]- , ftString = ["\"" ~~ "\""]- , ftRawString = []- , ftIdentifierChars = Nothing- , ftKeywords =- types- [ "std_logic"- , "std_logic_vector"- , "integer"- , "real"- , "boolean"- , "character"- , "string"- , "time"- , "bit"- , "bit_vector"- , "signed"- , "unsigned"- ]- <> reserved- [ "abs"- , "access"- , "after"- , "alias"- , "all"- , "and"- , "architecture"- , "array"- , "assert"- , "attribute"- , "begin"- , "block"- , "body"- , "buffer"- , "bus"- , "case"- , "component"- , "configuration"- , "constant"- , "disconnect"- , "downto"- , "else"- , "elsif"- , "end"- , "entity"- , "exit"- , "file"- , "for"- , "function"- , "generate"- , "generic"- , "group"- , "guarded"- , "if"- , "impure"- , "in"- , "inertial"- , "inout"- , "is"- , "label"- , "library"- , "linkage"- , "literal"- , "loop"- , "map"- , "mod"- , "nand"- , "new"- , "next"- , "nor"- , "not"- , "null"- , "of"- , "on"- , "open"- , "or"- , "others"- , "out"- , "package"- , "port"- , "postponed"- , "procedure"- , "process"- , "pure"- , "range"- , "record"- , "register"- , "reject"- , "return"- , "rol"- , "ror"- , "select"- , "severity"- , "signal"- , "shared"- , "sla"- , "sli"- , "sra"- , "srl"- , "subtype"- , "then"- , "to"- , "transport"- , "type"- , "unaffected"- , "units"- , "until"- , "use"- , "variable"- , "wait"- , "when"- , "while"- , "with"- , "xnor"- , "xor"- ]- }- )- ,- ( Verilog- , FileTypeInfo- { ftSelector = [Ext "v", Ext "vh", Ext "sv"]- , ftKind = KindLanguage- , ftComment = ["/*" ~~ "*/", "//" ~~ "\n"]- , ftChar = []- , ftString = ["\"" ~~ "\""]- , ftRawString = []- , ftIdentifierChars = Nothing- , ftKeywords =- types- [ "wire"- , "reg"- , "integer"- , "real"- , "time"- , "parameter"- , "event"- , "genvar"- , "string"- ]- <> reserved- [ "always"- , "end"- , "ifnone"- , "or"- , "rpmos"- , "tranif1"- , "and"- , "endcase"- , "initial"- , "output"- , "rtran"- , "tri"- , "assign"- , "endmodule"- , "inout"- , "rtranif0"- , "tri0"- , "begin"- , "endfunction"- , "input"- , "pmos"- , "rtranif1"- , "tri1"- , "buf"- , "endprimitive"- , "posedge"- , "scalared"- , "triand"- , "bufif0"- , "endspecify"- , "join"- , "primitive"- , "small"- , "trior"- , "bufif1"- , "endtable"- , "large"- , "pull0"- , "specify"- , "trireg"- , "case"- , "endtask"- , "macromodule"- , "pull1"- , "specparam"- , "vectored"- , "casex"- , "medium"- , "pullup"- , "strong0"- , "wait"- , "casez"- , "for"- , "module"- , "pulldown"- , "strong1"- , "wand"- , "cmos"- , "force"- , "nand"- , "rcmos"- , "supply0"- , "weak0"- , "deassign"- , "forever"- , "negedge"- , "supply1"- , "weak1"- , "default"- , "for"- , "nmos"- , "realtime"- , "table"- , "while"- , "defparam"- , "function"- , "nor"- , "task"- , "disable"- , "highz0"- , "not"- , "release"- , "wor"- , "edge"- , "highz1"- , "notif0"- , "repeat"- , "tran"- , "xnor"- , "else"- , "if"- , "notif1"- , "rnmos"- , "tranif0"- , "xor"- ]- }- )- ,- ( Yaml- , FileTypeInfo- { ftSelector = [Ext "yaml", Ext "yml"]- , ftKind = KindConfig- , ftComment = ["#" ~~ "\n"]- , ftChar = []- , ftString = ["\"" ~~ "\"", "'" ~~ "'"]- , ftRawString = []- , ftIdentifierChars = Just (const False, const False)- , ftKeywords = HM.empty- }- )- ,- ( Toml- , FileTypeInfo- { ftSelector = [Ext "toml", Name "Cargo.lock"]- , ftKind = KindConfig- , ftComment = ["#" ~~ "\n"]- , ftChar = []- , ftString = ["\"" ~~ "\"", "\"\"\"" ~~ "\"\"\""]- , ftRawString = ["'" ~~ "'", "'''" ~~ "'''"]- , ftIdentifierChars = Just (const False, const False)- , ftKeywords = HM.empty- }- )- ,- ( Ini- , FileTypeInfo- { ftSelector = [Ext "ini"]- , ftKind = KindConfig- , ftComment = [";" ~~ "\n", "#" ~~ "\n"]- , ftChar = []- , ftString = ["\"" ~~ "\""]- , ftRawString = []- , ftIdentifierChars = Just (const False, const False)- , ftKeywords = HM.empty- }- )- ,- ( Zig- , FileTypeInfo- { ftSelector = [Ext "zig"]- , ftKind = KindLanguage- , ftComment = ["//" ~~ "\n"]- , ftChar = ["'" ~~ "'"]- , ftString = ["\"" ~~ "\""]- , ftRawString = ["\\" ~~ "\n"]- , ftIdentifierChars = Nothing- , ftKeywords =- types- [ "i8"- , "u8"- , "i16"- , "u16"- , "i32"- , "u32"- , "i64"- , "u64"- , "i128"- , "u128"- , "isize"- , "usize"- , "c_short"- , "c_ushort"- , "c_int"- , "c_uint"- , "c_long"- , "c_ulong"- , "c_longlong"- , "c_ulonglong"- , "c_longdouble"- , "f16"- , "f32"- , "f64"- , "f80"- , "f128"- , "bool"- , "anyopaque"- , "void"- , "noreturn"- , "type"- , "anyerror"- , "comptime_int"- , "comptime_float"- ]- <> reserved- [ "addrspace"- , "align"- , "allowzero"- , "and"- , "anyframe"- , "anytype"- , "asm"- , "async"- , "await"- , "break"- , "catch"- , "comptime"- , "const"- , "continue"- , "defer"- , "else"- , "enum"- , "errdefer"- , "error"- , "export"- , "extern"- , "fn"- , "for"- , "if"- , "inline"- , "linksection"- , "noalias"- , "noinline"- , "nosuspend"- , "or"- , "orelse"- , "packed"- , "pub"- , "resume"- , "return"- , "struct"- , "suspend"- , "switch"- , "test"- , "threadlocal"- , "try"- , "union"- , "unreachable"- , "usingnamespace"- , "var"- , "volatile"- , "while"- ]- }- )- ,- ( Zsh- , FileTypeInfo- { ftSelector = [Ext "zsh"]- , ftKind = KindScript- , ftComment = ["#" ~~ "\n"]- , ftChar = []- , ftString = ["'" ~~ "'", "\"" ~~ "\""]- , ftRawString = []- , ftIdentifierChars = Nothing- , ftKeywords =- reserved- [ "do"- , "done"- , "esac"- , "then"- , "elif"- , "else"- , "fi"- , "for"- , "case"- , "if"- , "while"- , "function"- , "repeat"- , "time"- , "until"- , "select"- , "coproc"- , "nocorrect"- , "foreach"- , "end"- ]- }- )- ]---reserved :: [C.ByteString] -> HM.HashMap C.ByteString WordType-reserved = HM.fromList . map (,Keyword)-{-# INLINE reserved #-}--types :: [C.ByteString] -> HM.HashMap C.ByteString WordType-types = HM.fromList . map (,NativeType)-{-# INLINE types #-}--mkFilterFunction :: Bool -> FileTypeInfo -> Maybe FilterFunction-mkFilterFunction alterBoundary FileTypeInfo{..} =- Just $- runContextFilter (mkParConfig ftComment ftString ftRawString ftChar alterBoundary)-{-# INLINE mkFilterFunction #-}--contextFilter :: Maybe FileType -> ContextFilter -> Bool -> Text8 -> Text8-contextFilter _ (isContextFilterAll -> True) False txt = txt-contextFilter Nothing _ _ txt = txt-contextFilter (Just ftype) filt alterBoundary txt- | Just fun <- parFunc = fun filt txt- | otherwise = txt- where- parFunc = mkFilterFunction alterBoundary =<< Map.lookup ftype (unMapInfo fileTypeInfoMap)-{-# INLINE contextFilter #-}--fileTypeLookup :: Options -> RawFilePath -> Maybe (FileType, FileKind)-fileTypeLookup opts f = forcedType opts <|> lookupFileType f- where- lookupFileType :: RawFilePath -> Maybe (FileType, FileKind)- lookupFileType f = Map.lookup (Name $ takeFileName f) m <|> Map.lookup (Ext (C.dropWhile (== '.') $ takeExtension f)) m- m = unMap fileTypeMap-{-# INLINE fileTypeLookup #-}---fileTypeInfoLookup :: Options -> RawFilePath -> Maybe (FileType, FileTypeInfo)-fileTypeInfoLookup opts f = fileTypeLookup opts f >>= \(typ, kid) -> (typ,) <$> Map.lookup typ (unMapInfo fileTypeInfoMap)-{-# INLINE fileTypeInfoLookup #-}--fileTypeMap :: FileTypeMap-fileTypeMap = FileTypeMap $ Map.fromList $ concatMap (\(typ, FileTypeInfo{..}) -> map (, (typ, ftKind)) ftSelector) $ Map.toList (unMapInfo fileTypeInfoMap)-{-# NOINLINE fileTypeMap #-}--dumpFileTypeInfoMap :: FileTypeInfoMap -> IO ()-dumpFileTypeInfoMap m = forM_ ((Map.toList . unMapInfo) m) $ \(l, ex) ->- putStrLn $ show l <> [' ' | _ <- [length (show l) .. 12]] <> "-> " <> show (ftSelector ex)--dumpFileTypeMap :: FileTypeMap -> IO ()-dumpFileTypeMap m = forM_ (Map.toList (unMap m)) $ \(ext, l) ->- putStrLn $ show ext <> [' ' | _ <- [length (show ext) .. 12]] <> "-> " <> show l--forcedType :: Options -> Maybe (FileType, FileKind)-forcedType Options{type_force = l}- | Just typ <- l = Map.lookup (Ext $ C.pack typ) m <|> Map.lookup (Name $ C.pack typ) m- | otherwise = Nothing- where m = unMap fileTypeMap--(~~) :: C.ByteString -> C.ByteString -> Boundary-(~~) = Boundary-{-# INLINE (~~) #-}+{-# LANGUAGE AllowAmbiguousTypes #-}+{-# LANGUAGE DataKinds #-}+{-# LANGUAGE DeriveLift #-}+{-# LANGUAGE KindSignatures #-}+--+-- Copyright (c) 2013-2025 Nicola Bonelli <nicola@larthia.com>+--+-- This program is free software; you can redistribute it and/or modify+-- it under the terms of the GNU General Public License as published by+-- the Free Software Foundation; either version 2 of the License, or+-- (at your option) any later version.+--+-- This program is distributed in the hope that it will be useful,+-- but WITHOUT ANY WARRANTY; without even the implied warranty of+-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the+-- GNU General Public License for more details.+--+-- You should have received a copy of the GNU General Public License+-- along with this program; if not, write to the Free Software+-- Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.+--+{-# LANGUAGE ScopedTypeVariables #-}+{-# LANGUAGE TemplateHaskell #-}+{-# LANGUAGE TypeApplications #-}++module CGrep.FileTypeMap (+ FileTypeMap (..),+ FileTypeInfo (..),+ FileTypeInfoMap (..),+ CharSet (..),+ WordType (..),+ IsCharSet (..),+ CharIdentifierF,+ (~~),+ reserved,+ types,+) where++import CGrep.Boundary (Boundary (..))+import CGrep.FileKind+import CGrep.FileType (FileSelector (..), FileType (..))+import CGrep.Parser.Char+import Language.Haskell.TH.Syntax (Lift)++import qualified Data.HashMap.Strict as HM+import qualified Data.Map as Map+import qualified Data.Text as T++newtype FileTypeInfoMap = FileTypeInfoMap+ { unMapInfo :: Map.Map FileType FileTypeInfo+ }+ deriving stock (Eq, Show, Lift)++newtype FileTypeMap = FileTypeMap+ { unMap :: Map.Map FileSelector (FileType, FileKind)+ }++type CharIdentifierF = (Char -> Bool)++data WordType = Keyword | NativeType+ deriving stock (Eq, Ord, Show, Lift)++data FileTypeInfo = FileTypeInfo+ { ftSelector :: [FileSelector]+ , ftKind :: FileKind+ , ftChar :: [Boundary]+ , ftString :: [Boundary]+ , ftRawString :: [Boundary]+ , ftComment :: [Boundary]+ , ftIdentCharSet :: Maybe (CharSet, CharSet)+ , ftKeywords :: HM.HashMap T.Text WordType+ }+ deriving stock (Eq, Show, Lift)++data CharSet+ = None+ | Alpha+ | Alpha_+ | AlphaDollar_+ | AlphaNum+ | AlphaNum_+ | AlphaNum_'+ | AlphaNumDollar_+ | AlphaNumClojure_+ | AlphaDash_+ | AlphaNumDash_+ | Unicode_+ | UnicodeNum_+ | UnicodeDollar_+ | UnicodeNumDollar_+ | UnicodeNum_'+ | UnicodeXIDStart_+ | UnicodeNumXIDCont_+ | ClojureIdentStart+ | ClojureIdentCont+ | CSharpIdentStart+ | CSharpIdentCont+ | HtmlIdentStart+ | HtmlIdentCont+ | JavaIdentStart+ | JavaIdentCont+ | JuliaIdentStart+ | JuliaIdentCont+ | ListIdent+ | AgdaIdent+ deriving stock (Eq, Show, Lift)++class IsCharSet (cs :: CharSet) where+ isValidChar :: Char -> Bool++instance IsCharSet 'None where+ isValidChar _ = False+ {-# INLINE isValidChar #-}++instance IsCharSet 'Alpha where+ isValidChar = isAlpha+ {-# INLINE isValidChar #-}++instance IsCharSet 'Alpha_ where+ isValidChar = isAlpha_+ {-# INLINE isValidChar #-}++instance IsCharSet 'AlphaNum where+ isValidChar = isAlphaNum+ {-# INLINE isValidChar #-}++instance IsCharSet 'AlphaNum_ where+ isValidChar = isAlphaNum_+ {-# INLINE isValidChar #-}++instance IsCharSet 'AlphaNum_' where+ isValidChar = isAlphaNum_'+ {-# INLINE isValidChar #-}++instance IsCharSet 'AlphaDollar_ where+ isValidChar = isAlphaDollar_+ {-# INLINE isValidChar #-}++instance IsCharSet 'AlphaNumDollar_ where+ isValidChar = isAlphaNumDollar_+ {-# INLINE isValidChar #-}++instance IsCharSet 'ClojureIdentStart where+ isValidChar = isClojureIdentStart+ {-# INLINE isValidChar #-}++instance IsCharSet 'ClojureIdentCont where+ isValidChar = isClojureIdentCont+ {-# INLINE isValidChar #-}++instance IsCharSet 'CSharpIdentStart where+ isValidChar = isCSharpIdentStart+ {-# INLINE isValidChar #-}++instance IsCharSet 'CSharpIdentCont where+ isValidChar = isCSharpIdentCont+ {-# INLINE isValidChar #-}++instance IsCharSet 'AlphaDash_ where+ isValidChar = isAlphaDash_+ {-# INLINE isValidChar #-}++instance IsCharSet 'AlphaNumDash_ where+ isValidChar = isAlphaNumDash_+ {-# INLINE isValidChar #-}++instance IsCharSet 'HtmlIdentStart where+ isValidChar = isHtmlIdentStart+ {-# INLINE isValidChar #-}++instance IsCharSet 'HtmlIdentCont where+ isValidChar = isHtmlIdentCont+ {-# INLINE isValidChar #-}++instance IsCharSet 'JavaIdentStart where+ isValidChar = isJavaIdentStart+ {-# INLINE isValidChar #-}++instance IsCharSet 'JavaIdentCont where+ isValidChar = isJavaIdentCont+ {-# INLINE isValidChar #-}++instance IsCharSet 'JuliaIdentStart where+ isValidChar = isJuliaIdentStart+ {-# INLINE isValidChar #-}++instance IsCharSet 'JuliaIdentCont where+ isValidChar = isJuliaIdentCont+ {-# INLINE isValidChar #-}++instance IsCharSet 'ListIdent where+ isValidChar = isLispIdent+ {-# INLINE isValidChar #-}++instance IsCharSet 'Unicode_ where+ isValidChar = isUnicode_+ {-# INLINE isValidChar #-}++instance IsCharSet 'UnicodeNum_ where+ isValidChar = isUnicodeNum_+ {-# INLINE isValidChar #-}++instance IsCharSet 'UnicodeNum_' where+ isValidChar = isUnicodeNum_'+ {-# INLINE isValidChar #-}++instance IsCharSet 'UnicodeDollar_ where+ isValidChar = isUnicodeDollar_+ {-# INLINE isValidChar #-}++instance IsCharSet 'UnicodeNumDollar_ where+ isValidChar = isUnicodeNumDollar_+ {-# INLINE isValidChar #-}++instance IsCharSet 'UnicodeXIDStart_ where+ isValidChar = isUnicodeXIDStart_+ {-# INLINE isValidChar #-}++instance IsCharSet 'UnicodeNumXIDCont_ where+ isValidChar = isUnicodeNumXIDCont_+ {-# INLINE isValidChar #-}++instance IsCharSet 'AgdaIdent where+ isValidChar = isAgdaIdent+ {-# INLINE isValidChar #-}++(~~) :: T.Text -> T.Text -> Boundary+b ~~ e = Boundary{bBegin = b, bBeginLen = T.length b, bEnd = e, bEndLen = T.length e}+{-# INLINE (~~) #-}++reserved :: [T.Text] -> HM.HashMap T.Text WordType+reserved = HM.fromList . map (,Keyword)+{-# INLINE reserved #-}++types :: [T.Text] -> HM.HashMap T.Text WordType+types = HM.fromList . map (,NativeType)+{-# INLINE types #-}
+ src/CGrep/FileTypeMapTH.hs view
@@ -0,0 +1,4856 @@+{-# LANGUAGE AllowAmbiguousTypes #-}+{-# LANGUAGE DataKinds #-}+{-# LANGUAGE KindSignatures #-}+--+-- Copyright (c) 2013-2025 Nicola Bonelli <nicola@larthia.com>+--+-- This program is free software; you can redistribute it and/or modify+-- it under the terms of the GNU General Public License as published by+-- the Free Software Foundation; either version 2 of the License, or+-- (at your option) any later version.+--+-- This program is distributed in the hope that it will be useful,+-- but WITHOUT ANY WARRANTY; without even the implied warranty of+-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the+-- GNU General Public License for more details.+--+-- You should have received a copy of the GNU General Public License+-- along with this program; if not, write to the Free Software+-- Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.+--+{-# LANGUAGE ScopedTypeVariables #-}+{-# LANGUAGE TemplateHaskell #-}+{-# LANGUAGE TypeApplications #-}++module CGrep.FileTypeMapTH (+ fileTypeInfoMap,+ mkContextFilterFn,+ fileTypeLookup,+ fileTypeInfoLookup,+ dumpFileTypeInfoMap,+) where++import CGrep.Semantic.ContextFilter (+ ContextFilter,+ FilterFunction,+ isContextFilterAll,+ mkParConfig,+ runContextFilter,+ )++import Control.Applicative (Alternative ((<|>)))+import Control.Monad (forM_)+import Language.Haskell.TH.Syntax (lift)+import Options (Options (Options, code_only, force_type, hdr_only, keyword))+import qualified System.OsPath as OS++import qualified Data.HashMap.Strict as HM+import qualified Data.Map as Map++import CGrep.FileKind+import CGrep.FileType (FileSelector (..), FileType (..), ext, hdr, name)+import CGrep.FileTypeMap++import qualified Data.Text as T++fileTypeInfoMap :: FileTypeInfoMap+fileTypeInfoMap =+ $( lift $+ FileTypeInfoMap $+ Map.fromList+ [+ ( Agda+ , FileTypeInfo+ { ftSelector = [ext ".agda", ext ".lagda"]+ , ftKind = KindLanguage+ , ftComment = ["{-" ~~ "-}", "--" ~~ "\n"]+ , ftChar = ["'" ~~ "'"]+ , ftString = ["\"" ~~ "\""]+ , ftRawString = []+ , ftIdentCharSet = Just (AgdaIdent, AgdaIdent)+ , ftKeywords =+ reserved+ [ "abstract"+ , "codata"+ , "constructor"+ , "data"+ , "eta-equality"+ , "field"+ , "forall"+ , "hiding"+ , "import"+ , "in"+ , "inductive"+ , "infix"+ , "infixl"+ , "infixr"+ , "instance"+ , "let"+ , "module"+ , "mutual"+ , "no-eta-equality"+ , "open"+ , "pattern"+ , "postulate"+ , "primitive"+ , "private"+ , "public"+ , "quoteContext"+ , "quoteGoal"+ , "record"+ , "renaming"+ , "rewrite"+ , "Set"+ , "syntax"+ , "tactic"+ , "using"+ , "where"+ , "with"+ ]+ }+ )+ ,+ ( Assembly+ , FileTypeInfo+ { ftSelector = [ext ".s", ext ".S", ext ".asm", ext ".ASM"]+ , ftKind = KindLanguage+ , ftComment = ["#" ~~ "\n", ";" ~~ "\n", "|" ~~ "\n", "!" ~~ "\n", "/*" ~~ "*/"]+ , ftChar = []+ , ftString = ["\"" ~~ "\""]+ , ftRawString = []+ , ftIdentCharSet = Nothing+ , ftKeywords = HM.empty+ }+ )+ ,+ ( Awk+ , FileTypeInfo+ { ftSelector = [ext ".awk", ext ".mawk", ext ".gawk"]+ , ftKind = KindScript+ , ftComment = ["{-" ~~ "-}", "--" ~~ "\n", "#" ~~ "\n"]+ , ftChar = []+ , ftString = ["\"" ~~ "\""]+ , ftRawString = []+ , ftIdentCharSet = Just (Alpha_, AlphaNum_)+ , ftKeywords =+ reserved+ [ "BEGIN"+ , "END"+ , "if"+ , "else"+ , "while"+ , "do"+ , "for"+ , "in"+ , "break"+ , "continue"+ , "delete"+ , "next"+ , "nextfile"+ , "function"+ , "func"+ , "exit"+ ]+ }+ )+ ,+ ( Bash+ , FileTypeInfo+ { ftSelector = [ext ".sh", ext ".bash"]+ , ftKind = KindScript+ , ftComment = ["#" ~~ "\n"]+ , ftChar = []+ , ftString = ["'" ~~ "'", "\"" ~~ "\""]+ , ftRawString = []+ , ftIdentCharSet = Just (Alpha_, AlphaNum_)+ , ftKeywords =+ reserved+ [ "if"+ , "then"+ , "elif"+ , "else"+ , "fi"+ , "time"+ , "for"+ , "in"+ , "until"+ , "while"+ , "do"+ , "done"+ , "case"+ , "esac"+ , "coproc"+ , "select"+ , "function"+ ]+ }+ )+ ,+ ( C+ , FileTypeInfo+ { ftSelector = [ext ".c", ext ".C", hdr ".inc"]+ , ftKind = KindLanguage+ , ftComment = ["/*" ~~ "*/", "//" ~~ "\n"]+ , ftChar = ["'" ~~ "'"]+ , ftString = ["\"" ~~ "\""]+ , ftRawString = ["R\"" ~~ "\""]+ , ftIdentCharSet = Just (Alpha_, AlphaNum_)+ , ftKeywords =+ types+ [ "int"+ , "short"+ , "long"+ , "float"+ , "double"+ , "char"+ , "bool"+ , "int8_t"+ , "int16_t"+ , "int32_t"+ , "int64_t"+ , "int_fast8_t"+ , "int_fast16_t"+ , "int_fast32_t"+ , "int_fast64_t"+ , "int_least8_t"+ , "int_least16_t"+ , "int_least32_t"+ , "int_least64_t"+ , "intmax_t"+ , "intptr_t"+ , "uint8_t"+ , "uint16_t"+ , "uint32_t"+ , "uint64_t"+ , "uint_fast8_t"+ , "uint_fast16_t"+ , "uint_fast32_t"+ , "uint_fast64_t"+ , "uint_least8_t"+ , "uint_least16_t"+ , "uint_least32_t"+ , "uint_least64_t"+ , "uintmax_t"+ , "uintptr_t"+ ]+ <> reserved+ [ "auto"+ , "break"+ , "case"+ , "const"+ , "continue"+ , "default"+ , "do"+ , "else"+ , "enum"+ , "extern"+ , "for"+ , "goto"+ , "if"+ , "inline"+ , "register"+ , "restrict"+ , "return"+ , "signed"+ , "sizeof"+ , "static"+ , "struct"+ , "switch"+ , "typedef"+ , "union"+ , "unsigned"+ , "void"+ , "volatile"+ , "while"+ , "_Alignas"+ , "_Alignof"+ , "_Atomic"+ , "_Bool"+ , "_Complex"+ , "_Decimal128"+ , "_Decimal32"+ , "_Decimal64"+ , "_Generic"+ , "_Imaginary"+ , "_Noreturn"+ , "_Static_assert"+ , "_Thread_local"+ , "if"+ , "elif"+ , "else"+ , "endif"+ , "ifdef"+ , "ifndef"+ , "define"+ , "undef"+ , "include"+ , "line"+ , "error"+ , "pragma"+ , "defined"+ , "__has_c_attribute"+ , "_Pragma"+ , "asm"+ , "fortran"+ ]+ }+ )+ ,+ ( Cpp+ , FileTypeInfo+ { ftSelector =+ [ ext ".cpp"+ , ext ".CPP"+ , ext ".cxx"+ , ext ".cc"+ , ext ".cp"+ , ext ".c++"+ , ext ".cu"+ , ext ".cuh"+ , hdr ".tcc"+ , hdr ".h"+ , hdr ".H"+ , hdr ".hpp"+ , hdr ".ipp"+ , hdr ".HPP"+ , hdr ".hxx"+ , hdr ".hh"+ , hdr ".hp"+ , hdr ".h++"+ ]+ , ftKind = KindLanguage+ , ftComment = ["/*" ~~ "*/", "//" ~~ "\n"]+ , ftChar = ["'" ~~ "'"]+ , ftString = ["\"" ~~ "\""]+ , ftRawString = ["R\"(" ~~ ")\"", "R\"-(" ~~ ")-\"", "R\"--(" ~~ ")--\""]+ , ftIdentCharSet = Just (Alpha_, AlphaNum_)+ , ftKeywords =+ types+ [ "int"+ , "short"+ , "long"+ , "float"+ , "double"+ , "char"+ , "bool"+ , "int8_t"+ , "int16_t"+ , "int32_t"+ , "int64_t"+ , "int_fast8_t"+ , "int_fast16_t"+ , "int_fast32_t"+ , "int_fast64_t"+ , "int_least8_t"+ , "int_least16_t"+ , "int_least32_t"+ , "int_least64_t"+ , "intmax_t"+ , "intptr_t"+ , "uint8_t"+ , "uint16_t"+ , "uint32_t"+ , "uint64_t"+ , "uint_fast8_t"+ , "uint_fast16_t"+ , "uint_fast32_t"+ , "uint_fast64_t"+ , "uint_least8_t"+ , "uint_least16_t"+ , "uint_least32_t"+ , "uint_least64_t"+ , "uintmax_t"+ , "uintptr_t"+ , "nullptr_t"+ ]+ <> reserved+ [ "alignas"+ , "alignof"+ , "and"+ , "and_eq"+ , "asm"+ , "atomic_cancel"+ , "atomic_commit"+ , "atomic_noexcept"+ , "auto"+ , "bitand"+ , "bitor"+ , "break"+ , "case"+ , "catch"+ , "class"+ , "compl"+ , "concept"+ , "const"+ , "consteval"+ , "constexpr"+ , "constinit"+ , "const_cast"+ , "continue"+ , "co_await"+ , "co_return"+ , "co_yield"+ , "decltype"+ , "default"+ , "delete"+ , "do"+ , "dynamic_cast"+ , "else"+ , "enum"+ , "explicit"+ , "export"+ , "extern"+ , "false"+ , "for"+ , "friend"+ , "goto"+ , "if"+ , "inline"+ , "mutable"+ , "namespace"+ , "new"+ , "noexcept"+ , "not"+ , "not_eq"+ , "nullptr"+ , "operator"+ , "or"+ , "or_eq"+ , "private"+ , "protected"+ , "public"+ , "reflexpr"+ , "register"+ , "reinterpret_cast"+ , "requires"+ , "return"+ , "signed"+ , "sizeof"+ , "static"+ , "static_assert"+ , "static_cast"+ , "struct"+ , "switch"+ , "synchronized"+ , "template"+ , "this"+ , "thread_local"+ , "throw"+ , "true"+ , "try"+ , "typedef"+ , "typeid"+ , "typename"+ , "union"+ , "unsigned"+ , "using"+ , "virtual"+ , "void"+ , "volatile"+ , "wchar_t"+ , "while"+ , "xor"+ , "xor_eq"+ , "final"+ , "override"+ , "transaction_safe"+ , "transaction_safe_dynamic"+ , "import"+ , "module"+ , "elif"+ , "endif"+ , "ifdef"+ , "ifndef"+ , "define"+ , "undef"+ , "include"+ , "line"+ , "error"+ , "pragma"+ , "defined"+ , "__has_include"+ , "__has_cpp_attribute"+ , "export"+ , "import"+ , "module"+ ]+ }+ )+ ,+ ( CMake+ , FileTypeInfo+ { ftSelector = [name "CMakeLists.txt", ext ".cmake"]+ , ftKind = KindScript+ , ftComment = ["#" ~~ "\n"]+ , ftChar = ["'" ~~ "'"]+ , ftString = ["\"" ~~ "\""]+ , ftRawString = []+ , ftIdentCharSet = Just (Alpha_, AlphaNumDash_)+ , ftKeywords = HM.empty+ }+ )+ ,+ ( Cabal+ , FileTypeInfo+ { ftSelector = [ext ".cabal"]+ , ftKind = KindConfig+ , ftComment = ["--" ~~ "\n"]+ , ftChar = []+ , ftString = ["\"" ~~ "\""]+ , ftRawString = []+ , ftIdentCharSet = Just (Unicode_, UnicodeNum_')+ , ftKeywords = HM.empty+ }+ )+ ,+ ( Chapel+ , FileTypeInfo+ { ftSelector = [ext ".chpl"]+ , ftKind = KindLanguage+ , ftComment = ["/*" ~~ "*/", "//" ~~ "\n"]+ , ftChar = []+ , ftString = ["\"" ~~ "\"", "'" ~~ "'"]+ , ftRawString = ["\"\"\"" ~~ "\"\"\"", "'''" ~~ "'''"]+ , ftIdentCharSet = Just (Unicode_, UnicodeNumDollar_)+ , ftKeywords =+ types+ [ "void"+ , "nothing"+ , "bool"+ , "int"+ , "uint"+ , "real"+ , "imag"+ , "complex"+ , "string"+ , "bytes"+ ]+ <> reserved+ [ "align"+ , "as"+ , "atomic"+ , "begin"+ , "bool"+ , "borrowed"+ , "break"+ , "by"+ , "catch"+ , "class"+ , "cobegin"+ , "coforall"+ , "config"+ , "const"+ , "continue"+ , "defer"+ , "delete"+ , "dmapped"+ , "do"+ , "domain"+ , "else"+ , "enum"+ , "except"+ , "export"+ , "extern"+ , "false"+ , "for"+ , "forall"+ , "forwarding"+ , "if"+ , "in"+ , "index"+ , "inline"+ , "inout"+ , "iter"+ , "label"+ , "let"+ , "lifetime"+ , "local"+ , "locale"+ , "module"+ , "new"+ , "nil"+ , "noinit"+ , "on"+ , "only"+ , "otherwise"+ , "out"+ , "override"+ , "owned"+ , "param"+ , "private"+ , "prototype"+ , "proc"+ , "public"+ , "record"+ , "reduce"+ , "ref"+ , "require"+ , "return"+ , "scan"+ , "select"+ , "serial"+ , "shared"+ , "single"+ , "sparse"+ , "subdomain"+ , "sync"+ , "then"+ , "this"+ , "throw"+ , "throws"+ , "true"+ , "try"+ , "type"+ , "union"+ , "unmanaged"+ , "use"+ , "var"+ , "when"+ , "where"+ , "while"+ , "with"+ , "yield"+ , "zip"+ ]+ }+ )+ ,+ ( Clojure+ , FileTypeInfo+ { ftSelector = [ext ".clj", ext ".cljs", ext ".cljc", ext ".edn"]+ , ftKind = KindLanguage+ , ftComment = [";" ~~ "\n", ";;" ~~ "\n", ";;;" ~~ "\n"]+ , ftChar = []+ , ftString = ["\"" ~~ "\""]+ , ftRawString = []+ , ftIdentCharSet = Just (ClojureIdentStart, ClojureIdentCont)+ , ftKeywords =+ types+ [ "nil"+ , "boolean"+ , "char"+ , "byte"+ , "short"+ , "int"+ , "long"+ , "float"+ , "double"+ ]+ <> reserved+ [ "and"+ , "let"+ , "def"+ , "defn"+ , "if"+ , "else"+ , "do"+ , "quote"+ , "var"+ , "fn"+ , "loop"+ , "recur"+ , "throw"+ , "try"+ , "monitor-enter"+ , "monitor-exit"+ ]+ }+ )+ ,+ ( Coffee+ , FileTypeInfo+ { ftSelector = [ext ".coffee"]+ , ftKind = KindScript+ , ftComment = ["#" ~~ "\n", "###" ~~ "###"]+ , ftChar = []+ , ftString = ["'" ~~ "'", "\"" ~~ "\""]+ , ftRawString = ["'''" ~~ "'''", "\"\"\"" ~~ "\"\"\""]+ , ftIdentCharSet = Just (UnicodeDollar_, UnicodeNumDollar_)+ , ftKeywords =+ reserved+ [ "case"+ , "default"+ , "function"+ , "var"+ , "void"+ , "with"+ , "const"+ , "let"+ , "enum"+ , "export"+ , "import"+ , "native"+ , "__hasProp"+ , "__extends"+ , "__slice"+ , "__bind"+ , "__indexOf"+ , "implements"+ , "interface"+ , "package"+ , "private"+ , "protected"+ , "public"+ , "static"+ , "yield"+ , "true"+ , "false"+ , "null"+ , "this"+ , "new"+ , "delete"+ , "typeof"+ , "in"+ , "arguments"+ , "eval"+ , "instanceof"+ , "return"+ , "throw"+ , "break"+ , "continue"+ , "debugger"+ , "if"+ , "else"+ , "switch"+ , "for"+ , "while"+ , "do"+ , "try"+ , "catch"+ , "finally"+ , "class"+ , "extends"+ , "super"+ , "undefined"+ , "then"+ , "unless"+ , "until"+ , "loop"+ , "of"+ , "by"+ , "when"+ , "and"+ , "or"+ , "is"+ , "isnt"+ , "not"+ , "yes"+ , "no"+ , "on"+ , "off"+ ]+ }+ )+ ,+ ( Conf+ , FileTypeInfo+ { ftSelector = [ext ".config", ext ".conf", ext ".cfg", ext ".doxy"]+ , ftKind = KindConfig+ , ftComment = ["#" ~~ "\n"]+ , ftChar = []+ , ftString = ["'" ~~ "'", "\"" ~~ "\""]+ , ftRawString = []+ , ftIdentCharSet = Nothing+ , ftKeywords = HM.empty+ }+ )+ ,+ ( Csharp+ , FileTypeInfo+ { ftSelector = [ext ".cs", ext ".CS"]+ , ftKind = KindLanguage+ , ftComment = ["/*" ~~ "*/", "//" ~~ "\n"]+ , ftChar = ["'" ~~ "'"]+ , ftString = ["\"" ~~ "\""]+ , ftRawString = ["\"\"\"" ~~ "\"\"\""]+ , ftIdentCharSet = Just (CSharpIdentStart, CSharpIdentCont)+ , ftKeywords =+ types+ [ "bool"+ , "byte"+ , "sbyte"+ , "char"+ , "decimal"+ , "double"+ , "float"+ , "int"+ , "uint"+ , "nint"+ , "nuint"+ , "long"+ , "ulong"+ , "short"+ , "ushort"+ ]+ <> reserved+ [ "abstract"+ , "as"+ , "base"+ , "break"+ , "case"+ , "catch"+ , "checked"+ , "class"+ , "const"+ , "continue"+ , "default"+ , "delegate"+ , "do"+ , "else"+ , "enum"+ , "event"+ , "explicit"+ , "extern"+ , "false"+ , "finally"+ , "fixed"+ , "for"+ , "foreach"+ , "goto"+ , "if"+ , "implicit"+ , "in"+ , "interface"+ , "internal"+ , "is"+ , "lock"+ , "namespace"+ , "new"+ , "null"+ , "object"+ , "operator"+ , "out"+ , "override"+ , "params"+ , "private"+ , "protected"+ , "public"+ , "readonly"+ , "ref"+ , "return"+ , "sealed"+ , "sizeof"+ , "stackalloc"+ , "static"+ , "string"+ , "struct"+ , "switch"+ , "this"+ , "throw"+ , "true"+ , "try"+ , "typeof"+ , "unchecked"+ , "unsafe"+ , "using"+ , "virtual"+ , "void"+ , "volatile"+ , "while"+ ]+ }+ )+ ,+ ( Csh+ , FileTypeInfo+ { ftSelector = [ext ".csh", ext ".tcsh"]+ , ftKind = KindScript+ , ftComment = ["#" ~~ "\n"]+ , ftChar = []+ , ftString = ["'" ~~ "'", "\"" ~~ "\""]+ , ftRawString = []+ , ftIdentCharSet = Just (Alpha, AlphaNum_)+ , ftKeywords =+ reserved+ [ "alias"+ , "alloc"+ , "bg"+ , "bindkey"+ , "break"+ , "breaksw"+ , "built-ins"+ , "bye"+ , "case"+ , "cd"+ , "chdir"+ , "complete"+ , "continue"+ , "default"+ , "dirs"+ , "echo"+ , "echotc"+ , "else"+ , "end"+ , "endif"+ , "endsw"+ , "eval"+ , "exec"+ , "exit"+ , "fg"+ , "filetest"+ , "foreach"+ , "glob"+ , "goto"+ , "hashstat"+ , "history"+ , "hup"+ , "if"+ , "jobs"+ , "kill"+ , "limit"+ , "log"+ , "login"+ , "logout"+ , "ls-F"+ , "newgrp"+ , "nice"+ , "nohup"+ , "notify"+ , "onintr"+ , "popd"+ , "printenv"+ , "pushd"+ , "rehash"+ , "repeat"+ , "sched"+ , "set"+ , "setenv"+ , "settc"+ , "setty"+ , "shift"+ , "source"+ , "stop"+ , "suspend"+ , "switch"+ , "telltc"+ , "time"+ , "umask"+ , "unalias"+ , "uncomplete"+ , "unhash"+ , "unlimit"+ , "unset"+ , "unsetenv"+ , "wait"+ , "watchlog"+ , "where"+ , "which"+ , "while"+ ]+ }+ )+ ,+ ( Css+ , FileTypeInfo+ { ftSelector = [ext ".css"]+ , ftKind = KindMarkup+ , ftComment = ["/*" ~~ "*/"]+ , ftChar = []+ , ftString = ["\"" ~~ "\""]+ , ftRawString = []+ , ftIdentCharSet = Just (AlphaDash_, AlphaNumDash_)+ , ftKeywords = HM.empty+ }+ )+ ,+ ( Cql+ , FileTypeInfo+ { ftSelector = [ext ".cql"]+ , ftKind = KindLanguage+ , ftComment = ["--" ~~ "\n"]+ , ftChar = []+ , ftString = ["'" ~~ "'"]+ , ftRawString = []+ , ftIdentCharSet = Just (Alpha_, AlphaNum_)+ , ftKeywords =+ reserved+ [ "ASCII"+ , "ADD"+ , "AGGREGATE"+ , "ALL"+ , "ALLOW"+ , "ALTER"+ , "AND"+ , "ANY"+ , "APPLY"+ , "AS"+ , "ASC"+ , "AUTHORIZE"+ , "BATCH"+ , "BEGIN"+ , "BIGINT"+ , "BY"+ , "CLUSTERING"+ , "COLUMNFAMILY"+ , "COMPACT"+ , "CONSISTENCY"+ , "COUNT"+ , "COUNTER"+ , "CREATE"+ , "CUSTOM"+ , "DATE"+ , "DECIMAL"+ , "DELETE"+ , "DESC"+ , "DISTINCT"+ , "DOUBLE"+ , "DROP"+ , "EACH_QUORUM"+ , "ENTRIES"+ , "EXISTS"+ , "FILTERING"+ , "FLOAT"+ , "FROM"+ , "FROZEN"+ , "FULL"+ , "GRANT"+ , "IF"+ , "IN"+ , "INDEX"+ , "INET"+ , "INFINITY"+ , "INSERT"+ , "INT"+ , "INTO"+ , "KEY"+ , "KEYSPACE"+ , "KEYSPACES"+ , "LEVEL"+ , "LIMIT"+ , "LIST"+ , "LOCAL_ONE"+ , "LOCAL_QUORUM"+ , "MAP"+ , "MATERIALIZED"+ , "MODIFY"+ , "NAN"+ , "OF"+ , "ON"+ , "ONE"+ , "ORDER"+ , "PARTITION"+ , "PASSWORD"+ , "PER"+ , "PERMISSION"+ , "PERMISSIONS"+ , "PRIMARY"+ , "QUORUM"+ , "RECURSIVE"+ , "RENAME"+ , "REVOKE"+ , "SCHEMA"+ , "SELECT"+ , "SET"+ , "SMALLINT"+ , "STATIC"+ , "STORAGE"+ , "SUPERUSER"+ , "T"+ , "TABLE"+ , "TEXT"+ , "THREE"+ , "TIME"+ , "TIMESTAMP"+ , "TIMEUUID"+ , "TINYINT"+ , "TO"+ , "TOKEN"+ , "TRUNCATE"+ , "TTL"+ , "TUPLE"+ , "TWO"+ , "TYPE"+ , "UNLOGGED"+ , "UPDATE"+ , "USE"+ , "USER"+ , "USERS"+ , "USING"+ , "UUID"+ , "VALUES"+ , "VARCHAR"+ , "VARINT"+ , "VIEW"+ , "WHERE"+ , "WITH"+ , "WRITETIME"+ ]+ }+ )+ ,+ ( D+ , FileTypeInfo+ { ftSelector = [ext ".d", ext ".D", hdr ".di"]+ , ftKind = KindLanguage+ , ftComment = ["/*" ~~ "*/", "//" ~~ "\n"]+ , ftChar = ["'" ~~ "'"]+ , ftString = ["\"" ~~ "\""]+ , ftRawString = ["r\"" ~~ "\"", "`" ~~ "`"]+ , ftIdentCharSet = Just (Unicode_, UnicodeNum_)+ , ftKeywords =+ types+ [ "bool"+ , "byte"+ , "ubyte"+ , "short"+ , "ushort"+ , "int"+ , "uint"+ , "long"+ , "ulong"+ , "cent"+ , "ucent"+ , "char"+ , "wchar"+ , "dchar"+ , "float"+ , "double"+ , "real"+ , "ifloat"+ , "idouble"+ , "ireal"+ , "cfloat"+ , "cdouble"+ , "creal"+ , "void"+ ]+ <> reserved+ [ "abstract"+ , "alias"+ , "align"+ , "asm"+ , "assert"+ , "auto"+ , "body"+ , "break"+ , "case"+ , "cast"+ , "catch"+ , "class"+ , "const"+ , "continue"+ , "debug"+ , "default"+ , "delegate"+ , "delete"+ , "deprecated"+ , "do"+ , "else"+ , "enum"+ , "export"+ , "extern"+ , "false"+ , "final"+ , "finally"+ , "for"+ , "foreach"+ , "foreach_reverse"+ , "function"+ , "goto"+ , "if"+ , "immutable"+ , "import"+ , "in"+ , "inout"+ , "interface"+ , "invariant"+ , "is"+ , "lazy"+ , "macro"+ , "mixin"+ , "module"+ , "new"+ , "nothrow"+ , "null"+ , "out"+ , "override"+ , "package"+ , "pragma"+ , "private"+ , "protected"+ , "public"+ , "pure"+ , "ref"+ , "return"+ , "scope"+ , "shared"+ , "static"+ , "struct"+ , "super"+ , "switch"+ , "synchronized"+ , "template"+ , "this"+ , "throw"+ , "true"+ , "try"+ , "typeid"+ , "typeof"+ , "union"+ , "unittest"+ , "version"+ , "while"+ , "with"+ , "__FILE__"+ , "__FILE_FULL_PATH__"+ , "__MODULE__"+ , "__LINE__"+ , "__FUNCTION__"+ , "__PRETTY_FUNCTION__"+ , "__gshared"+ , "__traits"+ , "__vector"+ , "__parameters"+ ]+ }+ )+ ,+ ( Dart+ , FileTypeInfo+ { ftSelector = [ext ".dart"]+ , ftKind = KindLanguage+ , ftComment = ["/*" ~~ "*/", "//" ~~ "\n"]+ , ftChar = []+ , ftString = ["\"" ~~ "\"", "'" ~~ "'"]+ , ftRawString = ["'''" ~~ "'''", "\"\"\"" ~~ "\"\"\""]+ , ftIdentCharSet = Just (Unicode_, UnicodeNum_)+ , ftKeywords =+ types ["int", "double", "num", "bool", "null"]+ <> reserved+ [ "assert"+ , "break"+ , "case"+ , "catch"+ , "class"+ , "const"+ , "continue"+ , "default"+ , "do"+ , "else"+ , "enum"+ , "extends"+ , "false"+ , "final"+ , "finally"+ , "for"+ , "if"+ , "in"+ , "is"+ , "new"+ , "rethrow"+ , "return"+ , "super"+ , "switch"+ , "this"+ , "throw"+ , "true"+ , "try"+ , "var"+ , "void"+ , "while"+ , "with"+ , "async"+ , "hide"+ , "on"+ , "show"+ , "sync"+ , "abstract"+ , "as"+ , "covariant"+ , "deferred"+ , "dynamic"+ , "export"+ , "extension"+ , "external"+ , "factory"+ , "function"+ , "get"+ , "implements"+ , "import"+ , "interface"+ , "library"+ , "mixin"+ , "operator"+ , "part"+ , "set"+ , "static"+ , "typedef"+ , "await"+ , "yield"+ ]+ }+ )+ ,+ ( Dhall+ , FileTypeInfo+ { ftSelector = [ext ".dhall"]+ , ftKind = KindConfig+ , ftComment = ["{-" ~~ "-}", "--" ~~ "\n"]+ , ftChar = []+ , ftString = ["\"" ~~ "\""]+ , ftRawString = []+ , ftIdentCharSet = Just (Unicode_, UnicodeNum_')+ , ftKeywords =+ types ["Bool", "Natural", "Integer", "Double", "Text", "Date", "Time", "List", "Optional"]+ <> reserved+ [ "if"+ , "then"+ , "else"+ , "toMap"+ , "with"+ , "merge"+ , "showConstructor"+ , "missing"+ , "as"+ , "using"+ , "let"+ , "assert"+ ]+ }+ )+ ,+ ( Elixir+ , FileTypeInfo+ { ftSelector = [ext ".ex", ext ".exs"]+ , ftKind = KindLanguage+ , ftComment = ["#" ~~ "\n"]+ , ftChar = []+ , ftString = ["\"" ~~ "\"", "'" ~~ "'"]+ , ftRawString = []+ , ftIdentCharSet = Just (Unicode_, UnicodeNum_)+ , ftKeywords =+ types+ [ "integer"+ , "float"+ , "boolean"+ , "atom"+ , "binary"+ , "string"+ , "list"+ , "tuple"+ , "map"+ ]+ <> reserved+ [ "true"+ , "false"+ , "nil"+ , "when"+ , "and"+ , "or"+ , "not"+ , "in"+ , "fn"+ , "do"+ , "end"+ , "catch"+ , "rescue"+ , "after"+ , "else"+ ]+ }+ )+ ,+ ( Elm+ , FileTypeInfo+ { ftSelector = [ext ".elm"]+ , ftKind = KindLanguage+ , ftComment = ["{-" ~~ "-}", "--" ~~ "\n"]+ , ftChar = ["'" ~~ "'"]+ , ftString = ["\"" ~~ "\""]+ , ftRawString = ["\"\"\"" ~~ "\"\"\""]+ , ftIdentCharSet = Just (Alpha_, AlphaNum_')+ , ftKeywords =+ types+ [ "Bool"+ , "Int"+ , "Float"+ , "String"+ , "Char"+ , "List"+ , "Maybe"+ , "Result"+ , "Order"+ , "Never"+ , "Html"+ , "msg"+ , "Cmd"+ , "Sub"+ ]+ <> reserved+ [ "type"+ , "alias"+ , "port"+ , "if"+ , "then"+ , "else"+ , "case"+ , "of"+ , "let"+ , "in"+ , "infix"+ , "left"+ , "right"+ , "non"+ , "module"+ , "import"+ , "exposing"+ , "as"+ , "where"+ , "effect"+ , "command"+ , "subscription"+ , "true"+ , "false"+ , "null"+ ]+ }+ )+ ,+ ( Erlang+ , FileTypeInfo+ { ftSelector = [ext ".erl", ext ".ERL", ext ".hrl", ext ".HRL"]+ , ftKind = KindLanguage+ , ftComment = ["%" ~~ "\n"]+ , ftChar = []+ , ftString = ["\"" ~~ "\""]+ , ftRawString = []+ , ftIdentCharSet = Just (Alpha_, AlphaNum_)+ , ftKeywords =+ types+ [ "Atom"+ , "Integer"+ , "Float"+ , "Boolean"+ , "String"+ , "Tuple"+ , "List"+ , "Function"+ , "Binary"+ , "PID"+ , "Port"+ , "Reference"+ , "Map"+ ]+ <> reserved+ [ "after"+ , "and"+ , "andalso"+ , "band"+ , "begin"+ , "bnot"+ , "bor"+ , "bsl"+ , "bsr"+ , "bxor"+ , "case"+ , "catch"+ , "cond"+ , "div"+ , "end"+ , "fun"+ , "if"+ , "let"+ , "not"+ , "of"+ , "or"+ , "orelse"+ , "receive"+ , "rem"+ , "try"+ , "when"+ , "xor"+ ]+ }+ )+ ,+ ( Fish+ , FileTypeInfo+ { ftSelector = [ext ".fish"]+ , ftKind = KindScript+ , ftComment = ["#" ~~ "\n"]+ , ftChar = []+ , ftString = ["'" ~~ "'", "\"" ~~ "\""]+ , ftRawString = []+ , ftIdentCharSet = Just (Alpha_, AlphaNum_)+ , ftKeywords =+ reserved+ [ "and"+ , "argparse"+ , "begin"+ , "break"+ , "builtin"+ , "case"+ , "command"+ , "continue"+ , "else"+ , "end"+ , "eval"+ , "exec"+ , "for"+ , "function"+ , "if"+ , "not"+ , "or"+ , "read"+ , "return"+ , "set"+ , "status"+ , "string"+ , "switch"+ , "test"+ , "time"+ , "while"+ ]+ }+ )+ ,+ ( Fortran+ , FileTypeInfo+ { ftSelector =+ [ ext ".f"+ , ext ".for"+ , ext ".ftn"+ , ext ".F"+ , ext ".FOR"+ , ext ".FTN"+ , ext ".fpp"+ , ext ".FPP"+ , ext ".f90"+ , ext ".f95"+ , ext ".f03"+ , ext ".f08"+ , ext ".F90"+ , ext ".F95"+ , ext ".F03"+ , ext ".F08"+ ]+ , ftKind = KindLanguage+ , ftComment = ["!" ~~ "\n"]+ , ftChar = []+ , ftString = ["\"" ~~ "\"", "'" ~~ "'"]+ , ftRawString = []+ , ftIdentCharSet = Just (Alpha, AlphaNum_)+ , ftKeywords =+ types ["integer", "real", "complex", "logical", "character"]+ <> reserved+ [ -- fortran77+ "assign"+ , "backspace"+ , "block"+ , "data"+ , "call"+ , "close"+ , "common"+ , "continue"+ , "data"+ , "dimension"+ , "do"+ , "else"+ , "else"+ , "if"+ , "end"+ , "endfile"+ , "endif"+ , "entry"+ , "equivalence"+ , "external"+ , "format"+ , "function"+ , "goto"+ , "if"+ , "implicit"+ , "inquire"+ , "intrinsic"+ , "open"+ , "parameter"+ , "pause"+ , "print"+ , "program"+ , "read"+ , "return"+ , "rewind"+ , "rewrite"+ , "save"+ , "stop"+ , "subroutine"+ , "then"+ , "write"+ , -- fortran 90+ "allocatable"+ , "allocate"+ , "case"+ , "contains"+ , "cycle"+ , "deallocate"+ , "elsewhere"+ , "exit?"+ , "include"+ , "interface"+ , "intent"+ , "module"+ , "namelist"+ , "nullify"+ , "only"+ , "operator"+ , "optional"+ , "pointer"+ , "private"+ , "procedure"+ , "public"+ , "recursive"+ , "result"+ , "select"+ , "sequence"+ , "target"+ , "use"+ , "while"+ , "where"+ , -- fortran 95+ "elemental"+ , "forall"+ , "pure"+ , -- fortran 03+ "abstract"+ , "associate"+ , "asynchronous"+ , "bind"+ , "class"+ , "deferred"+ , "enum"+ , "enumerator"+ , "extends"+ , "final"+ , "flush"+ , "generic"+ , "import"+ , "non_overridable"+ , "nopass"+ , "pass"+ , "protected"+ , "value"+ , "volatile"+ , "wait"+ , -- fortran 08+ "block"+ , "codimension"+ , "do"+ , "concurrent"+ , "contiguous"+ , "critical"+ , "error"+ , "stop"+ , "submodule"+ , "sync"+ , "all"+ , "sync"+ , "images"+ , "sync"+ , "memory"+ , "lock"+ , "unlock"+ ]+ }+ )+ ,+ ( Fsharp+ , FileTypeInfo+ { ftSelector = [ext ".fs", ext ".fsx", ext ".fsi"]+ , ftKind = KindLanguage+ , ftComment = ["(*" ~~ "*)", "//" ~~ "\n"]+ , ftChar = ["'" ~~ "'"]+ , ftString = ["\"" ~~ "\""]+ , ftRawString = ["\"\"\"" ~~ "\"\"\""]+ , ftIdentCharSet = Just (Unicode_, UnicodeNum_')+ , ftKeywords =+ types+ [ "bool"+ , "byte"+ , "sbyte"+ , "int16"+ , "uint16"+ , "int"+ , "uint"+ , "int64"+ , "uint64"+ , "nativeint"+ , "unativeint"+ , "decimal"+ , "float"+ , "double"+ , "float32"+ , "single"+ , "char"+ , "string"+ ]+ <> reserved+ [ "abstract"+ , "and"+ , "as"+ , "assert"+ , "base"+ , "begin"+ , "class"+ , "default"+ , "delegate"+ , "do"+ , "done"+ , "downcast"+ , "downto"+ , "elif"+ , "else"+ , "end"+ , "exception"+ , "extern"+ , "FALSE"+ , "finally"+ , "fixed"+ , "for"+ , "fun"+ , "function"+ , "global"+ , "if"+ , "in"+ , "inherit"+ , "inline"+ , "interface"+ , "internal"+ , "lazy"+ , "let"+ , "let!"+ , "match"+ , "match!"+ , "member"+ , "module"+ , "mutable"+ , "namespace"+ , "new"+ , "not"+ , "null"+ , "of"+ , "open"+ , "or"+ , "override"+ , "private"+ , "public"+ , "rec"+ , "return"+ , "return!"+ , "select"+ , "static"+ , "struct"+ , "then"+ , "to"+ , "TRUE"+ , "try"+ , "type"+ , "upcast"+ , "use"+ , "use!"+ , "val"+ , "void"+ , "when"+ , "while"+ , "with"+ , "yield"+ , "yield!"+ , "const"+ , "asr"+ , "land"+ , "lor"+ , "lsl"+ , "lsr"+ , "lxor"+ , "mod"+ , "sig"+ , "break"+ , "checked"+ , "component"+ , "const"+ , "constraint"+ , "continue"+ , "event"+ , "external"+ , "include"+ , "mixin"+ , "parallel"+ , "process"+ , "protected"+ , "pure"+ , "sealed"+ , "tailcall"+ , "trait"+ , "virtual"+ ]+ }+ )+ ,+ ( Go+ , FileTypeInfo+ { ftSelector = [ext ".go"]+ , ftKind = KindLanguage+ , ftComment = ["/*" ~~ "*/", "//" ~~ "\n"]+ , ftChar = ["'" ~~ "'"]+ , ftString = ["\"" ~~ "\""]+ , ftRawString = ["`" ~~ "`"]+ , ftIdentCharSet = Just (UnicodeXIDStart_, UnicodeNumXIDCont_)+ , ftKeywords =+ types+ [ "int8"+ , "int16"+ , "int32"+ , "int64"+ , "uint8"+ , "uint16"+ , "uint32"+ , "uint64"+ , "int"+ , "uint"+ , "rune"+ , "byte"+ , "uintptr"+ , "float32"+ , "float64"+ , "complex64"+ , "complex128"+ , "string"+ , "error"+ , "bool"+ ]+ <> reserved+ [ "break"+ , "default"+ , "func"+ , "interface"+ , "select"+ , "case"+ , "defer"+ , "go"+ , "map"+ , "struct"+ , "chan"+ , "else"+ , "goto"+ , "package"+ , "switch"+ , "const"+ , "fallthrough"+ , "if"+ , "range"+ , "type"+ , "continue"+ , "for"+ , "import"+ , "return"+ , "var"+ , "append"+ , "cap"+ , "close"+ , "copy"+ , "false"+ , "float32"+ , "float64"+ , "imag"+ , "iota"+ , "len"+ , "make"+ , "new"+ , "nil"+ , "panic"+ , "print"+ , "println"+ , "real"+ , "recover"+ , "true"+ ]+ }+ )+ ,+ ( GoMod+ , FileTypeInfo+ { ftSelector = [name "go.mod"]+ , ftKind = KindConfig+ , ftComment = ["//" ~~ "\n"]+ , ftChar = []+ , ftString = ["\"" ~~ "\""]+ , ftRawString = []+ , ftIdentCharSet = Nothing+ , ftKeywords =+ reserved+ [ "module"+ , "go"+ , "require"+ , "exclude"+ , "replace"+ , "retract"+ ]+ }+ )+ ,+ ( Haskell+ , FileTypeInfo+ { ftSelector = [ext ".hs", ext ".lhs", ext ".hsc"]+ , ftKind = KindLanguage+ , ftComment = ["{-" ~~ "-}", "--" ~~ "\n"]+ , ftChar = ["'" ~~ "'"]+ , ftString = ["\"" ~~ "\""]+ , ftRawString = ["[r|" ~~ "|]", "[q|" ~~ "|]", "[s|" ~~ "|]", "[here|" ~~ "|]", "[i|" ~~ "|]"]+ , ftIdentCharSet = Just (Unicode_, UnicodeNum_')+ , ftKeywords =+ types+ [ "Bool"+ , "Char"+ , "String"+ , "ByteString"+ , "Text"+ , "Int"+ , "Int8"+ , "Int16"+ , "Int32"+ , "Int64"+ , "Word8"+ , "Word16"+ , "Word32"+ , "Word64"+ , "Integer"+ , "Float"+ , "Double"+ , "Complex"+ ]+ <> reserved+ [ "as"+ , "case"+ , "class"+ , "data"+ , "default"+ , "deriving"+ , "do"+ , "else"+ , "hiding"+ , "if"+ , "import"+ , "in"+ , "infix"+ , "infixl"+ , "infixr"+ , "instance"+ , "let"+ , "module"+ , "newtype"+ , "of"+ , "qualified"+ , "then"+ , "type"+ , "where"+ , "forall"+ , "mdo"+ , "family"+ , "role"+ , "pattern"+ , "static"+ , "group"+ , "by"+ , "using"+ , "foreign"+ , "export"+ , "label"+ , "dynamic"+ , "safe"+ , "interruptible"+ , "unsafe"+ , "stdcall"+ , "ccall"+ , "capi"+ , "prim"+ , "javascript"+ , "rec"+ , "proc"+ ]+ }+ )+ ,+ ( Html+ , FileTypeInfo+ { ftSelector = [ext ".htm", ext ".html"]+ , ftKind = KindMarkup+ , ftComment = ["<!--" ~~ "-->"]+ , ftChar = []+ , ftString = ["\"" ~~ "\""]+ , ftRawString = []+ , ftIdentCharSet = Just (HtmlIdentStart, HtmlIdentCont)+ , ftKeywords = HM.empty+ }+ )+ ,+ ( Idris+ , FileTypeInfo+ { ftSelector = [ext ".idr", ext ".lidr"]+ , ftKind = KindLanguage+ , ftComment = ["{-" ~~ "-}", "--" ~~ "\n", "|||" ~~ "\n"]+ , ftChar = ["'" ~~ "'"]+ , ftString = ["\"" ~~ "\""]+ , ftRawString = []+ , ftIdentCharSet = Just (Alpha_, AlphaNum_')+ , ftKeywords =+ types ["Int", "Integer", "Double", "Char", "String", "Ptr", "Bool"]+ <> reserved+ [ "abstract"+ , "auto"+ , "codata"+ , "data"+ , "if"+ , "parameters"+ , "public"+ , "then"+ , "total"+ , "using"+ , "case"+ , "class"+ , "concrete"+ , "covering"+ , "default"+ , "do"+ , "else"+ , "export"+ , "failing"+ , "forall"+ , "implementation"+ , "implicit"+ , "import"+ , "impossible"+ , "in"+ , "incomplete"+ , "instance"+ , "interface"+ , "let"+ , "module"+ , "mutual"+ , "namespace"+ , "of"+ , "open"+ , "params"+ , "partial"+ , "postulate"+ , "private"+ , "proof"+ , "public"+ , "record"+ , "return"+ , "rewrite"+ , "syntax"+ , "then"+ , "total"+ , "unreachable"+ , "where"+ , "with"+ , "using"+ ]+ }+ )+ ,+ ( Java+ , FileTypeInfo+ { ftSelector = [ext ".java"]+ , ftKind = KindLanguage+ , ftComment = ["/*" ~~ "*/", "//" ~~ "\n"]+ , ftChar = ["'" ~~ "'"]+ , ftString = ["\"" ~~ "\""]+ , ftRawString = []+ , ftIdentCharSet = Just (JavaIdentStart, JavaIdentCont)+ , ftKeywords =+ types+ [ "bytes"+ , "char"+ , "short"+ , "int"+ , "long"+ , "float"+ , "double"+ , "boolean"+ , "void"+ ]+ <> reserved+ [ "abstract"+ , "assert"+ , "break"+ , "case"+ , "catch"+ , "class"+ , "continue"+ , "const"+ , "default"+ , "do"+ , "else"+ , "enum"+ , "exports"+ , "extends"+ , "final"+ , "finally"+ , "for"+ , "goto"+ , "if"+ , "implements"+ , "import"+ , "instanceof"+ , "int"+ , "interface"+ , "module"+ , "native"+ , "new"+ , "package"+ , "private"+ , "protected"+ , "public"+ , "requires"+ , "return"+ , "static"+ , "strictfp"+ , "super"+ , "switch"+ , "synchronized"+ , "this"+ , "throw"+ , "throws"+ , "transient"+ , "try"+ , "var"+ , "volatile"+ , "while"+ , "true"+ , "false"+ , "null"+ ]+ }+ )+ ,+ ( Javascript+ , FileTypeInfo+ { ftSelector = [ext ".js"]+ , ftKind = KindLanguage+ , ftComment = ["/*" ~~ "*/", "//" ~~ "\n"]+ , ftChar = ["'" ~~ "'"]+ , ftString = ["\"" ~~ "\""]+ , ftRawString = []+ , ftIdentCharSet = Just (JavaIdentStart, JavaIdentCont)+ , ftKeywords =+ reserved+ [ "abstract"+ , "arguments"+ , "await"+ , "break"+ , "byte"+ , "case"+ , "catch"+ , "char"+ , "class"+ , "const"+ , "continue"+ , "debugger"+ , "default"+ , "delete"+ , "do"+ , "double"+ , "else"+ , "enum"+ , "eval"+ , "export"+ , "extends"+ , "false"+ , "final"+ , "finally"+ , "float"+ , "for"+ , "function"+ , "goto"+ , "if"+ , "implements"+ , "import"+ , "in"+ , "instanceof"+ , "int"+ , "interface"+ , "let"+ , "long"+ , "native"+ , "new"+ , "null"+ , "package"+ , "private"+ , "protected"+ , "public"+ , "return"+ , "short"+ , "static"+ , "super"+ , "switch"+ , "synchronized"+ , "this"+ , "throw"+ , "throws"+ , "transient"+ , "true"+ , "try"+ , "typeof"+ , "var"+ , "volatile"+ , "while"+ , "with"+ , "yield"+ ]+ }+ )+ ,+ ( Json+ , FileTypeInfo+ { ftSelector = [ext ".json", ext ".ndjson"]+ , ftKind = KindData+ , ftComment = []+ , ftChar = []+ , ftString = ["\"" ~~ "\""]+ , ftRawString = []+ , ftIdentCharSet = Nothing+ , ftKeywords = HM.empty+ }+ )+ ,+ ( Julia+ , FileTypeInfo+ { ftSelector = [ext ".jl"]+ , ftKind = KindLanguage+ , ftComment = ["#" ~~ "\n", "#-" ~~ "-#"]+ , ftChar = ["'" ~~ "'"]+ , ftString = ["\"" ~~ "\""]+ , ftRawString = ["\"\"\"" ~~ "\"\"\""]+ , ftIdentCharSet = Just (JuliaIdentStart, JuliaIdentCont)+ , ftKeywords =+ types+ [ "Float16"+ , "Float32"+ , "Float64"+ , "Bool"+ , "Char"+ , "Int8"+ , "UInt8"+ , "Int16"+ , "UInt16"+ , "Int32"+ , "UInt32"+ , "Int64"+ , "UInt64"+ , "Int128"+ , "UInt128"+ , -- super types+ "Integer"+ , "AbstractChar"+ , "AbstractFloat"+ , "Signed"+ , "Unsigned"+ ]+ <> reserved+ [ "baremodule"+ , "begin"+ , "break"+ , "catch"+ , "const"+ , "continue"+ , "do"+ , "else"+ , "elseif"+ , "end"+ , "export"+ , "false"+ , "finally"+ , "for"+ , "function"+ , "global"+ , "if"+ , "import"+ , "let"+ , "local"+ , "macro"+ , "module"+ , "quote"+ , "return"+ , "struct"+ , "true"+ , "try"+ , "using"+ , "while"+ ]+ }+ )+ ,+ ( Kotlin+ , FileTypeInfo+ { ftSelector = [ext ".kt", ext ".kts", ext ".ktm"]+ , ftKind = KindLanguage+ , ftComment = ["/*" ~~ "*/", "//" ~~ "\n"]+ , ftChar = ["'" ~~ "'"]+ , ftString = ["\"" ~~ "\""]+ , ftRawString = ["\"\"\"" ~~ "\"\"\""]+ , ftIdentCharSet = Just (Unicode_, UnicodeNum_)+ , ftKeywords =+ types+ [ "Byte"+ , "Short"+ , "Int"+ , "Long"+ , "Float"+ , "Double"+ , "UByte"+ , "UShort"+ , "UInt"+ , "ULong"+ , "Boolean"+ , "Char"+ ]+ <> reserved+ [ "as"+ , "break"+ , "class"+ , "continue"+ , "do"+ , "else"+ , "false"+ , "for"+ , "fun"+ , "if"+ , "in"+ , "interface"+ , "is"+ , "null"+ , "object"+ , "package"+ , "return"+ , "super"+ , "this"+ , "throw"+ , "true"+ , "try"+ , "typealias"+ , "typeof"+ , "val"+ , "var"+ , "when"+ , "while"+ , "by"+ , "catch"+ , "constructor"+ , "delegate"+ , "dynamic"+ , "field"+ , "file"+ , "finally"+ , "get"+ , "import"+ , "init"+ , "param"+ , "property"+ , "receiver"+ , "set"+ , "setparam"+ , "value"+ , "where"+ , "abstract"+ , "actual"+ , "annotation"+ , "companion"+ , "const"+ , "crossinline"+ , "data"+ , "enum"+ , "expect"+ , "external"+ , "final"+ , "infix"+ , "inline"+ , "inner"+ , "internal"+ , "lateinit"+ , "noinline"+ , "open"+ , "operator"+ , "out"+ , "override"+ , "private"+ , "protected"+ , "public"+ , "reified"+ , "sealed"+ , "suspend"+ , "tailrec"+ , "vararg"+ , "field"+ , "it"+ ]+ }+ )+ ,+ ( Ksh+ , FileTypeInfo+ { ftSelector = [ext ".ksh"]+ , ftKind = KindScript+ , ftComment = ["#" ~~ "\n"]+ , ftChar = []+ , ftString = ["'" ~~ "'", "\"" ~~ "\""]+ , ftRawString = []+ , ftIdentCharSet = Just (Alpha_, AlphaNum_)+ , ftKeywords =+ reserved+ [ "case"+ , "do"+ , "done"+ , "elif"+ , "else"+ , "esac"+ , "fi"+ , "for"+ , "function"+ , "if"+ , "in"+ , "select"+ , "then"+ , "time"+ , "until"+ , "while"+ ]+ }+ )+ ,+ ( Latex+ , FileTypeInfo+ { ftSelector = [ext ".latex", ext ".tex"]+ , ftKind = KindMarkup+ , ftComment = ["%" ~~ "\n"]+ , ftChar = []+ , ftString = ["\"" ~~ "\""]+ , ftRawString = []+ , ftIdentCharSet = Nothing+ , ftKeywords = HM.empty+ }+ )+ ,+ ( Lisp+ , FileTypeInfo+ { ftSelector = [ext ".lisp", ext ".cl"]+ , ftKind = KindLanguage+ , ftComment = [";" ~~ "\n", "#|" ~~ "|#"]+ , ftChar = []+ , ftString = ["\"" ~~ "\""]+ , ftRawString = []+ , ftIdentCharSet = Just (ListIdent, ListIdent)+ , ftKeywords =+ types+ [ "array"+ , "atom"+ , "bignum"+ , "bit"+ , "bit-vector"+ , "character"+ , "compiled-function"+ , "complex"+ , "cons"+ , "double-float"+ , "fixnum"+ , "float"+ , "function"+ , "hash-table"+ , "integer"+ , "keyword"+ , "list"+ , "long-float"+ , "nil"+ , "null"+ , "number"+ , "package"+ , "pathname"+ , "random-state"+ , "ratio"+ , "rational"+ , "readtable"+ , "sequence"+ , "short-float"+ , "signed-byte"+ , "simple-array"+ , "simple-bit-vector"+ , "simple-string"+ , "simple-vector"+ , "single-float"+ , "standard-char"+ , "stream"+ , "string"+ , "symbol"+ , "t"+ , "unsigned-byte"+ , "vector"+ ]+ }+ )+ ,+ ( Lua+ , FileTypeInfo+ { ftSelector = [ext ".lua"]+ , ftKind = KindLanguage+ , ftComment = ["--[[" ~~ "--]]", "--" ~~ "\n"]+ , ftChar = []+ , ftString = ["'" ~~ "'", "\"" ~~ "\""]+ , ftRawString = ["[===[" ~~ "]===]", "[==[" ~~ "]==]", "[=[" ~~ "]=]", "[[" ~~ "]]"]+ , ftIdentCharSet = Just (Unicode_, UnicodeNum_)+ , ftKeywords =+ reserved+ [ "and"+ , "break"+ , "do"+ , "else"+ , "elseif"+ , "end"+ , "false"+ , "for"+ , "function"+ , "if"+ , "in"+ , "local"+ , "nil"+ , "not"+ , "or"+ , "repeat"+ , "return"+ , "then"+ , "true"+ , "until"+ , "while"+ ]+ }+ )+ ,+ ( Make+ , FileTypeInfo+ { ftSelector = [name "Makefile", name "makefile", name "GNUmakefile", ext ".mk", ext ".mak", ext ".make"]+ , ftKind = KindScript+ , ftComment = ["#" ~~ "\n"]+ , ftChar = []+ , ftString = ["'" ~~ "'", "\"" ~~ "\""]+ , ftRawString = []+ , ftIdentCharSet = Just (AlphaDash_, AlphaNumDash_)+ , ftKeywords = HM.empty+ }+ )+ ,+ ( Nmap+ , FileTypeInfo+ { ftSelector = [ext ".nse"]+ , ftKind = KindScript+ , ftComment = ["--" ~~ "\n", "[[" ~~ "]]"]+ , ftChar = []+ , ftString = ["'" ~~ "'", "\"" ~~ "\""]+ , ftRawString = []+ , ftIdentCharSet = Nothing+ , ftKeywords = HM.empty+ }+ )+ ,+ ( Nim+ , FileTypeInfo+ { ftSelector = [ext ".nim"]+ , ftKind = KindLanguage+ , ftComment = ["#[" ~~ "#]", "#" ~~ "\n"]+ , ftChar = ["'" ~~ "'"]+ , ftString = ["\"" ~~ "\""]+ , ftRawString = ["\"\"\"" ~~ "\"\"\""]+ , ftIdentCharSet = Just (Unicode_, UnicodeNum_)+ , ftKeywords =+ types+ [ "int"+ , "int8"+ , "int16"+ , "int32"+ , "int64"+ , "uint"+ , "uint8"+ , "uint16"+ , "uint32"+ , "uint64"+ , "float"+ , "float32"+ , "float64"+ , "bool"+ , "string"+ , "cstring"+ ]+ <> reserved+ [ "addr"+ , "and"+ , "as"+ , "asm"+ , "bind"+ , "block"+ , "break"+ , "case"+ , "cast"+ , "concept"+ , "const"+ , "continue"+ , "converter"+ , "defer"+ , "discard"+ , "distinct"+ , "div"+ , "do"+ , "elif"+ , "else"+ , "end"+ , "enum"+ , "except"+ , "export"+ , "finally"+ , "for"+ , "from"+ , "func"+ , "if"+ , "import"+ , "in"+ , "include"+ , "interface"+ , "is"+ , "isnot"+ , "iterator"+ , "let"+ , "macro"+ , "method"+ , "mixin"+ , "mod"+ , "nil"+ , "not"+ , "notin"+ , "object"+ , "of"+ , "or"+ , "out"+ , "proc"+ , "ptr"+ , "raise"+ , "ref"+ , "return"+ , "shl"+ , "shr"+ , "static"+ , "template"+ , "try"+ , "tuple"+ , "type"+ , "using"+ , "var"+ , "when"+ , "while"+ , "xor"+ , "yield"+ ]+ }+ )+ ,+ ( OCaml+ , FileTypeInfo+ { ftSelector = [ext ".ml", hdr ".mli"]+ , ftKind = KindLanguage+ , ftComment = ["(*" ~~ "*)"]+ , ftChar = ["'" ~~ "'"]+ , ftString = ["\"" ~~ "\""]+ , ftRawString = ["{id|" ~~ "|id}"]+ , ftIdentCharSet = Just (Alpha_, AlphaNum_')+ , ftKeywords =+ types+ [ "int"+ , "float"+ , "char"+ , "string"+ , "bool"+ , "unit"+ , "list"+ , "array"+ , "exn"+ , "option"+ , "ref"+ ]+ <> reserved+ [ "and"+ , "as"+ , "assert"+ , "asr"+ , "begin"+ , "class"+ , "constraint"+ , "do"+ , "done"+ , "downto"+ , "else"+ , "end"+ , "exception"+ , "external"+ , "false"+ , "for"+ , "fun"+ , "function"+ , "functor"+ , "if"+ , "in"+ , "include"+ , "inherit"+ , "initializer"+ , "land"+ , "lazy"+ , "let"+ , "lor"+ , "lsl"+ , "lsr"+ , "lxor"+ , "match"+ , "method"+ , "mod"+ , "module"+ , "mutable"+ , "new"+ , "nonrec"+ , "object"+ , "of"+ , "open"+ , "or"+ , "private"+ , "rec"+ , "sig"+ , "struct"+ , "then"+ , "to"+ , "true"+ , "try"+ , "type"+ , "val"+ , "virtual"+ , "when"+ , "while"+ , "with"+ ]+ }+ )+ ,+ ( ObjectiveC+ , FileTypeInfo+ { ftSelector = [ext ".m", ext ".mi"]+ , ftKind = KindLanguage+ , ftComment = ["/*" ~~ "*/", "//" ~~ "\n"]+ , ftChar = ["'" ~~ "'"]+ , ftString = ["\"" ~~ "\""]+ , ftRawString = []+ , ftIdentCharSet = Just (Unicode_, UnicodeNum_)+ , ftKeywords =+ types+ [ "char"+ , "int"+ , "short"+ , "long"+ , "float"+ , "double"+ , "signed"+ , "unsigned"+ ]+ <> reserved+ [ "void"+ , "id"+ , "const"+ , "volatile"+ , "in"+ , "out"+ , "inout"+ , "bycopy"+ , "byref"+ , "oneway"+ , "self"+ , "super"+ , "interface"+ , "end"+ , "@implementation"+ , "@end"+ , "@interface"+ , "@end"+ , "@implementation"+ , "@end"+ , "@protoco"+ , "@end"+ , "@class"+ ]+ }+ )+ ,+ ( PHP+ , FileTypeInfo+ { ftSelector = [ext ".php", ext ".php3", ext ".php4", ext ".php5", ext ".phtml"]+ , ftKind = KindScript+ , ftComment = ["/*" ~~ "*/", "//" ~~ "\n", "#" ~~ "\n"]+ , ftChar = ["'" ~~ "'"]+ , ftString = ["'" ~~ "'", "\"" ~~ "\""]+ , ftRawString = ["<<END" ~~ "END;", "<<'END'" ~~ "END;"]+ , ftIdentCharSet = Just (Unicode_, UnicodeNum_)+ , ftKeywords =+ reserved+ [ "__halt_compiler"+ , "abstract"+ , "and"+ , "array"+ , "as"+ , "break"+ , "callable"+ , "case"+ , "catch"+ , "class"+ , "clone"+ , "const"+ , "continue"+ , "declare"+ , "default"+ , "die"+ , "do"+ , "echo"+ , "else"+ , "elseif"+ , "empty"+ , "enddeclare"+ , "endfor"+ , "endforeach"+ , "endif"+ , "endswitch"+ , "endwhile"+ , "eval"+ , "exit"+ , "extends"+ , "final"+ , "finally"+ , "fn"+ , "for"+ , "foreach"+ , "function"+ , "global"+ , "goto"+ , "if"+ , "implements"+ , "include"+ , "include_once"+ , "instanceof"+ , "insteadof"+ , "interface"+ , "isset"+ , "list"+ , "match"+ , "namespace"+ , "new"+ , "or"+ , "print"+ , "private"+ , "protected"+ , "public"+ , "readonly"+ , "require"+ , "require_once"+ , "return"+ , "static"+ , "switch"+ , "throw"+ , "trait"+ , "try"+ , "unset"+ , "use"+ , "var"+ , "while"+ , "xor"+ , "yield"+ , "yield"+ , "from"+ ]+ }+ )+ ,+ ( Perl+ , FileTypeInfo+ { ftSelector = [ext ".pl", ext ".pm", ext ".pm6", ext ".plx", ext ".perl"]+ , ftKind = KindScript+ , ftComment = ["=pod" ~~ "=cut", "#" ~~ "\n"]+ , ftChar = []+ , ftString = ["'" ~~ "'", "\"" ~~ "\""]+ , ftRawString = ["<<\"END\";" ~~ "END", "<<'END'" ~~ "END", "<<'EOT';" ~~ "EOT", "<<\"EOT\";" ~~ "EOT"]+ , ftIdentCharSet = Just (Alpha_, AlphaNum_)+ , ftKeywords = reserved []+ }+ )+ ,+ ( Python+ , FileTypeInfo+ { ftSelector = [ext ".py", ext ".pyx", ext ".pxd", ext ".pxi", ext ".scons"]+ , ftKind = KindLanguage+ , ftComment = ["#" ~~ "\n"]+ , ftChar = []+ , ftString = ["'" ~~ "'", "\"" ~~ "\""]+ , ftRawString = ["\"\"\"" ~~ "\"\"\"", "'''" ~~ "'''", "r'" ~~ "'"]+ , ftIdentCharSet = Just (Unicode_, UnicodeNum_)+ , ftKeywords =+ reserved+ [ "False"+ , "await"+ , "else"+ , "import"+ , "pass"+ , "None"+ , "break"+ , "except"+ , "in"+ , "raise"+ , "True"+ , "class"+ , "finally"+ , "is"+ , "return"+ , "and"+ , "continue"+ , "for"+ , "lambda"+ , "try"+ , "as"+ , "def"+ , "from"+ , "nonlocal"+ , "while"+ , "assert"+ , "del"+ , "global"+ , "not"+ , "with"+ , "async"+ , "elif"+ , "if"+ , "or"+ , "yield"+ ]+ }+ )+ ,+ ( R+ , FileTypeInfo+ { ftSelector = [ext ".r", ext ".rdata", ext ".rds", ext ".rda"]+ , ftKind = KindLanguage+ , ftComment = ["#" ~~ "\n"]+ , ftChar = []+ , ftString = ["\"" ~~ "\"", "'" ~~ "'"]+ , ftRawString = []+ , ftIdentCharSet = Just (Unicode_, UnicodeNum_)+ , ftKeywords =+ reserved+ [ "if"+ , "else"+ , "repeat"+ , "while"+ , "function"+ , "for"+ , "in"+ , "next"+ , "break"+ , "TRUE"+ , "FALSE"+ , "NULL"+ , "Inf"+ , "NaN"+ , "NA"+ , "NA_integer_"+ , "NA_real_"+ , "NA_complex_"+ , "NA_character_"+ , "…"+ ]+ }+ )+ ,+ ( Ruby+ , FileTypeInfo+ { ftSelector = [ext ".rb", ext ".ruby"]+ , ftKind = KindLanguage+ , ftComment = ["=begin" ~~ "=end", "#" ~~ "\n"]+ , ftChar = ["'" ~~ "'"]+ , ftString = ["'" ~~ "'", "\"" ~~ "\"", "%|" ~~ "|", "%q(" ~~ ")", "%Q(" ~~ ")"]+ , ftRawString = []+ , ftIdentCharSet = Just (Unicode_, UnicodeNum_)+ , ftKeywords =+ reserved+ [ "BEGIN"+ , "END"+ , "alias"+ , "and"+ , "begin"+ , "break"+ , "case"+ , "class"+ , "def"+ , "module"+ , "next"+ , "nil"+ , "not"+ , "or"+ , "redo"+ , "rescue"+ , "retry"+ , "return"+ , "elsif"+ , "end"+ , "false"+ , "ensure"+ , "for"+ , "if"+ , "true"+ , "undef"+ , "unless"+ , "do"+ , "else"+ , "super"+ , "then"+ , "until"+ , "when"+ , "while"+ , "defined?"+ , "self"+ ]+ }+ )+ ,+ ( Rust+ , FileTypeInfo+ { ftSelector = [ext ".rs"]+ , ftKind = KindLanguage+ , ftComment = ["/*" ~~ "*/", "//" ~~ "\n"]+ , ftChar = ["'" ~~ "'"]+ , ftString = ["\"" ~~ "\""]+ , ftRawString = ["r##\"" ~~ "\"##", "r#\"" ~~ "\"#", "r\"" ~~ "\""]+ , ftIdentCharSet = Just (UnicodeXIDStart_, UnicodeNumXIDCont_)+ , ftKeywords =+ types+ [ "i8"+ , "u8"+ , "i16"+ , "u16"+ , "i32"+ , "u32"+ , "i64"+ , "u64"+ , "i128"+ , "u128"+ , "isize"+ , "usize"+ , "bool"+ , "char"+ , "str"+ , "String"+ ]+ <> reserved+ [ "as"+ , "use"+ , "extern"+ , "crate"+ , "break"+ , "const"+ , "continue"+ , "crate"+ , "else"+ , "if"+ , "let"+ , "enum"+ , "extern"+ , "false"+ , "fn"+ , "for"+ , "if"+ , "impl"+ , "in"+ , "for"+ , "let"+ , "loop"+ , "match"+ , "mod"+ , "move"+ , "mut"+ , "pub"+ , "impl"+ , "ref"+ , "return"+ , "Self"+ , "self"+ , "static"+ , "struct"+ , "super"+ , "trait"+ , "true"+ , "type"+ , "unsafe"+ , "use"+ , "where"+ , "while"+ , "abstract"+ , "alignof"+ , "become"+ , "box"+ , "do"+ , "final"+ , "macro"+ , "offsetof"+ , "override"+ , "priv"+ , "proc"+ , "pure"+ , "sizeof"+ , "typeof"+ , "unsized"+ , "virtual"+ , "yield"+ , "async"+ , "await"+ , "dyn"+ , -- weak keywords+ "macro_rules"+ , "union"+ , "'static"+ , -- reserved for future use+ "abstract"+ , "become"+ , "box"+ , "do"+ , "final"+ , "macro"+ , "override"+ , "priv"+ , "typeof"+ , "unsized"+ , "virtual"+ , "yield"+ , "try"+ ]+ }+ )+ ,+ ( Scala+ , FileTypeInfo+ { ftSelector = [ext ".scala"]+ , ftKind = KindLanguage+ , ftComment = ["/*" ~~ "*/", "//" ~~ "\n"]+ , ftChar = ["'" ~~ "'"]+ , ftString = ["\"" ~~ "\"", "'" ~~ "'"]+ , ftRawString = []+ , ftIdentCharSet = Just (Unicode_, UnicodeNum_)+ , ftKeywords =+ types+ [ "Byte"+ , "Short"+ , "Int"+ , "Long"+ , "Float"+ , "Double"+ , "Char"+ , "String"+ , "Boolean"+ , "Unit"+ , "Null"+ , "Nothing"+ , "Any"+ , "AnyRef"+ ]+ <> reserved+ [ "abstract"+ , "case"+ , "catch"+ , "class"+ , "def"+ , "do"+ , "else"+ , "extends"+ , "false"+ , "final"+ , "finally"+ , "for"+ , "forSome"+ , "if"+ , "implicit"+ , "import"+ , "lazy"+ , "match"+ , "new"+ , "null"+ , "object"+ , "override"+ , "package"+ , "private"+ , "protected"+ , "return"+ , "sealed"+ , "super"+ , "this"+ , "throw"+ , "trait"+ , "true"+ , "try"+ , "type"+ , "val"+ , "var"+ , "while"+ , "with"+ , "yield"+ ]+ }+ )+ ,+ ( SmallTalk+ , FileTypeInfo+ { ftSelector = [ext ".st", ext ".gst"]+ , ftKind = KindLanguage+ , ftComment = ["\"" ~~ "\""]+ , ftChar = ["$" ~~ ""]+ , ftString = ["'" ~~ "'"]+ , ftRawString = []+ , ftIdentCharSet = Just (Unicode_, UnicodeNum_)+ , ftKeywords =+ reserved+ [ "true"+ , "false"+ , "nil"+ , "self"+ , "super"+ , "thisContext"+ ]+ }+ )+ ,+ ( Swift+ , FileTypeInfo+ { ftSelector = [ext ".swift"]+ , ftKind = KindLanguage+ , ftComment = ["/*" ~~ "*/", "//" ~~ "\n"]+ , ftChar = []+ , ftString = ["\"" ~~ "\""]+ , ftRawString = ["\"\"\"" ~~ "\"\"\""]+ , ftIdentCharSet = Just (Unicode_, UnicodeNum_)+ , ftKeywords =+ types+ [ "Int"+ , "Int8"+ , "Int16"+ , "Int32"+ , "Int64"+ , "UInt"+ , "UInt8"+ , "UInt16"+ , "UInt32"+ , "UInt64"+ , "Float"+ , "Double"+ , "Float80"+ , "Float16"+ , "Bool"+ , "String"+ , "Character"+ , "Array"+ , "Dictionary"+ , "Set"+ , "Optional"+ , "Any"+ , "AnyObject"+ ]+ <> reserved+ [ "associatedtype"+ , "class"+ , "deinit"+ , "enum"+ , "extension"+ , "fileprivate"+ , "func"+ , "import"+ , "init"+ , "inout"+ , "internal"+ , "let"+ , "open"+ , "operator"+ , "private"+ , "precedencegroup"+ , "protocol"+ , "public"+ , "rethrows"+ , "static"+ , "struct"+ , "subscript"+ , "typealias"+ , "var"+ , "break"+ , "case"+ , "catch"+ , "continue"+ , "default"+ , "defer"+ , "do"+ , "else"+ , "fallthrough"+ , "for"+ , "guard"+ , "if"+ , "in"+ , "repeat"+ , "return"+ , "throw"+ , "switch"+ , "where"+ , "while"+ , "Any"+ , "as"+ , "catch"+ , "false"+ , "is"+ , "nil"+ , "rethrows"+ , "self"+ , "Self"+ , "super"+ , "throw"+ , "throws"+ , "true"+ , "try"+ , "#available"+ , "#colorLiteral"+ , "#column"+ , "#dsohandle"+ , "#elseif"+ , "#else"+ , "#endif"+ , "#error"+ , "#fileID"+ , "#fileLiteral"+ , "#filePath"+ , "#file"+ , "#function"+ , "#if"+ , "#imageLiteral"+ , "#keyPath"+ , "#line"+ , "#selector"+ , "#sourceLocation"+ , "#warning"+ , "associativity"+ , "convenience"+ , "didSet"+ , "dynamic"+ , "final"+ , "get"+ , "indirect"+ , "infix"+ , "lazy"+ , "left"+ , "mutating"+ , "none"+ , "nonmutating"+ , "optional"+ , "override"+ , "postfix"+ , "precedence"+ , "prefix"+ , "Protocol"+ , "required"+ , "right"+ , "set"+ , "some"+ , "Type"+ , "unowned"+ , "weak"+ , "willSet"+ ]+ }+ )+ ,+ ( Sql+ , FileTypeInfo+ { ftSelector = [ext ".sql"]+ , ftKind = KindLanguage+ , ftComment = ["--" ~~ "\n"]+ , ftChar = []+ , ftString = ["'" ~~ "'"]+ , ftRawString = []+ , ftIdentCharSet = Just (Alpha_, AlphaNum_)+ , ftKeywords =+ reserved+ [ "ABORT"+ , "ABORTSESSION"+ , "ABS"+ , "ABSOLUTE"+ , "ACCESS"+ , "ACCESSIBLE"+ , "ACCESS_LOCK"+ , "ACCOUNT"+ , "ACOS"+ , "ACOSH"+ , "ACTION"+ , "ADD"+ , "ADD_MONTHS"+ , "ADMIN"+ , "AFTER"+ , "AGGREGATE"+ , "ALIAS"+ , "ALL"+ , "ALLOCATE"+ , "ALLOW"+ , "ALTER"+ , "ALTERAND"+ , "AMP"+ , "ANALYSE"+ , "ANALYZE"+ , "AND"+ , "ANSIDATE"+ , "ANY"+ , "ARE"+ , "ARRAY"+ , "ARRAY_AGG"+ , "ARRAY_EXISTS"+ , "ARRAY_MAX_CARDINALITY"+ , "AS"+ , "ASC"+ , "ASENSITIVE"+ , "ASIN"+ , "ASINH"+ , "ASSERTION"+ , "ASSOCIATE"+ , "ASUTIME"+ , "ASYMMETRIC"+ , "AT"+ , "ATAN"+ , "ATAN2"+ , "ATANH"+ , "ATOMIC"+ , "AUDIT"+ , "AUTHORIZATION"+ , "AUX"+ , "AUXILIARY"+ , "AVE"+ , "AVERAGE"+ , "AVG"+ , "BACKUP"+ , "BEFORE"+ , "BEGIN"+ , "BEGIN_FRAME"+ , "BEGIN_PARTITION"+ , "BETWEEN"+ , "BIGINT"+ , "BINARY"+ , "BIT"+ , "BLOB"+ , "BOOLEAN"+ , "BOTH"+ , "BREADTH"+ , "BREAK"+ , "BROWSE"+ , "BT"+ , "BUFFERPOOL"+ , "BULK"+ , "BUT"+ , "BY"+ , "BYTE"+ , "BYTEINT"+ , "BYTES"+ , "CALL"+ , "CALLED"+ , "CAPTURE"+ , "CARDINALITY"+ , "CASCADE"+ , "CASCADED"+ , "CASE"+ , "CASESPECIFIC"+ , "CASE_N"+ , "CAST"+ , "CATALOG"+ , "CCSID"+ , "CD"+ , "CEIL"+ , "CEILING"+ , "CHANGE"+ , "CHAR"+ , "CHAR2HEXINT"+ , "CHARACTER"+ , "CHARACTERS"+ , "CHARACTER_LENGTH"+ , "CHARS"+ , "CHAR_LENGTH"+ , "CHECK"+ , "CHECKPOINT"+ , "CLASS"+ , "CLASSIFIER"+ , "CLOB"+ , "CLONE"+ , "CLOSE"+ , "CLUSTER"+ , "CLUSTERED"+ , "CM"+ , "COALESCE"+ , "COLLATE"+ , "COLLATION"+ , "COLLECT"+ , "COLLECTION"+ , "COLLID"+ , "COLUMN"+ , "COLUMN_VALUE"+ , "COMMENT"+ , "COMMIT"+ , "COMPLETION"+ , "COMPRESS"+ , "COMPUTE"+ , "CONCAT"+ , "CONCURRENTLY"+ , "CONDITION"+ , "CONNECT"+ , "CONNECTION"+ , "CONSTRAINT"+ , "CONSTRAINTS"+ , "CONSTRUCTOR"+ , "CONTAINS"+ , "CONTAINSTABLE"+ , "CONTENT"+ , "CONTINUE"+ , "CONVERT"+ , "CONVERT_TABLE_HEADER"+ , "COPY"+ , "CORR"+ , "CORRESPONDING"+ , "COS"+ , "COSH"+ , "COUNT"+ , "COVAR_POP"+ , "COVAR_SAMP"+ , "CREATE"+ , "CROSS"+ , "CS"+ , "CSUM"+ , "CT"+ , "CUBE"+ , "CUME_DIST"+ , "CURRENT"+ , "CURRENT_CATALOG"+ , "CURRENT_DATE"+ , "CURRENT_DEFAULT_TRANSFORM_GROUP"+ , "CURRENT_LC_CTYPE"+ , "CURRENT_PATH"+ , "CURRENT_ROLE"+ , "CURRENT_ROW"+ , "CURRENT_SCHEMA"+ , "CURRENT_SERVER"+ , "CURRENT_TIME"+ , "CURRENT_TIMESTAMP"+ , "CURRENT_TIMEZONE"+ , "CURRENT_TRANSFORM_GROUP_FOR_TYPE"+ , "CURRENT_USER"+ , "CURRVAL"+ , "CURSOR"+ , "CV"+ , "CYCLE"+ , "DATA"+ , "DATABASE"+ , "DATABASES"+ , "DATABLOCKSIZE"+ , "DATE"+ , "DATEFORM"+ , "DAY"+ , "DAYS"+ , "DAY_HOUR"+ , "DAY_MICROSECOND"+ , "DAY_MINUTE"+ , "DAY_SECOND"+ , "DBCC"+ , "DBINFO"+ , "DEALLOCATE"+ , "DEC"+ , "DECFLOAT"+ , "DECIMAL"+ , "DECLARE"+ , "DEFAULT"+ , "DEFERRABLE"+ , "DEFERRED"+ , "DEFINE"+ , "DEGREES"+ , "DEL"+ , "DELAYED"+ , "DELETE"+ , "DENSE_RANK"+ , "DENY"+ , "DEPTH"+ , "DEREF"+ , "DESC"+ , "DESCRIBE"+ , "DESCRIPTOR"+ , "DESTROY"+ , "DESTRUCTOR"+ , "DETERMINISTIC"+ , "DIAGNOSTIC"+ , "DIAGNOSTICS"+ , "DICTIONARY"+ , "DISABLE"+ , "DISABLED"+ , "DISALLOW"+ , "DISCONNECT"+ , "DISK"+ , "DISTINCT"+ , "DISTINCTROW"+ , "DISTRIBUTED"+ , "DIV"+ , "DO"+ , "DOCUMENT"+ , "DOMAIN"+ , "DOUBLE"+ , "DROP"+ , "DSSIZE"+ , "DUAL"+ , "DUMP"+ , "DYNAMIC"+ , "EACH"+ , "ECHO"+ , "EDITPROC"+ , "ELEMENT"+ , "ELSE"+ , "ELSEIF"+ , "EMPTY"+ , "ENABLED"+ , "ENCLOSED"+ , "ENCODING"+ , "ENCRYPTION"+ , "END"+ , "END"+ , "EXEC"+ , "ENDING"+ , "END_FRAME"+ , "END_PARTITION"+ , "EQ"+ , "EQUALS"+ , "ERASE"+ , "ERRLVL"+ , "ERROR"+ , "ERRORFILES"+ , "ERRORTABLES"+ , "ESCAPE"+ , "ESCAPED"+ , "ET"+ , "EVERY"+ , "EXCEPT"+ , "EXCEPTION"+ , "EXCLUSIVE"+ , "EXEC"+ , "EXECUTE"+ , "EXISTS"+ , "EXIT"+ , "EXP"+ , "EXPLAIN"+ , "EXTERNAL"+ , "EXTRACT"+ , "FALLBACK"+ , "FALSE"+ , "FASTEXPORT"+ , "FENCED"+ , "FETCH"+ , "FIELDPROC"+ , "FILE"+ , "FILLFACTOR"+ , "FILTER"+ , "FINAL"+ , "FIRST"+ , "FIRST_VALUE"+ , "FLOAT"+ , "FLOAT4"+ , "FLOAT8"+ , "FLOOR"+ , "FOR"+ , "FORCE"+ , "FOREIGN"+ , "FORMAT"+ , "FOUND"+ , "FRAME_ROW"+ , "FREE"+ , "FREESPACE"+ , "FREETEXT"+ , "FREETEXTTABLE"+ , "FREEZE"+ , "FROM"+ , "FULL"+ , "FULLTEXT"+ , "FUNCTION"+ , "FUSION"+ , "GE"+ , "GENERAL"+ , "GENERATED"+ , "GET"+ , "GIVE"+ , "GLOBAL"+ , "GO"+ , "GOTO"+ , "GRANT"+ , "GRAPHIC"+ , "GROUP"+ , "GROUPING"+ , "GROUPS"+ , "GT"+ , "HANDLER"+ , "HASH"+ , "HASHAMP"+ , "HASHBAKAMP"+ , "HASHBUCKET"+ , "HASHROW"+ , "HAVING"+ , "HELP"+ , "HIGH_PRIORITY"+ , "HOLD"+ , "HOLDLOCK"+ , "HOST"+ , "HOUR"+ , "HOURS"+ , "HOUR_MICROSECOND"+ , "HOUR_MINUTE"+ , "HOUR_SECOND"+ , "IDENTIFIED"+ , "IDENTITY"+ , "IDENTITYCOL"+ , "IDENTITY_INSERT"+ , "IF"+ , "IGNORE"+ , "ILIKE"+ , "IMMEDIATE"+ , "IN"+ , "INCLUSIVE"+ , "INCONSISTENT"+ , "INCREMENT"+ , "INDEX"+ , "INDICATOR"+ , "INFILE"+ , "INHERIT"+ , "INITIAL"+ , "INITIALIZE"+ , "INITIALLY"+ , "INITIATE"+ , "INNER"+ , "INOUT"+ , "INPUT"+ , "INS"+ , "INSENSITIVE"+ , "INSERT"+ , "INSTEAD"+ , "INT"+ , "INT1"+ , "INT2"+ , "INT3"+ , "INT4"+ , "INT8"+ , "INTEGER"+ , "INTEGERDATE"+ , "INTERSECT"+ , "INTERSECTION"+ , "INTERVAL"+ , "INTO"+ , "IO_AFTER_GTIDS"+ , "IO_BEFORE_GTIDS"+ , "IS"+ , "ISNULL"+ , "ISOBID"+ , "ISOLATION"+ , "ITERATE"+ , "JAR"+ , "JOIN"+ , "JOURNAL"+ , "JSON_ARRAY"+ , "JSON_ARRAYAGG"+ , "JSON_EXISTS"+ , "JSON_OBJECT"+ , "JSON_OBJECTAGG"+ , "JSON_QUERY"+ , "JSON_TABLE"+ , "JSON_TABLE_PRIMITIVE"+ , "JSON_VALUE"+ , "KEEP"+ , "KEY"+ , "KEYS"+ , "KILL"+ , "KURTOSIS"+ , "LABEL"+ , "LAG"+ , "FTUAGE"+ , "LARGE"+ , "LAST"+ , "LAST_VALUE"+ , "LATERAL"+ , "LC_CTYPE"+ , "LE"+ , "LEAD"+ , "LEADING"+ , "LEAVE"+ , "LEFT"+ , "LESS"+ , "LEVEL"+ , "LIKE"+ , "LIKE_REGEX"+ , "LIMIT"+ , "LINEAR"+ , "LINENO"+ , "LINES"+ , "LISTAGG"+ , "LN"+ , "LOAD"+ , "LOADING"+ , "LOCAL"+ , "LOCALE"+ , "LOCALTIME"+ , "LOCALTIMESTAMP"+ , "LOCATOR"+ , "LOCATORS"+ , "LOCK"+ , "LOCKING"+ , "LOCKMAX"+ , "LOCKSIZE"+ , "LOG"+ , "LOG10"+ , "LOGGING"+ , "LOGON"+ , "LONG"+ , "LONGBLOB"+ , "LONGTEXT"+ , "LOOP"+ , "LOWER"+ , "LOW_PRIORITY"+ , "LT"+ , "MACRO"+ , "MAINTAINED"+ , "MAP"+ , "MASTER_BIND"+ , "MASTER_SSL_VERIFY_SERVER_CERT"+ , "MATCH"+ , "MATCHES"+ , "MATCH_NUMBER"+ , "MATCH_RECOGNIZE"+ , "MATERIALIZED"+ , "MAVG"+ , "MAX"+ , "MAXEXTENTS"+ , "MAXIMUM"+ , "MAXVALUE"+ , "MCHARACTERS"+ , "MDIFF"+ , "MEDIUMBLOB"+ , "MEDIUMINT"+ , "MEDIUMTEXT"+ , "MEMBER"+ , "MERGE"+ , "METHOD"+ , "MICROSECOND"+ , "MICROSECONDS"+ , "MIDDLEINT"+ , "MIN"+ , "MINDEX"+ , "MINIMUM"+ , "MINUS"+ , "MINUTE"+ , "MINUTES"+ , "MINUTE_MICROSECOND"+ , "MINUTE_SECOND"+ , "MLINREG"+ , "MLOAD"+ , "MLSLABEL"+ , "MOD"+ , "MODE"+ , "MODIFIES"+ , "MODIFY"+ , "MODULE"+ , "MONITOR"+ , "MONRESOURCE"+ , "MONSESSION"+ , "MONTH"+ , "MONTHS"+ , "MSUBSTR"+ , "MSUM"+ , "MULTISET"+ , "NAMED"+ , "NAMES"+ , "NATIONAL"+ , "NATURAL"+ , "NCHAR"+ , "NCLOB"+ , "NE"+ , "NESTED_TABLE_ID"+ , "NEW"+ , "NEW_TABLE"+ , "NEXT"+ , "NEXTVAL"+ , "NO"+ , "NOAUDIT"+ , "NOCHECK"+ , "NOCOMPRESS"+ , "NONCLUSTERED"+ , "NONE"+ , "NORMALIZE"+ , "NOT"+ , "NOTNULL"+ , "NOWAIT"+ , "NO_WRITE_TO_BINLOG"+ , "NTH_VALUE"+ , "NTILE"+ , "NULL"+ , "NULLIF"+ , "NULLIFZERO"+ , "NULLS"+ , "NUMBER"+ , "NUMERIC"+ , "NUMPARTS"+ , "OBID"+ , "OBJECT"+ , "OBJECTS"+ , "OCCURRENCES_REGEX"+ , "OCTET_LENGTH"+ , "OF"+ , "OFF"+ , "OFFLINE"+ , "OFFSET"+ , "OFFSETS"+ , "OLD"+ , "OLD_TABLE"+ , "OMIT"+ , "ON"+ , "ONE"+ , "ONLINE"+ , "ONLY"+ , "OPEN"+ , "OPENDATASOURCE"+ , "OPENQUERY"+ , "OPENROWSET"+ , "OPENXML"+ , "OPERATION"+ , "OPTIMIZATION"+ , "OPTIMIZE"+ , "OPTIMIZER_COSTS"+ , "OPTION"+ , "OPTIONALLY"+ , "OR"+ , "ORDER"+ , "ORDINALITY"+ , "ORGANIZATION"+ , "OUT"+ , "OUTER"+ , "OUTFILE"+ , "OUTPUT"+ , "OVER"+ , "OVERLAPS"+ , "OVERLAY"+ , "OVERRIDE"+ , "PACKAGE"+ , "PAD"+ , "PADDED"+ , "PARAMETER"+ , "PARAMETERS"+ , "PART"+ , "PARTIAL"+ , "PARTITION"+ , "PARTITIONED"+ , "PARTITIONING"+ , "PASSWORD"+ , "PATH"+ , "PATTERN"+ , "PCTFREE"+ , "PER"+ , "PERCENT"+ , "PERCENTILE_CONT"+ , "PERCENTILE_DISC"+ , "PERCENT_RANK"+ , "PERIOD"+ , "PERM"+ , "PERMANENT"+ , "PIECESIZE"+ , "PIVOT"+ , "PLACING"+ , "PLAN"+ , "PORTION"+ , "POSITION"+ , "POSITION_REGEX"+ , "POSTFIX"+ , "POWER"+ , "PRECEDES"+ , "PRECISION"+ , "PREFIX"+ , "PREORDER"+ , "PREPARE"+ , "PRESERVE"+ , "PREVVAL"+ , "PRIMARY"+ , "PRINT"+ , "PRIOR"+ , "PRIQTY"+ , "PRIVATE"+ , "PRIVILEGES"+ , "PROC"+ , "PROCEDURE"+ , "PROFILE"+ , "PROGRAM"+ , "PROPORTIONAL"+ , "PROTECTION"+ , "PSID"+ , "PTF"+ , "PUBLIC"+ , "PURGE"+ , "QUALIFIED"+ , "QUALIFY"+ , "QUANTILE"+ , "QUERY"+ , "QUERYNO"+ , "RADIANS"+ , "RAISERROR"+ , "RANDOM"+ , "RANGE"+ , "RANGE_N"+ , "RANK"+ , "RAW"+ , "READ"+ , "READS"+ , "READTEXT"+ , "READ_WRITE"+ , "REAL"+ , "RECONFIGURE"+ , "RECURSIVE"+ , "REF"+ , "REFERENCES"+ , "REFERENCING"+ , "REFRESH"+ , "REGEXP"+ , "REGR_AVGX"+ , "REGR_AVGY"+ , "REGR_COUNT"+ , "REGR_INTERCEPT"+ , "REGR_R2"+ , "REGR_SLOPE"+ , "REGR_SXX"+ , "REGR_SXY"+ , "REGR_SYY"+ , "RELATIVE"+ , "RELEASE"+ , "RENAME"+ , "REPEAT"+ , "REPLACE"+ , "REPLICATION"+ , "REPOVERRIDE"+ , "REQUEST"+ , "REQUIRE"+ , "RESIGNAL"+ , "RESOURCE"+ , "RESTART"+ , "RESTORE"+ , "RESTRICT"+ , "RESULT"+ , "RESULT_SET_LOCATOR"+ , "RESUME"+ , "RET"+ , "RETRIEVE"+ , "RETURN"+ , "RETURNING"+ , "RETURNS"+ , "REVALIDATE"+ , "REVERT"+ , "REVOKE"+ , "RIGHT"+ , "RIGHTS"+ , "RLIKE"+ , "ROLE"+ , "ROLLBACK"+ , "ROLLFORWARD"+ , "ROLLUP"+ , "ROUND_CEILING"+ , "ROUND_DOWN"+ , "ROUND_FLOOR"+ , "ROUND_HALF_DOWN"+ , "ROUND_HALF_EVEN"+ , "ROUND_HALF_UP"+ , "ROUND_UP"+ , "ROUTINE"+ , "ROW"+ , "ROWCOUNT"+ , "ROWGUIDCOL"+ , "ROWID"+ , "ROWNUM"+ , "ROWS"+ , "ROWSET"+ , "ROW_NUMBER"+ , "RULE"+ , "RUN"+ , "RUNNING"+ , "SAMPLE"+ , "SAMPLEID"+ , "SAVE"+ , "SAVEPOINT"+ , "SCHEMA"+ , "SCHEMAS"+ , "SCOPE"+ , "SCRATCHPAD"+ , "SCROLL"+ , "SEARCH"+ , "SECOND"+ , "SECONDS"+ , "SECOND_MICROSECOND"+ , "SECQTY"+ , "SECTION"+ , "SECURITY"+ , "SECURITYAUDIT"+ , "SEEK"+ , "SEL"+ , "SELECT"+ , "SEMANTICKEYPHRASETABLE"+ , "SEMANTICSIMILARITYDETAILSTABLE"+ , "SEMANTICSIMILARITYTABLE"+ , "SENSITIVE"+ , "SEPARATOR"+ , "SEQUENCE"+ , "SESSION"+ , "SESSION_USER"+ , "SET"+ , "SETRESRATE"+ , "SETS"+ , "SETSESSRATE"+ , "SETUSER"+ , "SHARE"+ , "SHOW"+ , "SHUTDOWN"+ , "SIGNAL"+ , "SIMILAR"+ , "SIMPLE"+ , "SIN"+ , "SINH"+ , "SIZE"+ , "SKEW"+ , "SKIP"+ , "SMALLINT"+ , "SOME"+ , "SOUNDEX"+ , "SOURCE"+ , "SPACE"+ , "SPATIAL"+ , "SPECIFIC"+ , "SPECIFICTYPE"+ , "SPOOL"+ , "SQL"+ , "SQLEXCEPTION"+ , "SQLSTATE"+ , "SQLTEXT"+ , "SQLWARNING"+ , "SQL_BIG_RESULT"+ , "SQL_CALC_FOUND_ROWS"+ , "SQL_SMALL_RESULT"+ , "SQRT"+ , "SS"+ , "SSL"+ , "STANDARD"+ , "START"+ , "STARTING"+ , "STARTUP"+ , "STATE"+ , "STATEMENT"+ , "STATIC"+ , "STATISTICS"+ , "STAY"+ , "STDDEV_POP"+ , "STDDEV_SAMP"+ , "STEPINFO"+ , "STOGROUP"+ , "STORED"+ , "STORES"+ , "STRAIGHT_JOIN"+ , "STRING_CS"+ , "STRUCTURE"+ , "STYLE"+ , "SUBMULTISET"+ , "SUBSCRIBER"+ , "SUBSET"+ , "SUBSTR"+ , "SUBSTRING"+ , "SUBSTRING_REGEX"+ , "SUCCEEDS"+ , "SUCCESSFUL"+ , "SUM"+ , "SUMMARY"+ , "SUSPEND"+ , "SYMMETRIC"+ , "SYNONYM"+ , "SYSDATE"+ , "SYSTEM"+ , "SYSTEM_TIME"+ , "SYSTEM_USER"+ , "SYSTIMESTAMP"+ , "TABLE"+ , "TABLESAMPLE"+ , "TABLESPACE"+ , "TAN"+ , "TANH"+ , "TBL_CS"+ , "TEMPORARY"+ , "TERMINATE"+ , "TERMINATED"+ , "TEXTSIZE"+ , "THAN"+ , "THEN"+ , "THRESHOLD"+ , "TIME"+ , "TIMESTAMP"+ , "TIMEZONE_HOUR"+ , "TIMEZONE_MINUTE"+ , "TINYBLOB"+ , "TINYINT"+ , "TINYTEXT"+ , "TITLE"+ , "TO"+ , "TOP"+ , "TRACE"+ , "TRAILING"+ , "TRAN"+ , "TRANSACTION"+ , "TRANSLATE"+ , "TRANSLATE_CHK"+ , "TRANSLATE_REGEX"+ , "TRANSLATION"+ , "TREAT"+ , "TRIGGER"+ , "TRIM"+ , "TRIM_ARRAY"+ , "TRUE"+ , "TRUNCATE"+ , "TRY_CONVERT"+ , "TSEQUAL"+ , "TYPE"+ , "UC"+ , "UESCAPE"+ , "UID"+ , "UNDEFINED"+ , "UNDER"+ , "UNDO"+ , "UNION"+ , "UNIQUE"+ , "UNKNOWN"+ , "UNLOCK"+ , "UNNEST"+ , "UNPIVOT"+ , "UNSIGNED"+ , "UNTIL"+ , "UPD"+ , "UPDATE"+ , "UPDATETEXT"+ , "UPPER"+ , "UPPERCASE"+ , "USAGE"+ , "USE"+ , "USER"+ , "USING"+ , "UTC_DATE"+ , "UTC_TIME"+ , "UTC_TIMESTAMP"+ , "VALIDATE"+ , "VALIDPROC"+ , "VALUE"+ , "VALUES"+ , "VALUE_OF"+ , "VARBINARY"+ , "VARBYTE"+ , "VARCHAR"+ , "VARCHAR2"+ , "VARCHARACTER"+ , "VARGRAPHIC"+ , "VARIABLE"+ , "VARIADIC"+ , "VARIANT"+ , "VARYING"+ , "VAR_POP"+ , "VAR_SAMP"+ , "VCAT"+ , "VERBOSE"+ , "VERSIONING"+ , "VIEW"+ , "VIRTUAL"+ , "VOLATILE"+ , "VOLUMES"+ , "WAIT"+ , "WAITFOR"+ , "WHEN"+ , "WHENEVER"+ , "WHERE"+ , "WHILE"+ , "WIDTH_BUCKET"+ , "WINDOW"+ , "WITH"+ , "WITHIN"+ , "WITHIN_GROUP"+ , "WITHOUT"+ , "WLM"+ , "WORK"+ , "WRITE"+ , "WRITETEXT"+ , "XMLCAST"+ , "XMLEXISTS"+ , "XMLNAMESPACES"+ , "XOR"+ , "YEAR"+ , "YEARS"+ , "YEAR_MONTH"+ , "ZEROFILL"+ , "ZEROIFNULL"+ , "ZONE"+ ]+ }+ )+ ,+ ( Tcl+ , FileTypeInfo+ { ftSelector = [ext ".tcl", ext ".tk"]+ , ftKind = KindScript+ , ftComment = ["#" ~~ "\n"]+ , ftChar = []+ , ftString = ["\"" ~~ "\""]+ , ftRawString = []+ , ftKeywords = HM.empty+ , ftIdentCharSet = Just (Unicode_, UnicodeNum_)+ }+ )+ ,+ ( Text+ , FileTypeInfo+ { ftSelector =+ [ ext ".txt"+ , ext ".md"+ , ext ".markdown"+ , ext ".mdown"+ , ext ".mkdn"+ , ext ".mkd"+ , ext ".mdwn"+ , ext ".mdtxt"+ , ext ".mdtext"+ , ext ".text"+ , name "README"+ , name "INSTALL"+ , name "VERSION"+ , name "LICENSE"+ , name "AUTHORS"+ , name "CHANGELOG"+ , name "go.sum"+ ]+ , ftKind = KindText+ , ftComment = []+ , ftChar = []+ , ftString = []+ , ftRawString = []+ , ftIdentCharSet = Nothing+ , ftKeywords = HM.empty+ }+ )+ ,+ ( Unison+ , FileTypeInfo+ { ftSelector = [ext ".u"]+ , ftKind = KindLanguage+ , ftComment = ["{-" ~~ "-}", "--" ~~ "\n"]+ , ftChar = ["'" ~~ "'"]+ , ftString = ["\"" ~~ "\"", "\"\"\"" ~~ "\"\"\""]+ , ftRawString = []+ , ftIdentCharSet = Just (Alpha_, AlphaNum_')+ , ftKeywords =+ types+ [ "Text"+ , "Number"+ , "Boolean"+ , "List"+ , "Optional"+ , "Maybe"+ , "Either"+ , "Tuple"+ , "Function"+ ]+ <> reserved+ [ "type"+ , "ability"+ , "structural"+ , "unique"+ , "if"+ , "then"+ , "else"+ , "forall"+ , "handle"+ , "with"+ , "where"+ , "use"+ , "true"+ , "false"+ , "alias"+ , "typeLink"+ , "termLink"+ , "let"+ , "namespace"+ , "match"+ , "cases"+ ]+ }+ )+ ,+ ( VHDL+ , FileTypeInfo+ { ftSelector = [ext ".vhd", ext ".vhdl"]+ , ftKind = KindLanguage+ , ftComment = ["--" ~~ "\n"]+ , ftChar = ["'" ~~ "'"]+ , ftString = ["\"" ~~ "\""]+ , ftRawString = []+ , ftIdentCharSet = Just (Alpha, AlphaNum_)+ , ftKeywords =+ types+ [ "std_logic"+ , "std_logic_vector"+ , "integer"+ , "real"+ , "boolean"+ , "character"+ , "string"+ , "time"+ , "bit"+ , "bit_vector"+ , "signed"+ , "unsigned"+ ]+ <> reserved+ [ "abs"+ , "access"+ , "after"+ , "alias"+ , "all"+ , "and"+ , "architecture"+ , "array"+ , "assert"+ , "attribute"+ , "begin"+ , "block"+ , "body"+ , "buffer"+ , "bus"+ , "case"+ , "component"+ , "configuration"+ , "constant"+ , "disconnect"+ , "downto"+ , "else"+ , "elsif"+ , "end"+ , "entity"+ , "exit"+ , "file"+ , "for"+ , "function"+ , "generate"+ , "generic"+ , "group"+ , "guarded"+ , "if"+ , "impure"+ , "in"+ , "inertial"+ , "inout"+ , "is"+ , "label"+ , "library"+ , "linkage"+ , "literal"+ , "loop"+ , "map"+ , "mod"+ , "nand"+ , "new"+ , "next"+ , "nor"+ , "not"+ , "null"+ , "of"+ , "on"+ , "open"+ , "or"+ , "others"+ , "out"+ , "package"+ , "port"+ , "postponed"+ , "procedure"+ , "process"+ , "pure"+ , "range"+ , "record"+ , "register"+ , "reject"+ , "return"+ , "rol"+ , "ror"+ , "select"+ , "severity"+ , "signal"+ , "shared"+ , "sla"+ , "sli"+ , "sra"+ , "srl"+ , "subtype"+ , "then"+ , "to"+ , "transport"+ , "type"+ , "unaffected"+ , "units"+ , "until"+ , "use"+ , "variable"+ , "wait"+ , "when"+ , "while"+ , "with"+ , "xnor"+ , "xor"+ ]+ }+ )+ ,+ ( Verilog+ , FileTypeInfo+ { ftSelector = [ext ".v", ext ".vh", ext ".sv"]+ , ftKind = KindLanguage+ , ftComment = ["/*" ~~ "*/", "//" ~~ "\n"]+ , ftChar = []+ , ftString = ["\"" ~~ "\""]+ , ftRawString = []+ , ftIdentCharSet = Just (Alpha_, AlphaNum_)+ , ftKeywords =+ types+ [ "wire"+ , "reg"+ , "integer"+ , "real"+ , "time"+ , "parameter"+ , "event"+ , "genvar"+ , "string"+ ]+ <> reserved+ [ "always"+ , "end"+ , "ifnone"+ , "or"+ , "rpmos"+ , "tranif1"+ , "and"+ , "endcase"+ , "initial"+ , "output"+ , "rtran"+ , "tri"+ , "assign"+ , "endmodule"+ , "inout"+ , "rtranif0"+ , "tri0"+ , "begin"+ , "endfunction"+ , "input"+ , "pmos"+ , "rtranif1"+ , "tri1"+ , "buf"+ , "endprimitive"+ , "posedge"+ , "scalared"+ , "triand"+ , "bufif0"+ , "endspecify"+ , "join"+ , "primitive"+ , "small"+ , "trior"+ , "bufif1"+ , "endtable"+ , "large"+ , "pull0"+ , "specify"+ , "trireg"+ , "case"+ , "endtask"+ , "macromodule"+ , "pull1"+ , "specparam"+ , "vectored"+ , "casex"+ , "medium"+ , "pullup"+ , "strong0"+ , "wait"+ , "casez"+ , "for"+ , "module"+ , "pulldown"+ , "strong1"+ , "wand"+ , "cmos"+ , "force"+ , "nand"+ , "rcmos"+ , "supply0"+ , "weak0"+ , "deassign"+ , "forever"+ , "negedge"+ , "supply1"+ , "weak1"+ , "default"+ , "for"+ , "nmos"+ , "realtime"+ , "table"+ , "while"+ , "defparam"+ , "function"+ , "nor"+ , "task"+ , "disable"+ , "highz0"+ , "not"+ , "release"+ , "wor"+ , "edge"+ , "highz1"+ , "notif0"+ , "repeat"+ , "tran"+ , "xnor"+ , "else"+ , "if"+ , "notif1"+ , "rnmos"+ , "tranif0"+ , "xor"+ ]+ }+ )+ ,+ ( Yaml+ , FileTypeInfo+ { ftSelector = [ext ".yaml", ext ".yml"]+ , ftKind = KindConfig+ , ftComment = ["#" ~~ "\n"]+ , ftChar = []+ , ftString = ["\"" ~~ "\"", "'" ~~ "'"]+ , ftRawString = []+ , ftIdentCharSet = Nothing+ , ftKeywords = HM.empty+ }+ )+ ,+ ( Toml+ , FileTypeInfo+ { ftSelector = [ext ".toml", name "Cargo.lock"]+ , ftKind = KindConfig+ , ftComment = ["#" ~~ "\n"]+ , ftChar = []+ , ftString = ["\"" ~~ "\"", "\"\"\"" ~~ "\"\"\""]+ , ftRawString = ["'" ~~ "'", "'''" ~~ "'''"]+ , ftIdentCharSet = Nothing+ , ftKeywords = HM.empty+ }+ )+ ,+ ( Ini+ , FileTypeInfo+ { ftSelector = [ext ".ini"]+ , ftKind = KindConfig+ , ftComment = [";" ~~ "\n", "#" ~~ "\n"]+ , ftChar = []+ , ftString = ["\"" ~~ "\""]+ , ftRawString = []+ , ftIdentCharSet = Nothing+ , ftKeywords = HM.empty+ }+ )+ ,+ ( Zig+ , FileTypeInfo+ { ftSelector = [ext ".zig"]+ , ftKind = KindLanguage+ , ftComment = ["//" ~~ "\n"]+ , ftChar = ["'" ~~ "'"]+ , ftString = ["\"" ~~ "\""]+ , ftRawString = ["\\" ~~ "\n"]+ , ftIdentCharSet = Just (Alpha_, AlphaNum_)+ , ftKeywords =+ types+ [ "i8"+ , "u8"+ , "i16"+ , "u16"+ , "i32"+ , "u32"+ , "i64"+ , "u64"+ , "i128"+ , "u128"+ , "isize"+ , "usize"+ , "c_short"+ , "c_ushort"+ , "c_int"+ , "c_uint"+ , "c_long"+ , "c_ulong"+ , "c_longlong"+ , "c_ulonglong"+ , "c_longdouble"+ , "f16"+ , "f32"+ , "f64"+ , "f80"+ , "f128"+ , "bool"+ , "anyopaque"+ , "void"+ , "noreturn"+ , "type"+ , "anyerror"+ , "comptime_int"+ , "comptime_float"+ ]+ <> reserved+ [ "addrspace"+ , "align"+ , "allowzero"+ , "and"+ , "anyframe"+ , "anytype"+ , "asm"+ , "async"+ , "await"+ , "break"+ , "catch"+ , "comptime"+ , "const"+ , "continue"+ , "defer"+ , "else"+ , "enum"+ , "errdefer"+ , "error"+ , "export"+ , "extern"+ , "fn"+ , "for"+ , "if"+ , "inline"+ , "linksection"+ , "noalias"+ , "noinline"+ , "nosuspend"+ , "or"+ , "orelse"+ , "packed"+ , "pub"+ , "resume"+ , "return"+ , "struct"+ , "suspend"+ , "switch"+ , "test"+ , "threadlocal"+ , "try"+ , "union"+ , "unreachable"+ , "usingnamespace"+ , "var"+ , "volatile"+ , "while"+ ]+ }+ )+ ,+ ( Zsh+ , FileTypeInfo+ { ftSelector = [ext ".zsh"]+ , ftKind = KindScript+ , ftComment = ["#" ~~ "\n"]+ , ftChar = []+ , ftString = ["'" ~~ "'", "\"" ~~ "\""]+ , ftRawString = []+ , ftIdentCharSet = Just (Alpha_, AlphaNum_)+ , ftKeywords =+ reserved+ [ "do"+ , "done"+ , "esac"+ , "then"+ , "elif"+ , "else"+ , "fi"+ , "for"+ , "case"+ , "if"+ , "while"+ , "function"+ , "repeat"+ , "time"+ , "until"+ , "select"+ , "coproc"+ , "nocorrect"+ , "foreach"+ , "end"+ ]+ }+ )+ ]+ )++mkContextFilterFn :: Maybe FileType -> ContextFilter -> Bool -> (T.Text -> T.Text)+mkContextFilterFn _ (isContextFilterAll -> True) False = id+mkContextFilterFn Nothing _ _ = id+mkContextFilterFn (Just ftype) filt useMarkers+ | Just fun <- parFunc = fun filt+ | otherwise = id+ where+ parFunc = mkFilterFunction useMarkers =<< Map.lookup ftype (unMapInfo fileTypeInfoMap)++fileTypeLookup :: Options -> OS.OsPath -> Maybe (FileType, FileKind)+fileTypeLookup opts f = forcedType opts <|> lookupFileType f (code_only opts) (hdr_only opts)+ where+ lookupFileType :: OS.OsPath -> Bool -> Bool -> Maybe (FileType, FileKind)+ lookupFileType file False False = Map.lookup (Name $ OS.takeFileName file) m <|> Map.lookup (Ext e) m <|> Map.lookup (Hdr e) m+ lookupFileType file True False = Map.lookup (Name $ OS.takeFileName file) m <|> Map.lookup (Ext e) m+ lookupFileType file False True = Map.lookup (Name $ OS.takeFileName file) m <|> Map.lookup (Hdr e) m+ lookupFileType _ True True = errorWithoutStackTrace "CGrep: code-only and hdr-only are mutually exclusive!"+ e = OS.takeExtension f+ m = unMap fileTypeMap+{-# INLINE fileTypeLookup #-}++fileTypeInfoLookup :: Options -> OS.OsPath -> Maybe (FileType, FileTypeInfo)+fileTypeInfoLookup opts f = fileTypeLookup opts f >>= \(typ, _kid) -> (typ,) <$> Map.lookup typ (unMapInfo fileTypeInfoMap)+{-# INLINE fileTypeInfoLookup #-}++fileTypeMap :: FileTypeMap+fileTypeMap = FileTypeMap $ Map.fromList $ concatMap (\(typ, FileTypeInfo{..}) -> map (,(typ, ftKind)) ftSelector) $ Map.toList (unMapInfo fileTypeInfoMap)+{-# NOINLINE fileTypeMap #-}++dumpFileTypeInfoMap :: FileTypeInfoMap -> IO ()+dumpFileTypeInfoMap m = forM_ ((Map.toList . unMapInfo) m) $ \(l, ex) ->+ putStrLn $ show l <> [' ' | _ <- [length (show l) .. 12]] <> "-> " <> show (ftSelector ex)++-- dumpFileTypeMap :: FileTypeMap -> IO ()+-- dumpFileTypeMap m = forM_ (Map.toList (unMap m)) $ \(e, l) ->+-- putStrLn $ show e <> [' ' | _ <- [length (show e) .. 12]] <> "-> " <> show l++forcedType :: Options -> Maybe (FileType, FileKind)+forcedType Options{..}+ | Just typ <- force_type = Map.lookup (ext typ) m <|> Map.lookup (name typ) m+ | otherwise = Nothing+ where+ m = unMap fileTypeMap++mkFilterFunction :: Bool -> FileTypeInfo -> Maybe FilterFunction+mkFilterFunction useMarkers FileTypeInfo{..} =+ Just $+ runContextFilter (mkParConfig ftComment ftString ftRawString ftChar useMarkers)+{-# INLINE mkFilterFunction #-}
+ src/CGrep/Line.hs view
@@ -0,0 +1,135 @@+--+-- Copyright (c) 2013-2025 Nicola Bonelli <nicola@larthia.com>+--+-- This program is free software; you can redistribute it and/or modify+-- it under the terms of the GNU General Public License as published by+-- the Free Software Foundation; either version 2 of the License, or+-- (at your option) any later version.+--+-- This program is distributed in the hope that it will be useful,+-- but WITHOUT ANY WARRANTY; without even the implied warranty of+-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the+-- GNU General Public License for more details.+--+-- You should have received a copy of the GNU General Public License+-- along with this program; if not, write to the Free Software+-- Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.+--++module CGrep.Line (+ LineIndex,+ buildIndex,+ totalLines,+ lookupLineAndPosition,+ getLineByOffset',+ ------------------------+ getLineOffsets,+ getLineByOffset,+ lowerBound,+) where++import CGrep.Text (textSlice)+import Data.Bits+import qualified Data.Text as T+import qualified Data.Text.Internal.Search as T+import qualified Data.Text.Unsafe as TU+import qualified Data.Vector.Unboxed as UV++-- A LineIndex holds the original text and a vector of line start offsets.+data LineIndex+ = LineIndex+ T.Text+ (UV.Vector (Int))+ deriving stock (Show)++totalLines :: LineIndex -> Int+totalLines (LineIndex _ vec) = UV.length vec+{-# INLINE totalLines #-}++-- | Build a LineIndex from the given Text.+buildIndex :: T.Text -> LineIndex+buildIndex txt = LineIndex txt $ UV.fromList $ 0 : (map (+ 1) $ T.indices (T.singleton '\n') txt)+{-# INLINE buildIndex #-}++-- | Given a LineIndex and a 0-based offset, return the (1-based) line number and column number.+lookupLineAndPosition :: LineIndex -> Int -> (# Int, Int #)+lookupLineAndPosition (LineIndex _ vec) !queryOffset+ | UV.null vec = (# 1, queryOffset + 1 #) -- Edge case: indice vuoto+ | otherwise =+ -- 1. find the 0-based index of the line.+ let !lineIndex = findLineIndex vec queryOffset+ in if lineIndex < 0+ then (# 1, queryOffset + 1 #)+ else+ let !lineStartOffset = vec `UV.unsafeIndex` lineIndex+ !lineNum = lineIndex + 1 -- 1-based+ !colNum = (queryOffset - lineStartOffset) + 1 -- 1-based+ in (# lineNum, colNum #)++-- | Given a LineIndex and a 0-based offset, return the line Text.+getLineByOffset' :: LineIndex -> Int -> T.Text+getLineByOffset' (LineIndex originalText vec) !offset+ | UV.null vec = T.empty+ | otherwise =+ let !lineIndex = findLineIndex vec offset+ in if lineIndex < 0+ then T.empty+ else+ let !offsetStart = vec `UV.unsafeIndex` lineIndex+ !numLines = UV.length vec+ in if lineIndex == numLines - 1+ then+ let !len = T.length originalText - offsetStart+ in textSlice originalText offsetStart len+ else+ let !offsetNext = vec `UV.unsafeIndex` (lineIndex + 1)+ !len = offsetNext - offsetStart - 1+ in textSlice originalText offsetStart len++-- Binary search to find the greatest index i such that vec[i] <= v+findLineIndex :: UV.Vector Int -> Int -> Int+findLineIndex vec v = findLineIndexGo vec v 0 (UV.length vec - 1)+{-# INLINE findLineIndex #-}++findLineIndexGo :: UV.Vector Int -> Int -> Int -> Int -> Int+findLineIndexGo vec v !left !right+ | left > right = right+ | otherwise = case v `compare` midValue of+ LT -> findLineIndexGo vec v left (mid - 1)+ EQ -> mid+ GT -> findLineIndexGo vec v (mid + 1) right+ where+ !mid = (left + right) `div` 2+ !midValue = vec `UV.unsafeIndex` mid++--------------------------------------------------------------------------------------------++--- >>> getLineOffsets "Hello ©\nWorld\nThis is a test\n"+-- [0,9,15,30]+getLineOffsets :: T.Text -> UV.Vector Int+getLineOffsets txt = UV.fromList $ 0 : (map (+ 1) $ T.indices (T.singleton '\n') txt)+{-# INLINE getLineOffsets #-}++getLineByOffset :: Int -> T.Text -> UV.Vector Int -> (# T.Text, Int #)+getLineByOffset off text vec = (# line, lb #)+ where+ !lb = lowerBound vec off+ !dropped = TU.dropWord8 lb text+ !line = T.takeWhile (/= '\n') dropped+{-# INLINE getLineByOffset #-}++lowerBound :: UV.Vector Int -> Int -> Int+lowerBound vec v+ | UV.null vec = 0 -- caso edge+ | otherwise = lowerBoundGo vec v 0 (UV.length vec - 1)+{-# INLINE lowerBound #-}++lowerBoundGo :: UV.Vector Int -> Int -> Int -> Int -> Int+lowerBoundGo vec v !left !right+ | left > right = vec `UV.unsafeIndex` right+ | midValue > v = lowerBoundGo vec v left (mid - 1)+ | midValue == v = midValue+ | otherwise = lowerBoundGo vec v (mid + 1) right+ where+ !mid = left + ((right - left) `shiftR` 1)+ !midValue = vec `UV.unsafeIndex` mid
+ src/CGrep/Match.hs view
@@ -0,0 +1,297 @@+--+-- Copyright (c) 2013-2025 Nicola Bonelli <nicola@larthia.com>+--+-- This program is free software; you can redistribute it and/or modify+-- it under the terms of the GNU General Public License as published by+-- the Free Software Foundation; either version 2 of the License, or+-- (at your option) any later version.+--+-- This program is distributed in the hope that it will be useful,+-- but WITHOUT ANY WARRANTY; without even the implied warranty of+-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the+-- GNU General Public License for more details.+--+-- You should have received a copy of the GNU General Public License+-- along with this program; if not, write to the Free Software+-- Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.+--+{-# OPTIONS_GHC -Wno-incomplete-uni-patterns #-}++module CGrep.Match (+ Match (..),+ mkMatches,+ putMatches,+ prettyFileName,+ prettyBold,+) where++import CGrep.Line (LineIndex, getLineByOffset', lookupLineAndPosition, totalLines)+import qualified Data.Text as T+import qualified Data.Text.Lazy as TL+import qualified Data.Text.Lazy.Builder as TLB+import qualified Data.Text.Lazy.Builder.Int as TLB+import qualified Data.Text.Unsafe as TU+import System.OsPath++import CGrep.Parser.Chunk (Chunk (..), MatchLine (..), cOffset)+import CGrep.Text (textOffsetWord8)+import Config+import Control.Monad.Reader+import Data.Function (on)+import Data.List (groupBy, nub, sortBy, sortOn)+import Data.List.Extra (intersperse)+import Options (Options (..))+import qualified OsPath as OS+import Reader (Env (..), ReaderIO)+import System.Console.ANSI (SGR (..), setSGRCode)+import System.Console.ANSI.Codes (ConsoleIntensity (..))+import Util (unsafeHead)++data Match = Match+ { mFilePath :: OsPath+ , mLineNumb :: {-# UNPACK #-} !Int+ , mLine :: {-# UNPACK #-} !T.Text+ , mChunks :: ![Chunk]+ }+ deriving stock (Show, Eq)++mTokens :: Match -> [T.Text]+mTokens (Match _ _ _ cs) = cToken <$> cs+{-# INLINE mTokens #-}++mkMatches :: LineIndex -> OsPath -> T.Text -> [Chunk] -> ReaderIO [Match]+mkMatches lindex f txt chunks = do+ invert <- invert_match <$> reader opt+ return $+ if invert+ then+ map+ ( \(MatchLine n xs) ->+ Match f n txt xs+ )+ . invertLines (totalLines lindex)+ $ mkMatchLines lindex chunks+ else+ map+ ( \(MatchLine n xs) ->+ let line = getLineByOffset' lindex ((cOffset . unsafeHead) xs)+ in Match f n line xs+ )+ $ mkMatchLines lindex chunks++mkMatchLines :: LineIndex -> [Chunk] -> [MatchLine]+mkMatchLines _ [] = []+mkMatchLines lindex chunks =+ map mergeGroup $+ groupBy ((==) `on` mlOffset) . sortBy (compare `on` mlOffset) $+ ( \chunk ->+ let (# r, _ #) = lookupLineAndPosition lindex (cOffset chunk)+ in MatchLine r [Chunk (cTyp chunk) (cToken chunk)]+ )+ <$> chunks+ where+ mergeGroup :: [MatchLine] -> MatchLine+ mergeGroup [] = error "mergeGroup: empty list"+ mergeGroup ls@(x : _) = MatchLine (mlOffset x) (foldl' (\l m -> l <> mlChunks m) [] ls)++invertLines :: Int -> [MatchLine] -> [MatchLine]+invertLines n xs = filter (\(MatchLine i _) -> i `notElem` idx) $ take n [MatchLine i [] | i <- [1 ..]]+ where+ idx = mlOffset <$> xs+{-# INLINE invertLines #-}++putMatches :: [Match] -> ReaderIO (Maybe TLB.Builder)+putMatches [] = pure Nothing+putMatches out = do+ Env{..} <- ask+ if+ | null_output opt -> pure Nothing+ | json opt -> Just <$> jsonMatch out+ | filename_only opt -> Just <$> filenameMatch out+ | otherwise -> Just <$> defPutMatches out++defPutMatches :: [Match] -> ReaderIO TLB.Builder+defPutMatches xs = do+ Env{..} <- ask+ if+ | Options{no_filename = False, no_numbers = False, count = False} <- opt ->+ pure $ mconcat . intersperse (TLB.singleton '\n') $ map (\out -> buildFileName conf opt out <> TLB.singleton ':' <> buildLineCol opt out <> TLB.singleton ':' <> buildTokens opt out <> buildLine conf opt out) xs+ | Options{no_filename = False, no_numbers = True, count = False} <- opt ->+ pure $ mconcat . intersperse (TLB.singleton '\n') $ map (\out -> buildFileName conf opt out <> TLB.singleton ':' <> buildTokens opt out <> buildLine conf opt out) xs+ | Options{no_filename = True, no_numbers = False, count = False} <- opt ->+ pure $ mconcat . intersperse (TLB.singleton '\n') $ map (\out -> buildTokens opt out <> buildLine conf opt out) xs+ | Options{no_filename = True, no_numbers = True, count = False} <- opt ->+ pure $ mconcat . intersperse (TLB.singleton '\n') $ map (\out -> buildTokens opt out <> buildLine conf opt out) xs+ | Options{no_filename = False, count = True} <- opt ->+ do+ let gs = groupBy (\(Match f1 _ _ _) (Match f2 _ _ _) -> f1 == f2) xs++ pure $+ mconcat . intersperse (TLB.singleton '\n') $+ ( \ys -> case ys of+ (y : _) -> buildFileName conf opt y <> TLB.singleton ':' <> TLB.decimal (length ys)+ [] -> mempty+ )+ <$> gs+ | Options{count = True} <- opt ->+ do+ let gs = groupBy (\(Match f1 _ _ _) (Match f2 _ _ _) -> f1 == f2) xs+ pure $ mconcat . intersperse (TLB.singleton '\n') $ (\ys -> TLB.decimal (length ys)) <$> gs++filenameMatch :: [Match] -> ReaderIO TLB.Builder+filenameMatch outs = do+ return $ mconcat . intersperse (TLB.singleton '\n') $ TLB.fromText <$> nub ((\(Match fname _ _ _) -> (OS.toText fname)) <$> outs)+{-# INLINE filenameMatch #-}++jsonMatch :: [Match] -> ReaderIO TLB.Builder+jsonMatch [] = pure mempty+jsonMatch outs@(Match fname _ _ _ : _) = do+ strname <- liftIO $ decodeUtf fname+ pure $+ mconcat . intersperse (TLB.singleton '\n') $+ [TLB.fromString "{ \"file\":\"" <> TLB.fromString strname <> TLB.fromString "\", \"matches\":["]+ <> [mconcat $ intersperse (TLB.singleton ',') (foldl mkMatch [] outs)]+ <> [TLB.fromString "]}"]+ where+ mkJToken chunk = TLB.fromString "{ \"col\":" <> TLB.decimal (cOffset chunk) <> TLB.fromString ", \"token\":\"" <> TLB.fromText (cToken chunk) <> TLB.fromString "\" }"+ mkMatch xs (Match _ n _ ts) =+ xs+ <> [ TLB.fromString "{ \"row\": "+ <> TLB.decimal n+ <> TLB.fromString ", \"tokens\":["+ <> mconcat (intersperse (TLB.fromString ",") (map mkJToken ts))+ <> TLB.fromString "] }"+ ]++--------------------------------------------------------------------++buildFileName :: Config -> Options -> Match -> TLB.Builder+buildFileName conf opt out =+ let str = OS.toText (mFilePath out)+ in buildFileName' conf opt $ str+ where+ buildFileName' :: Config -> Options -> T.Text -> TLB.Builder+ buildFileName' conf' opts = buildColoredText opts $ T.pack (setSGRCode (configColorFile conf'))+{-# INLINE buildFileName #-}++buildColoredText :: Options -> ColorCode -> T.Text -> TLB.Builder+buildColoredText opt colorCode txt+ | color opt && not (no_color opt) = TLB.fromText colorCode <> TLB.fromText txt <> resetBuilder+ | otherwise = TLB.fromText txt+{-# INLINE buildColoredText #-}++buildLineCol :: Options -> Match -> TLB.Builder+buildLineCol Options{no_numbers = True} _ = mempty+buildLineCol Options{no_numbers = False, no_column = True} (Match _ n _ _) = TLB.decimal n+buildLineCol Options{no_numbers = False, no_column = False} (Match _ n _ []) = TLB.decimal n+buildLineCol Options{no_numbers = False, no_column = False} (Match _ n l (t : _)) = TLB.decimal n <> TLB.singleton ':' <> TLB.decimal (bytesToCharOffset l (cOffset t) + 1)+{-# INLINE buildLineCol #-}++bytesToCharOffset :: T.Text -> Int -> Int+bytesToCharOffset line byteOffset = go 0 0+ where+ go !charIdx !byteIdx+ | byteIdx >= byteOffset = charIdx+ | byteIdx >= TU.lengthWord8 line = charIdx+ | otherwise =+ let TU.Iter _ delta = TU.iter line byteIdx+ in go (charIdx + 1) (byteIdx + delta)+{-# INLINE bytesToCharOffset #-}++buildTokens :: Options -> Match -> TLB.Builder+buildTokens Options{show_match = st} out+ | st = boldBuilder <> mconcat (TLB.fromText <$> mTokens out) <> resetBuilder <> TLB.singleton ':'+ | otherwise = mempty+{-# INLINE buildTokens #-}++buildLine :: Config -> Options -> Match -> TLB.Builder+buildLine conf Options{color = c, no_color = no_c} out+ | c && not no_c = buildColoredLine conf (sortBy (flip compare `on` (T.length . cToken)) (mChunks out)) (mLine out)+ | otherwise = TLB.fromText $ mLine out+{-# INLINE buildLine #-}++buildColoredLine :: Config -> [Chunk] -> T.Text -> TLB.Builder+buildColoredLine conf chunks line =+ let+ lineOffset = textOffsetWord8 line+ lineByteLen = TU.lengthWord8 line+ lineEndOffset = lineOffset + lineByteLen++ events :: [(Int, Int)]+ events = sortOn fst $ concatMap chunkToEvents chunks++ chunkToEvents :: Chunk -> [(Int, Int)]+ chunkToEvents chunk =+ let chunkStartAbs = cOffset chunk+ chunkLen = TU.lengthWord8 (cToken chunk)+ chunkEndAbs = chunkStartAbs + chunkLen+ overlaps = chunkEndAbs > lineOffset && chunkStartAbs < lineEndOffset+ in if not overlaps || chunkLen == 0+ then []+ else+ let relStart = max 0 (chunkStartAbs - lineOffset)+ relEnd = min lineByteLen (chunkEndAbs - lineOffset)+ in if relStart < relEnd+ then [(relStart, 1), (relEnd, -1)]+ else []++ colorMatch = TLB.fromString $ setSGRCode (configColorMatch conf)++ processEvent :: (Int, Int, TLB.Builder) -> (Int, Int) -> (Int, Int, TLB.Builder)+ processEvent (lastIdx, level, accBuilder) (eventIdx, delta) =+ let+ accBuilder'+ | eventIdx > lastIdx =+ let+ chunkLen = eventIdx - lastIdx+ textChunk = TU.takeWord8 chunkLen (TU.dropWord8 lastIdx line)++ coloredChunk+ | level > 0 = colorMatch <> TLB.fromText textChunk <> resetBuilder+ | otherwise = TLB.fromText textChunk+ in+ accBuilder <> coloredChunk+ | otherwise = accBuilder++ newLevel = level + delta+ colorChange+ | newLevel > 0 && level == 0 = colorMatch+ | newLevel == 0 && level > 0 = resetBuilder+ | otherwise = mempty+ in+ (eventIdx, newLevel, accBuilder' <> colorChange)++ (finalIdx, finalLevel, mainBuilder) = foldl' processEvent (0, 0, mempty) events++ remainingText = TU.dropWord8 finalIdx line+ finalBuilder+ | finalLevel > 0 = colorMatch <> TLB.fromText remainingText <> resetBuilder+ | otherwise = TLB.fromText remainingText+ in+ mainBuilder <> finalBuilder++--------------------------------------------------------------------++prettyFileName :: Config -> Options -> OsPath -> TL.Text+prettyFileName conf opt path = TLB.toLazyText $ buildColoredText opt (T.pack $ setSGRCode (configColorFile conf)) (OS.toText path)+{-# INLINE prettyFileName #-}++prettyBold :: Options -> T.Text -> TL.Text+prettyBold opt txt = TLB.toLazyText $ buildColoredText opt bold txt+{-# INLINE prettyBold #-}++--------------------------------------------------------------------++type ColorCode = T.Text++boldBuilder, resetBuilder :: TLB.Builder+boldBuilder = TLB.fromText bold+resetBuilder = TLB.fromText reset+{-# NOINLINE boldBuilder #-}+{-# NOINLINE resetBuilder #-}++bold, reset :: T.Text+bold = T.pack $ setSGRCode [SetConsoleIntensity BoldIntensity]+reset = T.pack $ setSGRCode []+{-# NOINLINE bold #-}+{-# NOINLINE reset #-}
− src/CGrep/Output.hs
@@ -1,273 +0,0 @@------ Copyright (c) 2013-2023 Nicola Bonelli <nicola@larthia.com>------ This program is free software; you can redistribute it and/or modify--- it under the terms of the GNU General Public License as published by--- the Free Software Foundation; either version 2 of the License, or--- (at your option) any later version.------ This program is distributed in the hope that it will be useful,--- but WITHOUT ANY WARRANTY; without even the implied warranty of--- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the--- GNU General Public License for more details.------ You should have received a copy of the GNU General Public License--- along with this program; if not, write to the Free Software--- Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.-----{-# LANGUAGE ExistentialQuantification #-}--module CGrep.Output ( Output(..)- , mkOutputElements- , putOutputElements- , runSearch- , showFileName- , showBold) where--import qualified Data.ByteString as B-import qualified Data.ByteString.Builder as B--import qualified Data.ByteString.Char8 as C-import qualified Data.ByteString.Lazy.Char8 as LC-import qualified Data.ByteString.Unsafe as BU--import qualified Data.Vector.Unboxed as UV--import Data.Vector.Unboxed ( (!) )--import System.Console.ANSI- ( setSGRCode,- ConsoleIntensity(BoldIntensity),- SGR(SetConsoleIntensity) )--import Control.Monad.Trans.Reader ( ask, reader )-import Control.Monad.IO.Class ( MonadIO(liftIO) )--import Data.List- ( foldl', sortBy, groupBy, isPrefixOf, nub, sort, genericLength, intersperse )-import Data.Function ( on )--import CGrep.Types ( Text8, Offset )-import CGrep.Parser.Chunk ( Chunk(..), MatchLine(..) )--import Config ( Config(configColorFile, configColorMatch) )-import Reader ( ReaderIO, Env(..) )-import Data.Int ( Int64 )-import Data.Word ( Word8 )-import Data.ByteString.Internal (c2w)-import qualified Data.Vector.Fusion.Util as VU (Box(..))--import System.Posix.FilePath (RawFilePath)--import qualified Data.Vector.Generic as GV-import CGrep.Parser.Line ( getLineOffsets )-import Options- ( Options(Options, invert_match, json, filename_only, no_shallow,- no_filename, count, no_color, no_numbers, no_column, show_match,- color) )--data Output = Output- { outFilePath :: RawFilePath- , outLineNumb :: {-# UNPACK #-} !Int64- , outLine :: {-# UNPACK #-} !Text8- , outChunks :: ![Chunk]- }--outTokens :: Output -> [Text8]-outTokens (Output fp ln l cs) = cToken <$> cs-{-# INLINE outTokens #-}---insertIndex :: UV.Vector Offset -> Offset -> Int-insertIndex vs x = search vs 0 (UV.length vs)- where search xs !lo !hi- | lo == hi = lo- | otherwise = let !mid = (lo + hi) `quot` 2- in if x < VU.unBox(xs `GV.basicUnsafeIndexM` mid)- then search xs lo mid- else search xs (mid+1) hi---getLineNumberAndOffset :: UV.Vector Offset -> Offset -> (# Int, Offset #)-getLineNumberAndOffset xs x =- let idx = insertIndex xs x- in (# idx, x - xs `UV.unsafeIndex` (idx-1) #)-{-# INLINE getLineNumberAndOffset #-}---mkOutputElements :: UV.Vector Int64 -> RawFilePath -> Text8 -> Text8 -> [Chunk] -> ReaderIO [Output]-mkOutputElements lineOffsets f text multi ts = do- invert <- invert_match <$> reader opt- return $ if invert then map (\(MatchLine n xs) -> Output f n (ls !! fromIntegral (n-1)) xs) . invertLines (length ls) $ mkMatchLines lineOffsets multi ts- else map (\(MatchLine n xs) -> Output f n (ls !! fromIntegral (n-1)) xs) $ mkMatchLines lineOffsets multi ts- where ls = C.lines text-{-# INLINE mkOutputElements #-}---mkMatchLines :: UV.Vector Int64 -> Text8 -> [Chunk] -> [MatchLine]-mkMatchLines lineOffsets _ [] = []-mkMatchLines lineOffsets text ts = map mergeGroup $ groupBy ((==) `on` lOffset) . sortBy (compare `on` lOffset) $- (\chunk -> let (# r, c #) = getLineNumberAndOffset lineOffsets (cOffset chunk) in MatchLine (fromIntegral r) [Chunk (cTyp chunk) (cToken chunk) c]) <$> ts- where mergeGroup :: [MatchLine] -> MatchLine- mergeGroup ls = MatchLine ((lOffset . head) ls) (foldl' (\l m -> l <> lChunks m) [] ls)---invertLines :: Int -> [MatchLine] -> [MatchLine]-invertLines n xs = filter (\(MatchLine i _) -> i `notElem` idx ) $ take n [ MatchLine i [] | i <- [1..]]- where idx = lOffset <$> xs-{-# INLINE invertLines #-}---putOutputElements :: [Output] -> ReaderIO (Maybe B.Builder)-putOutputElements [] = pure Nothing-putOutputElements out = do- Env{..} <- ask- if | json opt -> Just <$> jsonOutput out- | filename_only opt -> Just <$> filenameOutput out- | otherwise -> Just <$> defaultOutput out--runSearch :: Options- -> RawFilePath- -> Bool- -> ReaderIO [Output]- -> ReaderIO [Output]-runSearch opt filename eligible doSearch =- if eligible || no_shallow opt- then doSearch- else mkOutputElements UV.empty filename C.empty C.empty ([] :: [Chunk])---defaultOutput :: [Output] -> ReaderIO B.Builder-defaultOutput xs = do- Env{..} <- ask- if | Options{ no_filename = False, no_numbers = False , count = False } <- opt- -> pure $ mconcat . intersperse (B.char8 '\n') $ map (\out -> buildFileName conf opt out <> B.char8 ':' <> buildLineCol opt out <> B.char8 ':' <> buildTokens opt out <> buildLine conf opt out) xs-- | Options{ no_filename = False, no_numbers = True , count = False } <- opt- -> pure $ mconcat . intersperse (B.char8 '\n') $ map (\out -> buildFileName conf opt out <> B.char8 ':' <> buildTokens opt out <> buildLine conf opt out) xs-- | Options{ no_filename = True , no_numbers = False , count = False } <- opt- -> pure $ mconcat . intersperse (B.char8 '\n') $ map (\out -> buildTokens opt out <> buildLine conf opt out) xs-- | Options{ no_filename = True , no_numbers = True , count = False } <- opt- -> pure $ mconcat . intersperse (B.char8 '\n') $ map (\out -> buildTokens opt out <> buildLine conf opt out) xs-- | Options{ no_filename = False, count = True } <- opt- -> do- let gs = groupBy (\(Output f1 _ _ _) (Output f2 _ _ _) -> f1 == f2) xs- pure $ mconcat . intersperse (B.char8 '\n') $ (\ys@(y:_) -> buildFileName conf opt y <> B.char8 ':' <> B.intDec (length ys)) <$> gs- | Options{ count = True } <- opt- -> do- let gs = groupBy (\(Output f1 _ _ _) (Output f2 _ _ _) -> f1 == f2) xs- pure $ mconcat . intersperse (B.char8 '\n') $ (\ys@(y:_) -> B.intDec (length ys)) <$> gs---jsonOutput :: [Output] -> ReaderIO B.Builder-jsonOutput [] = pure mempty-jsonOutput outs = pure $ mconcat . intersperse (B.char8 '\n') $- [B.byteString "{ \"file\":\"" <> B.byteString fname <> B.byteString "\", \"matches\":["] <>- [ mconcat $ intersperse (B.char8 ',') (foldl mkMatch [] outs) ] <> [B.byteString "]}"]- where fname | (Output f _ _ _) <- head outs = f- mkJToken chunk = B.byteString "{ \"col\":" <> B.int64Dec (cOffset chunk) <> B.byteString ", \"token\":\"" <> B.byteString (cToken chunk) <> B.byteString "\" }"- mkMatch xs (Output _ n _ ts) =- xs <> [B.byteString "{ \"row\": " <> B.int64Dec n <> B.byteString ", \"tokens\":[" <>- mconcat (intersperse (B.byteString ",") (map mkJToken ts)) <> B.byteString "] }" ]---filenameOutput :: [Output] -> ReaderIO B.Builder-filenameOutput outs = return $ mconcat . intersperse (B.char8 '\n') $ B.byteString <$> nub ((\(Output fname _ _ _) -> fname) <$> outs)-{-# INLINE filenameOutput #-}---bold, reset :: C.ByteString-bold = C.pack $ setSGRCode [SetConsoleIntensity BoldIntensity]-reset = C.pack $ setSGRCode []-{-# NOINLINE bold #-}-{-# NOINLINE reset #-}--boldBuilder, resetBuilder :: B.Builder-boldBuilder = B.byteString bold-resetBuilder = B.byteString reset-{-# NOINLINE boldBuilder #-}-{-# NOINLINE resetBuilder #-}--type ColorString = C.ByteString---buildFileName :: Config -> Options -> Output -> B.Builder-buildFileName conf opt = buildFileName' conf opt . outFilePath- where buildFileName' :: Config -> Options -> B.ByteString -> B.Builder- buildFileName' conf opt = buildColoredAs opt $ C.pack (setSGRCode (configColorFile conf))-{-# INLINE buildFileName #-}---buildColoredAs :: Options -> ColorString -> B.ByteString -> B.Builder-buildColoredAs Options { color = c, no_color = c'} colorCode str- | c && not c'= B.byteString colorCode <> B.byteString str <> resetBuilder- | otherwise = B.byteString str-{-# INLINE buildColoredAs #-}---buildLineCol :: Options -> Output -> B.Builder-buildLineCol Options{no_numbers = True } _ = mempty-buildLineCol Options{no_numbers = False, no_column = True } (Output _ n _ _) = B.int64Dec n-buildLineCol Options{no_numbers = False, no_column = False } (Output _ n _ []) = B.int64Dec n-buildLineCol Options{no_numbers = False, no_column = False } (Output _ n _ ts) = B.int64Dec n <> B.char8 ':' <> B.int64Dec ((+1) . cOffset . head $ ts)-{-# INLINE buildLineCol #-}---buildTokens :: Options -> Output -> B.Builder-buildTokens Options { show_match = st } out- | st = boldBuilder <> mconcat (B.byteString <$> outTokens out) <> resetBuilder <> B.char8 ':'- | otherwise = mempty---buildLine :: Config -> Options -> Output -> B.Builder-buildLine conf Options { color = c, no_color = c' } out- | c && not c'= highlightLine conf (sortBy (flip compare `on` (C.length . cToken)) (outChunks out)) (outLine out)- | otherwise = B.byteString $ outLine out-{-# INLINE buildLine #-}---showFileName :: Config -> Options -> RawFilePath -> RawFilePath-showFileName conf opt = showColoredAs opt $ C.pack (setSGRCode (configColorFile conf))-{-# INLINE showFileName #-}---showBold :: Options -> C.ByteString -> C.ByteString-showBold opt = showColoredAs opt bold-{-# INLINE showBold #-}---showColoredAs :: Options -> C.ByteString -> C.ByteString -> C.ByteString-showColoredAs Options { color = c, no_color = c'} colorCode str- | c && not c'= colorCode <> str <> reset- | otherwise = str-{-# INLINE showColoredAs #-}---highlightLine :: Config -> [Chunk] -> Text8 -> B.Builder-highlightLine conf ts = highlightLine' (highlightIndexes ts, 0, 0)- where highlightLine' :: ([(Int64, Int64)], Int64, Int) -> C.ByteString -> B.Builder- highlightLine' _ (C.uncons -> Nothing) = mempty- highlightLine' (ns, !n, !bs) s@(C.uncons -> Just (x,_)) =- (if | check && bs' == 0 -> if fst stack > 0 then B.string8 colorMatch <> B.char8 x <> resetBuilder else B.char8 x <> resetBuilder- | check && bs' > 0 -> B.string8 colorMatch <> B.char8 x- | otherwise -> B.byteString next) <> highlightLine' (ns, n + nn, bs') rest- where stack = foldr (\(a, b) (c, d) -> (c + fromEnum (a == n), d + fromEnum (b == n))) (0, 0) ns- check = fst stack > 0 || snd stack > 0- colorMatch = setSGRCode (configColorMatch conf)- bs' = bs + fst stack - snd stack- plain = nub . sort $ foldr (\(a, b) acc -> a : b : acc) [] ns- nn | check = 1- | null plain' = fromIntegral (C.length s)- | otherwise = head plain' - n- where plain' = dropWhile (<=n) plain- (next, rest) = C.splitAt (fromIntegral nn) s- highlightLine' _ _ = undefined---highlightIndexes :: [Chunk] -> [(Int64, Int64)]-highlightIndexes = foldr (\chunk a -> let b = cOffset chunk in (fromIntegral b, b + fromIntegral (C.length (cToken chunk)) - 1) : a) [] . filter (not. B.null . cToken)-{-# INLINE highlightIndexes #-}
src/CGrep/Parser/Atom.hs view
@@ -1,5 +1,5 @@ ----- Copyright (c) 2013-2023 Nicola Bonelli <nicola@larthia.com>+-- Copyright (c) 2013-2025 Nicola Bonelli <nicola@larthia.com> -- -- This program is free software; you can redistribute it and/or modify -- it under the terms of the GNU General Public License as published by@@ -16,206 +16,165 @@ -- Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. -- -module CGrep.Parser.Atom (Atom(..), Atoms,- mkAtomFromToken,- combineAtoms,- filterTokensWithAtoms,- wildCardMap,- wildCardMatch,- wildCardsMatch) where+module CGrep.Parser.Atom (+ Atom (..),+ mkAtomFromToken,+ findAllMatches,+ wildCardMap,+) where import qualified Data.Map as M -import CGrep.Common ( trim, trim8 )-import CGrep.Distance ( (~==) )-import CGrep.Parser.Char ( isDigit )+import CGrep.Common (trimT)+import CGrep.Distance ((~==))+import CGrep.Parser.Char (isDigit) -import Data.List- ( isSuffixOf, findIndices, isInfixOf, isPrefixOf, subsequences )-import Options- ( Options(edit_dist, word_match, prefix_match, suffix_match) )-import Util ( spanGroup, rmQuote8 )+import Data.List (tails) +import Options (+ Options (edit_dist, prefix_match, suffix_match, word_match),+ )+import Util (unquoteT)+ import qualified CGrep.Parser.Token as T-import qualified Data.ByteString.Char8 as C-import qualified CGrep.Parser.Chunk as T -data Atom =- Any |- Keyword |- Number |- Oct |- Hex |- String |- Literal |- Identifier C.ByteString |- Raw T.Token- deriving stock (Eq, Ord, Show)+import qualified Data.Text as T -type Atoms = [Atom]+data Atom+ = Any+ | Keyword+ | Number+ | Oct+ | Hex+ | String+ | Literal+ | Placeholder T.Text+ | Exact T.Token+ deriving stock (Eq, Ord, Show) -wildCardMap :: M.Map C.ByteString Atom-wildCardMap = M.fromList- [- ("ANY", Any ),- ("KEY", Keyword ),- ("OCT", Oct ),- ("HEX", Hex ),- ("NUM", Number ),- ("STR", String ),- ("LIT", String )- ]+wildCardMap :: M.Map T.Text Atom+wildCardMap =+ M.fromList+ [ ("ANY", Any)+ , ("KEY", Keyword)+ , ("OCT", Oct)+ , ("HEX", Hex)+ , ("NUM", Number)+ , ("STR", String)+ , ("LIT", String)+ ] mkAtomFromToken :: T.Token -> Atom mkAtomFromToken t | T.isTokenIdentifier t = case () of- _ | Just wc <- M.lookup str wildCardMap -> wc- | isAtomIdentifier str -> Identifier str- | otherwise -> Raw $ T.mkTokenIdentifier (rmAtomEscape str) (T.tOffset t)- where str = T.tToken t- | otherwise = Raw t---combineAtoms :: [Atoms] -> [Atoms]-combineAtoms (m1:r@(m2:m3:ms))- | [Raw b] <- m2, T.tToken b == "OR" = combineAtoms $ (m1<>m3):ms- | otherwise = m1 : combineAtoms r-combineAtoms [m1,m2] = [m1,m2]-combineAtoms [m1] = [m1]-combineAtoms [] = []---{-# INLINE filterTokensWithAtoms #-}-filterTokensWithAtoms :: Options -> [Atoms] -> [T.Token] -> [T.Token]-filterTokensWithAtoms opt ws ts = go opt (spanOptionalCards ws) ts- where go :: Options -> [[Atoms]] -> [T.Token] -> [T.Token]- go _ [] _ = []- go opt (g:gs) ts =- {-# SCC "atom_find_total" #-} concatMap (take grpLen . (`drop` ts)) ({-# SCC "atom_find_indices" #-} findIndices (wildCardsCompare opt g) grp) <> {-# SCC atom_find_req #-} go opt gs ts- where grp = {-# SCC "atomSpanGroup" #-} spanGroup grpLen ts- grpLen = length g---spanOptionalCards :: [Atoms] -> [[Atoms]]-spanOptionalCards wc = map (`filterCardIndices` wc') idx- where wc' = zip [0..] wc- idx = subsequences $- findIndices (\case- [Identifier (C.uncons -> Just ('$', _))] -> True- _ -> False) wc---filterCardIndices :: [Int] -> [(Int, Atoms)] -> [Atoms]-filterCardIndices ns ps = map snd $ filter (\(n, _) -> n `notElem` ns) ps-{-# INLINE filterCardIndices #-}---wildCardsCompare :: Options -> [Atoms] -> [T.Token] -> Bool-wildCardsCompare opt l r =- wildCardsCompareAll ts && wildCardsCheckOccurrences ts- where ts = wildCardsGroupCompare opt l r-{-# INLINE wildCardsCompare #-}-+ _+ | Just wc <- M.lookup txt wildCardMap -> wc+ | isAtomPlaceholder txt -> Placeholder txt+ | otherwise -> Exact $ T.mkTokenIdentifier (unescapeAtom txt)+ where+ txt = T.tToken t+ | otherwise = Exact t -isAtomIdentifier :: C.ByteString -> Bool-isAtomIdentifier s =- if | Just (x, C.uncons -> Just (y, xs)) <- C.uncons s -> wprefix x && isDigit y- | Just (x, "") <- C.uncons s -> wprefix x- | otherwise -> error "isAtomIdentifier"- where wprefix x = x == '$' || x == '_'+{-# INLINE isAtomPlaceholder #-}+isAtomPlaceholder :: T.Text -> Bool+isAtomPlaceholder s =+ if+ | Just (x, T.uncons -> Just (y, _)) <- T.uncons s -> wprefix x && isDigit y+ | Just (x, "") <- T.uncons s -> wprefix x+ | otherwise -> errorWithoutStackTrace "CGrep: isAtomIdentifier"+ where+ wprefix x = x == '$' || x == '_' +unescapeAtom :: T.Text -> T.Text+unescapeAtom (T.uncons -> Just ('$', xs)) = xs+unescapeAtom (T.uncons -> Just ('_', xs)) = xs+unescapeAtom xs = xs+{-# INLINE unescapeAtom #-} -rmAtomEscape :: C.ByteString -> C.ByteString-rmAtomEscape (C.uncons -> Just ('$',xs)) = xs-rmAtomEscape (C.uncons -> Just ('_',xs)) = xs-rmAtomEscape xs = xs-{-# INLINE rmAtomEscape #-}+findAllMatches :: Options -> [[Atom]] -> [T.Token] -> [T.Token]+findAllMatches opt ws ts = concatMap (\w -> findAllMatches' opt w ts) ws+{-# INLINE findAllMatches #-} +findAllMatches' :: Options -> [Atom] -> [T.Token] -> [T.Token]+findAllMatches' opt as ts =+ let indices = findIndicesBy (doesAtomMatchToken opt) as ts+ in concatMap+ ( \i ->+ let s = extractSlice i (length as) ts+ in if atomsCheckOccurrences as s+ then s+ else []+ )+ indices -wildCardsCompareAll :: [(Bool, (Atoms, [C.ByteString]))] -> Bool-wildCardsCompareAll = all fst-{-# INLINE wildCardsCompareAll #-}-{-# SCC wildCardsCompareAll #-}+extractSlice :: Int -> Int -> [a] -> [a]+extractSlice i n xs = take n (drop i xs)+{-# INLINE extractSlice #-} --- Note: pattern $ and _ match any token, whereas $1 $2 (_1 _2 etc.) match tokens--- that must compare equal in the respective occurrences+-- The pattern _ matches any token, whereas _1, _2, etc. match tokens that must be equal across their respective occurrences. -wildCardsCheckOccurrences :: [(Bool, (Atoms, [C.ByteString]))] -> Bool-wildCardsCheckOccurrences ts = M.foldr (\xs r -> r && all (== head xs) xs) True m- where m = M.mapWithKey (\k xs ->+atomsCheckOccurrences :: [Atom] -> [T.Token] -> Bool+atomsCheckOccurrences as ts = M.foldr checkAndFold True (m)+ where+ checkAndFold xs acc =+ acc && case xs of+ [] -> True+ (y : ys) -> all (== y) ys+ m =+ M.mapWithKey+ ( \k xs -> case k of- [Identifier "_0"] -> xs- [Identifier "_1"] -> xs- [Identifier "_2"] -> xs- [Identifier "_3"] -> xs- [Identifier "_4"] -> xs- [Identifier "_5"] -> xs- [Identifier "_6"] -> xs- [Identifier "_7"] -> xs- [Identifier "_8"] -> xs- [Identifier "_9"] -> xs- [Identifier "$0"] -> xs- [Identifier "$1"] -> xs- [Identifier "$2"] -> xs- [Identifier "$3"] -> xs- [Identifier "$4"] -> xs- [Identifier "$5"] -> xs- [Identifier "$6"] -> xs- [Identifier "$7"] -> xs- [Identifier "$8"] -> xs- [Identifier "$9"] -> xs- _ -> []- ) $ M.fromListWith (<>) (map snd ts)-{-# INLINE wildCardsCheckOccurrences #-}-{-# SCC wildCardsCheckOccurrences #-}---wildCardsGroupCompare :: Options -> [Atoms] -> [T.Token] -> [(Bool, (Atoms, [C.ByteString]))]-wildCardsGroupCompare opt ls rs- | length rs >= length ls = zipWith (tokensZip opt) ls rs- | otherwise = [ (False, ([Any], [])) ]-{-# INLINE wildCardsGroupCompare #-}-{-# SCC wildCardsGroupCompare #-}---tokensZip :: Options -> Atoms -> T.Token -> (Bool, (Atoms, [C.ByteString]))-tokensZip opt l r- | wildCardsMatch opt l r = (True, (l, [T.tToken r]))- | otherwise = (False, ([Any],[] ))-{-# INLINE tokensZip #-}-{-# SCC tokensZip #-}---wildCardsMatch :: Options -> Atoms -> T.Token -> Bool-wildCardsMatch opt m t = any (\w -> wildCardMatch opt w t) m-{-# INLINE wildCardsMatch #-}-{-# SCC wildCardsMatch #-}+ Placeholder "_0" -> xs+ Placeholder "_1" -> xs+ Placeholder "_2" -> xs+ Placeholder "_3" -> xs+ Placeholder "_4" -> xs+ Placeholder "_5" -> xs+ Placeholder "_6" -> xs+ Placeholder "_7" -> xs+ Placeholder "_8" -> xs+ Placeholder "_9" -> xs+ _ -> []+ )+ $ M.fromListWith (<>)+ $ zip as (take (length as) (map (: []) ts)) +doesAtomMatchToken :: Options -> Atom -> T.Token -> Bool+doesAtomMatchToken opt (Exact l) r+ | T.isTokenIdentifier l && T.isTokenIdentifier r =+ if+ | word_match opt -> T.tToken l == T.tToken r+ | prefix_match opt -> T.tToken l `T.isPrefixOf` T.tToken r+ | suffix_match opt -> T.tToken l `T.isSuffixOf` T.tToken r+ | edit_dist opt -> (T.unpack . T.tToken) l ~== T.unpack (T.tToken r)+ | otherwise -> T.tToken l `T.isInfixOf` T.tToken r+ | T.isTokenString l && T.isTokenString r =+ if+ | word_match opt -> ls == rs+ | prefix_match opt -> ls `T.isPrefixOf` rs+ | suffix_match opt -> ls `T.isSuffixOf` rs+ | edit_dist opt -> T.unpack ls ~== T.unpack rs+ | otherwise -> ls `T.isInfixOf` rs+ | otherwise = l `T.eqToken` r+ where+ ls = (unquoteT . trimT) (T.tToken l)+ rs = (unquoteT . trimT) (T.tToken r)+doesAtomMatchToken _ Any _ = True+doesAtomMatchToken _ (Placeholder _) t = T.isTokenIdentifier t+doesAtomMatchToken _ Keyword t = T.isTokenKeyword t+doesAtomMatchToken _ String t = T.isTokenString t+doesAtomMatchToken _ Literal t = T.isTokenString t+doesAtomMatchToken _ Number t = T.isTokenNumber t+doesAtomMatchToken _ Oct t = T.isTokenNumber t && case T.uncons (T.tToken t) of Just ('0', T.uncons -> Just (d, _)) -> isDigit d; _ -> False+doesAtomMatchToken _ Hex t = T.isTokenNumber t && case T.uncons (T.tToken t) of Just ('0', T.uncons -> Just ('x', _)) -> True; _ -> False -{-# SCC wildCardMatch #-}-wildCardMatch :: Options -> Atom -> T.Token -> Bool-wildCardMatch opt (Raw l) r- | T.isTokenIdentifier l && T.isTokenIdentifier r = {-# SCC wildcard_raw_0 #-}- if | word_match opt -> T.tToken l == T.tToken r- | prefix_match opt -> T.tToken l `C.isPrefixOf` T.tToken r- | suffix_match opt -> T.tToken l `C.isSuffixOf` T.tToken r- | edit_dist opt -> (C.unpack . T.tToken) l ~== C.unpack (T.tToken r)- | otherwise -> T.tToken l `C.isInfixOf` T.tToken r- | T.isTokenString l && T.isTokenString r = {-# SCC wildcard_raw_1 #-}- if | word_match opt -> ls == rs- | prefix_match opt -> ls `C.isPrefixOf` rs- | suffix_match opt -> ls `C.isSuffixOf` rs- | edit_dist opt -> C.unpack ls ~== C.unpack rs- | otherwise -> ls `C.isInfixOf` rs- | otherwise = {-# SCC wildcard_raw_2 #-} l `T.eqToken` r- where ls = rmQuote8 $ trim8 (T.tToken l)- rs = rmQuote8 $ trim8 (T.tToken r)+isPrefixOfBy :: (a -> b -> Bool) -> [a] -> [b] -> Bool+isPrefixOfBy _ [] _ = True+isPrefixOfBy _ (_ : _) [] = False+isPrefixOfBy p (x : xs) (y : ys) = p x y && isPrefixOfBy p xs ys+{-# INLINEABLE isPrefixOfBy #-} -wildCardMatch _ Any _ = {-# SCC wildcard_any #-} True-wildCardMatch _ (Identifier _) t = {-# SCC wildcard_identifier #-} T.isTokenIdentifier t-wildCardMatch _ Keyword t = {-# SCC wildcard_keyword #-} T.isTokenKeyword t-wildCardMatch _ String t = {-# SCC wildcard_string #-} T.isTokenString t-wildCardMatch _ Literal t = {-# SCC wildcard_lit #-} T.isTokenString t-wildCardMatch _ Number t = {-# SCC wildcard_number #-} T.isTokenNumber t-wildCardMatch _ Oct t = {-# SCC wildcard_octal #-} T.isTokenNumber t && case C.uncons (T.tToken t) of Just ('0', C.uncons -> Just (d, _)) -> isDigit d; _ -> False-wildCardMatch _ Hex t = {-# SCC wildcard_hex #-} T.isTokenNumber t && case C.uncons (T.tToken t) of Just ('0', C.uncons -> Just ('x',_)) -> True; _ -> False+findIndicesBy :: (a -> b -> Bool) -> [a] -> [b] -> [Int]+findIndicesBy p needle haystack =+ [i | (i, tail') <- zip [0 ..] (tails haystack), isPrefixOfBy p needle tail']+{-# INLINE findIndicesBy #-}
src/CGrep/Parser/Char.hs view
@@ -1,5 +1,5 @@ ----- Copyright (c) 2013-2023 Nicola Bonelli <nicola@larthia.com>+-- Copyright (c) 2013-2025 Nicola Bonelli <nicola@larthia.com> -- -- This program is free software; you can redistribute it and/or modify -- it under the terms of the GNU General Public License as published by@@ -15,46 +15,76 @@ -- along with this program; if not, write to the Free Software -- Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. --+{-# LANGUAGE OverloadedStrings #-} module CGrep.Parser.Char (- chr- , ord- , isDigit- , isSpace- , isHexDigit- , isCharNumber- , isAlphaNum- , isAlpha- , isAlphaNum_- , isAlpha_- , isAlpha_'- , isAlphaNum_'- , isBracket'- , isAlpha_and- , isAlphaNum_and+ chr,+ ord,+ isDigit,+ isSpace,+ isHexDigit,+ isCharNumber,+ isAlphaNum,+ isAlpha,+ isAlphaNum_,+ isAlpha_,+ isAlpha_',+ isAlphaNum_',+ isAlphaDollar_,+ isAlphaNumDollar_,+ isUnicode_,+ isUnicodeNum_,+ isUnicodeDollar_,+ isUnicodeNumDollar_,+ isUnicodeNum_',+ isUnicodeXIDStart_,+ isUnicodeNumXIDCont_,+ isBracket',+ isPunctuation,+ isClojureIdentStart,+ isClojureIdentCont,+ isAlphaDash_,+ isAlphaNumDash_,+ isCSharpIdentStart,+ isCSharpIdentCont,+ isHtmlIdentStart,+ isHtmlIdentCont,+ isJavaIdentStart,+ isJavaIdentCont,+ isJuliaIdentStart,+ isJuliaIdentCont,+ isLispIdent,+ isAgdaIdent, ) where -import GHC.Exts ( Char(C#), Int(I#), ord# )-import GHC.Base (isTrue#, int2Word#, leWord#, chr#)+import Data.Char (GeneralCategory (..), generalCategory, isLetter)+import GHC.Base (chr#, int2Word#, isTrue#, leWord#)+import GHC.Exts (Char (C#), Int (I#), ord#)+import GHC.Unicode (isAsciiLower) ord :: Char -> Int ord (C# c#) = I# (ord# c#) {-# INLINE ord #-} - chr :: Int -> Char chr i@(I# i#)- | isTrue# (int2Word# i# `leWord#` 0x10FFFF##) = C# (chr# i#)- | otherwise- = errorWithoutStackTrace ("CGrep.Char: bad argument: " ++ show i)+ | isTrue# (int2Word# i# `leWord#` 0x10FFFF##) = C# (chr# i#)+ | otherwise =+ errorWithoutStackTrace ("CGrep: chr bad argument: " <> show i) {-# INLINE chr #-} +isBracket' :: Char -> Bool+isBracket' c = c `elem` ("[]{}()" :: String)+{-# INLINE isBracket' #-} +isPunctuation :: Char -> Bool+isPunctuation c = c `elem` (";,." :: String)+{-# INLINE isPunctuation #-}+ isDigit :: Char -> Bool-isDigit c = (fromIntegral (ord c - ord '0') :: Word) <= 9+isDigit c = (fromIntegral (ord c - ord '0') :: Word) <= 9 {-# INLINE isDigit #-} - isSpace :: Char -> Bool isSpace c = uc == 32 || uc == 0xa0 || (uc - 0x9 <= 4) && not ctrl where@@ -62,69 +92,214 @@ ctrl = uc == 2 || uc == 3 {-# INLINE isSpace #-} --isHexDigit :: Char -> Bool-isHexDigit c = isDigit c ||- (fromIntegral (ord c - ord 'A')::Word) <= 5 ||- (fromIntegral (ord c - ord 'a')::Word) <= 5+isHexDigit :: Char -> Bool+isHexDigit c =+ isDigit c+ || (fromIntegral (ord c - ord 'A') :: Word) <= 5+ || (fromIntegral (ord c - ord 'a') :: Word) <= 5 {-# INLINE isHexDigit #-} - isCharNumber :: Char -> Bool isCharNumber c = isHexDigit c || c `elem` (".xX" :: String) {-# INLINE isCharNumber #-} - isAlphaNum :: Char -> Bool-isAlphaNum c = o >= 97 && o <= 122 ||- o >= 65 && o <= 90 ||- o >= 48 && o <= 57- where o = ord c+isAlphaNum c =+ o >= 97 && o <= 122+ || o >= 65 && o <= 90+ || o >= 48 && o <= 57+ where+ o = ord c {-# INLINE isAlphaNum #-} - isAlpha :: Char -> Bool-isAlpha c = o >= 97 && o <= 122 ||- o >= 65 && o <= 90- where o = ord c+isAlpha c =+ o >= 97 && o <= 122+ || o >= 65 && o <= 90+ where+ o = ord c {-# INLINE isAlpha #-} - isAlphaNum_ :: Char -> Bool-isAlphaNum_ c = o >= 97 && o <= 122 ||- o >= 65 && o <= 90 ||- o >= 48 && o <= 57 || c == '_'- where o = ord c+isAlphaNum_ c =+ o >= 97 && o <= 122+ || o >= 65 && o <= 90+ || o >= 48 && o <= 57+ || c == '_'+ where+ o = ord c {-# INLINE isAlphaNum_ #-} +isAlphaNumDash_ :: Char -> Bool+isAlphaNumDash_ c =+ o >= 97 && o <= 122+ || o >= 65 && o <= 90+ || o >= 48 && o <= 57+ || c == '_'+ || c == '-'+ where+ o = ord c+{-# INLINE isAlphaNumDash_ #-} isAlpha_ :: Char -> Bool-isAlpha_ c = o >= 97 && o <= 122 ||- o >= 65 && o <= 90 || c == '_'- where o = ord c+isAlpha_ c =+ o >= 97 && o <= 122+ || o >= 65 && o <= 90+ || c == '_'+ where+ o = ord c {-# INLINE isAlpha_ #-} +isAlphaDash_ :: Char -> Bool+isAlphaDash_ c =+ o >= 97 && o <= 122+ || o >= 65 && o <= 90+ || c == '_'+ || c == '-'+ where+ o = ord c+{-# INLINE isAlphaDash_ #-} isAlpha_' :: Char -> Bool isAlpha_' c = isAlpha_ c || c == '_' || c == '\'' {-# INLINE isAlpha_' #-} - isAlphaNum_' :: Char -> Bool isAlphaNum_' c = isAlphaNum_ c || c == '_' || c == '\'' {-# INLINE isAlphaNum_' #-} +isAlphaDollar_ :: Char -> Bool+isAlphaDollar_ c = isAlphaNum_ c || c == '$'+{-# INLINE isAlphaDollar_ #-} -isBracket' :: Char -> Bool-isBracket' c = c `elem` ("[]{}()" :: String)-{-# INLINE isBracket' #-}+isAlphaNumDollar_ :: Char -> Bool+isAlphaNumDollar_ c = isAlphaNum_ c || c == '_' || c == '$'+{-# INLINE isAlphaNumDollar_ #-} +isClojureIdentStart :: Char -> Bool+isClojureIdentStart c+ | isSpace c = False+ | isClojureDelimiter c = False+ | isDigit c = False+ | c `elem` ("#:" :: String) = False+ | otherwise = True+ where+ isClojureDelimiter x = x `elem` ("()[]{}@;~`'\"\\" :: String)+{-# INLINE isClojureIdentStart #-} -isAlpha_and :: String -> Char -> Bool-isAlpha_and s c = isAlpha_ c || c == '_' || c `elem` s-{-# INLINE isAlpha_and #-}+isClojureIdentCont :: Char -> Bool+isClojureIdentCont c = isClojureIdentStart c || isDigit c+{-# INLINE isClojureIdentCont #-} +isCSharpIdentStart :: Char -> Bool+isCSharpIdentStart c =+ case generalCategory c of+ UppercaseLetter -> True -- Lu+ LowercaseLetter -> True -- Ll+ TitlecaseLetter -> True -- Lt+ ModifierLetter -> True -- Lm+ OtherLetter -> True -- Lo+ LetterNumber -> True -- Nl+ _ -> c == '_' -- underscore permesso+{-# INLINE isCSharpIdentStart #-} -isAlphaNum_and :: String -> Char -> Bool-isAlphaNum_and s c = isAlphaNum_ c || c == '_' || c `elem` s-{-# INLINE isAlphaNum_and #-}+isCSharpIdentCont :: Char -> Bool+isCSharpIdentCont c =+ case generalCategory c of+ UppercaseLetter -> True -- Lu+ LowercaseLetter -> True -- Ll+ TitlecaseLetter -> True -- Lt+ ModifierLetter -> True -- Lm+ OtherLetter -> True -- Lo+ LetterNumber -> True -- Nl+ DecimalNumber -> True -- Nd+ ConnectorPunctuation -> True -- Pc (include _)+ NonSpacingMark -> True -- Mn+ SpacingCombiningMark -> True -- Mc+ Format -> True -- Cf+ _ -> False+{-# INLINE isCSharpIdentCont #-}++isHtmlIdentStart :: Char -> Bool+isHtmlIdentStart = isAlpha+{-# INLINE isHtmlIdentStart #-}++isHtmlIdentCont :: Char -> Bool+isHtmlIdentCont c = isAlphaNum c || c == '_' || c == '-' || c == '.' || c == ':'+{-# INLINE isHtmlIdentCont #-}++isJavaIdentStart :: Char -> Bool+isJavaIdentStart c = c == '_' || c == '$' || isLetter c+{-# INLINE isJavaIdentStart #-}++isJavaIdentCont :: Char -> Bool+isJavaIdentCont c = c == '_' || c == '$' || isLetter c || isDigit c+{-# INLINE isJavaIdentCont #-}++isJuliaIdentStart :: Char -> Bool+isJuliaIdentStart c = c == '_' || isLetter c+{-# INLINE isJuliaIdentStart #-}++isJuliaIdentCont :: Char -> Bool+isJuliaIdentCont c = c == '_' || c == '!' || c == '?' || isLetter c || isDigit c+{-# INLINE isJuliaIdentCont #-}++isUnicode_ :: Char -> Bool+isUnicode_ c = c == '_' || isLetter c+{-# INLINE isUnicode_ #-}++isUnicodeNum_ :: Char -> Bool+isUnicodeNum_ c = c == '_' || isLetter c || isDigit c+{-# INLINE isUnicodeNum_ #-}++isUnicodeDollar_ :: Char -> Bool+isUnicodeDollar_ c = c == '_' || c == '\'' || isLetter c || isDigit c+{-# INLINE isUnicodeDollar_ #-}++isUnicodeNumDollar_ :: Char -> Bool+isUnicodeNumDollar_ c = c == '_' || c == '$' || isLetter c || isDigit c+{-# INLINE isUnicodeNumDollar_ #-}++isUnicodeNum_' :: Char -> Bool+isUnicodeNum_' c = c == '_' || c == '\'' || isLetter c || isDigit c+{-# INLINE isUnicodeNum_' #-}++isLispIdent :: Char -> Bool+isLispIdent c =+ isAsciiLower c+ || isDigit c+ || c `elem` ("-+*/@$%^&_=<>~!?[]{}" :: String)++isUnicodeXIDStart_ :: Char -> Bool+isUnicodeXIDStart_ x = x == '_' || isXIDStart x+ where+ isXIDStart c = case generalCategory c of+ UppercaseLetter -> True+ LowercaseLetter -> True+ TitlecaseLetter -> True+ ModifierLetter -> True+ OtherLetter -> True+ LetterNumber -> True+ _ -> False+{-# INLINE isUnicodeXIDStart_ #-}++isUnicodeNumXIDCont_ :: Char -> Bool+isUnicodeNumXIDCont_ c = c == '_' || isXIDContinue c+ where+ isXIDContinue x = case generalCategory x of+ UppercaseLetter -> True+ LowercaseLetter -> True+ TitlecaseLetter -> True+ ModifierLetter -> True+ OtherLetter -> True+ LetterNumber -> True+ DecimalNumber -> True+ NonSpacingMark -> True+ SpacingCombiningMark -> True+ ConnectorPunctuation -> True+ _ -> False+{-# INLINE isUnicodeNumXIDCont_ #-}++isAgdaIdent :: Char -> Bool+isAgdaIdent c = not (c `elem` ("@.(){};_" :: String))+{-# INLINE isAgdaIdent #-}
src/CGrep/Parser/Chunk.hs view
@@ -1,5 +1,7 @@+{-# LANGUAGE AllowAmbiguousTypes #-}+{-# LANGUAGE DataKinds #-} ----- Copyright (c) 2013-2023 Nicola Bonelli <nicola@larthia.com>+-- Copyright (c) 2013-2025 Nicola Bonelli <nicola@larthia.com> -- -- This program is free software; you can redistribute it and/or modify -- it under the terms of the GNU General Public License as published by@@ -15,73 +17,52 @@ -- along with this program; if not, write to the Free Software -- Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. --+{-# LANGUAGE KindSignatures #-} module CGrep.Parser.Chunk (- parseChunks- , Chunk(..)- , MatchLine(..)- , ChunkType- , pattern ChunkIdentifier- , pattern ChunkKeyword- , pattern ChunkDigit- , pattern ChunkBracket- , pattern ChunkString- , pattern ChunkNativeType- , pattern ChunkOperator- , pattern ChunkUnspec- ) where---import CGrep.Parser.Char- ( isDigit,- isSpace,- isCharNumber,- isAlphaNum_,- isAlpha_,- isBracket',- isCharNumber,- isBracket',- isAlpha_,- isAlphaNum_- )--import CGrep.Types ( Text8, Offset )-import Data.List (genericLength)-import CGrep.FileTypeMap ( FileTypeInfo(..) )--import qualified Data.ByteString.Char8 as C-import qualified Data.ByteString.Lazy as LB-import qualified Data.ByteString.Internal as BI--import qualified ByteString.StrictBuilder as B+ parseChunks,+ Chunk (..),+ cOffset,+ MatchLine (..),+ ChunkType,+ pattern ChunkIdentifier,+ pattern ChunkKeyword,+ pattern ChunkDigit,+ pattern ChunkBracket,+ pattern ChunkString,+ pattern ChunkNativeType,+ pattern ChunkOperator,+ pattern ChunkUnspec,+) where -import Data.MonoTraversable ( MonoFoldable(oforM_) )+import CGrep.FileTypeMap (CharSet (..), FileTypeInfo (..), IsCharSet (..)) -import Data.STRef ( STRef, newSTRef, writeSTRef, readSTRef )-import Control.Monad.ST ( ST, runST )+import Control.Monad.ST (ST, runST)+import Data.STRef (STRef, newSTRef, readSTRef, writeSTRef) +import Data.Sequence ((|>)) import qualified Data.Sequence as S-import Data.Sequence ((|>), Seq((:<|), (:|>), Empty))-import Data.Maybe ( fromMaybe )-import Data.Word ( Word8 )+import qualified Data.Text as T+import qualified Data.Text.Unsafe as TU+import Data.Word (Word8) +import CGrep.Parser.Char (isBracket', isCharNumber, isDigit, isSpace)+import CGrep.Text (iterM, textOffsetWord8) -newtype ChunkType = ChunkType {unChunkType :: Word8}+newtype ChunkType = ChunkType {_unChunkType :: Word8} deriving newtype (Eq, Ord) - instance Show ChunkType where- show ChunkUnspec = "*"+ show ChunkUnspec = "*" show ChunkIdentifier = "identifier"- show ChunkKeyword = "keyword"- show ChunkDigit = "digit"- show ChunkBracket = "bracket"- show ChunkOperator = "operator"- show ChunkString = "string"+ show ChunkKeyword = "keyword"+ show ChunkDigit = "digit"+ show ChunkBracket = "bracket"+ show ChunkOperator = "operator"+ show ChunkString = "string" show ChunkNativeType = "native-type" {-# INLINE show #-} - pattern ChunkUnspec :: ChunkType pattern ChunkUnspec = ChunkType 0 @@ -108,28 +89,31 @@ {-# COMPLETE ChunkIdentifier, ChunkKeyword, ChunkDigit, ChunkBracket, ChunkOperator, ChunkString, ChunkNativeType, ChunkUnspec #-} -data Chunk = Chunk {- cTyp :: {-# UNPACK #-} !ChunkType- , cToken :: {-# UNPACK #-} !Text8- , cOffset :: {-# UNPACK #-} !Offset-} deriving stock (Eq, Show, Ord)-+data Chunk = Chunk+ { cTyp :: {-# UNPACK #-} !ChunkType+ , cToken :: {-# UNPACK #-} !T.Text+ }+ deriving stock (Eq, Show, Ord) -data MatchLine = MatchLine {- lOffset :: {-# UNPACK #-} !Offset,- lChunks :: [Chunk]-} deriving stock (Eq, Show)+cOffset :: Chunk -> Int+cOffset (Chunk _ t) = textOffsetWord8 t+{-# INLINE cOffset #-} +data MatchLine = MatchLine+ { mlOffset :: {-# UNPACK #-} !Int+ , mlChunks :: [Chunk]+ }+ deriving stock (Eq, Show) -newtype ChunkState = ChunkState { unChunkState :: Word8 }+newtype ChunkState = ChunkState {_unChunkState :: Word8} deriving newtype (Eq, Ord) instance Show ChunkState where- show StateSpace = "space"- show StateAlpha = "alpha"- show StateDigit = "digit"- show StateBracket = "bracket"- show StateOther = "other"+ show StateSpace = "space"+ show StateAlpha = "alpha"+ show StateDigit = "digit"+ show StateBracket = "bracket"+ show StateOther = "other" {-# INLINE show #-} pattern StateSpace :: ChunkState@@ -149,75 +133,97 @@ {-# COMPLETE StateSpace, StateAlpha, StateDigit, StateBracket, StateOther #-} - (<~) :: STRef s a -> a -> ST s () ref <~ !x = writeSTRef ref x {-# INLINE (<~) #-} +toChunk :: T.Text -> Maybe Int -> Int -> Chunk+toChunk _ Nothing _ = error "CGrep.Parser.Chunk.toChunk: Nothing (Internal Error)"+toChunk text (Just beg) cur =+ let chunk = TU.takeWord8 (cur - beg) (TU.dropWord8 beg text)+ in Chunk ChunkUnspec chunk+{-# INLINE toChunk #-} -{-# INLINE parseChunks #-}-parseChunks :: Maybe FileTypeInfo -> Text8 -> S.Seq Chunk-parseChunks l t = runST $ case l >>= \FileTypeInfo {..} -> ftIdentifierChars of- Just (isAlpha1, isAlphaN) -> parseChunks' isAlpha_ isAlphaNum_ t- _ -> parseChunks' isAlpha_ isAlphaNum_ t- where parseChunks' :: (Char->Bool) -> (Char -> Bool) -> C.ByteString -> ST s (S.Seq Chunk)- parseChunks' isAlpha1 isAlphaN txt = do- stateR <- newSTRef StateSpace- offR <- newSTRef 0- accR <- newSTRef (mempty :: B.Builder)- tokensR <- newSTRef S.empty- oforM_ txt $ \w -> do- let x = BI.w2c w- state <- readSTRef stateR- off <- readSTRef offR- acc <- readSTRef accR- tokens <- readSTRef tokensR- case state of- StateSpace ->- if | isSpace x -> do stateR <~ StateSpace ; accR <~ mempty- | isAlpha1 x -> do stateR <~ StateAlpha ; accR <~ B.asciiChar x- | isDigit x -> do stateR <~ StateDigit ; accR <~ B.asciiChar x- | isBracket' x -> do stateR <~ StateBracket ; accR <~ B.asciiChar x- | otherwise -> do stateR <~ StateOther ; accR <~ B.asciiChar x- StateAlpha ->- if | isAlphaN x -> do stateR <~ StateAlpha ; accR <~ (acc <> B.asciiChar x)- | isSpace x -> do stateR <~ StateSpace ; accR <~ mempty ; tokensR <~ (tokens |> toChunk off acc)- | isBracket' x -> do stateR <~ StateBracket ; accR <~ B.asciiChar x ; tokensR <~ (tokens |> toChunk off acc)- | otherwise -> do stateR <~ StateOther ; accR <~ B.asciiChar x ; tokensR <~ (tokens |> toChunk off acc)- StateDigit ->- if | isCharNumber x -> do stateR <~ StateDigit ; accR <~ (acc <> B.asciiChar x)- | isSpace x -> do stateR <~ StateSpace ; accR <~ mempty ; tokensR <~ (tokens |> toChunk off acc)- | isAlpha1 x -> do stateR <~ StateAlpha ; accR <~ B.asciiChar x ; tokensR <~ (tokens |> toChunk off acc)- | isBracket' x -> do stateR <~ StateBracket ; accR <~ B.asciiChar x ; tokensR <~ (tokens |> toChunk off acc)- | otherwise -> do stateR <~ StateOther ; accR <~ B.asciiChar x ; tokensR <~ (tokens |> toChunk off acc)- StateBracket ->- if | isSpace x -> do stateR <~ StateSpace ; accR <~ mempty ; tokensR <~ (tokens |> toChunk off acc)- | isAlpha1 x -> do stateR <~ StateAlpha ; accR <~ B.asciiChar x ; tokensR <~ (tokens |> toChunk off acc)- | isDigit x -> do stateR <~ StateDigit ; accR <~ B.asciiChar x ; tokensR <~ (tokens |> toChunk off acc)- | isBracket' x -> do stateR <~ StateBracket ; accR <~ B.asciiChar x ; tokensR <~ (tokens |> toChunk off acc)- | otherwise -> do stateR <~ StateOther ; accR <~ B.asciiChar x ; tokensR <~ (tokens |> toChunk off acc)- StateOther ->- if | isSpace x -> do stateR <~ StateSpace ; accR <~ mempty ; tokensR <~ (tokens |> toChunk off acc)- | isAlpha1 x -> do stateR <~ StateAlpha ; accR <~ B.asciiChar x ; tokensR <~ (tokens |> toChunk off acc)- | isDigit x -> if B.builderBytes acc == "."- then do stateR <~ StateDigit ; accR <~ (acc <> B.asciiChar x)- else do stateR <~ StateDigit ; accR <~ B.asciiChar x ; tokensR <~ (tokens |> toChunk off acc)- | isBracket' x -> do stateR <~ StateBracket ; accR <~ B.asciiChar x ; tokensR <~ (tokens |> toChunk off acc)- | otherwise -> do stateR <~ StateOther ; accR <~ (acc <> B.asciiChar x)- offR <~ (off+1)+{-# COMPLETE StateSpace, StateAlpha, StateDigit, StateBracket, StateOther #-} - lastAcc <- readSTRef accR- tokens <- readSTRef tokensR+{-# INLINE parseChunks #-}+parseChunks :: Maybe FileTypeInfo -> T.Text -> S.Seq Chunk+parseChunks l txt = runST $ case l >>= \FileTypeInfo{..} -> ftIdentCharSet of+ Just (None, None) -> parseChunks' @'None @'None txt+ Just (Alpha_, AlphaNum_') -> parseChunks' @'Alpha_ @'AlphaNum_' txt+ Just (Alpha_, AlphaNum_) -> parseChunks' @'Alpha_ @'AlphaNum_ txt+ Just (Alpha, AlphaNum_) -> parseChunks' @'Alpha @'AlphaNum_ txt+ Just (Alpha_, AlphaNumDash_) -> parseChunks' @'Alpha_ @'AlphaNumDash_ txt+ Just (AlphaDash_, AlphaNumDash_) -> parseChunks' @'AlphaDash_ @'AlphaNumDash_ txt+ Just (ClojureIdentStart, ClojureIdentCont) -> parseChunks' @'ClojureIdentStart @'ClojureIdentCont txt+ Just (CSharpIdentStart, CSharpIdentCont) -> parseChunks' @'CSharpIdentStart @'CSharpIdentCont txt+ Just (HtmlIdentStart, HtmlIdentCont) -> parseChunks' @'HtmlIdentStart @'HtmlIdentCont txt+ Just (JavaIdentStart, JavaIdentCont) -> parseChunks' @'JavaIdentStart @'JavaIdentCont txt+ Just (JuliaIdentStart, JuliaIdentCont) -> parseChunks' @'JuliaIdentStart @'JuliaIdentCont txt+ Just (ListIdent, ListIdent) -> parseChunks' @'ListIdent @'ListIdent txt+ Just (Unicode_, UnicodeNum_) -> parseChunks' @'Unicode_ @'UnicodeNum_ txt+ Just (UnicodeDollar_, UnicodeNumDollar_) -> parseChunks' @'UnicodeDollar_ @'UnicodeNumDollar_ txt+ Just (Unicode_, UnicodeNum_') -> parseChunks' @'Unicode_ @'UnicodeNum_' txt+ Just (Unicode_, UnicodeNumDollar_) -> parseChunks' @'Unicode_ @'UnicodeNumDollar_ txt+ Just (UnicodeXIDStart_, UnicodeNumXIDCont_) -> parseChunks' @'UnicodeXIDStart_ @'UnicodeNumXIDCont_ txt+ Just (AgdaIdent, AgdaIdent) -> parseChunks' @'AgdaIdent @'AgdaIdent txt+ Nothing -> parseChunks' @'None @'None txt+ charsets -> error $ "CGrep: unsupported CharSet combination: " <> show charsets - if B.builderLength lastAcc == 0- then return tokens- else do- state <- readSTRef stateR- off <- readSTRef offR- return $ tokens |> toChunk off lastAcc+parseChunks' :: forall (cs1 :: CharSet) (csN :: CharSet) s. (IsCharSet cs1, IsCharSet csN) => T.Text -> ST s (S.Seq Chunk)+parseChunks' txt = do+ stateR <- newSTRef StateSpace+ accR <- newSTRef Nothing+ tokensR <- newSTRef S.empty+ iterM txt $ \(# x, offset, _delta #) -> do+ state <- readSTRef stateR+ acc <- readSTRef accR+ tokens <- readSTRef tokensR+ case state of+ StateSpace ->+ if+ | isSpace x -> do stateR <~ StateSpace; accR <~ Nothing+ | isValidChar @cs1 x -> do stateR <~ StateAlpha; accR <~ Just offset+ | isDigit x -> do stateR <~ StateDigit; accR <~ Just offset+ | isBracket' x -> do stateR <~ StateBracket; accR <~ Just offset+ | otherwise -> do stateR <~ StateOther; accR <~ Just offset+ StateAlpha ->+ if+ | isValidChar @csN x -> do stateR <~ StateAlpha+ | isSpace x -> do stateR <~ StateSpace; accR <~ Nothing; tokensR <~ (tokens |> toChunk txt acc offset)+ | isBracket' x -> do stateR <~ StateBracket; accR <~ Just offset; tokensR <~ (tokens |> toChunk txt acc offset)+ | otherwise -> do stateR <~ StateOther; accR <~ Just offset; tokensR <~ (tokens |> toChunk txt acc offset)+ StateDigit ->+ if+ | isCharNumber x -> do stateR <~ StateDigit+ | isSpace x -> do stateR <~ StateSpace; accR <~ Nothing; tokensR <~ (tokens |> toChunk txt acc offset)+ | isValidChar @cs1 x -> do stateR <~ StateAlpha; accR <~ Just offset; tokensR <~ (tokens |> toChunk txt acc offset)+ | isBracket' x -> do stateR <~ StateBracket; accR <~ Just offset; tokensR <~ (tokens |> toChunk txt acc offset)+ | otherwise -> do stateR <~ StateOther; accR <~ Just offset; tokensR <~ (tokens |> toChunk txt acc offset)+ StateBracket ->+ if+ | isSpace x -> do stateR <~ StateSpace; accR <~ Nothing; tokensR <~ (tokens |> toChunk txt acc offset)+ | isValidChar @cs1 x -> do stateR <~ StateAlpha; accR <~ Just offset; tokensR <~ (tokens |> toChunk txt acc offset)+ | isDigit x -> do stateR <~ StateDigit; accR <~ Just offset; tokensR <~ (tokens |> toChunk txt acc offset)+ | isBracket' x -> do stateR <~ StateBracket; accR <~ Just offset; tokensR <~ (tokens |> toChunk txt acc offset)+ | otherwise -> do stateR <~ StateOther; accR <~ Just offset; tokensR <~ (tokens |> toChunk txt acc offset)+ StateOther ->+ if+ | isSpace x -> do stateR <~ StateSpace; accR <~ Nothing; tokensR <~ (tokens |> toChunk txt acc offset)+ | isValidChar @cs1 x -> do stateR <~ StateAlpha; accR <~ Just offset; tokensR <~ (tokens |> toChunk txt acc offset)+ | isDigit x ->+ if acc == Just (offset - 1)+ && TU.unsafeHead (TU.dropWord8 (offset - 1) txt) == '.'+ then do stateR <~ StateDigit+ else do stateR <~ StateDigit; accR <~ Just offset; tokensR <~ (tokens |> toChunk txt acc offset)+ | isBracket' x -> do stateR <~ StateBracket; accR <~ Just offset; tokensR <~ (tokens |> toChunk txt acc offset)+ | otherwise -> do stateR <~ StateOther + tokens <- readSTRef tokensR+ lastAcc <- readSTRef accR -toChunk :: Offset -> B.Builder -> Chunk-toChunk off b = Chunk ChunkUnspec str (off - fromIntegral (B.builderLength b))- where str = B.builderBytes b-{-# INLINE toChunk #-}+ case lastAcc of+ Nothing -> return tokens+ Just off+ | off == TU.lengthWord8 txt -> return tokens+ | otherwise -> return $ tokens |> toChunk txt lastAcc (TU.lengthWord8 txt)
− src/CGrep/Parser/Line.hs
@@ -1,91 +0,0 @@------ Copyright (c) 2013-2023 Nicola Bonelli <nicola@larthia.com>------ This program is free software; you can redistribute it and/or modify--- it under the terms of the GNU General Public License as published by--- the Free Software Foundation; either version 2 of the License, or--- (at your option) any later version.------ This program is distributed in the hope that it will be useful,--- but WITHOUT ANY WARRANTY; without even the implied warranty of--- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the--- GNU General Public License for more details.------ You should have received a copy of the GNU General Public License--- along with this program; if not, write to the Free Software--- Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.-----module CGrep.Parser.Line (- getLineOffsets- , getAllLineOffsets- , getLineByOffset- , lowerBound- ) where--import qualified Data.ByteString.Char8 as C-import qualified Data.ByteString.Lazy.Char8 as LC--import qualified Data.Vector.Unboxed as UV-import Data.Vector.Unboxed ((!))-import qualified Data.ByteString.Unsafe as BU-import Data.ByteString.Internal (c2w)-import CGrep.Types ( Offset, Text8, LText8 )-import Data.Int ( Int64 )----- Returns a vector of offsets for a given character in a ByteString, up to the given maximum offset.-charOffsets :: Char -> Int64 -> C.ByteString -> UV.Vector Int64-charOffsets c maxOff bs = UV.unfoldrN (fromIntegral maxOff) (findOffsets bs maxOff) 0- where findOffsets :: C.ByteString -> Int64 -> Int64 -> Maybe (Int64, Int64)- findOffsets bs' maxOff' i- | i >= maxOff' = Nothing- | BU.unsafeIndex bs' (fromIntegral i) == c2w c = Just (fromIntegral i, i + 1)- | otherwise = findOffsets bs' maxOff' (i + 1)---getLineOffsets :: Int64 -> Text8 -> UV.Vector Offset-getLineOffsets maxOff text =- let idx = nlOffsets (fromIntegral maxOff) text- in if UV.null idx- then idx- else if UV.last idx == fromIntegral (C.length text -1)- then UV.init idx- else idx--{-# INLINE nlOffsets #-}-nlOffsets :: Int -> Text8 -> UV.Vector Int64-nlOffsets maxOff' bs' = UV.unfoldrN maxOff' (findOffsets maxOff' bs') (-1)--findOffsets :: Int -> Text8 -> Int -> Maybe (Int64, Int)-findOffsets max ts !i- | i == -1 = Just (0, 0)- | i >= max = Nothing- | BU.unsafeIndex ts (fromIntegral i) == c2w '\n' = Just (fromIntegral i + 1, i + 1)- | otherwise = findOffsets max ts (i + 1)---getAllLineOffsets :: Text8 -> UV.Vector Offset-getAllLineOffsets ts = getLineOffsets (fromIntegral $ C.length ts) ts-{-# INLINE getAllLineOffsets #-}---lowerBound :: UV.Vector Int64 -> Int64 -> Int64-lowerBound vec v = lowerBoundGo vec v 0 (UV.length vec-1)--lowerBoundGo :: UV.Vector Int64 -> Int64 -> Int -> Int -> Int64-lowerBoundGo vec v !left !right- | left > right = if right >= 0 then vec `UV.unsafeIndex` right else -1- | otherwise = case v `compare` midValue of- LT -> lowerBoundGo vec v left (mid - 1)- EQ -> midValue- _ -> lowerBoundGo vec v (mid + 1) right- where- mid = (left + right) `div` 2- midValue = vec `UV.unsafeIndex` mid---getLineByOffset :: Offset -> Text8 -> UV.Vector Int64 -> (# Text8, Offset #)-getLineByOffset off text vec = (# (head . C.lines) (C.drop (fromIntegral lb) text), lb #)- where lb = lowerBound vec off-{-# INLINE getLineByOffset #-}
src/CGrep/Parser/Token.hs view
@@ -1,5 +1,8 @@+{-# LANGUAGE AllowAmbiguousTypes #-}+{-# LANGUAGE DataKinds #-}+{-# LANGUAGE KindSignatures #-} ----- Copyright (c) 2013-2023 Nicola Bonelli <nicola@larthia.com>+-- Copyright (c) 2013-2025 Nicola Bonelli <nicola@larthia.com> -- -- This program is free software; you can redistribute it and/or modify -- it under the terms of the GNU General Public License as published by@@ -15,183 +18,184 @@ -- along with this program; if not, write to the Free Software -- Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. ----{-# LANGUAGE OverloadedRecordDot #-}-{-# LANGUAGE DuplicateRecordFields #-}+{-# LANGUAGE ScopedTypeVariables #-}+{-# LANGUAGE TypeApplications #-} module CGrep.Parser.Token (- parseTokens- , filterToken- , Token(..)- , TokenFilter(..)- , mkTokenFilter- , eqToken- , isTokenIdentifier- , isTokenKeyword- , isTokenNumber- , isTokenBracket- , isTokenString- , isTokenOperator- , isTokenUnspecified- , tTyp- , tToken- , tOffset- , mkTokenIdentifier- , mkTokenKeyword- , mkTokenDigit- , mkTokenBracket- , mkTokenString- , mkTokenOperator-- ) where+ parseTokens,+ filterToken,+ Token (..),+ TokenFilter (..),+ mkTokenFilter,+ eqToken,+ isTokenIdentifier,+ isTokenKeyword,+ isTokenNumber,+ isTokenBracket,+ isTokenString,+ isTokenOperator,+ isTokenUnspecified,+ tTyp,+ tToken,+ tOffset,+ mkTokenIdentifier,+ mkTokenKeyword,+ mkTokenDigit,+ mkTokenBracket,+ mkTokenString,+ mkTokenOperator,+) where -import qualified Data.ByteString.Char8 as C-import qualified Data.ByteString.Lazy as LB-import qualified Data.ByteString.Internal as BI-import qualified Data.DList as DL+-- import qualified Data.DList as DL -import CGrep.Parser.Char- ( chr,- isAlphaNum_,- isAlpha_,- isBracket',- isCharNumber,- isDigit,- isSpace,- isCharNumber,- isBracket',- isAlpha_,- isAlphaNum_ )+-- import CGrep.Parser.Char (+-- chr,+-- isAlphaNum_,+-- isAlpha_,+-- isBracket',+-- isCharNumber,+-- isDigit,+-- isPunctuation,+-- isSpace,+-- ) -import CGrep.Types (Offset, Text8)-import Data.List (genericLength)+-- import Data.List (genericLength) -import CGrep.FileTypeMap- ( CharIdentifierF,- FileTypeInfo(ftKeywords, ftIdentifierChars), WordType (..) )+import CGrep.FileTypeMap (+ -- CharIdentifierF,+ CharSet (..),+ FileTypeInfo (..),+ IsCharSet (..),+ WordType (..),+ ) import qualified Data.HashMap.Strict as HM++-- import Data.Sequence (Seq (Empty, (:<|), (:|>)), (|>))+import Data.Sequence (Seq, (|>)) import qualified Data.Sequence as S-import Data.Sequence ((|>), Seq((:<|), (:|>), Empty)) -import Control.Monad.ST ( ST, runST )-import Data.STRef ( STRef, newSTRef, readSTRef, writeSTRef, modifySTRef', modifySTRef )-import Data.MonoTraversable ( MonoFoldable(oforM_) )-import Data.Word (Word8)+import Control.Monad.ST (ST, runST)+import Data.STRef (STRef, modifySTRef', newSTRef, readSTRef, writeSTRef) -import qualified ByteString.StrictBuilder as B import CGrep.Parser.Chunk -import GHC.Exts ( inline )-import Data.Coerce ( coerce )-import Data.Text.Internal.Read (T)+import CGrep.Semantic.ContextFilter+import CGrep.Parser.Char (isBracket', isCharNumber, isDigit, isPunctuation, isSpace)+import CGrep.Text (iterM, textOffsetWord8, textSlice)+import Data.Coerce (coerce)+import Data.Maybe (fromMaybe)+import qualified Data.Text as T+import qualified Data.Text.Unsafe as TU+import GHC.Exts (inline) -newtype TokenState = TokenState { unTokenState :: Int }+newtype TokenState = TokenState {_unTokenState :: Int} deriving newtype (Eq) instance Show TokenState where- show StateSpace = "space"+ show StateSpace = "space" show StateIdentifier = "identifier"- show StateDigit = "digit"- show StateBracket = "bracket"- show StateLiteral = "literal"- show StateOther = "other"-+ show StateDigit = "digit"+ show StateBracket = "bracket"+ show StateLiteral = "literal"+ show StateOther = "other" {-# INLINE show #-} -pattern StateSpace :: TokenState+pattern StateSpace :: TokenState pattern StateSpace = TokenState 0 -pattern StateIdentifier :: TokenState+pattern StateIdentifier :: TokenState pattern StateIdentifier = TokenState 1 -pattern StateDigit :: TokenState+pattern StateDigit :: TokenState pattern StateDigit = TokenState 2 -pattern StateBracket :: TokenState+pattern StateBracket :: TokenState pattern StateBracket = TokenState 3 -pattern StateLiteral :: TokenState+pattern StateLiteral :: TokenState pattern StateLiteral = TokenState 4 -pattern StateOther :: TokenState+pattern StateOther :: TokenState pattern StateOther = TokenState 5 +{-# COMPLETE StateSpace, StateIdentifier, StateDigit, StateBracket, StateLiteral, StateOther #-}+ newtype Token = Token Chunk deriving newtype (Eq, Ord) instance Show Token where- show (Token (Chunk typ bs off)) = "(" ++ show typ ++ " " ++ C.unpack bs ++ " @" ++ show off ++ ")"+ show (Token (Chunk typ txt)) = case typ of+ ChunkUnspec -> "(*)"+ _ -> "(" <> show typ <> " '" <> T.unpack txt <> "' @" <> show (textOffsetWord8 txt) <> ")" {-# INLINE show #-} - eqToken :: Token -> Token -> Bool-eqToken a b = tToken a == tToken b &&- tTyp a == tTyp b+eqToken a b =+ tToken a == tToken b+ && tTyp a == tTyp b {-# INLINE eqToken #-} --mkTokenIdentifier :: C.ByteString -> Offset -> Token-mkTokenIdentifier bs off = Token $ Chunk ChunkIdentifier bs off+mkTokenIdentifier :: T.Text -> Token+mkTokenIdentifier bs = Token $ Chunk ChunkIdentifier bs {-# INLINE mkTokenIdentifier #-} -mkTokenKeyword :: C.ByteString -> Offset -> Token-mkTokenKeyword bs off = Token $ Chunk ChunkKeyword bs off+mkTokenKeyword :: T.Text -> Token+mkTokenKeyword bs = Token $ Chunk ChunkKeyword bs {-# INLINE mkTokenKeyword #-} -mkTokenDigit :: C.ByteString -> Offset -> Token-mkTokenDigit bs off = Token $ Chunk ChunkDigit bs off+mkTokenDigit :: T.Text -> Token+mkTokenDigit bs = Token $ Chunk ChunkDigit bs {-# INLINE mkTokenDigit #-} -mkTokenBracket :: C.ByteString -> Offset -> Token-mkTokenBracket bs off = Token $ Chunk ChunkBracket bs off+mkTokenBracket :: T.Text -> Token+mkTokenBracket bs = Token $ Chunk ChunkBracket bs {-# INLINE mkTokenBracket #-} -mkTokenOperator :: C.ByteString -> Offset -> Token-mkTokenOperator bs off = Token $ Chunk ChunkOperator bs off+mkTokenOperator :: T.Text -> Token+mkTokenOperator bs = Token $ Chunk ChunkOperator bs {-# INLINE mkTokenOperator #-} -mkTokenString :: C.ByteString -> Offset -> Token-mkTokenString bs off = Token $ Chunk ChunkString bs off+mkTokenString :: T.Text -> Token+mkTokenString bs = Token $ Chunk ChunkString bs {-# INLINE mkTokenString #-} -mkTokenNativeType :: C.ByteString -> Offset -> Token-mkTokenNativeType bs off = Token $ Chunk ChunkNativeType bs off+mkTokenNativeType :: T.Text -> Token+mkTokenNativeType bs = Token $ Chunk ChunkNativeType bs {-# INLINE mkTokenNativeType #-} -mkTokenFromWord :: Maybe FileTypeInfo -> C.ByteString -> Offset -> Token-mkTokenFromWord Nothing txt off = mkTokenIdentifier txt off-mkTokenFromWord (Just info) txt off =+mkTokenFromWord :: Maybe FileTypeInfo -> T.Text -> Token+mkTokenFromWord Nothing txt = mkTokenIdentifier txt+mkTokenFromWord (Just info) txt = case HM.lookup txt (ftKeywords info) of- Just typ -> case typ of- Keyword -> mkTokenKeyword txt off- NativeType -> mkTokenNativeType txt off- _ -> mkTokenIdentifier txt off-{-# INLINABLE mkTokenFromWord #-}---mkToken :: Maybe FileTypeInfo -> TokenState -> C.ByteString -> Offset -> Token-mkToken _ StateSpace = mkTokenOperator-mkToken info StateIdentifier = mkTokenFromWord info-mkToken _ StateDigit = mkTokenDigit-mkToken _ StateBracket = mkTokenBracket-mkToken _ StateLiteral = mkTokenString-mkToken _ StateOther = mkTokenOperator+ Just typ -> case typ of+ Keyword -> mkTokenKeyword txt+ NativeType -> mkTokenNativeType txt+ _ -> mkTokenIdentifier txt+{-# INLINEABLE mkTokenFromWord #-} +mkToken :: Maybe FileTypeInfo -> TokenState -> T.Text -> Token+mkToken info state = case state of+ StateSpace -> mkTokenOperator+ StateIdentifier -> mkTokenFromWord info+ StateDigit -> mkTokenDigit+ StateBracket -> mkTokenBracket+ StateLiteral -> mkTokenString+ StateOther -> mkTokenOperator+{-# INLINE mkToken #-} tTyp :: Token -> ChunkType tTyp = cTyp . coerce {-# INLINE tTyp #-} -tOffset :: Token -> Offset+tOffset :: Token -> Int tOffset t = cOffset (coerce t :: Chunk) {-# INLINE tOffset #-} -tToken :: Token -> Text8+tToken :: Token -> T.Text tToken t = cToken (coerce t :: Chunk) {-# INLINE tToken #-} - isTokenIdentifier :: Token -> Bool isTokenIdentifier t = cTyp (coerce t) == ChunkIdentifier {-# INLINE isTokenIdentifier #-}@@ -224,191 +228,197 @@ isTokenUnspecified t = cTyp (coerce t) == ChunkUnspec {-# INLINE isTokenUnspecified #-} - data TokenFilter = TokenFilter- { tfIdentifier :: !Bool- , tfKeyword :: !Bool- , tfNativeType :: !Bool- , tfString :: !Bool- , tfNumber :: !Bool- , tfOperator :: !Bool- , tfBracket :: !Bool- } deriving stock (Eq, Show)-+ { tfIdentifier :: {-# UNPACK #-} !Bool+ , tfKeyword :: {-# UNPACK #-} !Bool+ , tfNativeType :: {-# UNPACK #-} !Bool+ , tfString :: {-# UNPACK #-} !Bool+ , tfNumber :: {-# UNPACK #-} !Bool+ , tfOperator :: {-# UNPACK #-} !Bool+ , tfBracket :: {-# UNPACK #-} !Bool+ }+ deriving stock (Eq, Show) filterToken :: TokenFilter -> Token -> Bool filterToken f t = case cTyp (coerce t :: Chunk) of ChunkIdentifier -> tfIdentifier f- ChunkKeyword -> tfKeyword f- ChunkDigit -> tfNumber f- ChunkOperator -> tfOperator f- ChunkString -> tfString f+ ChunkKeyword -> tfKeyword f+ ChunkDigit -> tfNumber f+ ChunkOperator -> tfOperator f+ ChunkString -> tfString f ChunkNativeType -> tfNativeType f- ChunkBracket -> tfBracket f- ChunkUnspec -> False-+ ChunkBracket -> tfBracket f+ ChunkUnspec -> False mkTokenFilter :: (Traversable t) => t ChunkType -> TokenFilter mkTokenFilter = foldr go (TokenFilter False False False False False False False)- where- go ChunkIdentifier f = f { tfIdentifier = True }- go ChunkKeyword f = f { tfKeyword = True }- go ChunkNativeType f = f { tfNativeType = True }- go ChunkDigit f = f { tfNumber = True }- go ChunkOperator f = f { tfOperator = True }- go ChunkString f = f { tfString = True }- go ChunkBracket f = f { tfBracket = True }- go ChunkUnspec f = f-+ where+ go ChunkIdentifier f = f{tfIdentifier = True}+ go ChunkKeyword f = f{tfKeyword = True}+ go ChunkNativeType f = f{tfNativeType = True}+ go ChunkDigit f = f{tfNumber = True}+ go ChunkOperator f = f{tfOperator = True}+ go ChunkString f = f{tfString = True}+ go ChunkBracket f = f{tfBracket = True}+ go ChunkUnspec f = f (<~) :: STRef s a -> a -> ST s () ref <~ !x = writeSTRef ref x {-# INLINE (<~) #-} --data TokenIdx = TokenIdx {- offset :: {-# UNPACK #-}!Int- , len :: {-# UNPACK #-}!Int-}+data TokenIdx = TokenIdx+ { _offset :: {-# UNPACK #-} !Int+ , len :: {-# UNPACK #-} !Int+ } -tkString :: TokenIdx -> C.ByteString -> C.ByteString-tkString (TokenIdx off len) = C.take len . C.drop off+tkString :: TokenIdx -> T.Text -> T.Text+tkString (TokenIdx off len) = TU.takeWord8 len . TU.dropWord8 off {-# INLINE tkString #-} --data AccOp = Reset | Start {-# UNPACK #-}!Int | Append {-# UNPACK #-} !Int-+data AccOp = Reset | Start {-# UNPACK #-} !Int !Int | Append {-# UNPACK #-} !Int !Int (<<~) :: STRef s TokenIdx -> AccOp -> ST s ()-ref <<~ Reset = writeSTRef ref (TokenIdx (-1) 0)-ref <<~ Start cur = writeSTRef ref (TokenIdx cur 1)-ref <<~ Append cur = modifySTRef' ref $ \case- TokenIdx (-1) 0 -> TokenIdx cur 1- TokenIdx off len -> TokenIdx off (len + 1)-+ref <<~ Reset = writeSTRef ref (TokenIdx (-1) 0)+ref <<~ Start offset delta = writeSTRef ref (TokenIdx offset delta)+ref <<~ Append offset delta = modifySTRef' ref $ \case+ TokenIdx (-1) 0 -> TokenIdx offset delta+ TokenIdx offset' delta' -> TokenIdx offset' (delta + delta') {-# INLINE (<<~) #-} - {-# INLINE parseTokens #-}-parseTokens :: TokenFilter -> Maybe FileTypeInfo -> C.ByteString -> S.Seq Token-parseTokens f@TokenFilter{..} l t = runST (case l >>= ftIdentifierChars of- Nothing -> parseToken' isAlpha_ isAlphaNum_ l t- Just (isAlpha1, isAlphaN) -> parseToken' isAlpha1 isAlphaN l t)- where parseToken' :: CharIdentifierF -> CharIdentifierF -> Maybe FileTypeInfo -> C.ByteString -> ST a (S.Seq Token)- parseToken' isAlpha1 isAlphaN info txt = do-- stateR <- newSTRef StateSpace- accR <- newSTRef (TokenIdx (-1) (-1))- tokensR <- newSTRef S.empty- curR <- newSTRef 0-- oforM_ txt $ \w -> do-- let x = BI.w2c w- cur <- readSTRef curR- state <- readSTRef stateR-- case state of- StateSpace -> {-# SCC "StateSpace" #-}- if | isSpace x -> do accR <<~ Reset- | inline isAlpha1 x -> do stateR <~ StateIdentifier ; accR <<~ Start cur- | x == chr 2 -> do stateR <~ StateLiteral ; accR <<~ Reset- | isDigit x -> do stateR <~ StateDigit ; accR <<~ Start cur- | isBracket' x -> do stateR <~ StateBracket ; accR <<~ Start cur- | otherwise -> do stateR <~ StateOther ; accR <<~ Start cur-- StateIdentifier -> {-# SCC "StateIdentifier" #-}- if isAlphaN x- then accR <<~ Append cur- else do- acc <- readSTRef accR- tokens <- readSTRef tokensR- if | isSpace x -> do stateR <~ StateSpace ; accR <<~ Reset ; tokensR <~ (tokens |> buildToken_ tfIdentifier tfKeyword tfNativeType (mkTokenFromWord info) acc txt)- | x == chr 2 -> do stateR <~ StateLiteral ; accR <<~ Reset ; tokensR <~ (tokens |> buildToken_ tfIdentifier tfKeyword tfNativeType (mkTokenFromWord info) acc txt)- | isBracket' x -> do stateR <~ StateBracket ; accR <<~ Start cur ; tokensR <~ (tokens |> buildToken_ tfIdentifier tfKeyword tfNativeType (mkTokenFromWord info) acc txt)- | otherwise -> do stateR <~ StateOther ; accR <<~ Start cur ; tokensR <~ (tokens |> buildToken_ tfIdentifier tfKeyword tfNativeType (mkTokenFromWord info) acc txt)-- StateDigit -> {-# SCC "StateDigit" #-}- if isCharNumber x- then accR <<~ Append cur- else do- acc <- readSTRef accR- tokens <- readSTRef tokensR- if | isSpace x -> do stateR <~ StateSpace ; accR <<~ Reset ; tokensR <~ (tokens |> buildToken tfNumber mkTokenDigit acc txt)- | x == chr 2 -> do stateR <~ StateLiteral ; accR <<~ Reset ; tokensR <~ (tokens |> buildToken tfNumber mkTokenDigit acc txt)- | inline isAlpha1 x -> do stateR <~ StateIdentifier ; accR <<~ Start cur ; tokensR <~ (tokens |> buildToken tfNumber mkTokenDigit acc txt)- | isBracket' x -> do stateR <~ StateBracket ; accR <<~ Start cur ; tokensR <~ (tokens |> buildToken tfNumber mkTokenDigit acc txt)- | otherwise -> do stateR <~ StateOther ; accR <<~ Start cur ; tokensR <~ (tokens |> buildToken tfNumber mkTokenDigit acc txt)+parseTokens :: TokenFilter -> Maybe FileTypeInfo -> Bool -> T.Text -> Seq Token+parseTokens tf info strict txt =+ runST $ do+ let (runtimeCS1, runtimeCS2) = fromMaybe (None, None) (info >>= ftIdentCharSet)+ case (runtimeCS1, runtimeCS2) of+ (None, None) -> parseToken' @'None @'None tf info True txt+ (Alpha_, AlphaNum_') -> parseToken' @'Alpha_ @'AlphaNum_' tf info strict txt+ (Alpha_, AlphaNum_) -> parseToken' @'Alpha_ @'AlphaNum_ tf info strict txt+ (Alpha, AlphaNum_) -> parseToken' @'Alpha @'AlphaNum_ tf info strict txt+ (Alpha_, AlphaNumDash_) -> parseToken' @'Alpha_ @'AlphaNumDash_ tf info strict txt+ (AlphaDash_, AlphaNumDash_) -> parseToken' @'AlphaDash_ @'AlphaNumDash_ tf info strict txt+ (ClojureIdentStart, ClojureIdentCont) -> parseToken' @'ClojureIdentStart @'ClojureIdentCont tf info strict txt+ (CSharpIdentStart, CSharpIdentCont) -> parseToken' @'CSharpIdentStart @'CSharpIdentCont tf info strict txt+ (HtmlIdentStart, HtmlIdentCont) -> parseToken' @'HtmlIdentStart @'HtmlIdentCont tf info strict txt+ (JavaIdentStart, JavaIdentCont) -> parseToken' @'JavaIdentStart @'JavaIdentCont tf info strict txt+ (JuliaIdentStart, JuliaIdentCont) -> parseToken' @'JuliaIdentStart @'JuliaIdentCont tf info strict txt+ (ListIdent, ListIdent) -> parseToken' @'ListIdent @'ListIdent tf info strict txt+ (Unicode_, UnicodeNum_) -> parseToken' @'Unicode_ @'UnicodeNum_ tf info strict txt+ (UnicodeDollar_, UnicodeNumDollar_) -> parseToken' @'UnicodeDollar_ @'UnicodeNumDollar_ tf info strict txt+ (Unicode_, UnicodeNum_') -> parseToken' @'Unicode_ @'UnicodeNum_' tf info strict txt+ (Unicode_, UnicodeNumDollar_) -> parseToken' @'Unicode_ @'UnicodeNumDollar_ tf info strict txt+ (UnicodeXIDStart_, UnicodeNumXIDCont_) -> parseToken' @'UnicodeXIDStart_ @'UnicodeNumXIDCont_ tf info strict txt+ (AgdaIdent, AgdaIdent) -> parseToken' @'AgdaIdent @'AgdaIdent tf info strict txt+ _ -> error $ "CGrep: unsupported CharSet combination: " <> show (runtimeCS1, runtimeCS2) - StateLiteral -> {-# SCC "StateLiteral" #-}- if x == chr 3- then do- acc <- readSTRef accR- tokens <- readSTRef tokensR- stateR <~ StateSpace ; accR <<~ Reset; tokensR <~ (tokens |> buildToken tfString mkTokenString acc txt)- else do accR <<~ Append cur+parseToken' :: forall (cs1 :: CharSet) (cs :: CharSet) a. (IsCharSet cs1, IsCharSet cs) => TokenFilter -> Maybe FileTypeInfo -> Bool -> T.Text -> ST a (S.Seq Token)+parseToken' tf@TokenFilter{..} info strict txt = do+ stateR <- newSTRef StateSpace+ accR <- newSTRef (TokenIdx (-1) (-1))+ tokensR <- newSTRef S.empty - StateBracket -> {-# SCC "StateBracket" #-} do- acc <- readSTRef accR+ iterM txt $ \(# x, cur, delta #) -> do+ state <- readSTRef stateR+ case state of+ StateSpace ->+ if+ | isSpace x -> do accR <<~ Reset+ | inline (isValidChar @cs1 x) -> do stateR <~ StateIdentifier; accR <<~ Start cur delta+ | x == startLiteralMarker -> do stateR <~ StateLiteral; accR <<~ Reset+ | isDigit x -> do stateR <~ StateDigit; accR <<~ Start cur delta+ | isBracket' x -> do stateR <~ StateBracket; accR <<~ Start cur delta+ | otherwise -> do stateR <~ StateOther; accR <<~ Start cur delta+ StateIdentifier ->+ if isValidChar @cs x+ then accR <<~ Append cur delta+ else do+ acc <- readSTRef accR+ tokens <- readSTRef tokensR+ if+ | isSpace x -> do stateR <~ StateSpace; accR <<~ Reset; tokensR <~ (tokens |> buildTokenIf tfIdentifier tfKeyword tfNativeType (mkTokenFromWord info) acc txt)+ | x == startLiteralMarker -> do stateR <~ StateLiteral; accR <<~ Reset; tokensR <~ (tokens |> buildTokenIf tfIdentifier tfKeyword tfNativeType (mkTokenFromWord info) acc txt)+ | isBracket' x -> do stateR <~ StateBracket; accR <<~ Start cur delta; tokensR <~ (tokens |> buildTokenIf tfIdentifier tfKeyword tfNativeType (mkTokenFromWord info) acc txt)+ | otherwise -> do stateR <~ StateOther; accR <<~ Start cur delta; tokensR <~ (tokens |> buildTokenIf tfIdentifier tfKeyword tfNativeType (mkTokenFromWord info) acc txt)+ StateDigit ->+ if isCharNumber x+ then accR <<~ Append cur delta+ else do+ acc <- readSTRef accR+ tokens <- readSTRef tokensR+ if+ | isSpace x -> do stateR <~ StateSpace; accR <<~ Reset; tokensR <~ (tokens |> buildToken tfNumber mkTokenDigit acc txt)+ | x == startLiteralMarker -> do stateR <~ StateLiteral; accR <<~ Reset; tokensR <~ (tokens |> buildToken tfNumber mkTokenDigit acc txt)+ | inline (isValidChar @cs1 x) -> do stateR <~ StateIdentifier; accR <<~ Start cur delta; tokensR <~ (tokens |> buildToken tfNumber mkTokenDigit acc txt)+ | isBracket' x -> do stateR <~ StateBracket; accR <<~ Start cur delta; tokensR <~ (tokens |> buildToken tfNumber mkTokenDigit acc txt)+ | otherwise -> do stateR <~ StateOther; accR <<~ Start cur delta; tokensR <~ (tokens |> buildToken tfNumber mkTokenDigit acc txt)+ StateLiteral ->+ if x == endLiteralMarker+ then do+ acc <- readSTRef accR+ tokens <- readSTRef tokensR+ stateR <~ StateSpace+ accR <<~ Reset+ tokensR <~ (tokens |> buildToken tfString mkTokenString acc txt)+ else do accR <<~ Append cur delta+ StateBracket ->+ do+ acc <- readSTRef accR tokens <- readSTRef tokensR- if | isSpace x -> do stateR <~ StateSpace ; accR <<~ Reset ; tokensR <~ (tokens |> buildToken tfBracket mkTokenBracket acc txt)- | inline isAlpha1 x -> do stateR <~ StateIdentifier ; accR <<~ Start cur ; tokensR <~ (tokens |> buildToken tfBracket mkTokenBracket acc txt)- | isDigit x -> do stateR <~ StateDigit ; accR <<~ Start cur ; tokensR <~ (tokens |> buildToken tfBracket mkTokenBracket acc txt)- | isBracket' x -> do accR <<~ Start cur ; tokensR <~ (tokens |> buildToken tfBracket mkTokenBracket acc txt)- | x == chr 2 -> do stateR <~ StateLiteral ; accR <<~ Reset ; tokensR <~ (tokens |> buildToken tfBracket mkTokenBracket acc txt)- | otherwise -> do stateR <~ StateOther ; accR <<~ Start cur ; tokensR <~ (tokens |> buildToken tfBracket mkTokenBracket acc txt)-- StateOther -> {-# SCC "StateOther" #-} do- acc <- readSTRef accR+ if+ | isSpace x -> do stateR <~ StateSpace; accR <<~ Reset; tokensR <~ (tokens |> buildToken tfBracket mkTokenBracket acc txt)+ | inline (isValidChar @cs1 x) -> do stateR <~ StateIdentifier; accR <<~ Start cur delta; tokensR <~ (tokens |> buildToken tfBracket mkTokenBracket acc txt)+ | isDigit x -> do stateR <~ StateDigit; accR <<~ Start cur delta; tokensR <~ (tokens |> buildToken tfBracket mkTokenBracket acc txt)+ | isBracket' x -> do accR <<~ Start cur delta; tokensR <~ (tokens |> buildToken tfBracket mkTokenBracket acc txt)+ | x == startLiteralMarker -> do stateR <~ StateLiteral; accR <<~ Reset; tokensR <~ (tokens |> buildToken tfBracket mkTokenBracket acc txt)+ | otherwise -> do stateR <~ StateOther; accR <<~ Start cur delta; tokensR <~ (tokens |> buildToken tfBracket mkTokenBracket acc txt)+ StateOther ->+ do+ acc <- readSTRef accR tokens <- readSTRef tokensR- if | isSpace x -> do stateR <~ StateSpace ; accR <<~ Reset ; tokensR <~ (tokens |> buildToken tfOperator mkTokenOperator acc txt)- | inline isAlpha1 x -> do stateR <~ StateIdentifier ; accR <<~ Start cur ; tokensR <~ (tokens |> buildToken tfOperator mkTokenOperator acc txt)- | isDigit x -> if tkString acc txt == "."- then do stateR <~ StateDigit ; accR <<~ Append cur- else do stateR <~ StateDigit ; accR <<~ Start cur ; tokensR <~ (tokens |> buildToken tfOperator mkTokenOperator acc txt)- | isBracket' x -> do stateR <~ StateBracket ; accR <<~ Append cur ; tokensR <~ (tokens |> buildToken tfOperator mkTokenOperator acc txt)- | x == chr 2 -> do stateR <~ StateLiteral ; accR <<~ Reset ; tokensR <~ (tokens |> buildToken tfBracket mkTokenBracket acc txt)- | otherwise -> do accR <<~ Append cur-- curR <~ (cur + 1)-- lastAcc <- readSTRef accR- tokens <- readSTRef tokensR-- if lastAcc.len == 0- then return tokens- else do- state <- readSTRef stateR- cur <- readSTRef curR- return $ tokens |> buildFilteredToken f (mkToken info state) lastAcc txt+ if+ | isSpace x -> do stateR <~ StateSpace; accR <<~ Reset; tokensR <~ (tokens |> buildToken tfOperator mkTokenOperator acc txt)+ | inline (isValidChar @cs1 x) -> do stateR <~ StateIdentifier; accR <<~ Start cur delta; tokensR <~ (tokens |> buildToken tfOperator mkTokenOperator acc txt)+ | isDigit x ->+ if tkString acc txt == "."+ then do stateR <~ StateDigit; accR <<~ Append cur delta+ else do stateR <~ StateDigit; accR <<~ Start cur delta; tokensR <~ (tokens |> buildToken tfOperator mkTokenOperator acc txt)+ | isBracket' x -> do stateR <~ StateBracket; accR <<~ Start cur delta; tokensR <~ (tokens |> buildToken tfOperator mkTokenOperator acc txt)+ | x == startLiteralMarker -> do stateR <~ StateLiteral; accR <<~ Reset; tokensR <~ (tokens |> buildToken tfBracket mkTokenBracket acc txt)+ | isPunctuation x -> do accR <<~ Start cur delta; tokensR <~ (tokens |> buildToken tfOperator mkTokenOperator acc txt)+ | otherwise ->+ if strict+ then do accR <<~ Append cur delta+ else do accR <<~ Start cur delta; tokensR <~ (tokens |> buildToken tfOperator mkTokenOperator acc txt) + lastAcc <- readSTRef accR+ tokens <- readSTRef tokensR+ if lastAcc.len == 0+ then return tokens+ else do+ state <- readSTRef stateR+ return $ tokens |> buildFilteredToken tf (mkToken info state) lastAcc txt -buildFilteredToken :: TokenFilter -> (C.ByteString -> Offset -> Token) -> TokenIdx -> C.ByteString -> Token+buildFilteredToken :: TokenFilter -> (T.Text -> Token) -> TokenIdx -> T.Text -> Token buildFilteredToken tf f (TokenIdx start len) txt =- let t = f (subByteString start len txt) (fromIntegral start)- in if filterToken tf t+ let t = f (textSlice txt start len)+ in if filterToken tf t then t else unspecifiedToken {-# INLINE buildFilteredToken #-} --buildToken :: Bool -> (C.ByteString -> Offset -> Token) -> TokenIdx -> C.ByteString -> Token-buildToken True f (TokenIdx start len) txt = f (subByteString start len txt) (fromIntegral start)-buildToken False f (TokenIdx start len) txt = unspecifiedToken+buildToken :: Bool -> (T.Text -> Token) -> TokenIdx -> T.Text -> Token+buildToken True f (TokenIdx start len) txt = f (textSlice txt start len)+buildToken False _ (TokenIdx _start _len) _txt = unspecifiedToken {-# INLINE buildToken #-} -buildToken_ :: Bool -> Bool -> Bool -> (C.ByteString -> Offset -> Token) -> TokenIdx -> C.ByteString -> Token-buildToken_ i k t f (TokenIdx start len) txt =+buildTokenIf :: Bool -> Bool -> Bool -> (T.Text -> Token) -> TokenIdx -> T.Text -> Token+buildTokenIf i k t f (TokenIdx start len) txt = if i && isTokenIdentifier tok || k && isTokenKeyword tok || t && isTokenNativeType tok- then tok- else unspecifiedToken- where tok = f (subByteString start len txt) (fromIntegral start)---subByteString :: Int -> Int -> C.ByteString -> C.ByteString-subByteString i n = C.take n . C.drop i-{-# INLINE subByteString #-}-+ then tok+ else unspecifiedToken+ where+ tok = f (textSlice txt start len) unspecifiedToken :: Token-unspecifiedToken = Token $ Chunk ChunkUnspec C.empty 0+unspecifiedToken = Token $ Chunk ChunkUnspec T.empty+{-# INLINE unspecifiedToken #-}
src/CGrep/Search.hs view
@@ -1,5 +1,8 @@+{-# LANGUAGE AllowAmbiguousTypes #-}+{-# LANGUAGE DataKinds #-}+{-# LANGUAGE KindSignatures #-} ----- Copyright (c) 2013-2023 Nicola Bonelli <nicola@larthia.com>+-- Copyright (c) 2013-2025 Nicola Bonelli <nicola@larthia.com> -- -- This program is free software; you can redistribute it and/or modify -- it under the terms of the GNU General Public License as published by@@ -14,65 +17,402 @@ -- You should have received a copy of the GNU General Public License -- along with this program; if not, write to the Free Software -- Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.+-- nc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. --+{-# LANGUAGE NumericUnderscores #-}+{-# LANGUAGE QuasiQuotes #-} -module CGrep.Search ( searchStringIndices- , searchStringTaggedIndices- , eligibleForSearch- , TaggedIx(..)- ) where+module CGrep.Search (+ startSearch,+ isRegexp,+)+where -import CGrep.Types ( Text8 )-import Data.Int ( Int64 )-import GHC.Exts ( groupWith )+import CGrep.Common (getTargetContents, getTargetName, takeN)+import CGrep.FileKind (FileKind)+import CGrep.FileType (FileType)+import CGrep.FileTypeMap (+ FileTypeInfo,+ )+import CGrep.FileTypeMapTH (+ fileTypeInfoLookup,+ fileTypeLookup,+ )+import CGrep.Match (+ Match (..),+ prettyFileName,+ putMatches,+ )+import qualified CGrep.Strategy.BoyerMoore as BoyerMoore+import qualified CGrep.Strategy.Levenshtein as Levenshtein+import qualified CGrep.Strategy.Regex as Regex+import qualified CGrep.Strategy.Semantic as Semantic+import qualified CGrep.Strategy.Tokenizer as Tokenizer+import Config (+ Config (+ Config,+ configColorFile,+ configColorMatch,+ configColors,+ configFileLine,+ configFileTypes,+ configPruneDirs+ ),+ ) -import qualified Data.ByteString.Char8 as C-import qualified Data.ByteString.Search as BM-import qualified Data.ByteString.Search.DFA as DFA+import Control.Arrow (Arrow ((&&&)))+import Control.Concurrent (forkIO)+import Control.Concurrent.Async (Async, async, asyncOn, forConcurrently_, wait)+import Control.Exception as E (SomeException, catch)+import Control.Monad (forM, forM_, forever, replicateM_, unless, void, when)+import Control.Monad.Extra (partitionM)+import Control.Monad.IO.Class (MonadIO (liftIO))+import Control.Monad.Trans.Except (runExceptT, throwE)+import Control.Monad.Trans.Reader (+ ReaderT (runReaderT),+ ask,+ )+import qualified Data.Bifunctor+import Data.Functor as F (unzip)+import qualified Data.HashSet as S+import Data.IORef (+ IORef,+ modifyIORef,+ modifyIORef',+ newIORef,+ readIORef,+ )+import Data.List.Split (chunksOf)+import Data.Maybe (catMaybes, fromJust, fromMaybe)+import Data.Tuple.Extra ()+import GHC.Conc (getNumCapabilities)+import Options (Options (..))+import PutMessage (putMessageLn)+import Reader (+ Env (..),+ ReaderIO,+ )+import System.Directory.OsPath (doesDirectoryExist, getDirectoryContents, makeAbsolute)+import System.Environment (lookupEnv)+import System.IO (+ stderr,+ stdin,+ stdout,+ ) -import qualified Data.ByteString.Lazy.Search as LBM-import qualified Data.ByteString.Lazy.Search.DFA as LDFA-import Data.List.Extra ( notNull )+import Control.Applicative ((<|>))+import Control.Concurrent (MVar)+import Control.Concurrent.Classy (newBoundedChan, readBoundedChan)+import Control.Concurrent.Classy.BoundedChan (writeBoundedChan)+import Control.Concurrent.Extra (newMVar)+import Data.Atomics.Counter (incrCounter, newCounter, readCounter)+import Data.List.Extra (notNull)+import qualified Data.Text as T+import qualified Data.Text.Lazy as TL+import qualified Data.Text.Lazy.Builder as TLB+import qualified Data.Text.Lazy.IO as LTIO+import qualified OsPath as OS+import System.Clock+import System.OsPath (OsPath, OsString, osp, takeBaseName, unsafeEncodeUtf, (</>))+import System.OsString as OS (isPrefixOf, isSuffixOf)+import System.Process (runProcess, waitForProcess) +data SearcherKind+ = Levenshtein+ | BoyerMoore+ | Semantic+ | Tokenizer+ | Regex -findIndices :: Text8 -> Text8 -> [Int]-findIndices p = if C.length p <= 3- then DFA.indices p- else BM.indices p-{-# INLINE findIndices #-}+class Searcher (s :: SearcherKind) where+ run :: (MVar () -> Maybe (FileType, FileTypeInfo) -> OsPath -> T.Text -> [T.Text] -> Bool -> ReaderIO [Match]) +instance Searcher 'Levenshtein where+ run = Levenshtein.search -searchStringIndices :: [Text8] -> Text8 -> [[Int64]]-searchStringIndices ps text = ps >>= \p -> [fromIntegral <$> p `findIndices` text]-{-# INLINE searchStringIndices #-}+instance Searcher 'BoyerMoore where+ run = BoyerMoore.search +instance Searcher 'Semantic where+ run = Semantic.search -data TaggedIx a = TaggedIx {- index :: {-# UNPACK #-} !Int- , tags :: [a]-} deriving stock (Show)+instance Searcher 'Tokenizer where+ run = Tokenizer.search -instance Eq (TaggedIx a) where- (TaggedIx i1 _) == (TaggedIx i2 _) = i1 == i2+instance Searcher 'Regex where+ run = Regex.search -instance Ord (TaggedIx a) where- compare (TaggedIx i1 _) (TaggedIx i2 _) = compare i1 i2+getSearcher :: Env -> (MVar () -> Maybe (FileType, FileTypeInfo) -> OsPath -> T.Text -> [T.Text] -> Bool -> ReaderIO [Match])+getSearcher Env{..} = do+ if+ | (not . isRegexp) opt && not (hasTokenizerOpt opt) && not (semantic opt) && edit_dist opt -> run @Levenshtein+ | (not . isRegexp) opt && not (hasTokenizerOpt opt) && not (semantic opt) -> run @BoyerMoore+ | (not . isRegexp) opt && semantic opt -> run @Semantic+ | (not . isRegexp) opt -> run @Tokenizer+ | isRegexp opt -> run @Regex+ | otherwise -> undefined --- >>> searchStringTaggedIndices [("a",2),("b",1),("a",0), ("he", 42)] "aheba"--- [TaggedIx {index = 0, tags = [2,0]},TaggedIx {index = 1, tags = [42]},TaggedIx {index = 3, tags = [1]},TaggedIx {index = 4, tags = [2,0]}]+data RecursiveContext = RecursiveContext+ { rcFileTypes :: [FileType]+ , rcFileKinds :: [FileKind]+ , rcPrunableDirs :: [OsPath]+ , rcParallel :: Bool+ } -searchStringTaggedIndices :: [(Text8, a)] -> Text8 -> [TaggedIx a]-searchStringTaggedIndices ps text =- let res = ps >>= \p -> let pat = fst p- tag = snd p- ids = findIndices pat text- in (\i -> TaggedIx (fromIntegral i) [tag]) <$> ids- in fuseGroup <$> groupWith index res- where {-# INLINE fuseGroup #-}- fuseGroup :: [TaggedIx a] -> TaggedIx a- fuseGroup xs = TaggedIx (index $ head xs) $ concatMap tags xs+withRecursiveContents ::+ RecursiveContext ->+ Options ->+ OsPath ->+ S.HashSet OsPath ->+ ([OsPath] -> IO ()) ->+ IO ()+withRecursiveContents ctx@RecursiveContext{..} opt@Options{..} dir visited action = do+ xs <- getDirectoryContents dir+ (dirs, files) <- partitionM (\p -> doesDirectoryExist (dir </> p)) xs -eligibleForSearch :: [a] -> [[Int64]] -> Bool-eligibleForSearch [_] = all notNull-eligibleForSearch _ = any notNull-{-# INLINE eligibleForSearch #-}+ -- filter the list of files+ let files' :: [OsPath] = (dir </>) <$> filter (\f -> fileFilter opt rcFileTypes rcFileKinds f) files+ let dirs' :: [OsPath] = (dir </>) <$> filter (\d -> not $ isDot d) dirs++ -- run IO action+ mapM_ action (chunksOf 16 files')++ foreach <-+ if rcParallel+ then pure (forConcurrently_ @[])+ else pure forM_++ foreach dirs' $ \dirPath -> do+ unless (isPrunableDir dirPath rcPrunableDirs) $+ -- this is a good directory, unless already visited...+ makeAbsolute dirPath >>= \cpath -> do+ unless (cpath `S.member` visited) $ withRecursiveContents ctx opt dirPath (S.insert cpath visited) action++startSearch :: [OsPath] -> [T.Text] -> [FileType] -> [FileKind] -> Bool -> ReaderIO ()+startSearch paths patterns fTypes fKinds isTermIn = do+ Env{..} <- ask++ startTime <- liftIO $ getTime Monotonic+ totalFiles <- liftIO $ newCounter 0+ totalMatchingFiles <- liftIO $ newCounter 0+ totalMatches <- liftIO $ newCounter 0++ lock <- liftIO $ newMVar ()++ let Config{..} = conf+ Options{..} = opt++ numCaps <- liftIO getNumCapabilities++ let multiplier = 4+ let totalJobs = multiplier * fromMaybe (max (numCaps - 1) 1) jobs++ -- create channels ...+ fileCh <- liftIO $ newBoundedChan 65536++ -- recursively traverse the filesystem ...+ _ <- liftIO . forkIO $ do+ if recursive || follow+ then forM_ (if null paths then [unsafeEncodeUtf "."] else paths) $ \path ->+ doesDirectoryExist path >>= \case+ True ->+ withRecursiveContents+ RecursiveContext+ { rcFileTypes = fTypes+ , rcFileKinds = fKinds+ , rcPrunableDirs = (mkPrunableDirName <$> (unsafeEncodeUtf <$> configPruneDirs) <> (unsafeEncodeUtf <$> prune_dir))+ , rcParallel = True+ }+ opt+ path+ (S.singleton path)+ ( \paths' -> do+ when (notNull paths') $+ -- putMessageLn @T.Text lock stderr $ "Discovered empty directory!"+ incrCounter (length paths') totalFiles >> writeBoundedChan fileCh paths'+ )+ _ -> incrCounter 1 totalFiles >> writeBoundedChan fileCh [path]+ else+ forM_+ ( if null paths && not isTermIn+ then [(unsafeEncodeUtf "", 0 :: Int)]+ else paths `zip` [0 ..]+ )+ (\(p, _idx) -> incrCounter 1 totalFiles >> writeBoundedChan fileCh [p])++ when verbose $+ putMessageLn @T.Text lock stderr $+ "File discovery completed..."++ replicateM_ totalJobs $ writeBoundedChan fileCh []++ -- launch the worker threads...+ matchingFiles :: IORef (S.HashSet (OsPath, Int)) <- liftIO $ newIORef S.empty++ let env = Env conf opt+ runSearch = getSearcher env++ workers <- forM ([0 .. totalJobs - 1] :: [Int]) $ \idx -> do+ let processor = 1 + idx `div` multiplier+ liftIO . asyncOn processor $ void . runExceptT $ do+ asRef <- liftIO $ newIORef ([] :: [Async ()])+ forever $ do+ paths' <- liftIO $ readBoundedChan fileCh+ case paths' of+ [] -> liftIO $ do+ -- when verbose $+ -- putMessageLn @T.Text lock stderr $ "worker_" <> T.pack (show idx) <> "@" <> T.pack (show processor) <> " received termination signal!"+ readIORef asRef >>= mapM_ wait+ fs' -> do+ out <-+ liftIO $+ runReaderT+ ( catMaybes+ <$> forM+ fs'+ ( \f ->+ liftIO $+ E.catch+ ( runReaderT+ ( do+ text <- liftIO $ getTargetContents f+ !matches <- take max_count <$> runSearch lock (fileTypeInfoLookup opt f) (getTargetName f) text patterns strict++ when (vim || editor) $+ liftIO $+ mapM_ (modifyIORef matchingFiles . S.insert . (mFilePath &&& mLineNumb)) matches+ -- putMessage @T.Text lock stderr $ "."+ _ <- liftIO $ incrCounter (length matches) totalMatches+ when (notNull matches) $ void $ liftIO $ incrCounter 1 totalMatchingFiles+ putMatches matches+ )+ env+ )+ ( \e -> do+ let msg = show (e :: SomeException)+ putMessageLn lock stderr (prettyFileName conf opt (getTargetName f) <> ": error: " <> TL.pack (takeN 120 msg))+ return Nothing+ )+ )+ )+ env+ unless (null out) $+ liftIO $+ async+ ( do+ let !dump = TLB.toLazyText (mconcat ((<> TLB.singleton '\n') <$> out))+ LTIO.hPutStr stdout dump+ )+ >>= \a -> modifyIORef' asRef (a :)+ when (null paths') $ do+ when verbose $+ putMessageLn lock stderr $+ "worker_" <> T.pack (show idx) <> "@" <> T.pack (show processor) <> " terminated!"+ throwE ()++ -- wait workers to complete the job+ liftIO $ do+ mapM_ wait workers+ when verbose $ putMessageLn @T.Text lock stderr $ "All workers terminated!"++ endTime <- liftIO $ getTime Monotonic+ when stats $ liftIO $ do+ let diffTime = fromIntegral (toNanoSecs (diffTimeSpec endTime startTime)) / 1_000_000_000 :: Double+ total <- (readCounter totalFiles)+ matches <- (readCounter totalMatches)+ mfiles <- (readCounter totalMatchingFiles)+ putMessageLn @T.Text lock stderr $ "\nTotal matches " <> T.pack (show matches)+ putMessageLn @T.Text lock stderr $ "Matching files " <> T.pack (show mfiles)+ putMessageLn @T.Text lock stderr $ "Total files searched " <> T.pack (show total)+ putMessageLn @T.Text lock stderr $ "Elapsed time " <> T.pack (show diffTime) <> " seconds"++ -- run editor...+ when (vim || editor) $ liftIO $ do+ editor' <-+ if vim+ then return (Just "vim")+ else lookupEnv "EDITOR"++ files <- S.toList <$> readIORef matchingFiles+ let filesUnpacked = Data.Bifunctor.first (T.unpack . OS.toText) <$> files++ let editFiles =+ if fileline || configFileLine+ then fmap (\(a, b) -> a <> ":" <> show b) filesUnpacked+ else fmap fst filesUnpacked++ putStrLn $ "cgrep: open files " <> unwords editFiles <> "..."++ void $+ runProcess+ (fromJust $ editor' <|> Just "vi")+ editFiles+ Nothing+ Nothing+ (Just stdin)+ (Just stdout)+ (Just stderr)+ >>= waitForProcess++fileFilter :: Options -> [FileType] -> [FileKind] -> OsPath -> Bool+fileFilter opt fTypes fKinds filename = (fileFilterTypes typ) && (fileFilterKinds kin)+ where+ (typ, kin) = F.unzip $ fileTypeLookup opt filename+ fileFilterTypes = maybe False (liftA2 (||) (const $ null fTypes) (`elem` fTypes))+ fileFilterKinds = maybe False (liftA2 (||) (const $ null fKinds) (`elem` fKinds))++isNotTestFile :: OsPath -> Bool+isNotTestFile f =+ let fs =+ [ ([osp|_test|] `OS.isSuffixOf`)+ , ([osp|-test|] `OS.isSuffixOf`)+ , ([osp|test-|] `OS.isPrefixOf`)+ , ([osp|test_|] `OS.isPrefixOf`)+ , ([osp|test|] ==)+ ] ::+ [OsString -> Bool]+ basename = takeBaseName f+ in not $ any ($ basename) fs+{-# INLINE isNotTestFile #-}++isDot :: OsPath -> Bool+isDot p = p == [osp|.|] || p == [osp|..|]+{-# INLINE isDot #-}++isPrunableDir :: OsPath -> [OsPath] -> Bool+isPrunableDir dir = any (`OS.isSuffixOf` pdir)+ where+ pdir = mkPrunableDirName dir+{-# INLINE isPrunableDir #-}++mkPrunableDirName :: OsPath -> OsPath+mkPrunableDirName xs+ | unsafeEncodeUtf "/" `OS.isSuffixOf` xs = xs+ | otherwise = xs <> unsafeEncodeUtf "/"+{-# INLINE mkPrunableDirName #-}++-- (.!.) :: V.Vector a -> Int -> a+-- v .!. i = v ! (i `mod` V.length v)+-- {-# INLINE (.!.) #-}++-- hasFileType :: OsPath -> Options -> [FileType] -> Bool+-- hasFileType path opt xs = isJust $ fileTypeLookup opt path >>= (\(typ, _) -> typ `elemIndex` xs)+-- {-# INLINE hasFileType #-}++hasTokenizerOpt :: Options -> Bool+hasTokenizerOpt Options{..} =+ identifier+ || nativeType+ || keyword+ || number+ || string+ || operator++#ifdef ENABLE_PCRE+isRegexp :: Options -> Bool+isRegexp opt = regex_posix opt || regex_pcre opt+#else+isRegexp :: Options -> Bool+isRegexp opt = regex_posix opt+#endif+{-# INLINE isRegexp #-}
+ src/CGrep/Semantic/ContextFilter.hs view
@@ -0,0 +1,386 @@+--+-- Copyright (c) 2013-2025 Nicola Bonelli <nicola@larthia.com>+--+-- This program is free software; you can redistribute it and/or modify+-- it under the terms of the GNU General Public License as published by+-- the Free Software Foundation; either version 2 of the License, or+-- (at your option) any later version.+--+-- This program is distributed in the hope that it will be useful,+-- but WITHOUT ANY WARRANTY; without even the implied warranty of+-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the+-- GNU General Public License for more details.+--+-- You should have received a copy of the GNU General Public License+-- along with this program; if not, write to the Free Software+-- Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.+--+{-# LANGUAGE BangPatterns #-}+{-# LANGUAGE OverloadedStrings #-}+{-# LANGUAGE UnboxedTuples #-}++module CGrep.Semantic.ContextFilter (+ FilterFunction,+ ContextFilter (..),+ isContextFilterAll,+ contextBitCode,+ contextBitComment,+ contextBitLiteral,+ mkContextFilter,+ mkParConfig,+ (~!),+ (~?),+ runContextFilter,+ startLiteralMarker,+ endLiteralMarker,+) where++import CGrep.Boundary (Boundary (..))+import CGrep.Parser.Char (chr, isSpace)+import CGrep.Text (blankByWidth)+import Data.Bits (Bits (complement, (.&.), (.|.)))+import Data.Int (Int32)+import qualified Data.Set as Set+import qualified Data.Text as T+import qualified Data.Vector as V+import Options (Options (..))++startLiteralMarker :: Char+startLiteralMarker = chr 2++endLiteralMarker :: Char+endLiteralMarker = chr 3++type FilterFunction = ContextFilter -> T.Text -> T.Text++data Context = Code | Comment | Literal+ deriving stock (Eq, Show)++newtype ContextBit = ContextBit Int32+ deriving stock (Show)+ deriving newtype (Eq, Bits)++contextBitEmpty :: ContextBit+contextBitEmpty = ContextBit 0++contextBitCode :: ContextBit+contextBitCode = ContextBit 0x1++contextBitComment :: ContextBit+contextBitComment = ContextBit 0x2++contextBitLiteral :: ContextBit+contextBitLiteral = ContextBit 0x4++(~=) :: ContextBit -> Bool -> ContextBit+b ~= True = b+_ ~= False = contextBitEmpty+{-# INLINE (~=) #-}++(~?) :: ContextFilter -> ContextBit -> Bool+f ~? b = (unFilter f .&. b) /= contextBitEmpty+{-# INLINE (~?) #-}++(~!) :: ContextFilter -> ContextBit -> ContextFilter+a ~! b = ContextFilter $ unFilter a .&. complement b+{-# INLINE (~!) #-}++newtype ContextFilter = ContextFilter {unFilter :: ContextBit}+ deriving stock (Show)+ deriving newtype (Eq, Bits)++contextFilterAll :: ContextFilter+contextFilterAll = ContextFilter (contextBitCode .|. contextBitComment .|. contextBitLiteral)+{-# NOINLINE contextFilterAll #-}++isContextFilterAll :: ContextFilter -> Bool+isContextFilterAll f = f == contextFilterAll+{-# INLINE isContextFilterAll #-}++codeFilter :: ContextFilter -> Bool+codeFilter f = (unFilter f .&. contextBitCode) /= contextBitEmpty+{-# INLINE codeFilter #-}++commentFilter :: ContextFilter -> Bool+commentFilter f = (unFilter f .&. contextBitComment) /= contextBitEmpty+{-# INLINE commentFilter #-}++literalFilter :: ContextFilter -> Bool+literalFilter f = (unFilter f .&. contextBitLiteral) /= contextBitEmpty+{-# INLINE literalFilter #-}++data RegionBoundary = RegionBoundary+ { rbBegin :: !(# Char, T.Text #)+ , rbBeginLen :: {-# UNPACK #-} !Int+ , rbEnd :: !(# Char, T.Text #)+ , rbEndLen :: {-# UNPACK #-} !Int+ }++toRegionBoundary :: Boundary -> RegionBoundary+toRegionBoundary Boundary{..} =+ RegionBoundary+ { rbBegin = (# T.head bBegin, T.tail bBegin #)+ , rbBeginLen = bBeginLen+ , rbEnd = (# T.head bEnd, T.tail bEnd #)+ , rbEndLen = bEndLen+ }++data ParConfig = ParConfig+ { commBound :: !(V.Vector RegionBoundary)+ , litrBound :: !(V.Vector RegionBoundary)+ , rawBound :: !(V.Vector RegionBoundary)+ , chrBound :: !(V.Vector RegionBoundary)+ , inits :: !T.Text+ , useMakers :: !Bool+ }++mkParConfig :: [Boundary] -> [Boundary] -> [Boundary] -> [Boundary] -> Bool -> ParConfig+mkParConfig cs ls rs chs ab =+ ParConfig+ { commBound = V.fromList $ toRegionBoundary <$> cs+ , litrBound = V.fromList $ toRegionBoundary <$> ls+ , rawBound = V.fromList $ toRegionBoundary <$> rs+ , chrBound = V.fromList $ toRegionBoundary <$> chs+ , inits =+ T.concat . Set.toList . Set.fromList $+ (safeHead . bBegin <$> cs)+ <> (safeHead . bBegin <$> ls)+ <> (safeHead . bBegin <$> rs)+ <> (safeHead . bBegin <$> chs)+ , useMakers = ab+ }++data ParState = ParState+ { parCtxState :: !ContextState+ , parNextState :: !ContextState+ , parDisplay :: !Bool+ , parSkip :: {-# UNPACK #-} !Int+ , parText :: !T.Text+ }+ deriving stock (Show)++data ContextState+ = CodeState1+ | CodeStateN+ | CommState1 {-# UNPACK #-} !Int+ | CommStateN {-# UNPACK #-} !Int+ | ChrState {-# UNPACK #-} !Int+ | LitrState1 {-# UNPACK #-} !Int+ | LitrStateN {-# UNPACK #-} !Int+ | RawState {-# UNPACK #-} !Int+ deriving stock (Show, Eq, Ord)++mkContextFilter :: Options -> ContextFilter+mkContextFilter Options{..} =+ if not (code || comment || literal)+ then contextFilterAll+ else ContextFilter $ contextBitCode ~= code .|. contextBitComment ~= comment .|. contextBitLiteral ~= literal++getContext :: ContextState -> Context+getContext CodeState1 = Code+getContext CodeStateN = Code+getContext (CommState1 _) = Comment+getContext (CommStateN _) = Comment+getContext (LitrState1 _) = Literal+getContext (LitrStateN _) = Literal+getContext (RawState _) = Literal+getContext (ChrState _) = Literal+{-# INLINE getContext #-}++-- contextFilterFun:+--++runContextFilter :: ParConfig -> ContextFilter -> T.Text -> T.Text+runContextFilter conf@ParConfig{..} f txt+ | useMakers = T.unfoldr contextFilter' initialState+ | otherwise = T.unfoldr contextFilter'' initialState+ where+ !initialState =+ ParState+ { parCtxState = CodeState1+ , parNextState = CodeState1+ , parDisplay = codeFilter f+ , parSkip = 0+ , parText = txt+ }++ contextFilter' :: ParState -> Maybe (Char, ParState)+ contextFilter' !state = case T.uncons (parText state) of+ Nothing -> Nothing+ Just (!x, !xs) ->+ let !nextState = nextContextState conf state x xs f+ !nextState' = nextState{parText = xs}+ !disp = parDisplay nextState+ in if disp+ then+ let !currCtx = getContext (parCtxState state)+ !nextCtx = getContext (parCtxState nextState)+ in case currCtx of+ Code | nextCtx == Literal -> Just (startLiteralMarker, nextState')+ Literal | nextCtx == Code -> Just (endLiteralMarker, nextState')+ _ -> Just (x, nextState')+ else Just (if isSpace x then x else blankByWidth x, nextState')+ {-# INLINE contextFilter' #-}++ contextFilter'' :: ParState -> Maybe (Char, ParState)+ contextFilter'' !state = case T.uncons (parText state) of+ Nothing -> Nothing+ Just (!x, !xs) ->+ let !nextState = nextContextState conf state x xs f+ !nextState' = nextState{parText = xs}+ !shouldDisplay = parDisplay nextState || isSpace x+ in Just (if shouldDisplay then x else blankByWidth x, nextState')+ {-# INLINE contextFilter'' #-}++nextContextState :: ParConfig -> ParState -> Char -> T.Text -> ContextFilter -> ParState+nextContextState !conf !s@ParState{..} !c !cont !f+ | parSkip > 0 =+ let !newSkip = parSkip - 1+ in if newSkip == 0+ then s{parCtxState = parNextState, parSkip = 0}+ else s{parSkip = newSkip}+ | CodeState1 <- parCtxState = handleCodeState True+ | CodeStateN <- parCtxState = handleCodeState False+ | CommState1 n <- parCtxState = handleCommentState n True+ | CommStateN n <- parCtxState = handleCommentState n False+ | LitrState1 n <- parCtxState = handleLiteralState n True+ | LitrStateN n <- parCtxState = handleLiteralState n False+ | ChrState n <- parCtxState = handleCharState n+ | RawState n <- parCtxState = handleRawState n+ where+ !codeDisp = codeFilter f+ !commDisp = commentFilter f+ !litrDisp = literalFilter f++ {-# INLINE transitionWith #-}+ transitionWith !nextSt !disp !skipLen =+ let !skip = skipLen - 1+ in if skip == 0+ then s{parCtxState = nextSt, parNextState = nextSt, parDisplay = disp, parSkip = 0}+ else s{parNextState = nextSt, parDisplay = disp, parSkip = skip}++ {-# INLINE handleCodeState #-}+ handleCodeState !isFirst =+ if c `T.elem` inits conf+ then case findPrefixBoundary c cont (commBound conf) of+ (# i, Just !b #) -> transitionWith (CommState1 i) commDisp (rbBeginLen b)+ _ -> case findPrefixBoundary c cont (litrBound conf) of+ (# i, Just !b #) -> transitionWith (LitrState1 i) codeDisp (rbBeginLen b)+ _ -> case findPrefixBoundary c cont (rawBound conf) of+ (# i, Just !b #) -> transitionWith (RawState i) codeDisp (rbBeginLen b)+ _ -> case findPrefixBoundary' c cont (chrBound conf) of+ (# i, Just !b #) -> transitionWith (ChrState i) codeDisp (rbBeginLen b)+ _ ->+ if isFirst+ then s{parCtxState = CodeStateN, parNextState = CodeStateN, parDisplay = codeDisp, parSkip = 0}+ else s+ else+ if isFirst+ then s{parCtxState = CodeStateN, parNextState = CodeStateN, parDisplay = codeDisp, parSkip = 0}+ else s++ {-# INLINE handleCommentState #-}+ handleCommentState !n !isFirst =+ let !RegionBoundary{..} = V.unsafeIndex (commBound conf) n+ (# ec, es #) = rbEnd+ in if c == ec && es `T.isPrefixOf` cont+ then transitionWith CodeState1 commDisp rbEndLen+ else+ if isFirst+ then s{parCtxState = CommStateN n, parNextState = CommStateN n, parDisplay = commDisp, parSkip = 0}+ else s++ {-# INLINE handleLiteralState #-}+ handleLiteralState !n !isFirst =+ if c == '\\'+ then s{parDisplay = litrDisp, parSkip = 1}+ else+ let !RegionBoundary{..} = V.unsafeIndex (litrBound conf) n+ (# ec, es #) = rbEnd+ in if c == ec && es `T.isPrefixOf` cont+ then+ let !skip = rbEndLen - 1+ in if skip == 0+ then s{parCtxState = CodeState1, parNextState = CodeState1, parDisplay = codeDisp, parSkip = 0}+ else s{parCtxState = CodeState1, parNextState = CodeState1, parDisplay = codeDisp, parSkip = skip}+ else+ if isFirst+ then s{parCtxState = LitrStateN n, parNextState = LitrStateN n, parDisplay = litrDisp, parSkip = 0}+ else s++ {-# INLINE handleCharState #-}+ handleCharState !n =+ if c == '\\'+ then s{parDisplay = litrDisp, parSkip = 1}+ else+ let !RegionBoundary{..} = V.unsafeIndex (chrBound conf) n+ (# ec, es #) = rbEnd+ in if c == ec && es `T.isPrefixOf` cont+ then+ let !skip = rbEndLen - 1+ in if skip == 0+ then s{parCtxState = CodeState1, parNextState = CodeState1, parDisplay = codeDisp, parSkip = 0}+ else s{parCtxState = CodeState1, parNextState = CodeState1, parDisplay = codeDisp, parSkip = skip}+ else s{parDisplay = litrDisp, parSkip = 0}++ {-# INLINE handleRawState #-}+ handleRawState !n =+ let !RegionBoundary{..} = V.unsafeIndex (rawBound conf) n+ (# ec, es #) = rbEnd+ in if c == ec && es `T.isPrefixOf` cont+ then+ let !skip = rbEndLen - 1+ in if skip == 0+ then s{parCtxState = CodeState1, parNextState = CodeState1, parDisplay = codeDisp, parSkip = 0}+ else s{parCtxState = CodeState1, parNextState = CodeState1, parDisplay = codeDisp, parSkip = skip}+ else s{parDisplay = litrDisp, parSkip = 0}+{-# INLINE nextContextState #-}++findPrefixBoundary :: Char -> T.Text -> V.Vector RegionBoundary -> (# Int, Maybe RegionBoundary #)+findPrefixBoundary !x !xs !vb = go 0+ where+ !len = V.length vb+ go !i+ | i >= len = (# 0, Nothing #)+ | otherwise =+ let !RegionBoundary{..} = V.unsafeIndex vb i+ (# !c, !cont #) = rbBegin+ in if c == x+ then+ if cont `T.isPrefixOf` xs+ then (# i, Just (V.unsafeIndex vb i) #)+ else go (i + 1)+ else go (i + 1)+{-# INLINE findPrefixBoundary #-}++findPrefixBoundary' :: Char -> T.Text -> V.Vector RegionBoundary -> (# Int, Maybe RegionBoundary #)+findPrefixBoundary' !x !xs !vb = go 0+ where+ !len = V.length vb+ go !i+ | i >= len = (# 0, Nothing #)+ | otherwise =+ let !RegionBoundary{..} = V.unsafeIndex vb i+ (# !c, !cont #) = rbBegin+ in if c /= x+ then go (i + 1)+ else+ if not (cont `T.isPrefixOf` xs)+ then go (i + 1)+ else case xs of+ (T.uncons -> Just (!y, !ys)) ->+ let !skip = if y == '\\' then 1 else 0+ in case T.drop skip ys of+ (T.uncons -> Just (!z, !zs)) ->+ let (# !e, !es #) = rbEnd+ in if z == e && es `T.isPrefixOf` zs+ then (# i, Just (V.unsafeIndex vb i) #)+ else go (i + 1)+ _ -> go (i + 1)+ _ -> go (i + 1)++safeHead :: T.Text -> T.Text+safeHead txt = case T.uncons txt of+ Just (x, _) -> T.singleton x+ Nothing -> T.empty+{-# INLINE safeHead #-}
+ src/CGrep/Semantic/Tests.hs view
@@ -0,0 +1,1374 @@+{-# LANGUAGE KindSignatures #-}+{-# LANGUAGE DefaultSignatures #-}+{-# LANGUAGE DataKinds #-}+{-# LANGUAGE AllowAmbiguousTypes #-}++module CGrep.Semantic.Tests (+ filterTests,+) where+import CGrep.FileType (FileType (..))+import CGrep.Parser.Token (Token, isTokenOperator, tToken, isTokenBracket, isTokenIdentifier, isTokenKeyword)+import qualified Data.Text as T++class LanguageTestFilter (lang :: FileType) where+ langFilter :: Maybe Bool -> [Token] -> [Token]++-- | Rust-specific instance using the helper functions.+instance LanguageTestFilter 'Rust where+ langFilter Nothing tokens = tokens+ langFilter (Just keepTests) tokens = processOutsideRust keepTests tokens++instance LanguageTestFilter 'Go where+ langFilter Nothing tokens = tokens+ langFilter (Just keepTests) tokens = processOutsideGo keepTests tokens++instance LanguageTestFilter 'Java where+ langFilter Nothing tokens = tokens+ langFilter (Just keepTests) tokens = processOutsideJava keepTests tokens++instance LanguageTestFilter 'Kotlin where+ langFilter Nothing tokens = tokens+ langFilter (Just keepTests) tokens = processOutsideJava keepTests tokens++instance LanguageTestFilter 'C where+ langFilter Nothing tokens = tokens+ langFilter (Just keepTests) tokens = processOutsideC keepTests tokens++instance LanguageTestFilter 'Cpp where+ langFilter Nothing tokens = tokens+ langFilter (Just keepTests) tokens = processOutsideC keepTests tokens++instance LanguageTestFilter 'Python where+ langFilter Nothing tokens = tokens+ langFilter (Just keepTests) tokens = processOutsidePython keepTests tokens++instance LanguageTestFilter 'Zig where+ langFilter Nothing tokens = tokens+ langFilter (Just keepTests) tokens = processOutsideZig keepTests tokens++instance LanguageTestFilter 'Javascript where+ langFilter Nothing tokens = tokens+ langFilter (Just keepTests) tokens = processOutsideJavascript keepTests tokens++instance LanguageTestFilter 'Scala where+ langFilter Nothing tokens = tokens+ langFilter (Just keepTests) tokens = processOutsideScala keepTests tokens++instance LanguageTestFilter 'Haskell where+ langFilter Nothing tokens = tokens+ langFilter (Just keepTests) tokens = processOutsideHaskell keepTests tokens++instance LanguageTestFilter 'Csharp where+ langFilter Nothing tokens = tokens+ langFilter (Just keepTests) tokens = processOutsideCsharp keepTests tokens++instance LanguageTestFilter 'Fsharp where+ langFilter Nothing tokens = tokens+ langFilter (Just keepTests) tokens = processOutsideFsharp keepTests tokens++instance LanguageTestFilter 'Dart where+ langFilter Nothing tokens = tokens+ langFilter (Just keepTests) tokens = processOutsideDart keepTests tokens++instance LanguageTestFilter 'Elixir where+ langFilter Nothing tokens = tokens+ langFilter (Just keepTests) tokens = processOutsideElixir keepTests tokens++instance LanguageTestFilter 'Ruby where+ langFilter Nothing tokens = tokens+ langFilter (Just keepTests) tokens = processOutsideRuby keepTests tokens++instance LanguageTestFilter 'PHP where+ langFilter Nothing tokens = tokens+ langFilter (Just keepTests) tokens = processOutsidePHP keepTests tokens++instance LanguageTestFilter 'Swift where+ langFilter Nothing tokens = tokens+ langFilter (Just keepTests) tokens = processOutsideSwift keepTests tokens++instance LanguageTestFilter 'ObjectiveC where+ langFilter Nothing tokens = tokens+ langFilter (Just keepTests) tokens = processOutsideSwift keepTests tokens -- Reuse Swift implementation++instance LanguageTestFilter 'R where+ langFilter Nothing tokens = tokens+ langFilter (Just keepTests) tokens = processOutsideR keepTests tokens++instance LanguageTestFilter 'Julia where+ langFilter Nothing tokens = tokens+ langFilter (Just keepTests) tokens = processOutsideJulia keepTests tokens++instance LanguageTestFilter 'Perl where+ langFilter Nothing tokens = tokens+ langFilter (Just keepTests) tokens = processOutsidePerl keepTests tokens++instance LanguageTestFilter 'OCaml where+ langFilter Nothing tokens = tokens+ langFilter (Just keepTests) tokens = processOutsideOCaml keepTests tokens++instance LanguageTestFilter 'Erlang where+ langFilter Nothing tokens = tokens+ langFilter (Just keepTests) tokens = processOutsideErlang keepTests tokens++instance LanguageTestFilter 'Nim where+ langFilter Nothing tokens = tokens+ langFilter (Just keepTests) tokens = processOutsideNim keepTests tokens++instance LanguageTestFilter 'Clojure where+ langFilter Nothing tokens = tokens+ langFilter (Just keepTests) tokens = processOutsideClojure keepTests tokens++instance LanguageTestFilter 'D where+ langFilter Nothing tokens = tokens+ langFilter (Just keepTests) tokens = processOutsideD keepTests tokens++-- ------------------------------------------------------------------+-- Main Dispatcher Function (Runtime-to-Type bridge)+-- ------------------------------------------------------------------++filterTests :: FileType -> Maybe Bool -> [Token] -> [Token]+filterTests fileType flag tokens =+ case fileType of+ Rust -> langFilter @'Rust flag tokens+ Go -> langFilter @'Go flag tokens+ Java -> langFilter @'Java flag tokens+ Kotlin -> langFilter @'Kotlin flag tokens+ C -> langFilter @'C flag tokens+ Cpp -> langFilter @'Cpp flag tokens+ Python -> langFilter @'Python flag tokens+ Zig -> langFilter @'Zig flag tokens+ Javascript -> langFilter @'Javascript flag tokens+ Scala -> langFilter @'Scala flag tokens+ Haskell -> langFilter @'Haskell flag tokens+ Csharp -> langFilter @'Csharp flag tokens+ Fsharp -> langFilter @'Fsharp flag tokens+ Dart -> langFilter @'Dart flag tokens+ Elixir -> langFilter @'Elixir flag tokens+ Ruby -> langFilter @'Ruby flag tokens+ PHP -> langFilter @'PHP flag tokens+ Swift -> langFilter @'Swift flag tokens+ ObjectiveC -> langFilter @'ObjectiveC flag tokens+ R -> langFilter @'R flag tokens+ Julia -> langFilter @'Julia flag tokens+ Perl -> langFilter @'Perl flag tokens+ OCaml -> langFilter @'OCaml flag tokens+ Erlang -> langFilter @'Erlang flag tokens+ Nim -> langFilter @'Nim flag tokens+ Clojure -> langFilter @'Clojure flag tokens+ D -> langFilter @'D flag tokens+ _ -> tokens++-- ------------------------------------------------------------------+-- Rust-Specific Implementation Helpers (Moved from old function)+-- ------------------------------------------------------------------++-- | (Rust) Helper: Processes tokens *outside* a test block.+processOutsideRust :: Bool -> [Token] -> [Token]+processOutsideRust _ [] = [] -- End of stream+processOutsideRust keepTests (t1:t2:t3:t4:t5:t6:t7:t8:t9:t10:ts)+ -- Look for the exact sequence: #[cfg(test)] mod <name> {+ | (isTokenOperator t1 && tToken t1 == "#") &&+ (isTokenBracket t2 && tToken t2 == "[") &&+ (isTokenIdentifier t3 && tToken t3 == "cfg") &&+ (isTokenBracket t4 && tToken t4 == "(") &&+ (isTokenIdentifier t5 && tToken t5 == "test") &&+ (isTokenBracket t6 && tToken t6 == ")") &&+ (isTokenBracket t7 && tToken t7 == "]") &&+ (isTokenKeyword t8 && tToken t8 == "mod") &&+ isTokenIdentifier t9 && -- Module name (any identifier)+ (isTokenBracket t10 && tToken t10 == "{")+ =+ -- Found the start of a test block.+ -- Find the matching closing brace.+ let (insideTokens, remainingTokens) = processInsideBraces 1 ts+ in if keepTests+ then -- We want test tokens, so keep the *entire* block+ t1:t2:t3:t4:t5:t6:t7:t8:t9:t10:insideTokens ++ processOutsideRust keepTests remainingTokens+ else -- We don't want test tokens, so discard the entire block+ processOutsideRust keepTests remainingTokens++-- No test block marker found, process the current token+processOutsideRust keepTests (t:ts) =+ if keepTests+ then -- We want test tokens, so discard this "outside" token+ processOutsideRust keepTests ts+ else -- We don't want test tokens, so keep this "outside" token+ t : processOutsideRust keepTests ts++-- ------------------------------------------------------------------+-- Go-Specific Implementation Helpers+-- ------------------------------------------------------------------++-- | (Go) Helper: Processes tokens *outside* a test function.+processOutsideGo :: Bool -> [Token] -> [Token]+processOutsideGo _ [] = [] -- End of stream+processOutsideGo keepTests (t1:t2:ts)+ -- Look for "func Test..."+ | isTokenKeyword t1 && tToken t1 == "func" &&+ isTokenIdentifier t2 && "Test" `T.isPrefixOf` (tToken t2)+ =+ -- Found the start of a test function. Now find its opening brace.+ case findOpeningBrace ts of+ Nothing -> -- Malformed, no '{' found. Treat as non-test code.+ if keepTests then processOutsideGo keepTests (t2:ts) else t1 : processOutsideGo keepTests (t2:ts)+ Just (signatureTokens, tokensAfterBrace) ->+ -- We found the function body's opening brace.+ -- signatureTokens *includes* the opening brace.+ -- tokensAfterBrace starts *after* the opening brace.+ let (bodyTokens, remainingTokens) = processInsideBraces 1 tokensAfterBrace+ in if keepTests+ then -- Keep func + name + signature + body+ t1 : t2 : signatureTokens ++ bodyTokens ++ processOutsideGo keepTests remainingTokens+ else -- Discard the whole function+ processOutsideGo keepTests remainingTokens++-- No test function found, process the current token+processOutsideGo keepTests (t:ts) =+ if keepTests+ then -- We want test tokens, so discard this "outside" token+ processOutsideGo keepTests ts+ else -- We don't want test tokens, so keep this "outside" token+ t : processOutsideGo keepTests ts++-- ------------------------------------------------------------------+-- Java-Specific Implementation Helpers+-- ------------------------------------------------------------------++-- | (Java) Helper: Processes tokens *outside* a test method.+processOutsideJava :: Bool -> [Token] -> [Token]+processOutsideJava _ [] = [] -- End of stream+processOutsideJava keepTests (t1:t2:ts)+ -- Look for "@Test"+ | isTokenOperator t1 && tToken t1 == "@" &&+ isTokenIdentifier t2 && tToken t2 == "Test"+ =+ -- Found @Test annotation. Now find its opening brace.+ case findOpeningBrace ts of+ Nothing -> -- Malformed, no '{' found. Treat as non-test code.+ if keepTests then processOutsideJava keepTests (t2:ts) else t1 : processOutsideJava keepTests (t2:ts)+ Just (signatureTokens, tokensAfterBrace) ->+ -- We found the method body's opening brace.+ -- signatureTokens *includes* the opening brace.+ -- tokensAfterBrace starts *after* the opening brace.+ let (bodyTokens, remainingTokens) = processInsideBraces 1 tokensAfterBrace+ in if keepTests+ then -- Keep @Test + signature + body+ t1 : t2 : signatureTokens ++ bodyTokens ++ processOutsideJava keepTests remainingTokens+ else -- Discard the whole method+ processOutsideJava keepTests remainingTokens++-- No test method found, process the current token+processOutsideJava keepTests (t:ts) =+ if keepTests+ then -- We want test tokens, so discard this "outside" token+ processOutsideJava keepTests ts+ else -- We don't want test tokens, so keep this "outside" token+ t : processOutsideJava keepTests ts++-- ------------------------------------------------------------------+-- C/C++-Specific Implementation Helpers+-- ------------------------------------------------------------------++-- | (C/C++) Helper: Processes tokens *outside* a test function.+-- Recognizes:+-- 1. Functions starting with "test_" (snake_case, common in C)+-- 2. Functions starting with "Test" (PascalCase, common in C++)+-- 3. TEST(...) macro (Google Test)+-- 4. TEST_F(...) macro (Google Test with fixture)+-- 5. TEST_CASE(...) macro (Catch2)+processOutsideC :: Bool -> [Token] -> [Token]+processOutsideC _ [] = [] -- End of stream++-- Pattern 1: TEST( macro (Google Test)+processOutsideC keepTests (t1:t2:ts)+ | isTokenIdentifier t1 && tToken t1 == "TEST" &&+ isTokenBracket t2 && tToken t2 == "("+ =+ -- Found TEST( macro. Find the opening brace of the test body.+ case findOpeningBrace ts of+ Nothing -> -- Malformed, no '{' found. Treat as non-test code.+ if keepTests then processOutsideC keepTests (t2:ts) else t1 : processOutsideC keepTests (t2:ts)+ Just (signatureTokens, tokensAfterBrace) ->+ let (bodyTokens, remainingTokens) = processInsideBraces 1 tokensAfterBrace+ in if keepTests+ then t1 : t2 : signatureTokens ++ bodyTokens ++ processOutsideC keepTests remainingTokens+ else processOutsideC keepTests remainingTokens++-- Pattern 2: TEST_F( macro (Google Test with fixture)+processOutsideC keepTests (t1:t2:ts)+ | isTokenIdentifier t1 && tToken t1 == "TEST_F" &&+ isTokenBracket t2 && tToken t2 == "("+ =+ case findOpeningBrace ts of+ Nothing ->+ if keepTests then processOutsideC keepTests (t2:ts) else t1 : processOutsideC keepTests (t2:ts)+ Just (signatureTokens, tokensAfterBrace) ->+ let (bodyTokens, remainingTokens) = processInsideBraces 1 tokensAfterBrace+ in if keepTests+ then t1 : t2 : signatureTokens ++ bodyTokens ++ processOutsideC keepTests remainingTokens+ else processOutsideC keepTests remainingTokens++-- Pattern 3: TEST_CASE( macro (Catch2)+processOutsideC keepTests (t1:t2:ts)+ | isTokenIdentifier t1 && tToken t1 == "TEST_CASE" &&+ isTokenBracket t2 && tToken t2 == "("+ =+ case findOpeningBrace ts of+ Nothing ->+ if keepTests then processOutsideC keepTests (t2:ts) else t1 : processOutsideC keepTests (t2:ts)+ Just (signatureTokens, tokensAfterBrace) ->+ let (bodyTokens, remainingTokens) = processInsideBraces 1 tokensAfterBrace+ in if keepTests+ then t1 : t2 : signatureTokens ++ bodyTokens ++ processOutsideC keepTests remainingTokens+ else processOutsideC keepTests remainingTokens++-- Pattern 4: Function starting with "test_" or "Test"+processOutsideC keepTests (t1:ts)+ | isTokenIdentifier t1 &&+ (("test_" `T.isPrefixOf` tToken t1) || ("Test" `T.isPrefixOf` tToken t1))+ =+ -- Found a test function. Find the opening brace.+ case findOpeningBrace ts of+ Nothing -> -- Malformed, no '{' found. Treat as non-test code.+ if keepTests then processOutsideC keepTests ts else t1 : processOutsideC keepTests ts+ Just (signatureTokens, tokensAfterBrace) ->+ let (bodyTokens, remainingTokens) = processInsideBraces 1 tokensAfterBrace+ in if keepTests+ then t1 : signatureTokens ++ bodyTokens ++ processOutsideC keepTests remainingTokens+ else processOutsideC keepTests remainingTokens++-- No test found, process the current token+processOutsideC keepTests (t:ts) =+ if keepTests+ then -- We want test tokens, so discard this "outside" token+ processOutsideC keepTests ts+ else -- We don't want test tokens, so keep this "outside" token+ t : processOutsideC keepTests ts++-- ------------------------------------------------------------------+-- Python-Specific Implementation Helpers+-- ------------------------------------------------------------------++-- | (Python) Helper: Processes tokens *outside* a test function.+-- Recognizes:+-- 1. Functions starting with "test_" (pytest, unittest convention)+-- 2. Classes starting with "Test" (pytest convention)+-- 3. Decorators @pytest, @unittest, @pytest.mark.* (common test decorators)+--+processOutsidePython :: Bool -> [Token] -> [Token]+processOutsidePython _ [] = [] -- End of stream++-- Pattern 1: @pytest or @unittest decorator+processOutsidePython keepTests (t1:t2:ts)+ | isTokenOperator t1 && tToken t1 == "@" &&+ isTokenIdentifier t2 && (tToken t2 == "pytest" || tToken t2 == "unittest")+ =+ -- Found a test decorator. Collect decorator line and the function/class.+ let (decoratorTokens, afterDecorator) = collectDecoratorAndFunction ts+ in if keepTests+ then t1 : t2 : decoratorTokens ++ processOutsidePython keepTests afterDecorator+ else processOutsidePython keepTests afterDecorator++-- Pattern 2: @pytest.mark.* decorator+processOutsidePython keepTests (t1:t2:t3:t4:t5:ts)+ | isTokenOperator t1 && tToken t1 == "@" &&+ isTokenIdentifier t2 && tToken t2 == "pytest" &&+ isTokenOperator t3 && tToken t3 == "." &&+ isTokenIdentifier t4 && tToken t4 == "mark" &&+ isTokenOperator t5 && tToken t5 == "."+ =+ -- Found @pytest.mark.* decorator+ let (decoratorTokens, afterDecorator) = collectDecoratorAndFunction ts+ in if keepTests+ then t1 : t2 : t3 : t4 : t5 : decoratorTokens ++ processOutsidePython keepTests afterDecorator+ else processOutsidePython keepTests afterDecorator++-- Pattern 3: def test_*+processOutsidePython keepTests (t1:t2:ts)+ | isTokenKeyword t1 && tToken t1 == "def" &&+ isTokenIdentifier t2 && "test_" `T.isPrefixOf` tToken t2+ =+ -- Found a test function. Collect tokens until next def/class.+ let (testTokens, remainingTokens) = collectUntilNextDefOrClass ts+ in if keepTests+ then t1 : t2 : testTokens ++ processOutsidePython keepTests remainingTokens+ else processOutsidePython keepTests remainingTokens++-- Pattern 4: class Test*+processOutsidePython keepTests (t1:t2:ts)+ | isTokenKeyword t1 && tToken t1 == "class" &&+ isTokenIdentifier t2 && "Test" `T.isPrefixOf` tToken t2+ =+ -- Found a test class. Collect tokens until next class/def at same level.+ let (testTokens, remainingTokens) = collectUntilNextDefOrClass ts+ in if keepTests+ then t1 : t2 : testTokens ++ processOutsidePython keepTests remainingTokens+ else processOutsidePython keepTests remainingTokens++-- No test found, process the current token+processOutsidePython keepTests (t:ts) =+ if keepTests+ then -- We want test tokens, so discard this "outside" token+ processOutsidePython keepTests ts+ else -- We don't want test tokens, so keep this "outside" token+ t : processOutsidePython keepTests ts++-- | Helper: Collect decorator tokens and the following function/class definition.+-- This handles decorators like @pytest.fixture, @unittest.skip, etc.+collectDecoratorAndFunction :: [Token] -> ([Token], [Token])+collectDecoratorAndFunction ts =+ -- First, collect tokens until we hit 'def' or 'class'+ let (beforeDef, fromDef) = collectUntilDefOrClass ts+ in case fromDef of+ [] -> (beforeDef, []) -- No def/class found+ _ ->+ -- Now collect the actual function/class body+ let (bodyTokens, remaining) = collectUntilNextDefOrClass (drop 2 fromDef) -- skip 'def'/'class' and name+ in (beforeDef ++ take 2 fromDef ++ bodyTokens, remaining)++-- | Helper: Collect tokens until we find 'def' or 'class' keyword.+collectUntilDefOrClass :: [Token] -> ([Token], [Token])+collectUntilDefOrClass [] = ([], [])+collectUntilDefOrClass (t:ts)+ | isTokenKeyword t && (tToken t == "def" || tToken t == "class") =+ ([], t:ts) -- Found definition, stop here+ | otherwise =+ let (collected, remaining) = collectUntilDefOrClass ts+ in (t : collected, remaining)++-- | Helper: Collect tokens until we find 'def' or 'class' keyword.+-- This is a simplified heuristic for Python's indentation-based blocks.+collectUntilNextDefOrClass :: [Token] -> ([Token], [Token])+collectUntilNextDefOrClass [] = ([], [])+collectUntilNextDefOrClass (t:ts)+ | isTokenKeyword t && (tToken t == "def" || tToken t == "class") =+ ([], t:ts) -- Found next definition, stop here+ | otherwise =+ let (collected, remaining) = collectUntilNextDefOrClass ts+ in (t : collected, remaining)++-- ------------------------------------------------------------------+-- Zig-Specific Implementation Helpers+-- ------------------------------------------------------------------++-- | (Zig) Helper: Processes tokens *outside* a test block.+-- Recognizes:+-- 1. test "description" { ... } - Zig's built-in test syntax+-- +-- Zig has a very clean built-in test syntax that's easy to recognize.+processOutsideZig :: Bool -> [Token] -> [Token]+processOutsideZig _ [] = [] -- End of stream++-- Pattern: test "..." {+processOutsideZig keepTests (t1:ts)+ | isTokenKeyword t1 && tToken t1 == "test"+ =+ -- Found a test block. Find the opening brace.+ case findOpeningBrace ts of+ Nothing -> -- Malformed, no '{' found. Treat as non-test code.+ if keepTests then processOutsideZig keepTests ts else t1 : processOutsideZig keepTests ts+ Just (signatureTokens, tokensAfterBrace) ->+ -- signatureTokens includes the test name/description and the opening brace+ -- tokensAfterBrace starts after the opening brace+ let (bodyTokens, remainingTokens) = processInsideBraces 1 tokensAfterBrace+ in if keepTests+ then -- Keep test keyword + description + body+ t1 : signatureTokens ++ bodyTokens ++ processOutsideZig keepTests remainingTokens+ else -- Discard the whole test block+ processOutsideZig keepTests remainingTokens++-- No test found, process the current token+processOutsideZig keepTests (t:ts) =+ if keepTests+ then -- We want test tokens, so discard this "outside" token+ processOutsideZig keepTests ts+ else -- We don't want test tokens, so keep this "outside" token+ t : processOutsideZig keepTests ts++-- ------------------------------------------------------------------+-- Javascript-Specific Implementation Helpers+-- ------------------------------------------------------------------++-- | (Javascript) Helper: Processes tokens *outside* a test block.+-- Recognizes:+-- 1. describe('...', function() { ... }) - Mocha/Jasmine/Jest+-- 2. describe('...', () => { ... }) - Modern syntax+-- 3. it('...', function() { ... }) - Test cases+-- 4. it('...', () => { ... }) - Modern syntax+-- 5. test('...', function() { ... }) - Jest+-- 6. test('...', () => { ... }) - Jest modern syntax+-- 7. context('...', ...) - Mocha/Jasmine+-- +-- All these frameworks use similar patterns with function calls.+processOutsideJavascript :: Bool -> [Token] -> [Token]+processOutsideJavascript _ [] = [] -- End of stream++-- Pattern 1: describe( or it( or test( or context(+processOutsideJavascript keepTests (t1:t2:ts)+ | isTokenIdentifier t1 && + (tToken t1 == "describe" || tToken t1 == "it" || tToken t1 == "test" || tToken t1 == "context") &&+ isTokenBracket t2 && tToken t2 == "("+ =+ -- Found a test function call. Find the opening brace of the test body.+ case findOpeningBrace ts of+ Nothing -> -- Malformed, no '{' found. Treat as non-test code.+ if keepTests then processOutsideJavascript keepTests (t2:ts) else t1 : processOutsideJavascript keepTests (t2:ts)+ Just (signatureTokens, tokensAfterBrace) ->+ -- signatureTokens includes everything up to and including the opening brace+ -- tokensAfterBrace starts after the opening brace+ let (bodyTokens, remainingTokens) = processInsideBraces 1 tokensAfterBrace+ in if keepTests+ then -- Keep test function call + signature + body+ t1 : t2 : signatureTokens ++ bodyTokens ++ processOutsideJavascript keepTests remainingTokens+ else -- Discard the whole test block+ processOutsideJavascript keepTests remainingTokens++-- No test found, process the current token+processOutsideJavascript keepTests (t:ts) =+ if keepTests+ then -- We want test tokens, so discard this "outside" token+ processOutsideJavascript keepTests ts+ else -- We don't want test tokens, so keep this "outside" token+ t : processOutsideJavascript keepTests ts++-- ------------------------------------------------------------------+-- Scala-Specific Implementation Helpers+-- ------------------------------------------------------------------++-- | (Scala) Helper: Processes tokens *outside* a test block.+-- Recognizes:+-- 1. test("...") { ... } - ScalaTest FunSuite, MUnit+-- 2. it("...") { ... } - ScalaTest FunSpec+-- 3. describe("...") { ... } - ScalaTest FunSpec+-- 4. scenario("...") { ... } - ScalaTest FeatureSpec+-- 5. feature("...") { ... } - ScalaTest FeatureSpec+-- +-- ScalaTest and MUnit are the most popular testing frameworks for Scala.+processOutsideScala :: Bool -> [Token] -> [Token]+processOutsideScala _ [] = [] -- End of stream++-- Pattern: test( or it( or describe( or scenario( or feature(+processOutsideScala keepTests (t1:t2:ts)+ | isTokenIdentifier t1 && + (tToken t1 == "test" || tToken t1 == "it" || tToken t1 == "describe" || + tToken t1 == "scenario" || tToken t1 == "feature") &&+ isTokenBracket t2 && tToken t2 == "("+ =+ -- Found a test function call. Find the opening brace of the test body.+ case findOpeningBrace ts of+ Nothing -> -- Malformed, no '{' found. Treat as non-test code.+ if keepTests then processOutsideScala keepTests (t2:ts) else t1 : processOutsideScala keepTests (t2:ts)+ Just (signatureTokens, tokensAfterBrace) ->+ -- signatureTokens includes everything up to and including the opening brace+ -- tokensAfterBrace starts after the opening brace+ let (bodyTokens, remainingTokens) = processInsideBraces 1 tokensAfterBrace+ in if keepTests+ then -- Keep test function call + signature + body+ t1 : t2 : signatureTokens ++ bodyTokens ++ processOutsideScala keepTests remainingTokens+ else -- Discard the whole test block+ processOutsideScala keepTests remainingTokens++-- No test found, process the current token+processOutsideScala keepTests (t:ts) =+ if keepTests+ then -- We want test tokens, so discard this "outside" token+ processOutsideScala keepTests ts+ else -- We don't want test tokens, so keep this "outside" token+ t : processOutsideScala keepTests ts++-- ------------------------------------------------------------------+-- Haskell-Specific Implementation Helpers+-- ------------------------------------------------------------------++-- | (Haskell) Helper: Processes tokens *outside* a test block.+-- Recognizes:+-- 1. describe "..." - HSpec+-- 2. it "..." - HSpec+-- 3. context "..." - HSpec (alias for describe)+-- 4. testCase "..." - Tasty/HUnit+-- 5. testGroup "..." - Tasty+-- 6. testProperty "..." - Tasty/QuickCheck+-- 7. prop_* functions - QuickCheck convention+-- +-- Haskell tests often use do-notation or $ operator, so we look for+-- these patterns and collect tokens until the next top-level definition.+processOutsideHaskell :: Bool -> [Token] -> [Token]+processOutsideHaskell _ [] = [] -- End of stream++-- Pattern 1: describe/it/context/testCase/testGroup/testProperty followed by string+processOutsideHaskell keepTests (t1:ts)+ | isTokenIdentifier t1 && + (tToken t1 == "describe" || tToken t1 == "it" || tToken t1 == "context" ||+ tToken t1 == "testCase" || tToken t1 == "testGroup" || tToken t1 == "testProperty")+ =+ -- Found a test block. Try to find opening brace, otherwise collect until next definition.+ case findOpeningBrace ts of+ Just (signatureTokens, tokensAfterBrace) ->+ -- Found braces, use them+ let (bodyTokens, remainingTokens) = processInsideBraces 1 tokensAfterBrace+ in if keepTests+ then t1 : signatureTokens ++ bodyTokens ++ processOutsideHaskell keepTests remainingTokens+ else processOutsideHaskell keepTests remainingTokens+ Nothing ->+ -- No braces, collect until next top-level definition+ let (testTokens, remainingTokens) = collectUntilNextHaskellDef ts+ in if keepTests+ then t1 : testTokens ++ processOutsideHaskell keepTests remainingTokens+ else processOutsideHaskell keepTests remainingTokens++-- Pattern 2: prop_* function (QuickCheck convention)+processOutsideHaskell keepTests (t1:ts)+ | isTokenIdentifier t1 && "prop_" `T.isPrefixOf` tToken t1+ =+ -- Found a property test function. Collect until next definition.+ let (propTokens, remainingTokens) = collectUntilNextHaskellDef ts+ in if keepTests+ then t1 : propTokens ++ processOutsideHaskell keepTests remainingTokens+ else processOutsideHaskell keepTests remainingTokens++-- No test found, process the current token+processOutsideHaskell keepTests (t:ts) =+ if keepTests+ then -- We want test tokens, so discard this "outside" token+ processOutsideHaskell keepTests ts+ else -- We don't want test tokens, so keep this "outside" token+ t : processOutsideHaskell keepTests ts++-- | Helper: Collect tokens until we find the next top-level Haskell definition.+-- This looks for common patterns that indicate a new definition:+-- - describe, it, context, testCase, testGroup, testProperty (test keywords)+-- - main (main function)+-- - Keywords that start definitions at module level+collectUntilNextHaskellDef :: [Token] -> ([Token], [Token])+collectUntilNextHaskellDef [] = ([], [])+collectUntilNextHaskellDef (t:ts)+ -- Found another test keyword or common top-level definition+ | isTokenIdentifier t && + (tToken t == "describe" || tToken t == "it" || tToken t == "context" ||+ tToken t == "testCase" || tToken t == "testGroup" || tToken t == "testProperty" ||+ tToken t == "main" || "prop_" `T.isPrefixOf` tToken t) =+ ([], t:ts) -- Found next definition, stop here+ -- Found a type signature (identifier followed by ::)+ | isTokenIdentifier t =+ case ts of+ (t2:_) | isTokenOperator t2 && tToken t2 == "::" ->+ ([], t:ts) -- Likely a new function definition+ _ ->+ let (collected, remaining) = collectUntilNextHaskellDef ts+ in (t : collected, remaining)+ | otherwise =+ let (collected, remaining) = collectUntilNextHaskellDef ts+ in (t : collected, remaining)++-- ------------------------------------------------------------------+-- C#-Specific Implementation Helpers+-- ------------------------------------------------------------------++-- | (C#) Helper: Processes tokens *outside* a test method.+-- Recognizes:+-- 1. [Test] - NUnit+-- 2. [TestFixture] - NUnit+-- 3. [Fact] - xUnit+-- 4. [Theory] - xUnit+-- 5. [TestMethod] - MSTest+-- 6. [TestClass] - MSTest+-- +-- C# uses attributes (annotations) similar to Java.+processOutsideCsharp :: Bool -> [Token] -> [Token]+processOutsideCsharp _ [] = [] -- End of stream++-- Pattern: [Test] or [Fact] or [Theory] or [TestMethod] etc.+processOutsideCsharp keepTests (t1:t2:t3:ts)+ | isTokenBracket t1 && tToken t1 == "[" &&+ isTokenIdentifier t2 && + (tToken t2 == "Test" || tToken t2 == "TestFixture" || tToken t2 == "Fact" || + tToken t2 == "Theory" || tToken t2 == "TestMethod" || tToken t2 == "TestClass") &&+ isTokenBracket t3 && tToken t3 == "]"+ =+ -- Found a test attribute. Find the opening brace of the method/class.+ case findOpeningBrace ts of+ Nothing -> -- Malformed, no '{' found. Treat as non-test code.+ if keepTests then processOutsideCsharp keepTests (t3:ts) else t1 : processOutsideCsharp keepTests (t2:t3:ts)+ Just (signatureTokens, tokensAfterBrace) ->+ let (bodyTokens, remainingTokens) = processInsideBraces 1 tokensAfterBrace+ in if keepTests+ then t1 : t2 : t3 : signatureTokens ++ bodyTokens ++ processOutsideCsharp keepTests remainingTokens+ else processOutsideCsharp keepTests remainingTokens++-- No test found, process the current token+processOutsideCsharp keepTests (t:ts) =+ if keepTests+ then processOutsideCsharp keepTests ts+ else t : processOutsideCsharp keepTests ts++-- ------------------------------------------------------------------+-- F#-Specific Implementation Helpers+-- ------------------------------------------------------------------++-- | (F#) Helper: Processes tokens *outside* a test function.+-- Recognizes:+-- 1. [<Test>] - NUnit/xUnit (F# attribute syntax)+-- 2. [<Fact>] - xUnit+-- 3. [<Theory>] - xUnit+-- 4. testCase "..." - Expecto+-- 5. testList "..." - Expecto+-- 6. test "..." - Expecto+-- +-- F# uses [< >] syntax for attributes and Expecto functions.+processOutsideFsharp :: Bool -> [Token] -> [Token]+processOutsideFsharp _ [] = [] -- End of stream++-- Pattern 1: [<Test>] or [<Fact>] or [<Theory>]+processOutsideFsharp keepTests (t1:t2:t3:t4:t5:ts)+ | isTokenBracket t1 && tToken t1 == "[" &&+ isTokenOperator t2 && tToken t2 == "<" &&+ isTokenIdentifier t3 && + (tToken t3 == "Test" || tToken t3 == "Fact" || tToken t3 == "Theory") &&+ isTokenOperator t4 && tToken t4 == ">" &&+ isTokenBracket t5 && tToken t5 == "]"+ =+ -- Found F# test attribute. Find the function body.+ case findOpeningBrace ts of+ Just (signatureTokens, tokensAfterBrace) ->+ let (bodyTokens, remainingTokens) = processInsideBraces 1 tokensAfterBrace+ in if keepTests+ then t1 : t2 : t3 : t4 : t5 : signatureTokens ++ bodyTokens ++ processOutsideFsharp keepTests remainingTokens+ else processOutsideFsharp keepTests remainingTokens+ Nothing ->+ -- No braces, collect until next definition+ let (testTokens, remainingTokens) = collectUntilNextHaskellDef ts+ in if keepTests+ then t1 : t2 : t3 : t4 : t5 : testTokens ++ processOutsideFsharp keepTests remainingTokens+ else processOutsideFsharp keepTests remainingTokens++-- Pattern 2: testCase or testList or test (Expecto)+processOutsideFsharp keepTests (t1:ts)+ | isTokenIdentifier t1 && + (tToken t1 == "testCase" || tToken t1 == "testList" || tToken t1 == "test")+ =+ case findOpeningBrace ts of+ Just (signatureTokens, tokensAfterBrace) ->+ let (bodyTokens, remainingTokens) = processInsideBraces 1 tokensAfterBrace+ in if keepTests+ then t1 : signatureTokens ++ bodyTokens ++ processOutsideFsharp keepTests remainingTokens+ else processOutsideFsharp keepTests remainingTokens+ Nothing ->+ let (testTokens, remainingTokens) = collectUntilNextHaskellDef ts+ in if keepTests+ then t1 : testTokens ++ processOutsideFsharp keepTests remainingTokens+ else processOutsideFsharp keepTests remainingTokens++-- No test found, process the current token+processOutsideFsharp keepTests (t:ts) =+ if keepTests+ then processOutsideFsharp keepTests ts+ else t : processOutsideFsharp keepTests ts++-- ------------------------------------------------------------------+-- Dart-Specific Implementation Helpers+-- ------------------------------------------------------------------++-- | (Dart) Helper: Processes tokens *outside* a test block.+-- Recognizes:+-- 1. test('...') or test("...") - Dart test package+-- 2. group('...') or group("...") - Test group+-- 3. testWidgets('...') - Flutter widget tests+-- +-- Similar to JavaScript but with Dart-specific functions.+processOutsideDart :: Bool -> [Token] -> [Token]+processOutsideDart _ [] = [] -- End of stream++-- Pattern: test( or group( or testWidgets(+processOutsideDart keepTests (t1:t2:ts)+ | isTokenIdentifier t1 && + (tToken t1 == "test" || tToken t1 == "group" || tToken t1 == "testWidgets") &&+ isTokenBracket t2 && tToken t2 == "("+ =+ case findOpeningBrace ts of+ Nothing ->+ if keepTests then processOutsideDart keepTests (t2:ts) else t1 : processOutsideDart keepTests (t2:ts)+ Just (signatureTokens, tokensAfterBrace) ->+ let (bodyTokens, remainingTokens) = processInsideBraces 1 tokensAfterBrace+ in if keepTests+ then t1 : t2 : signatureTokens ++ bodyTokens ++ processOutsideDart keepTests remainingTokens+ else processOutsideDart keepTests remainingTokens++-- No test found, process the current token+processOutsideDart keepTests (t:ts) =+ if keepTests+ then processOutsideDart keepTests ts+ else t : processOutsideDart keepTests ts++-- ------------------------------------------------------------------+-- Elixir-Specific Implementation Helpers+-- ------------------------------------------------------------------++-- | (Elixir) Helper: Processes tokens *outside* a test block.+-- Recognizes:+-- 1. test "..." - ExUnit test+-- 2. describe "..." - ExUnit describe block+-- 3. defmodule *Test - Test module+-- +-- Elixir uses ExUnit with do...end blocks.+processOutsideElixir :: Bool -> [Token] -> [Token]+processOutsideElixir _ [] = [] -- End of stream++-- Pattern 1: test or describe followed by string+processOutsideElixir keepTests (t1:ts)+ | isTokenIdentifier t1 && (tToken t1 == "test" || tToken t1 == "describe")+ =+ -- Collect until we find 'end' keyword or next test/describe+ let (testTokens, remainingTokens) = collectUntilElixirEnd ts+ in if keepTests+ then t1 : testTokens ++ processOutsideElixir keepTests remainingTokens+ else processOutsideElixir keepTests remainingTokens++-- Pattern 2: defmodule *Test+processOutsideElixir keepTests (t1:t2:ts)+ | isTokenKeyword t1 && tToken t1 == "defmodule" &&+ isTokenIdentifier t2 && "Test" `T.isSuffixOf` tToken t2+ =+ -- Collect test module until 'end'+ let (moduleTokens, remainingTokens) = collectUntilElixirEnd ts+ in if keepTests+ then t1 : t2 : moduleTokens ++ processOutsideElixir keepTests remainingTokens+ else processOutsideElixir keepTests remainingTokens++-- No test found, process the current token+processOutsideElixir keepTests (t:ts) =+ if keepTests+ then processOutsideElixir keepTests ts+ else t : processOutsideElixir keepTests ts++-- | Helper: Collect tokens until we find 'end' keyword in Elixir.+-- This handles do...end blocks.+collectUntilElixirEnd :: [Token] -> ([Token], [Token])+collectUntilElixirEnd = go 0+ where+ go :: Int -> [Token] -> ([Token], [Token])+ go _ [] = ([], [])+ go depth (t:ts)+ -- Found 'do' - increase depth+ | isTokenKeyword t && tToken t == "do" =+ let (collected, remaining) = go (depth + 1) ts+ in (t : collected, remaining)+ -- Found 'end' - decrease depth or finish+ | isTokenKeyword t && tToken t == "end" =+ if depth <= 1+ then ([t], ts) -- Include final 'end' and finish+ else let (collected, remaining) = go (depth - 1) ts+ in (t : collected, remaining)+ -- Other tokens+ | otherwise =+ let (collected, remaining) = go depth ts+ in (t : collected, remaining)++-- ------------------------------------------------------------------+-- Ruby-Specific Implementation Helpers+-- ------------------------------------------------------------------++-- | (Ruby) Helper: Processes tokens *outside* a test block.+-- Recognizes:+-- 1. describe "..." - RSpec+-- 2. context "..." - RSpec (alias for describe)+-- 3. it "..." - RSpec+-- 4. def test_* - Minitest+-- 5. class Test* - Minitest+-- +-- Ruby uses RSpec (BDD) and Minitest frameworks.+processOutsideRuby :: Bool -> [Token] -> [Token]+processOutsideRuby _ [] = [] -- End of stream++-- Pattern 1: describe/context/it (RSpec)+processOutsideRuby keepTests (t1:ts)+ | isTokenIdentifier t1 && + (tToken t1 == "describe" || tToken t1 == "context" || tToken t1 == "it")+ =+ -- Collect until 'end' keyword+ let (testTokens, remainingTokens) = collectUntilElixirEnd ts -- Reuse Elixir helper+ in if keepTests+ then t1 : testTokens ++ processOutsideRuby keepTests remainingTokens+ else processOutsideRuby keepTests remainingTokens++-- Pattern 2: def test_* (Minitest)+processOutsideRuby keepTests (t1:t2:ts)+ | isTokenKeyword t1 && tToken t1 == "def" &&+ isTokenIdentifier t2 && "test_" `T.isPrefixOf` tToken t2+ =+ let (testTokens, remainingTokens) = collectUntilElixirEnd ts+ in if keepTests+ then t1 : t2 : testTokens ++ processOutsideRuby keepTests remainingTokens+ else processOutsideRuby keepTests remainingTokens++-- Pattern 3: class Test* (Minitest)+processOutsideRuby keepTests (t1:t2:ts)+ | isTokenKeyword t1 && tToken t1 == "class" &&+ isTokenIdentifier t2 && "Test" `T.isPrefixOf` tToken t2+ =+ let (testTokens, remainingTokens) = collectUntilElixirEnd ts+ in if keepTests+ then t1 : t2 : testTokens ++ processOutsideRuby keepTests remainingTokens+ else processOutsideRuby keepTests remainingTokens++-- No test found, process the current token+processOutsideRuby keepTests (t:ts) =+ if keepTests+ then processOutsideRuby keepTests ts+ else t : processOutsideRuby keepTests ts++-- ------------------------------------------------------------------+-- PHP-Specific Implementation Helpers+-- ------------------------------------------------------------------++-- | (PHP) Helper: Processes tokens *outside* a test method.+-- Recognizes:+-- 1. @test annotation in docblock+-- 2. test* method naming+-- 3. class *Test+-- +-- PHP uses PHPUnit framework.+processOutsidePHP :: Bool -> [Token] -> [Token]+processOutsidePHP _ [] = [] -- End of stream++-- Pattern 1: @test annotation (in comment/docblock)+processOutsidePHP keepTests (t1:t2:ts)+ | isTokenOperator t1 && tToken t1 == "@" &&+ isTokenIdentifier t2 && tToken t2 == "test"+ =+ case findOpeningBrace ts of+ Nothing ->+ if keepTests then processOutsidePHP keepTests (t2:ts) else t1 : processOutsidePHP keepTests (t2:ts)+ Just (signatureTokens, tokensAfterBrace) ->+ let (bodyTokens, remainingTokens) = processInsideBraces 1 tokensAfterBrace+ in if keepTests+ then t1 : t2 : signatureTokens ++ bodyTokens ++ processOutsidePHP keepTests remainingTokens+ else processOutsidePHP keepTests remainingTokens++-- Pattern 2: function/method starting with test+processOutsidePHP keepTests (t1:t2:ts)+ | isTokenKeyword t1 && (tToken t1 == "function" || tToken t1 == "public" || tToken t1 == "protected") &&+ isTokenIdentifier t2 && "test" `T.isPrefixOf` tToken t2+ =+ case findOpeningBrace ts of+ Nothing ->+ if keepTests then processOutsidePHP keepTests (t2:ts) else t1 : processOutsidePHP keepTests (t2:ts)+ Just (signatureTokens, tokensAfterBrace) ->+ let (bodyTokens, remainingTokens) = processInsideBraces 1 tokensAfterBrace+ in if keepTests+ then t1 : t2 : signatureTokens ++ bodyTokens ++ processOutsidePHP keepTests remainingTokens+ else processOutsidePHP keepTests remainingTokens++-- Pattern 3: class *Test+processOutsidePHP keepTests (t1:t2:ts)+ | isTokenKeyword t1 && tToken t1 == "class" &&+ isTokenIdentifier t2 && "Test" `T.isSuffixOf` tToken t2+ =+ case findOpeningBrace ts of+ Nothing ->+ if keepTests then processOutsidePHP keepTests (t2:ts) else t1 : processOutsidePHP keepTests (t2:ts)+ Just (signatureTokens, tokensAfterBrace) ->+ let (bodyTokens, remainingTokens) = processInsideBraces 1 tokensAfterBrace+ in if keepTests+ then t1 : t2 : signatureTokens ++ bodyTokens ++ processOutsidePHP keepTests remainingTokens+ else processOutsidePHP keepTests remainingTokens++-- No test found, process the current token+processOutsidePHP keepTests (t:ts) =+ if keepTests+ then processOutsidePHP keepTests ts+ else t : processOutsidePHP keepTests ts++-- ------------------------------------------------------------------+-- Swift/Objective-C-Specific Implementation Helpers+-- ------------------------------------------------------------------++-- | (Swift/Objective-C) Helper: Processes tokens *outside* a test method.+-- Recognizes:+-- 1. class *Test: XCTestCase+-- 2. func test*()+-- +-- Swift and Objective-C use XCTest framework.+processOutsideSwift :: Bool -> [Token] -> [Token]+processOutsideSwift _ [] = [] -- End of stream++-- Pattern 1: func test*+processOutsideSwift keepTests (t1:t2:ts)+ | isTokenKeyword t1 && tToken t1 == "func" &&+ isTokenIdentifier t2 && "test" `T.isPrefixOf` tToken t2+ =+ case findOpeningBrace ts of+ Nothing ->+ if keepTests then processOutsideSwift keepTests (t2:ts) else t1 : processOutsideSwift keepTests (t2:ts)+ Just (signatureTokens, tokensAfterBrace) ->+ let (bodyTokens, remainingTokens) = processInsideBraces 1 tokensAfterBrace+ in if keepTests+ then t1 : t2 : signatureTokens ++ bodyTokens ++ processOutsideSwift keepTests remainingTokens+ else processOutsideSwift keepTests remainingTokens++-- Pattern 2: class *Test (could also check for XCTestCase inheritance)+processOutsideSwift keepTests (t1:t2:ts)+ | isTokenKeyword t1 && tToken t1 == "class" &&+ isTokenIdentifier t2 && "Test" `T.isSuffixOf` tToken t2+ =+ case findOpeningBrace ts of+ Nothing ->+ if keepTests then processOutsideSwift keepTests (t2:ts) else t1 : processOutsideSwift keepTests (t2:ts)+ Just (signatureTokens, tokensAfterBrace) ->+ let (bodyTokens, remainingTokens) = processInsideBraces 1 tokensAfterBrace+ in if keepTests+ then t1 : t2 : signatureTokens ++ bodyTokens ++ processOutsideSwift keepTests remainingTokens+ else processOutsideSwift keepTests remainingTokens++-- No test found, process the current token+processOutsideSwift keepTests (t:ts) =+ if keepTests+ then processOutsideSwift keepTests ts+ else t : processOutsideSwift keepTests ts++-- ------------------------------------------------------------------+-- R-Specific Implementation Helpers+-- ------------------------------------------------------------------++-- | (R) Helper: Processes tokens *outside* a test block.+-- Recognizes:+-- 1. test_that("...", { ... })+-- 2. describe("...", { ... })+-- 3. context("...", { ... })+-- +-- R uses testthat framework.+processOutsideR :: Bool -> [Token] -> [Token]+processOutsideR _ [] = [] -- End of stream++-- Pattern: test_that( or describe( or context(+processOutsideR keepTests (t1:t2:ts)+ | isTokenIdentifier t1 && + (tToken t1 == "test_that" || tToken t1 == "describe" || tToken t1 == "context") &&+ isTokenBracket t2 && tToken t2 == "("+ =+ case findOpeningBrace ts of+ Nothing ->+ if keepTests then processOutsideR keepTests (t2:ts) else t1 : processOutsideR keepTests (t2:ts)+ Just (signatureTokens, tokensAfterBrace) ->+ let (bodyTokens, remainingTokens) = processInsideBraces 1 tokensAfterBrace+ in if keepTests+ then t1 : t2 : signatureTokens ++ bodyTokens ++ processOutsideR keepTests remainingTokens+ else processOutsideR keepTests remainingTokens++-- No test found, process the current token+processOutsideR keepTests (t:ts) =+ if keepTests+ then processOutsideR keepTests ts+ else t : processOutsideR keepTests ts++-- ------------------------------------------------------------------+-- Julia-Specific Implementation Helpers+-- ------------------------------------------------------------------++-- | (Julia) Helper: Processes tokens *outside* a test block.+-- Recognizes:+-- 1. @testset "..." begin ... end+-- 2. @test expression+-- +-- Julia uses Test standard library.+processOutsideJulia :: Bool -> [Token] -> [Token]+processOutsideJulia _ [] = [] -- End of stream++-- Pattern 1: @testset+processOutsideJulia keepTests (t1:t2:ts)+ | isTokenOperator t1 && tToken t1 == "@" &&+ isTokenIdentifier t2 && tToken t2 == "testset"+ =+ -- Collect until 'end' keyword+ let (testTokens, remainingTokens) = collectUntilElixirEnd ts+ in if keepTests+ then t1 : t2 : testTokens ++ processOutsideJulia keepTests remainingTokens+ else processOutsideJulia keepTests remainingTokens++-- Pattern 2: @test+processOutsideJulia keepTests (t1:t2:ts)+ | isTokenOperator t1 && tToken t1 == "@" &&+ isTokenIdentifier t2 && tToken t2 == "test"+ =+ -- Single line test, collect until newline or next statement+ let (testTokens, remainingTokens) = collectUntilNextHaskellDef ts+ in if keepTests+ then t1 : t2 : testTokens ++ processOutsideJulia keepTests remainingTokens+ else processOutsideJulia keepTests remainingTokens++-- No test found, process the current token+processOutsideJulia keepTests (t:ts) =+ if keepTests+ then processOutsideJulia keepTests ts+ else t : processOutsideJulia keepTests ts++-- ------------------------------------------------------------------+-- Perl-Specific Implementation Helpers+-- ------------------------------------------------------------------++-- | (Perl) Helper: Processes tokens *outside* a test block.+-- Recognizes:+-- 1. Test files (*.t)+-- 2. subtest blocks+-- +-- Perl uses Test::More, Test::Simple frameworks.+-- Note: Perl tests are less structured, so this is a simplified approach.+processOutsidePerl :: Bool -> [Token] -> [Token]+processOutsidePerl _ [] = [] -- End of stream++-- Pattern: subtest+processOutsidePerl keepTests (t1:ts)+ | isTokenIdentifier t1 && tToken t1 == "subtest"+ =+ case findOpeningBrace ts of+ Nothing ->+ -- Collect until next top-level statement+ let (testTokens, remainingTokens) = collectUntilNextHaskellDef ts+ in if keepTests+ then t1 : testTokens ++ processOutsidePerl keepTests remainingTokens+ else processOutsidePerl keepTests remainingTokens+ Just (signatureTokens, tokensAfterBrace) ->+ let (bodyTokens, remainingTokens) = processInsideBraces 1 tokensAfterBrace+ in if keepTests+ then t1 : signatureTokens ++ bodyTokens ++ processOutsidePerl keepTests remainingTokens+ else processOutsidePerl keepTests remainingTokens++-- No test found, process the current token+processOutsidePerl keepTests (t:ts) =+ if keepTests+ then processOutsidePerl keepTests ts+ else t : processOutsidePerl keepTests ts++-- ------------------------------------------------------------------+-- OCaml-Specific Implementation Helpers+-- ------------------------------------------------------------------++-- | (OCaml) Helper: Processes tokens *outside* a test block.+-- Recognizes:+-- 1. let test_* = ... - OUnit convention+-- 2. test_case - Alcotest+-- +-- OCaml uses OUnit and Alcotest frameworks.+processOutsideOCaml :: Bool -> [Token] -> [Token]+processOutsideOCaml _ [] = [] -- End of stream++-- Pattern 1: let test_*+processOutsideOCaml keepTests (t1:t2:ts)+ | isTokenKeyword t1 && tToken t1 == "let" &&+ isTokenIdentifier t2 && "test_" `T.isPrefixOf` tToken t2+ =+ -- Collect until next let or end of block+ let (testTokens, remainingTokens) = collectUntilNextHaskellDef ts+ in if keepTests+ then t1 : t2 : testTokens ++ processOutsideOCaml keepTests remainingTokens+ else processOutsideOCaml keepTests remainingTokens++-- Pattern 2: test_case+processOutsideOCaml keepTests (t1:ts)+ | isTokenIdentifier t1 && tToken t1 == "test_case"+ =+ let (testTokens, remainingTokens) = collectUntilNextHaskellDef ts+ in if keepTests+ then t1 : testTokens ++ processOutsideOCaml keepTests remainingTokens+ else processOutsideOCaml keepTests remainingTokens++-- No test found, process the current token+processOutsideOCaml keepTests (t:ts) =+ if keepTests+ then processOutsideOCaml keepTests ts+ else t : processOutsideOCaml keepTests ts++-- ------------------------------------------------------------------+-- Erlang-Specific Implementation Helpers+-- ------------------------------------------------------------------++-- | (Erlang) Helper: Processes tokens *outside* a test function.+-- Recognizes:+-- 1. *_test() - EUnit convention+-- 2. *_test_() - EUnit generator convention+-- +-- Erlang uses EUnit framework.+processOutsideErlang :: Bool -> [Token] -> [Token]+processOutsideErlang _ [] = [] -- End of stream++-- Pattern: function ending with _test or _test_+processOutsideErlang keepTests (t1:t2:ts)+ | isTokenIdentifier t1 && + ("_test" `T.isSuffixOf` tToken t1 || "_test_" `T.isSuffixOf` tToken t1) &&+ isTokenBracket t2 && tToken t2 == "("+ =+ -- Find the function body (might use -> or after parameters)+ case findOpeningBrace ts of+ Nothing ->+ -- Collect until next function definition+ let (testTokens, remainingTokens) = collectUntilNextHaskellDef ts+ in if keepTests+ then t1 : t2 : testTokens ++ processOutsideErlang keepTests remainingTokens+ else processOutsideErlang keepTests remainingTokens+ Just (signatureTokens, tokensAfterBrace) ->+ let (bodyTokens, remainingTokens) = processInsideBraces 1 tokensAfterBrace+ in if keepTests+ then t1 : t2 : signatureTokens ++ bodyTokens ++ processOutsideErlang keepTests remainingTokens+ else processOutsideErlang keepTests remainingTokens++-- No test found, process the current token+processOutsideErlang keepTests (t:ts) =+ if keepTests+ then processOutsideErlang keepTests ts+ else t : processOutsideErlang keepTests ts++-- ------------------------------------------------------------------+-- Nim-Specific Implementation Helpers+-- ------------------------------------------------------------------++-- | (Nim) Helper: Processes tokens *outside* a test block.+-- Recognizes:+-- 1. suite "..."+-- 2. test "..."+-- +-- Nim uses unittest module.+processOutsideNim :: Bool -> [Token] -> [Token]+processOutsideNim _ [] = [] -- End of stream++-- Pattern: suite or test+processOutsideNim keepTests (t1:ts)+ | isTokenIdentifier t1 && (tToken t1 == "suite" || tToken t1 == "test")+ =+ -- Collect until next suite/test or end+ let (testTokens, remainingTokens) = collectUntilNextHaskellDef ts+ in if keepTests+ then t1 : testTokens ++ processOutsideNim keepTests remainingTokens+ else processOutsideNim keepTests remainingTokens++-- No test found, process the current token+processOutsideNim keepTests (t:ts) =+ if keepTests+ then processOutsideNim keepTests ts+ else t : processOutsideNim keepTests ts++-- ------------------------------------------------------------------+-- Clojure-Specific Implementation Helpers+-- ------------------------------------------------------------------++-- | (Clojure) Helper: Processes tokens *outside* a test block.+-- Recognizes:+-- 1. (deftest ...)+-- 2. (testing ...)+-- +-- Clojure uses clojure.test.+processOutsideClojure :: Bool -> [Token] -> [Token]+processOutsideClojure _ [] = [] -- End of stream++-- Pattern 1: (deftest+processOutsideClojure keepTests (t1:t2:ts)+ | isTokenBracket t1 && tToken t1 == "(" &&+ isTokenIdentifier t2 && tToken t2 == "deftest"+ =+ -- Collect until matching closing paren+ let (testTokens, remainingTokens) = collectUntilMatchingParen 1 ts+ in if keepTests+ then t1 : t2 : testTokens ++ processOutsideClojure keepTests remainingTokens+ else processOutsideClojure keepTests remainingTokens++-- Pattern 2: (testing+processOutsideClojure keepTests (t1:t2:ts)+ | isTokenBracket t1 && tToken t1 == "(" &&+ isTokenIdentifier t2 && tToken t2 == "testing"+ =+ let (testTokens, remainingTokens) = collectUntilMatchingParen 1 ts+ in if keepTests+ then t1 : t2 : testTokens ++ processOutsideClojure keepTests remainingTokens+ else processOutsideClojure keepTests remainingTokens++-- No test found, process the current token+processOutsideClojure keepTests (t:ts) =+ if keepTests+ then processOutsideClojure keepTests ts+ else t : processOutsideClojure keepTests ts++-- | Helper: Collect tokens until matching closing parenthesis+collectUntilMatchingParen :: Int -> [Token] -> ([Token], [Token])+collectUntilMatchingParen 0 ts = ([], ts)+collectUntilMatchingParen _ [] = ([], [])+collectUntilMatchingParen depth (t:ts)+ | isTokenBracket t && tToken t == "(" =+ let (collected, remaining) = collectUntilMatchingParen (depth + 1) ts+ in (t : collected, remaining)+ | isTokenBracket t && tToken t == ")" =+ if depth == 1+ then ([t], ts)+ else let (collected, remaining) = collectUntilMatchingParen (depth - 1) ts+ in (t : collected, remaining)+ | otherwise =+ let (collected, remaining) = collectUntilMatchingParen depth ts+ in (t : collected, remaining)++-- ------------------------------------------------------------------+-- D-Specific Implementation Helpers+-- ------------------------------------------------------------------++-- | (D) Helper: Processes tokens *outside* a test block.+-- Recognizes:+-- 1. unittest { ... } - D's built-in test blocks+-- +-- D has built-in unittest blocks.+processOutsideD :: Bool -> [Token] -> [Token]+processOutsideD _ [] = [] -- End of stream++-- Pattern: unittest {+processOutsideD keepTests (t1:ts)+ | isTokenKeyword t1 && tToken t1 == "unittest"+ =+ case findOpeningBrace ts of+ Nothing ->+ if keepTests then processOutsideD keepTests ts else t1 : processOutsideD keepTests ts+ Just (signatureTokens, tokensAfterBrace) ->+ let (bodyTokens, remainingTokens) = processInsideBraces 1 tokensAfterBrace+ in if keepTests+ then t1 : signatureTokens ++ bodyTokens ++ processOutsideD keepTests remainingTokens+ else processOutsideD keepTests remainingTokens++-- No test found, process the current token+processOutsideD keepTests (t:ts) =+ if keepTests+ then processOutsideD keepTests ts+ else t : processOutsideD keepTests ts++-- ------------------------------------------------------------------+-- Generic Helper Functions+-- ------------------------------------------------------------------++-- | (Generic) Helper: Processes tokens *inside* a braced block, handling nesting.+-- Renamed from processInsideRust+processInsideBraces :: Int -> [Token] -> ([Token], [Token])+processInsideBraces 0 ts = ([], ts) -- Base case: found matching '}'+processInsideBraces _ [] = ([], []) -- Error case: unexpected EOF+processInsideBraces nestingLevel (t:ts)+ -- Found a nested opening brace+ | isTokenBracket t && tToken t == "{" =+ let (nestedInside, remaining) = processInsideBraces (nestingLevel + 1) ts+ in (t : nestedInside, remaining) -- Keep '{' as part of inside tokens++ -- Found a closing brace+ | isTokenBracket t && tToken t == "}" =+ if nestingLevel == 1+ then -- This is the final '}' we were looking for+ ([t], ts) -- Include the final '}' and return the rest+ else -- This is a nested '}'+ let (nestedInside, remaining) = processInsideBraces (nestingLevel - 1) ts+ in (t : nestedInside, remaining) -- Keep '}'++ -- Any other token inside the block+ | otherwise =+ let (nestedInside, remaining) = processInsideBraces nestingLevel ts+ in (t : nestedInside, remaining) -- Keep the token++-- | Helper to find the first opening brace, returning tokens before/at brace, and tokens after.+-- Returns (tokens_including_brace, tokens_after_brace)+findOpeningBrace :: [Token] -> Maybe ([Token], [Token])+findOpeningBrace = go []+ where+ go :: [Token] -> [Token] -> Maybe ([Token], [Token])+ go _ [] = Nothing -- Reached end without finding a brace+ go acc (t:ts)+ | isTokenBracket t && tToken t == "{" = Just (reverse (t:acc), ts)+ | otherwise = go (t:acc) ts
src/CGrep/Strategy/BoyerMoore.hs view
@@ -1,5 +1,4 @@------ Copyright (c) 2013-2023 Nicola Bonelli <nicola@larthia.com>+-- Copyright (c) 2013-2025 Nicola Bonelli <nicola@larthia.com> -- -- This program is free software; you can redistribute it and/or modify -- it under the terms of the GNU General Public License as published by@@ -18,91 +17,98 @@ module CGrep.Strategy.BoyerMoore (search) where -import qualified Data.ByteString.Char8 as C-import qualified Data.ByteString.Lazy.Char8 as LC+import Control.Monad.Trans.Reader (ask) -import Control.Monad.Trans.Reader ( reader, ask )-import Control.Monad.IO.Class ( MonadIO(liftIO) )-import Data.List ( isSuffixOf, isPrefixOf, genericLength )+-- expandMultiline, -import CGrep.Common- ( Text8,- expandMultiline,- getTargetContents,- getTargetName,- ignoreCase)-import CGrep.Output ( Output, mkOutputElements, runSearch )-import CGrep.ContextFilter ( mkContextFilter)-import CGrep.FileType ( FileType )-import CGrep.FileTypeMap ( fileTypeLookup, contextFilter, FileTypeInfo )-import CGrep.Types ( Offset )-import CGrep.Search+import CGrep.Common (ignoreCase, runSearch)+import CGrep.Semantic.ContextFilter (mkContextFilter)+import CGrep.FileType (FileType)+import CGrep.FileTypeMap (FileTypeInfo)+import CGrep.FileTypeMapTH (mkContextFilterFn)+import CGrep.Match (Match, mkMatches) -import Reader ( ReaderIO, Env(..) )-import Options ( Options(word_match, prefix_match, suffix_match) )-import Verbose ( putMsgLnVerbose ) import CGrep.Parser.Chunk-import Data.Int ( Int64 )+import Options (Options (..))+import PutMessage (putMessageLnVerb)+import Reader (Env (..), ReaderIO) -import System.Posix.FilePath ( RawFilePath )-import System.IO ( stderr )+import System.IO (stderr)+import System.OsPath (OsPath) -import CGrep.Parser.Line ( getLineOffsets, getLineByOffset )+import CGrep.Line (buildIndex, getLineByOffset, getLineOffsets)+import CGrep.Text (textContainsOneOf, textIndices, textSlice)+import Control.Concurrent (MVar)+import qualified Data.Text as T+import qualified Data.Text.Unsafe as TU import qualified Data.Vector.Unboxed as UV-import Data.Array (indices) --search :: Maybe (FileType, FileTypeInfo) -> RawFilePath -> [Text8] -> ReaderIO [Output]-search info f patterns = do-+search :: MVar () -> Maybe (FileType, FileTypeInfo) -> OsPath -> T.Text -> [T.Text] -> Bool -> ReaderIO [Match]+search lock info filename text patterns _strict = do Env{..} <- ask-- text <- liftIO $ getTargetContents f-- let filename = getTargetName f-- -- transform text+ let lindex = buildIndex text - let ctxFilter = mkContextFilter opt+ -- transform text...+ let !contextFilter = mkContextFilterFn (fst <$> info) (mkContextFilter opt) False - let [text''', _ , text', _] = scanr ($) text [ expandMultiline opt- , contextFilter (fst <$> info) ctxFilter False- , ignoreCase opt- ]+ let text' = ignoreCase opt text+ let text'' = contextFilter $ text' -- make shallow search-- let indices' = searchStringIndices patterns text'- let indices''' = searchStringIndices patterns text'''+ let !eligibleForSearch = textContainsOneOf patterns text' -- search for matching tokens-- let ctor = Chunk ChunkUnspec-- let chunks = concat $ zipWith (\p xs -> (p `ctor` ) <$> xs ) patterns indices'''+ putMessageLnVerb 3 lock stderr $ "---\n" <> text'' <> "\n---"+ putMessageLnVerb 1 lock stderr $ "strategy : running Boyer-Moore search on " <> show filename - -- filter exact/partial matching tokens+ runSearch lindex filename eligibleForSearch $ do+ let lineOffsets = getLineOffsets text''+ let indices = textIndices patterns text'' - let lineOffsets = getLineOffsets (fromIntegral $ C.length text) text+ let !chunks =+ concat $+ zipWith+ ( \p offsets ->+ let blen = TU.lengthWord8 p+ in map (\offset -> Chunk ChunkUnspec (textSlice text'' offset blen)) offsets+ )+ patterns+ indices - let chunks' = if word_match opt || prefix_match opt || suffix_match opt- then filter (checkChunk opt lineOffsets (snd <$> info) text''') chunks+ let !chunks' =+ if word_match opt || prefix_match opt || suffix_match opt+ then filter (filterChunk opt lineOffsets (snd <$> info) text'') chunks else chunks - putMsgLnVerbose 2 stderr $ "strategy : running Boyer-Moore search on " <> filename- putMsgLnVerbose 3 stderr $ "---\n" <> text''' <> "\n---"-- runSearch opt filename (eligibleForSearch patterns indices') $ do-- putMsgLnVerbose 2 stderr $ "chunks' : " <> show chunks'- mkOutputElements lineOffsets filename text text''' chunks'-+ putMessageLnVerb 2 lock stderr $ "matches : " <> show chunks'+ putMessageLnVerb 2 lock stderr $ "lindex : " <> show lindex+ mkMatches lindex filename text'' chunks' -checkChunk :: Options -> UV.Vector Int64 -> Maybe FileTypeInfo -> Text8 -> Chunk -> Bool-checkChunk opt vec info text chunk- | word_match opt = let !off = cOffset chunk - off' in any (\chunk' -> cOffset chunk' == off && cToken chunk' == cToken chunk) cs- | prefix_match opt = any (\chunk' -> cToken chunk `C.isPrefixOf` cToken chunk' && cOffset chunk' + off' == cOffset chunk) cs- | suffix_match opt = any (\chunk' -> cToken chunk `C.isSuffixOf` cToken chunk' && cOffset chunk' + off' + fromIntegral (C.length (cToken chunk') - C.length (cToken chunk)) == cOffset chunk) cs- | otherwise = undefined- where (# line',off' #) = getLineByOffset (cOffset chunk) text vec- cs = parseChunks info line'+filterChunk :: Options -> UV.Vector Int -> Maybe FileTypeInfo -> T.Text -> Chunk -> Bool+filterChunk opts loff info text chunk+ | word_match opts =+ let (# line', _ #) = getLineByOffset ((cOffset chunk)) text loff+ !cs = parseChunks info line'+ !off = cOffset chunk+ in any (\chunk' -> cOffset chunk' == off && cToken chunk' == cToken chunk) $ cs+ | prefix_match opts =+ let !(# line', _ #) = getLineByOffset (cOffset chunk) text loff+ !cs = parseChunks info line'+ in any+ ( \chunk' ->+ cToken chunk `T.isPrefixOf` cToken chunk'+ && cOffset chunk' == cOffset chunk+ )+ cs+ | suffix_match opts =+ let !(# line', _ #) = getLineByOffset (cOffset chunk) text loff+ !cs = parseChunks info line'+ !tokLen = T.length (cToken chunk)+ in any+ ( \chunk' ->+ let !tokLen' = T.length (cToken chunk')+ in cToken chunk `T.isSuffixOf` cToken chunk'+ && cOffset chunk' + (tokLen' - tokLen) == cOffset chunk+ )+ cs+ | otherwise = undefined
src/CGrep/Strategy/Levenshtein.hs view
@@ -1,5 +1,5 @@ ----- Copyright (c) 2013-2023 Nicola Bonelli <nicola@larthia.com>+-- Copyright (c) 2013-2025 Nicola Bonelli <nicola@larthia.com> -- -- This program is free software; you can redistribute it and/or modify -- it under the terms of the GNU General Public License as published by@@ -17,66 +17,58 @@ -- module CGrep.Strategy.Levenshtein (search) where-import CGrep.Parser.Line ( getAllLineOffsets ) -import qualified Data.ByteString.Char8 as C+import CGrep.Line (buildIndex) -import Control.Monad.Trans.Reader ( reader, ask )-import Control.Monad.IO.Class ( MonadIO(liftIO) )+import Control.Monad.Trans.Reader (ask) -import CGrep.ContextFilter ( mkContextFilter )-import CGrep.Common- ( Text8,- getTargetName,- getTargetContents,- expandMultiline,- ignoreCase )-import CGrep.Output ( Output, mkOutputElements )-import CGrep.Distance ( (~==) )-import CGrep.Parser.Chunk ( Chunk, cToken, parseChunks )-import CGrep.FileType ( FileType )-import CGrep.FileTypeMap- ( fileTypeLookup, FileTypeInfo, contextFilter )+import CGrep.Common (+ ignoreCase,+ )+import CGrep.Semantic.ContextFilter (mkContextFilter)+import CGrep.Distance ((~==))+import CGrep.FileType (FileType)+import CGrep.FileTypeMap (+ FileTypeInfo,+ )+import CGrep.FileTypeMapTH (+ mkContextFilterFn,+ ) -import Reader ( ReaderIO, Env (..) )-import Verbose ( putMsgLnVerbose )-import System.Posix.FilePath (RawFilePath)-import System.IO (stderr)-import Data.Foldable ( Foldable(toList) )+import CGrep.Match (Match, mkMatches)+import CGrep.Parser.Chunk (cToken, parseChunks) -search :: Maybe (FileType, FileTypeInfo) -> RawFilePath -> [Text8] -> ReaderIO [Output]-search info f patterns = do+import Control.Concurrent (MVar)+import Data.Foldable (Foldable (toList))+import qualified Data.Text as T+import PutMessage (putMessageLnVerb)+import Reader (Env (..), ReaderIO)+import System.IO (stderr)+import System.OsPath (OsPath) +search :: MVar () -> Maybe (FileType, FileTypeInfo) -> OsPath -> T.Text -> [T.Text] -> Bool -> ReaderIO [Match]+search lock info filename text patterns _strict = do Env{..} <- ask-- text <- liftIO $ getTargetContents f-- let filename = getTargetName f+ let lindex = buildIndex text -- transform text-- let ctxFilter = mkContextFilter opt+ let !contextFilter = mkContextFilterFn (fst <$> info) (mkContextFilter opt) False - let [text''', _ , _ , _] = scanr ($) text [ expandMultiline opt- , contextFilter (fst <$> fileTypeLookup opt filename) ctxFilter False- , ignoreCase opt- ]+ let text' = ignoreCase opt text+ let text'' = contextFilter $ text' -- parse source code, get the Cpp.Token list... - tokens' = parseChunks (snd <$> info) text'''+ let tokens' = parseChunks (snd <$> info) text'' -- filter tokens... - patterns' = map C.unpack patterns- matches = filter (\t -> any (\p -> p ~== C.unpack (cToken t)) patterns') (toList tokens')-- putMsgLnVerbose 2 stderr $ "strategy : running edit-distance (Levenshtein) search on " <> filename <> "..."- putMsgLnVerbose 3 stderr $ "---\n" <> text''' <> "\n---"-- putMsgLnVerbose 2 stderr $ "tokens : " <> show tokens'- putMsgLnVerbose 2 stderr $ "matches : " <> show matches+ let patterns' = map T.unpack patterns+ let matches = filter (\t -> any (\p -> p ~== T.unpack (cToken t)) patterns') (toList tokens') - let lineOffsets = getAllLineOffsets text+ putMessageLnVerb 3 lock stderr $ "---\n" <> text'' <> "\n---"+ putMessageLnVerb 1 lock stderr $ "strategy : running edit-distance (Levenshtein) search on " <> show filename+ putMessageLnVerb 2 lock stderr $ "tokens : " <> show tokens'+ putMessageLnVerb 2 lock stderr $ "matches : " <> show matches - mkOutputElements lineOffsets filename text text''' matches+ mkMatches lindex filename text'' matches
src/CGrep/Strategy/Regex.hs view
@@ -1,5 +1,5 @@ ----- Copyright (c) 2013-2023 Nicola Bonelli <nicola@larthia.com>+-- Copyright (c) 2013-2025 Nicola Bonelli <nicola@larthia.com> -- -- This program is free software; you can redistribute it and/or modify -- it under the terms of the GNU General Public License as published by@@ -17,69 +17,72 @@ -- module CGrep.Strategy.Regex (search) where+import Control.Monad.Trans.Reader (ask) -import qualified Data.ByteString.Char8 as C+import Text.Regex.Base (+ AllTextMatches (getAllTextMatches),+ MatchText,+ ) -import Control.Monad.Trans.Reader ( reader, ask )-import Control.Monad.IO.Class ( MonadIO(liftIO) )+#ifdef ENABLE_PCRE+import qualified Text.Regex.PCRE ((=~))+#endif+import qualified Text.Regex.TDFA ((=~)) -import Text.Regex.Base- ( AllTextMatches(getAllTextMatches), MatchText )-import Text.Regex.Posix ( (=~) )-import Text.Regex.PCRE ( (=~) )+import Data.Array (Array, elems) -import Data.Array ( Array, elems )+import CGrep.Common (+ ignoreCase,+ )+import CGrep.Semantic.ContextFilter (mkContextFilter)+import CGrep.FileType (FileType)+import CGrep.FileTypeMap (FileTypeInfo (..))+import CGrep.FileTypeMapTH (mkContextFilterFn)+import CGrep.Match (Match, mkMatches) -import CGrep.Common- ( Text8,- getTargetName,- getTargetContents,- expandMultiline,- ignoreCase )-import CGrep.Output ( Output, mkOutputElements )-import CGrep.ContextFilter ( mkContextFilter)-import CGrep.FileType ( FileType )-import CGrep.FileTypeMap ( FileTypeInfo(..), fileTypeLookup, contextFilter )+#ifdef ENABLE_PCRE+import Options (Options (regex_pcre))+#endif -import Reader ( ReaderIO, Env (..) )-import Options ( Options(regex_pcre) )-import Verbose ( putMsgLnVerbose )+import PutMessage (putMessageLnVerb)+import Reader (Env (..), ReaderIO) import CGrep.Parser.Chunk-import CGrep.Parser.Line ( getAllLineOffsets ) -import System.Posix.FilePath (RawFilePath) import System.IO (stderr)--search :: Maybe (FileType, FileTypeInfo) -> RawFilePath -> [Text8] -> ReaderIO [Output]-search info f patterns = do+import System.OsPath (OsPath)+import qualified Data.Text as T+import CGrep.Line (buildIndex)+import Control.Concurrent (MVar) +search :: MVar () -> Maybe (FileType, FileTypeInfo) -> OsPath -> T.Text -> [T.Text] -> Bool -> ReaderIO [Match]+search lock info filename text patterns _strict = do Env{..} <- ask-- text <- liftIO $ getTargetContents f+ let lindex = buildIndex text - let filename = getTargetName f+ let !contextFilter = mkContextFilterFn (fst <$> info) (mkContextFilter opt) False -- transform text - let ctxFilter = mkContextFilter opt-- let [text''', _ , _ , _] = scanr ($) text [ expandMultiline opt- , contextFilter (fst <$> fileTypeLookup opt filename) ctxFilter False- , ignoreCase opt- ]-- -- search for matching tokens-- (=~~~) = if regex_pcre opt then (Text.Regex.PCRE.=~) else (Text.Regex.Posix.=~)-- tokens = map (\(str, (off,_)) -> Chunk ChunkUnspec str (fromIntegral off)) $- concatMap elems $ patterns >>= (\p -> elems (getAllTextMatches $ text''' =~~~ p :: (Array Int) (MatchText Text8)))+ let text' = contextFilter . ignoreCase opt $ text - putMsgLnVerbose 2 stderr $ "strategy : running regex " <> (if regex_pcre opt then "(pcre)" else "(posix)") <> " search on " <> filename <> "..."- putMsgLnVerbose 3 stderr $ "---\n" <> text''' <> "\n---"- putMsgLnVerbose 2 stderr $ "tokens : " <> show tokens+ -- search for matching tokens+#ifdef ENABLE_PCRE+ (=~~~) = if regex_pcre opt then (Text.Regex.PCRE.=~) else (Text.Regex.TDFA.=~)+#else+ (=~~~) = (Text.Regex.TDFA.=~)+#endif+ tokens =+ map (\(str, (_, _)) -> Chunk ChunkUnspec str) $+ concatMap elems $+ patterns >>= (\p -> elems (getAllTextMatches $ text' =~~~ p :: (Array Int) (MatchText T.Text))) - let lineOffsets = getAllLineOffsets text+ putMessageLnVerb 3 lock stderr $ "---\n" <> text' <> "\n---"+ #ifdef ENABLE_PCRE+ putMessageLnVerb 1 lock stderr $ "strategy : running regex " <> (if regex_pcre opt then "(pcre)" else "(posix)") <> " search on " <> show filename+ #else+ putMessageLnVerb 1 lock stderr $ "strategy : running regex (posix) search on " <> show filename+ #endif+ putMessageLnVerb 2 lock stderr $ "tokens : " <> show tokens - mkOutputElements lineOffsets filename text text''' tokens+ mkMatches lindex filename text' tokens
src/CGrep/Strategy/Semantic.hs view
@@ -1,5 +1,5 @@ ----- Copyright (c) 2013-2023 Nicola Bonelli <nicola@larthia.com>+-- Copyright (c) 2013-2025 Nicola Bonelli <nicola@larthia.com> -- -- This program is free software; you can redistribute it and/or modify -- it under the terms of the GNU General Public License as published by@@ -18,118 +18,111 @@ module CGrep.Strategy.Semantic (search) where -import qualified Data.ByteString.Char8 as C-import CGrep.Parser.Token--import CGrep.ContextFilter- ( contextBitComment, mkContextFilter, (~!) )-import CGrep.Common- ( Text8,- trim,- getTargetName,- getTargetContents,- expandMultiline,- ignoreCase, trim8, subText )--import CGrep.Search ( eligibleForSearch, searchStringIndices )-import CGrep.Output ( Output, mkOutputElements, runSearch )-import CGrep.Parser.Line ( getAllLineOffsets )--import CGrep.Parser.Atom- ( Atom(..),- mkAtomFromToken,- combineAtoms,- filterTokensWithAtoms)--import Control.Monad.Trans.Reader ( reader, ask )-import Control.Monad.IO.Class ( MonadIO(liftIO) )--import Data.List ( sortBy, nub )-import Data.Function ( on )-import Data.Maybe ( mapMaybe )--import Reader ( ReaderIO, Env (..) )-import Verbose ( putMsgLnVerbose )-import Util ( rmQuote8 )+import CGrep.Common (ignoreCase, runSearch, sliceToMaxIndex, trimT)+import CGrep.Semantic.ContextFilter (+ contextBitComment,+ mkContextFilter,+ (~!),+ )+import CGrep.FileType (FileType)+import CGrep.FileTypeMap (+ FileTypeInfo,+ )+import CGrep.FileTypeMapTH (+ mkContextFilterFn,+ )+import CGrep.Line (buildIndex)+import CGrep.Match (Match, mkMatches)+import CGrep.Parser.Atom (+ Atom (..),+ findAllMatches,+ mkAtomFromToken,+ ) import CGrep.Parser.Chunk--import System.Posix.FilePath ( RawFilePath, takeBaseName )--import CGrep.FileType ( FileType )-import CGrep.FileTypeMap- ( fileTypeLookup, FileTypeInfo, contextFilter )-import System.IO ( stderr )--import qualified Data.Sequence as S-import Data.Foldable ( Foldable(toList) )-import Data.Coerce ( coerce )--search :: Maybe (FileType, FileTypeInfo) -> RawFilePath -> [Text8] -> ReaderIO [Output]-search info f ps = do+import CGrep.Parser.Token+import CGrep.Text (textContainsOneOf, textIndices)+import Control.Concurrent (MVar)+import Control.Monad.Trans.Reader (ask)+import Data.Coerce (coerce)+import Data.Foldable (Foldable (toList))+import Data.Function (on)+import Data.List (sortBy)+import Data.Maybe (mapMaybe)+import qualified Data.Text as T+import PutMessage (putMessageLnVerb)+import Reader (Env (..), ReaderIO)+import System.IO (stderr)+import System.OsPath (OsPath)+import Util (unquoteT)+import CGrep.Semantic.Tests (filterTests)+import Options (Options(..)) +search :: MVar () -> Maybe (FileType, FileTypeInfo) -> OsPath -> T.Text -> [T.Text] -> Bool -> ReaderIO [Match]+search lock info filename text patterns strict = do Env{..} <- ask-- text <- liftIO $ getTargetContents f-- let filename = getTargetName f+ let lindex = buildIndex text - let [text''', _, text', _ ] = scanr ($) text [ expandMultiline opt- , contextFilter (fst <$> fileTypeLookup opt filename) filt True- , ignoreCase opt- ]+ let filt = mkContextFilter opt ~! contextBitComment+ let !contextFilter = mkContextFilterFn (fst <$> info) filt True - filt = mkContextFilter opt ~! contextBitComment+ let text' = ignoreCase opt text+ let text'' = contextFilter $ text' -- pre-process patterns-- pfilter = TokenFilter {- tfIdentifier = True,- tfKeyword = True,- tfNativeType = True,- tfString = True,- tfNumber = True,- tfOperator = True,- tfBracket = True}-- patterns = map (parseTokens pfilter (snd <$> info) . contextFilter (fst <$> fileTypeLookup opt filename) filt True) ps- patterns' = map (mkAtomFromToken <$>) patterns- patterns'' = map (combineAtoms . map (:[])) (toList <$> patterns')-- identifiers = mapMaybe- (\case- Raw (Token (Chunk ChunkString xs _)) -> Just (rmQuote8 $ trim8 xs)- Raw (Token (Chunk ChunkIdentifier "OR" _)) -> Nothing- Raw t -> Just (tToken t)- _ -> Nothing)- (concatMap toList patterns')+ let pfilter =+ TokenFilter+ { tfIdentifier = True+ , tfKeyword = True+ , tfNativeType = True+ , tfString = True+ , tfNumber = True+ , tfOperator = True+ , tfBracket = True+ } - -- put banners...+ let patterns' = map (parseTokens pfilter (snd <$> info) strict . contextFilter) patterns+ patterns'' = map (toList . (mkAtomFromToken <$>)) patterns' - putMsgLnVerbose 2 stderr $ "strategy : running generic semantic search on " <> filename <> "..."- putMsgLnVerbose 2 stderr $ "atoms : " <> show patterns'' <> " -> identifiers: " <> show identifiers- putMsgLnVerbose 3 stderr $ "---\n" <> text''' <> "\n---"+ let matchers =+ mapMaybe+ ( \case+ Exact (Token (Chunk ChunkString xs)) -> Just ((unquoteT . trimT) xs)+ Exact t -> Just (tToken t)+ _ -> Nothing+ )+ (concatMap toList patterns'') - let indices' = searchStringIndices identifiers text'+ let !eligibleForSearch = textContainsOneOf matchers text' - runSearch opt filename (eligibleForSearch identifiers indices') $ do+ runSearch lindex filename eligibleForSearch $ do+ putMessageLnVerb 3 lock stderr $ "---\n" <> text'' <> "\n---"+ putMessageLnVerb 1 lock stderr $ "strategy : running generic semantic search on " <> show filename+ putMessageLnVerb 2 lock stderr $ "atoms : " <> show patterns''+ putMessageLnVerb 2 lock stderr $ "matchers : " <> show matchers -- parse source code, get the Generic.Chunk list...+ let indices' = textIndices matchers text'' - let tfilter = mkTokenFilter $ cTyp . coerce <$> concatMap toList patterns+ -- parse source code, get the Generic.Token list...+ -- let tfilter = mkTokenFilter $ cTyp . coerce <$> concatMap toList patterns'+ -- putMessageLnVerb 3 lock stderr $ "filter : " <> show tfilter - let tokens = toList $ parseTokens tfilter (snd <$> info) (subText indices' text''')+ let tokens = toList $ parseTokens pfilter (snd <$> info) strict (sliceToMaxIndex indices' text'')+ putMessageLnVerb 3 lock stderr $ "indices : " <> show indices' -- get matching tokens ... - let tokens' = sortBy (compare `on` tOffset) $ nub $ concatMap (\ms -> filterTokensWithAtoms opt ms tokens) patterns''-- -- convert Tokens to Chunks+ let tokens' = case (fst <$> info) of+ Nothing -> tokens+ Just ft -> filterTests ft (tests opt) tokens - let matches = coerce tokens' :: [Chunk]+ putMessageLnVerb 3 lock stderr $ "tokens : " <> show tokens+ putMessageLnVerb 3 lock stderr $ "tokens' : " <> show tokens' - putMsgLnVerbose 2 stderr $ "tokens : " <> show tokens- putMsgLnVerbose 2 stderr $ "matches : " <> show matches+ let allMatches = sortBy (compare `on` tOffset) $ findAllMatches opt patterns'' tokens' - let lineOffsets = getAllLineOffsets text+ -- convert Tokens to Chunks+ let matches = coerce allMatches :: [Chunk]+ putMessageLnVerb 2 lock stderr $ "matches : " <> show matches - mkOutputElements lineOffsets filename text text''' matches+ mkMatches lindex filename text'' matches
src/CGrep/Strategy/Tokenizer.hs view
@@ -1,5 +1,5 @@ ----- Copyright (c) 2013-2023 Nicola Bonelli <nicola@larthia.com>+-- Copyright (c) 2013-2025 Nicola Bonelli <nicola@larthia.com> -- -- This program is free software; you can redistribute it and/or modify -- it under the terms of the GNU General Public License as published by@@ -18,113 +18,113 @@ module CGrep.Strategy.Tokenizer (search) where -import qualified Data.ByteString.Char8 as C-import Control.Monad.Trans.Reader ( reader, ask )-import Control.Monad.IO.Class ( MonadIO(liftIO) )+import CGrep.Common (+ ignoreCase,+ sliceToMaxIndex,+ )+import CGrep.Semantic.ContextFilter (+ contextBitComment,+ mkContextFilter,+ (~!),+ )+import CGrep.Distance ((~==))+import CGrep.FileType (FileType)+import CGrep.FileTypeMap (+ FileTypeInfo,+ ) -import CGrep.ContextFilter- ( contextBitComment, (~!), mkContextFilter )-import CGrep.Common- ( Text8,- expandMultiline,- getTargetContents,- getTargetName,- ignoreCase,- subText- )-import CGrep.Output ( Output, mkOutputElements, runSearch )-import CGrep.Distance ( (~==) )+import CGrep.FileTypeMapTH (+ mkContextFilterFn,+ ) -import CGrep.Parser.Line+import CGrep.Common (runSearch)+import CGrep.Line+import CGrep.Match (Match, mkMatches)+import CGrep.Parser.Chunk (Chunk (..)) import CGrep.Parser.Token--import CGrep.FileType ( FileType )-import CGrep.FileTypeMap- ( fileTypeLookup, FileTypeInfo, contextFilter )--import CGrep.Search ( eligibleForSearch, searchStringIndices )-import Data.List ( isSuffixOf, isInfixOf, isPrefixOf )--import Reader ( ReaderIO, Env (..) )-import Options- ( Options(identifier, keyword, string, number, operator, edit_dist,- word_match, prefix_match, suffix_match, nativeType) )-import Verbose ( putMsgLnVerbose )--import CGrep.Parser.Chunk (Chunk(..))-import System.Posix.FilePath (RawFilePath)+import CGrep.Text (textContainsOneOf, textIndices)+import Control.Concurrent (MVar)+import Control.Monad.Trans.Reader (ask)+import Data.Coerce (coerce)+import qualified Data.Text as T+import Options (+ Options (+ edit_dist,+ identifier,+ keyword,+ nativeType,+ number,+ operator,+ prefix_match,+ string,+ suffix_match,+ word_match+ ),+ )+import PutMessage (putMessageLnVerb)+import Reader (Env (..), ReaderIO) import System.IO (stderr)--import Data.Foldable ( Foldable(toList) )-import CGrep.Types (Offset)-import Data.Coerce ( coerce )--import qualified Data.Sequence as S-import Util ( mapMaybe' )--search :: Maybe (FileType, FileTypeInfo) -> RawFilePath -> [Text8] -> ReaderIO [Output]-search info f ps = do+import System.OsPath (OsPath)+import Util (mapMaybe') +search :: MVar () -> Maybe (FileType, FileTypeInfo) -> OsPath -> T.Text -> [T.Text] -> Bool -> ReaderIO [Match]+search lock info filename text patterns strict = do Env{..} <- ask-- text <- liftIO $ getTargetContents f-- let filename = getTargetName f+ let lindex = buildIndex text -- transform text let filt = mkContextFilter opt ~! contextBitComment - let [text''', _ , text', _] = scanr ($) text [ expandMultiline opt- , contextFilter (fst <$> fileTypeLookup opt filename) filt True- , ignoreCase opt- ]+ let !contextFilter = mkContextFilterFn (fst <$> info) filt True + let text' = ignoreCase opt text+ let text'' = contextFilter $ text' - putMsgLnVerbose 2 stderr $ "strategy: running token search on " <> filename <> "..."- putMsgLnVerbose 3 stderr $ "---\n" <> text''' <> "\n---"+ -- make shallow search+ let !eligibleForSearch = textContainsOneOf patterns text' - let indices' = searchStringIndices ps text'+ putMessageLnVerb 3 lock stderr $ "---\n" <> text'' <> "\n---"+ putMessageLnVerb 1 lock stderr $ "strategy: running token search on " <> show filename <> "..." - runSearch opt filename (eligibleForSearch ps indices') $ do+ runSearch lindex filename eligibleForSearch $ do+ let indices = textIndices patterns text'' -- parse source code, get the token list... - let tfilter = TokenFilter {- tfIdentifier = identifier opt,- tfKeyword = keyword opt,- tfNativeType = nativeType opt,- tfString = string opt,- tfNumber = number opt,- tfOperator = operator opt,- tfBracket = False }-- let tokens = {-# SCC tok_0 #-} parseTokens tfilter (snd <$> info) (subText indices' text''')-- -- filter tokens and make chunks+ let tfilter =+ TokenFilter+ { tfIdentifier = identifier opt+ , tfKeyword = keyword opt+ , tfNativeType = nativeType opt+ , tfString = string opt+ , tfNumber = number opt+ , tfOperator = operator opt+ , tfBracket = False+ } - matches = {-# SCC tok_3 #-} mapMaybe' (tokenizerFilter opt ps) tokens+ let tokens = parseTokens tfilter (snd <$> info) strict (sliceToMaxIndex indices text'') - putMsgLnVerbose 2 stderr $ "tokens : " <> show tokens- putMsgLnVerbose 2 stderr $ "matches : " <> show matches+ -- filter tokens and make chunks - let lineOffsets = getAllLineOffsets text+ chunks = mapMaybe' (tokenizerFilter opt patterns) tokens - mkOutputElements lineOffsets filename text text''' matches+ putMessageLnVerb 2 lock stderr $ "tokens : " <> show tokens+ putMessageLnVerb 2 lock stderr $ "matches : " <> show chunks + mkMatches lindex filename text'' chunks -tokenizerFilter :: Options -> [C.ByteString] -> Token -> Maybe Chunk+tokenizerFilter :: Options -> [T.Text] -> Token -> Maybe Chunk tokenizerFilter opt patterns token | isTokenUnspecified token = Nothing | tokenPredicate opt patterns token = Just $ coerce token | otherwise = Nothing {-# INLINE tokenizerFilter #-} --tokenPredicate :: Options -> [C.ByteString] -> Token -> Bool+tokenPredicate :: Options -> [T.Text] -> Token -> Bool tokenPredicate opt patterns tokens- | edit_dist opt = (\t -> any (\p -> C.unpack p ~== (C.unpack . tToken) t) patterns) tokens- | word_match opt = ((`elem` patterns) . tToken) tokens- | prefix_match opt = ((\t -> any (`C.isPrefixOf`t) patterns) . tToken) tokens- | suffix_match opt = ((\t -> any (`C.isSuffixOf`t) patterns) . tToken) tokens- | otherwise = ((\t -> any (`C.isInfixOf` t) patterns) . tToken) tokens+ | edit_dist opt = (\t -> any (\p -> T.unpack p ~== (T.unpack . tToken) t) patterns) tokens+ | word_match opt = ((`elem` patterns) . tToken) tokens+ | prefix_match opt = ((\t -> any (`T.isPrefixOf` t) patterns) . tToken) tokens+ | suffix_match opt = ((\t -> any (`T.isSuffixOf` t) patterns) . tToken) tokens+ | otherwise = ((\t -> any (`T.isInfixOf` t) patterns) . tToken) tokens
+ src/CGrep/Text.hs view
@@ -0,0 +1,159 @@+--+-- Copyright (c) 2013-2025 Nicola Bonelli <nicola@larthia.com>+--+-- This program is free software; you can redistribute it and/or modify+-- it under the terms of the GNU General Public License as published by+-- the Free Software Foundation; either version 2 of the License, or+-- (at your option) any later version.+--+-- This program is distributed in the hope that it will be useful,+-- but WITHOUT ANY WARRANTY; without even the implied warranty of+-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the+-- GNU General Public License for more details.+--+-- You should have received a copy of the GNU General Public License+-- along with this program; if not, write to the Free Software+-- Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.+--++module CGrep.Text (+ iterM,+ textIndices,+ textSlice,+ textOffsetWord8,+ textContainsOneOf,+ firstIndex,+ charUtf8Length,+ blankByWidth,+) where++import Data.Bits (unsafeShiftL, (.&.), (.|.))+import Data.Char (ord)+import Data.Maybe (isJust)+import qualified Data.Text as T+import qualified Data.Text.Array as A+import qualified Data.Text.Internal as TI+import Data.Text.Internal.ArrayUtils (memchr)+import qualified Data.Text.Internal.Search as TIS+import qualified Data.Text.Unsafe as TU+import Data.Word (Word64)+import GHC.Word (Word8)++iterM :: (Monad m) => T.Text -> ((# Char, Int, Int #) -> m ()) -> m ()+iterM txt f = go 0 (return ())+ where+ !len = TU.lengthWord8 txt+ go !off !cont+ | off >= len = cont+ | otherwise =+ let TU.Iter !c !delta = TU.iter txt off+ in f (# c, off, delta #) >> go (off + delta) cont+{-# INLINE iterM #-}++textIndices :: [T.Text] -> T.Text -> [[Int]]+textIndices ps text = (`TIS.indices` text) <$> ps+{-# INLINE textIndices #-}++textContainsOneOf :: [T.Text] -> T.Text -> Bool+textContainsOneOf [] _ = True+textContainsOneOf ps text = any isJust ((`firstIndex` text) <$> ps)+{-# INLINE textContainsOneOf #-}++textSlice :: T.Text -> Int -> Int -> T.Text+textSlice txt start len = TU.takeWord8 len $ TU.dropWord8 start txt+{-# INLINE textSlice #-}++textOffsetWord8 :: T.Text -> Int+textOffsetWord8 (TI.Text _ off _) = off+{-# INLINE textOffsetWord8 #-}++charUtf8Length :: Char -> Int+charUtf8Length c+ | n <= 0x7F = 1 -- 0-127+ | n <= 0x7FF = 2 -- 128-2047+ | n <= 0xFFFF = 3 -- 2048-65535+ | otherwise = 4 -- 65536-1114111 (max Unicode code point)+ where+ -- Get the integer Unicode code point from the Char+ n = ord c+{-# INLINE charUtf8Length #-}++blankByWidth :: Char -> Char+blankByWidth c+ | n <= 0x7F = '\x0020' -- SPACE+ | n <= 0x7FF = '\x00A0' -- NO-BREAK SPACE+ | n <= 0xFFFF = '\x3000' -- IDEOGRAPHIC SPACE+ | otherwise = '\x100000' -- (Placeholder PUA)+ where+ -- Get the integer Unicode code point from the Char+ n = ord c+{-# INLINE blankByWidth #-}++data T = {-# UNPACK #-} !Word64 :* {-# UNPACK #-} !Int++{- | /O(n+m)/ Find the offsets of all non-overlapping indices of+@needle@ within @haystack@.++In (unlikely) bad cases, this algorithm's complexity degrades+towards /O(n*m)/.+-}+firstIndex ::+ -- | Substring to search for (@needle@)+ T.Text ->+ -- | Text to search in (@haystack@)+ T.Text ->+ Maybe Int+firstIndex needle@(TI.Text narr noff nlen)+ | nlen == 1 = scanOne (A.unsafeIndex narr noff)+ | nlen <= 0 = const Nothing+ | otherwise = firstIndex' needle+{-# INLINE firstIndex #-}++-- | nlen must be >= 2, otherwise nindex causes access violation+firstIndex' :: T.Text -> T.Text -> Maybe Int+firstIndex' (TI.Text narr noff nlen) (TI.Text harr@(A.ByteArray harr#) hoff hlen) = loop (hoff + nlen)+ where+ nlast = nlen - 1+ !z = nindex nlast+ nindex k = A.unsafeIndex narr (noff + k)+ buildTable !i !msk !skp+ | i >= nlast = (msk .|. swizzle z) :* skp+ | otherwise = buildTable (i + 1) (msk .|. swizzle c) skp'+ where+ !c = nindex i+ skp'+ | c == z = nlen - i - 2+ | otherwise = skp+ !(mask :* skip) = buildTable 0 0 (nlen - 2)++ swizzle :: Word8 -> Word64+ swizzle !k = 1 `unsafeShiftL` (word8ToInt k .&. 0x3f)++ loop !i+ | i > hlen + hoff =+ Nothing+ | A.unsafeIndex harr (i - 1) == z =+ if A.equal narr noff harr (i - nlen) nlen+ then Just $ i - nlen - hoff+ else loop (i + skip + 1)+ | i == hlen + hoff =+ Nothing+ | mask .&. swizzle (A.unsafeIndex harr i) == 0 =+ loop (i + nlen + 1)+ | otherwise =+ case memchr harr# i (hlen + hoff - i) z of+ -1 -> Nothing+ x -> loop (i + x + 1)+{-# INLINE firstIndex' #-}++scanOne :: Word8 -> T.Text -> Maybe Int+scanOne c (TI.Text harr hoff hlen) = loop 0+ where+ loop !i+ | i >= hlen = Nothing+ | A.unsafeIndex harr (hoff + i) == c = Just i+ | otherwise = loop (i + 1)+{-# INLINE scanOne #-}++word8ToInt :: Word8 -> Int+word8ToInt = fromIntegral
− src/CGrep/Types.hs
@@ -1,31 +0,0 @@------ Copyright (c) 2013-2023 Nicola Bonelli <nicola@larthia.com>------ This program is free software; you can redistribute it and/or modify--- it under the terms of the GNU General Public License as published by--- the Free Software Foundation; either version 2 of the License, or--- (at your option) any later version.------ This program is distributed in the hope that it will be useful,--- but WITHOUT ANY WARRANTY; without even the implied warranty of--- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the--- GNU General Public License for more details.------ You should have received a copy of the GNU General Public License--- along with this program; if not, write to the Free Software--- Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.-----module CGrep.Types (- Offset- , Text8- , LText8-) where--import Data.ByteString.Char8 as C ( ByteString )-import Data.ByteString.Lazy.Char8 as LC ( ByteString )-import Data.Int ( Int64 )--type Offset = Int64-type Text8 = C.ByteString-type LText8 = LC.ByteString
src/CmdOptions.hs view
@@ -1,5 +1,5 @@ ----- Copyright (c) 2013-2023 Nicola Bonelli <nicola@larthia.com>+-- Copyright (c) 2013-2025 Nicola Bonelli <nicola@larthia.com> -- -- This program is free software; you can redistribute it and/or modify -- it under the terms of the GNU General Public License as published by@@ -17,73 +17,241 @@ -- module CmdOptions (- options+ parserInfo, ) where -import Data.Version(showVersion)-import System.Console.CmdArgs- ( (&=),- cmdArgsMode,- args,- explicit,- groupname,- help,- name,- program,- summary,- typ,- Mode,- CmdArgs )+import Options.Applicative (+ Parser,+ ParserInfo,+ argument,+ auto,+ fullDesc,+ header,+ help,+ helper,+ info,+ long,+ many,+ metavar,+ optional,+ option,+ progDesc,+ short,+ str,+ strOption,+ switch,+ value,+ (<**>),+ ) -import Paths_cgrep ( version )-import Options ( Options(..) )+import Options (Options (..))+import Paths_cgrep (version)+import Options.Applicative.Builder (infoOption)+import Options.Applicative (hidden)+import Data.Version (showVersion) -options :: Mode (CmdArgs Options)-options = cmdArgsMode $ Options- { file = "" &= typ "FILE" &= groupname "Pattern" &= help "Read PATTERNs from file (one per line)"- , word_match = False &= help "Force word matching" &=explicit &= name "word" &= name "w"- , prefix_match = False &= help "Force prefix matching" &=explicit &= name "prefix" &= name "p"- , suffix_match = False &= help "Force suffix matching" &=explicit &= name "suffix" &= name "s"- , edit_dist = False &= help "Use edit distance" &=explicit &= name "edit" &= name "e"- , regex_posix = False &= help "Use regex matching (posix)" &= explicit &= name "G" &=name "regex"- , regex_pcre = False &= help "Use regex matching (pcre)" &= explicit &= name "P" &=name "pcre"- , ignore_case = False &= help "Ignore case distinctions"- , code = False &= groupname "\nContext filters" &= help "Enable search in source code" &= explicit &= name "c" &= name "code"- , comment = False &= help "Enable search in comments" &= explicit &= name "m" &= name "comment"- , literal = False &= help "Enable search in string literals" &= explicit &= name "l" &= name "literal"- , identifier = False &= groupname "\nToken filters" &= help "Identifiers" &= explicit &= name "identifier" &= name "name"- , nativeType = False &= help "Native Types" &= explicit &= name "native" &= name "type"- , keyword = False &= help "Keywords" &= explicit &= name "keyword"- , number = False &= help "Literal numbers" &= explicit &= name "number"- , string = False &= help "Literal strings" &= explicit &= name "string"- , operator = False &= help "Operators" &= explicit &= name "op"- , semantic = False &= groupname "\nSemantic" &= help "\"code\" pattern: _, _1, _2... (identifiers), $, $1, $2... (optionals), ANY, KEY, STR, LIT, NUM, HEX, OCT, OR" &= explicit &= name "S" &= name "semantic"- , max_count = maxBound &= groupname "\nOutput control" &= help "Stop search in files after INT matches" &= explicit &= name "max-count"- , type_filter = [] &= help "Specify file types. ie: Cpp, +Haskell, -Makefile"- , kind_filter = [] &= help "Specify file kinds. Text, Config, Language, Data, Markup or Script"- , type_force = Nothing &= help "Force the type of file" &= explicit &= name "force-type"- , type_map = False &= help "List the supported file types" &= explicit &= name "type-list"- , invert_match = False &= help "Select non-matching lines" &= explicit &= name "invert-match" &= name "v"- , multiline = 1 &= help "Enable multi-line matching"- , recursive = False &= help "Enable recursive search (don't follow symlinks)" &= explicit &= name "recursive" &= name "r"- , skip_test = False &= help "Skip files that have 'test' in the name" &= explicit &= name "skip-test" &= name "T"- , prune_dir = [] &= help "Do not descend into dir" &= explicit &= name "prune-dir"- , follow = False &= help "Follow symlinks" &= explicit &= name "follow" &= name "L"- , show_match = False &= groupname "\nOutput format" &= help "Show list of matching tokens" &= explicit &= name "show-match"- , color = False &= help "Use colors to highlight the match strings" &= explicit &= name "color"- , no_color = False &= help "Do not use colors (override config file)" &= explicit &= name "no-color"- , no_filename = False &= help "Suppress the file name prefix on output" &= explicit &= name "h" &= name "no-filename"- , no_numbers = False &= help "Suppress both line and column numbers on output" &= explicit &= name "no-numbers"- , no_column = False &= help "Suppress the column number on output" &= explicit &= name "no-column"- , count = False &= help "Print only a count of matching lines per file" &= explicit &= name "count"- , filename_only = False &= help "Print only the name of files containing matches" &= explicit &= name "filename-only"- , vim = False &= help "Run vim editor passing the files that match" &= explicit &=name "vim"- , editor = False &= help "Run the editor specified by EDITOR var., passing the files that match" &= explicit &=name "editor"- , fileline = False &= help "When edit option is specified, pass the list of matching files in file:line format (e.g. vim 'file-line' plugin)" &= explicit &=name "fileline"- , json = False &= help "Format output as json object" &= explicit &= name "json"- , jobs = Nothing &= groupname "\nConcurrency" &= help "Number threads to run in parallel" &= explicit &= name "threads" &= name "j"- , verbose = 0 &= groupname "\nMiscellaneous" &= help "Verbose level: 1, 2 or 3" &= explicit &= name "verbose"- , no_shallow = False &= help "Disable shallow-search" &= explicit &= name "no-shallow"- , show_palette = False &= help "Show color palette" &= explicit &= name "palette"- , others = [] &= args- } &= summary ("Cgrep " <> showVersion version <> ". Usage: cgrep [OPTION] [PATTERN] files...") &= program "cgrep"+-- Parser per le opzioni+options :: Parser Options+options = Options+ <$> optional(strOption+ ( long "file"+ <> metavar "FILE"+ <> help "Read PATTERNs from file (one per line)" ))+ <*> switch+ ( long "word"+ <> short 'w'+ <> help "Force word matching" )+ <*> switch+ ( long "prefix"+ <> short 'p'+ <> help "Force prefix matching" )+ <*> switch+ ( long "suffix"+ <> short 's'+ <> help "Force suffix matching" )+ <*> switch+ ( long "edit"+ <> short 'e'+ <> help "Use edit distance" )+ <*> switch+ ( long "regex"+ <> short 'G'+ <> help "Use regex matching (posix)" )+#ifdef ENABLE_PCRE+ <*> switch+ ( long "pcre"+ <> short 'P'+ <> help "Use regex matching (pcre)" )+#endif+ <*> switch+ ( long "ignore-case"+ <> short 'i'+ <> help "Ignore case distinctions" )+ -- Context filters+ <*> switch+ ( long "code"+ <> short 'c'+ <> help "Enable search in source code" )+ <*> switch+ ( long "comment"+ <> short 'm'+ <> help "Enable search in comments" )+ <*> switch+ ( long "literal"+ <> short 'l'+ <> help "Enable search in string literals" )+ -- Token filters+ <*> switch+ ( long "identifier"+ <> long "name"+ <> help "Identifiers" )+ <*> switch+ ( long "native"+ <> long "type"+ <> help "Native Types" )+ <*> switch+ ( long "keyword"+ <> help "Keywords" )+ <*> switch+ ( long "number"+ <> help "Literal numbers" )+ <*> switch+ ( long "string"+ <> help "Literal strings" )+ <*> switch+ ( long "op"+ <> help "Operators" )+ -- File filters+ <*> many (strOption+ ( long "type"+ <> metavar "TYPE"+ <> help "Specify file types. ie: Cpp, +Haskell, -Makefile" ))+ <*> many (strOption+ ( long "kind"+ <> metavar "KIND"+ <> help "Specify file kinds. Text, Config, Language, Data, Markup or Script" ))+ <*> switch+ ( long "code-only"+ <> help "Parse code modules only (skip headers/interfaces)" )+ <*> switch+ ( long "hdr-only"+ <> help "Parse headers/interfaces only (skip modules)" )+ <*> optional (option auto+ ( long "tests"+ <> short 'T'+ <> help "Filter tests: 'True' tests only, 'False' code only, omitted (search all)" ))+ <*> many (strOption+ ( long "prune-dir"+ <> metavar "DIR"+ <> help "Do not descend into dir" ))+ <*> switch+ ( long "recursive"+ <> short 'r'+ <> help "Enable recursive search (don't follow symlinks)" )+ <*> switch+ ( long "follow"+ <> short 'L'+ <> help "Follow symlinks" )+ -- Semantic+ <*> switch+ ( long "semantic"+ <> short 'S'+ <> help "\"code\" pattern: _, _1, _2... (identifiers), ANY, KEY, STR, LIT, NUM, HEX, OCT" )+ <*> switch+ ( long "strict"+ <> help "Enable strict semantic for operators" )+ -- Control+ <*> option auto+ ( long "max-count"+ <> metavar "INT"+ <> value maxBound+ <> help "Stop search in files after INT matches" )+ <*> optional (strOption+ ( long "force-type"+ <> metavar "TYPE"+ <> help "Force the type of file" ))+ <*> switch+ ( long "type-list"+ <> help "List the supported file types" )+ <*> switch+ ( long "invert-match"+ <> short 'v'+ <> help "Select non-matching lines" )+ <*> optional (option auto+ ( long "threads"+ <> short 'j'+ <> metavar "INT"+ <> help "Approximate number of threads to run search" ))+ -- Output format+ <*> switch+ ( long "show-match"+ <> help "Show list of matching tokens" )+ <*> switch+ ( long "color"+ <> help "Use colors to highlight the match strings" )+ <*> switch+ ( long "no-color"+ <> help "Do not use colors (override config file)" )+ <*> switch+ ( long "no-filename"+ <> short 'h'+ <> help "Suppress the file name prefix on output" )+ <*> switch+ ( long "no-numbers"+ <> help "Suppress both line and column numbers on output" )+ <*> switch+ ( long "no-column"+ <> help "Suppress the column number on output" )+ <*> switch+ ( long "count"+ <> help "Print only a count of matching lines per file" )+ <*> switch+ ( long "filename-only"+ <> help "Print only the name of files containing matches" )+ <*> switch+ ( long "json"+ <> help "Format output as json object" )+ <*> switch+ ( long "vim"+ <> help "Run vim editor passing the files that match" )+ <*> switch+ ( long "editor"+ <> help "Run the editor specified by EDITOR var., passing the files that match" )+ <*> switch+ ( long "fileline"+ <> help "When edit option is specified, pass the list of matching files in file:line format (e.g. vim 'file-line' plugin)" )+ -- Miscellaneous+ <*> switch+ ( long "verbose"+ <> help "Enable verbose mode"+ )+ <*> switch+ ( long "stats"+ <> help "Print statistics about the search"+ )+ <*> option auto+ ( long "debug"+ <> metavar "INT"+ <> value 0+ -- <> help "debug level: 1, 2, 3 or 4"+ <> hidden)+ <*> switch+ ( long "null-output"+ <> help "Disable output for performance evaluation" )+ <*> switch+ ( long "palette"+ <> help "Show color palette" )+ <*> many (argument str (metavar "PATTERN FILES..."))++parseVersion :: Parser (a -> a)+parseVersion = infoOption ("cgrep " <> showVersion version)+ ( long "version"+ <> help "Show version information and exit"+ -- <> hidden -- Uncomment to hide it from the main help message+ )++parserInfo :: ParserInfo Options+parserInfo = info (options <**> helper <**> parseVersion)+ ( fullDesc+ <> progDesc "Context-aware grep for source codes"+ <> header ("cgrep " <> showVersion version <> " - Usage: cgrep [OPTION] [PATTERN] files...") )
src/Config.hs view
@@ -1,5 +1,5 @@ ----- Copyright (c) 2013-2023 Nicola Bonelli <nicola@larthia.com>+-- Copyright (c) 2013-2025 Nicola Bonelli <nicola@larthia.com> -- -- This program is free software; you can redistribute it and/or modify -- it under the terms of the GNU General Public License as published by@@ -16,146 +16,125 @@ -- Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. -- -{-# LANGUAGE DeriveGeneric #-}- module Config (- Config(..)- , dumpPalette- , getConfig+ Config (..),+ dumpPalette,+ getConfig, ) where -import Control.Monad ( MonadPlus(mzero), filterM, forM_ )-import System.Directory ( doesFileExist, getHomeDirectory )-import System.Console.ANSI- ( Color(White, Red, Green, Yellow, Blue, Magenta, Cyan),- ColorIntensity(Vivid),- ConsoleIntensity(BoldIntensity),- ConsoleLayer(Foreground),- SGR(SetColor, SetConsoleIntensity), setSGRCode )--import System.Console.ANSI.Types- ( SGR(SetPaletteColor, SetColor, SetConsoleIntensity),- xterm6LevelRGB,- Color(White, Red, Green, Yellow, Blue, Magenta, Cyan),- ColorIntensity(Vivid),- ConsoleIntensity(BoldIntensity),- ConsoleLayer(Foreground) )+import Control.Monad (MonadPlus (mzero), filterM, forM_)+import System.Console.ANSI.Types (ConsoleIntensity (BoldIntensity), xterm6LevelRGB)+import System.Directory (doesFileExist, getHomeDirectory) -import qualified Data.Yaml as Y-import Data.Aeson ( (.!=), (.:?), FromJSON(parseJSON) )-import Data.Maybe ( fromMaybe, mapMaybe )+import Data.Aeson (FromJSON (parseJSON), (.!=), (.:?))+import Data.Maybe (fromMaybe, listToMaybe, mapMaybe)+import qualified Data.Yaml as Y -import GHC.Generics ( Generic )-import CGrep.FileType ( FileType )+import CGrep.FileType (FileType)+import GHC.Generics (Generic) -import Data.List.Split ( splitOn )-import qualified Data.ByteString as B+import Data.List.Split (splitOn) import System.FilePath ((</>))-import Data.ByteString.RawFilePath (RawFilePath)-import qualified Data.ByteString.Char8 as C -import Data.List.Extra (notNull) import CGrep.FileKind (FileKind)+import System.Console.ANSI (Color (..), ColorIntensity (..), ConsoleLayer (..), SGR (..))+import System.Console.ANSI.Codes (setSGRCode) import Text.Read (readMaybe) cgreprc :: FilePath cgreprc = "cgreprc" - data Config = Config- { configFileTypes :: [FileType]- , configFileKinds :: [FileKind]- , configPruneDirs :: [RawFilePath]- , configColors :: Bool- , configColorFile :: [SGR]- , configColorMatch :: [SGR]- , configFileLine :: Bool- , configJobs :: Maybe Int- } deriving stock (Show, Read)-+ { configFileTypes :: [FileType]+ , configFileKinds :: [FileKind]+ , configPruneDirs :: [String]+ , configColors :: Bool+ , configColorFile :: [SGR]+ , configColorMatch :: [SGR]+ , configFileLine :: Bool+ , configJobs :: Maybe Int+ }+ deriving stock (Show, Read) defaultConfig :: Config-defaultConfig = Config- { configFileTypes = []- , configFileKinds = []- , configPruneDirs = []- , configColors = False- , configColorFile = [SetConsoleIntensity BoldIntensity, SetColor Foreground Vivid Blue]- , configColorMatch = [SetConsoleIntensity BoldIntensity]- , configFileLine = False- , configJobs = Nothing- }-+defaultConfig =+ Config+ { configFileTypes = []+ , configFileKinds = []+ , configPruneDirs = []+ , configColors = False+ , configColorFile = [SetConsoleIntensity BoldIntensity, SetColor Foreground Vivid Blue]+ , configColorMatch = [SetConsoleIntensity BoldIntensity]+ , configFileLine = False+ , configJobs = Nothing+ } mkConfig :: YamlConfig -> Config mkConfig YamlConfig{..} =- let configFileTypes = mapMaybe readMaybe yamlFileTypes- configFileKinds = mapMaybe readMaybe yamlFileKinds- configPruneDirs = C.pack <$> yamlPruneDirs- configColors = yamlColors- configColorFile = fromMaybe [] (yamlColorFileName >>= readColor)- configColorMatch = fromMaybe [] (yamlColorMatch >>= readColor)- configFileLine = yamlFileLine- configJobs = yamlJobs- in Config {..}-+ let configFileTypes = mapMaybe readMaybe yamlFileTypes+ configFileKinds = mapMaybe readMaybe yamlFileKinds+ configPruneDirs = yamlPruneDirs+ configColors = yamlColors+ configColorFile = fromMaybe [] (yamlColorFileName >>= readColor)+ configColorMatch = fromMaybe [] (yamlColorMatch >>= readColor)+ configFileLine = yamlFileLine+ configJobs = yamlJobs+ in Config{..} data YamlConfig = YamlConfig- { yamlFileTypes :: [String]- , yamlFileKinds :: [String]- , yamlPruneDirs :: [String]- , yamlColors :: Bool- , yamlColorFileName :: Maybe String- , yamlColorMatch :: Maybe String- , yamlFileLine :: Bool- , yamlJobs :: Maybe Int- } deriving stock (Show, Generic)-+ { yamlFileTypes :: [String]+ , yamlFileKinds :: [String]+ , yamlPruneDirs :: [String]+ , yamlColors :: Bool+ , yamlColorFileName :: Maybe String+ , yamlColorMatch :: Maybe String+ , yamlFileLine :: Bool+ , yamlJobs :: Maybe Int+ }+ deriving stock (Show, Generic) instance Y.FromJSON YamlConfig where- parseJSON (Y.Object v) =- YamlConfig <$> v .:? "file_types" .!= []- <*> v .:? "file_kinds" .!= []- <*> v .:? "prune_dirs" .!= []- <*> v .:? "colors" .!= False- <*> v .:? "color_filename" .!= Nothing- <*> v .:? "color_match" .!= Nothing- <*> v .:? "file_line" .!= False- <*> v .:? "threads" .!= Nothing- parseJSON _ = mzero-+ parseJSON (Y.Object v) =+ YamlConfig+ <$> v .:? "file_types" .!= []+ <*> v .:? "file_kinds" .!= []+ <*> v .:? "prune_dirs" .!= []+ <*> v .:? "colors" .!= False+ <*> v .:? "color_filename" .!= Nothing+ <*> v .:? "color_match" .!= Nothing+ <*> v .:? "file_line" .!= False+ <*> v .:? "threads" .!= Nothing+ parseJSON _ = mzero getConfig :: IO (Config, Maybe FilePath) getConfig = do- home <- getHomeDirectory- confs <- filterM doesFileExist [cgreprc, "." <> cgreprc, home </> "." <> cgreprc, "/etc" </> cgreprc]- if notNull confs- then do- conf <- Y.decodeFileEither (head confs)+ home <- getHomeDirectory+ confs <- filterM doesFileExist ["." <> cgreprc, home </> "." <> cgreprc, "/etc" </> cgreprc]+ case (listToMaybe confs) of+ Just fp -> do+ conf <- Y.decodeFileEither fp case conf of- Left e -> errorWithoutStackTrace $ Y.prettyPrintParseException e- Right yconf -> return (mkConfig yconf, Just (head confs))- else return (defaultConfig, Nothing)-+ Left e -> errorWithoutStackTrace $ "CGrep:" <> Y.prettyPrintParseException e+ Right yconf -> return (mkConfig yconf, Just fp)+ Nothing -> return (defaultConfig, Nothing) readColor :: String -> Maybe [SGR]-readColor "Bold" = Just [SetConsoleIntensity BoldIntensity]-readColor "Red" = Just [SetConsoleIntensity BoldIntensity, SetColor Foreground Vivid Red]-readColor "Green" = Just [SetConsoleIntensity BoldIntensity, SetColor Foreground Vivid Green]-readColor "Yellow" = Just [SetConsoleIntensity BoldIntensity, SetColor Foreground Vivid Yellow]-readColor "Blue" = Just [SetConsoleIntensity BoldIntensity, SetColor Foreground Vivid Blue]-readColor "Magenta" = Just [SetConsoleIntensity BoldIntensity, SetColor Foreground Vivid Magenta]-readColor "Cyan" = Just [SetConsoleIntensity BoldIntensity, SetColor Foreground Vivid Cyan]-readColor "White" = Just [SetConsoleIntensity BoldIntensity, SetColor Foreground Vivid White]-readColor "Orange" = Just [SetConsoleIntensity BoldIntensity, SetPaletteColor Foreground $ xterm6LevelRGB 5 2 0]-readColor "Acqua" = Just [SetConsoleIntensity BoldIntensity, SetPaletteColor Foreground $ xterm6LevelRGB 2 5 4]-readColor xs = case splitOn ":" xs of- [r, g, b] -> Just [SetConsoleIntensity BoldIntensity, SetPaletteColor Foreground $ xterm6LevelRGB (read r) (read g) (read b)]- _ -> Nothing-+readColor "Bold" = Just [SetConsoleIntensity BoldIntensity]+readColor "Red" = Just [SetConsoleIntensity BoldIntensity, SetColor Foreground Vivid Red]+readColor "Green" = Just [SetConsoleIntensity BoldIntensity, SetColor Foreground Vivid Green]+readColor "Yellow" = Just [SetConsoleIntensity BoldIntensity, SetColor Foreground Vivid Yellow]+readColor "Blue" = Just [SetConsoleIntensity BoldIntensity, SetColor Foreground Vivid Blue]+readColor "Magenta" = Just [SetConsoleIntensity BoldIntensity, SetColor Foreground Vivid Magenta]+readColor "Cyan" = Just [SetConsoleIntensity BoldIntensity, SetColor Foreground Vivid Cyan]+readColor "White" = Just [SetConsoleIntensity BoldIntensity, SetColor Foreground Vivid White]+readColor "Orange" = Just [SetConsoleIntensity BoldIntensity, SetPaletteColor Foreground $ xterm6LevelRGB 5 2 0]+readColor "Acqua" = Just [SetConsoleIntensity BoldIntensity, SetPaletteColor Foreground $ xterm6LevelRGB 2 5 4]+readColor xs = case splitOn ":" xs of+ [r, g, b] -> Just [SetConsoleIntensity BoldIntensity, SetPaletteColor Foreground $ xterm6LevelRGB (read r) (read g) (read b)]+ _ -> Nothing dumpPalette :: IO () dumpPalette = do- let palette = [(r, g, b) | r <- [0..5], g <- [0..5], b <- [0..5]]- forM_ palette $ \(r, g, b) -> do- putStrLn $ setSGRCode [SetConsoleIntensity BoldIntensity, SetPaletteColor Foreground $ xterm6LevelRGB r g b] <> "COLOR " <> show r <> ":" <> show g <> ":" <> show b <> setSGRCode []+ let palette = [(r, g, b) | r <- [0 .. 5], g <- [0 .. 5], b <- [0 .. 5]]+ forM_ palette $ \(r, g, b) -> do+ putStrLn $ setSGRCode [SetConsoleIntensity BoldIntensity, SetPaletteColor Foreground $ xterm6LevelRGB r g b] <> "COLOR " <> show r <> ":" <> show g <> ":" <> show b <> setSGRCode []
src/Main.hs view
@@ -1,5 +1,5 @@ ----- Copyright (c) 2013-2023 Nicola Bonelli <nicola@larthia.com>+-- Copyright (c) 2013-2025 Nicola Bonelli <nicola@larthia.com> -- -- This program is free software; you can redistribute it and/or modify -- it under the terms of the GNU General Public License as published by@@ -16,66 +16,64 @@ -- Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. -- -module Main where--import qualified Data.ByteString.Char8 as C-import qualified Codec.Binary.UTF8.String as UC+module Main (main) where -import Data.List ( isSuffixOf, (\\), isInfixOf, nub, sort, union, isPrefixOf, genericLength, partition, elemIndex )-import Data.Maybe ( catMaybes )-import Data.Char ( toLower )-import Data.Version ( showVersion )+import Data.Char (toLower) -import Control.Monad.Trans.Reader ( ReaderT(runReaderT), ask )-import Control.Monad ( when, void )+import Control.Monad (void, when)+import Control.Monad.Trans.Reader (ReaderT (runReaderT)) import qualified Data.Map as M-import GHC.Conc ( getNumCapabilities, setNumCapabilities )-import GHC.IO.Handle ( hIsTerminalDevice )+import GHC.Conc (getNumCapabilities, setNumCapabilities)+import GHC.IO.Handle (hIsTerminalDevice) -import System.IO ( stdin, stdout, stderr )-import System.Console.CmdArgs ( cmdArgsRun )-import System.Exit ( exitSuccess )-import System.Environment ( withArgs )+import Options.Applicative (execParser)+import System.Environment (withArgs)+import System.Exit (exitSuccess)+import System.IO (stdin, stdout) -import CGrep.FileType ( readTypeList, readKindList )-import CGrep.FileTypeMap ( dumpFileTypeInfoMap, fileTypeInfoMap )-import CGrep.Parser.Atom ( wildCardMap )-import CGrep.Common ( trim8 )+import CGrep.Common (trimT)+import CGrep.FileType (readKindList, readTypeList)+import CGrep.FileTypeMapTH (dumpFileTypeInfoMap, fileTypeInfoMap)+import CGrep.Parser.Atom (wildCardMap)+import CGrep.Search (isRegexp, startSearch) -import Verbose ( putMsgLnVerbose )-import Paths_cgrep ( version )-import CmdOptions ( options )-import Options ( Options(..) )-import Config- ( dumpPalette, getConfig, Config(configFileTypes, configColors, configJobs, configFileKinds) )+import CmdOptions (parserInfo)+import Config (+ Config (configColors, configFileKinds, configFileTypes, configJobs),+ dumpPalette,+ getConfig,+ )+import Options (Options (..)) -import Util ( partitionM)-import Reader ( ReaderIO, Env (..) )-import Search ( parallelSearch, isRegexp )-import System.Posix.FilePath (RawFilePath)+import Reader (Env (..)) -import Data.List.Extra (notNull)-import Data.Functor ( ($>), void )-import Control.Applicative ( Alternative((<|>)) )+import Control.Applicative (Alternative ((<|>)))+import Data.Functor (($>))+import qualified Data.Text as T+import qualified Data.Text.IO as TIO +import Data.List (union, (\\))+import qualified OsPath as OS+import System.OsPath (OsPath)+import qualified System.OsString as OS+ main :: IO () main = do -- check whether this is a terminal device- isTermIn <- hIsTerminalDevice stdin+ isTermIn <- hIsTerminalDevice stdin isTermOut <- hIsTerminalDevice stdout -- read config options (conf, _) <- getConfig -- read command-line options- opt@Options{..} <- (if isTermOut- then \o -> o { color = color o || configColors conf }- else id) <$> cmdArgsRun options-- -- check for multiple backends...- when (length (catMaybes [ if json then Just "" else Nothing ]) > 1) $- error "Cgrep: you can use one back-end at time!"+ opt@Options{..} <-+ ( if isTermOut+ then \o -> o{color = color o || configColors conf}+ else id+ )+ <$> execParser parserInfo -- display lang-map and exit... when type_map $@@ -87,26 +85,24 @@ -- check whether the pattern list is empty, display help message if it's the case when (null others && isTermIn && null file) $- withArgs ["--help"] $ void (cmdArgsRun options)-- let others' = C.pack <$> others-- -- load patterns- patterns <- if null file then pure $ readPatternsFromCommandLine others'- else readPatternsFromFile (C.pack file)+ withArgs ["--help"] $+ void (execParser parserInfo) - let patterns' = map (if ignore_case then ic else id) patterns- where ic | (not . isRegexp) opt && semantic = C.unwords . map (\p -> if p `elem` wildCardTokens then p else C.map toLower p) . C.words- | otherwise = C.map toLower- where wildCardTokens = "OR" : M.keys wildCardMap -- "OR" is not included in wildCardMap+ let others' = T.pack <$> others - -- display the configuration in use+ -- load patterns and filepaths: - -- when (isJust confpath) $- -- hPutStrLn stderr $ showBold opt ("Using '" <> fromJust confpath <> "' configuration file...")+ (patterns, paths) <- case file of+ Nothing -> do+ pure $ splitPatternsAndFiles others'+ Just f -> do+ readPatternsFromFile (OS.unsafeEncodeUtf f) >>= \ps -> pure (ps, OS.fromText <$> others') - -- load files to parse:- let paths = getFilePaths (notNull file) others'+ let patterns' = map (if ignore_case then ic else id) patterns+ where+ ic+ | (not . isRegexp) opt && semantic = T.unwords . map (\p -> if p `elem` (M.keys wildCardMap) then p else T.map toLower p) . T.words+ | otherwise = T.map toLower -- parse cmd line language list: let (l0, l1, l2) = readTypeList type_filter@@ -115,35 +111,20 @@ let types = (if null l0 then configFileTypes conf else l0 `union` l1) \\ l2 kinds = if null kind_filter then configFileKinds conf else readKindList kind_filter -- runReaderT (do- putMsgLnVerbose 1 stderr $ "cgrep " <> showVersion version <> "!"- putMsgLnVerbose 1 stderr $ "File types: " <> show type_filter- putMsgLnVerbose 1 stderr $ "File kinds: " <> show kinds- ) (Env conf opt)- -- specify number of cores cap <- case jobs <|> configJobs conf of- (Just j) -> setNumCapabilities (j+1) $> j- Nothing -> getNumCapabilities+ (Just j) -> setNumCapabilities (j + 1) $> j+ Nothing -> getNumCapabilities -- run search- runReaderT (parallelSearch paths patterns' types kinds isTermIn) (Env conf opt {jobs = Just cap})---readPatternsFromFile :: RawFilePath -> IO [C.ByteString]-readPatternsFromFile "" = return []-readPatternsFromFile f = map trim8 . C.lines <$> C.readFile (C.unpack f)---readPatternsFromCommandLine :: [C.ByteString] -> [C.ByteString]-readPatternsFromCommandLine [] = []-readPatternsFromCommandLine xs | ":" `elem` xs = takeWhile (/= ":") xs- | otherwise = [ head xs ]+ runReaderT (startSearch paths patterns' types kinds isTermIn) (Env conf opt{jobs = Just cap}) +readPatternsFromFile :: OsPath -> IO [T.Text]+readPatternsFromFile f+ | OS.null f = return []+ | otherwise = map trimT . T.lines <$> TIO.readFile (OS.toFilePath f) -getFilePaths :: Bool -> [RawFilePath] -> [RawFilePath]-getFilePaths False xs = case ":" `elemIndex` xs of- Nothing -> if null xs then [] else tail xs- (Just n) -> drop (n+1) xs-getFilePaths True xs = xs+splitPatternsAndFiles :: [T.Text] -> ([T.Text], [OsPath])+splitPatternsAndFiles [] = ([], [])+splitPatternsAndFiles [x] = ([x], [])+splitPatternsAndFiles (x : xs) = ([x], OS.fromText <$> xs)
src/Options.hs view
@@ -1,5 +1,5 @@ ----- Copyright (c) 2013-2023 Nicola Bonelli <nicola@larthia.com>+-- Copyright (c) 2013-2025 Nicola Bonelli <nicola@larthia.com> -- -- This program is free software; you can redistribute it and/or modify -- it under the terms of the GNU General Public License as published by@@ -16,65 +16,69 @@ -- Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. -- -{-# LANGUAGE DeriveDataTypeable #-}--module Options where--import Data.Data ( Data, Typeable )+module Options (+ Options (..),+) where data Options = Options- -- Pattern:- { file :: String- , word_match :: Bool- , prefix_match :: Bool- , suffix_match :: Bool- , edit_dist :: Bool- , regex_posix :: Bool- , regex_pcre :: Bool- , ignore_case :: Bool- -- Context:- , code :: Bool- , comment :: Bool- , literal :: Bool- -- Tokenizer:- , identifier :: Bool- , nativeType :: Bool- , keyword :: Bool- , number :: Bool- , string :: Bool- , operator :: Bool- -- Semantic:- , semantic :: Bool- -- Output control:- , max_count :: Int- , type_filter :: [String]- , kind_filter :: [String]- , type_force :: Maybe String- , type_map :: Bool- , invert_match :: Bool- , multiline :: Int- , recursive :: Bool- , skip_test :: Bool- , prune_dir :: [FilePath]- , follow :: Bool- -- Output format:- , show_match :: Bool- , color :: Bool- , no_color :: Bool- , no_filename :: Bool- , no_numbers :: Bool- , no_column :: Bool- , count :: Bool- , filename_only :: Bool- , json :: Bool- , vim :: Bool- , editor :: Bool- , fileline :: Bool- -- Parallel:- , jobs :: Maybe Int- -- Misc:- , verbose :: Int- , no_shallow :: Bool- , show_palette :: Bool- , others :: [String]- } deriving stock (Data, Typeable, Show)+ { file :: Maybe String+ , word_match :: Bool+ , prefix_match :: Bool+ , suffix_match :: Bool+ , edit_dist :: Bool+ , regex_posix :: Bool+#ifdef ENABLE_PCRE+ , regex_pcre :: Bool+#endif+ , ignore_case :: Bool+ , -- Context:+ code :: Bool+ , comment :: Bool+ , literal :: Bool+ , -- Token filters:+ identifier :: Bool+ , nativeType :: Bool+ , keyword :: Bool+ , number :: Bool+ , string :: Bool+ , operator :: Bool+ , -- File filters:+ type_filter :: [String]+ , kind_filter :: [String]+ , code_only :: Bool+ , hdr_only :: Bool+ , tests :: Maybe Bool+ , prune_dir :: [FilePath]+ , recursive :: Bool+ , follow :: Bool+ , -- Semantic:+ semantic :: Bool+ , strict :: Bool+ , -- Control:+ max_count :: Int+ , force_type :: Maybe String+ , type_map :: Bool+ , invert_match :: Bool+ , jobs :: Maybe Int+ , -- Output format:+ show_match :: Bool+ , color :: Bool+ , no_color :: Bool+ , no_filename :: Bool+ , no_numbers :: Bool+ , no_column :: Bool+ , count :: Bool+ , filename_only :: Bool+ , json :: Bool+ , vim :: Bool+ , editor :: Bool+ , fileline :: Bool+ , -- Misc:+ verbose :: Bool+ , stats :: Bool+ , debug :: Int+ , null_output :: Bool+ , show_palette :: Bool+ , others :: [String]+ }+ deriving stock (Show)
+ src/OsPath.hs view
@@ -0,0 +1,52 @@+--+-- Copyright (c) 2013-2025 Nicola Bonelli <nicola@larthia.com>+--+-- This program is free software; you can redistribute it and/or modify+-- it under the terms of the GNU General Public License as published by+-- the Free Software Foundation; either version 2 of the License, or+-- (at your option) any later version.+--+-- This program is distributed in the hope that it will be useful,+-- but WITHOUT ANY WARRANTY; without even the implied warranty of+-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the+-- GNU General Public License for more details.+--+-- You should have received a copy of the GNU General Public License+-- along with this program; if not, write to the Free Software+-- Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.+--+--+module OsPath (+ fromByteString,+ fromText,+ toShortByteString,+ toText,+ toFilePath,+) where++import Data.ByteString (ByteString)+import Data.ByteString.Short (ShortByteString, toShort)+import qualified Data.Text as T+import qualified Data.Text.Encoding as TE+import System.OsString.Data.ByteString.Short (fromShort)+import System.OsString.Internal.Types++toText :: OsString -> T.Text+toText = TE.decodeUtf8Lenient . fromShort . getPosixString . getOsString+{-# INLINE toText #-}++toShortByteString :: OsString -> ShortByteString+toShortByteString = getPosixString . getOsString+{-# INLINE toShortByteString #-}++toFilePath :: OsString -> FilePath+toFilePath = show . getPosixString . getOsString+{-# INLINE toFilePath #-}++fromByteString :: ByteString -> OsString+fromByteString bs = OsString $ PosixString (toShort bs)+{-# INLINE fromByteString #-}++fromText :: T.Text -> OsString+fromText = OsString . PosixString . toShort . TE.encodeUtf8+{-# INLINE fromText #-}
+ src/PutMessage.hs view
@@ -0,0 +1,79 @@+--+-- Copyright (c) 2013-2025 Nicola Bonelli <nicola@larthia.com>+--+-- This program is free software; you can redistribute it and/or modify+-- it under the terms of the GNU General Public License as published by+-- the Free Software Foundation; either version 2 of the License, or+-- (at your option) any later version.+--+-- This program is distributed in the hope that it will be useful,+-- but WITHOUT ANY WARRANTY; without even the implied warranty of+-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the+-- GNU General Public License for more details.+--+-- You should have received a copy of the GNU General Public License+-- along with this program; if not, write to the Free Software+-- Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.+--++module PutMessage (+ PutStr (..),+ putMessageLnVerb,+ putMessageLn,+ putMessage,+) where++import Control.Concurrent (MVar)+import Control.Concurrent.MVar (withMVar)+import Control.Monad (when)+import Control.Monad.IO.Class (MonadIO (liftIO))+import Control.Monad.Trans.Reader (reader)+import Data.String (IsString)+import qualified Data.Text as T+import qualified Data.Text.IO as TIO+import qualified Data.Text.Lazy as TL+import qualified Data.Text.Lazy.IO as TLIO+import GHC.IO.Handle (Handle)+import Options (Options (..))+import Reader (Env (..), ReaderIO)+import System.IO (hPutStr, hPutStrLn)++class (IsString a) => PutStr a where+ putMsgLnLock :: MVar () -> Handle -> a -> IO ()+ putMsgLock :: MVar () -> Handle -> a -> IO ()++instance PutStr String where+ putMsgLnLock lock h msg = withMVar lock (\_ -> hPutStrLn h msg)+ {-# INLINE putMsgLnLock #-}+ putMsgLock lock h msg = withMVar lock (\_ -> hPutStr h msg)+ {-# INLINE putMsgLock #-}++instance PutStr T.Text where+ putMsgLnLock lock h msg = withMVar lock (\_ -> TIO.hPutStrLn h msg)+ {-# INLINE putMsgLnLock #-}+ putMsgLock lock h msg = withMVar lock (\_ -> TIO.hPutStr h msg)+ {-# INLINE putMsgLock #-}++instance PutStr TL.Text where+ putMsgLnLock lock h msg = withMVar lock (\_ -> TLIO.hPutStrLn h msg)+ {-# INLINE putMsgLnLock #-}+ putMsgLock lock h msg = withMVar lock (\_ -> TLIO.hPutStr h msg)+ {-# INLINE putMsgLock #-}++putMessageLnVerb :: (PutStr a) => Int -> MVar () -> Handle -> a -> ReaderIO ()+putMessageLnVerb lvl lock h xs = do+ n <- reader $ debug . opt+ when (n >= lvl) $+ liftIO $+ putMsgLnLock lock h xs+{-# INLINE putMessageLnVerb #-}++putMessageLn :: (PutStr a, MonadIO m) => MVar () -> Handle -> a -> m ()+putMessageLn lock h xs =+ liftIO $ putMsgLnLock lock h xs+{-# INLINE putMessageLn #-}++putMessage :: (PutStr a, MonadIO m) => MVar () -> Handle -> a -> m ()+putMessage lock h xs =+ liftIO $ putMsgLock lock h xs+{-# INLINE putMessage #-}
src/Reader.hs view
@@ -1,5 +1,5 @@ ----- Copyright (c) 2013-2023 Nicola Bonelli <nicola@larthia.com>+-- Copyright (c) 2013-2025 Nicola Bonelli <nicola@larthia.com> -- -- This program is free software; you can redistribute it and/or modify -- it under the terms of the GNU General Public License as published by@@ -16,16 +16,18 @@ -- Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. -- -module Reader where--import Control.Monad.Trans.Reader ( ReaderT )+module Reader (+ Env (..),+ ReaderIO,+) where -import Config ( Config )-import Options ( Options )+import Config (Config)+import Control.Monad.Trans.Reader (ReaderT)+import Options (Options) -data Env = Env {- conf :: Config- , opt :: Options-}+data Env = Env+ { conf :: Config+ , opt :: Options+ } type ReaderIO = ReaderT Env IO
− src/Search.hs
@@ -1,331 +0,0 @@------ Copyright (c) 2013-2023 Nicola Bonelli <nicola@larthia.com>------ This program is free software; you can redistribute it and/or modify--- it under the terms of the GNU General Public License as published by--- the Free Software Foundation; either version 2 of the License, or--- (at your option) any later version.------ This program is distributed in the hope that it will be useful,--- but WITHOUT ANY WARRANTY; without even the implied warranty of--- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the--- GNU General Public License for more details.------ You should have received a copy of the GNU General Public License--- along with this program; if not, write to the Free Software--- Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.---nc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.-----module Search (- parallelSearch- , isRegexp-) where--import Data.List ( isPrefixOf, isSuffixOf, partition, elemIndex, intersperse )-import Data.List.Split ( chunksOf )-import qualified Data.Map as M-import Data.Maybe ( fromJust, isJust, catMaybes, fromMaybe )-import Data.Function ( fix )-import qualified Data.Set as S--import Control.Exception as E ( catch, SomeException )-import Control.Concurrent ( forkIO, MVar, putMVar, forkOn, threadDelay )--import Control.Monad ( when, forM_, forever, unless, void, forM, replicateM_ )-import Control.Monad.Trans ( MonadIO(liftIO) )-import Control.Monad.Trans.Except ( runExceptT, throwE )-import Control.Monad.Trans.Reader- ( ReaderT(runReaderT), ask, reader, ask, local )-import Control.Applicative- ( Applicative(liftA2), Alternative((<|>)) )--import System.Environment ( lookupEnv )-import System.PosixCompat.Files as PC- ( getFileStatus, getSymbolicLinkStatus, isDirectory, FileStatus )-import System.IO- ( BufferMode(BlockBuffering),- hSetBinaryMode,- hSetBuffering,- hPutStrLn,- stderr,- stdin,- stdout,- stderr,- hPutStrLn )--import System.Process ( runProcess, waitForProcess )--import CGrep.Output- ( putOutputElements,- showFileName,- Output(..),- showFileName )-import CGrep.Common ( takeN, getTargetName, Text8, takeN )-import Options ( Options(..) )-import Config- ( Config(Config, configFileTypes, configFileLine, configColorMatch,- configColorFile, configColors, configPruneDirs),- dumpPalette )-import Reader- ( Env(Env, opt, conf),- ReaderIO,- Env(..) )--import qualified Data.ByteString as B-import qualified Data.ByteString.Builder as B-import qualified Data.ByteString.Builder.Extra as B--import qualified Data.ByteString.Lazy.Char8 as LB-import qualified Data.ByteString.Char8 as C-import qualified Codec.Binary.UTF8.String as UC--import Data.Tuple.Extra ( )-import qualified Data.Bifunctor--import qualified Data.Vector as V hiding ((!))-import Data.Vector ((!))--import Data.IORef- ( modifyIORef, modifyIORef', newIORef, readIORef, IORef, atomicModifyIORef, atomicModifyIORef' )-import Control.Concurrent.Chan.Unagi.Bounded- ( newChan, readChan, writeChan )--import System.Posix.Directory.Traversals ( getDirectoryContents )-import System.Posix.FilePath ( RawFilePath, takeBaseName, (</>) )-import System.Posix.Directory.Foreign (dtDir)-import System.Directory (makeAbsolute, canonicalizePath)-import Data.Functor ( void, (<&>), ($>))-import RawFilePath.Directory (doesDirectoryExist)-import Control.Arrow ( Arrow((&&&)) )-import Control.Concurrent.Async (forConcurrently, forConcurrently_, mapConcurrently_, async, Async, wait, asyncOn)--import qualified CGrep.Strategy.BoyerMoore as BoyerMoore-import qualified CGrep.Strategy.Levenshtein as Levenshtein-import qualified CGrep.Strategy.Regex as Regex-import qualified CGrep.Strategy.Tokenizer as Tokenizer-import qualified CGrep.Strategy.Semantic as Semantic-import Control.Monad.Catch ( SomeException, MonadCatch(catch) )-import Control.Monad.IO.Class ( MonadIO(liftIO) )--import CGrep.FileType ( FileType )-import CGrep.FileTypeMap- ( fileTypeInfoLookup, fileTypeLookup, FileTypeInfo )--import Control.Monad.Loops ( whileM_ )-import Verbose (putMsgLnVerbose, putMsgLn)-import Control.Concurrent.MVar ( newMVar, takeMVar )-import Data.IORef.Extra (atomicWriteIORef')-import CGrep.FileKind ( FileKind )-import qualified Data.List.NonEmpty as NE (unzip)---withRecursiveContents :: Options- -> RawFilePath- -> [FileType]- -> [FileKind]- -> [RawFilePath]- -> S.Set RawFilePath- -> IORef Int- -> ([RawFilePath] -> IO ()) -> IO ()-withRecursiveContents opt@Options{..} dir fTypes fKinds pdirs visited walkers action = do- xs <- getDirectoryContents dir- let (dirs, files) = partition ((== dtDir) . fst) xs-- -- filter the list of files- let files' = (dir </>) . snd <$> filter (\f -> fileFilter opt fTypes fKinds (snd f) && (not skip_test || isNotTestFile (snd f))) files- let dirs' = (dir </>) . snd <$> dirs-- -- run IO action- mapM_ action (chunksOf 8 files')-- -- process directories recursively...- foreach <- readIORef walkers >>= \tot -> do- if tot < 64 then pure (forConcurrently_ @[])- else pure forM_-- foreach dirs' $ \dirPath -> do- unless (isPrunableDir dirPath pdirs) $- -- this is a good directory, unless already visited...- -- this is a good directory, unless already visited...- -- this is a good directory, unless already visited...- -- this is a good directory, unless already visited...-- -- this is a good directory, unless already visited...-- -- this is a good directory, unless already visited...-- -- this is a good directory, unless already visited...- -- this is a good directory, unless already visited...-- -- this is a good directory, unless already visited...- makeRawAbsolute dirPath >>= \cpath ->- unless (cpath `S.member` visited) $ incrRef walkers *>- withRecursiveContents opt dirPath fTypes fKinds pdirs (S.insert cpath visited) walkers action-- decrRef walkers---parallelSearch :: [RawFilePath] -> [C.ByteString] -> [FileType] -> [FileKind] -> Bool -> ReaderIO ()-parallelSearch paths patterns fTypes fKinds isTermIn = do- Env{..} <- ask-- let Config{..} = conf- Options{..} = opt-- let multiplier = 4- jobs' = fromMaybe 1 jobs- totalJobs = jobs' * multiplier-- -- create channels ...- fileCh <- liftIO $ newChan 65536-- -- recursively traverse the filesystem ...- _ <- liftIO . forkOn 0 $ do- walkers <- newIORef (0 :: Int)- if recursive || follow- then forM_ (if null paths then ["."] else paths) $ \p ->- doesDirectoryExist p >>= \case- True -> incrRef walkers *>- withRecursiveContents opt p fTypes fKinds- (mkPrunableDirName <$> configPruneDirs <> (C.pack <$> prune_dir)) (S.singleton p) walkers (do- writeChan (fst fileCh))- _ -> writeChan (fst fileCh) [p]- else forM_ (if null paths && not isTermIn- then [("", 0)]- else paths `zip` [0..]) (\(p, idx) -> writeChan (fst fileCh) [p])-- -- enqueue EOF messages...- when (verbose > 0) $ putMsgLn @Text8 stderr "filesystem traversal completed!"- replicateM_ totalJobs $ writeChan (fst fileCh) []-- -- launch the worker threads...- matchingFiles <- liftIO $ newIORef S.empty-- let env = Env conf opt- runSearch = getSearcher env-- workers <- forM ([0 .. totalJobs-1] :: [Int]) $ \idx -> do- let processor = 1 + idx `div` multiplier- liftIO . asyncOn processor $ void . runExceptT $ do- asRef <- liftIO $ newIORef ([] :: [Async ()])- forever $ do- fs <- liftIO $ readChan (snd fileCh)- liftIO $ E.catch (- case fs of- [] -> liftIO $ readIORef asRef >>= mapM_ wait- fs -> runReaderT (do- out <- catMaybes <$> forM fs (\f -> do- out' <- take max_count <$> runSearch (fileTypeInfoLookup opt f) f patterns- when (vim || editor) $- liftIO $ mapM_ (modifyIORef matchingFiles . S.insert . (outFilePath &&& outLineNumb)) out'- putOutputElements out')- unless (null out) $- liftIO $ async (do- let !dump = LB.toStrict $ B.toLazyByteString (mconcat ((<> B.char8 '\n') <$> out))- B.hPut stdout dump) >>= \a -> modifyIORef' asRef (a:)- ) env- ) (\e -> let msg = show (e :: SomeException) in- C.hPutStrLn stderr (showFileName conf opt (getTargetName (head fs)) <> ": error: " <> C.pack (takeN 120 msg)))- when (null fs) $ do- when (verbose > 0) $ putMsgLn stderr $ "[" <> C.pack (show idx) <> "]@" <> C.pack (show processor) <>" searcher done!"- throwE ()-- -- wait workers to complete the job- liftIO $ mapM_ wait workers-- -- run editor...- when (vim || editor ) $ liftIO $ do- editor' <- if vim- then return (Just "vim")- else lookupEnv "EDITOR"-- files <- S.toList <$> readIORef matchingFiles- let filesUnpacked = Data.Bifunctor.first C.unpack <$> files-- let editFiles = (if fileline || configFileLine- then fmap (\(a,b) -> a <> ":" <> show b)- else fmap fst) filesUnpacked-- putStrLn $ "cgrep: open files " <> unwords editFiles <> "..."-- void $ runProcess (fromJust $ editor' <|> Just "vi")- editFiles- Nothing- Nothing- (Just stdin)- (Just stdout)- (Just stderr) >>= waitForProcess--getSearcher :: Env -> (Maybe (FileType, FileTypeInfo) -> RawFilePath -> [Text8] -> ReaderIO [Output])-getSearcher Env{..} = do- if | (not . isRegexp) opt && not (hasTokenizerOpt opt) && not (semantic opt) && edit_dist opt -> Levenshtein.search- | (not . isRegexp) opt && not (hasTokenizerOpt opt) && not (semantic opt) -> BoyerMoore.search- | (not . isRegexp) opt && semantic opt -> Semantic.search- | (not . isRegexp) opt -> Tokenizer.search- | isRegexp opt -> Regex.search- | otherwise -> undefined---makeRawAbsolute :: RawFilePath -> IO RawFilePath-makeRawAbsolute p = makeAbsolute (C.unpack p) <&> C.pack-{-# INLINE makeRawAbsolute #-}--incrRef :: IORef Int -> IO ()-incrRef ref = atomicModifyIORef' ref (\n -> (n+1, ()))-{-# INLINE incrRef #-}--decrRef :: IORef Int -> IO ()-decrRef ref = atomicModifyIORef' ref (\n -> (n-1, ()))-{-# INLINE decrRef #-}---fileFilter :: Options -> [FileType] -> [FileKind] -> RawFilePath -> Bool-fileFilter opt fTypes fKinds filename = fileFilterTypes typ && fileFilterKinds kin- where (typ, kin) = NE.unzip $ fileTypeLookup opt filename- fileFilterTypes = maybe False (liftA2 (||) (const $ null fTypes) (`elem` fTypes))- fileFilterKinds = maybe False (liftA2 (||) (const $ null fKinds) (`elem` fKinds))---isNotTestFile :: RawFilePath -> Bool-isNotTestFile f =- let fs = [("_test" `C.isSuffixOf`), ("-test" `C.isSuffixOf`), ("test-" `C.isPrefixOf`), ("test_" `C.isPrefixOf`), ("test" == )] :: [C.ByteString -> Bool]- in not $ any ($ takeBaseName f) fs-{-# INLINE isNotTestFile #-}---isPrunableDir:: RawFilePath -> [RawFilePath] -> Bool-isPrunableDir dir = any (`C.isSuffixOf` pdir)- where pdir = mkPrunableDirName dir-{-# INLINE isPrunableDir #-}---mkPrunableDirName :: RawFilePath -> RawFilePath-mkPrunableDirName xs | "/" `C.isSuffixOf` xs = xs- | otherwise = xs <> "/"-{-# INLINE mkPrunableDirName #-}---(.!.) :: V.Vector a -> Int -> a-v .!. i = v ! (i `mod` V.length v)-{-# INLINE (.!.) #-}---hasFileType :: RawFilePath -> Options -> [FileType] -> Bool-hasFileType path opt xs = isJust $ fileTypeLookup opt path >>= (\(typ, _) -> typ `elemIndex` xs)-{-# INLINE hasFileType #-}---hasTokenizerOpt :: Options -> Bool-hasTokenizerOpt Options{..} =- identifier ||- nativeType ||- keyword ||- number ||- string ||- operator---isRegexp :: Options -> Bool-isRegexp opt = regex_posix opt || regex_pcre opt-{-# INLINE isRegexp #-}
src/Util.hs view
@@ -1,5 +1,5 @@ ----- Copyright (c) 2013-2023 Nicola Bonelli <nicola@larthia.com>+-- Copyright (c) 2013-2025 Nicola Bonelli <nicola@larthia.com> -- -- This program is free software; you can redistribute it and/or modify -- it under the terms of the GNU General Public License as published by@@ -16,43 +16,44 @@ -- Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. -- -module Util where--import Data.Maybe ( listToMaybe )-import Data.Char ( toLower )+module Util (+ partitionM,+ xor,+ prettyRead,+ spanGroup,+ spanGroupSeq,+ unquoteT,+ mapMaybe',+ findWithIndex,+ unsafeHead,+) where -import qualified Data.ByteString.Char8 as C import qualified Data.Sequence as S-import Data.Sequence ((|>), Seq((:<|), (:|>), Empty))+import qualified Data.Text as T import Text.Read (readMaybe) -partitionM :: Monad m => (a -> m Bool) -> [a] -> m ([a], [a])+partitionM :: (Monad m) => (a -> m Bool) -> [a] -> m ([a], [a]) partitionM _ [] = return ([], [])-partitionM f (x:xs) = do+partitionM f (x : xs) = do res <- f x- (as,bs) <- partitionM f xs- return ([x | res]<>as, [x | not res]<>bs)+ (as, bs) <- partitionM f xs+ return ([x | res] <> as, [x | not res] <> bs) {-# INLINE partitionM #-} - xor :: Bool -> Bool -> Bool a `xor` b = a && not b || not a && b {-# INLINE xor #-} --prettyRead :: Read a => String -> String -> a+prettyRead :: (Read a) => String -> String -> a prettyRead xs err = case readMaybe xs of Just v -> v- _ -> error $ err <> ": parse error near '" <> take 40 xs <> "'"-+ _ -> errorWithoutStackTrace $ err <> ": parse error near '" <> take 40 xs <> "'" spanGroup :: Int -> [a] -> [[a]]-spanGroup _ [] = []-spanGroup 1 xs = map (: []) xs-spanGroup n xs = take n xs : spanGroup n (tail xs)-{-# INLINE spanGroup #-}-+spanGroup n xs+ | length xs < n = [] -- Stop if the remaining list is shorter than n+ | otherwise = take n xs : spanGroup n (drop 1 xs) spanGroupSeq :: Int -> S.Seq a -> [S.Seq a] spanGroupSeq _ S.Empty = []@@ -60,39 +61,37 @@ spanGroupSeq n xs = S.take n xs : spanGroupSeq n (S.drop 1 xs) {-# INLINE spanGroupSeq #-} --rmQuote :: String -> String-rmQuote [] = []-rmQuote [x] = [x]-rmQuote y@(x:xs)- | x == '"' || x == '\'' = if x == last xs then init xs- else y- | otherwise = y-{-# INLINE rmQuote #-}---rmQuote8 :: C.ByteString -> C.ByteString-rmQuote8 b | C.length b < 2 = b- | otherwise =- case C.uncons b of- Just (x,xs) -> if (x == '"' || x == '\'') && (x == C.last b) then C.init xs else b- _ -> b-{-# INLINE rmQuote8 #-}-+{- | Removes a single pair of matching quotes ('"' or '\'')+| from the beginning and end of a Text.+-}+unquoteT :: T.Text -> T.Text+unquoteT txt =+ case T.uncons txt of+ -- Check if 'x' is a quote we care about+ Just (x, xs) | x == '"' || x == '\'' ->+ case T.unsnoc xs of+ Just (inner, y) | x == y -> inner+ _ -> txt+ _ -> txt -- Text was empty or first char wasn't a quote+{-# INLINE unquoteT #-} -mapMaybe' :: Foldable f => (a -> Maybe b) -> f a -> [b]+mapMaybe' :: (Foldable f) => (a -> Maybe b) -> f a -> [b] mapMaybe' f = foldr g [] where g x rest- | Just y <- f x = y : rest- | otherwise = rest-+ | Just y <- f x = y : rest+ | otherwise = rest findWithIndex :: forall a. (a -> Bool) -> [a] -> (# Int, Maybe a #) findWithIndex predicate = go predicate 0 where- go :: forall a. (a -> Bool) -> Int -> [a] -> (# Int, Maybe a #)- go p _ [] = (# 0, Nothing #)- go p !index (x:xs)- | p x = (# index, Just x #)- | otherwise = go p (index + 1) xs+ go :: (a -> Bool) -> Int -> [a] -> (# Int, Maybe a #)+ go _ _ [] = (# 0, Nothing #)+ go p !index (x : xs)+ | p x = (# index, Just x #)+ | otherwise = go p (index + 1) xs++unsafeHead :: [a] -> a+unsafeHead [] = error "unsafeHead: empty list"+unsafeHead (!x : _) = x+{-# INLINE unsafeHead #-}
− src/Verbose.hs
@@ -1,66 +0,0 @@------ Copyright (c) 2013-2023 Nicola Bonelli <nicola@larthia.com>------ This program is free software; you can redistribute it and/or modify--- it under the terms of the GNU General Public License as published by--- the Free Software Foundation; either version 2 of the License, or--- (at your option) any later version.------ This program is distributed in the hope that it will be useful,--- but WITHOUT ANY WARRANTY; without even the implied warranty of--- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the--- GNU General Public License for more details.------ You should have received a copy of the GNU General Public License--- along with this program; if not, write to the Free Software--- Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.-----module Verbose where--import Control.Monad.Trans.Reader ( reader )-import Control.Monad.IO.Class ( MonadIO(liftIO) )-import Control.Monad ( when )--import Options ( Options(verbose) )-import Reader ( ReaderIO, Env(..) )--import qualified Data.ByteString as C (hPutStr, hPut)-import GHC.IO.Handle ( Handle )-import System.IO ( Handle, hPutStrLn, hPutStr )-import Data.String ( IsString )--import qualified Data.ByteString.Char8 as C-import qualified Data.Text as T-import qualified Data.Text.IO as T---class (IsString a) => PutStr a where- putStringLn :: Handle -> a -> IO ()- putString :: Handle -> a -> IO ()--instance PutStr String where- putStringLn = hPutStrLn- putString = hPutStr--instance PutStr C.ByteString where- putStringLn = C.hPutStrLn- putString = C.hPutStr--instance PutStr T.Text where- putStringLn = T.hPutStrLn- putString = T.hPutStr---putMsgLnVerbose :: (PutStr a) => Int -> Handle -> a -> ReaderIO ()-putMsgLnVerbose l h xs = do- n <- reader $ verbose . opt- when (n >= l) $- liftIO $ putStringLn h xs-{-# INLINE putMsgLnVerbose #-}---putMsgLn :: (PutStr a, MonadIO m) => Handle -> a -> m ()-putMsgLn h xs =- liftIO $ putStringLn h xs-{-# INLINE putMsgLn #-}