sphinxesc (empty) → 0.1.0.0
raw patch · 6 files changed
+351/−0 lines, 6 filesdep +basedep +optparse-applicativedep +parsecsetup-changed
Dependencies added: base, optparse-applicative, parsec, sphinxesc
Files
- LICENSE +24/−0
- Main.hs +40/−0
- README.md +117/−0
- Setup.hs +2/−0
- SphinxEscape.hs +130/−0
- sphinxesc.cabal +38/−0
+ LICENSE view
@@ -0,0 +1,24 @@+The MIT License (MIT)++Copyright (c) 2016 Daniel Choi <dhchoi@gmail.com>++Permission is hereby granted, free of charge, to any person obtaining a copy+of this software and associated documentation files (the "Software"), to deal+in the Software without restriction, including without limitation the rights+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell+copies of the Software, and to permit persons to whom the Software is+furnished to do so, subject to the following conditions:++The above copyright notice and this permission notice shall be included in+all copies or substantial portions of the Software.++THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN+THE SOFTWARE.+++
+ Main.hs view
@@ -0,0 +1,40 @@+{-# LANGUAGE OverloadedStrings, RecordWildCards, ScopedTypeVariables #-} +module Main where+import SphinxEscape (escapeSphinxQueryString, parseQuery)+import System.Environment+import Data.List+import Options.Applicative+import Control.Applicative+import Data.Monoid((<>))++data Options = Options {+ optMode :: Mode+ , optInput :: Maybe String+ }++data Mode = Convert | Parse ++mode :: Parser Options+mode = Options+ <$> flag Convert Parse (short 'p' <> help "Show parser evaluation")+ <*> (+ (Just <$> strArgument (metavar "RAW-STRING" <> help "sphinx raw input expression"))+ <|> pure Nothing+ )++opt :: ParserInfo Options+opt = info (helper <*> mode) (header "sphinxesc")+ ++main = do+ Options{..} <- execParser opt+ input <- maybe getContents return optInput+ case optMode of + Convert -> do+ putStrLn $ escapeSphinxQueryString input+ Parse -> do+ print $ parseQuery input+++ +
+ README.md view
@@ -0,0 +1,117 @@+# sphinxesc++A small module to prevent user-submitted search expressions from being +mis-parsed into invalid Sphinx Extended Query Expressions.++The module provides a function ++ module SphinxEscape where+ escapeSphinxQueryString :: String -> String++that sanitizes the Sphinx query expression in a way that can be safely submitted to the Sphinx API. ++## Synopsis++Example from ghci:++```+ghci> :m SphinxEscape +ghci> putStrLn $ escapeSphinxQueryString "@tag_list hello OR quick brown fox 7/11"+@tag_list hello | quick brown fox 7 11+ghci> +ghci> putStrLn $ escapeSphinxQueryString "hello AND quick brown fox 7/11"+hello & quick brown fox 7 11+ghci> ++```++## Explanation++`escapeSphinxQueryString` performs very simple escaping with the help of a+simplified abtract syntax tree. The abstract syntax tree it builds is:++```+data Expression = + TagFieldSearch String + | Literal String+ | Phrase String+ | AndOrExpr Conj Expression Expression + deriving Show++```++The escaping does not parse more advanced Sphinx query expressions such as+`NEAR/n`, quorum, etc., nor does it recognize arbitrary `@field` expressions.+The only special expressions recognized are `& (AND)`, `| (OR)` and `@tag_list+WORDS`. Except for quoted phrases, non-alphanumeric characters that do not+form part of these specific expressions are simply turned into whitespace. ++See the **Testing** section below for examples of conversions.++Obviously these rules are quite domain specific. The rules can be+made more configurable later.+++## Testing+++The command line executable `sphinxesc` can be used to test the expression parser +and escaping of the input to the final sphinx search expression.++```+$ sphinxesc "test OR hello"+test | hello++# -p option shows the parsing result++$ sphinxesc -p "test OR hello"+AndOrExpr Or (Literal "test") (Literal "hello")+```++There is a suite of Bash-based regression tests in `tests.txt`, where the input+is on the left, followed by `::` surrounded by any whitespace, followed by the+expected escaped output result. To run the tests, execute the script+`./test.sh`+++**NOTE** This test output may be outdated. Please look at the `tests.txt` for +the current tests.++```+./test.sh++INPUT EXPECTED RESULT PASS +7/11 7 11 7 11 PASS +hello 7/11 hello 7 11 hello 7 11 PASS +hello OR 7/11 hello | 7 11 hello | 7 11 PASS +hello or 7/11 hello | 7 11 hello | 7 11 PASS +hello | 7/11 hello | 7 11 hello | 7 11 PASS +hello AND 7/11 hello & 7 11 hello & 7 11 PASS +@tag_list fox tango 7/11 @tag_list fox tango 7 11 @tag_list fox tango 7 11 PASS +@(tag_list) fox tango 7/11 @tag_list fox tango 7 11 @tag_list fox tango 7 11 PASS +@(tag_list) AND @tag_list AND @tag_list AND PASS +@other_field AND other field AND other field AND PASS +hello & @other_field AND hello & other field AND hello & other field AND PASS +hello & hello hello PASS +& hello & hello hello PASS +& & hello & hello hello PASS +| | hello | hello hello PASS +"hello" hello hello hello hello hello PASS +hello" hello hello hello hello hello PASS +hello' hello hello hello hello hello PASS +hello' @tag_list fox hello @tag_list fox hello @tag_list fox PASS +hello' @tag_list fox & hello @tag_list fox hello @tag_list fox PASS + PASS +```++(The last case is hard to see, but the input is a blank string "" and the output is a blank string "".)++## Future directions++The escaping function can be made more configurable. The parser and AST data+structure can also be made more sophisticated, so that the AST can cover more+of the Sphinx Extended Query syntax. ++## Reference++* <http://sphinxsearch.com/docs/latest/extended-syntax.html> Sphinx Extended Syntax docs
+ Setup.hs view
@@ -0,0 +1,2 @@+import Distribution.Simple+main = defaultMain
+ SphinxEscape.hs view
@@ -0,0 +1,130 @@+{-# LANGUAGE OverloadedStrings, RecordWildCards, ScopedTypeVariables, FlexibleContexts #-} +module SphinxEscape where+import Control.Applicative+import Data.Functor.Identity (Identity )+import Text.Parsec hiding (many, (<|>)) +import Data.Char+import Data.List+ ++-- Main function+escapeSphinxQueryString :: String -> String+escapeSphinxQueryString s = intercalate " " . map expressionToString . parseQuery $ s+++-- Just a simplified syntax tree. Besides this, all other input has its+-- non-alphanumeric characters stripped, including double and single quotes and+-- parentheses++data Expression = + TagFieldSearch String + | Literal String+ | Phrase String+ | AndOrExpr Conj Expression Expression + deriving Show++data Conj = And | Or+ deriving Show++parseQuery :: String -> [Expression]+parseQuery inp =+ case Text.Parsec.parse (many expression) "" inp of+ Left x -> error $ "parser failed: " ++ show x+ Right xs -> xs++-- escapes expression to string to pass to sphinx+expressionToString :: Expression -> String+expressionToString (TagFieldSearch s) = "@tag_list" ++ escapeString s+expressionToString (Literal s) = escapeString s+expressionToString (Phrase s) = "\"" ++ s ++ "\"" -- no need to escape the contents+expressionToString (AndOrExpr c a b) = + let a' = expressionToString a + b' = expressionToString b+ c' = conjToString c + -- if either a' or b' is just whitespace, just choose one or the other+ in case (all isSpace a', all isSpace b') of+ (True, False) -> b'+ (False, True) -> a'+ (False, False) -> a' ++ c' ++ b'+ _ -> ""++conjToString :: Conj -> String+conjToString And = " & "+conjToString Or = " | "++-- removes all non-alphanumerics from literal strings that could be parsed+-- mistakenly as Sphinx Extended Query operators+escapeString :: String -> String+escapeString s = map (stripAlphaNum) s++stripAlphaNum :: Char -> Char+stripAlphaNum s | isAlphaNum s = s+ | otherwise = ' '+++type Parser' = ParsecT String () Identity ++-- | can be literal or tag field or nothing, followed an expression+topLevelExpression :: Parser' [Expression]+topLevelExpression = do+ a <- option [] ((:[]) <$> (tagField <|> literal))+ xs <- many expression+ return $ a ++ xs+++expression :: Parser' Expression+expression = (try andOrExpr) <|> try tagField <|> try phrase <|> literal ++tagField :: Parser' Expression+tagField = do+ char '@'+ string "tag_list" <|> string "(tag_list)"+ s <- manyTill anyChar (try literalStop)+ return $ TagFieldSearch s+++andOrExpr :: Parser' Expression+andOrExpr = do + a <- (try tagField <|> try phrase <|> literal)+ x <- try conjExpr+ b <- expression -- recursion+ return $ AndOrExpr x a b++conjExpr :: Parser' Conj+conjExpr = andExpr <|> orExpr++andExpr :: Parser' Conj+andExpr = mkConjExpr ["and", "AND", "&"] And++orExpr :: Parser' Conj+orExpr = mkConjExpr ["or", "OR", "|"] Or+++mkConjExpr :: [String] -> Conj -> Parser' Conj+mkConjExpr xs t = + try (many1 space >> choice (map (string . (++" ")) xs))+ >> return t++phrase :: Parser' Expression+phrase = do+ _ <- char '"'+ xs <- manyTill anyChar (char '"')+ return . Phrase $ xs++literalStop :: Parser' ()+literalStop = (choice [ + lookAhead (tagField >> return ()) + , lookAhead (conjExpr >> return ())+ , lookAhead (phrase >> return ())+ , eof+ ])+ <?> "literalStop"++literal :: Parser' Expression+literal = do+ a <- anyChar+ notFollowedBy literalStop+ xs <- manyTill anyChar (try literalStop)+ return . Literal $ a:xs++
+ sphinxesc.cabal view
@@ -0,0 +1,38 @@+name: sphinxesc+version: 0.1.0.0+synopsis: Transform queries for sphinx input+description: Transform queries for sphinx input+category: Text+author: Daniel Choi+maintainer: Mackey RMS+bug-reports: https://github.com/mackeyrms/sphinxesc/issues+homepage: https://github.com/mackeyrms/sphinxesc#readme+cabal-version: >=1.10+tested-with: GHC == 8.0.2+build-type: Simple+extra-source-files: README.md+license: MIT+license-file: LICENSE+-- copyright: ++source-repository head+ type: git+ location: https://github.com/mackeyrms/sphinxesc++library+ build-depends: base >=4.7 && <5.0+ , parsec+ exposed-modules: SphinxEscape+ hs-source-dirs: .+ default-language: Haskell2010++executable sphinxesc+ build-depends: base+ , parsec+ , sphinxesc+ , optparse-applicative+ hs-source-dirs: .+ default-language: Haskell2010+ main-is: Main.hs++