diff --git a/LICENSE b/LICENSE new file mode 100644 --- /dev/null +++ b/LICENSE @@ -0,0 +1,24 @@ +The MIT License (MIT) + +Copyright (c) 2016 Daniel Choi + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + + diff --git a/Main.hs b/Main.hs new file mode 100644 --- /dev/null +++ b/Main.hs @@ -0,0 +1,40 @@ +{-# LANGUAGE OverloadedStrings, RecordWildCards, ScopedTypeVariables #-} +module Main where +import SphinxEscape (escapeSphinxQueryString, parseQuery) +import System.Environment +import Data.List +import Options.Applicative +import Control.Applicative +import Data.Monoid((<>)) + +data Options = Options { + optMode :: Mode + , optInput :: Maybe String + } + +data Mode = Convert | Parse + +mode :: Parser Options +mode = Options + <$> flag Convert Parse (short 'p' <> help "Show parser evaluation") + <*> ( + (Just <$> strArgument (metavar "RAW-STRING" <> help "sphinx raw input expression")) + <|> pure Nothing + ) + +opt :: ParserInfo Options +opt = info (helper <*> mode) (header "sphinxesc") + + +main = do + Options{..} <- execParser opt + input <- maybe getContents return optInput + case optMode of + Convert -> do + putStrLn $ escapeSphinxQueryString input + Parse -> do + print $ parseQuery input + + + + diff --git a/README.md b/README.md new file mode 100644 --- /dev/null +++ b/README.md @@ -0,0 +1,117 @@ +# sphinxesc + +A small module to prevent user-submitted search expressions from being +mis-parsed into invalid Sphinx Extended Query Expressions. + +The module provides a function + + module SphinxEscape where + escapeSphinxQueryString :: String -> String + +that sanitizes the Sphinx query expression in a way that can be safely submitted to the Sphinx API. + +## Synopsis + +Example from ghci: + +``` +ghci> :m SphinxEscape +ghci> putStrLn $ escapeSphinxQueryString "@tag_list hello OR quick brown fox 7/11" +@tag_list hello | quick brown fox 7 11 +ghci> +ghci> putStrLn $ escapeSphinxQueryString "hello AND quick brown fox 7/11" +hello & quick brown fox 7 11 +ghci> + +``` + +## Explanation + +`escapeSphinxQueryString` performs very simple escaping with the help of a +simplified abtract syntax tree. The abstract syntax tree it builds is: + +``` +data Expression = + TagFieldSearch String + | Literal String + | Phrase String + | AndOrExpr Conj Expression Expression + deriving Show + +``` + +The escaping does not parse more advanced Sphinx query expressions such as +`NEAR/n`, quorum, etc., nor does it recognize arbitrary `@field` expressions. +The only special expressions recognized are `& (AND)`, `| (OR)` and `@tag_list +WORDS`. Except for quoted phrases, non-alphanumeric characters that do not +form part of these specific expressions are simply turned into whitespace. + +See the **Testing** section below for examples of conversions. + +Obviously these rules are quite domain specific. The rules can be +made more configurable later. + + +## Testing + + +The command line executable `sphinxesc` can be used to test the expression parser +and escaping of the input to the final sphinx search expression. + +``` +$ sphinxesc "test OR hello" +test | hello + +# -p option shows the parsing result + +$ sphinxesc -p "test OR hello" +AndOrExpr Or (Literal "test") (Literal "hello") +``` + +There is a suite of Bash-based regression tests in `tests.txt`, where the input +is on the left, followed by `::` surrounded by any whitespace, followed by the +expected escaped output result. To run the tests, execute the script +`./test.sh` + + +**NOTE** This test output may be outdated. Please look at the `tests.txt` for +the current tests. + +``` +./test.sh + +INPUT EXPECTED RESULT PASS +7/11 7 11 7 11 PASS +hello 7/11 hello 7 11 hello 7 11 PASS +hello OR 7/11 hello | 7 11 hello | 7 11 PASS +hello or 7/11 hello | 7 11 hello | 7 11 PASS +hello | 7/11 hello | 7 11 hello | 7 11 PASS +hello AND 7/11 hello & 7 11 hello & 7 11 PASS +@tag_list fox tango 7/11 @tag_list fox tango 7 11 @tag_list fox tango 7 11 PASS +@(tag_list) fox tango 7/11 @tag_list fox tango 7 11 @tag_list fox tango 7 11 PASS +@(tag_list) AND @tag_list AND @tag_list AND PASS +@other_field AND other field AND other field AND PASS +hello & @other_field AND hello & other field AND hello & other field AND PASS +hello & hello hello PASS +& hello & hello hello PASS +& & hello & hello hello PASS +| | hello | hello hello PASS +"hello" hello hello hello hello hello PASS +hello" hello hello hello hello hello PASS +hello' hello hello hello hello hello PASS +hello' @tag_list fox hello @tag_list fox hello @tag_list fox PASS +hello' @tag_list fox & hello @tag_list fox hello @tag_list fox PASS + PASS +``` + +(The last case is hard to see, but the input is a blank string "" and the output is a blank string "".) + +## Future directions + +The escaping function can be made more configurable. The parser and AST data +structure can also be made more sophisticated, so that the AST can cover more +of the Sphinx Extended Query syntax. + +## Reference + +* Sphinx Extended Syntax docs diff --git a/Setup.hs b/Setup.hs new file mode 100644 --- /dev/null +++ b/Setup.hs @@ -0,0 +1,2 @@ +import Distribution.Simple +main = defaultMain diff --git a/SphinxEscape.hs b/SphinxEscape.hs new file mode 100644 --- /dev/null +++ b/SphinxEscape.hs @@ -0,0 +1,130 @@ +{-# LANGUAGE OverloadedStrings, RecordWildCards, ScopedTypeVariables, FlexibleContexts #-} +module SphinxEscape where +import Control.Applicative +import Data.Functor.Identity (Identity ) +import Text.Parsec hiding (many, (<|>)) +import Data.Char +import Data.List + + +-- Main function +escapeSphinxQueryString :: String -> String +escapeSphinxQueryString s = intercalate " " . map expressionToString . parseQuery $ s + + +-- Just a simplified syntax tree. Besides this, all other input has its +-- non-alphanumeric characters stripped, including double and single quotes and +-- parentheses + +data Expression = + TagFieldSearch String + | Literal String + | Phrase String + | AndOrExpr Conj Expression Expression + deriving Show + +data Conj = And | Or + deriving Show + +parseQuery :: String -> [Expression] +parseQuery inp = + case Text.Parsec.parse (many expression) "" inp of + Left x -> error $ "parser failed: " ++ show x + Right xs -> xs + +-- escapes expression to string to pass to sphinx +expressionToString :: Expression -> String +expressionToString (TagFieldSearch s) = "@tag_list" ++ escapeString s +expressionToString (Literal s) = escapeString s +expressionToString (Phrase s) = "\"" ++ s ++ "\"" -- no need to escape the contents +expressionToString (AndOrExpr c a b) = + let a' = expressionToString a + b' = expressionToString b + c' = conjToString c + -- if either a' or b' is just whitespace, just choose one or the other + in case (all isSpace a', all isSpace b') of + (True, False) -> b' + (False, True) -> a' + (False, False) -> a' ++ c' ++ b' + _ -> "" + +conjToString :: Conj -> String +conjToString And = " & " +conjToString Or = " | " + +-- removes all non-alphanumerics from literal strings that could be parsed +-- mistakenly as Sphinx Extended Query operators +escapeString :: String -> String +escapeString s = map (stripAlphaNum) s + +stripAlphaNum :: Char -> Char +stripAlphaNum s | isAlphaNum s = s + | otherwise = ' ' + + +type Parser' = ParsecT String () Identity + +-- | can be literal or tag field or nothing, followed an expression +topLevelExpression :: Parser' [Expression] +topLevelExpression = do + a <- option [] ((:[]) <$> (tagField <|> literal)) + xs <- many expression + return $ a ++ xs + + +expression :: Parser' Expression +expression = (try andOrExpr) <|> try tagField <|> try phrase <|> literal + +tagField :: Parser' Expression +tagField = do + char '@' + string "tag_list" <|> string "(tag_list)" + s <- manyTill anyChar (try literalStop) + return $ TagFieldSearch s + + +andOrExpr :: Parser' Expression +andOrExpr = do + a <- (try tagField <|> try phrase <|> literal) + x <- try conjExpr + b <- expression -- recursion + return $ AndOrExpr x a b + +conjExpr :: Parser' Conj +conjExpr = andExpr <|> orExpr + +andExpr :: Parser' Conj +andExpr = mkConjExpr ["and", "AND", "&"] And + +orExpr :: Parser' Conj +orExpr = mkConjExpr ["or", "OR", "|"] Or + + +mkConjExpr :: [String] -> Conj -> Parser' Conj +mkConjExpr xs t = + try (many1 space >> choice (map (string . (++" ")) xs)) + >> return t + +phrase :: Parser' Expression +phrase = do + _ <- char '"' + xs <- manyTill anyChar (char '"') + return . Phrase $ xs + +literalStop :: Parser' () +literalStop = (choice [ + lookAhead (tagField >> return ()) + , lookAhead (conjExpr >> return ()) + , lookAhead (phrase >> return ()) + , eof + ]) + "literalStop" + +literal :: Parser' Expression +literal = do + a <- anyChar + notFollowedBy literalStop + xs <- manyTill anyChar (try literalStop) + return . Literal $ a:xs + + diff --git a/sphinxesc.cabal b/sphinxesc.cabal new file mode 100644 --- /dev/null +++ b/sphinxesc.cabal @@ -0,0 +1,38 @@ +name: sphinxesc +version: 0.1.0.0 +synopsis: Transform queries for sphinx input +description: Transform queries for sphinx input +category: Text +author: Daniel Choi +maintainer: Mackey RMS +bug-reports: https://github.com/mackeyrms/sphinxesc/issues +homepage: https://github.com/mackeyrms/sphinxesc#readme +cabal-version: >=1.10 +tested-with: GHC == 8.0.2 +build-type: Simple +extra-source-files: README.md +license: MIT +license-file: LICENSE +-- copyright: + +source-repository head + type: git + location: https://github.com/mackeyrms/sphinxesc + +library + build-depends: base >=4.7 && <5.0 + , parsec + exposed-modules: SphinxEscape + hs-source-dirs: . + default-language: Haskell2010 + +executable sphinxesc + build-depends: base + , parsec + , sphinxesc + , optparse-applicative + hs-source-dirs: . + default-language: Haskell2010 + main-is: Main.hs + +