diff --git a/README.md b/README.md new file mode 100644 --- /dev/null +++ b/README.md @@ -0,0 +1,976 @@ +Bloodhound [![TravisCI](https://travis-ci.org/bitemyapp/bloodhound.svg)](https://travis-ci.org/bitemyapp/bloodhound) [![Hackage](https://img.shields.io/hackage/v/bloodhound.svg?style=flat)](https://hackage.haskell.org/package/bloodhound) +========== + +![Bloodhound (dog)](./bloodhound.jpg) + +Elasticsearch client and query DSL for Haskell +============================================== + +Why? +---- + +Search doesn't have to be hard. Let the dog do it. + +Endorsements +------------ + +"Bloodhound makes Elasticsearch almost tolerable!" - Almost-gruntled user + +"ES is a nightmare but Bloodhound at least makes it tolerable." - Same user, later opinion. + +Version compatibility +--------------------- + +Elasticsearch \>= 1.0 is recommended. Bloodhound mostly works with 0.9.x, but I don't recommend it if you expect everything to work. As of Bloodhound 0.3 all \>=1.0 versions of Elasticsearch work. + +Current versions we test against are 1.0.3, 1.1.2, 1.2.3, 1.3.2, and 1.4.0. We also check that GHC 7.6 and 7.8 both build and pass tests. See our [TravisCI](https://travis-ci.org/bitemyapp/bloodhound) to learn more. + +Stability +--------- + +Bloodhound is stable for production use. I will strive to avoid breaking API compatibility from here on forward, but dramatic features like a type-safe, fully integrated mapping API may require breaking things in the future. + +Hackage page and Haddock documentation +====================================== + + + +Elasticsearch Tutorial +====================== + +It's not using Bloodhound, but if you need an introduction to or overview of Elasticsearch and how to use it, you can use [this screencast](http://vimeo.com/106463167). + +Examples +======== + +Index Operations +---------------- + +### Create Index + +``` {.haskell} + +-- Formatted for use in ghci, so there are "let"s in front of the decls. + +-- if you see :{ and :}, they're so you can copy-paste +-- the multi-line examples into your ghci REPL. + +:set -XDeriveGeneric +:{ +import Control.Applicative +import Database.Bloodhound +import Data.Aeson +import Data.Either (Either(..)) +import Data.Maybe (fromJust) +import Data.Time.Calendar (Day(..)) +import Data.Time.Clock (secondsToDiffTime, UTCTime(..)) +import Data.Text (Text) +import GHC.Generics (Generic) +import Network.HTTP.Client +import qualified Network.HTTP.Types.Status as NHTS + +-- no trailing slashes in servers, library handles building the path. +let testServer = (Server "http://localhost:9200") +let testIndex = IndexName "twitter" +let testMapping = MappingName "tweet" +let withBH' = withBH defaultManagerSettings testServer + +-- defaultIndexSettings is exported by Database.Bloodhound as well +let defaultIndexSettings = IndexSettings (ShardCount 3) (ReplicaCount 2) + +-- createIndex returns MonadBH m => m Reply. You can use withBH for + one-off commands or you can use runBH to group together commands + and to pass in your own HTTP manager for pipelining. + +-- response :: Reply, Reply is a synonym for Network.HTTP.Conduit.Response +response <- withBH' $ createIndex defaultIndexSettings testIndex +:} + +``` + +### Delete Index + +#### Code + +``` {.haskell} + +-- response :: Reply +response <- withBH' $ deleteIndex testIndex + +``` + +#### Example Response + +``` {.haskell} + +-- print response if it was a success +Response {responseStatus = Status {statusCode = 200, statusMessage = "OK"} + , responseVersion = HTTP/1.1 + , responseHeaders = [("Content-Type", "application/json; charset=UTF-8") + , ("Content-Length", "21")] + , responseBody = "{\"acknowledged\":true}" + , responseCookieJar = CJ {expose = []} + , responseClose' = ResponseClose} + +-- if the index to be deleted didn't exist anyway +Response {responseStatus = Status {statusCode = 404, statusMessage = "Not Found"} + , responseVersion = HTTP/1.1 + , responseHeaders = [("Content-Type", "application/json; charset=UTF-8") + , ("Content-Length","65")] + , responseBody = "{\"error\":\"IndexMissingException[[twitter] missing]\",\"status\":404}" + , responseCookieJar = CJ {expose = []} + , responseClose' = ResponseClose} + +``` + +### Refresh Index + +#### Note, you **have** to do this if you expect to read what you just wrote + +``` {.haskell} + +resp <- withBH' $ refreshIndex testIndex + +``` + +#### Example Response + +``` {.haskell} + +-- print resp on success +Response {responseStatus = Status {statusCode = 200, statusMessage = "OK"} + , responseVersion = HTTP/1.1 + , responseHeaders = [("Content-Type", "application/json; charset=UTF-8") + , ("Content-Length","50")] + , responseBody = "{\"_shards\":{\"total\":10,\"successful\":5,\"failed\":0}}" + , responseCookieJar = CJ {expose = []} + , responseClose' = ResponseClose} + +``` + +Mapping Operations +------------------ + +### Create Mapping + +``` {.haskell} + +-- don't forget imports and the like at the top. + +data TweetMapping = TweetMapping deriving (Eq, Show) + +-- I know writing the JSON manually sucks. +-- I don't have a proper data type for Mappings yet. +-- Let me know if this is something you need. + +:{ +instance ToJSON TweetMapping where + toJSON TweetMapping = + object ["tweet" .= + object ["properties" .= + object ["location" .= + object ["type" .= ("geo_point" :: Text)]]]] +:} + +resp <- withBH' $ putMapping testIndex testMapping TweetMapping + +``` + +### Delete Mapping + +``` {.haskell} + +resp <- withBH' $ deleteMapping testIndex testMapping + +``` + +Document Operations +------------------- + +### Indexing Documents + +``` {.haskell} + +-- don't forget the imports and derive generic setting for ghci +-- at the beginning of the examples. + +:{ +data Location = Location { lat :: Double + , lon :: Double } deriving (Eq, Generic, Show) + +data Tweet = Tweet { user :: Text + , postDate :: UTCTime + , message :: Text + , age :: Int + , location :: Location } deriving (Eq, Generic, Show) + +exampleTweet = Tweet { user = "bitemyapp" + , postDate = UTCTime + (ModifiedJulianDay 55000) + (secondsToDiffTime 10) + , message = "Use haskell!" + , age = 10000 + , location = Location 40.12 (-71.34) } + +-- automagic (generic) derivation of instances because we're lazy. +instance ToJSON Tweet +instance FromJSON Tweet +instance ToJSON Location +instance FromJSON Location +:} + +-- Should be able to toJSON and encode the data structures like this: +-- λ> toJSON $ Location 10.0 10.0 +-- Object fromList [("lat",Number 10.0),("lon",Number 10.0)] +-- λ> encode $ Location 10.0 10.0 +-- "{\"lat\":10,\"lon\":10}" + +resp <- withBH' $ indexDocument testIndex testMapping defaultIndexDocumentSettings exampleTweet (DocId "1") + +``` + +#### Example Response + +``` {.haskell} + +Response {responseStatus = + Status {statusCode = 200, statusMessage = "OK"} + , responseVersion = HTTP/1.1, responseHeaders = + [("Content-Type","application/json; charset=UTF-8"), + ("Content-Length","75")] + , responseBody = "{\"_index\":\"twitter\",\"_type\":\"tweet\",\"_id\":\"1\",\"_version\":2,\"created\":false}" + , responseCookieJar = CJ {expose = []}, responseClose' = ResponseClose} + +``` + +### Deleting Documents + +``` {.haskell} + +resp <- withBH' $ deleteDocument testIndex testMapping (DocId "1") + +``` + +### Getting Documents + +``` {.haskell} + +-- n.b., you'll need the earlier imports. responseBody is from http-conduit + +resp <- withBH' $ getDocument testIndex testMapping (DocId "1") + +-- responseBody :: Response body -> body +let body = responseBody resp + +-- you have two options, you use decode and just get Maybe (EsResult Tweet) +-- or you can use eitherDecode and get Either String (EsResult Tweet) + +let maybeResult = decode body :: Maybe (EsResult Tweet) +-- the explicit typing is so Aeson knows how to parse the JSON. + +-- use either if you want to know why something failed to parse. +-- (string errors, sadly) +let eitherResult = eitherDecode body :: Either String (EsResult Tweet) + +-- print eitherResult should look like: +Right (EsResult {_index = "twitter" + , _type = "tweet" + , _id = "1" + , foundResult = Just (EsResultFound { _version = 2 + , _source = Tweet {user = "bitemyapp" + , postDate = 2009-06-18 00:00:10 UTC + , message = "Use haskell!" + , age = 10000 + , location = Location {lat = 40.12, lon = -71.34}}})}) + +-- _source in EsResultFound is parametric, we dispatch the type by passing in what we expect (Tweet) as a parameter to EsResult. + +-- use the _source record accessor to get at your document +fmap (fmap _source . foundResult) eitherResult +Right (Just (Tweet {user = "bitemyapp" + , postDate = 2009-06-18 00:00:10 UTC + , message = "Use haskell!" + , age = 10000 + , location = Location {lat = 40.12, lon = -71.34}})) + +``` + +Bulk Operations +--------------- + +### Bulk create, index + +``` {.haskell} + +-- don't forget the imports and derive generic setting for ghci +-- at the beginning of the examples. + +:{ +-- Using the earlier Tweet datatype and exampleTweet data + +-- just changing up the data a bit. +let bulkTest = exampleTweet { user = "blah" } +let bulkTestTwo = exampleTweet { message = "woohoo!" } + +-- create only bulk operation +-- BulkCreate :: IndexName -> MappingName -> DocId -> Value -> BulkOperation +let firstOp = BulkCreate testIndex + testMapping (DocId "3") (toJSON bulkTest) + +-- index operation "create or update" +let sndOp = BulkIndex testIndex + testMapping (DocId "4") (toJSON bulkTestTwo) + +-- Some explanation, the final "Value" type that BulkIndex, +-- BulkCreate, and BulkUpdate accept is the actual document +-- data that your operation applies to. BulkDelete doesn't +-- take a value because it's just deleting whatever DocId +-- you pass. + +-- list of bulk operations +let stream = [firstDoc, secondDoc] + +-- Fire off the actual bulk request +-- bulk :: Vector BulkOperation -> IO Reply +resp <- withBH' $ bulk stream +:} + +``` + +### Encoding individual bulk API operations + +``` {.haskell} +-- the following functions are exported in Bloodhound so +-- you can build up bulk operations yourself +encodeBulkOperations :: V.Vector BulkOperation -> L.ByteString +encodeBulkOperation :: BulkOperation -> L.ByteString + +-- How to use the above: +data BulkTest = BulkTest { name :: Text } deriving (Eq, Generic, Show) +instance FromJSON BulkTest +instance ToJSON BulkTest + +_ <- insertData +let firstTest = BulkTest "blah" +let secondTest = BulkTest "bloo" +let firstDoc = BulkIndex testIndex + testMapping (DocId "2") (toJSON firstTest) +let secondDoc = BulkCreate testIndex + testMapping (DocId "3") (toJSON secondTest) +let stream = V.fromList [firstDoc, secondDoc] :: V.Vector BulkOperation + +-- to encode yourself +let firstDocEncoded = encode firstDoc :: L.ByteString + +-- to encode a vector of bulk operations +let encodedOperations = encodeBulkOperations stream + +-- to insert into a particular server +-- bulk :: V.Vector BulkOperation -> IO Reply +_ <- withBH' $ bulk streamp + +``` + +Search +------ + +### Querying + +#### Term Query + +``` {.haskell} + +-- exported by the Client module, just defaults some stuff. +-- mkSearch :: Maybe Query -> Maybe Filter -> Search +-- mkSearch query filter = Search query filter Nothing False (From 0) (Size 10) Nothing + +let query = TermQuery (Term "user" "bitemyapp") Nothing + +-- AND'ing identity filter with itself and then tacking it onto a query +-- search should be a null-operation. I include it for the sake of example. +-- <||> (or/plus) should make it into a search that returns everything. + +let filter = IdentityFilter <&&> IdentityFilter + +-- constructing the search object the searchByIndex function dispatches on. +let search = mkSearch (Just query) (Just filter) + +-- you can also searchByType and specify the mapping name. +reply <- withBH' $ searchByIndex testIndex search + +let result = eitherDecode (responseBody reply) :: Either String (SearchResult Tweet) + +λ> fmap (hits . searchHits) result +Right [Hit {hitIndex = IndexName "twitter" + , hitType = MappingName "tweet" + , hitDocId = DocId "1" + , hitScore = 0.30685282 + , hitSource = Tweet {user = "bitemyapp" + , postDate = 2009-06-18 00:00:10 UTC + , message = "Use haskell!" + , age = 10000 + , location = Location {lat = 40.12, lon = -71.34}}}] + +``` + +#### Match Query + +``` {.haskell} + +let query = QueryMatchQuery $ mkMatchQuery (FieldName "user") (QueryString "bitemyapp") +let search = mkSearch (Just query) Nothing + +``` + +#### Multi-Match Query + +``` {.haskell} + +let fields = [FieldName "user", FieldName "message"] +let query = QueryMultiMatchQuery $ mkMultiMatchQuery fields (QueryString "bitemyapp") +let search = mkSearch (Just query) Nothing + +``` + +#### Bool Query + +``` {.haskell} + +let innerQuery = QueryMatchQuery $ + mkMatchQuery (FieldName "user") (QueryString "bitemyapp") +let query = QueryBoolQuery $ + mkBoolQuery [innerQuery] [] [] +let search = mkSearch (Just query) Nothing + +``` + +#### Boosting Query + +``` {.haskell} + +let posQuery = QueryMatchQuery $ + mkMatchQuery (FieldName "user") (QueryString "bitemyapp") +let negQuery = QueryMatchQuery $ + mkMatchQuery (FieldName "user") (QueryString "notmyapp") +let query = QueryBoostingQuery $ + BoostingQuery posQuery negQuery (Boost 0.2) + +``` + +#### Rest of the query/filter types + +Just follow the pattern you've seen here and check the Hackage API documentation. + +### Sorting + +``` {.haskell} + +let sortSpec = DefaultSortSpec $ mkSort (FieldName "age") Ascending + +-- mkSort is a shortcut function that takes a FieldName and a SortOrder +-- to generate a vanilla DefaultSort. +-- checkt the DefaultSort type for the full list of customizable options. + +-- From and size are integers for pagination. + +-- When sorting on a field, scores are not computed. By setting TrackSortScores to true, scores will still be computed and tracked. + +-- type Sort = [SortSpec] +-- type TrackSortScores = Bool +-- type From = Int +-- type Size = Int + +-- Search takes Maybe Query +-- -> Maybe Filter +-- -> Maybe Sort +-- -> TrackSortScores +-- -> From -> Size +-- -> Maybe [FieldName] + +-- just add more sortspecs to the list if you want tie-breakers. +let search = Search Nothing (Just IdentityFilter) (Just [sortSpec]) False (From 0) (Size 10) Nothing + +``` + +### Field selection + +If you only want certain fields from the source document returned, you can +set the "fields" field of the Search record. + +``` {.haskell} + +let search' = mkSearch (Just (MatchAllQuery Nothing)) Nothing + search = search' { fields = Just [FieldName "updated"] } + +``` + +### Filtering + +#### And, Not, and Or filters + +Filters form a monoid and seminearring. + +``` {.haskell} + +instance Monoid Filter where + mempty = IdentityFilter + mappend a b = AndFilter [a, b] defaultCache + +instance Seminearring Filter where + a <||> b = OrFilter [a, b] defaultCache + +-- AndFilter and OrFilter take [Filter] as an argument. + +-- This will return anything, because IdentityFilter returns everything +OrFilter [IdentityFilter, someOtherFilter] False + +-- This will return exactly what someOtherFilter returns +AndFilter [IdentityFilter, someOtherFilter] False + +-- Thanks to the seminearring and monoid, the above can be expressed as: + +-- "and" +IdentityFilter <&&> someOtherFilter + +-- "or" +IdentityFilter <||> someOtherFilter + +-- Also there is a NotFilter, it only accepts a single filter, not a list. + +NotFilter someOtherFilter False + +``` + +#### Identity Filter + +``` {.haskell} + +-- And'ing two Identity +let queryFilter = IdentityFilter <&&> IdentityFilter + +let search = mkSearch Nothing (Just queryFilter) + +reply <- withBH' $ searchByType testIndex testMapping search + +``` + +#### Boolean Filter + +Similar to boolean queries. + +``` {.haskell} + +-- Will return only items whose "user" field contains the term "bitemyapp" +let queryFilter = BoolFilter (MustMatch (Term "user" "bitemyapp") False) + +-- Will return only items whose "user" field does not contain the term "bitemyapp" +let queryFilter = BoolFilter (MustNotMatch (Term "user" "bitemyapp") False) + +-- The clause (query) should appear in the matching document. +-- In a boolean query with no must clauses, one or more should +-- clauses must match a document. The minimum number of should +-- clauses to match can be set using the minimum_should_match parameter. +let queryFilter = BoolFilter (ShouldMatch [(Term "user" "bitemyapp")] False) + +``` + +#### Exists Filter + +``` {.haskell} + +-- Will filter for documents that have the field "user" +let existsFilter = ExistsFilter (FieldName "user") + +``` + +#### Geo BoundingBox Filter + +``` {.haskell} + +-- topLeft and bottomRight +let box = GeoBoundingBox (LatLon 40.73 (-74.1)) (LatLon 40.10 (-71.12)) + +let constraint = GeoBoundingBoxConstraint (FieldName "tweet.location") box False GeoFilterMemory + +``` + +#### Geo Distance Filter + +``` {.haskell} + +let geoPoint = GeoPoint (FieldName "tweet.location") (LatLon 40.12 (-71.34)) + +-- coefficient and units +let distance = Distance 10.0 Miles + +-- GeoFilterType or NoOptimizeBbox +let optimizeBbox = OptimizeGeoFilterType GeoFilterMemory + +-- SloppyArc is the usual/default optimization in Elasticsearch today +-- but pre-1.0 versions will need to pick Arc or Plane. + +let geoFilter = GeoDistanceFilter geoPoint distance SloppyArc optimizeBbox False + +``` + +#### Geo Distance Range Filter + +Think of a donut and you won't be far off. + +``` {.haskell} + +let geoPoint = GeoPoint (FieldName "tweet.location") (LatLon 40.12 (-71.34)) + +let distanceRange = DistanceRange (Distance 0.0 Miles) (Distance 10.0 Miles) + +let geoFilter = GeoDistanceRangeFilter geoPoint distanceRange + +``` + +#### Geo Polygon Filter + +``` {.haskell} + +-- I think I drew a square here. +let points = [LatLon 40.0 (-70.00), + LatLon 40.0 (-72.00), + LatLon 41.0 (-70.00), + LatLon 41.0 (-72.00)] + +let geoFilter = GeoPolygonFilter (FieldName "tweet.location") points + +``` + +#### Document IDs filter + +``` {.haskell} + +-- takes a mapping name and a list of DocIds +IdsFilter (MappingName "tweet") [DocId "1"] + +``` + +#### Range Filter + +``` {.haskell} + +-- RangeFilter :: FieldName +-- -> RangeValue +-- -> RangeExecution +-- -> Cache -> Filter + +let filter = RangeFilter (FieldName "age") + (RangeGtLt (GreaterThan 1000.0) (LessThan 100000.0)) + RangeExecutionIndex False + +``` + +``` {.haskell} + +let filter = RangeFilter (FieldName "age") + (RangeLte (LessThanEq 100000.0)) + RangeExecutionIndex False + +``` + +##### Date Ranges + +Date ranges are expressed in UTCTime. Date ranges use the same range bound constructors as numerics, except that they end in "D". + +Note that compatibility with ES is tested only down to seconds. + +``` {.haskell} + +let filter = RangeFilter (FieldName "postDate") + (RangeDateGtLte + (GreaterThanD (UTCTime + (ModifiedJulianDay 55000) + (secondsToDiffTime 9))) + (LessThanEqD (UTCTime + (ModifiedJulianDay 55000) + (secondsToDiffTime 11)))) + RangeExecutionIndex False +``` + +#### Regexp Filter + +``` {.haskell} + +-- RegexpFilter +-- :: FieldName +-- -> Regexp +-- -> RegexpFlags +-- -> CacheName +-- -> Cache +-- -> CacheKey +-- -> Filter +let filter = RegexpFilter (FieldName "user") (Regexp "bite.*app") + AllRegexpFlags (CacheName "test") False (CacheKey "key") + +-- n.b. +-- data RegexpFlags = AllRegexpFlags +-- | NoRegexpFlags +-- | SomeRegexpFlags (NonEmpty RegexpFlag) deriving (Eq, Show) + +-- data RegexpFlag = AnyString +-- | Automaton +-- | Complement +-- | Empty +-- | Intersection +-- | Interval deriving (Eq, Show) + +``` + +### Aggregations + +#### Adding aggregations to search + +Aggregations can now be added to search queries, or made on their own. + +``` {.haskell} +type Aggregations = M.Map Text Aggregation +data Aggregation + = TermsAgg TermsAggregation + | DateHistogramAgg DateHistogramAggregation +``` + +For convenience, \`\`\`mkAggregations\`\`\` exists, that will create an \`\`\`Aggregations\`\`\` with the aggregation provided. + +For example: + +``` {.haskell} + let a = mkAggregations "users" $ TermsAgg $ mkTermsAggregation "user" + let search = mkAggregateSearch Nothing a +``` + +Aggregations can be added to an existing search, using the \`\`\`aggBody\`\`\` field + +``` {.haskell} + let search = mkSearch (Just (MatchAllQuery Nothing)) Nothing + let search' = search {aggBody = Just a} +``` + +Since the \`\`\`Aggregations\`\`\` structure is just a Map Text Aggregation, M.insert can be used to add additional aggregations. + +``` {.haskell} + let a' = M.insert "age" (TermsAgg $ mkTermsAggregation "age") a +``` + +#### Extracting aggregations from results + +Aggregations are part of the reply structure of every search, in the form of + +``` {.haskell} +-- Lift decode and response body to be in the IO monad. +let decode' = liftM decode +let responseBody' = liftM responseBody +let reply = withBH' $ searchByIndex testIndex search +let response = decode' $ responseBody' reply :: IO (Maybe (SearchResult Tweet)) + +-- Now that we have our response, we can extract our terms aggregation result -- which is a list of buckets. + +let terms = do { response' <- response; return $ response' >>= aggregations >>= toTerms "users" } +terms +Just (Bucket {buckets = [TermsResult {termKey = "bitemyapp", termsDocCount = 1, termsAggs = Nothing}]}) +``` + +Note that bucket aggregation results, such as the TermsResult is a member of the type class : + +``` {.haskell} +class BucketAggregation a where + key :: a -> Text + docCount :: a -> Int + aggs :: a -> Maybe AggregationResults +``` + +haskell + +You can use the function to get any nested results, if there were any. For example, if there were a nested terms aggregation keyed to "age" in a TermsResult named , you would call + +#### Terms Aggregation + +``` {.haskell} +data TermsAggregation + = TermsAggregation {term :: Either Text Text, + termInclude :: Maybe TermInclusion, + termExclude :: Maybe TermInclusion, + termOrder :: Maybe TermOrder, + termMinDocCount :: Maybe Int, + termSize :: Maybe Int, + termShardSize :: Maybe Int, + termCollectMode :: Maybe CollectionMode, + termExecutionHint :: Maybe ExecutionHint, + termAggs :: Maybe Aggregations} +``` + +Term Aggregations have two factory functions, , and , and can be used as follows: + +``` {.haskell} +let ta = TermsAgg $ mkTermsAggregation "user" +``` + +There are of course other options that can be added to a Terms Aggregation, such as the collection mode: + +``` {.haskell} +let ta = mkTermsAggregation "user" +let ta' = ta { termCollectMode = Just BreadthFirst } +let ta'' = TermsAgg ta' +``` + +For more documentation on how the Terms Aggregation works, see + +#### Date Histogram Aggregation + +``` {.haskell} +data DateHistogramAggregation + = DateHistogramAggregation {dateField :: FieldName, + dateInterval :: Interval, + dateFormat :: Maybe Text, + datePreZone :: Maybe Text, + datePostZone :: Maybe Text, + datePreOffset :: Maybe Text, + datePostOffset :: Maybe Text, + dateAggs :: Maybe Aggregations} +``` + +haskell + +The Date Histogram Aggregation works much the same as the Terms Aggregation. + +Relevant functions include , and + +``` {.haskell} +let dh = DateHistogramAgg (mkDateHistogram (FieldName "postDate") Minute) +``` + +Date histograms also accept a : + +``` {.haskell} +FractionalInterval :: Float -> TimeInterval -> Interval +-- TimeInterval is the following: +data TimeInterval = Weeks | Days | Hours | Minutes | Seconds +``` + +It can be used as follows: + +``` {.haskell} +let dh = DateHistogramAgg (mkDateHistogram (FieldName "postDate") (FractionalInterval 1.5 Minutes)) +``` + +The is defined as: + +``` {.haskell} +data DateHistogramResult + = DateHistogramResult {dateKey :: Int, + dateKeyStr :: Maybe Text, + dateDocCount :: Int, + dateHistogramAggs :: Maybe AggregationResults} +``` + +It is an instance of , and can have nested aggregations in each bucket. + +Buckets can be extracted from a using + +For more information on the Date Histogram Aggregation, see: + + +Contributors +============ + +* [Chris Allen](https://github.com/bitemyapp) +* [Liam Atkinson](https://github.com/latkins) +* [Christopher Guiney](https://github.com/chrisguiney) +* [Curtis Carter](https://github.com/ccarter) +* [Michael Xavier](https://github.com/MichaelXavier) +* [Bob Long](https://github.com/bobjflong) +* [Maximilian Tagher](https://github.com/MaxGabriel) +* [Anna Kopp](https://github.com/annakopp) +* [Matvey B. Aksenov](https://github.com/supki) +* [Jan-Philip Loos](https://github.com/MaxDaten) + +Possible future functionality +============================= + +Span Queries +------------ + +Beginning here: + +Function Score Query +-------------------- + + + +Node discovery and failover +--------------------------- + +Might require TCP support. + +Support for TCP access to Elasticsearch +--------------------------------------- + +Pretend to be a transport client? + +Bulk cluster-join merge +----------------------- + +Might require making a lucene index on disk with the appropriate format. + +GeoShapeQuery +------------- + + + +GeoShapeFilter +-------------- + + + +Geohash cell filter +------------------- + + + +HasChild Filter +--------------- + + + +HasParent Filter +---------------- + + + +Indices Filter +-------------- + + + +Query Filter +------------ + + + +Script based sorting +-------------------- + + + +Collapsing redundantly nested and/or structures +----------------------------------------------- + +The Seminearring instance, if deeply nested can possibly produce nested structure that is redundant. Depending on how this affects ES performance, reducing this structure might be valuable. + +Runtime checking for cycles in data structures +---------------------------------------------- + +check for n \> 1 occurrences in DFS: + + + + + +Photo Origin +============ + +Photo from HA! Designs: diff --git a/bloodhound.cabal b/bloodhound.cabal --- a/bloodhound.cabal +++ b/bloodhound.cabal @@ -1,5 +1,5 @@ name: bloodhound -version: 0.8.0.0 +version: 0.9.0.0 synopsis: ElasticSearch client library for Haskell description: ElasticSearch made awesome for Haskell hackers homepage: https://github.com/bitemyapp/bloodhound @@ -12,6 +12,11 @@ build-type: Custom cabal-version: >=1.10 +extra-source-files: + README.md + changelog.md + + source-repository head type: git location: https://github.com/bitemyapp/bloodhound.git @@ -26,16 +31,16 @@ build-depends: base >= 4.3 && <5, bytestring >= 0.10.0 && <0.11, containers >= 0.5.0.0 && <0.6, - aeson >= 0.7 && <0.10, + aeson >= 0.10 && <0.11, http-client >= 0.3 && <0.5, + network-uri >= 2.6 && <2.7, semigroups >= 0.15 && <0.18, time >= 1.4 && <1.6, text >= 0.11 && <1.3, mtl >= 1.0 && <2.3, transformers >= 0.2 && <0.5, - http-types >= 0.8 && <0.9, + http-types >= 0.8 && <0.10, vector >= 0.10.9 && <0.12, - uri-bytestring >= 0.1 && <0.2, exceptions, data-default-class, blaze-builder, diff --git a/changelog.md b/changelog.md new file mode 100644 --- /dev/null +++ b/changelog.md @@ -0,0 +1,109 @@ +0.8.0.0 +=================== + +Thanks to the following people, Bloodhound 0.8.0.0 is being released! + +* @MichaelXavier + - #67: Deriving Monad(Throw|Catch|Mask) + - #64: Export BH constructor + - #61: Filter aggregation support + - #60: Add value_count aggregation support + - #58: Eliminate partiality in EsResult + +* @centromere + - #59: Fixed bug with IndexSettings serialization + - #56: Added fields support to Search + - #55: Added ability to specify a parent document + - #54: Fixed IndexTemplate serialization bug + - #52: Added ability to manipulate templates + - #51: Fixed mapping API + - #50: Fixed problem with put sending POST + +* @bermanjosh + - #63: Url query encoding bug + - #53: Scan type + +* @sjakobi + - #69: Replace Control.Monad.Error with CM.Except via mtl-compat + - #70: Silence redundant import warning with base-4.8.* + - #71: Use "newManager" instead of deprecated "withManager" + +0.7.0.0 +=================== + +* Added QueryFilter thanks to Bjørn Nordbø! + +* Support for optimistic concurrency control thanks again to @MichaelXavier! + +0.6.0.1 +=================== + +* Allow Aeson 0.9 + +0.6.0.0 +=================== + +* Moved to BHMonad, thanks to @MichaelXavier! Now there's a reader of config information and IO is lifted. + +* SearchHits have a Monoid now, makes combining search results nicer, allows for defaulting when a search cannot be performed. + +0.5.0.0 +=================== + +* Fixed and changed TermsQuery (This caused the major bump) + +* Removed benchmarks from travis.yml + +* Added doctests, examples for Database.Bloodhound.Client. Haddocks should be much nicer. + +* Various fixes, reformatting + +0.4.0.0 +=================== + +* Term and date aggregation - thanks to Christopher Guiney! (@chrisguiney) + +Following three thanks to Liam Atkins (@latkins) + +* omitNulls changed to exclude empty lists and null values + +* BoolQuery must/mustNot/Should changed from Maybe (Query|[Query]) to [Query] thanks to @latkins + +* Added vector dependency so we can check for V.null/V.empty on JSON arrays + +* Highlighting, thanks to @latkins! See http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-highlighting.html and http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/highlighting-intro.html for more + +* Added 1.4.0 support and CI integration + +* Can generate individual bulk operations, https://github.com/bitemyapp/bloodhound/issues/17, bulk requests should be more efficient now too - Vector instead of List. + +0.3.0.0 +=================== + +* Status "ok" changed from Bool to Maybe Bool thanks to @borisyukd + +* Elasticsearch 1.3.x compatibility fixed with changes to geo bounding boxes - thanks to Curtis Carter! (@ccarter) + +* CI coverage expanded to 1.0.x -> 1.3.x + +0.2.0.1 +=================== + +* Killed off maybeJson/mField/catMaybes in favor of omitNulls + +* Experimenting with RecordWildcards + +* Merged Types and Instances module into Types to prevent possibility of orphans and cull orphan instance warnings. + +* Added note about current supported Elasticsearch version. + +0.2.0.0 +=================== + +* Added TermFilter + +* Renamed createMapping to putMapping + +* Fixed and rebuilt documentation + +* RegexpFlags changed to a sum type instead of Text, thanks to @MichaelXavier! diff --git a/src/Database/Bloodhound/Client.hs b/src/Database/Bloodhound/Client.hs --- a/src/Database/Bloodhound/Client.hs +++ b/src/Database/Bloodhound/Client.hs @@ -66,9 +66,9 @@ import Data.Aeson import Data.ByteString.Lazy.Builder import qualified Data.ByteString.Lazy.Char8 as L -import Data.Default.Class import Data.Ix -import Data.Maybe (fromMaybe) +import qualified Data.List as LS (filter) +import Data.Maybe (fromMaybe, isJust) import Data.Monoid import Data.Text (Text) import qualified Data.Text as T @@ -78,8 +78,8 @@ import qualified Network.HTTP.Types.Method as NHTM import qualified Network.HTTP.Types.Status as NHTS import qualified Network.HTTP.Types.URI as NHTU +import qualified Network.URI as URI import Prelude hiding (filter, head) -import URI.ByteString hiding (Query) import Database.Bloodhound.Types @@ -499,14 +499,16 @@ -- | 'documentExists' enables you to check if a document exists. Returns 'Bool' -- in IO -- --- >>> exists <- runBH' $ documentExists testIndex testMapping (DocId "1") +-- >>> exists <- runBH' $ documentExists testIndex testMapping Nothing (DocId "1") documentExists :: MonadBH m => IndexName -> MappingName - -> DocId -> m Bool -documentExists (IndexName indexName) - (MappingName mappingName) (DocId docId) = do + -> Maybe DocumentParent -> DocId -> m Bool +documentExists (IndexName indexName) (MappingName mappingName) + parent (DocId docId) = do (_, exists) <- existentialQuery =<< url return exists - where url = joinPath [indexName, mappingName, docId] + where url = addQuery params <$> joinPath [indexName, mappingName, docId] + parentParam = fmap (\(DocumentParent (DocId p)) -> p) parent + params = LS.filter (\(_, v) -> isJust v) [("parent", parentParam)] dispatchSearch :: MonadBH m => Text -> Search -> m Reply dispatchSearch url search = post url' (Just (encode search)) @@ -628,32 +630,7 @@ pageSearch resultOffset pageSize search = search { from = resultOffset, size = pageSize } parseUrl' :: MonadThrow m => Text -> m Request -parseUrl' t = - case parseURI laxURIParserOptions (T.encodeUtf8 t) of - Right uri -> setURI def uri - Left e -> throwM $ InvalidUrlException (T.unpack t) ("Invalid URL: " ++ show e) - -setURI :: MonadThrow m => Request -> URI -> m Request -setURI req URI{..} = do - Authority {..} <- maybe missingUA return uriAuthority - let req' = req { secure = isSecure - , host = hostBS authorityHost - , port = thePort - , path = uriPath - } - thePort = maybe defPort portNumber authorityPort - addAuth = maybe id addAuth' authorityUserInfo - return $ setQueryString theQueryString $ addAuth req' - where - missingUA = throwM $ InvalidUrlException "N/A" "Missing URI host/port" - addAuth' UserInfo {..} = applyBasicProxyAuth uiUsername uiPassword - defPort - | isSecure = 443 - | otherwise = 80 - isSecure = case uriScheme of - Scheme "https" -> True - _ -> False - theQueryString = [(k , Just v) | (k, v) <- queryPairs uriQuery] +parseUrl' t = parseUrl (URI.escapeURIString URI.isAllowedInURI (T.unpack t)) -- | Was there an optimistic concurrency control conflict when -- indexing a document? diff --git a/src/Database/Bloodhound/Types.hs b/src/Database/Bloodhound/Types.hs --- a/src/Database/Bloodhound/Types.hs +++ b/src/Database/Bloodhound/Types.hs @@ -61,6 +61,7 @@ , Reply , EsResult(..) , EsResultFound(..) + , EsError(..) , DocVersion , ExternalDocVersion(..) , VersionControl(..) @@ -416,13 +417,18 @@ , _id :: Text , foundResult :: Maybe (EsResultFound a)} deriving (Eq, Show) - {-| 'EsResultFound' contains the document and its metadata inside of an 'EsResult' when the document was successfully found. -} data EsResultFound a = EsResultFound { _version :: DocVersion , _source :: a } deriving (Eq, Show) +{-| 'EsError' is the generic type that will be returned when there was a + problem. If you can't parse the expected response, its a good idea to + try parsing this. +-} +data EsError = EsError { errorStatus :: Int + , errorMessage :: Text } deriving (Eq, Show) {-| 'DocVersion' is an integer version number for a document between 1 @@ -756,7 +762,7 @@ deriving (Eq, Show) data Source = - NoSource + NoSource | SourcePatterns PatternOrPatterns | SourceIncludeExclude Include Exclude deriving (Show, Eq) @@ -1305,7 +1311,7 @@ , hitType :: MappingName , hitDocId :: DocId , hitScore :: Score - , hitSource :: a + , hitSource :: Maybe a , hitHighlight :: Maybe HitHighlight } deriving (Eq, Show) data ShardResult = @@ -2133,6 +2139,12 @@ v .: "_source" parseJSON _ = empty +instance FromJSON EsError where + parseJSON (Object v) = EsError <$> + v .: "status" <*> + v .: "error" + parseJSON _ = empty + instance ToJSON Search where toJSON (Search query sFilter sort searchAggs highlight sTrackSortScores sFrom sSize _ sFields sSource) = omitNulls [ "query" .= query @@ -2396,11 +2408,11 @@ instance (FromJSON a) => FromJSON (Hit a) where parseJSON (Object v) = Hit <$> - v .: "_index" <*> - v .: "_type" <*> - v .: "_id" <*> - v .: "_score" <*> - v .: "_source" <*> + v .: "_index" <*> + v .: "_type" <*> + v .: "_id" <*> + v .: "_score" <*> + v .:? "_source" <*> v .:? "highlight" parseJSON _ = empty diff --git a/tests/tests.hs b/tests/tests.hs --- a/tests/tests.hs +++ b/tests/tests.hs @@ -14,6 +14,7 @@ import Data.List.NonEmpty (NonEmpty (..)) import qualified Data.List.NonEmpty as NE import qualified Data.Map.Strict as M +import qualified Data.Maybe as MY import Data.Monoid import Data.Text (Text) import qualified Data.Text as T @@ -27,7 +28,7 @@ import qualified Network.HTTP.Types.Status as NHTS import Prelude hiding (filter) import Test.Hspec -import Test.QuickCheck.Property.Monoid (prop_Monoid, eq, T(..)) +import Test.QuickCheck.Property.Monoid (T (..), eq, prop_Monoid) import Test.Hspec.QuickCheck (prop) import Test.QuickCheck @@ -116,6 +117,20 @@ instance ToJSON Location instance FromJSON Location +data ParentMapping = ParentMapping deriving (Eq, Show) + +instance ToJSON ParentMapping where + toJSON ParentMapping = + object ["parent" .= Null ] + +data ChildMapping = ChildMapping deriving (Eq, Show) + +instance ToJSON ChildMapping where + toJSON ChildMapping = + object ["child" .= + object ["_parent" .= object ["type" .= ("parent" :: Text)]] + ] + data TweetMapping = TweetMapping deriving (Eq, Show) instance ToJSON TweetMapping where @@ -173,10 +188,17 @@ _ <- refreshIndex testIndex return () +insertWithSpaceInId :: BH IO () +insertWithSpaceInId = do + _ <- indexDocument testIndex testMapping defaultIndexDocumentSettings exampleTweet (DocId "Hello World") + _ <- refreshIndex testIndex + return () + searchTweet :: Search -> BH IO (Either String Tweet) searchTweet search = do result <- searchTweets search - let myTweet = fmap (hitSource . head . hits . searchHits) result + let myTweet :: Either String Tweet + myTweet = grabFirst result return myTweet searchTweets :: Search -> BH IO (Either String (SearchResult Tweet)) @@ -227,7 +249,7 @@ let search = (mkSearch (Just query) Nothing) { source = Just src } reply <- searchAll search let result = eitherDecode (responseBody reply) :: Either String (SearchResult Value) - let value = fmap (hitSource . head . hits . searchHits) result + let value = grabFirst result liftIO $ value `shouldBe` expected @@ -293,6 +315,13 @@ getSource :: EsResult a -> Maybe a getSource = fmap _source . foundResult +-- grabFirst :: Either String (SearchResult a) -> Either String a +grabFirst r = + case fmap (hitSource . head . hits . searchHits) r of + (Left e) -> Left e + (Right Nothing) -> Left "Source was missing" + (Right (Just x)) -> Right x + main :: IO () main = hspec $ do @@ -315,6 +344,13 @@ (responseBody docInserted) :: Either String (EsResult Tweet) liftIO $ (fmap getSource newTweet `shouldBe` Right (Just exampleTweet)) + it "indexes, gets, and then deletes the generated document with a DocId containing a space" $ withTestEnv $ do + _ <- insertWithSpaceInId + docInserted <- getDocument testIndex testMapping (DocId "Hello World") + let newTweet = eitherDecode + (responseBody docInserted) :: Either String (EsResult Tweet) + liftIO $ (fmap getSource newTweet `shouldBe` Right (Just exampleTweet)) + it "produces a parseable result when looking up a bogus document" $ withTestEnv $ do doc <- getDocument testIndex testMapping (DocId "bogus") let noTweet = eitherDecode @@ -330,6 +366,18 @@ res' <- insertData' cfg liftIO $ isVersionConflict res' `shouldBe` True + it "indexes two documents in a parent/child relationship and checks that the child exists" $ withTestEnv $ do + resetIndex + _ <- putMapping testIndex (MappingName "parent") ParentMapping + _ <- putMapping testIndex (MappingName "child") ChildMapping + _ <- indexDocument testIndex (MappingName "parent") defaultIndexDocumentSettings exampleTweet (DocId "1") + let parent = (Just . DocumentParent . DocId) "1" + ids = IndexDocumentSettings NoVersionControl parent + _ <- indexDocument testIndex (MappingName "child") ids otherTweet (DocId "2") + _ <- refreshIndex testIndex + exists <- documentExists testIndex (MappingName "child") parent (DocId "2") + liftIO $ exists `shouldBe` True + describe "template API" $ do it "can create a template" $ withTestEnv $ do let idxTpl = IndexTemplate (TemplatePattern "tweet-*") (Just (IndexSettings (ShardCount 1) (ReplicaCount 1))) [toJSON TweetMapping] @@ -448,7 +496,7 @@ (Just IdentityFilter) (Just [sortSpec]) Nothing Nothing False (From 0) (Size 10) SearchTypeQueryThenFetch Nothing Nothing result <- searchTweets search - let myTweet = fmap (hitSource . head . hits . searchHits) result + let myTweet = grabFirst result liftIO $ myTweet `shouldBe` Right otherTweet @@ -665,26 +713,26 @@ fmap aggregations res `shouldBe` Right (Just (M.fromList [ docCountPair "bitemyapps" 1 , docCountPair "notmyapps" 1 ])) - -- Interaction of date serialization and date histogram aggregation is broken. - -- it "returns date histogram aggregation results" $ withTestEnv $ do - -- _ <- insertData - -- let histogram = DateHistogramAgg $ mkDateHistogram (FieldName "postDate") Minute - -- let search = mkAggregateSearch Nothing (mkAggregations "byDate" histogram) - -- searchExpectAggs search - -- searchValidBucketAgg search "byDate" toDateHistogram - -- it "returns date histogram using fractional date" $ withTestEnv $ do - -- _ <- insertData - -- let periods = [Year, Quarter, Month, Week, Day, Hour, Minute, Second] - -- let fractionals = map (FractionalInterval 1.5) [Weeks, Days, Hours, Minutes, Seconds] - -- let intervals = periods ++ fractionals - -- let histogram = mkDateHistogram (FieldName "postDate") - -- let search interval = mkAggregateSearch Nothing $ mkAggregations "byDate" $ DateHistogramAgg (histogram interval) - -- let expect interval = searchExpectAggs (search interval) - -- let valid interval = searchValidBucketAgg (search interval) "byDate" toDateHistogram - -- forM_ intervals expect - -- forM_ intervals valid + it "returns date histogram aggregation results" $ withTestEnv $ do + _ <- insertData + let histogram = DateHistogramAgg $ mkDateHistogram (FieldName "postDate") Minute + let search = mkAggregateSearch Nothing (mkAggregations "byDate" histogram) + searchExpectAggs search + searchValidBucketAgg search "byDate" toDateHistogram + it "returns date histogram using fractional date" $ withTestEnv $ do + _ <- insertData + let periods = [Year, Quarter, Month, Week, Day, Hour, Minute, Second] + let fractionals = map (FractionalInterval 1.5) [Weeks, Days, Hours, Minutes, Seconds] + let intervals = periods ++ fractionals + let histogram = mkDateHistogram (FieldName "postDate") + let search interval = mkAggregateSearch Nothing $ mkAggregations "byDate" $ DateHistogramAgg (histogram interval) + let expect interval = searchExpectAggs (search interval) + let valid interval = searchValidBucketAgg (search interval) "byDate" toDateHistogram + forM_ intervals expect + forM_ intervals valid + describe "Highlights API" $ do it "returns highlight from query when there should be one" $ withTestEnv $ do @@ -714,7 +762,7 @@ it "doesn't include source when sources are disabled" $ withTestEnv $ do searchExpectSource NoSource - (Left "key \"_source\" not present") + (Left "Source was missing") it "includes a source" $ withTestEnv $ do searchExpectSource @@ -808,6 +856,6 @@ scan_search' <- scanSearch testIndex testMapping search :: BH IO [Hit Tweet] let scan_search = map hitSource scan_search' liftIO $ - regular_search `shouldBe` Right exampleTweet -- Check that the size restrtiction is being honored + regular_search `shouldBe` Right exampleTweet -- Check that the size restrtiction is being honored liftIO $ - scan_search `shouldMatchList` [exampleTweet, otherTweet] + scan_search `shouldMatchList` [Just exampleTweet, Just otherTweet]