bloodhound 0.8.0.0 → 0.9.0.0

raw patch · 6 files changed

+1198/−71 lines, 6 filesdep +network-uridep −uri-bytestringdep ~aesondep ~http-typesPVP ok

version bump matches the API change (PVP)

Dependencies added: network-uri

Dependencies removed: uri-bytestring

Dependency ranges changed: aeson, http-types

API changes (from Hackage documentation)

Files

• README.md +976/−0
• bloodhound.cabal +9/−4
• changelog.md +109/−0
• src/Database/Bloodhound/Client.hs +11/−34
• src/Database/Bloodhound/Types.hs +20/−8
• tests/tests.hs +73/−25

@@ -0,0 +1,976 @@+Bloodhound [![TravisCI](https://travis-ci.org/bitemyapp/bloodhound.svg)](https://travis-ci.org/bitemyapp/bloodhound) [![Hackage](https://img.shields.io/hackage/v/bloodhound.svg?style=flat)](https://hackage.haskell.org/package/bloodhound)+==========++![Bloodhound (dog)](./bloodhound.jpg)++Elasticsearch client and query DSL for Haskell+==============================================++Why?+----++Search doesn't have to be hard. Let the dog do it.++Endorsements+------------++"Bloodhound makes Elasticsearch almost tolerable!" - Almost-gruntled user++"ES is a nightmare but Bloodhound at least makes it tolerable." - Same user, later opinion.++Version compatibility+---------------------++Elasticsearch \>= 1.0 is recommended. Bloodhound mostly works with 0.9.x, but I don't recommend it if you expect everything to work. As of Bloodhound 0.3 all \>=1.0 versions of Elasticsearch work.++Current versions we test against are 1.0.3, 1.1.2, 1.2.3, 1.3.2, and 1.4.0. We also check that GHC 7.6 and 7.8 both build and pass tests. See our [TravisCI](https://travis-ci.org/bitemyapp/bloodhound) to learn more.++Stability+---------++Bloodhound is stable for production use. I will strive to avoid breaking API compatibility from here on forward, but dramatic features like a type-safe, fully integrated mapping API may require breaking things in the future.++Hackage page and Haddock documentation+======================================++<http://hackage.haskell.org/package/bloodhound>++Elasticsearch Tutorial+======================++It's not using Bloodhound, but if you need an introduction to or overview of Elasticsearch and how to use it, you can use [this screencast](http://vimeo.com/106463167).++Examples+========++Index Operations+----------------++### Create Index++``` {.haskell}++-- Formatted for use in ghci, so there are "let"s in front of the decls.++-- if you see :{ and :}, they're so you can copy-paste+-- the multi-line examples into your ghci REPL.++:set -XDeriveGeneric+:{+import Control.Applicative+import Database.Bloodhound+import Data.Aeson+import Data.Either (Either(..))+import Data.Maybe (fromJust)+import Data.Time.Calendar (Day(..))+import Data.Time.Clock (secondsToDiffTime, UTCTime(..))+import Data.Text (Text)+import GHC.Generics (Generic)+import Network.HTTP.Client+import qualified Network.HTTP.Types.Status as NHTS++-- no trailing slashes in servers, library handles building the path.+let testServer = (Server "http://localhost:9200")+let testIndex = IndexName "twitter"+let testMapping = MappingName "tweet"+let withBH' = withBH defaultManagerSettings testServer++-- defaultIndexSettings is exported by Database.Bloodhound as well+let defaultIndexSettings = IndexSettings (ShardCount 3) (ReplicaCount 2)++-- createIndex returns MonadBH m => m Reply. You can use withBH for+   one-off commands or you can use runBH to group together commands+   and to pass in your own HTTP manager for pipelining.++-- response :: Reply, Reply is a synonym for Network.HTTP.Conduit.Response+response <- withBH' $ createIndex defaultIndexSettings testIndex+:}++```++### Delete Index++#### Code++``` {.haskell}++-- response :: Reply+response <- withBH' $ deleteIndex testIndex++```++#### Example Response++``` {.haskell}++-- print response if it was a success+Response {responseStatus = Status {statusCode = 200, statusMessage = "OK"}+        , responseVersion = HTTP/1.1+        , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")+                           , ("Content-Length", "21")]+        , responseBody = "{\"acknowledged\":true}"+        , responseCookieJar = CJ {expose = []}+        , responseClose' = ResponseClose}++-- if the index to be deleted didn't exist anyway+Response {responseStatus = Status {statusCode = 404, statusMessage = "Not Found"}+        , responseVersion = HTTP/1.1+        , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")+                           , ("Content-Length","65")]+        , responseBody = "{\"error\":\"IndexMissingException[[twitter] missing]\",\"status\":404}"+        , responseCookieJar = CJ {expose = []}+        , responseClose' = ResponseClose}++```++### Refresh Index++#### Note, you **have** to do this if you expect to read what you just wrote++``` {.haskell}++resp <- withBH' $ refreshIndex testIndex++```++#### Example Response++``` {.haskell}++-- print resp on success+Response {responseStatus = Status {statusCode = 200, statusMessage = "OK"}+        , responseVersion = HTTP/1.1+        , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")+                           , ("Content-Length","50")]+        , responseBody = "{\"_shards\":{\"total\":10,\"successful\":5,\"failed\":0}}"+        , responseCookieJar = CJ {expose = []}+        , responseClose' = ResponseClose}++```++Mapping Operations+------------------++### Create Mapping++``` {.haskell}++-- don't forget imports and the like at the top.++data TweetMapping = TweetMapping deriving (Eq, Show)++-- I know writing the JSON manually sucks.+-- I don't have a proper data type for Mappings yet.+-- Let me know if this is something you need.++:{+instance ToJSON TweetMapping where+  toJSON TweetMapping =+    object ["tweet" .=+      object ["properties" .=+        object ["location" .=+          object ["type" .= ("geo_point" :: Text)]]]]+:}++resp <- withBH' $ putMapping testIndex testMapping TweetMapping++```++### Delete Mapping++``` {.haskell}++resp <- withBH' $ deleteMapping testIndex testMapping++```++Document Operations+-------------------++### Indexing Documents++``` {.haskell}++-- don't forget the imports and derive generic setting for ghci+-- at the beginning of the examples.++:{+data Location = Location { lat :: Double+                         , lon :: Double } deriving (Eq, Generic, Show)++data Tweet = Tweet { user     :: Text+                   , postDate :: UTCTime+                   , message  :: Text+                   , age      :: Int+                   , location :: Location } deriving (Eq, Generic, Show)++exampleTweet = Tweet { user     = "bitemyapp"+                     , postDate = UTCTime+                                  (ModifiedJulianDay 55000)+                                  (secondsToDiffTime 10)+                     , message  = "Use haskell!"+                     , age      = 10000+                     , location = Location 40.12 (-71.34) }++-- automagic (generic) derivation of instances because we're lazy.+instance ToJSON   Tweet+instance FromJSON Tweet+instance ToJSON   Location+instance FromJSON Location+:}++-- Should be able to toJSON and encode the data structures like this:+-- λ> toJSON $ Location 10.0 10.0+-- Object fromList [("lat",Number 10.0),("lon",Number 10.0)]+-- λ> encode $ Location 10.0 10.0+-- "{\"lat\":10,\"lon\":10}"++resp <- withBH' $ indexDocument testIndex testMapping defaultIndexDocumentSettings exampleTweet (DocId "1")++```++#### Example Response++``` {.haskell}++Response {responseStatus =+  Status {statusCode = 200, statusMessage = "OK"}+    , responseVersion = HTTP/1.1, responseHeaders =+    [("Content-Type","application/json; charset=UTF-8"),+     ("Content-Length","75")]+    , responseBody = "{\"_index\":\"twitter\",\"_type\":\"tweet\",\"_id\":\"1\",\"_version\":2,\"created\":false}"+    , responseCookieJar = CJ {expose = []}, responseClose' = ResponseClose}++```++### Deleting Documents++``` {.haskell}++resp <- withBH' $ deleteDocument testIndex testMapping (DocId "1")++```++### Getting Documents++``` {.haskell}++-- n.b., you'll need the earlier imports. responseBody is from http-conduit++resp <- withBH' $ getDocument testIndex testMapping (DocId "1")++-- responseBody :: Response body -> body+let body = responseBody resp++-- you have two options, you use decode and just get Maybe (EsResult Tweet)+-- or you can use eitherDecode and get Either String (EsResult Tweet)++let maybeResult = decode body :: Maybe (EsResult Tweet)+-- the explicit typing is so Aeson knows how to parse the JSON.++-- use either if you want to know why something failed to parse.+-- (string errors, sadly)+let eitherResult = eitherDecode body :: Either String (EsResult Tweet)++-- print eitherResult should look like:+Right (EsResult {_index = "twitter"+               , _type = "tweet"+               , _id = "1"+               , foundResult = Just (EsResultFound { _version = 2+                                                   , _source = Tweet {user = "bitemyapp"+                                                                     , postDate = 2009-06-18 00:00:10 UTC+                                                                     , message = "Use haskell!"+                                                                     , age = 10000+                                                                     , location = Location {lat = 40.12, lon = -71.34}}})})++-- _source in EsResultFound is parametric, we dispatch the type by passing in what we expect (Tweet) as a parameter to EsResult.++-- use the _source record accessor to get at your document+fmap (fmap _source . foundResult) eitherResult+Right (Just (Tweet {user = "bitemyapp"+                   , postDate = 2009-06-18 00:00:10 UTC+                   , message = "Use haskell!"+                   , age = 10000+                   , location = Location {lat = 40.12, lon = -71.34}}))++```++Bulk Operations+---------------++### Bulk create, index++``` {.haskell}++-- don't forget the imports and derive generic setting for ghci+-- at the beginning of the examples.++:{+-- Using the earlier Tweet datatype and exampleTweet data++-- just changing up the data a bit.+let bulkTest = exampleTweet { user = "blah" }+let bulkTestTwo = exampleTweet { message = "woohoo!" }++-- create only bulk operation+-- BulkCreate :: IndexName -> MappingName -> DocId -> Value -> BulkOperation+let firstOp = BulkCreate testIndex+              testMapping (DocId "3") (toJSON bulkTest)++-- index operation "create or update"+let sndOp   = BulkIndex testIndex+              testMapping (DocId "4") (toJSON bulkTestTwo)++-- Some explanation, the final "Value" type that BulkIndex,+-- BulkCreate, and BulkUpdate accept is the actual document+-- data that your operation applies to. BulkDelete doesn't+-- take a value because it's just deleting whatever DocId+-- you pass.++-- list of bulk operations+let stream = [firstDoc, secondDoc]++-- Fire off the actual bulk request+-- bulk :: Vector BulkOperation -> IO Reply+resp <- withBH' $ bulk stream+:}++```++### Encoding individual bulk API operations++``` {.haskell}+-- the following functions are exported in Bloodhound so+-- you can build up bulk operations yourself+encodeBulkOperations :: V.Vector BulkOperation -> L.ByteString+encodeBulkOperation :: BulkOperation -> L.ByteString++-- How to use the above:+data BulkTest = BulkTest { name :: Text } deriving (Eq, Generic, Show)+instance FromJSON BulkTest+instance ToJSON BulkTest++_ <- insertData+let firstTest = BulkTest "blah"+let secondTest = BulkTest "bloo"+let firstDoc = BulkIndex testIndex+               testMapping (DocId "2") (toJSON firstTest)+let secondDoc = BulkCreate testIndex+               testMapping (DocId "3") (toJSON secondTest)+let stream = V.fromList [firstDoc, secondDoc] :: V.Vector BulkOperation++-- to encode yourself+let firstDocEncoded = encode firstDoc :: L.ByteString++-- to encode a vector of bulk operations+let encodedOperations = encodeBulkOperations stream++-- to insert into a particular server+-- bulk :: V.Vector BulkOperation -> IO Reply+_ <- withBH' $ bulk streamp++```++Search+------++### Querying++#### Term Query++``` {.haskell}++-- exported by the Client module, just defaults some stuff.+-- mkSearch :: Maybe Query -> Maybe Filter -> Search+-- mkSearch query filter = Search query filter Nothing False (From 0) (Size 10) Nothing++let query = TermQuery (Term "user" "bitemyapp") Nothing++-- AND'ing identity filter with itself and then tacking it onto a query+-- search should be a null-operation. I include it for the sake of example.+-- <||> (or/plus) should make it into a search that returns everything.++let filter = IdentityFilter <&&> IdentityFilter++-- constructing the search object the searchByIndex function dispatches on.+let search = mkSearch (Just query) (Just filter)++-- you can also searchByType and specify the mapping name.+reply <- withBH' $ searchByIndex testIndex search++let result = eitherDecode (responseBody reply) :: Either String (SearchResult Tweet)++λ> fmap (hits . searchHits) result+Right [Hit {hitIndex = IndexName "twitter"+          , hitType = MappingName "tweet"+          , hitDocId = DocId "1"+          , hitScore = 0.30685282+          , hitSource = Tweet {user = "bitemyapp"+                             , postDate = 2009-06-18 00:00:10 UTC+                             , message = "Use haskell!"+                             , age = 10000+                             , location = Location {lat = 40.12, lon = -71.34}}}]++```++#### Match Query++``` {.haskell}++let query = QueryMatchQuery $ mkMatchQuery (FieldName "user") (QueryString "bitemyapp")+let search = mkSearch (Just query) Nothing++```++#### Multi-Match Query++``` {.haskell}++let fields = [FieldName "user", FieldName "message"]+let query = QueryMultiMatchQuery $ mkMultiMatchQuery fields (QueryString "bitemyapp")+let search = mkSearch (Just query) Nothing++```++#### Bool Query++``` {.haskell}++let innerQuery = QueryMatchQuery $+                 mkMatchQuery (FieldName "user") (QueryString "bitemyapp")+let query = QueryBoolQuery $+            mkBoolQuery [innerQuery] [] []+let search = mkSearch (Just query) Nothing++```++#### Boosting Query++``` {.haskell}++let posQuery = QueryMatchQuery $+               mkMatchQuery (FieldName "user") (QueryString "bitemyapp")+let negQuery = QueryMatchQuery $+               mkMatchQuery (FieldName "user") (QueryString "notmyapp")+let query = QueryBoostingQuery $+            BoostingQuery posQuery negQuery (Boost 0.2)++```++#### Rest of the query/filter types++Just follow the pattern you've seen here and check the Hackage API documentation.++### Sorting++``` {.haskell}++let sortSpec = DefaultSortSpec $ mkSort (FieldName "age") Ascending++-- mkSort is a shortcut function that takes a FieldName and a SortOrder+-- to generate a vanilla DefaultSort.+-- checkt the DefaultSort type for the full list of customizable options.++-- From and size are integers for pagination.++-- When sorting on a field, scores are not computed. By setting TrackSortScores to true, scores will still be computed and tracked.++-- type Sort = [SortSpec]+-- type TrackSortScores = Bool+-- type From = Int+-- type Size = Int++-- Search takes Maybe Query+--              -> Maybe Filter+--              -> Maybe Sort+--              -> TrackSortScores+--              -> From -> Size+--              -> Maybe [FieldName]++-- just add more sortspecs to the list if you want tie-breakers.+let search = Search Nothing (Just IdentityFilter) (Just [sortSpec]) False (From 0) (Size 10) Nothing++```++### Field selection++If you only want certain fields from the source document returned, you can+set the "fields" field of the Search record.++``` {.haskell}++let search' = mkSearch (Just (MatchAllQuery Nothing)) Nothing+    search  = search' { fields = Just [FieldName "updated"] }++```++### Filtering++#### And, Not, and Or filters++Filters form a monoid and seminearring.++``` {.haskell}++instance Monoid Filter where+  mempty = IdentityFilter+  mappend a b = AndFilter [a, b] defaultCache++instance Seminearring Filter where+  a <||> b = OrFilter [a, b] defaultCache++-- AndFilter and OrFilter take [Filter] as an argument.++-- This will return anything, because IdentityFilter returns everything+OrFilter [IdentityFilter, someOtherFilter] False++-- This will return exactly what someOtherFilter returns+AndFilter [IdentityFilter, someOtherFilter] False++-- Thanks to the seminearring and monoid, the above can be expressed as:++-- "and"+IdentityFilter <&&> someOtherFilter++-- "or"+IdentityFilter <||> someOtherFilter++-- Also there is a NotFilter, it only accepts a single filter, not a list.++NotFilter someOtherFilter False++```++#### Identity Filter++``` {.haskell}++-- And'ing two Identity+let queryFilter = IdentityFilter <&&> IdentityFilter++let search = mkSearch Nothing (Just queryFilter)++reply <- withBH' $ searchByType testIndex testMapping search++```++#### Boolean Filter++Similar to boolean queries.++``` {.haskell}++-- Will return only items whose "user" field contains the term "bitemyapp"+let queryFilter = BoolFilter (MustMatch (Term "user" "bitemyapp") False)++-- Will return only items whose "user" field does not contain the term "bitemyapp"+let queryFilter = BoolFilter (MustNotMatch (Term "user" "bitemyapp") False)++-- The clause (query) should appear in the matching document.+-- In a boolean query with no must clauses, one or more should+-- clauses must match a document. The minimum number of should+-- clauses to match can be set using the minimum_should_match parameter.+let queryFilter = BoolFilter (ShouldMatch [(Term "user" "bitemyapp")] False)++```++#### Exists Filter++``` {.haskell}++-- Will filter for documents that have the field "user"+let existsFilter = ExistsFilter (FieldName "user")++```++#### Geo BoundingBox Filter++``` {.haskell}++-- topLeft and bottomRight+let box = GeoBoundingBox (LatLon 40.73 (-74.1)) (LatLon 40.10 (-71.12))++let constraint = GeoBoundingBoxConstraint (FieldName "tweet.location") box False GeoFilterMemory++```++#### Geo Distance Filter++``` {.haskell}++let geoPoint = GeoPoint (FieldName "tweet.location") (LatLon 40.12 (-71.34))++-- coefficient and units+let distance = Distance 10.0 Miles++-- GeoFilterType or NoOptimizeBbox+let optimizeBbox = OptimizeGeoFilterType GeoFilterMemory++-- SloppyArc is the usual/default optimization in Elasticsearch today+-- but pre-1.0 versions will need to pick Arc or Plane.++let geoFilter = GeoDistanceFilter geoPoint distance SloppyArc optimizeBbox False++```++#### Geo Distance Range Filter++Think of a donut and you won't be far off.++``` {.haskell}++let geoPoint = GeoPoint (FieldName "tweet.location") (LatLon 40.12 (-71.34))++let distanceRange = DistanceRange (Distance 0.0 Miles) (Distance 10.0 Miles)++let geoFilter = GeoDistanceRangeFilter geoPoint distanceRange++```++#### Geo Polygon Filter++``` {.haskell}++-- I think I drew a square here.+let points = [LatLon 40.0 (-70.00),+              LatLon 40.0 (-72.00),+              LatLon 41.0 (-70.00),+              LatLon 41.0 (-72.00)]++let geoFilter = GeoPolygonFilter (FieldName "tweet.location") points++```++#### Document IDs filter++``` {.haskell}++-- takes a mapping name and a list of DocIds+IdsFilter (MappingName "tweet") [DocId "1"]++```++#### Range Filter++``` {.haskell}++-- RangeFilter :: FieldName+--                -> RangeValue+--                -> RangeExecution+--                -> Cache -> Filter++let filter = RangeFilter (FieldName "age")+             (RangeGtLt (GreaterThan 1000.0) (LessThan 100000.0))+             RangeExecutionIndex False++```++``` {.haskell}++let filter = RangeFilter (FieldName "age")+             (RangeLte (LessThanEq 100000.0))+             RangeExecutionIndex False++```++##### Date Ranges++Date ranges are expressed in UTCTime. Date ranges use the same range bound constructors as numerics, except that they end in "D".++Note that compatibility with ES is tested only down to seconds.++``` {.haskell}++let filter = RangeFilter (FieldName "postDate")+             (RangeDateGtLte+              (GreaterThanD (UTCTime+                          (ModifiedJulianDay 55000)+                          (secondsToDiffTime 9)))+              (LessThanEqD (UTCTime+                            (ModifiedJulianDay 55000)+                            (secondsToDiffTime 11))))+             RangeExecutionIndex False+```++#### Regexp Filter++``` {.haskell}++-- RegexpFilter+--   :: FieldName+--      -> Regexp+--      -> RegexpFlags+--      -> CacheName+--      -> Cache+--      -> CacheKey+--      -> Filter+let filter = RegexpFilter (FieldName "user") (Regexp "bite.*app")+             AllRegexpFlags (CacheName "test") False (CacheKey "key")++-- n.b.+-- data RegexpFlags = AllRegexpFlags+--                 | NoRegexpFlags+--                 | SomeRegexpFlags (NonEmpty RegexpFlag) deriving (Eq, Show)++-- data RegexpFlag = AnyString+--                | Automaton+--                | Complement+--                | Empty+--                | Intersection+--                | Interval deriving (Eq, Show)++```++### Aggregations++#### Adding aggregations to search++Aggregations can now be added to search queries, or made on their own.++``` {.haskell}+type Aggregations = M.Map Text Aggregation+data Aggregation+  = TermsAgg TermsAggregation+  | DateHistogramAgg DateHistogramAggregation+```++For convenience, \`\`\`mkAggregations\`\`\` exists, that will create an \`\`\`Aggregations\`\`\` with the aggregation provided.++For example:++``` {.haskell}+ let a = mkAggregations "users" $ TermsAgg $ mkTermsAggregation "user"+ let search = mkAggregateSearch Nothing a+```++Aggregations can be added to an existing search, using the \`\`\`aggBody\`\`\` field++``` {.haskell}+ let search  = mkSearch (Just (MatchAllQuery Nothing)) Nothing+ let search' = search {aggBody = Just a}+```++Since the \`\`\`Aggregations\`\`\` structure is just a Map Text Aggregation, M.insert can be used to add additional aggregations.++``` {.haskell}+ let a' = M.insert "age" (TermsAgg $ mkTermsAggregation "age") a+```++#### Extracting aggregations from results++Aggregations are part of the reply structure of every search, in the form of++``` {.haskell}+-- Lift decode and response body to be in the IO monad.+let decode' = liftM decode+let responseBody' = liftM responseBody+let reply = withBH' $ searchByIndex testIndex search+let response = decode' $ responseBody' reply :: IO (Maybe (SearchResult Tweet))++-- Now that we have our response, we can extract our terms aggregation result -- which is a list of buckets.++let terms = do { response' <- response; return $ response' >>= aggregations >>= toTerms "users" }+terms+Just (Bucket {buckets = [TermsResult {termKey = "bitemyapp", termsDocCount = 1, termsAggs = Nothing}]})+```++Note that bucket aggregation results, such as the TermsResult is a member of the type class :++``` {.haskell}+class BucketAggregation a where+  key :: a -> Text+  docCount :: a -> Int+  aggs :: a -> Maybe AggregationResults+```++haskell++You can use the function to get any nested results, if there were any. For example, if there were a nested terms aggregation keyed to "age" in a TermsResult named , you would call++#### Terms Aggregation++``` {.haskell}+data TermsAggregation+  = TermsAggregation {term :: Either Text Text,+                      termInclude :: Maybe TermInclusion,+                      termExclude :: Maybe TermInclusion,+                      termOrder :: Maybe TermOrder,+                      termMinDocCount :: Maybe Int,+                      termSize :: Maybe Int,+                      termShardSize :: Maybe Int,+                      termCollectMode :: Maybe CollectionMode,+                      termExecutionHint :: Maybe ExecutionHint,+                      termAggs :: Maybe Aggregations}+```++Term Aggregations have two factory functions, , and , and can be used as follows:++``` {.haskell}+let ta = TermsAgg $ mkTermsAggregation "user"+```++There are of course other options that can be added to a Terms Aggregation, such as the collection mode:++``` {.haskell}+let ta   = mkTermsAggregation "user"+let ta'  = ta { termCollectMode = Just BreadthFirst }+let ta'' = TermsAgg ta'+```++For more documentation on how the Terms Aggregation works, see <http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html>++#### Date Histogram Aggregation++``` {.haskell}+data DateHistogramAggregation+  = DateHistogramAggregation {dateField :: FieldName,+                              dateInterval :: Interval,+                              dateFormat :: Maybe Text,+                              datePreZone :: Maybe Text,+                              datePostZone :: Maybe Text,+                              datePreOffset :: Maybe Text,+                              datePostOffset :: Maybe Text,+                              dateAggs :: Maybe Aggregations}+```++haskell++The Date Histogram Aggregation works much the same as the Terms Aggregation.++Relevant functions include , and++``` {.haskell}+let dh = DateHistogramAgg (mkDateHistogram (FieldName "postDate") Minute)+```++Date histograms also accept a :++``` {.haskell}+FractionalInterval :: Float -> TimeInterval -> Interval+-- TimeInterval is the following:+data TimeInterval = Weeks | Days | Hours | Minutes | Seconds+```++It can be used as follows:++``` {.haskell}+let dh = DateHistogramAgg (mkDateHistogram (FieldName "postDate") (FractionalInterval 1.5 Minutes))+```++The is defined as:++``` {.haskell}+data DateHistogramResult+  = DateHistogramResult {dateKey :: Int,+                         dateKeyStr :: Maybe Text,+                         dateDocCount :: Int,+                         dateHistogramAggs :: Maybe AggregationResults}+```++It is an instance of , and can have nested aggregations in each bucket.++Buckets can be extracted from a using++For more information on the Date Histogram Aggregation, see: <http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-datehistogram-aggregation.html>+++Contributors+============++* [Chris Allen](https://github.com/bitemyapp)+* [Liam Atkinson](https://github.com/latkins)+* [Christopher Guiney](https://github.com/chrisguiney)+* [Curtis Carter](https://github.com/ccarter)+* [Michael Xavier](https://github.com/MichaelXavier)+* [Bob Long](https://github.com/bobjflong)+* [Maximilian Tagher](https://github.com/MaxGabriel)+* [Anna Kopp](https://github.com/annakopp)+* [Matvey B. Aksenov](https://github.com/supki)+* [Jan-Philip Loos](https://github.com/MaxDaten)++Possible future functionality+=============================++Span Queries+------------++Beginning here: <http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-span-first-query.html>++Function Score Query+--------------------++<http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html>++Node discovery and failover+---------------------------++Might require TCP support.++Support for TCP access to Elasticsearch+---------------------------------------++Pretend to be a transport client?++Bulk cluster-join merge+-----------------------++Might require making a lucene index on disk with the appropriate format.++GeoShapeQuery+-------------++<http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-query.html>++GeoShapeFilter+--------------++<http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-filter.html>++Geohash cell filter+-------------------++<http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geohash-cell-filter.html>++HasChild Filter+---------------++<http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-has-child-filter.html>++HasParent Filter+----------------++<http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-has-parent-filter.html>++Indices Filter+--------------++<http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-indices-filter.html>++Query Filter+------------++<http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-query-filter.html>++Script based sorting+--------------------++<http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-sort.html#_script_based_sorting>++Collapsing redundantly nested and/or structures+-----------------------------------------------++The Seminearring instance, if deeply nested can possibly produce nested structure that is redundant. Depending on how this affects ES performance, reducing this structure might be valuable.++Runtime checking for cycles in data structures+----------------------------------------------++check for n \> 1 occurrences in DFS:++<http://hackage.haskell.org/package/stable-maps-0.0.5/docs/System-Mem-StableName-Dynamic.html>++<http://hackage.haskell.org/package/stable-maps-0.0.5/docs/System-Mem-StableName-Dynamic-Map.html>++Photo Origin+============++Photo from HA! Designs: <https://www.flickr.com/photos/hadesigns/>

@@ -1,5 +1,5 @@ name:                bloodhound-version:             0.8.0.0+version:             0.9.0.0 synopsis:            ElasticSearch client library for Haskell description:         ElasticSearch made awesome for Haskell hackers homepage:            https://github.com/bitemyapp/bloodhound@@ -12,6 +12,11 @@ build-type:          Custom cabal-version:       >=1.10 +extra-source-files:+  README.md+  changelog.md++ source-repository head   type:     git   location: https://github.com/bitemyapp/bloodhound.git@@ -26,16 +31,16 @@   build-depends:       base             >= 4.3     && <5,                        bytestring       >= 0.10.0  && <0.11,                        containers       >= 0.5.0.0 && <0.6,-                       aeson            >= 0.7     && <0.10,+                       aeson            >= 0.10    && <0.11,                        http-client      >= 0.3     && <0.5,+                       network-uri      >= 2.6     && <2.7,                        semigroups       >= 0.15    && <0.18,                        time             >= 1.4     && <1.6,                        text             >= 0.11    && <1.3,                        mtl              >= 1.0     && <2.3,                        transformers     >= 0.2     && <0.5,-                       http-types       >= 0.8     && <0.9,+                       http-types       >= 0.8     && <0.10,                        vector           >= 0.10.9  && <0.12,-                       uri-bytestring   >= 0.1     && <0.2,                        exceptions,                        data-default-class,                        blaze-builder,

@@ -0,0 +1,109 @@+0.8.0.0+===================++Thanks to the following people, Bloodhound 0.8.0.0 is being released!++* @MichaelXavier+  - #67: Deriving Monad(Throw|Catch|Mask)+  - #64: Export BH constructor+  - #61: Filter aggregation support+  - #60: Add value_count aggregation support+  - #58: Eliminate partiality in EsResult+  +* @centromere+  - #59: Fixed bug with IndexSettings serialization+  - #56: Added fields support to Search+  - #55: Added ability to specify a parent document+  - #54: Fixed IndexTemplate serialization bug+  - #52: Added ability to manipulate templates+  - #51: Fixed mapping API+  - #50: Fixed problem with put sending POST+  +* @bermanjosh+  - #63: Url query encoding bug+  - #53: Scan type++* @sjakobi+  - #69: Replace Control.Monad.Error with CM.Except via mtl-compat+  - #70: Silence redundant import warning with base-4.8.*+  - #71: Use "newManager" instead of deprecated "withManager"++0.7.0.0+===================++* Added QueryFilter thanks to Bjørn Nordbø!++* Support for optimistic concurrency control thanks again to @MichaelXavier!++0.6.0.1+===================++* Allow Aeson 0.9++0.6.0.0+===================++* Moved to BHMonad, thanks to @MichaelXavier! Now there's a reader of config information and IO is lifted.++* SearchHits have a Monoid now, makes combining search results nicer, allows for defaulting when a search cannot be performed.++0.5.0.0+===================++* Fixed and changed TermsQuery (This caused the major bump)++* Removed benchmarks from travis.yml++* Added doctests, examples for Database.Bloodhound.Client. Haddocks should be much nicer.++* Various fixes, reformatting++0.4.0.0+===================++* Term and date aggregation - thanks to Christopher Guiney! (@chrisguiney)++Following three thanks to Liam Atkins (@latkins)++* omitNulls changed to exclude empty lists and null values++* BoolQuery must/mustNot/Should changed from Maybe (Query|[Query]) to [Query] thanks to @latkins++* Added vector dependency so we can check for V.null/V.empty on JSON arrays++* Highlighting, thanks to @latkins! See http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-highlighting.html and http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/highlighting-intro.html for more++* Added 1.4.0 support and CI integration++* Can generate individual bulk operations, https://github.com/bitemyapp/bloodhound/issues/17, bulk requests should be more efficient now too - Vector instead of List.++0.3.0.0+===================++* Status "ok" changed from Bool to Maybe Bool thanks to @borisyukd++* Elasticsearch 1.3.x compatibility fixed with changes to geo bounding boxes - thanks to Curtis Carter! (@ccarter)++* CI coverage expanded to 1.0.x -> 1.3.x++0.2.0.1+===================++* Killed off maybeJson/mField/catMaybes in favor of omitNulls++* Experimenting with RecordWildcards++* Merged Types and Instances module into Types to prevent possibility of orphans and cull orphan instance warnings.++* Added note about current supported Elasticsearch version.++0.2.0.0+===================++* Added TermFilter++* Renamed createMapping to putMapping++* Fixed and rebuilt documentation++* RegexpFlags changed to a sum type instead of Text, thanks to @MichaelXavier!

@@ -66,9 +66,9 @@ import           Data.Aeson import           Data.ByteString.Lazy.Builder import qualified Data.ByteString.Lazy.Char8   as L-import           Data.Default.Class import           Data.Ix-import           Data.Maybe                   (fromMaybe)+import qualified Data.List                    as LS (filter)+import           Data.Maybe                   (fromMaybe, isJust) import           Data.Monoid import           Data.Text                    (Text) import qualified Data.Text                    as T@@ -78,8 +78,8 @@ import qualified Network.HTTP.Types.Method    as NHTM import qualified Network.HTTP.Types.Status    as NHTS import qualified Network.HTTP.Types.URI       as NHTU+import qualified Network.URI                  as URI import           Prelude                      hiding (filter, head)-import           URI.ByteString               hiding (Query)  import           Database.Bloodhound.Types @@ -499,14 +499,16 @@ -- | 'documentExists' enables you to check if a document exists. Returns 'Bool' --   in IO ----- >>> exists <- runBH' $ documentExists testIndex testMapping (DocId "1")+-- >>> exists <- runBH' $ documentExists testIndex testMapping Nothing (DocId "1") documentExists :: MonadBH m => IndexName -> MappingName-                  -> DocId -> m Bool-documentExists (IndexName indexName)-  (MappingName mappingName) (DocId docId) = do+               -> Maybe DocumentParent -> DocId -> m Bool+documentExists (IndexName indexName) (MappingName mappingName)+               parent (DocId docId) = do   (_, exists) <- existentialQuery =<< url   return exists-  where url = joinPath [indexName, mappingName, docId]+  where url = addQuery params <$> joinPath [indexName, mappingName, docId]+        parentParam = fmap (\(DocumentParent (DocId p)) -> p) parent+        params = LS.filter (\(_, v) -> isJust v) [("parent", parentParam)]  dispatchSearch :: MonadBH m => Text -> Search -> m Reply dispatchSearch url search = post url' (Just (encode search))@@ -628,32 +630,7 @@ pageSearch resultOffset pageSize search = search { from = resultOffset, size = pageSize }  parseUrl' :: MonadThrow m => Text -> m Request-parseUrl' t =-  case parseURI laxURIParserOptions (T.encodeUtf8 t) of-    Right uri -> setURI def uri-    Left e -> throwM $ InvalidUrlException (T.unpack t) ("Invalid URL: " ++ show e)--setURI :: MonadThrow m => Request -> URI -> m Request-setURI req URI{..} = do-  Authority {..} <- maybe missingUA return uriAuthority-  let req' = req { secure = isSecure-                 , host   = hostBS authorityHost-                 , port   = thePort-                 , path   = uriPath-                 }-      thePort = maybe defPort portNumber authorityPort-      addAuth = maybe id addAuth' authorityUserInfo-  return $ setQueryString theQueryString $ addAuth req'-  where-    missingUA = throwM $ InvalidUrlException "N/A" "Missing URI host/port"-    addAuth' UserInfo {..} = applyBasicProxyAuth uiUsername uiPassword-    defPort-      | isSecure  = 443-      | otherwise = 80-    isSecure = case uriScheme of-      Scheme "https" -> True-      _              -> False-    theQueryString = [(k , Just v) | (k, v) <- queryPairs uriQuery]+parseUrl' t = parseUrl (URI.escapeURIString URI.isAllowedInURI (T.unpack t))  -- | Was there an optimistic concurrency control conflict when -- indexing a document?

@@ -61,6 +61,7 @@        , Reply        , EsResult(..)        , EsResultFound(..)+       , EsError(..)        , DocVersion        , ExternalDocVersion(..)        , VersionControl(..)@@ -416,13 +417,18 @@                            , _id         :: Text                            , foundResult :: Maybe (EsResultFound a)} deriving (Eq, Show) - {-| 'EsResultFound' contains the document and its metadata inside of an     'EsResult' when the document was successfully found. -} data EsResultFound a = EsResultFound {  _version :: DocVersion                                      , _source   :: a } deriving (Eq, Show) +{-| 'EsError' is the generic type that will be returned when there was a+    problem. If you can't parse the expected response, its a good idea to+    try parsing this.+-}+data EsError = EsError { errorStatus  :: Int+                       , errorMessage :: Text } deriving (Eq, Show)   {-| 'DocVersion' is an integer version number for a document between 1@@ -756,7 +762,7 @@   deriving (Eq, Show)  data Source =-  NoSource+    NoSource   | SourcePatterns PatternOrPatterns   | SourceIncludeExclude Include Exclude     deriving (Show, Eq)@@ -1305,7 +1311,7 @@       , hitType      :: MappingName       , hitDocId     :: DocId       , hitScore     :: Score-      , hitSource    :: a+      , hitSource    :: Maybe a       , hitHighlight :: Maybe HitHighlight } deriving (Eq, Show)  data ShardResult =@@ -2133,6 +2139,12 @@                          v .: "_source"   parseJSON _          = empty +instance FromJSON EsError where+  parseJSON (Object v) = EsError <$>+                         v .: "status" <*>+                         v .: "error"+  parseJSON _ = empty+ instance ToJSON Search where   toJSON (Search query sFilter sort searchAggs highlight sTrackSortScores sFrom sSize _ sFields sSource) =     omitNulls [ "query"        .= query@@ -2396,11 +2408,11 @@  instance (FromJSON a) => FromJSON (Hit a) where   parseJSON (Object v) = Hit <$>-                         v .:  "_index"  <*>-                         v .:  "_type"   <*>-                         v .:  "_id"     <*>-                         v .:  "_score"  <*>-                         v .:  "_source" <*>+                         v .:  "_index"   <*>+                         v .:  "_type"    <*>+                         v .:  "_id"      <*>+                         v .:  "_score"   <*>+                         v .:?  "_source" <*>                          v .:? "highlight"   parseJSON _          = empty 

@@ -14,6 +14,7 @@ import           Data.List.NonEmpty              (NonEmpty (..)) import qualified Data.List.NonEmpty              as NE import qualified Data.Map.Strict                 as M+import qualified Data.Maybe                      as MY import           Data.Monoid import           Data.Text                       (Text) import qualified Data.Text                       as T@@ -27,7 +28,7 @@ import qualified Network.HTTP.Types.Status       as NHTS import           Prelude                         hiding (filter) import           Test.Hspec-import           Test.QuickCheck.Property.Monoid (prop_Monoid, eq, T(..))+import           Test.QuickCheck.Property.Monoid (T (..), eq, prop_Monoid)  import           Test.Hspec.QuickCheck           (prop) import           Test.QuickCheck@@ -116,6 +117,20 @@ instance ToJSON   Location instance FromJSON Location +data ParentMapping = ParentMapping deriving (Eq, Show)++instance ToJSON ParentMapping where+  toJSON ParentMapping =+    object ["parent" .= Null ]++data ChildMapping = ChildMapping deriving (Eq, Show)++instance ToJSON ChildMapping where+  toJSON ChildMapping =+    object ["child" .=+      object ["_parent" .= object ["type" .= ("parent" :: Text)]]+    ]+ data TweetMapping = TweetMapping deriving (Eq, Show)  instance ToJSON TweetMapping where@@ -173,10 +188,17 @@   _ <- refreshIndex testIndex   return () +insertWithSpaceInId :: BH IO ()+insertWithSpaceInId = do+  _ <- indexDocument testIndex testMapping defaultIndexDocumentSettings exampleTweet (DocId "Hello World")+  _ <- refreshIndex testIndex+  return ()+ searchTweet :: Search -> BH IO (Either String Tweet) searchTweet search = do   result <- searchTweets search-  let myTweet = fmap (hitSource . head . hits . searchHits) result+  let myTweet :: Either String Tweet+      myTweet = grabFirst result   return myTweet  searchTweets :: Search -> BH IO (Either String (SearchResult Tweet))@@ -227,7 +249,7 @@   let search = (mkSearch (Just query) Nothing) { source = Just src }   reply <- searchAll search   let result = eitherDecode (responseBody reply) :: Either String (SearchResult Value)-  let value = fmap (hitSource . head . hits . searchHits) result+  let value = grabFirst result   liftIO $     value `shouldBe` expected @@ -293,6 +315,13 @@ getSource :: EsResult a -> Maybe a getSource = fmap _source . foundResult +-- grabFirst :: Either String (SearchResult a) -> Either String a+grabFirst r =+  case fmap (hitSource . head . hits . searchHits) r of+    (Left e) -> Left e+    (Right Nothing) -> Left "Source was missing"+    (Right (Just x)) -> Right x+ main :: IO () main = hspec $ do @@ -315,6 +344,13 @@                      (responseBody docInserted) :: Either String (EsResult Tweet)       liftIO $ (fmap getSource newTweet `shouldBe` Right (Just exampleTweet)) +    it "indexes, gets, and then deletes the generated document with a DocId containing a space" $ withTestEnv $ do+      _ <- insertWithSpaceInId+      docInserted <- getDocument testIndex testMapping (DocId "Hello World")+      let newTweet = eitherDecode+                     (responseBody docInserted) :: Either String (EsResult Tweet)+      liftIO $ (fmap getSource newTweet `shouldBe` Right (Just exampleTweet))+     it "produces a parseable result when looking up a bogus document" $ withTestEnv $ do       doc <- getDocument testIndex testMapping  (DocId "bogus")       let noTweet = eitherDecode@@ -330,6 +366,18 @@       res' <- insertData' cfg       liftIO $ isVersionConflict res' `shouldBe` True +    it "indexes two documents in a parent/child relationship and checks that the child exists" $ withTestEnv $ do+      resetIndex+      _ <- putMapping testIndex (MappingName "parent") ParentMapping+      _ <- putMapping testIndex (MappingName "child") ChildMapping+      _ <- indexDocument testIndex (MappingName "parent") defaultIndexDocumentSettings exampleTweet (DocId "1")+      let parent = (Just . DocumentParent . DocId) "1"+          ids = IndexDocumentSettings NoVersionControl parent+      _ <- indexDocument testIndex (MappingName "child") ids otherTweet (DocId "2")+      _ <- refreshIndex testIndex+      exists <- documentExists testIndex (MappingName "child") parent (DocId "2")+      liftIO $ exists `shouldBe` True+   describe "template API" $ do     it "can create a template" $ withTestEnv $ do       let idxTpl = IndexTemplate (TemplatePattern "tweet-*") (Just (IndexSettings (ShardCount 1) (ReplicaCount 1))) [toJSON TweetMapping]@@ -448,7 +496,7 @@                    (Just IdentityFilter) (Just [sortSpec]) Nothing Nothing                    False (From 0) (Size 10) SearchTypeQueryThenFetch Nothing Nothing       result <- searchTweets search-      let myTweet = fmap (hitSource . head . hits . searchHits) result+      let myTweet = grabFirst result       liftIO $         myTweet `shouldBe` Right otherTweet @@ -665,26 +713,26 @@         fmap aggregations res `shouldBe` Right (Just (M.fromList [ docCountPair "bitemyapps" 1                                                                  , docCountPair "notmyapps" 1                                                                  ]))-    -- Interaction of date serialization and date histogram aggregation is broken.-    -- it "returns date histogram aggregation results" $ withTestEnv $ do-    --   _ <- insertData-    --   let histogram = DateHistogramAgg $ mkDateHistogram (FieldName "postDate") Minute-    --   let search = mkAggregateSearch Nothing (mkAggregations "byDate" histogram)-    --   searchExpectAggs search-    --   searchValidBucketAgg search "byDate" toDateHistogram -    -- it "returns date histogram using fractional date" $ withTestEnv $ do-    --   _ <- insertData-    --   let periods            = [Year, Quarter, Month, Week, Day, Hour, Minute, Second]-    --   let fractionals        = map (FractionalInterval 1.5) [Weeks, Days, Hours, Minutes, Seconds]-    --   let intervals          = periods ++ fractionals-    --   let histogram          = mkDateHistogram (FieldName "postDate")-    --   let search interval    = mkAggregateSearch Nothing $ mkAggregations "byDate" $ DateHistogramAgg (histogram interval)-    --   let expect interval    = searchExpectAggs (search interval)-    --   let valid interval     = searchValidBucketAgg (search interval) "byDate" toDateHistogram-    --   forM_ intervals expect-    --   forM_ intervals valid+    it "returns date histogram aggregation results" $ withTestEnv $ do+      _ <- insertData+      let histogram = DateHistogramAgg $ mkDateHistogram (FieldName "postDate") Minute+      let search = mkAggregateSearch Nothing (mkAggregations "byDate" histogram)+      searchExpectAggs search+      searchValidBucketAgg search "byDate" toDateHistogram +    it "returns date histogram using fractional date" $ withTestEnv $ do+      _ <- insertData+      let periods            = [Year, Quarter, Month, Week, Day, Hour, Minute, Second]+      let fractionals        = map (FractionalInterval 1.5) [Weeks, Days, Hours, Minutes, Seconds]+      let intervals          = periods ++ fractionals+      let histogram          = mkDateHistogram (FieldName "postDate")+      let search interval    = mkAggregateSearch Nothing $ mkAggregations "byDate" $ DateHistogramAgg (histogram interval)+      let expect interval    = searchExpectAggs (search interval)+      let valid interval     = searchValidBucketAgg (search interval) "byDate" toDateHistogram+      forM_ intervals expect+      forM_ intervals valid+   describe "Highlights API" $ do      it "returns highlight from query when there should be one" $ withTestEnv $ do@@ -714,7 +762,7 @@     it "doesn't include source when sources are disabled" $ withTestEnv $ do       searchExpectSource         NoSource-        (Left "key \"_source\" not present")+        (Left "Source was missing")      it "includes a source" $ withTestEnv $ do       searchExpectSource@@ -808,6 +856,6 @@       scan_search' <- scanSearch testIndex testMapping search :: BH IO [Hit Tweet]       let scan_search = map hitSource scan_search'       liftIO $-        regular_search `shouldBe` Right exampleTweet -- Check that the size restrtiction is being honored +        regular_search `shouldBe` Right exampleTweet -- Check that the size restrtiction is being honored       liftIO $-        scan_search `shouldMatchList` [exampleTweet, otherTweet]+        scan_search `shouldMatchList` [Just exampleTweet, Just otherTweet]