http-conduit-downloader 1.0.24 → 1.0.25
raw patch · 2 files changed
+13/−10 lines, 2 filesPVP: major bump suggested
API removals or changes: PVP suggests a major version bump
API changes (from Hackage documentation)
+ Network.HTTP.Conduit.Downloader: rdrCookieJar :: RawDownloadResult -> CookieJar
- Network.HTTP.Conduit.Downloader: RawDownloadResult :: Status -> HttpVersion -> ResponseHeaders -> ByteString -> RawDownloadResult
+ Network.HTTP.Conduit.Downloader: RawDownloadResult :: Status -> HttpVersion -> ResponseHeaders -> ByteString -> CookieJar -> RawDownloadResult
Files
Network/HTTP/Conduit/Downloader.hs view
@@ -8,13 +8,13 @@ -- human readable error messages. -- -- * Handles some web server bugs (returning @deflate@ data instead of @gzip@,--- invalid @gzip encoding).+-- invalid @gzip@ encoding). ----- * Uses OpenSSL instead of tls package since it doesn't handle all sites.+-- * Uses OpenSSL instead of @tls@ package (since @tls@ doesn't handle all sites). -- -- * Ignores invalid SSL sertificates. ----- * Receives data in 32k blocks internally to reduce memory fragmentation+-- * Receives data in 32k chunks internally to reduce memory fragmentation -- on many parallel downloads. -- -- * Download timeout.@@ -37,10 +37,10 @@ -- case ra of -- Left err -> ... -- uh oh, bad host -- Right ha -> do--- ... -- crawler politeness stuff (rate limits, domain queues)+-- ... -- crawler politeness stuff (rate limits, queues) -- dr <- download d url (Just ha) opts -- case dr of--- DROK dat redownloadOpts ->+-- DROK dat redownloadOptions -> -- ... -- analyze data, save redownloadOpts for next download -- DRRedirect .. -> ... -- DRNotModified -> ...@@ -48,7 +48,8 @@ -- @ -- -- It's highly recommended to use--- <http://hackage.haskell.org/package/hsdns-cache>+-- <http://hackage.haskell.org/package/concurrent-dns-cache>+-- (preferably with single resolver pointing to locally running BIND) -- for DNS resolution since @getAddrInfo@ used in @http-conduit@ can be -- buggy and ineffective when it needs to resolve many hosts per second for -- a long time.@@ -118,6 +119,7 @@ , rdrHttpVersion :: N.HttpVersion , rdrHeaders :: N.ResponseHeaders , rdrBody :: B.ByteString+ , rdrCookieJar :: C.CookieJar } deriving (Show, Eq) @@ -374,6 +376,7 @@ , rdrHttpVersion = C.responseVersion r , rdrHeaders = h , rdrBody = d+ , rdrCookieJar = C.responseCookieJar r }) Nothing -> return (DRError "Too much data", Nothing)) `E.catch`
http-conduit-downloader.cabal view
@@ -1,6 +1,6 @@ cabal-version: >= 1.6 name: http-conduit-downloader-version: 1.0.24+version: 1.0.25 author: Vladimir Shabanov <vshabanoff@gmail.com> maintainer: Vladimir Shabanov <vshabanoff@gmail.com> homepage: https://github.com/bazqux/http-conduit-downloader@@ -19,11 +19,11 @@ * Handles some web server bugs (returning 'deflate' data instead of 'gzip', invalid 'gzip' encoding). .- * Uses OpenSSL instead of tls package since it doesn't handle all sites.+ * Uses OpenSSL instead of 'tls' package (since 'tls' doesn't handle all sites). . * Ignores invalid SSL sertificates. .- * Receives data in 32k blocks internally to reduce memory fragmentation+ * Receives data in 32k chunks internally to reduce memory fragmentation on many parallel downloads. . * Download timeout.@@ -33,7 +33,7 @@ * Returns HTTP headers for subsequent redownloads and handles 'Not modified' results. .- * Can be used with external DNS resolver (hsdns-cache for example).+ * Can be used with external DNS resolver (e.g. concurrent-dns-cache). source-repository head type: git