cabal-version: >= 1.6
name: http-conduit-downloader
version: 1.0.16
author: Vladimir Shabanov <vshabanoff@gmail.com>
maintainer: Vladimir Shabanov <vshabanoff@gmail.com>
homepage: https://github.com/bazqux/http-conduit-downloader
license: BSD3
category: Web
license-file: LICENSE
build-type: Simple
synopsis: HTTP downloader tailored for web-crawler needs.
description:
HTTP/HTTPS downloader built on top of @http-conduit@
and used in <https://bazqux.com> crawler.
.
* Handles all possible http-conduit exceptions and returns
human readable error messages.
.
* Handles some web server bugs (returning 'deflate' data instead of 'gzip',
invalid 'gzip' encoding).
.
* Uses OpenSSL instead of tls package since it doesn't handle all sites.
.
* Ignores invalid SSL sertificates.
.
* Receives data in 32k blocks internally to reduce memory fragmentation
on many parallel downloads.
.
* Download timeout.
.
* Total download size limit.
.
* Returns HTTP headers for subsequent redownloads and handles
'Not modified' results.
.
* Can be used with external DNS resolver (hsdns-cache for example).
source-repository head
type: git
location: https://github.com/bazqux/http-conduit-downloader
library
build-depends:
base == 4.*, http-conduit >= 2.0.0.4, http-client, connection, network, zlib, lifted-base,
conduit, resourcet, http-types, data-default, bytestring, mtl,
time, old-locale, HsOpenSSL
exposed-modules:
Network.HTTP.Conduit.Downloader
ghc-options: -O2 -Wall