cabal-version: 2.4
name: http-conduit-downloader
version: 1.1.5
author: Vladimir Shabanov <dev@vshabanov.com>
maintainer: Vladimir Shabanov <dev@vshabanov.com>
homepage: https://github.com/bazqux/http-conduit-downloader
license: BSD-3-Clause
category: Web
license-file: LICENSE
build-type: Simple
synopsis: HTTP downloader tailored for web-crawler needs.
description:
HTTP/HTTPS downloader built on top of @http-client@
and used in <https://bazqux.com> crawler.
.
Previously it was based on @http-conduit@ (hence the name) but since
all the necessary parts are in @http-client@ now @http-conduit@ is no
longer used.
.
* Handles all possible http-client exceptions and returns
human readable error messages.
.
* Handles some web server bugs (returning 'deflate' data instead of 'gzip',
invalid 'gzip' encoding).
.
* Uses OpenSSL instead of 'tls' package (since 'tls' doesn't handle all sites and works slower than OpenSSL).
.
* Ignores invalid SSL sertificates.
.
* Receives data in 32k chunks internally to reduce memory fragmentation
on many parallel downloads.
.
* Download timeout.
.
* Total download size limit.
.
* Returns HTTP headers for subsequent redownloads and handles
'Not modified' results.
.
* Can be used with external DNS resolver (e.g. @concurrent-dns-cache@).
source-repository head
type: git
location: https://github.com/bazqux/http-conduit-downloader
library
default-language: Haskell2010
build-depends:
base == 4.*,
http-client == 0.7.16.*,
http-client-openssl == 0.3.3.*,
zlib,
http-types, data-default, bytestring, text,
time >= 1.5.0, HsOpenSSL >= 0.11.5, network-uri >= 2.6, network >= 2.6
exposed-modules:
Network.HTTP.Conduit.Downloader
ghc-options: -O2 -Wall