cabal-version: 2.4
name: scrappy-requests
-- PVP summary: +-+------- breaking API changes
-- | | +----- non-breaking API additions
-- | | | +--- code changes with no API change
version: 0.1.0.0
synopsis: html pattern matching library and high-level interface concurrent requests lib for webscraping
description:
Scrappy is meant to be a fully expressive library for all aspects of webscraping.
In this sense it is meant to be as undetectable as using Selenium but with a design
specifically for webscraping (where Selenium was never intended for webscraping nor do
it's maintainers seek to provide features that allow for more expressive scraping if it
would not help testing). The Elem.* modules provide a wide range of expressive pattern matching
functions while adding no more complexity in needing to learn this library over parsing.
.
In addition to expressive patterns to fit your specific patterns you hope to scrape, scrappy
provides helper functions for complex control flows such as running multiple different
parser-scrapers on respective sites based on a rotating ConcurrentStream when users have many
target sites that they can rotate working on to not overload a given site and thus avoid detection
.
For simpler control flows such as scraping a large number of pages on a single site, scrappy currently
provides functions like getHtml, which not only is a super simple interface to http requests,
but a persistent function that gurantees retrieval of the HTML document to be scraped
.
This package is labelled as uncurated, and so suggestions are very much welcome and this package is
expected to grow based on feedback. The primary focus will be on running Javascript to allow for greater
access to information on sites
-- URL for the project homepage or repository.
homepage: https://github.com/Ace-Interview-Prep/scrappy
license: BSD-3-Clause
license-file: LICENSE
author: Galen Sprout
maintainer: galen.sprout@gmail.com
bug-reports: https://github.com/Ace-Interview-Prep/scrappy/issues
x-curated: uncurated-seeking-adoption
stability: Experimental
category: Webscraping
build-type: Simple
extra-source-files: README.MD
source-repository head
type: git
location: https://github.com/Ace-Interview-Prep/scrappy-requests
library
exposed-modules: Scrappy.Requests.BuildActions
, Scrappy.Requests.Requests
, Scrappy.Requests.Types
-- Needs a lot of work, is more like an idea rn
-- , Concurrencies
, Scrappy.Requests.Proxies
--, Scrappy.JS
build-depends: aeson >= 2.2.3 && < 2.3
, base >= 4.20.2 && < 4.21
, bytestring >= 0.12.2 && < 0.13
, containers >= 0.7 && < 0.8
, directory >= 1.3.8 && < 1.4
, exceptions >= 0.10.9 && < 0.11
, http-client >= 0.7.19 && < 0.8
, http-client-tls >= 0.3.6 && < 0.4
, http-types >= 0.12.4 && < 0.13
, lens >= 5.3.6 && < 5.4
, modern-uri >= 0.3.6 && < 0.4
, mtl >= 2.3.1 && < 2.4
, network-uri >= 2.6.4 && < 2.7
, parsec >= 3.1.18 && < 3.2
, parser-combinators >= 1.3.1 && < 1.4
, scrappy-core >= 0.1.0 && < 0.2
, text >= 2.1.3 && < 2.2
, time >= 1.12.2 && < 1.13
, transformers >= 0.6.1 && < 0.7
hs-source-dirs: src
default-language: Haskell2010