diff --git a/LICENSE b/LICENSE
new file mode 100644
--- /dev/null
+++ b/LICENSE
@@ -0,0 +1,30 @@
+Copyright (c) 2021, Galen
+
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+ * Redistributions of source code must retain the above copyright
+ notice, this list of conditions and the following disclaimer.
+
+ * Redistributions in binary form must reproduce the above
+ copyright notice, this list of conditions and the following
+ disclaimer in the documentation and/or other materials provided
+ with the distribution.
+
+ * Neither the name of Galen nor the names of other
+ contributors may be used to endorse or promote products derived
+ from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
diff --git a/README.MD b/README.MD
new file mode 100644
--- /dev/null
+++ b/README.MD
@@ -0,0 +1,150 @@
+# Scrappy #
+
+# 
+
+
+provides a number of functions that allow for easily scraping certain patterns from websites as well control functions that allow you to rotate between multiple sites and rotate proxies in order to deal with common bot detection problems faced by scrapers.
+
+## Scraping ##
+
+Scraping is parsing, where we don't care about the placement of our match in a chunk of data. ScraperT is really just ParsecT (ParsecT String () Identity a)
+
+## How to depend on this project ##
+
+Scrappy is currently not on Hackage because I honestly don't have time to perfect version bounds however through [Nix](https://nixos.org/download) you can get the package the following way:
+
+OR!! Just clone scrappy-tutorial and rename it
+
+OR if you really want to learn (hey good for you!)
+
+```bash
+# assuming you've done nix-setup... I'd show you how but it depends what setup you prefer and its super simple
+nix-env -f https://github.com/obsidiansystems/nix-thunk/archive/master.tar.gz -iA command
+nix-shell -p cabal2nix
+cd yourProjectDirectory
+cabal init
+# follow prompts
+mkdir deps
+git clone https://github.com/Ace-Interview-Prep/scrappy.git deps/scrappy # or SSH url
+nix-thunk pack deps/scrappy # this is technically optional but recommended
+# add scrappy to your cabal file under build-depends
+cabal2nix . --shell > shell.nix -- this will create an easy to use shell.nix file with all of your build depends
+nix-shell #you could also do nix-shell shell.nix
+cabal run my-project-name-in-cabal-file
+```
+
+
+## Tutorials ##
+
+- https://medium.com/p/135283dc2af
+- https://github.com/Ace-Interview-Prep/scrappy-tutorial
+
+## Main functions, Types, and Classes ##
+
+```haskell
+
+class ElementRep (a :: * -> *) where
+--type InnerHTMLRep a = something
+ elTag :: a b -> Elem
+ attrs :: a b -> Attrs
+ innerText' :: a b -> String
+ matches' :: a b -> [b]
+
+instance ElementRep (Elem') where
+ elTag = _el
+ attrs = _attrs
+ innerText' = innerHtmlFull
+ matches' = innerMatches
+
+data Elem' a = Elem' { _el :: Elem
+ , _attrs :: Map String String
+ , innerMatches :: [a]
+ , innerHtmlFull :: String
+ } deriving Show
+
+-- | Manager returned may be a new manager, if the given manager failed
+getHtml :: Manager -> Url -> IO (Manager, Html)
+getHtml' :: Url -> IO Html
+
+scrape :: ScraperT a -> Html -> Maybe [a]
+
+-- | For more advanced control over inner structure see Elem.TreeElemParser
+elemParser :: Elem -> [(String, Maybe String)]
+ -> Maybe (ParsecT s u m a) -- Optionally scraped pattern inside this el, if specified, return element must have at least 1
+ -> ScraperT (Elem' a)
+
+el :: Elem -> [(String, String)] -> ScraperT (Elem' String)
+
+-- | Find any expressive pattern as long as it is inside of some HTML context
+contains :: ScraperT (Elem' a) -> ScraperT a -> ScraperT a
+
+
+:t fmap snd getHtml' "https://google.com" >>= return . (runScraperOnHtml (el "a" [])
+>>> IO (Maybe [Elem' a])
+
+-- Will get all 's on the url that are inside
+-- divs with a class of xyz
+example :: IO (Maybe [Elem' a])
+example = runScraperOnUrl url $ el "div" [("class", "xyz")] `contains`
+ el "a" [("id", "link")]
+
+
+```
+
+Currently with this library you can request HTML and run a multitude of scraper-patterns on it, for example:
+- Scrape an element
+- Scrape an element with a specific inner pattern
+- Scrape an element with an attribute that fits a specific parser-pattern
+- Scrape an element with a specific inner DOM-tree
+- Scrape a group of elements that *kinda* repeats
+ - For example, if you want a complex group that varies from page to page but to the user's eye looks the exact same (think of search results on a nice website) or even largely the same, then use the `htmlGroup` function from Scrappy.Elem.TreeElemParser
+- Scrape any **arbitrary** parser from the [parsec library](https://hackage.haskell.org/package/parsec)
+
+You can also deeply chain!
+- Inside any instance of the ElementRep class (Elem', TreeHTML) through contains and contains'!
+ - For example use: think of getting all prices
+- In a sequence!
+ - skipToBody: to avoid matches in the
+ - (>=) and (>) which are sequencers that take two parsers like Monadic actions (hence the characters chosen) but unlike running two parsers one after the other, there may be whatever random stuff in between!
+
+
+## TODO ##
+
+As this is an Open Source ambitious project, there is much that is left to do which follows from the potential of this library as a result of it's inner workings
+
+- Upload to Hackage
+- Be included in the haskellPackages attribute of nixpkgs
+- grep/regex replacement (and streamEdit across multiple files, see below)
+ - I hate ReGeX with a burning passion. No way will I ever come back to an expression and say "Oh that must match on emails!".
+ - It would be neat to have a way to write haskell at the command line like "grepScrappy -sR "char 'a' >> some digit >> char 'b'" but this string would quickly get massive so I wonder if there's a happy medium that exists such as shortforms for the most common parsers (char, many, some, digit, string, etc) that while we would still allow for any haskell string (which typifies to a ParsecT) there would be ways to shorten expressions
+- Improve `contains` and similar functions to be able to somehow apply the 'contained' parser to inside the start and end tags
+- More Generic typing and Monads for a SiteT, HistoryT, and WebT (an accumulator of new undiscovered sites)
+ - I have a number of complicated Monads all with their own varied implementations that could be greatly reduced to a SiteT which keeps state of what branches of the Site tree have been visited or seen.
+ - With a SiteT it would easily allow for logic to gather all new links (see Scrappy.Links) upon getting a new HTML via getHTML or sibling functions, and perhaps a new sibling function like getNextHTML which gets the next untouched HTML page
+ - SiteT extension would mean that we could create functions like `scrapeSite` which might look for all pattern matches across the site-tree. By extension this could apply to HistoryT and WebT via an incredibly simple interface.
+ - The concept of a SiteT implies there may be use in a HistoryT (ie all Sites visited with their respective site-trees) and a WebT which is meant to represent the entire web in the sense that it continues to extend and is not constrained to past history.
+ - Hopefully this gives intriguing ideas to why scrappy-nlp might be so powerful
+- Concurrency with streamEdits
+ - This functionality will help both for making it easier+quicker to perform concurrency when scraping (which is actually quite easy for different pages due to forkIO and standard library functions) but also makes a scrappy-DOM (see below) much easier to imagine as a robust web framework (imagine how long 10+ prerender-like scrapers would take or rather 10 passthroughs). I see this as the biggest blocker to a 1st version of scrappy-DOM (which wouldn't yet have FRP or GHCJS functionality.. you could still write raw JavaScript in V1 though)
+ - Extending this idea, lazy-js (see below) is effectively a streamEdit performer and so this logic would also provide incredible speed to this 'category' of the DOM as well.
+- Break up into scrappy-core and scrappy-requests
+ - This is so that scrappy-core can be used in frontend apps that are built using GHCJS. GHCJS purposely doesn't build networking packages.
+- Fork a streamEdit package
+ - A lot of streamEdit packages are exceptionally complex while library functions like Data.Text leave much to be desired. scrappy-streamEdit would be a perfect medium
+- Fork a scrappy-staleLink (or better name) package
+ - For use with [Obelisk](https://github.com/obsidiansystems/obelisk) the static directory which will provide a list of static assets which are no longer in use and might be good to delete
+- Fork a scrappy-templates package
+ - This is in the works currently for templating files/strings with variables such as large prompts for GPT-*, for example you could template a prompt for resumes that is job dependent in an ergonomic/easy-to-use way
+ - See Scrappy.Templates
+ - To reduce runtime errors from performing IO, it would be nice to have a staticFile finder like with staticWhich: a package that ensures an executable's existence *at compile time*
+- scrappy + nlp
+ - It would be advantageous to use certain functions like a recursive scraper from pageN -> page(N+M), where M is based on a stop condition and/or the arrow a successful pattern match (see Scrappy.Run). While this can be easily done with HTML patterns, this extends and lends well to NLP analysis for the text that is contained in the structural HTML
+- scrappy-DOM
+ - I currently have a bone to pick with the `prerender` function from [reflex-dom](https://github.com/reflex-frp/reflex-dom) as it can easily fail. For example, if the browser insists on inserting a inside a tag then reflex-dom panics because the DOM is not as expected, even if the DOM area is on a completely different branch and way earlier than it. Intuitively, using streamEdit on an element with an attribute like prerenderID="some unique string" would be an incredibly dependable solution.
+ - The blockers to this are of course that it's only worth doing if it will be an improvement to reflex-dom as a whole (or maybe we integrate with reflex-dom) and reflex-dom is truly exceptional with FRP, GHCJS bindings, and mobile-readiness
+- scrappy-headless
+ - This is effectively a headless browser which has the ability to read and run JS in response to events. A lot of work has been done on this in the Scrappy.BuildActions module and in [lazy-js](https://github.com/augyg/lazy-js) which is a fork of this project
+ - Also need to implement httpHistory in a more robust way to make a headless browser more legit
+- lazy-js
+ - This is a foundational project for both scrappy-headless and scrappy-DOM which is a lazy reader and writer of JavaScript. An example use is virtualizing clicking a link to bypass any JavaScript bot detectors and/or just to follow JavaScript based events like a button which calls some JS to change window.location (like a fancy href)
+
diff --git a/Setup.hs b/Setup.hs
new file mode 100644
--- /dev/null
+++ b/Setup.hs
@@ -0,0 +1,2 @@
+import Distribution.Simple
+main = defaultMain
diff --git a/scrappy-requests.cabal b/scrappy-requests.cabal
new file mode 100644
--- /dev/null
+++ b/scrappy-requests.cabal
@@ -0,0 +1,75 @@
+cabal-version: 2.4
+name: scrappy-requests
+-- PVP summary: +-+------- breaking API changes
+-- | | +----- non-breaking API additions
+-- | | | +--- code changes with no API change
+version: 0.1.0.0
+synopsis: html pattern matching library and high-level interface concurrent requests lib for webscraping
+description:
+ Scrappy is meant to be a fully expressive library for all aspects of webscraping.
+ In this sense it is meant to be as undetectable as using Selenium but with a design
+ specifically for webscraping (where Selenium was never intended for webscraping nor do
+ it's maintainers seek to provide features that allow for more expressive scraping if it
+ would not help testing). The Elem.* modules provide a wide range of expressive pattern matching
+ functions while adding no more complexity in needing to learn this library over parsing.
+ .
+ In addition to expressive patterns to fit your specific patterns you hope to scrape, scrappy
+ provides helper functions for complex control flows such as running multiple different
+ parser-scrapers on respective sites based on a rotating ConcurrentStream when users have many
+ target sites that they can rotate working on to not overload a given site and thus avoid detection
+ .
+ For simpler control flows such as scraping a large number of pages on a single site, scrappy currently
+ provides functions like getHtml, which not only is a super simple interface to http requests,
+ but a persistent function that gurantees retrieval of the HTML document to be scraped
+ .
+ This package is labelled as uncurated, and so suggestions are very much welcome and this package is
+ expected to grow based on feedback. The primary focus will be on running Javascript to allow for greater
+ access to information on sites
+
+-- URL for the project homepage or repository.
+homepage: https://github.com/Ace-Interview-Prep/scrappy
+license: BSD-3-Clause
+license-file: LICENSE
+author: Galen Sprout
+maintainer: galen.sprout@gmail.com
+bug-reports: https://github.com/Ace-Interview-Prep/scrappy/issues
+x-curated: uncurated-seeking-adoption
+stability: Experimental
+category: Webscraping
+build-type: Simple
+extra-source-files: README.MD
+source-repository head
+ type: git
+ location: https://github.com/Ace-Interview-Prep/scrappy-requests
+
+library
+ exposed-modules: Scrappy.Requests.BuildActions
+ , Scrappy.Requests.Requests
+ , Scrappy.Requests.Types
+ -- Needs a lot of work, is more like an idea rn
+ -- , Concurrencies
+ , Scrappy.Requests.Proxies
+ --, Scrappy.JS
+
+ build-depends: aeson >= 2.2.3 && < 2.3
+ , base >= 4.20.2 && < 4.21
+ , bytestring >= 0.12.2 && < 0.13
+ , containers >= 0.7 && < 0.8
+ , directory >= 1.3.8 && < 1.4
+ , exceptions >= 0.10.9 && < 0.11
+ , http-client >= 0.7.19 && < 0.8
+ , http-client-tls >= 0.3.6 && < 0.4
+ , http-types >= 0.12.4 && < 0.13
+ , lens >= 5.3.6 && < 5.4
+ , modern-uri >= 0.3.6 && < 0.4
+ , mtl >= 2.3.1 && < 2.4
+ , network-uri >= 2.6.4 && < 2.7
+ , parsec >= 3.1.18 && < 3.2
+ , parser-combinators >= 1.3.1 && < 1.4
+ , scrappy-core >= 0.1.0 && < 0.2
+ , text >= 2.1.3 && < 2.2
+ , time >= 1.12.2 && < 1.13
+ , transformers >= 0.6.1 && < 0.7
+
+ hs-source-dirs: src
+ default-language: Haskell2010
diff --git a/src/Scrappy/Requests/BuildActions.hs b/src/Scrappy/Requests/BuildActions.hs
new file mode 100644
--- /dev/null
+++ b/src/Scrappy/Requests/BuildActions.hs
@@ -0,0 +1,1676 @@
+{-# LANGUAGE LambdaCase #-}
+{-# LANGUAGE FlexibleContexts #-}
+{-# LANGUAGE OverloadedStrings #-}
+{-# LANGUAGE ScopedTypeVariables #-}
+{-# LANGUAGE RankNTypes #-}
+
+module Scrappy.Requests.BuildActions where
+
+--import Scrappy.Requests
+import Scrappy.Elem.SimpleElemParser (elemParser)
+-- import Elem.ElemHeadParse
+import Scrappy.Elem.Types (TreeHTML, Elem, Elem'(..), ElemHead, ElementRep, Attrs, ShowHTML, innerText', elTag, attrs, UrlPagination(..), matches', showH)
+import Scrappy.Elem.ElemHeadParse (hrefParser', hrefParser, attrsParser, parseOpeningTag, attrsMatch')
+-- import Links (Link, Option, Namespace, UrlPagination(..), maybeUsefulUrl)
+
+import Scrappy.Links (maybeUsefulUrl, Url, BaseUrl, Link(..) )
+import Scrappy.Find (findNaive, findSomeHTMLNaive)
+import Scrappy.Scrape
+import Scrappy.Requests.Types (CookieManager)
+
+import Network.HTTP.Client (Request, queryString, method, parseRequest)
+import Network.HTTP.Types.Method (Method, methodPost, methodGet)
+--import Control.Monad.Catch (MonadThrow)
+import Control.Monad (when)
+import Text.Parsec (ParsecT, ParseError, Stream, many, parse, string, (<|>), parserZero, try)
+import Text.Parsec.Error (Message (Message))
+import Data.Bifunctor (bimap)
+import Data.Either (fromRight)
+import Data.Text (Text, pack, unpack)
+import Data.Text.Encoding (encodeUtf8)
+import Data.Map as Map (Map, insert, adjust, findWithDefault, toList, fromList, lookup, union, empty, insertWith, keys)
+import Data.Maybe (fromMaybe, fromJust, catMaybes)
+import qualified Text.URI as URI
+import Data.Char (digitToInt, toLower)
+import Data.List (isPrefixOf)
+import Control.Monad.IO.Class
+
+
+
+-- | This process for processing forms to URLs is done by creating 4 different flows
+
+-- | 1. 1 =: 1 ; single valid value for the eventual key ;; typically for verification purposes
+-- | 2. 1 =: { Set of Strings } ; we have some amount of Text inputs which allow arbitrary input ;;
+-- --> we use this to skew search results to our best efforts through info on domain/keywords
+-- | 3 -- A) Select-like Elements
+-- | -- B) Radio Elements
+
+-- | 3A & 3B result in same type eventually -> Map (Namespace/Key) [Option]
+ --optimal solution begins generating "stream" of urls on new instance of (2)
+
+-- | In our business logic, it revolves around 2. ; we have maybe no way of knowing for sure which
+-- | textinput is which and how they interrelate
+-- | -> (eg. CASE1: 1st is paper title, 2nd is author\
+-- | CASE2: Only 1 text input for any)
+
+-- | We assume 1st is most likely but if it fails or gives unusual responses then we go to next
+-- | But for lack of free development time we are hoping this just works the first time and doesn't
+-- | need a ton of attention/error handling
+
+--for interacting with the web page solely through HTML
+
+-- When run, the most basic case will be chosen first
+data FilledForm = FilledForm { actionUrl :: Url -- | from action attribute
+ , reqMethod :: Method
+ , searchTerm :: Term
+ -- , actnAttr :: Action
+ , textInputOpts :: [TInputOpt]
+ , qStringVariants :: [QueryString] -- List of all possible queries given the parsed form.
+ }
+
+data EditableForm = EditableForm { _actionUrl :: Url
+ , rMethod :: Method
+ , params :: Map Namespace Option
+ } deriving (Eq, Show)
+
+-- data InputElem = Radio Namespace Option | SelectElem (Namespace, [Option]) | Basic Namespace (Maybe Option)
+data ParsedForm = ParsedForm Action Method [InputElem] deriving Show
+
+-- | Should make a record type for signins with an instance of Default
+
+
+
+type QueryString = [(Namespace, Option)]
+-- buildSearchUrlSubsets (mkSubsetVars variable radio)
+
+
+-- | This may introduce a small need for lenses
+mkEditableForm :: ParsedForm -> BaseUrl -> EditableForm
+mkEditableForm (ParsedForm act meth inpEls) (Link baseUrl) =
+ EditableForm (baseUrl <> "/" <> (unpack act)) meth map
+ where
+ map = simplify $ mkQParams "" inpEls
+ simplify :: ([QueryString], [QueryString]) -> Map Namespace Option
+ -- simplify = \(a,b) -> fromList $ zip a b
+ simplify (a,b) = fromList $ head a <> head b
+
+-- head of each listOfQueryStrings then concat them then toMap
+
+-- mkQParams :: String -> [InputElem] -> ([TInputOpt], [QueryString])
+-- mkQParams searchTerm inputs = mkFormInputs searchTerm $ sepElems' inputs mempty
+
+
+-- toKeyVal :: [InputElem] -> QueryString
+-- toKeyVal = \case
+-- -- we need to filter
+-- Radio n o
+
+-- renderEditableForm :: EditableForm -> Url
+-- renderEditableForm form =
+
+-- | Unless we want to use
+-- submitEditableForm :: EditableForm -> Html
+
+
+-- hypothesis: fillForm with searchTerm == ""
+
+
+-- not actually sure if i use this
+fillForm :: ParsedForm -> Url -> String -> FilledForm
+fillForm (ParsedForm act'' meth formInputs) baseUrl searchTerm =
+ let
+ act = unpack act''
+ act' = if isPrefixOf baseUrl act then act
+ else
+ if isPrefixOf "/" act then baseUrl <> act
+ else baseUrl <> "/" <> act
+ (tInputs, subsets) = mkQParams searchTerm formInputs
+ in
+ FilledForm act' meth searchTerm tInputs subsets
+
+-- | Lazily build FilledForm so that the full list of possible querystrings is not evaluated
+mkFilledForm :: Url -> String -> Elem' String -> Either FormError FilledForm
+mkFilledForm baseUrl searchTerm element = case elTag element of
+ "form" ->
+ case parse formElem "" (innerText' element) of
+ Right (ParsedForm act meth formInputs) ->
+ let
+ (tInputs, subsets) = mkQParams searchTerm formInputs
+ in
+ return $ FilledForm (baseUrl <> "/" <> (unpack act)) meth searchTerm tInputs subsets
+ Left err -> Left $ ParsecError err
+ _ -> Left InvalidElement
+
+
+
+mkQParams :: String -> [InputElem] -> ([TInputOpt], [QueryString])
+mkQParams searchTerm inputs = mkFormInputs searchTerm $ sepElems' inputs mempty
+
+
+
+-- | (Radio | Var | Basic | TInput)
+mkFormInputs :: String -> SepdStruct -> ([TInputOpt], [QueryString])
+mkFormInputs searchTerm (radio, var, basic, tInput) =
+ let
+ radio' = radiosToMap radio
+ variable' = fromList var
+ -- textInput' = ( fmap (pack . (fromMaybe "") . (Map.lookup "name") . attrs) tInput
+ -- , fmap (pack . (findWithDefault "" "value") . attrs) tInput)
+
+ textInput' = (tInput, replicate (length tInput) "")
+ textInputOpts' = fmap (basic <>) (genTInputOpts' (pack searchTerm) textInput')
+ subPaths' = buildSearchUrlSubsets (union variable' radio')
+ subPaths'' = case length (take 1 subPaths') == 1 of
+ True -> subPaths'
+ False -> mempty : mempty
+ in (textInputOpts', subPaths'')
+
+
+--temp
+type SelectElem = Elem'
+-- I could use datakinds to promote "select"
+
+
+
+-- | Name and Namespace are really same shit; might just converge
+-- | Refer to literally "name" attribute
+type Namespace = Text
+
+-- | This is an operationally focused type where
+-- | a certain namespace is found to have n num of Options
+type Option = Text
+
+
+
+data QParams = Opt (Map Namespace [Option]) | SimpleKV (Text, Text)
+
+
+data Signin = Signin { user :: String
+ , pass :: String
+ , signinLink :: Url
+ }
+
+executeSignin :: CookieManager -> Signin -> IO CookieManager
+executeSignin = undefined
+
+handleMKNewManagerWithChangeCookie :: CookieManager -> Signin -> IO CookieManager
+handleMKNewManagerWithChangeCookie = undefined
+
+
+
+-- pageKey=param
+
+
+
+
+
+
+--PLACEHOLDER
+type ReqBody = String
+
+
+
+
+-- | I believe this could instead compose Name/Namespace
+-- | OR!! Is the [String] the list of Options?
+type FormOptionRadio = Map String [String] -- ?
+type FormOptionRadio' = Map Namespace [Option]
+
+
+-- Note: Upon further research, it would be useful to generalize to FormOption which reps
+-- a well formed input to a URL generator
+ -- the only difference for type="radio" is how we get this info
+
+ -- radio:
+
+ -- elem + elem + << .. >> + elem + << .. >> + elem
+
+ -- select
+
+ -- InContext $ (elem + elem + elem + elem)
+
+
+ -- but both eval to a Query Param with options
+
+ -- We could even have:
+
+
+
+getContainedUrls :: Elem' a -> Maybe [String]
+getContainedUrls e = findSomeHTMLNaive hrefParser (innerText' e)
+
+
+
+
+-------------------------------------------------------------------------------------------------------
+-------------------------------------------------------------------------------------------------------
+-----Above is outside of research websites domain and below is actions on resDB's------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+
+
+
+
+inputTypesCondensed = []
+
+data FormRes = FormLinks [Text]
+
+-- type Hrefs = Link
+-- data PageLinker = PL' URL [FormRes] [Hrefs]
+-- |since it can have multiple forms
+-- | This is more for my understanding of the bigger picture of links management
+ -- if say:
+ -- -> we found ALL forms and created ALL links
+ -- but this would be full of trash ofc
+
+-- 2 Options
+-- 1) get next page -> do ...
+-- 1.5) pull from page >>cond=> get x page -> do ...
+-- 2) pull from page -> pdf
+
+-- 3) Error handling
+
+-- | A key goal of the link management system is to allow for precedence ; ie we've found browsing
+-- | page but prefer search page
+ -- therefore: we continue to look for hrefs in such a case until we've exhausted the basic site
+ -- tree
+type HrefsOfPage = [Text]
+
+data HrefsSite = HrefsSite [HrefsOfPage]
+
+type URL = Text
+data PageLinker' = PL URL [FormRes] [HrefsSite]
+
+
+-- runParserOnHTML :: ToHTML a => a -> [HtmlGenParser a]
+-- -- where goal would just be to take a specific collection of patterns then grab all in one go
+-- -- * this would neeeeed arrows due to garbage collection and size of said garbage
+
+
+
+
+
+finish :: [String] -> [String] -> String
+finish [] _ = ""
+finish (x:xs) (y:ys) = "&" <> x <> "=" <> y <> finish xs ys
+
+finish' :: ([String], [String]) -> Int -> String -> String
+finish' (paramA:paramAs, paramB:paramBs) idx term = case idx of
+ -- SO we are at the right index if 0
+ 0 -> "&" <> paramA <> "=" <> term <> finish paramAs paramBs
+ _ -> "&" <> paramA <> "=" <> paramB <> finish' (paramAs, paramBs) (idx-1) term
+
+
+
+
+finish'' :: [Text] -> [Text] -> QueryString
+finish'' [] _ = []
+finish'' (x:xs) (y:ys) = (x, y) : finish'' xs ys
+ --"&" <> x <> "=" <> y <> finish xs ys
+
+finish''' :: ([Text], [Text]) -> Int -> Text -> QueryString
+finish''' (paramA:paramAs, paramB:paramBs) idx term = case idx of
+ 0 -> (paramA, term) : finish'' paramAs paramBs
+ -- "&" <> paramA <> "=" <> term <> finish'' paramAs paramBs
+ _ -> (paramA, paramB) : finish''' (paramAs, paramBs) (idx - 1) term
+ -- "&" <> paramA <> "=" <> paramB <> finish''' (paramAs, paramBs) (idx-1) term
+
+
+
+-- | Basically just calls finish' with starting state
+genTInputOpts' :: Text -> ([Text], [Text]) -> [QueryString]
+genTInputOpts' searchTerm params =
+ if (length $ snd params) /= (length $ fst params)
+ then
+ undefined
+ else
+ reverse $ go searchTerm start params --(start, params)
+ where
+ start = (length . fst $ params) - 1
+ go :: Text -> Int -> ([Text], [Text]) -> [QueryString]
+ go term idx params = if idx == -1 then [] else finish''' params idx term : go term (idx-1) params
+
+
+
+-- | Basically just calls finish' with starting state
+genSerchStrm :: String -> ([String], [String]) -> [Text]
+genSerchStrm searchTerm params =
+ if (length $ snd params) /= (length $ fst params)
+ then
+ undefined
+ else
+ reverse $ fmap (pack ) $ go searchTerm start params --(start, params)
+ where
+ start = (length . fst $ params) - 1
+ go :: String -> Int -> ([String], [String]) -> [String]
+ go term idx params = if idx == -1 then [] else finish' params idx term : go term (idx-1) params
+
+ -- 1 use of idx sets the next eval'd string , the other subtracts until its at that piece and
+ -- sees 0 thus evaluating the logical placement
+
+
+ -- go _ _ ([],_) _ = []
+ -- -- go term 0 (paramA:paramAs, paramB:paramBs) (stateIdx, state1) =
+ -- -- --in this case, idx will always be 0
+ -- -- (paramA <> "=" <> term <> "&" <> finish paramAs paramBs) : go term (stateIdx+1) state1 (stateIdx,state1)
+
+ -- go term idx params =
+
+
+ -- Seems like I need a go-like function which is forced to return Text/String not [Text]
+ -- would need idx, params, term
+ -- would literally be iter1 <> iter2
+
+
+ -- so if they havent hit pattern: idx==0 yet but no more left to pull and create then the
+ -- idx must exceed the length of the list
+
+allElems :: [String]
+allElems = ["this is a placeholder for describing all html tags as a type system"]
+
+-- case 1
+
+-- 84 : input
+-- 7 : Select
+-- 8 : button
+-- else 0
+
+-- case 2
+
+-- 83 : select + input
+
+
+-- | input has attr: type="" which determines value type for form url
+-- | Attrs that I care about really are refs@( name || id ) , (Maybe) value, type,
+inputElems :: Maybe [String]
+inputElems = Just [ "input"
+ , "textarea"
+ , "button"
+ , "meter"
+ , "progress"
+ ]
+
+-- Another idea is a type family for Elem tags that allows for certain functions to specify
+-- if they only work with a subset of tags ; if it works with all : it specifies the type family
+-- as the input arg
+
+inputTypes :: [String]
+inputTypes = ["button", "checkbox", "color", "date", "datetime-local", "email", "file", "hidden", "image", "month", "number", "password", "radio", "range", "reset", "search", "submit", "tel", "text", "week", "time", "url"]
+
+
+ --
+ --
+ --
+ --
+ --
+ --
+ --
+ --
+ --
+ --
+ --
+ --
+ --
+ --
+ --
+ -- --> Is just a text box
+ --
+ --
+ --
+ --
+ --
+ --
+ --src: https://www.w3schools.com/html/html_form_input_types.asp
+
+ -- How are buttons handled?
+
+ -- How should we handle cases like file? Should we specify value = Nothing?
+
+-- | Seems sepElems is not yet perfected for the whole scope of HTML Possibilities but
+-- | All functions that pull from (,,,) / 4[]
+
+
+-- | Could we just use parseElemHead here?
+ -- probably gonna be waaaaay more efficient with self-closing tags and I dont think there's
+ -- any relevance of the (maybe innerText)
+-- | && shouldnt this specify certain attrs? 145
+
+-- both Option and Value ~ Text
+data InputElem = Radio Namespace Option
+ | SelectElem (Namespace, [Option])
+ | Basic Namespace (Maybe Option)
+ deriving (Show, Eq)
+ -- Maybe distinguishes between basic
+-- | SO:
+
+
+
+
+
+ -- Radio case is also "input" elem tag
+
+-- (radio, variable, (basic, tInput))
+
+-- structure based on filtering steps based on Constructors
+
+-- - Instead of searchForm as arg, will be filtered, parsed form including sorted input elems
+
+
+instance ShowHTML InputElem where
+ showH = show -- TEMPORARY
+
+
+-- data ParsedForm = ParsedForm Action Method [InputElem] deriving Show
+
+
+-- f :: ParsedForm -> FilledForm
+
+
+-- basic, textInput, variable, radio
+ ------ NOT READY READY AS IS
+
+-- | (Radio | Var | Basic | TInput)
+type SepdStruct = ([(Namespace, Option)], [(Namespace, [Option])], [(Namespace, Option)], [Namespace])
+
+
+
+sepElems' :: [InputElem] -> SepdStruct -> SepdStruct
+sepElems' [] s = s
+sepElems' (elem:elems) (a,b,c,d) =(case elem of
+ Radio nom opt -> sepElems' elems ((nom, opt) : a, b, c, d)
+ SelectElem selEl -> sepElems' elems (a, selEl : b, c, d)
+ Basic namesp (Just opt) -> sepElems' elems (a, b, (namesp, opt) : c, d)
+ Basic namesp (Nothing) -> sepElems' elems (a, b, c, namesp : d))
+
+
+ -- elems = ( filter fBasic elems
+ -- , filter fTextInp elems
+ -- , filter fVar elems
+ -- , filter fRadio elems
+ -- )
+ -- where
+ -- fBasic e =
+ -- (elTag e == "input")
+ -- && (let typ = fromJust $ Map.lookup "type" (attrs e) in not $ elem typ ["text", "radio"])
+
+ -- fTextInp e = ( elTag e == "textarea" )
+ -- || (elTag e == "input" && (fromJust (Map.lookup "type" (attrs e)) == "text"))
+
+ -- fVar e = elem (elTag e) ["select", "datalist"]
+
+ -- fRadio e = elTag e == "input" && (fromJust (Map.lookup "type" (attrs e)) == "radio")
+
+
+-- ...
+-- any ->
+-- fmap (any,) (getValue any)
+-- case getValue any of
+-- Just a -> (any, a)
+-- Nothing -> Nothing
+-- OR
+-- case type' of
+-- -- so back to the starting calc, just not handled specially or dynamically ; we do the bare minimum
+-- -- We throw out if we can
+
+
+instance ShowHTML a => ShowHTML (Maybe a) where
+ showH (Just a) = showH a
+ showH Nothing = ""
+
+invalidSearchElems :: [Maybe InputElem] -> ParsecT s u m [InputElem]
+invalidSearchElems matches =
+ if (length $ filter (==Nothing) matches) == 0
+ then return $ catMaybes matches
+ else parserZero
+
+ -- length (matches' e) < 3
+
+-- -- | Scape pattern that matches on search forms
+-- -- | is like formElem but forces find
+-- searchForm :: Stream s m Char => ParsecT s u m (Elem' String)
+-- searchForm = do
+-- let
+-- f = (try $ string "search") <|> (try $ string "Search")
+-- e <- elemParser (Just ["form"]) (Just $ f) []
+-- if length (matches' e) < 3
+-- then parserZero
+-- else return e
+
+-- | Like formElem except that it also checks if the form is for searching, via a dumb manner
+searchForm :: Stream s m Char => ParsecT s u m ParsedForm
+searchForm = do
+ Elem' _ as matches innerT <- elemParser (Just ["form"]) (Just inputElem) []
+ let
+ count = length $ fromMaybe [] $ runScraperOnHtml ((try $ string "Search") <|> string "search") innerT
+ getAction = pack . fromJust . (Map.lookup "action")
+ getMethod as = case fromJust ((Map.lookup "method") as) of
+ "get" -> methodGet
+ "post" -> methodPost
+
+ if count < (min count (length matches))
+ then parserZero
+ else return $ ParsedForm (getAction as) (getMethod as) (catMaybes matches)
+
+
+-- | Scape pattern that matches on search forms as well as extracting the action and method attributes
+formElem :: Stream s m Char => ParsecT s u m ParsedForm
+formElem = do
+ Elem' _ as matches innerT <- elemParser (Just ["form"]) (Just inputElem) []
+ return $ ParsedForm (getAction as) (getMethod as) (catMaybes matches)
+ where
+ getAction = pack . fromJust . (Map.lookup "action")
+ getMethod as = case fmap toLower $ fromJust ((Map.lookup "method") as) of
+ "get" -> methodGet
+ "post" -> methodPost
+
+ -- if count < (min count (length matches))
+ -- then parserZero
+ -- else return $ ParsedForm (getAction as) (getMethod as) matchesSafe
+
+
+-- | Scrape pattern for element
+inputElem :: Stream s m Char => ParsecT s u m (Maybe InputElem)
+inputElem = (try radioBasic') <|> (fmap Just selectEl)
+
+-- -- checks el tag then returns either radio or
+-- radioBasic :: Stream s m Char => ParsecT s u m InputElem
+-- radioBasic = do
+-- (e, attrs) <- parseOpeningTag (Just ["input"]) []
+-- if (fromMaybe "" $ Map.lookup "type" attrs) == "radio"
+-- then return $ Radio (pack . fromJust $ Map.lookup "name" attrs) (pack . fromJust $ Map.lookup "value" attrs)
+-- else return $ Basic (pack . fromJust $ Map.lookup "name" attrs) (fmap pack $ Map.lookup "value" attrs)
+
+
+
+-- checks el tag then returns either radio or
+-- | parserZero means we dont care, Nothing means this form is invalid
+radioBasic' :: (Stream s m Char) => ParsecT s u m (Maybe InputElem)
+radioBasic' = do
+ (e, attrs) <- parseOpeningTag (Just ["input", "textarea"]) []
+ let
+ type' = fromJust $ Map.lookup "type" attrs
+ case e of
+ "textarea" ->
+ return . Just $ Basic (pack . fromJust $ Map.lookup "name" attrs) (fmap pack $ Map.lookup "value" attrs)
+ "input" -> processInputEl attrs
+ _ -> undefined
+
+ -- case type' of
+ -- "date", "datetime", "MONTH"+"week", "number", "range",, "time" -> doSomething
+ -- "date" -> parserZero
+ -- "range" -> parserZero -- | speciallyHandleRange
+ -- "time" -> parserZero
+ -- (SomethingImportant ) -> ""
+ -- _ -> parserZero
+
+ --WEIIRDD
+ -- Date | Number ( Range ) | Search | Time
+
+-- | Handles differences in implications from all input types
+-- | When an input given its type, would not impact the query url, it's discarded
+processInputEl :: Attrs -> ParsecT s u m (Maybe InputElem)
+processInputEl attrs =
+ let
+ type' = fromJust $ Map.lookup "type" attrs
+ in
+ case type' of
+ "radio" ->
+ return . Just $ Radio (pack . fromJust $ Map.lookup "name" attrs) (pack . fromJust $ Map.lookup "value" attrs)
+ "text" ->
+ return . Just $ Basic (pack . fromJust $ Map.lookup "name" attrs) (fmap pack $ Map.lookup "value" attrs)
+ "hidden" ->
+ return . Just $ Basic (pack . fromJust $ Map.lookup "name" attrs) (fmap pack $ Map.lookup "value" attrs)
+ "submit" -> parserZero
+ "checkbox" ->
+ if (elem "checked" (keys attrs))
+ then return . Just $ Basic (pack . fromJust $ Map.lookup "name" attrs) (fmap pack $ Map.lookup "value" attrs)
+ else parserZero
+ -- DONT CURRRR
+ "button" -> parserZero
+ "color" -> parserZero
+ "submit" -> parserZero
+ "search" -> parserZero
+
+ -- WEIRDDDD -> 99% chance not a search form
+ "email" -> case (Map.lookup "value" attrs) of
+ -- check value, if its not set, this should throw parserZero unless definitely a search form
+ Just a -> return . Just $ Basic (pack . fromJust $ Map.lookup "name" attrs) (fmap pack $ Map.lookup "value" attrs) -- (type', a)
+ Nothing -> return Nothing
+
+ "file" -> return Nothing ---SHOULD FAIL WHOLE FORM
+ "image" -> return Nothing
+ "password" -> return . Just $ Basic (pack . fromMaybe "password" $ Map.lookup "name" attrs) Nothing
+ "reset" -> return Nothing
+ "tel" -> return Nothing
+ "url" -> return Nothing
+ "range" -> handleNumber attrs
+ "number" -> handleNumber attrs
+ _ ->
+ case (Map.lookup "value" attrs) of
+ Just a -> return . Just $ Basic (pack . fromJust $ Map.lookup "name" attrs) (Just $ pack a)
+ Nothing -> return . Just $ Basic (pack . fromJust $ Map.lookup "name" attrs) (Just "")
+
+handleNumber :: Attrs -> ParsecT s u m (Maybe InputElem)
+handleNumber attrs = do
+ let
+ name = (pack . fromJust . (Map.lookup "name")) attrs
+ value = Map.lookup "value" attrs
+ minn = Map.lookup "min" attrs
+ maxx = Map.lookup "max" attrs
+
+ case value of
+ Just val -> return . Just $ Basic name (Just $ pack val)
+ Nothing ->
+ case minn of
+ Just valMin -> return . Just $ Basic name (Just $ pack valMin)
+ Nothing ->
+ case maxx of
+ Just valMax -> return . Just $ Basic name (Just $ pack valMax)
+ Nothing -> parserZero
+
+-- speciallyHandleNumber = speciallyHandleRange
+
+-- speciallyHandleRange attrs = (name, min, max) --> then iterate some interval between the min and max
+
+-- SelectElem Name [Option]
+
+-- | formRaw <- elemParser (Just ["form"]) (Just $ inputElem) []
+-- | return ParsedForm Action $ build (matches formRaw)...
+
+-- | Matches on
+selectEl :: Stream s m Char => ParsecT s u m InputElem
+selectEl = do
+ e <- elemParser (Just ["select", "datalist"]) (Just optionParser) []
+ return $ SelectElem ((pack . fromJust . (Map.lookup "name") . attrs $ e), (reverse $ fmap pack $ matches' e))
+ where
+ name e = pack . fromJust . (Map.lookup "name") . attrs $ e
+ opts e = fmap pack (matches' e)
+
+-- parseOpeningTag (Just ["option"])
+
+optionParser :: Stream s m Char => ParsecT s u m String
+optionParser = do
+ (_, a) <- parseOpeningTag (Just ["option"]) []
+ return $ (fromJust . (Map.lookup "value")) a
+
+
+ -- _ <- string "