packages feed

gscholar-rss 0.1.0.0 → 0.2.0.0

raw patch · 4 files changed

+127/−7 lines, 4 filesbinary-added

Files

gscholar-rss.cabal view
@@ -1,5 +1,5 @@ name:                gscholar-rss-version:             0.1.0.0+version:             0.2.0.0 synopsis:            scrapes google scholar, provides RSS feed description:         A simple Google Scholar scraper, providing RSS/Atom                      feeds. Check homepage for manual and examples.@@ -7,12 +7,14 @@ license-file:        LICENSE author:              Francesco Ariis maintainer:          fa-ml@ariis.it-copyright:           2018 © Francesco Ariis+copyright:           2018-2019 © Francesco Ariis homepage:            http://www.ariis.it/static/articles/gscholar-rss/page.html category:            Web, Utils build-type:          Simple-extra-source-files:  README-cabal-version:       >=2.0+extra-source-files:  README,+                     manual/page.rst,+                     manual/gscholar.jpg+cabal-version:       2.0  executable gscholar-rss   main-is:             Main.hs
+ manual/gscholar.jpg view

binary file changed (absent → 13667 bytes)

+ manual/page.rst view
@@ -0,0 +1,99 @@+--------------------------------------------------------------------------------+title: gscholar-rss+author: Francesco Ariis+summary: Google Scholar to RSS feed scraper+tags: google scholar, scholar, rss, feed, atom+published: 2018-11-05 00:00:00+revised: 2018-11-10 00:00:00+--------------------------------------------------------------------------------++============+gscholar-rss+============++``gscholar-rss`` (`Hackage entry`__) is an RSS scraper for Google Scholar,+so you can be warned via RSS on the latest academic research of your choice.++.. __: http://hackage.haskell.org/package/gscholar-rss++Getting gscholar-rss+--------------------++To get ``gscholar-rss``, you can alternatively:++- download the appropriate binary from the `binaries page`__ and put it+  somewhere visibile in your ``$PATH``. **Or**++.. __: http://ariis.it/link/repos/gscholar-rss/bin/++- compile it yourself. This means getting Haskell for your platform+  (e.g. on Debian ``sudo apt-get install haskell-platform``).+  Once you installed it:++  ::++     cabal get gscholar-rss+     # now `cd` into the newly created folder and run+     cabal new-build++  will create a binary (the location is displayed upon completion -- check+  for a line similar to ``Linking /home/f/spool/gs-rss...``).++Using gscholar-rss+------------------++- *visit* `Google Scholar`__ and perform a search.++.. __: http://scholar.google.com++  .. image:: gscholar.jpg+    :alt: Google Scholar page, "sort by date highlighted++  as highlighted by the picture, most likely you want to select the+  "Sort by date" option.++- *copy* the search URL+  (in my case ``http://scholar.google.it/scholar?hl=en&scisbd=1&as...``)+  and pass it as an argument to ``gscholar-rss``. Don't forget to add+  quotations around your URL!++  ::++    ./gscholar-rss "http://scholar.google.it/scholar?hl=en&scisbd=1&as..."++This should generate a well-formed RSS feed.++Integration with feed-readers+-----------------------------++Many feed-readers allow taking advantage of external programs to fetch+a feed; this is exactly what ``gscholar-rss`` needs.+Refer to the documentation of your feed-reader of choice on how to do+that. As examples:++- **Newsbeuter/Newsboat**: the relevant manual section is `Scripts and+  Filters (Snownews Extensions)`__, which means writing entries like+  this in your url-file:++.. __: https://newsboat.org/releases/2.13/docs/newsboat.html#_scripts_and_filters_snownews_extensions++  ::++    "exec:gscholar-rss 'https://scholar.google.it/scholar?hl=en&as_sdt=0,5&q=haskell&scisbd=1'"++- **Liferea**: `Website scraping with liferea`__.++.. __: https://lzone.de/liferea/scraping.htm+++Repository and contact+----------------------++Here is the `darcs repository`_.++.. _`darcs repository`: http://ariis.it/link/repos/gscholar-rss/++If you want to report a bug or request a feature, `write to me`_.++.. _`write to me`: /static/articles/mail/page.html+
src/Main.hs view
@@ -11,12 +11,12 @@ import Text.HTML.Scalpel.Core import Text.RSS.Syntax import Text.RSS.Export-import Text.URI  import qualified Data.Text          as T import qualified Data.Text.Lazy.IO  as TLI import qualified Data.Text.Encoding as TE import qualified Data.String        as DS+import qualified Text.URI           as U   main :: IO ()@@ -70,7 +70,7 @@ entries = chroots ("div" @: [hasClass "gs_ri"]) paper     where           paper = Article <$> text ("h3" @: [hasClass "gs_rt"])-                          <*> attr "href" "a"+                          <*> (cleanGBooks <$> attr "href" "a")                           <*> text ("div" @: [hasClass "gs_a"])  buildRSS :: URL -> Page -> RSS@@ -97,9 +97,28 @@           req = DS.fromString (T.unpack url)  feedTitle :: URL -> Title-feedTitle url = case lookup "q" (queryToPairs $ T.unpack url) of+feedTitle url = case lookup "q" (U.queryToPairs $ T.unpack url) of                   Nothing -> "Error: title not found"                   Just t  -> "gscholar-rss: " <> T.pack t++-- removes 'ots' and 'sig' from a google books url+cleanGBooks :: URL -> URL+cleanGBooks url+        | U.uriRegName uri ==+            Just "books.google.it" =+            let+                qry = U.uriQueryItems uri+                fqr = filter filtFun qry+                u'  = uri { U.uriQuery = Just (U.pairsToQuery fqr)}+            in+            T.pack . show $ u'+        | otherwise = url+    where+          uri = maybe (error "gclean: misparse")+                      id+                      (U.parseURI $ T.unpack url)+          filtFun :: (String, String) -> Bool+          filtFun (l, _) = not (elem l ["ots", "sig"])  printRSS :: URL -> Page -> IO () printRSS url s =