A changelog is kept at http://community.haskell.org/~ndm/darcs/tagsoup/CHANGES.txt.
+A changelog is kept at https://github.com/ndmitchell/tagsoup/blob/master/CHANGES.txt.
- We first need to find where the information is displayed, and in what format. Taking a look at the front web page, when not logged in, you may notice that there is no hit count. However, looking at the source shows us: + We first need to find where the information is displayed, and in what format. Taking a look at the front web page, when not logged in, we see:
-<div class="printfooter"> -<p>Retrieved from "<a href="http://www.haskell.org/haskellwiki/Haskell"> -http://www.haskell.org/haskellwiki/Haskell</a>"</p> - -<p>This page has been accessed 615,165 times. -This page was last modified 15:44, 15 March 2007. -Recent content is available under <a href="/haskellwiki/HaskellWiki:Copyrights" -title="HaskellWiki:Copyrights">a simple permissive license</a>.</p> +<ul id="f-list"> + <li id="lastmod"> This page was last modified on 9 September 2013, at 22:38.</li> + <li id="viewcount">This page has been accessed 6,985,922 times.</li> + <li id="copyright">Recent content is available under <a href="/haskellwiki/HaskellWiki:Copyrights" title="HaskellWiki:Copyrights">a simple permissive license</a>.</li> + <li id="privacy"><a href="/haskellwiki/HaskellWiki:Privacy_policy" title="HaskellWiki:Privacy policy">Privacy policy</a></li> + <li id="about"><a href="/haskellwiki/HaskellWiki:About" title="HaskellWiki:About">About HaskellWiki</a></li> + <li id="disclaimer"><a href="/haskellwiki/HaskellWiki:General_disclaimer" title="HaskellWiki:General disclaimer">Disclaimers</a></li> +</ul>
- So we see that the hit count is available, but not shown. This leads us to rule 1: + So we see the hit count is available. This leads us to rule 1:
Rule 1:
@@ -143,7 +143,7 @@
openURL x = getResponseBody =<< simpleHTTP (getRequest x)
-main = do src <- openURL "http://haskell.org/haskellwiki/Haskell"
+main = do src <- openURL "http://www.haskell.org/haskellwiki/Haskell"
writeFile "temp.htm" src
@@ -155,7 +155,7 @@ Tagsoup installs both as a library and a program. The program contains all the examples mentioned on this page, along with a few other useful functions. In order to download a URL to a file:
-$ tagsoup grab http://haskell.org/haskellwiki/Haskell > temp.htm +$ tagsoup grab http://www.haskell.org/haskellwiki/Haskell > temp.htm@@ -172,35 +172,26 @@ If the user changes their website, they will do so in unpredictable ways. They may move the page, they may put the information somewhere else, they may remove the information entirely. If you want something robust talk to the site owner, or buy the data from someone. If you try and think about design changes, you will complicate your design, and it still won't work. It is better to write an extraction method quickly, and happily rewrite it when things change.
- So now, lets consider the fragment from above. It is useful to find a tag which is unique just above your snippet - something with a nice "id" property, or a "class" - something which is unlikely to occur multiple times. In the above example, "printfooter" as the class seems perfect. We decide that to find the snippet, we will start at a "div" tag, with a "class" attribute with the value "printfooter". + So now, lets consider the fragment from above. It is useful to find a tag which is unique just above your snippet - something with a nice "id" property, or a "class" - something which is unlikely to occur multiple times. In the above example, "viewcount" as the id seems perfect.
haskellHitCount = do - tags <- fmap parseTags $ openURL "http://haskell.org/haskellwiki/Haskell" - let count = fromFooter $ head $ sections (~== "<div class=printfooter>") tags - putStrLn $ "haskell.org has been hit " ++ show count ++ " times" + src <- openURL "http://haskell.org/haskellwiki/Haskell" + let count = fromFooter $ parseTags src + putStrLn $ "haskell.org has been hit " ++ count ++ " times" + where fromFooter = filter isDigit . innerText . take 2 . dropWhile (~/= "<li id=viewcount>")
- Now we start writing the code! The first thing to do is open the required URL, then we parse the code into a list of Tags. We then apply the sections function, which returns all the lists whose first element matches the query. We use the (~==) operator to construct the query - in this case asking for the "div" we mentioned earlier. This (~==) operator is very different from standard equality, it allows additional attributes to be present but does not match them. We write "<div class=printfooter>" as syntactic sugar for TagOpen "div" [("class","printfooter")]. If we just wanted any open tag with the given class we could have written (~== TagOpen "" [("class","printfooter")]) and this would have matched. Any empty strings in the second element of the match are considered as wildcards. -
-- Once we have a list of all matching prefixes, we take the head - assuming that only one will match. Then we apply fromFooter which needs to perform the traversal from the "printfooter" attribute onwards to the actual hit count data. -
- -- Now we have a stream starting at the right place, we generally mangle the code using standard list operators: + Now we start writing the code! The first thing to do is open the required URL, then we parse the code into a list of Tags with parseTags. The fromFooter function does the interesting thing, and can be read left to right:
--fromFooter x = read (filter isDigit num) :: Int - where num = ss !! (i - 1) - Just i = findIndex (== "times.") ss - ss = words s - TagText s = sections (~== "<p>") x !! 1 !! 1 -+
- This code finds s, the text inside the appropriate paragraph by knowing that its the second (!! 1) paragraph, and within that paragraph, its the second tag - the actual text. We then split up the text using words, find the message that comes after hit count, and read all the digits we can find - filtering out the comma. I'm pretty sure this could be done better using regular expressions, and I invite a reader to submit improved code. This code may seem slightly messy, and indeed it is - often that is the nature of extracting information from a tag soup. + This code may seem slightly messy, and indeed it is - often that is the nature of extracting information from a tag soup.
Rule 3:
@@ -220,8 +211,8 @@
First we spot that the page helpfully has named anchors, there is a current work anchor, and after that is one for Haskell. We can extract all the information between them with a simple take/drop pair:
-takeWhile (~/= "<A name=haskell>") $ -drop 5 $ dropWhile (~/= "<A name=current>") tags +takeWhile (~/= "<a name=haskell>") $ +drop 5 $ dropWhile (~/= "<a name=current>") tags
This code drops until you get to the "current" section, then takes until you get to the "haskell" section, ensuring we only look at the important bit of the page. Next we want to find all hyperlinks within this section: @@ -246,8 +237,8 @@ spjPapers = do tags <- fmap parseTags $ openURL "http://research.microsoft.com/en-us/people/simonpj/" let links = map f $ sections (~== "<A>") $ - takeWhile (~/= "<A name=haskell>") $ - drop 5 $ dropWhile (~/= "<A name=current>") tags + takeWhile (~/= "<a name=haskell>") $ + drop 5 $ dropWhile (~/= "<a name=current>") tags putStr $ unlines links where f :: [Tag] -> String