xpathdsv (empty) → 0.1.0.0
raw patch · 8 files changed
+256/−0 lines, 8 filesdep +basedep +hxtdep +hxt-xpathsetup-changed
Dependencies added: base, hxt, hxt-xpath, optparse-applicative, text
Files
- LICENSE +30/−0
- Main.hs +72/−0
- README.md +80/−0
- Setup.hs +2/−0
- sample.html +12/−0
- stack.yaml +36/−0
- test.sh +2/−0
- xpathdsv.cabal +22/−0
+ LICENSE view
@@ -0,0 +1,30 @@+Copyright Daniel Choi (c) 2016++All rights reserved.++Redistribution and use in source and binary forms, with or without+modification, are permitted provided that the following conditions are met:++ * Redistributions of source code must retain the above copyright+ notice, this list of conditions and the following disclaimer.++ * Redistributions in binary form must reproduce the above+ copyright notice, this list of conditions and the following+ disclaimer in the documentation and/or other materials provided+ with the distribution.++ * Neither the name of Daniel Choi nor the names of other+ contributors may be used to endorse or promote products derived+ from this software without specific prior written permission.++THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ Main.hs view
@@ -0,0 +1,72 @@+{-# LANGUAGE OverloadedStrings, RecordWildCards, ScopedTypeVariables, Arrows #-} +module Main where+import Text.XML.HXT.Core+import Data.Text (Text)+import qualified Data.Text as T+import Text.XML.HXT.XPath.Arrows+import Options.Applicative+import Data.List (intercalate)++data Options = Options {+ parseHtml :: Bool+ , outputDelimiter :: String+ , nullOutput :: String+ , xpathBase :: String+ , childXPaths :: [String]+ } deriving Show++options :: Parser Options+options = Options + <$> flag yes no (long "xml" <> help "Parse as XML, rather than HTML.")+ <*> strOption (short 'F' <> metavar "OUTPUT-DELIM" <> help "Default \\t" <> value "\t")+ <*> strOption (short 'n' <> metavar "NULL-OUTPUT" <> help "Null value output string. Default \"\"" <> value "")+ <*> strArgument (metavar "BASE-XPATH")+ <*> many (strArgument (metavar "CHILD-XPATH"))++opts = info (helper <*> options)+ (header "xpathdsv" <> fullDesc + <> progDesc "Extract DSV data from HTML or XML with XPath"+ <> footer "See https://github.com/danchoi/xpathdsv for more information.")++main :: IO ()+main = do+ o@Options{..} <- execParser opts+ let xpaths'' = mkXPaths nullOutput childXPaths+ r <- runX (readDocument [withValidate no, withWarnings no+ , withParseHTML parseHtml, withInputEncoding utf8] ""+ >>> extractDSV xpathBase xpaths''+ )+ mapM_ (putStrLn . intercalate outputDelimiter) $ concat r+++mkXPaths :: ArrowXml a => String -> [String] -> a XmlTree [String]+mkXPaths n [] = mkXPaths n ["."]+mkXPaths nullOutput' xpaths =+ let arrows :: ArrowXml a => [a XmlTree String]+ arrows = map (\x -> (listA+ (getXPathTrees x >>> getText' >>> cleanText)+ >>^ concat >>> replaceNull nullOutput')+ `orElse` constA nullOutput')+ xpaths+ in listA (catA arrows)++getText' :: ArrowXml a => a XmlTree String+getText' = (isText >>> getText) `orElse` xshow this++cleanText :: ArrowXml a => a String String+cleanText = arr (T.unpack . T.concatMap escapeNewLines . T.strip . T.pack)++replaceNull nullOutput' = arr (\x -> if null x then nullOutput' else x)++escapeNewLines '\n' = " "+escapeNewLines '\r' = " "+escapeNewLines '\t' = " "+escapeNewLines x = T.singleton x++++type Row = (String, [String]) -- output, one column minimum++extractDSV :: ArrowXml a => String -> a XmlTree [String] -> a XmlTree [[String]]+extractDSV xpath1 xpaths = + getXPathTrees xpath1 >>> listA xpaths
+ README.md view
@@ -0,0 +1,80 @@+# xpathdsv+++Extract DSV text from HTML and XML using XPATH expressions.++## Example++If you have an HTML file like this:++sample.html++```html+<html>+ <head><title>Test</title></head>+ <body>+ <h1>Some links</h1>+ <ul>+ <li><a href="http://news.ycombinator.com">Hacker News</a></li>+ <li><a href="http://yahoo.com">Yahoo</a>+ <li><a href="http://duckduckgo.com">Duck Duck Go</a>+ <li><a href="http://github.com">GitHub</a>+ </ul>+ </body>+</html>+```++You can extract a list of tab-separated values like this:++ xpathdsv '//a' '/a/text()' '/a/@href/text()' < sample.html++Output:++ Hacker News http://news.ycombinator.com+ Yahoo http://yahoo.com+ Duck Duck Go http://duckduckgo.com+ GitHub http://github.com+++The first XPATH expression in the command sets the base node on which all the+following XPATH expressions are applied. Each of the following XPATH expressions+then generate a column of the row of data.++If you don't specify a `text()` node at the end of an XPATH expression, you'll get +a string representation of a node, which may be useful for debugging:++ xpathdsv '//a' '/a' < sample.html++Output:++ <a href="http://news.ycombinator.com">Hacker News</a>+ <a href="http://yahoo.com">Yahoo</a>+ <a href="http://duckduckgo.com">Duck Duck Go</a>+ <a href="http://github.com">GitHub</a>++## Usage+++ xpathdsv+ + Usage: xpathdsv [--xml] [-F OUTPUT-DELIM] [-n NULL-OUTPUT] BASE-XPATH+ [CHILD-XPATH]+ Extract DSV data from HTML or XML with XPath+ + Available options:+ -h,--help Show this help text+ --xml Parse as XML, rather than HTML.+ -F OUTPUT-DELIM Default \t+ -n NULL-OUTPUT Null value output string. Default ""+ + See https://github.com/danchoi/xpathdsv for more information.+++## Author++Daniel Choi <https://github.com/danchoi>+++## References++* <https://en.wikipedia.org/wiki/XPath>
+ Setup.hs view
@@ -0,0 +1,2 @@+import Distribution.Simple+main = defaultMain
+ sample.html view
@@ -0,0 +1,12 @@+<html>+ <head><title>Test</title></head>+ <body>+ <h1>Some links</h1>+ <ul>+ <li><a href="http://news.ycombinator.com">Hacker News</a></li>+ <li><a href="http://yahoo.com">Yahoo</a>+ <li><a href="http://duckduckgo.com">Duck Duck Go</a>+ <li><a href="http://github.com">GitHub</a>+ </ul>+ </body>+</html>
+ stack.yaml view
@@ -0,0 +1,36 @@+# This file was automatically generated by stack init+# For more information, see: http://docs.haskellstack.org/en/stable/yaml_configuration/++# Specifies the GHC version and set of packages available (e.g., lts-3.5, nightly-2015-09-21, ghc-7.10.2)+resolver: lts-5.18++# Local packages, usually specified by relative directory name+packages:+- '.'+# Packages to be pulled from upstream that are not in the resolver (e.g., acme-missiles-0.3)+extra-deps:+- hxt-xpath-9.1.2.2++# Override default flag values for local packages and extra-deps+flags: {}++# Extra package databases containing global packages+extra-package-dbs: []++# Control whether we use the GHC we find on the path+# system-ghc: true++# Require a specific version of stack, using version ranges+# require-stack-version: -any # Default+# require-stack-version: >= 1.0.0++# Override the architecture used by stack, especially useful on Windows+# arch: i386+# arch: x86_64++# Extra directories used by stack for building+# extra-include-dirs: [/path/to/dir]+# extra-lib-dirs: [/path/to/dir]++# Allow a newer minor version of GHC than the snapshot specifies+# compiler-check: newer-minor
+ test.sh view
@@ -0,0 +1,2 @@+#!/bin/bash+xpathdsv '//a' '/a/@href/text()' '/a/text()' < sample.html
+ xpathdsv.cabal view
@@ -0,0 +1,22 @@+name: xpathdsv+version: 0.1.0.0+synopsis: Command line tool to extract DSV data from HTML and XML with XPATH expressions+description: Please see README.md+homepage: https://github.com/danchoi/xpathdsv#readme+license: BSD3+license-file: LICENSE+author: Daniel Choi+maintainer: dhchoi@gmail.com+copyright: Daniel Choi 2016+category: Text+build-type: Simple+cabal-version: >=1.10++executable xpathdsv+ main-is: Main.hs+ default-language: Haskell2010+ build-depends: base >= 4.7 && < 5+ , hxt + , hxt-xpath+ , text+ , optparse-applicative