hxt-8.3.1: doc/index.html
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<title>Haskell XML Toolbox 8.3.1</title>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
</head>
<body style="background-color: #ffffff;">
<h1>Haskell XML Toolbox 8.3.1</h1>
<h3>Contents</h3>
<ul>
<li><a href="#intro">Introduction</a></li>
<li><a href="#descr">Description</a></li>
<li><a href="#docu">Documentation</a></li>
<li><a href="#req">Requirements</a></li>
<li><a href="#down">Download</a></li>
<li><a href="#install">Installation</a></li>
<li><a href="#change">Change history</a></li>
<li><a href="#limit">Known problems and limitations</a></li>
<li><a href="#port">Portability</a></li>
<li><a href="#filter">HXT with Filters</a>
<li><a href="#rel">Related Work</a></li>
<li><a href="#feed">Feedback</a></li>
</ul>
<hr>
<a name="intro"></a>
<h3>Introduction</h3>
<p>
The <i>Haskell XML Toolbox</i> is a collection of tools for
processing XML with
<a href="http://www.haskell.org">Haskell</a>.
It is purely written in Haskell.
The <i>Haskell XML Toolbox</i> is a project of the
<a href="http://www.fh-wedel.de">University of Applied Sciences Wedel</a>,
</p>
<p>
The main design goal of the <i>Haskell XML Toolbox</i>
is the support of various XML standards including
<a href="http://www.w3.org/TR/REC-xml">Extensible Markup
Language (XML) 1.0 (Second Edition)</a> with DTD
processing and Validation,
<a href="http://www.w3.org/TR/REC-xml-names/">Namespaces in XML
1.0 (Second Edition)</a>,
<a href="http://www.w3.org/TR/xpath">XML Path Language
(XPath)</a>,
<a href="http://www.w3.org/TR/xslt">XSL Transformations
(XSLT)</a>,
<a
href="http://www.oasis-open.org/committees/relax-ng/spec-20011203.html">RELAX NG Specification</a>,
as well as HTML/XHTML processing.
</p>
<a name="descr"></a>
<h3>Description</h3>
<p>
The <i>Haskell XML Toolbox</i> bases on the ideas of
<a href="http://www.cs.york.ac.uk/fp/HaXml/">HaXml</a> and
<a href="http://www.flightlab.com/~joe/hxml/">HXML</a>,
but introduces a more general and flexible approach for processing XML with
Haskell.
The <i>Haskell XML Toolbox</i> uses a generic data model for
representing XML documents,
including the DTD subset and the document subset, in Haskell.
This data model makes it possible to use filter functions
as a uniform design of XML processing applications.
The processing filters are implemented as
<a href="http://www.haskell.org/haskellwiki/Arrow">arrows</a>.
This is more flexible than the filter approach from HXML and
HaXml,
but all filter applications can easily be transformed into
arrows.
</p>
<p>
Since version 5.2 HXT works with arrows instead of filters.
The filter part has been separated from this library and is
available in an extra package (see <a href="#filter">HXT with Filters</a>)
There is a <em>cookbook</em> for using this arrow interface
to build (nontrivial) applications. Manuel Ohlendorf
has developed examples for processing RDF and has documented the
development in his master thesis: <em>A Cookbook for the Haskell
XML Toolbox with Examples for Processing RDF Documents</em>
(the thesis as <a href="cookbook/doc/thesis.pdf">PDF</a>)
</p>
<p>Features:</p>
<ul>
<li>Unicode and UTF-8, US-ASCII and ISO-Laitin-1 support</li>
<li>http: and file: protocol support</li>
<li>http access via proxy</li>
<li>wellformed document parsing, validation</li>
<li>namespace support: namespace propagation and checking</li>
<li>XPath support for selection of document parts</li>
<li>liberal HTML parser for interpreting any text containing <
... > as HTML/XML</li>
<li>liberal and lasy lightweight HTML/XML parser based on
<a href="http://www-users.cs.york.ac.uk/~ndm/tagsoup/">tagsoup</a></li>
<li><a href="http://www.relaxng.org/">Relax NG</a> schema
validator</li>
<li>integrated <a href="http://www.w3.org/TR/xslt">XSLT</a> transformer</li>
<li>easy conversion between user defined data structures and XML
by the use of pickler functions</li>
</ul>
<a name="docu"></a>
<h3>Documentation</h3>
<p>
The <em><a href="./hdoc/index.html">HXT API Documentation</a></em> generated
with <a href="http://www.haskell.org/haddock/">Haddock</a>.
</p>
<p>
A (somewhat) gentle introduction to HXT is avalable in the
<a href="http://www.haskell.org/haskellwiki/HXT">Haskell
Wiki</a>.
There's also a page about
<a href="http://www.haskell.org/haskellwiki/HXT/Conversion_of_Haskell_data_from/to_XML">HXT: Conversion of Haskell data from/to XML</a> with picklers.
</p>
<p>
The XSLT transformer has been developed by Tim Walkenhorst in
this master thesis: <em><a href="xslt/thesis.pdf">Implementing an XSLT
processor for the Haskell XML Toolbox</a></em>. It's a
rather complete implementation, but it's of course not a
substitute for Xalan or other advanced XSLT systems. The XSLT
module consists of less than 2000 lines of code. Compared with
the more than 300,000 lines of Java for Xalan, this Haskell code
can be viewed as one of the first formal specifications for XSLT.
</p>
<p>
Manuel Ohlendorfs master thesis, describing the arrow interface
of the toolbox: <em>A Cookbook for the Haskell
XML Toolbox with Examples for Processing RDF Documents</em>
(the thesis as <a href="cookbook/doc/thesis.pdf">PDF</a>).
The source code of the example application is included in the
<tt>doc/cookbook</tt> directory of the distribution.
</p>
<p>
The master's thesis
<i>"Design and Implementation of a validating XML parser in
Haskell"</i>
by Martin Schmidt describes the design and motivation of the
<i>Haskell XML Toolbox</i>
(the thesis as <a href="thesis/index.html">HTML</a>
or <a href="thesis.pdf">PDF</a>) and the development of the DTD
validator module.
The documentation in the thesis is a bit out of date, the modules
and module names and some function names have been changed. For details the online
haddock documentation should be used.
</p>
<p>
The description of the development of the <a href="http://www.w3.org/TR/xpath">XPath</a> modules
is described (in german) in <em>Konzeption und Implementierung
eines XPath-Moduls für die Haskell XML Toolbox</em>
(<a href="./hxpath/diplomarbeit.pdf">PDF-document</a>).
</p>
<p>
The description of the internals of the <a href="http://www.relaxng.org/">Relax NG</a> validator
modules is described (in german) in <em>Design und Entwicklung
eines <a href="http://www.relaxng.org/">Relax NG</a> Schema Validators auf Basis der Haskell XML
Toolbox</em> (<a href="./relaxng/thesis.pdf">PDF-document</a>).
</p>
<a name="req"></a>
<h3>Requirements</h3>
<ul>
<li>ghc-6.10 with cabal 1.6</li>
<li><a href="http://hackage.haskell.org/cgi-bin/hackage-scripts/package/curl">curl package >= 1.3</a></li>
<li><a href="http://hackage.haskell.org/cgi-bin/hackage-scripts/package/tagsoup">tagsoup package >= 0.6</a></li>
</ul>
<p>
It is recommended to install the versions available from
<a href="http://hackage.haskell.org/packages/hackage.html">Hackage</a>.
</p>
<a name="down"></a>
<h3>Downloads</h3>
<p>
<a href="hxt-8.3.1.tar.gz">Haskell XML Toolbox 8.3.1</a>,
released 2009-06-16:
</p>
<ul>
<li>This version works with ghc 6.10 with cabal 1.6.
For ghc 6.8 and cabal & 1.6 please use HXT 8.1.1.</li>
<li>This version does not contain the filter part of hxt any
more.
That functionality is separated into a package <a
href="#filter">hxt-filter</a>.
</li>
<li>Includes sources for building a ghc package
<code>hxt</code> with Cabal or make.
This package contains a Haskell DOM, an XML parser, a HTML parser based on parsec,
a lightweight HTML/XML parser based on tagsoup, a
DTD validator, namespace processing functions, a Relax
NG validator, an XPath expression evaluator, an XSLT transformer,
and serialization/deserialisation of data to/from XML.
</li>
<li>HTTP access is done via the <a href="http://hackage.haskell.org/cgi-bin/hackage-scripts/package/curl">curl binding</a>.
</li>
<li>Includes various examples, e.g. in example dir
<tt>examples/arrows/hparser/</tt>
a validating parser, which can be used as a starting point for
a HXT command line application.
</li>
<li>Includes an <a href="http://www.haskell.org/arrows/">arrow interface</a> with type classes and
overloading for a more flexible use of the filter technique.
</li>
</ul>
<p>
A darcs repository is available under <a href="http://darcs2.fh-wedel.de/repos/hxt/">http://darcs2.fh-wedel.de/repos/hxt</a>,
the web interface is <a href="http://darcs2.fh-wedel.de/cgi-bin/darcsweb.cgi">http://darcs2.fh-wedel.de/cgi-bin/darcsweb.cgi</a>.
</p>
<a name="install"></a>
<h3>Installation</h3>
<p>
Before installing this version, install the curl and tagsoup modules.
For a quick install with Cabal execute the following commands in
<tt>.../HXT-8.3.1</tt>
</p>
<pre>
cabal configure
cabal build
sudo cabal install --global
</pre>
<p>
A quick test of the example programs:
</p>
<pre>
cd examples
make all
make test
</pre>
<p>
Installation without Cabal with GNU make:
</p>
<pre>
make all
make install # with root privileges
</pre>
<a name="change"></a>
<h3>Change History</h3>
<ul>
<li>In <a href="hxt-8.3.1.tar.gz">Version 8.3.1</a>
This is only a bug fix release.
<ul>
<li>
Interface and option handling for libcurl reworked.
New input option "a_no_redirect" for preventing autmatic
redirect added.
</li>
<li>
Encoding of none XML/HTML text data done with the same
encoding routines as for XML/HTML. This enables easy processing
of other text documents.
</li>
</ul>
</li>
<li>In <a href="hxt-8.3.0.tar.gz">Version 8.3.0</a>
<ul>
<li>
New output option "a_no_empty_elements" for preventing
the XML short format "<name/>" for HTML elements,
like "script", "p", and others. Especially a script tag
of the form "<script href="..."/>" does not work in
firefox.
Turning on this option gives the form "<script href="..."></script>".
</li>
<li>
An input option "a_strict_input" for bytestring input of
files added. Lazy input, especially when using the tagsoup
parser, can lead to error messages like "too many open
files" when processing a whole bunsh of documents.
</li>
<li>Internal representation of qualified names changed to gain
more space efficency.
</li>
</ul>
</li>
<li>In <a href="hxt-8.2.0.tar.gz">Version 8.2.0</a>
<ul>
<li>
Modifications to work with ghc-6.10.
</li>
<li>
A new module <tt>Data.Atom</tt> for dealing with names like LISP atoms
and for sharing the memory for these values. When using names as keys
in tables, trees or maps, it becomes much more efficient to represent these
names as atoms than as strings. Equality check on atoms is constant in time and really fast, and all occurences
of an atom share the same internal value.
</li>
<li>
Implementation of <tt>strictA</tt> changed. <tt>strictA</tt> is marked deprecated.
The implementation is not longer done with a <tt>DeepSeq</tt> function but with
<tt>Control.Parallel.Strategies</tt>, the <tt>NFData</tt> class and <tt>rnf</tt>.
There is a new combinator <tt>rnfA</tt> for complete <em>deep</em> evaluation of an
arrow result.
</li>
<li>
Further functions for working with <a href="http://www.w3.org/TR/xmlschema11-2/#regexs">W3C XML Schema Regular expressions</a>
in module Text.XML.HXT.RelaxNG.XmlSchema.RegexMatch,
especially for tokenizing and sed like editing of text.
</li>
</ul>
</li>
<li>In <a href="hxt-8.1.1.tar.gz">Version 8.1.1</a>
<ul>
<li>
New functions for working with <a href="http://www.w3.org/TR/xmlschema11-2/#regexs">W3C XML Schema Regular expressions</a>
in module Text.XML.HXT.RelaxNG.XmlSchema.RegexMatch.
</li>
<li>
Darcs server has changed, new server is <tt><a href="http://darcs2.fh-wedel.de/repos/hxt/">http://darcs2.fh-wedel.de/repos/hxt/</a></tt>.
</li>
</ul>
</li>
<li>In <a href="hxt-8.1.0.tar.gz">Version 8.1.0</a>
<ul>
<li>
HTTP interface changed to work with libcurl
via curl bindings with package <a href="http://hackage.haskell.org/cgi-bin/hackage-scripts/package/curl">curl</a>.
So the HTTP package is not longer needed, also the old
and somewhat inefficent interface to curl by starting an external process and
communicate via a pipe is not longer needed.
When installing the curl bindings, be aware that the
libcurl development packages including the C header files
must be installed. Otherwise the Setup.hs will complain of
missing files.
</li>
<li>
New input option for ignoring none XML/HTML contents
when reading documents (useful for crawler like applications).
</li>
<li>
Mime type support for the <tt>file:</tt> protocol. Mime type mapping
can be controlled by a config file in the format of
<tt>/etc/mime.types</tt> on some linux systems.
</li
<li>
A few more picklers for de-serialization from/to XML, e.g. for maps.
</li>
<li>
A new option for ignoring decoding errors when reading XML documents.
This may be useful for crawler like applications.
</li>
</ul>
</li>
<li>In <a href="hxt-8.0.0.tar.gz">Version 8.0.0</a>
<ul>
<li>
Old filter interface separated from the hxt package and moved
to an extra package <a href="#filter">hxt-filter</a>.
</li>
<li>
Version numbers added in hxt.cabal for required package versions.
</li>
<li>
DTD validation and XPath modules refactored to work with arrows instead of
filters.
This is done for separating the old hxt filter library
from the actively developed and maintained arrow part.
</li>
</ul>
</li>
<li>In <a href="hxt-7.5.tar.gz">Version 7.5</a>
<ul>
<li>
A module <tt>Text.XML.HXT.Arrow.XPathSimple</tt> for fast XPath selection for simple XPath queries
added. If the XPath query only contains for navigaion the axis from the
root down to the leaves, the query is evaluated by computing
a simple arrow and applying the arrow. This gives a
speedup for queries like <tt>/htm/body//h1/text()</tt> for
extraction of the headline text of a HTML document.
</li>
<li>
Option added to control the selection of the parser
(XML/HTML) by the mime type of the document.
</li>
<li>
New lasy and lightweight HTML/XML parser based on tagsoup library.
Useful especially for converting XML into native Haskell
data. (Example in examples/arrows/performance)
</li>
<li>
W3C XML Schema Datatype library for validating with RelaxNG extended.
Currently supported Datatypes: Strings, URIs, QNames,
binary string-encoded, decimal and all integer datatypes.
</li>
<li>
Regular expression pattern matcher for W3C XML Schema datatype patterns.
<li>
<li>New arrow combinator 'mergeA' in Control.Arrow.ArrowList
for combining the components of a tuple, resulting from
applying arrows constructed with (&&&), (***) and similar combinators.
</li>
</ul>
</li>
<li>In <a href="hxt-7.4.tar.gz">Version 7.4</a>
<ul>
<li>Configuration changes for ghc-6.8</li>
<li>Error in DTD validation algorithm resulting in exponential
growth of runtime during validating mixed content elements removed.
</li>
</ul>
</li>
<li>In <a href="hxt-7.3.tar.gz">Version 7.3</a>
<ul>
<li>Module <tt>Text.XML.HXT.Pickle</tt> for conversion between user
defined data types and the HXT DOM structure extendet.
The picklers are extended, such that a DTD can be derived
from a pickler and can be checked for consistency.
See Haddock doc (<tt>Text.XML.HXT.Pickle</tt>)
and example directory
<tt>examples/arrows/pickle</tt> for examples.
</li>
</ul>
</li>
<li>In <a href="HXT-7.2.tar.gz">Version 7.2</a>
<ul>
<li>New module Text.XML.HXT.XmlPickler for conversion between user
defined data types and the HXT DOM structure. This enables
the simple persistent storage and retrieval of arbitrary
data with XML documents. See example directory
<tt>examples/arrows/pickle</tt> for a none trivial example
of these picklers. These functions are an adaptation of
<a href="http://research.microsoft.com/~akenn/fun/picklercombinators.pdf">Andrew
Kennedy's pickler combinators</a>.
</li>
<li>UTF-8 decoding done with UTF8 module from
<a href="http://www.abridgegame.org/darcs/">darcs</a>.
Decoding errors are detected and issued.
US-ASCII decoding also checks encoding errors.
</li>
<li>
ISO-8859-X (X=2..11,13..16) deccoding of input documents implemented
</li>
<li>
Some bug fixes.
</li>
</ul>
</li>
<li>In <a href="HXT-7.1.tar.gz">Version 7.1</a>
<ul>
<li>Version control changed from CVS to darcs.
darcs repository is available under
<tt><a href="http://darcs2.fh-wedel.de/repos/hxt/">http://darcs2.fh-wedel.de/repos/hxt/</a></tt>.
</li>
<li><tt>deepSeq</tt> and <tt>strict</tt> added for <tt>XmlTree</tt>
and an arrow <tt>strictA</tt> for forcing the evaluation of a whole
XML tree. This sometimes saves space when applied after document input,
DTD processing and validation.
</li>
<li>Typeable instances added for all DOM data types.</li>
<li>HTTP access via curl extended to handle automatic redirects</li>
</ul>
</li>
<li>In <a href="HXT-7.0.tar.gz">Version 7.0</a>
<ul>
<li>New integrated XSLT transformer module. The example
parser (<tt>examples/arrows/hparser</tt>) is extended to act as an XSLT transformer.
</li>
<li>Errors concerning HTML parsing and implict closing of
elements have been fixed.
</li>
<li>Some minor changes of and additions to the arrow API.
</li>
<li>New functions <tt>readString</tt> and
<tt>readFromString</tt> for reading documents from Haskell
strings the same way as reading external documents with <tt>readDocument</tt>.
</li>
</ul>
</li>
<li>In <a href="HXT-6.1.tar.gz">Version 6.1</a>
<ul>
<li>HXT 6.1. contains only changes in the cabal installation process for
working with the <a href="http://www.haskell.org/http/">Haskell HTTP module</a> in tar archive http-20060707.tgz.
This module no longer depends on <a href="http://www.haskell.org/crypto/">The Haskell Cryptographic Library</a>,
and <a href="http://www.n-heptane.com/nhlab/">NewBinary package</a>.
</li>
</ul>
</li>
<li>In <a href="HXT-6.0.tar.gz">Version 6.0</a>
<ul>
<li>The arrow interface has changed slightly, especially the handling of user defined states in the state and IO arrows has been simplified.
This is the main reason for a 6.0 version.
</li>
<li>The XPath arrows have been extended.
There are arrows not only for selecting nodes via an XPath expression, but also
for processing and modifying all nodes selected
by an XPath expressions (see module Text.XML.HXT.Arrow.XmlNodeSet).
</li>
<li>Separation of the API documentation into two documents, one for the
old <a href="./hdoc-filter/index.html">Filter API</a>
and a separate one for the <a href="./hdoc_arrow/index.html">Arrow API</a>.
The <a href="./hdoc/index.htm">complete API documentation</a> is still available.
</li>
<li>DTD processing for the arrow part is done completely by arrow based routines.
</li>
<li>Cabal config file and dependencies change to work with ghc 6.4.2.
</ul>
</li>
</ul>
<a name="limit"></a>
<h3>Known problems and limitations</h3>
<p>
The parser has been tested with the XML Validation Suite form the
W3C. The following problems have been encountered:
</p>
<ul>
<li>Line numbers in XML parser do not always point to the
correct position of the syntax error.</li>
<li>Line numbers are not yet reported for validation constraint
errors.
</li>
<li>The standalone document check is not yet implemented.</li>
<li>The XSLT module does not support the complete XSLT standard.</li>
</ul>
<a name="port"></a>
<h3>Portability</h3>
<p>
Portability to Windows based systems has not been tested very
intensively, but did work on an XP system with the Cygwin tools installed.
Development was done under Linux with GHC 6.10 with the -Wall
flag. No warnings were issued, when compiling the toolbox sources.
</p>
<a name="filter"></a>
<h3>HXT with Filters</h3>
<p>
For older applications using the filter functionality,
there is an extra package <a href="hxt-filter-8.2.0.tar.gz">hxt-filter</a>.
This package must be installed on top of hxt.
The filter package will not be actively developed any more.
Please move to the arrow version for long term projects.
Installation works with cabal in the usual way.
Download archive is <a href="hxt-filter-8.3.0.tar.gz">hxt-filter-8.3.0.tar.gz</a>,
<em><a href="./hdoc-filter/index.html">HXT Filter API Documentation</a></em>
with source links is availabe as well as a
darcs repository under <a href="http://darcs2.fh-wedel.de/repos/hxt-filter/">http://darcs2.fh-wedel.de/repos/hxt-filter</a>.
</p>
<a name="rel"></a>
<h3>Related work</h3>
<ul>
<li>Malcolm Wallace and Colin Runciman wrote
<a href="http://www.cs.york.ac.uk/fp/HaXml/">HaXml</a>,
a collection of utilities for using Haskell and XML together.
The <i>Haskell XML Toolbox</i> is based on their idea
of using filter combinators for processing XML with Haskell.
</li>
<li>Joe English wrote
<a href="http://www.flightlab.com/~joe/hxml/">HXML</a>
- a non-validating XML parser in Haskell.
His
<a href="http://www.flightlab.com/~joe/sgml/validate.html">idea</a>
of validating XML by using derivatives of regular expressions
was
implemented in the validation functions of this software.
Also his ideas and sources for navigateble trees are used
in the hxpath modules.
</li>
</ul>
<a name="feed"></a>
<h3>Feedback</h3>
<p>
We are interested in hearing your feedback
on our <i>Haskell XML Toolbox</i>, suggestions
for improvements, comments and criticisms.<br>
</p>
<p>
Mail address is
<a
href="mailto:hxmltoolbox@fh-wedel.de?subject=Haskell%20XML%20Toolbox%20Feedback">hxmltoolbox@fh-wedel.de</a>
</p>
<hr>
<table width="100%">
<tr>
<td>
The <i>Haskell XML Toolbox</i>
is distributed under the
<a href="http://www.opensource.org/licenses/mit-license.php"><b>MIT License</b></a>.
</td>
<td align="right">
<a href="http://validator.w3.org/check/referer">
<img border="0" src="http://www.w3.org/Icons/valid-html401" alt="Valid HTML 4.01!" height="31" width="88">
</a>
</td>
</tr>
<tr><td><i>Last modified: 2009-06-16</i></td><td> </td></tr>
</table>
</body>
</html>