diff --git a/README.md b/README.md new file mode 100644 --- /dev/null +++ b/README.md @@ -0,0 +1,97 @@ +# epub-metadata + + +## Synopsis + +Library for parsing epub document metadata (Haskell) + + +## Description + +Library for parsing and manipulating epub document metadata. Supports epub versions 2 and 3. + +This library was constructed by studying the IDPF specifications for epub documents found [here for version 2.x](http://www.idpf.org/epub/20/spec/OPF_2.0.1_draft.htm) and [here for version 3.x](http://www.idpf.org/epub/30/spec/epub30-publications.html) + + +### Why was this done? + +The motivation for this project grew out of my desire to take charge +of missing or incorrect epub metadata in books I have purchased. I +started out using the Calibre open source tools for examining this +info. Limitations and incomplete implementation of those tools led +me here to build a more complete implementation in the programming +language that I love beyond all others. + + +### Why didn't I just use existing solutions? + +#### Calibre ebook-meta utility + +I experienced various problems using this software, such as: + +* Incomplete and in some cases incorrect handling of tags that can + exist more than once (creator, contributor), particularly when they + are differentiated using attributes according to the spec. + +* Unable to display many fields in the OPF Package Document metadata + specification. Unable to manipulate data that is represented as + attributes of tags in the OPF spec. + +* Astonishingly slow performance. A command-line renaming tool built with + epub-metadata is more than 45 times faster at parsing and displaying + epub metadata. I'm going to blame Python here for Calibre's + performance. This has had a big impact on projects where I've been + processing hundreds of epubs in batch operations. + + +#### epub on Hackage, epub E-Book construction support library + +* The focus of this project seems to be with building new documents, + not parsing existing files. And there is a specific attempt to do more + than the metadata, to gather up the content and other metafiles that + make up an epub for creation. + +* Examining Codec.Ebook.OPF.Types, most of the metadata fields + from the OPF Package Document spec are missing or aren't modeled + thoroughly. I felt that to contribute to this project, I would have + had to significantly rip up the types and redesign them. + +* At this time it seemed like a better solution for me to + start fresh with modelling the types and code to manipulate them. That + said, I would be very interested in combining the epub and epub-metadata + projects at some point in some way that makes sense. + + +### Using this library + + Please see the [Haddock documentation](http://hackage.haskell.org/package/epub-metadata-4.0/docs/Codec-Epub.html) or source code for + [Codec.Epub](http://ui3.info/darcs/epub-metadata/src/Codec/Epub.hs) + + This file has a working example of using this library. + + +## Getting source + +- Download the cabalized source package [from Hackage](http://hackage.haskell.org/package/epub-metadata) +- Download the cabalized source tarball [from here](http://ui3.info/d/proj/epub-metadata/epub-metadata-4.2.tar.gz) +- Get the source with darcs: `$ darcs get http://ui3.info/darcs/epub-metadata` +- If you're just looking, [browse the source](http://ui3.info/darcs/epub-metadata) + +And once you have it, building the usual way: + +> $ cabal configure --enable-tests +> $ cabal build +> $ cabal test +> $ cabal haddock +> $ cabal install + + +## Installing + +Build and install with cabal-install: + `$ cabal update ; cabal install epub-metadata` + + +## Contact + +Dino Morelli <[dino@ui3.info](mailto:dino@ui3.info)> diff --git a/TODO b/TODO new file mode 100644 --- /dev/null +++ b/TODO @@ -0,0 +1,9 @@ +- May need to add a special function that will try to locate modified even in the metaDate field (which is what an epub2 might look like) + +- Need more API docs. Detail the differences between epub2 and epub3 in the data structures, where the fields come from and which are only one or the other. + +- Separate some of the Metadata sub-types into their own modules, starting with Title. We've got some utility functions now for this, will help to clean it up some more. + +- Better errors on parse failure, this will emerge along with more unit testing + +- More unit testing diff --git a/changelog.md b/changelog.md new file mode 100644 --- /dev/null +++ b/changelog.md @@ -0,0 +1,129 @@ +4.2 (2014-05-09) + + * Added many files missing from extra-source-files + + +4.1 (2014-05-04) + + * Fix for Simplifier ticks exhausted problem that was exposed + after upgrade to GHC 7.8.2 + * Added README.md and changelog.md files, better docs! + * Some documentation fixes + * Fixed a bug reading books where the container XML document is malformed + + +4.0 (2013-09-20) + + * Added support for epub3 documents. This was done using a single + set of datatypes, not specific to either epub2 or epub3. + * Redesigned the book examining API to be an edsl. Actions are to be + combined together based on what the developer needs from the document. + * The data structures to contain epub metadata "sections" were + redesigned to no longer be nested. Part of this change includes a + typeclass-based pretty-print API for displaying this data. + * Documentation rewrites and additions, including a working code + example in the API docs. + + +3.0 (2013-04-13) + + * Repackaged this code to show that it's for epub v2.x only at + this time + * Fixed a bug where mimetype file not occurring first in the archive + on Windows + * Fixed support for all 0-or-more epub2 metadata fields + * Redesigned the Format (metadata display) code + * Modularized the unit tests so that more tests could be added that + aren't specifically about parsing metadata. + * Cosmetic updates to the epub metadata API including some + documentation clean-up + + +2.3.2 (2013-02-01) + + * Added code to strip illegal characters from the beginning of the + OPF metadata XML file + * Switched the encoding hack to be case-insensitive + + +2.3.1 (2012-10-31) + + * Added a fix for epub zip files with a damaged central directory + signature + + +2.3.0 (2012-01-29) + + * Encountered a problem in Windows where using a lazy ByteString + to read the entire epub zip file was never closing the file while + consumer applications were still running. Changed this library to + use a strict ByteString instead for reading the zip archive. + + +2.2.0.1 (2011-10-27) + + * Changed display output of OPF data to label the textual content of + a tag as "text" + * Extensive changes to the cabal build of this project to bring it + up to Cabal 1.10 + * Unit tests now use the test-suite cabal stanza + + +2.2.0.0 (2011-04-20) + + * Now includes some API functions for working directly with zip + archives and directories full of the loose files which will be made + into epubs + * Moved the epubmeta utility from this project. It's now in epub-tools + + +2.1.0 (2011-02-13) + + * Project now uses zip-archive instead of the libzip + library. zip-archive is a pure Haskell solution that's easier to + build on non-UNIX-like platforms + * Other changes to the build and project to make it buildable under + Windows + + +2.0.2 (2011-01-10) + + * This library now uses the haskell LibZip library instead of relying + on unzip shell command invocation + * Changed command-line utility name from epub-meta to epubmeta + + +2.0.1 (2011-01-08) + + * Now always using namespace qualification for all OPF Package + tags. Library would incorrectly fail if the OPF document was created + using explicit namespacing for the OPF package elements. + * Hack to avoid problems with UTF-8 encoding in OPF Package XML + documents + * Hack to avoid problems with !DOCTYPE declarations. Found some + books where a DTD was specified for a completely different type of + document. Discarding these declarations for now. + * Changed this code to use System.Process instead of HSH for unzip + invocation in order to have more control over interpreting exit + codes. + + +2.0.0 (2011-01-02) + + * Added parsing and output of the manifest, spine and guide portions + of the spec. Full read capability of OPF Package data is now + complete. + * Major redesign of the API. It's been split and organized into + smaller, more focused modules. + * Updated to work with the 9.x version of hxt + * Redesigned epub metadata formatting code to be much more efficient + using Writer monad and Data.Sequence + * Added support for new capabilities to command-line utility + * Clarified the Haddock documentation to describe how these data + types map to the source XML data + * Updated unit tests to reflect the large API changes + + +1.0.2 (2010-05-11) + + * Initial release diff --git a/doc/dev/notes b/doc/dev/notes new file mode 100644 --- /dev/null +++ b/doc/dev/notes @@ -0,0 +1,39 @@ +epub3 epub2 + metadata metadata ++ identifier + identifier +? id ? id +?m identifier-type +?m scheme ? scheme ++ title + title +? lang ? lang +?m title-type +?m display-seq ++ language + language +* contributor * contributor +?m role ? role +?m file-as ? file-as +?m display-seq +* creator * creator +?m role ? role +?m file-as ? file-as +?m display-seq +? date * date + [simpler than it used to be 1] ? event + m modified [this one is special 2] +? source * source +? type * type +* coverage * coverage +* description * description +* format * format +* publisher * publisher +* relation * relation +* rights * rights +* subject * subject + +[1] date is simpler than it used to be, I believe it means "when this book was originally written or created", and looks like this: + + 2012 + +[2] modified metadata looks like this: + + 2011-01-01T12:00:00Z diff --git a/doc/hcar/epubmetadata-De.tex b/doc/hcar/epubmetadata-De.tex new file mode 100644 --- /dev/null +++ b/doc/hcar/epubmetadata-De.tex @@ -0,0 +1,26 @@ +% epubmetadata-De.tex +\begin{hcarentry}{epub-metadata} +\report{Dino Morelli}%05/11 +\status{experimental, actively developed} +\makeheader + +Library for parsing and manipulating epub OPF package data. Now with epub3 support. + +* Added support for epub3 documents. This was done using a single set of datatypes, not specific to either epub2 or epub3. +* Redesigned the book file querying API to be an edsl. Actions are to be combined together based on what the developer needs from the document. +* Data structures to contain epub metadata "sections" were redesigned to no longer be nested. Part of this change includes a typeclass-based pretty-print API for displaying this data. +* Documentation rewrites and additions, including a working code example in the API docs. + +epub-metadata is available from Hackage and the Darcs repository below. + +See also epub-tools \cref{epubtools}. + +\FurtherReading +\begin{compactitem} +\item Project page: +\url{http://ui3.info/d/proj/epub-metadata.html} + +\item Source repository: +\texttt{darcs get} \url{http://ui3.info/darcs/epub-metadata} +\end{compactitem} +\end{hcarentry} diff --git a/epub-metadata.cabal b/epub-metadata.cabal --- a/epub-metadata.cabal +++ b/epub-metadata.cabal @@ -1,5 +1,5 @@ name: epub-metadata -version: 4.1 +version: 4.2 cabal-version: >= 1.8 build-type: Simple license: BSD3 @@ -15,9 +15,13 @@ This library was constructed by studying the IDPF specifications for epub documents found here and here category: Codec, Text tested-with: GHC >= 7.6.2 -extra-source-files: doc/examples/README +extra-source-files: changelog.md + doc/dev/notes + doc/examples/README doc/examples/*.cabal doc/examples/*.hs + doc/hcar/epubmetadata-De.tex + README.md testsuite/*.epub testsuite/*.hs testsuite/*.opf @@ -26,6 +30,10 @@ testsuite/bookfiles/content/bar testsuite/Epub2/*.hs testsuite/Epub3/*.hs + TODO + util/gentags.sh + util/prefs/boring + util/show-opf.hs source-repository head type: darcs diff --git a/util/gentags.sh b/util/gentags.sh new file mode 100644 --- /dev/null +++ b/util/gentags.sh @@ -0,0 +1,3 @@ +#! /bin/sh + +find src -regex '.*\..?hs' | xargs hasktags -c diff --git a/util/prefs/boring b/util/prefs/boring new file mode 100644 --- /dev/null +++ b/util/prefs/boring @@ -0,0 +1,117 @@ +# Boring file regexps: + +### compiler and interpreter intermediate files +# haskell (ghc) interfaces +\.hi$ +\.hi-boot$ +\.o-boot$ +# object files +\.o$ +\.o\.cmd$ +# profiling haskell +\.p_hi$ +\.p_o$ +# haskell program coverage resp. profiling info +\.tix$ +\.prof$ +# fortran module files +\.mod$ +# linux kernel +\.ko\.cmd$ +\.mod\.c$ +(^|/)\.tmp_versions($|/) +# *.ko files aren't boring by default because they might +# be Korean translations rather than kernel modules +# \.ko$ +# python, emacs, java byte code +\.py[co]$ +\.elc$ +\.class$ +# objects and libraries; lo and la are libtool things +\.(obj|a|exe|so|lo|la)$ +# compiled zsh configuration files +\.zwc$ +# Common LISP output files for CLISP and CMUCL +\.(fas|fasl|sparcf|x86f)$ + +### build and packaging systems +# cabal intermediates +\.installed-pkg-config +\.setup-config +# standard cabal build dir, might not be boring for everybody +# ^dist(/|$) +# autotools +(^|/)autom4te\.cache($|/) +(^|/)config\.(log|status)$ +# microsoft web expression, visual studio metadata directories +\_vti_cnf$ +\_vti_pvt$ +# gentoo tools +\.revdep-rebuild.* +# generated dependencies +^\.depend$ + +### version control systems +# cvs +(^|/)CVS($|/) +\.cvsignore$ +# cvs, emacs locks +^\.# +# rcs +(^|/)RCS($|/) +,v$ +# subversion +(^|/)\.svn($|/) +# mercurial +(^|/)\.hg($|/) +# git +(^|/)\.git($|/) +# bzr +\.bzr$ +# sccs +(^|/)SCCS($|/) +# darcs +(^|/)_darcs($|/) +(^|/)\.darcsrepo($|/) +^\.darcs-temp-mail$ +-darcs-backup[[:digit:]]+$ +# gnu arch +(^|/)(\+|,) +(^|/)vssver\.scc$ +\.swp$ +(^|/)MT($|/) +(^|/)\{arch\}($|/) +(^|/).arch-ids($|/) +# bitkeeper +(^|/)BitKeeper($|/) +(^|/)ChangeSet($|/) + +### miscellaneous +# backup files +~$ +\.bak$ +\.BAK$ +# patch originals and rejects +\.orig$ +\.rej$ +# X server +\..serverauth.* +# image spam +\# +(^|/)Thumbs\.db$ +# vi, emacs tags +(^|/)(tags|TAGS)$ +#(^|/)\.[^/] +# core dumps +(^|/|\.)core$ +# partial broken files (KIO copy operations) +\.part$ +# waf files, see http://code.google.com/p/waf/ +(^|/)\.waf-[[:digit:].]+-[[:digit:]]+($|/) +(^|/)\.lock-wscript$ +# mac os finder +(^|/)\.DS_Store$ +# cabal +(^|/)dist($|/) + +(^|/)bin($|/) diff --git a/util/show-opf.hs b/util/show-opf.hs new file mode 100644 --- /dev/null +++ b/util/show-opf.hs @@ -0,0 +1,17 @@ +#! /usr/bin/runhaskell -isrc + +-- Copyright: 2010-2013 Dino Morelli +-- License: BSD3 (see LICENSE) +-- Author: Dino Morelli + +import Control.Monad.Error ( runErrorT ) +import System.Environment ( getArgs ) + +import Codec.Epub.Opf.Parse ( parseXmlToOpf ) + + +main :: IO () +main = do + opfXmlContents <- (fmap head) getArgs >>= readFile + parseResult <- runErrorT $ parseXmlToOpf opfXmlContents + either print print parseResult