mnist-idx (empty) → 0.1.2.3
raw patch · 6 files changed
+677/−0 lines, 6 filesdep +basedep +binarydep +bytestringsetup-changed
Dependencies added: base, binary, bytestring, directory, hspec, mnist-idx, vector
Files
- LICENSE +166/−0
- Setup.hs +2/−0
- mnist-idx.cabal +82/−0
- src/Data/IDX.hs +264/−0
- src/Data/IDX/Internal.hs +103/−0
- test/Main.hs +60/−0
+ LICENSE view
@@ -0,0 +1,166 @@+ GNU LESSER GENERAL PUBLIC LICENSE+ Version 3, 29 June 2007++Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>+Everyone is permitted to copy and distribute verbatim copies+of this license document, but changing it is not allowed.+++ This version of the GNU Lesser General Public License incorporates+ the terms and conditions of version 3 of the GNU General Public+ License, supplemented by the additional permissions listed below.++ 0. Additional Definitions.++ As used herein, "this License" refers to version 3 of the GNU Lesser+ General Public License, and the "GNU GPL" refers to version 3 of the GNU+ General Public License.++ "The Library" refers to a covered work governed by this License,+ other than an Application or a Combined Work as defined below.++ An "Application" is any work that makes use of an interface provided+ by the Library, but which is not otherwise based on the Library.+ Defining a subclass of a class defined by the Library is deemed a mode+ of using an interface provided by the Library.++ A "Combined Work" is a work produced by combining or linking an+ Application with the Library. The particular version of the Library+ with which the Combined Work was made is also called the "Linked+ Version".++ The "Minimal Corresponding Source" for a Combined Work means the+ Corresponding Source for the Combined Work, excluding any source code+ for portions of the Combined Work that, considered in isolation, are+ based on the Application, and not on the Linked Version.++ The "Corresponding Application Code" for a Combined Work means the+ object code and/or source code for the Application, including any data+ and utility programs needed for reproducing the Combined Work from the+ Application, but excluding the System Libraries of the Combined Work.++ 1. Exception to Section 3 of the GNU GPL.++ You may convey a covered work under sections 3 and 4 of this License+ without being bound by section 3 of the GNU GPL.++ 2. Conveying Modified Versions.++ If you modify a copy of the Library, and, in your modifications, a+ facility refers to a function or data to be supplied by an Application+ that uses the facility (other than as an argument passed when the+ facility is invoked), then you may convey a copy of the modified+ version:++ a) under this License, provided that you make a good faith effort to+ ensure that, in the event an Application does not supply the+ function or data, the facility still operates, and performs+ whatever part of its purpose remains meaningful, or++ b) under the GNU GPL, with none of the additional permissions of+ this License applicable to that copy.++ 3. Object Code Incorporating Material from Library Header Files.++ The object code form of an Application may incorporate material from+ a header file that is part of the Library. You may convey such object+ code under terms of your choice, provided that, if the incorporated+ material is not limited to numerical parameters, data structure+ layouts and accessors, or small macros, inline functions and templates+ (ten or fewer lines in length), you do both of the following:++ a) Give prominent notice with each copy of the object code that the+ Library is used in it and that the Library and its use are+ covered by this License.++ b) Accompany the object code with a copy of the GNU GPL and this license+ document.++ 4. Combined Works.++ You may convey a Combined Work under terms of your choice that,+ taken together, effectively do not restrict modification of the+ portions of the Library contained in the Combined Work and reverse+ engineering for debugging such modifications, if you also do each of+ the following:++ a) Give prominent notice with each copy of the Combined Work that+ the Library is used in it and that the Library and its use are+ covered by this License.++ b) Accompany the Combined Work with a copy of the GNU GPL and this license+ document.++ c) For a Combined Work that displays copyright notices during+ execution, include the copyright notice for the Library among+ these notices, as well as a reference directing the user to the+ copies of the GNU GPL and this license document.++ d) Do one of the following:++ 0) Convey the Minimal Corresponding Source under the terms of this+ License, and the Corresponding Application Code in a form+ suitable for, and under terms that permit, the user to+ recombine or relink the Application with a modified version of+ the Linked Version to produce a modified Combined Work, in the+ manner specified by section 6 of the GNU GPL for conveying+ Corresponding Source.++ 1) Use a suitable shared library mechanism for linking with the+ Library. A suitable mechanism is one that (a) uses at run time+ a copy of the Library already present on the user's computer+ system, and (b) will operate properly with a modified version+ of the Library that is interface-compatible with the Linked+ Version.++ e) Provide Installation Information, but only if you would otherwise+ be required to provide such information under section 6 of the+ GNU GPL, and only to the extent that such information is+ necessary to install and execute a modified version of the+ Combined Work produced by recombining or relinking the+ Application with a modified version of the Linked Version. (If+ you use option 4d0, the Installation Information must accompany+ the Minimal Corresponding Source and Corresponding Application+ Code. If you use option 4d1, you must provide the Installation+ Information in the manner specified by section 6 of the GNU GPL+ for conveying Corresponding Source.)++ 5. Combined Libraries.++ You may place library facilities that are a work based on the+ Library side by side in a single library together with other library+ facilities that are not Applications and are not covered by this+ License, and convey such a combined library under terms of your+ choice, if you do both of the following:++ a) Accompany the combined library with a copy of the same work based+ on the Library, uncombined with any other library facilities,+ conveyed under the terms of this License.++ b) Give prominent notice with the combined library that part of it+ is a work based on the Library, and explaining where to find the+ accompanying uncombined form of the same work.++ 6. Revised Versions of the GNU Lesser General Public License.++ The Free Software Foundation may publish revised and/or new versions+ of the GNU Lesser General Public License from time to time. Such new+ versions will be similar in spirit to the present version, but may+ differ in detail to address new problems or concerns.++ Each version is given a distinguishing version number. If the+ Library as you received it specifies that a certain numbered version+ of the GNU Lesser General Public License "or any later version"+ applies to it, you have the option of following the terms and+ conditions either of that published version or of any later version+ published by the Free Software Foundation. If the Library as you+ received it does not specify a version number of the GNU Lesser+ General Public License, you may choose any version of the GNU Lesser+ General Public License ever published by the Free Software Foundation.++ If the Library as you received it specifies that a proxy can decide+ whether future versions of the GNU Lesser General Public License shall+ apply, that proxy's public statement of acceptance of any version is+ permanent authorization for you to choose that version for the+ Library.+
+ Setup.hs view
@@ -0,0 +1,2 @@+import Distribution.Simple+main = defaultMain
+ mnist-idx.cabal view
@@ -0,0 +1,82 @@+name: mnist-idx++version: 0.1.2.3++-- A short (one-line) description of the package.+synopsis: Read and write IDX data that is used in e.g. the MNIST database.++-- A longer description of the package.+description: This package provides functionality to read and write data in the IDX+ binary format. This format is relevant for machine learning applications,+ like the MNIST handwritten digit database.++-- URL for the project homepage or repository.+homepage: https://github.com/kryoxide/mnist-idx/++-- The license under which the package is released.+license: LGPL-3++-- The file containing the license text.+license-file: LICENSE++-- The package author(s).+author: Christof Schramm++-- An email address to which users can send suggestions, bug reports, and +-- patches.+maintainer: christof.schramm@campus.lmu.de++-- A copyright notice.+-- copyright: ++category: Data++build-type: Simple++-- Extra files to be distributed with the package, such as examples or a +-- README.+-- extra-source-files: ++-- Constraint on the version of Cabal needed to build this package.+cabal-version: >= 1.16++source-repository head+ type: git+ location: https://github.com/kryoxide/mnist-idx/++library+ -- Modules exported by the library.+ exposed-modules: Data.IDX+ Data.IDX.Internal+ + -- Modules included in this library but not exported.+ -- other-modules:+ + -- LANGUAGE extensions used by modules in this package.+ -- other-extensions: + + -- Other library packages from which modules are imported.+ build-depends: base >=4.6 && <4.9,+ binary >= 0.7 && < 0.8,+ vector >= 0.10 && < 0.11,+ bytestring >= 0.10 && < 0.11 + -- Directories containing source files.+ hs-source-dirs: src+ + -- Base language which the package is written in.+ default-language: Haskell2010++test-suite tests++ type: exitcode-stdio-1.0++ hs-source-dirs: test++ main-is: Main.hs++ build-depends: base >= 4.6+ , hspec >= 1.9+ , vector >= 0.10 && < 0.11+ , binary >= 0.7 && < 0.8+ , directory >= 1.2 && < 1.3+ , mnist-idx
+ src/Data/IDX.hs view
@@ -0,0 +1,264 @@+--------------------------------------------------------------------------------+-- |+-- Module : Data.IDX+-- Copyright : Christof Schramm+-- License : GPL v 3+--+-- Maintainer : Christof Schramm <christof.schramm@campus.lmu.de>+-- Stability : Experimental+-- Portability : Should work in all common Haskell implementations+--+-- A package for reading and writing data in the IDX format.+-- This data format is used for machine-learning data sets like the+-- MINST database of handwritten digits (<http://yann.lecun.com/exdb/mnist/>)+--------------------------------------------------------------------------------+module Data.IDX (+ -- * Data types+ IDXData+ , IDXLabels+ , IDXContentType(..)++ -- * Accessing data+ , idxType+ , idxDimensions++ , isIDXReal+ , isIDXIntegral++ -- * Raw data+ , idxDoubleContent+ , idxIntContent++ -- * Labeled data+ , labeledIntData+ , labeledDoubleData++ -- * IO / Serialization++ -- ** IDXLabels+ + -- *** ByteString serialization+ , encodeIDXLabels+ , decodeIDXLabels++ -- *** FileIO+ , encodeIDXLabelsFile+ , decodeIDXLabelsFile+ + -- ** IDXData (e.g. images)+ + -- *** ByteString serialization+ , encodeIDX+ , decodeIDX++ -- *** File IO+ , encodeIDXFile+ , decodeIDXFile+ )where++-- For compatibility with versions of base < 4.8+import Control.Applicative ((<$>))+import Control.Monad++import Data.Binary+import Data.ByteString.Lazy (ByteString)+import qualified Data.ByteString.Lazy as BL+import Data.IDX.Internal+import Data.Int+import Data.Traversable+import qualified Data.Vector.Unboxed as V+import Data.Vector.Unboxed ((!))+import Data.Word++import Debug.Trace++instance Binary IDXContentType where+ get = do+ w <- getWord8+ case w of+ 0x08 -> return IDXUnsignedByte+ 0x09 -> return IDXSignedByte+ 0x0B -> return IDXShort+ 0x0C -> return IDXInt+ 0x0D -> return IDXFloat+ 0x0E -> return IDXDouble+ _ -> fail $ "Unrecognized IDX content type: " ++ (show w)+ + put IDXUnsignedByte = putWord8 0x08+ put IDXSignedByte = putWord8 0x09+ put IDXShort = putWord8 0x0B+ put IDXInt = putWord8 0x0C+ put IDXFloat = putWord8 0x0D+ put IDXDouble = putWord8 0x0E++instance Binary IDXLabels where+ get = do+ getInt32+ nItems <- fromIntegral <$> getInt32+ let readEntries n = V.replicateM n $ fromIntegral <$> getWord8 >>= (return $!)+ v <- readContent readEntries 500 nItems+ return $ IDXLabels v++ put (IDXLabels v) = do+ put (0 :: Int32)+ let len = V.length v+ put (fromIntegral len :: Int32)+ V.forM_ v (\x -> put $! (fromIntegral x :: Word8))++-- | Partition a dataset and label each subpartition, return int values+labeledIntData:: IDXLabels -> IDXData -> Maybe [(Int, V.Vector Int)]+labeledIntData (IDXLabels v) dat =+ if (V.length v) == dim0+ then Just $ do+ i <- [0 .. dim0 - 1]+ let lab = v ! i+ return $ (lab,V.slice (i*entrySize) entrySize content)+ else Nothing+ where+ dim0 = (idxDimensions dat) ! 0+ content = idxIntContent dat+ entrySize = (V.product $ idxDimensions dat) `div` dim0++-- | Partition a dataset and label each subpartition, return double values+labeledDoubleData:: IDXLabels -> IDXData -> Maybe [(Int, V.Vector Double)]+labeledDoubleData (IDXLabels v) dat =+ if (V.length v) == dim0+ then Just $ do+ i <- [0 .. dim0 - 1]+ let lab = v ! i+ return $ (lab,V.slice (i*entrySize) entrySize content)+ else Nothing+ where+ dim0 = (idxDimensions dat) ! 0+ content = idxDoubleContent dat+ entrySize = (V.product $ idxDimensions dat) `div` dim0++instance Binary IDXData where+ get = do+ -- Get header information (4 bytes total)+ getWord8+ getWord8 + idxType <- get :: Get IDXContentType+ nDimensions <- fromIntegral <$> getWord8++ -- Each dimension size is encoded as a 32 bit integer+ dimensionSizes <- replicateM nDimensions (fromIntegral <$> getInt32) + let nEntries = fromIntegral $ product dimensionSizes+ dimV = V.fromList dimensionSizes++ -- Retrieve the data, depending on the type specified in the file+ -- Cast all integral types to Int and all decimal numbers tod double+ case idxType of+ t@IDXUnsignedByte -> buildIntResult nEntries t dimV getWord8+ t@IDXSignedByte -> buildIntResult nEntries t dimV getInt8+ t@IDXShort -> buildIntResult nEntries t dimV getInt16+ t@IDXInt -> buildIntResult nEntries t dimV getInt32++ t@IDXFloat -> buildDoubleResult nEntries t dimV getFloat+ t@IDXDouble -> buildDoubleResult nEntries t dimV getDouble+ + put d = do+ -- First four bytes are meta information+ putWord8 0+ putWord8 0+ -- Third byte is content type+ put $ idxType d+ + -- Fourth byte is number of dimensions+ let dimensions = idxDimensions d+ put $ (fromIntegral $ V.length dimensions :: Word8)++ -- Put size of each dimension as an Int32+ V.forM_ dimensions $ (\x -> put $! (fromIntegral x :: Int32))++ -- Put the individual values+ case d of+ IDXDoubles t _ content -> V.forM_ content $ putReal t+ IDXInts t _ content -> V.forM_ content $ putIntegral t++-- | Helper function for parsing integer data from the+-- IDX content. Returns a full IDX result.+buildIntResult :: Integral a+ => Int -- ^ Expected number of entries+ -> IDXContentType -- ^ Description of content+ -> V.Vector Int -- ^ Dimension sizes+ -> Get a -- ^ Monadic action to get content element+ -> Get IDXData+buildIntResult nEntries typ dimV getContent = do+ content <- readContent readEntries 500 nEntries+ return $ IDXInts typ dimV content+ where+ readEntries n = V.replicateM n $ fromIntegral <$> getContent >>= (return $!)++-- | Helper function for parsing real number data from+-- the IDX content.+buildDoubleResult :: Real a+ => Int -- ^ Expected number of entries+ -> IDXContentType -- ^ Description of content+ -> V.Vector Int -- ^ Dimension sizes+ -> Get a -- ^ Monadic action to get content element+ -> Get IDXData+buildDoubleResult nEntries typ dimV getContent = do+ content <- readContent readEntries 500 nEntries+ return $ IDXDoubles typ dimV content+ where+ readEntries n = V.replicateM n $ realToFrac <$> getContent >>= (return $!)++-- | Put values that are saved as Int+putIntegral :: IDXContentType -> Int -> Put+putIntegral IDXUnsignedByte n = put $! (fromIntegral n :: Word8)+putIntegral IDXSignedByte n = put $! (fromIntegral n :: Int8 )+putIntegral IDXShort n = put $! (fromIntegral n :: Int16)+putIntegral IDXInt n = put $! (fromIntegral n :: Int32)+putIntegral t _ = error $ "IDX.putIntegral " ++ show t ++-- | Put real values that are saved as Double+putReal :: IDXContentType -> Double -> Put+putReal IDXDouble n = put n+putReal IDXFloat n = put $! (realToFrac n :: Float )++-- Haskell's Data.Binary uses big-endian format+getInt8 :: Get Int8+getInt8 = get++getInt16 :: Get Int16+getInt16 = get++getInt32 :: Get Int32+getInt32 = get++getFloat :: Get Float+getFloat = get++getDouble :: Get Double+getDouble = get++-- | Read labels from a file, return 'Nothing' if something doesn't work+decodeIDXLabelsFile :: FilePath -> IO (Maybe IDXLabels)+decodeIDXLabelsFile path = BL.readFile path >>= return . decodeIDXLabels++decodeIDXLabels :: BL.ByteString -> Maybe IDXLabels+decodeIDXLabels content = case decodeOrFail content of+ Right (_,_,result) -> Just result+ Left _ -> Nothing++-- | Read data from a file, return 'Nothing' if something doesn't work+encodeIDXLabelsFile :: IDXLabels -> FilePath -> IO ()+encodeIDXLabelsFile labs path = encodeFile path labs++encodeIDXLabels :: IDXLabels -> BL.ByteString+encodeIDXLabels = encode++decodeIDXFile :: FilePath -> IO (Maybe IDXData)+decodeIDXFile path = BL.readFile path >>= return . decodeIDX++decodeIDX :: BL.ByteString -> Maybe IDXData+decodeIDX content = case decodeOrFail content of+ Right (_,_,result) -> Just result+ Left _ -> Nothing++encodeIDXFile :: IDXData -> FilePath -> IO ()+encodeIDXFile idx path = encodeFile path idx++encodeIDX :: IDXData -> BL.ByteString+encodeIDX = encode
+ src/Data/IDX/Internal.hs view
@@ -0,0 +1,103 @@+{-# LANGUAGE GADTs #-}+--------------------------------------------------------------------------------+-- |+-- Module: Data.IDX.Internal+-- Copyright: Christof Schramm+-- License: GPLv3+--+-- Maintainer: Christof Schramm <christof.schramm@campus.lmu.de>+-- Stability: Experimental+-- Portability: Shoud work with all major haskell implementations+--+-- An internal package, the API contained here should not be used+-- and is subject to possibly breaking changes. Use these functions+-- and types at your own risk.+--+-- The safe interface is in 'Data.IDX'+--------------------------------------------------------------------------------+module Data.IDX.Internal where++import Control.Monad (replicateM)+import Data.Binary++import qualified Data.Vector.Unboxed as V+import Data.Vector.Unboxed ((!))++-- | A type to describe the content, according to IDX spec+data IDXContentType where+ IDXUnsignedByte :: IDXContentType+ IDXSignedByte :: IDXContentType+ IDXShort :: IDXContentType+ IDXInt :: IDXContentType+ IDXFloat :: IDXContentType+ IDXDouble :: IDXContentType+ deriving Show+++-- | Datatype for storing IDXData. Internally data is always stored either+-- as 'Int' or 'Double' unboxed vectors. However when binary serialization+-- is used, the data is serialized according to the 'IDXContentType'.+data IDXData = IDXInts IDXContentType (V.Vector Int) (V.Vector Int )+ | IDXDoubles IDXContentType (V.Vector Int) (V.Vector Double)+ deriving Show++newtype IDXLabels = IDXLabels (V.Vector Int)+++-- | Return the what type the data is stored in+idxType :: IDXData -> IDXContentType+idxType (IDXInts t _ _) = t+idxType (IDXDoubles t _ _) = t++-- | Return an unboxed Vector of Int dimensions+idxDimensions :: IDXData -> V.Vector Int+idxDimensions (IDXInts _ ds _) = ds+idxDimensions (IDXDoubles _ ds _) = ds++-- | Return wether the data in this IDXData value is+-- stored as integral values+isIDXIntegral :: IDXData -> Bool+isIDXIntegral (IDXInts _ _ _) = True+isIDXIntegral (_ ) = False++-- | Return wether the data in this IDXData value is+-- stored as double values+isIDXReal :: IDXData -> Bool+isIDXReal (IDXDoubles _ _ _) = True+isIDXReal (_ ) = False++-- | Return contained ints, if no ints are contained,+-- convert content to ints by using 'round'. Data is stored like+-- in a C-array, i.e. the last index changes first.+idxIntContent :: IDXData -> V.Vector Int+idxIntContent (IDXInts _ _ v) = v+idxIntContent (IDXDoubles _ _ v) =+ V.fromList $ [round $ (v ! i) | i <- [0.. ((V.length v)-1)]]++-- | Return contained doubles, if no doubles are contained+-- convert the content to double by using 'fromIntegral'. Data is stored like+-- in a C-array, i.e. the last index changes first.+idxDoubleContent :: IDXData -> V.Vector Double+idxDoubleContent (IDXDoubles _ _ v) = v+idxDoubleContent (IDXInts _ _ v) =+ V.fromList $ [fromIntegral $ (v ! i) | i <- [0.. ((V.length v) - 1)]]++-- | Helper function to read a (possibly big) vector of binary+-- values as chunks. Strictly evaluates each chunk and then+-- concatenates the chunks, does not leak space.+readContent :: (V.Unbox a)+ => (Int -> Get (V.Vector a)) -- ^ To Get a chunk of size n+ -> Int -- ^ Chunk size+ -> Int -- ^ Expected input+ -> Get (V.Vector a) +readContent readEntries chunkSize n =+ if n > chunkSize+ then do+ headChunk <- readEntries (n `mod` chunkSize)+ let nChunks = n `div` chunkSize+ chunkList <- replicateM nChunks (readContent readEntries chunkSize chunkSize)+ return $! V.concat $ headChunk:chunkList+ else do+ rest <- readEntries n+ return $! rest+
+ test/Main.hs view
@@ -0,0 +1,60 @@+module Main where++import Test.Hspec++import Data.IDX+import Data.IDX.Internal++import qualified Data.Vector.Unboxed as V++import System.Directory+import System.IO++dataList = [1,2,3,4]++testData = IDXInts IDXUnsignedByte dims values+ where+ dims = V.fromList [2,2]+ values = V.fromList dataList++testLabels = IDXLabels $ V.fromList [0,1]++spec :: Spec+spec = do+ describe "IDX dataset" $ do+ it "should be created and deserialized correctly" $ do++ -- Get temporary directory+ tempdir <- getTemporaryDirectory++ -- Open a few temp files, immediately close the file handles+ -- (we don't need them)+ (imgPath,imgH) <- openTempFile tempdir "idx_test_img"+ hClose imgH++ (labPath,labH) <- openTempFile tempdir "idx_test_lab"+ hClose labH++ -- Save our test data+ encodeIDXFile testData imgPath+ encodeIDXLabelsFile testLabels labPath++ -- Read it again+ Just idata <- decodeIDXFile imgPath+ Just ils@(IDXLabels ilabs) <- decodeIDXLabelsFile labPath++ -- Remove the temp files+ removeFile imgPath+ removeFile labPath++ -- Label the image data with the label data+ let Just lst = labeledIntData ils idata++ -- See if everything matches up+ V.length ilabs `shouldBe` 2+ length lst `shouldBe` 2+ (fst $ head $ tail lst) `shouldBe` 1+ (V.toList $ idxIntContent idata) `shouldBe` dataList++main :: IO ()+main = hspec spec