diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,18 @@ +# Changelog + +## 0.2.3 2026-06-08 + +Initial release of `bytestring-mmap-compat`, forked from +`bytestring-mmap-0.2.2` on Hackage. + + - Adds `System.Posix.IO.Compat` shim so the package compiles + against `unix >= 2.8` (whose `openFd` lost the `Maybe FileMode` + argument). + - Widens the `unix` bound to `>= 2.7 && < 2.9`. + - Modernises the cabal file: `cabal-version: 2.0`, explicit + `default-language: Haskell2010`, explicit base / bytestring + bounds. + - Public API unchanged from upstream 0.2.2. + - Dropped `-O2` from `ghc-options` (a per-package `-O2` is rarely + a real win on a thin FFI wrapper and burns compile time on + every downstream user; the hot path is in `cbits/`). diff --git a/LICENSE b/LICENSE new file mode 100644 --- /dev/null +++ b/LICENSE @@ -0,0 +1,27 @@ +Copyright (c) Don Stewart + +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: +1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. +2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. +3. Neither the name of the author nor the names of his contributors + may be used to endorse or promote products derived from this software + without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE AUTHORS ``AS IS'' AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY +OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +SUCH DAMAGE. diff --git a/README.md b/README.md new file mode 100644 --- /dev/null +++ b/README.md @@ -0,0 +1,39 @@ +# bytestring-mmap-compat + +A fork of [`bytestring-mmap`](https://hackage.haskell.org/package/bytestring-mmap) +that compiles against `unix >= 2.8`. + +## Why this exists + +The original `bytestring-mmap` (Don Stewart, last Hackage upload in +2011) calls `System.Posix.openFd` with the pre-2.8 four-argument +signature. Upstream is unmaintained, and the latest Hackage revision +caps the `unix` bound at `< 2.8`, which leaves the package unusable +in any package set that pulls in a newer `unix` transitively. + +This fork: + + - Adds a tiny `System.Posix.IO.Compat` shim that selects the right + `openFd` signature via CPP. + - Widens the `unix` bound to cover both old and new signatures. + - Modernises the cabal file (cabal-version 2.0, sensible base / + bytestring bounds). + +The public API (`System.IO.Posix.MMap`, +`System.IO.Posix.MMap.Lazy`) is byte-for-byte compatible with the +upstream; the only difference is the package name. + +## When to switch back to upstream + +When (or if) upstream adopts a compatible `openFd` call site, switch +your `build-depends` from `bytestring-mmap-compat` to `bytestring-mmap` +and remove this dependency. No source-level changes required. + +## Credit + +All non-trivial code is by Don Stewart. This fork only adds the +`Compat` shim and updates packaging metadata. + +## License + +BSD3; see `LICENSE`. diff --git a/Setup.lhs b/Setup.lhs new file mode 100644 --- /dev/null +++ b/Setup.lhs @@ -0,0 +1,3 @@ +#!/usr/bin/env runhaskell +> import Distribution.Simple +> main = defaultMain diff --git a/System/IO/Posix/MMap.hs b/System/IO/Posix/MMap.hs new file mode 100644 --- /dev/null +++ b/System/IO/Posix/MMap.hs @@ -0,0 +1,114 @@ +{-# LANGUAGE ForeignFunctionInterface #-} +-------------------------------------------------------------------- +-- | +-- Module : System.IO.Posix.MMap +-- Copyright : (c) Galois, Inc. 2007 +-- License : BSD3 +-- +-- Maintainer: Don Stewart +-- Stability : provisional +-- Portability: non-portable -- posix only +-- +-- mmap a file or device into memory as a strict ByteString. +-- +module System.IO.Posix.MMap ( + + -- $mmap_intro + -- $mmap_unmap + + -- * Memory mapped files + unsafeMMapFile -- :: FilePath -> IO ByteString + +-- $mmap_intro +-- +-- 'unsafeMMapFile' mmaps a file or device into memory as a strict +-- 'ByteString'. The file is not actually copied strictly into memory, +-- but instead pages from the file will be loaded into the address +-- space on demand. +-- +-- We can consider mmap as lazy IO pushed into the virtual memory +-- subsystem. +-- +-- The file is mapped using MAP_SHARED: modifications to the file +-- will be immediately shared with any other process accessing the +-- file. This has no effect from the Haskell point of view, since +-- ByteStrings are treated as immutable values. +-- +-- However, if the file is written to by any other process on the +-- system while it is in use in Haskell, those changes will be +-- immediately reflected on the Haskell side, destroying referential +-- transparency. +-- +-- It is only safe to mmap a file if you know you are the sole user. +-- +-- For more details about mmap, and its consequences, see: +-- +-- * +-- +-- * +-- + +-- $mmap_unmap +-- +-- When the entire file is out of scope, the Haskell storage manager +-- will call munmap to free the file, using a finaliser. Until then, as +-- much of the file as you access will be allocated. +-- +-- Note that the Haskell storage manager doesn't know how large a +-- resource is associated with an mmapped file. If you allocate many +-- such files, the garbage collector will only see the 'ForeignPtr's +-- that have been allocated, not the corresponding ByteArrays. The +-- result will be that the GC runs less often that you hoped, as it +-- looks like only a few bytes have been allocated on the Haskell heap. +-- +-- Use of 'performGC' or 'finalizeForeignPtr' when you know that +-- the object is going out of scope can ensure that resources are +-- released appropriately. +-- + + ) where + +import System.IO.Posix.MMap.Internal + +-- import System.IO +-- import qualified System.IO as IO +import Foreign.Ptr + +import Control.Exception +import Data.ByteString + +import System.Posix hiding (openFd) +import System.Posix.IO.Compat (openFd) + +-- | The 'unsafeMMapFile' function maps a file or device into memory, +-- returning a strict 'ByteString' that accesses the mapped file. +-- If the mmap fails for some reason, an error is thrown. +-- +-- Memory mapped files will behave as if they were read lazily -- +-- pages from the file will be loaded into memory on demand. +-- +-- The storage manager is used to free the mapped memory. When +-- the garbage collector notices there are no further references to the +-- mapped memory, a call to munmap is made. It is not necessary to do +-- this yourself. In tight memory situations, it may be profitable to +-- use 'performGC' or 'finalizeForeignPtr' to force an unmap. +-- +-- Note: this operation may break referential transparency! If +-- any other process on the system changes the file when it is mapped +-- into Haskell, the contents of your 'ByteString' will change. +-- +unsafeMMapFile :: FilePath -> IO ByteString +unsafeMMapFile f = do + fd <- openFd f ReadOnly defaultFileFlags + always (closeFd fd) $ do + stat <- getFdStatus fd + let size = fromIntegral (fileSize stat) + if size <= 0 + then return empty -- BSD mmap won't accept a length of zero + else do + ptr <- c_mmap size (fromIntegral fd) + if ptr == nullPtr + then error "System.IO.Posix.MMap.mmapFile: unable to mmap file" + else unsafePackMMapPtr ptr size + + where always = flip finally diff --git a/System/IO/Posix/MMap/Internal.hs b/System/IO/Posix/MMap/Internal.hs new file mode 100644 --- /dev/null +++ b/System/IO/Posix/MMap/Internal.hs @@ -0,0 +1,54 @@ +{-# LANGUAGE ForeignFunctionInterface #-} +-------------------------------------------------------------------- +-- | +-- Module : System.IO.Posix.MMap.Internal +-- Copyright : (c) Galois, Inc. 2007 +-- License : BSD3 +-- +-- Maintainer: Don Stewart +-- Stability : provisional +-- Portability: non-portable -- posix only +-- +-- Low level mmap access. +-- +module System.IO.Posix.MMap.Internal ( + + -- * Converting an mmapped pointer to a 'ByteString' + unsafePackMMapPtr, -- :: Ptr Word8 -> CSize -> IO ByteString + + -- * Low level bindings + c_mmap, -- :: CSize -> CInt -> IO (Ptr Word8) + c_munmap -- :: Ptr Word8 -> CSize -> IO CInt + + ) where + +import System.IO +import qualified System.IO as IO +import Foreign.C.Types +import Foreign.Ptr +import qualified Foreign.Concurrent as FC + +import Control.Monad +import Data.Word +import Data.ByteString.Internal +-- import Data.ByteString + +-- | Create a bytestring from a memory mapped Ptr. +-- A finalizer will be associated with the resource, that will call +-- munmap when the storage manager detects that the resource is no longer +-- in use. +unsafePackMMapPtr :: Ptr Word8 -> CSize -> IO ByteString +unsafePackMMapPtr p s = do + fp <- FC.newForeignPtr p $ do + v <- c_munmap p s + when (v == -1) $ IO.hPutStrLn stderr $ + "System.IO.Posix.MMap: warning, failed to unmap " + ++ show s ++" bytes at "++show p + return (fromForeignPtr fp 0 (fromIntegral s)) +{-# INLINE unsafePackMMapPtr #-} + +foreign import ccall unsafe "hs_bytestring_mmap.h hs_bytestring_mmap" + c_mmap :: CSize -> CInt -> IO (Ptr Word8) + +foreign import ccall unsafe "hs_bytestring_mmap.h munmap" + c_munmap :: Ptr Word8 -> CSize -> IO CInt diff --git a/System/IO/Posix/MMap/Lazy.hs b/System/IO/Posix/MMap/Lazy.hs new file mode 100644 --- /dev/null +++ b/System/IO/Posix/MMap/Lazy.hs @@ -0,0 +1,130 @@ +{-# LANGUAGE CPP, BangPatterns, ForeignFunctionInterface #-} +-------------------------------------------------------------------- +-- | +-- Module : System.IO.Posix.MMap +-- Copyright : (c) Galois, Inc. 2007 +-- License : BSD3 +-- +-- Maintainer: Don Stewart +-- Stability : provisional +-- Portability: non-portable -- posix only +-- +-- Lazy, chunk-wise memory mapping. +-- +-- Memory map a file as a lazy ByteString. Finalisers are associated +-- cached-sized portions of the file, which will be deallocated as +-- those chunks go out of scope. +-- +-- Unlike strict Bytestrings, mmapFile for Lazy ByteStrings will +-- deallocate chunks of the file. +-- +-- The storage manager is used to free chunks of the mapped memory. When +-- the garbage collector notices there are no further references to +-- a chunk, a call to munmap is made. +-- +-- In effect, the file is mmapped once, lazily, then covered with finalizers +-- for each chunk. When any chunk goes out of scope, that part is +-- deallocated. We must allocate the spine of the structure strictly +-- though, to ensure finalizers are registered for the entire file. +-- +-- The Haskell garbage collector decides when to run based on heap +-- pressure, however the mmap stores memory outside the Haskell heap, +-- so those resources are not counted when deciding to run the garbage +-- collect. The result is that finalizers run less often than you might +-- expect, and it is possible to write a lazy bytestring mmap program +-- that never deallocates (and thus doesn't run in constant space). +-- 'performGC' or 'finalizerForeignPtr' can be used to trigger collection +-- at sensible points. +-- +-- Note: this operation may break referential transparency! If +-- any other process on the system changes the file when it is mapped +-- into Haskell, the contents of your 'ByteString' will change. +-- +module System.IO.Posix.MMap.Lazy ( + + unsafeMMapFile -- :: FilePath -> IO ByteString + + ) where + +import System.IO.Posix.MMap.Internal + +-- import System.IO +import Foreign.C.Types +import Foreign.Ptr +-- import Control.Monad + +import Control.Exception +import Data.Word +import Data.ByteString.Lazy.Internal + +import System.Posix hiding (openFd) +import System.Posix.IO.Compat (openFd) + +-- +-- | The 'unsafeMMapFile' function maps a file or device into memory as +-- a lazy ByteString, made of 64*pagesize unmappable chunks of bytes. +-- +-- Memory mapped files will behave as if they were read lazily -- +-- pages from the file will be loaded into memory on demand. +-- +-- The storage manager is used to free chunks that go out of scope, +-- and unlike strict bytestrings, memory mapped lazy ByteStrings will +-- be deallocated in chunks (so you can write traversals that run in +-- constant space). +-- +-- However, the size of the mmapped resource is not known by the Haskell +-- GC, it appears only as a small ForeignPtr. This means that the +-- Haskell GC may not not run as often as you'd like, leading to delays +-- in unmapping chunks. +-- +-- Appropriate use of performGC or finalizerForeignPtr may be required +-- to ensure deallocation, as resources allocated by mmap are not +-- tracked by the Haskell garbage collector. +-- +-- For example, when writing out a lazy bytestring allocated with mmap, +-- you may wish to finalizeForeignPtr when each chunk is written, as the +-- chunk goes out of scope, rather than relying on the garbage collector +-- to notice the chunk has gone. +-- +-- This operation is unsafe: if the file is written to by any other +-- process on the system, the 'ByteString' contents will change in +-- Haskell. +-- +unsafeMMapFile :: FilePath -> IO ByteString +unsafeMMapFile path = do + fd <- openFd path ReadOnly defaultFileFlags + always (closeFd fd) $ do + stat <- getFdStatus fd + let size = fromIntegral (fileSize stat) + ptr <- c_mmap size (fromIntegral fd) + if ptr == nullPtr + then error "System.IO.Posix.MMap.Lazy: unable to mmap file!" + else chunks chunk_size ptr (fromIntegral size) + where + always = flip finally + + -- must be page aligned. + chunk_size = 64 * fromIntegral pagesize -- empircally derived + +-- +-- Break the file up into chunks. + -- Have separate munmap finalizers for each chunk. +-- +chunks :: CSize -> Ptr Word8 -> CSize -> IO ByteString +chunks chunk_size p bytes = loop p bytes +#ifndef __HADDOCK__ + where + loop !ptr !rest + | rest <= 0 = return Empty + | otherwise = let s = min chunk_size rest + ptr' = ptr `plusPtr` fromIntegral s + rest' = rest - s + in do c <- unsafePackMMapPtr ptr s + cs <- loop ptr' rest' -- need to be strict + return (chunk c cs) -- to ensure we cover the whole file + -- with finalizers +#endif + +foreign import ccall unsafe "unistd.h getpagesize" + pagesize :: CInt + diff --git a/System/Posix/IO/Compat.hs b/System/Posix/IO/Compat.hs new file mode 100644 --- /dev/null +++ b/System/Posix/IO/Compat.hs @@ -0,0 +1,12 @@ +{-# LANGUAGE CPP #-} +module System.Posix.IO.Compat where + +import qualified System.Posix as Unix + + +openFd :: FilePath -> Unix.OpenMode -> Unix.OpenFileFlags -> IO Unix.Fd +#if MIN_VERSION_unix(2,8,0) +openFd = Unix.openFd +#else +openFd file openMode = Unix.openFd file openMode Nothing +#endif diff --git a/bytestring-mmap-compat.cabal b/bytestring-mmap-compat.cabal new file mode 100644 --- /dev/null +++ b/bytestring-mmap-compat.cabal @@ -0,0 +1,58 @@ +cabal-version: 2.0 +name: bytestring-mmap-compat +version: 0.2.3 +synopsis: mmap support for strict ByteStrings (unix >= 2.8 fork) +description: + A fork of @bytestring-mmap@ that compiles against @unix >= 2.8@. + . + The original package (Don Stewart, last uploaded 2011) calls + @System.Posix.openFd@ with the pre-2.8 four-argument signature. + Upstream is unmaintained and the latest Hackage revision restricts + the @unix@ bound to @< 2.8@, which makes the package unusable in + package sets that pull in a newer @unix@ transitively. + . + This fork adds a tiny @System.Posix.IO.Compat@ shim that selects + the right @openFd@ signature via CPP and widens the @unix@ bound, + leaving the public API (@System.IO.Posix.MMap@, + @System.IO.Posix.MMap.Lazy@) byte-for-byte compatible with the + upstream. + . + The package name is suffixed with @-compat@ so it can coexist on + Hackage with the original; switch back to upstream + @bytestring-mmap@ once it adopts a compatible @openFd@ call site. +category: System +homepage: https://github.com/NCrashed/bytestring-mmap-compat +bug-reports: https://github.com/NCrashed/bytestring-mmap-compat/issues +license: BSD3 +license-file: LICENSE +author: Don Stewart +maintainer: Anton Gushcha +copyright: (c) Don Stewart 2008-2011, (c) Anton Gushcha 2024-2026 +build-type: Simple +tested-with: GHC == 9.6.6 +extra-doc-files: README.md + CHANGELOG.md +extra-source-files: include/hs_bytestring_mmap.h + cbits/hs_bytestring_mmap.c + +source-repository head + type: git + location: https://github.com/NCrashed/bytestring-mmap-compat.git + +library + default-language: Haskell2010 + hs-source-dirs: . + exposed-modules: System.IO.Posix.MMap + System.IO.Posix.MMap.Lazy + System.IO.Posix.MMap.Internal + other-modules: System.Posix.IO.Compat + build-depends: base >= 4.14 && < 5 + , bytestring >= 0.10 && < 0.13 + , unix >= 2.7 && < 2.9 + default-extensions: CPP, ForeignFunctionInterface, BangPatterns, + NondecreasingIndentation + ghc-options: -Wall + c-sources: cbits/hs_bytestring_mmap.c + include-dirs: include + includes: hs_bytestring_mmap.h + install-includes: hs_bytestring_mmap.h diff --git a/cbits/hs_bytestring_mmap.c b/cbits/hs_bytestring_mmap.c new file mode 100644 --- /dev/null +++ b/cbits/hs_bytestring_mmap.c @@ -0,0 +1,22 @@ +/* + * hs_bytestring_mmap.c + * + * License : BSD3 + * + * Copyright (C) 2003 David Roundy + * 2005-7 Don Stewart + * + * Maintainer: Don Stewart + */ +#include "hs_bytestring_mmap.h" + +/* + * mmap len bytes from fd into memory, read only. + */ +unsigned char *hs_bytestring_mmap(size_t len, int fd) { + void *result = mmap(0, len, PROT_READ, MAP_SHARED, fd, 0); + if (result == MAP_FAILED) + return (unsigned char *)0; + else + return (unsigned char *)result; +} diff --git a/include/hs_bytestring_mmap.h b/include/hs_bytestring_mmap.h new file mode 100644 --- /dev/null +++ b/include/hs_bytestring_mmap.h @@ -0,0 +1,15 @@ +/* + * hs_bytestring_mmap.h + * + * License : BSD3 + * + * Copyright (C) 2003 David Roundy + * 2005-7 Don Stewart + * + * Maintainer: Don Stewart + */ + +#include +#include + +unsigned char *hs_bytestring_mmap(size_t len, int fd);