diff --git a/README.md b/README.md --- a/README.md +++ b/README.md @@ -1,36 +1,72 @@ # rawfilepath -A Haskell library for the mid-level system functions for the `RawFilePath` data type. +The `unix` package provides `RawFilePath` which is a type synonym of `ByteString`. Unlike `FilePath` (which is `String`), it has no performance issues because it is `ByteString`. It has no encoding issues because it is `ByteString` which is a sequence of bytes instead of characters. -## Background +That's all good. With `RawFilePath`, we can properly separate the "sequence of bytes" and the "sequence of Unicode characters." The control is yours. Properly encode or decode them with UTF-8 or UTF-16 or any codec of your choice. +However, + +- The functions in `unix` are low-level. +- The higher-level packages such as `process` and `directory` are strictly tied to `FilePath`. + +This library provides the higher-level interface with `RawFilePath`. + +## Advantages + +`rawfilepath` is easy to use. + +```haskell +{-# language OverloadedStrings #-} + +import RawFilePath +import System.IO +import qualified Data.ByteString as B + + +main :: IO () +main = do + p <- startProcess $ proc "sed" ["-e", "s/\\>/!/g"] + `setStdin` CreatePipe + `setStdout` CreatePipe + B.hPut (processStdin p) "Lorem ipsum dolor sit amet" + hClose (processStdin p) + result <- B.hGetContents (processStdout p) + print result + -- "Lorem! ipsum! dolor! sit! amet!" +``` + +- High performance +- No round-trip encoding issue +- Minimal dependencies (three packages: `bytestring`, `unix`, and `base`) +- Lightweight library (under 400 total lines of code) +- Type safety (inspired by typed-process) +- **Available now** + +## Rationale + +### Performance + Traditional `String` is notorious: - 24 bytes (three words) required for one character (the List constructor, the actual Char value, and the pointer to the next List constructor). 24x memory consumption. - Heap fragmentation causing malloc/free overhead -- A lot of pointer chasing for reading, devastating the cache hit rate +- A lot of pointer chasing for reading: Devastates the cache hit rate - A lot of pointer chasing plus a lot of heap object allocation for manipulation (appending, slicing, etc.) - Completely unnecessary but mandatory conversions and memory allocation when the data is sent to or received from the outside world -Transition to `Text` and `ByteString` began, but even after a dazzling community effort, `FilePath`, a key data type for programming anything useful, remained to be a type synonym of `String`. - -To put a cherry on top of creaking, fuming, dragging, and littering pointers all over the heap space, `String` had another fantastic nature to serve as a file path data type: Encoding blindness. All functions that return `FilePath` would actually take a series of bytes returned by a syscall and somehow magically "decode" it into a `String` which is surprising because no encoding information was given. Of course there is no magic and it's an abject fail. `FilePath` just wouldn't work. +This already makes us unhappy enough to avoid `String`. `FilePath` is a type synonym of `String`. Use `RawFilePath` instead. It's faster and occupies less memory. -In June 2015, three bright Haskell programmers came up with an elegant solution called the [Abstract FilePath Proposal] and met an immediate thunderous applause. Inspired by this enthusiasm, they further pursued the career of professional Haskell programming and focused on more interesting things. 16 months later, a programmer under the pseudonym XT got so sick and tired of the situation and released a package called `rawfilepath` that, despite being far from perfect, worked. +### Encoding -## So what is this? +`FilePath` is a type synonym of `String`. This is a bigger problem than what `String` already has, because it's not just a performance issue anymore; it's a correctness issue as there is no **encoding information**. -`RawFilePath` is a data type provided by the `unix` package. It has no performance issues because it is `ByteString` which is packed. It has no encoding issues because it is `ByteString` which is a sequence of bytes instead of characters. However, the functions in `unix` are low-level, and the higher-level packages such as `process` and `directory` are strictly tied to `FilePath`. +A syscall would give you (or expect from you) a series of bytes, but `String` is a series of characters. But how do you know the system's encoding? NTFS is UTF-16, and FAT32 uses the OEM character set. On Linux, there is no filesystem-level encoding. Would Haskell somehow magically figure out the system's encoding information and encode/decode accordingly? Well, there is no magic. `FilePath` has completely no guarantee of correct behavior at all, especially when there are non-ASCII letters. -So I decided to start writing the `RawFilePath` version of those functions. +### AFPP -## Advantages +In June 2015, three bright Haskell programmers came up with an elegant solution called the [Abstract FilePath Proposal] and met an immediate thunderous applause. Inspired by this enthusiasm, they further pursued the career of professional Haskell programming and focused on more interesting things. (sigh) -- High performance -- No round-trip encoding issue -- Minimal dependencies (`bytestring`, `unix`, and `base`) -- Lightweight library (under 400 total lines of code) -- Available now +This library provides a stable and high-performance API that is available now. ## Documentation @@ -38,8 +74,8 @@ ## To do -`rawfilepath` is in an early stage, although major backwards-incompatible changes are unlikely to happen. We can probably port more system functions that are present in `process` or `directory` +`rawfilepath` is stable. We don't expect any backward-incompatible changes. But we do want to port more system functions that are present in `process` or `directory`. We'll need to be a bit careful about their API for stability, though. Patches will be highly appreciated. -[Abstract FilePath Proposal]: https://ghc.haskell.org/trac/ghc/wiki/Proposal/AbstractFilePath +[Abstract FilePath Proposal]: https://gitlab.haskell.org/ghc/ghc/-/wikis/proposal/abstract-file-path diff --git a/rawfilepath.cabal b/rawfilepath.cabal --- a/rawfilepath.cabal +++ b/rawfilepath.cabal @@ -1,5 +1,5 @@ name: rawfilepath -version: 0.2.4 +version: 1.0.0 synopsis: Use RawFilePath instead of FilePath description: Please see README.md homepage: https://github.com/xtendo-org/rawfilepath#readme diff --git a/src/Data/ByteString/RawFilePath.hs b/src/Data/ByteString/RawFilePath.hs --- a/src/Data/ByteString/RawFilePath.hs +++ b/src/Data/ByteString/RawFilePath.hs @@ -4,7 +4,7 @@ -- License : Apache 2.0 -- -- Maintainer : e@xtendo.org --- Stability : experimental +-- Stability : stable -- Portability : POSIX -- -- A drop-in replacement of @Data.ByteString@ from the @bytestring@ package diff --git a/src/RawFilePath.hs b/src/RawFilePath.hs --- a/src/RawFilePath.hs +++ b/src/RawFilePath.hs @@ -5,7 +5,7 @@ -- License : BSD-style (see the file LICENSE) -- -- Maintainer : e@xtendo.org --- Stability : experimental +-- Stability : stable -- Portability : POSIX -- -- Welcome to @RawFilePath@, a small part of the Haskell community's effort to @@ -24,8 +24,12 @@ -- * A lot of pointer chasing plus a lot of heap object allocation for manipulation (appending, slicing, etc.) -- - Completely unnecessary but mandatory conversions and memory allocation when the data is sent to or received from the outside world -- --- 'String' has another problematic nature to serve as a file path data type: Encoding blindness. All functions that return 'FilePath' would actually take a series of bytes returned by a syscall and somehow magically "decode" it into a `String` which is surprising because no encoding information was given. Of course there is no magic and it's an abject fail. 'FilePath' just wouldn't work. +-- `FilePath` is a type synonym of `String`. This is a bigger problem than what `String` already has, because it's not just a performance issue anymore; it's a correctness issue as there is no encoding information. -- +-- A syscall would give you (or expect from you) a series of bytes, but `String` is a series of characters. But how do you know the system's encoding? NTFS is UTF-16, and FAT32 uses the OEM character set. On Linux, there is no filesystem-level encoding. Would Haskell somehow magically figure out the system's encoding information and encode/decode accordingly? Well, there is no magic. `FilePath` has completely no guarantee of correct behavior at all, especially when there are non-ASCII letters. +-- +-- With this library, you use 'RawFilePath' which is a sequence of bytes (instead of characters). You have the full control of decoding from (or encoding to) these bytes. This lets you do the job properly. +-- -- == Usage -- -- This is the top-level module that re-exports the sub-modules. Therefore, @@ -35,7 +39,13 @@ -- import RawFilePath -- @ -- --- to import all functions. +-- to import all functions. For documentation, see: +-- +-- * "RawFilePath.Directory" +-- * "RawFilePath.Process" +-- +-- For process-related functions, see "RawFilePath.Process" for a brief +-- introduction and an example code. -- ----------------------------------------------------------------------------- diff --git a/src/RawFilePath/Directory.hs b/src/RawFilePath/Directory.hs --- a/src/RawFilePath/Directory.hs +++ b/src/RawFilePath/Directory.hs @@ -5,7 +5,7 @@ -- License : BSD-style (see the LICENSE file) -- -- Maintainer : e@xtendo.org --- Stability : experimental +-- Stability : stable -- Portability : POSIX -- -- This is the module for the 'RawFilePath' version of functions in the diff --git a/src/RawFilePath/Import.hs b/src/RawFilePath/Import.hs --- a/src/RawFilePath/Import.hs +++ b/src/RawFilePath/Import.hs @@ -20,7 +20,14 @@ import GHC.IO.Device as Module hiding (close, getEcho, setEcho, Directory) import GHC.IO.Encoding as Module import GHC.IO.Exception as Module -import GHC.IO.Handle.FD as Module hiding (fdToHandle) +import GHC.IO.Handle.FD as Module hiding + ( fdToHandle + , openBinaryFile + , openFile + , stderr + , stdin + , stdout + ) import GHC.IO.Handle.Internals as Module import GHC.IO.Handle.Types as Module import GHC.IO.IOMode as Module diff --git a/src/RawFilePath/Process.hs b/src/RawFilePath/Process.hs --- a/src/RawFilePath/Process.hs +++ b/src/RawFilePath/Process.hs @@ -5,11 +5,11 @@ -- License : BSD-style (see the file LICENSE) -- -- Maintainer : e@xtendo.org --- Stability : experimental +-- Stability : stable -- Portability : POSIX -- --- Welcome to @RawFilePath.Process@, a small part of the Haskell --- community's effort to purge 'String' for the Greater Good. +-- Welcome to @RawFilePath.Process@, a small part of the Haskell community's +-- effort to replace 'String' for the Greater Good. -- -- With this module, you can create (and interact with) sub-processes without -- the encoding problem of 'String'. The command and its arguments, all @@ -27,23 +27,92 @@ -- example, use 'Data.ByteString.hGetContents' from "Data.ByteString" to read -- from a 'Handle' as a 'ByteString'. -- --- == Example +-- == Fast and Brief Example -- +-- If you have experience with Unix pipes, this example should be pretty +-- straightforward. In fact it is so simple that you don't need any type +-- theory or PL knowledge. It demonstrates how you can create a child process +-- and interact with it. +-- -- @ -- {-\# language OverloadedStrings \#-} -- -- import RawFilePath.Process +-- import System.IO -- import qualified Data.ByteString as B -- +-- -- main :: IO () -- main = do --- p <- 'startProcess' $ 'proc' "echo" ["hello"] --- \`setStdout\` 'CreatePipe' --- result <- B.hGetContents ('processStdout' p) --- _ <- 'waitForProcess' p +-- p \<- 'startProcess' $ 'proc' "sed" ["-e", "s\/\\\\\>\/!\/g"] +-- \`'setStdin'\` 'CreatePipe' +-- \`'setStdout'\` 'CreatePipe' +-- B.hPut ('processStdin' p) "Lorem ipsum dolor sit amet" +-- hClose ('processStdin' p) +-- result <- B.hGetContents ('processStdout' p) +-- print result +-- -- "Lorem! ipsum! dolor! sit! amet!" +-- @ -- --- print (result == "hello\\n") +-- That's it! You can totally skip the verbose explanation below. +-- +-- == Verbose Explanation of the Example +-- +-- We launch @sed@ as a child process. As we know, it is a regular expression +-- search and replacement tool. In the example, @sed@ is a simple Unix pipe +-- utility: Take some text from @stdin@ and output the processed text to +-- @stdout@. +-- +-- In @sed@ regex, @\\\>@ means "the end of the word." So, @"s\/\\\\\>\/!\/g"@ +-- means "substitute all ends of the words with an exclamation mark." Then, we +-- feed some text to its @stdin@, close @stdin@ (to send EOF to @sed@ EOF), +-- and read what it said to @stdout@. +-- +-- The interesting part is 'proc'. It is a simple function that takes a +-- command and its arguments and returns a 'ProcessConf' which defines the +-- properties of the child process you want to create. You can use +-- functions like 'setStdin' or 'setStdout' to change those properties. +-- +-- The advantage of this interface is type safety. Take @stdout@ for example. +-- There are four options: @Inherit@, @UseHandle@, @CreatePipe@, and +-- @NoStream@. If you want to read @stdout@ of the child process, you must set +-- it to @CreatePipe@. With the @process@ package, this is done by giving a +-- proper argument to @createProcess@. The trouble is, regardless of the +-- argument, @createProcess@ returns 'Maybe' 'Handle' as @stdout@. You may or +-- may not get a 'Handle'. +-- +-- This is not what we want with Haskell. We want to ensure that (1) we use +-- 'CreatePipe' and certainly get the @stdout@ 'Handle' without the fear of +-- 'Nothing', and (2) if we don't use 'CreatePipe' but still request the +-- @stdout@ 'Handle', it is an error, detected at compile time. +-- +-- So that's what @RawFilePath.Process@ does. In the above example, we use +-- functions like 'setStdout'. Later, you use the 'processStdout' family of +-- functions to get the process's standard stream handles. This requires that +-- the process was created with 'CreatePipe' appropriately set for that +-- stream. +-- +-- It sounds all complicated, but all you really need to do is as simple as: +-- -- @ +-- 'startProcess' $ 'proc' \"...\" [...] \`'setStdout'\` 'CreatePipe' +-- @ +-- +-- ... If you want to create a new pipe for the child process's @stdin@. Then +-- you can later use `processStdout` to get the 'Handle'. If you don't put the +-- @\`setStdout\` CreatePipe@ part or set it to something other than +-- @CreatePipe@, it will be a compile-time error to use 'processStdout' on +-- this process object. +-- +-- In short, it makes the correct code easy and the wrong code impossible. +-- This approach was inspired by the @typed-process@ package. Then why not +-- just @typed-process@? @rawfilepath@ offers +-- +-- 1. RawFilePath! +-- 2. A lot less dependency (only three packages) +-- 3. A lot more portability (doesn't require any language extension). +-- +-- Enjoy. -- -----------------------------------------------------------------------------