packages feed

rawfilepath 0.2.4 → 1.0.0

raw patch · 7 files changed

+156/−34 lines, 7 filesdep ~base

Dependency ranges changed: base

Files

README.md view
@@ -1,36 +1,72 @@ # rawfilepath -A Haskell library for the mid-level system functions for the `RawFilePath` data type.+The `unix` package provides `RawFilePath` which is a type synonym of `ByteString`. Unlike `FilePath` (which is `String`), it has no performance issues because it is `ByteString`. It has no encoding issues because it is `ByteString` which is a sequence of bytes instead of characters. -## Background+That's all good. With `RawFilePath`, we can properly separate the "sequence of bytes" and the "sequence of Unicode characters." The control is yours. Properly encode or decode them with UTF-8 or UTF-16 or any codec of your choice. +However,++- The functions in `unix` are low-level.+- The higher-level packages such as `process` and `directory` are strictly tied to `FilePath`.++This library provides the higher-level interface with `RawFilePath`.++## Advantages++`rawfilepath` is easy to use.++```haskell+{-# language OverloadedStrings #-}++import RawFilePath+import System.IO+import qualified Data.ByteString as B+++main :: IO ()+main = do+  p <- startProcess $ proc "sed" ["-e", "s/\\>/!/g"]+    `setStdin` CreatePipe+    `setStdout` CreatePipe+  B.hPut (processStdin p) "Lorem ipsum dolor sit amet"+  hClose (processStdin p)+  result <- B.hGetContents (processStdout p)+  print result+  -- "Lorem! ipsum! dolor! sit! amet!"+```++- High performance+- No round-trip encoding issue+- Minimal dependencies (three packages: `bytestring`, `unix`, and `base`)+- Lightweight library (under 400 total lines of code)+- Type safety (inspired by typed-process)+- **Available now**++## Rationale++### Performance+ Traditional `String` is notorious:  - 24 bytes (three words) required for one character (the List constructor, the actual Char value, and the pointer to the next List constructor). 24x memory consumption. - Heap fragmentation causing malloc/free overhead-- A lot of pointer chasing for reading, devastating the cache hit rate+- A lot of pointer chasing for reading: Devastates the cache hit rate - A lot of pointer chasing plus a lot of heap object allocation for manipulation (appending, slicing, etc.) - Completely unnecessary but mandatory conversions and memory allocation when the data is sent to or received from the outside world -Transition to `Text` and `ByteString` began, but even after a dazzling community effort, `FilePath`, a key data type for programming anything useful, remained to be a type synonym of `String`.--To put a cherry on top of creaking, fuming, dragging, and littering pointers all over the heap space, `String` had another fantastic nature to serve as a file path data type: Encoding blindness. All functions that return `FilePath` would actually take a series of bytes returned by a syscall and somehow magically "decode" it into a `String` which is surprising because no encoding information was given. Of course there is no magic and it's an abject fail. `FilePath` just wouldn't work.+This already makes us unhappy enough to avoid `String`. `FilePath` is a type synonym of `String`. Use `RawFilePath` instead. It's faster and occupies less memory. -In June 2015, three bright Haskell programmers came up with an elegant solution called the [Abstract FilePath Proposal] and met an immediate thunderous applause. Inspired by this enthusiasm, they further pursued the career of professional Haskell programming and focused on more interesting things. 16 months later, a programmer under the pseudonym XT got so sick and tired of the situation and released a package called `rawfilepath` that, despite being far from perfect, worked.+### Encoding -## So what is this?+`FilePath` is a type synonym of `String`. This is a bigger problem than what `String` already has, because it's not just a performance issue anymore; it's a correctness issue as there is no **encoding information**. -`RawFilePath` is a data type provided by the `unix` package. It has no performance issues because it is `ByteString` which is packed. It has no encoding issues because it is `ByteString` which is a sequence of bytes instead of characters. However, the functions in `unix` are low-level, and the higher-level packages such as `process` and `directory` are strictly tied to `FilePath`.+A syscall would give you (or expect from you) a series of bytes, but `String` is a series of characters. But how do you know the system's encoding? NTFS is UTF-16, and FAT32 uses the OEM character set. On Linux, there is no filesystem-level encoding. Would Haskell somehow magically figure out the system's encoding information and encode/decode accordingly? Well, there is no magic. `FilePath` has completely no guarantee of correct behavior at all, especially when there are non-ASCII letters. -So I decided to start writing the `RawFilePath` version of those functions.+### AFPP -## Advantages+In June 2015, three bright Haskell programmers came up with an elegant solution called the [Abstract FilePath Proposal] and met an immediate thunderous applause. Inspired by this enthusiasm, they further pursued the career of professional Haskell programming and focused on more interesting things. (sigh) -- High performance-- No round-trip encoding issue-- Minimal dependencies (`bytestring`, `unix`, and `base`)-- Lightweight library (under 400 total lines of code)-- Available now+This library provides a stable and high-performance API that is available now.  ## Documentation @@ -38,8 +74,8 @@  ## To do -`rawfilepath` is in an early stage, although major backwards-incompatible changes are unlikely to happen. We can probably port more system functions that are present in `process` or `directory`+`rawfilepath` is stable. We don't expect any backward-incompatible changes. But we do want to port more system functions that are present in `process` or `directory`. We'll need to be a bit careful about their API for stability, though.  Patches will be highly appreciated. -[Abstract FilePath Proposal]: https://ghc.haskell.org/trac/ghc/wiki/Proposal/AbstractFilePath+[Abstract FilePath Proposal]: https://gitlab.haskell.org/ghc/ghc/-/wikis/proposal/abstract-file-path
rawfilepath.cabal view
@@ -1,5 +1,5 @@ name:                rawfilepath-version:             0.2.4+version:             1.0.0 synopsis:            Use RawFilePath instead of FilePath description:         Please see README.md homepage:            https://github.com/xtendo-org/rawfilepath#readme
src/Data/ByteString/RawFilePath.hs view
@@ -4,7 +4,7 @@ -- License     : Apache 2.0 -- -- Maintainer  : e@xtendo.org--- Stability   : experimental+-- Stability   : stable -- Portability : POSIX -- -- A drop-in replacement of @Data.ByteString@ from the @bytestring@ package
src/RawFilePath.hs view
@@ -5,7 +5,7 @@ -- License     :  BSD-style (see the file LICENSE) -- -- Maintainer  :  e@xtendo.org--- Stability   :  experimental+-- Stability   :  stable -- Portability :  POSIX -- -- Welcome to @RawFilePath@, a small part of the Haskell community's effort to@@ -24,8 +24,12 @@ -- * A lot of pointer chasing plus a lot of heap object allocation for manipulation (appending, slicing, etc.) -- - Completely unnecessary but mandatory conversions and memory allocation when the data is sent to or received from the outside world ----- 'String' has another problematic nature to serve as a file path data type: Encoding blindness. All functions that return 'FilePath' would actually take a series of bytes returned by a syscall and somehow magically "decode" it into a `String` which is surprising because no encoding information was given. Of course there is no magic and it's an abject fail. 'FilePath' just wouldn't work.+-- `FilePath` is a type synonym of `String`. This is a bigger problem than what `String` already has, because it's not just a performance issue anymore; it's a correctness issue as there is no encoding information. --+-- A syscall would give you (or expect from you) a series of bytes, but `String` is a series of characters. But how do you know the system's encoding? NTFS is UTF-16, and FAT32 uses the OEM character set. On Linux, there is no filesystem-level encoding. Would Haskell somehow magically figure out the system's encoding information and encode/decode accordingly? Well, there is no magic. `FilePath` has completely no guarantee of correct behavior at all, especially when there are non-ASCII letters.+--+-- With this library, you use 'RawFilePath' which is a sequence of bytes (instead of characters). You have the full control of decoding from (or encoding to) these bytes. This lets you do the job properly.+-- -- == Usage -- -- This is the top-level module that re-exports the sub-modules. Therefore,@@ -35,7 +39,13 @@ -- import RawFilePath -- @ ----- to import all functions.+-- to import all functions. For documentation, see:+--+-- * "RawFilePath.Directory"+-- * "RawFilePath.Process"+--+-- For process-related functions, see "RawFilePath.Process" for a brief+-- introduction and an example code. -- -----------------------------------------------------------------------------
src/RawFilePath/Directory.hs view
@@ -5,7 +5,7 @@ -- License     :  BSD-style (see the LICENSE file) -- -- Maintainer  :  e@xtendo.org--- Stability   :  experimental+-- Stability   :  stable -- Portability :  POSIX -- -- This is the module for the 'RawFilePath' version of functions in the
src/RawFilePath/Import.hs view
@@ -20,7 +20,14 @@ import GHC.IO.Device as Module hiding (close, getEcho, setEcho, Directory) import GHC.IO.Encoding as Module import GHC.IO.Exception as Module-import GHC.IO.Handle.FD as Module hiding (fdToHandle)+import GHC.IO.Handle.FD as Module hiding+  ( fdToHandle+  , openBinaryFile+  , openFile+  , stderr+  , stdin+  , stdout+  ) import GHC.IO.Handle.Internals as Module import GHC.IO.Handle.Types as Module import GHC.IO.IOMode as Module
src/RawFilePath/Process.hs view
@@ -5,11 +5,11 @@ -- License     :  BSD-style (see the file LICENSE) -- -- Maintainer  :  e@xtendo.org--- Stability   :  experimental+-- Stability   :  stable -- Portability :  POSIX ----- Welcome to @RawFilePath.Process@, a small part of the Haskell--- community's effort to purge 'String' for the Greater Good.+-- Welcome to @RawFilePath.Process@, a small part of the Haskell community's+-- effort to replace 'String' for the Greater Good. -- -- With this module, you can create (and interact with) sub-processes without -- the encoding problem of 'String'. The command and its arguments, all@@ -27,23 +27,92 @@ -- example, use 'Data.ByteString.hGetContents' from "Data.ByteString" to read -- from a 'Handle' as a 'ByteString'. ----- == Example+-- == Fast and Brief Example --+-- If you have experience with Unix pipes, this example should be pretty+-- straightforward. In fact it is so simple that you don't need any type+-- theory or PL knowledge. It demonstrates how you can create a child process+-- and interact with it.+-- -- @ -- {-\# language OverloadedStrings \#-} -- -- import RawFilePath.Process+-- import System.IO -- import qualified Data.ByteString as B --+-- -- main :: IO () -- main = do---     p <- 'startProcess' $ 'proc' "echo" ["hello"]---         \`setStdout\` 'CreatePipe'---     result <- B.hGetContents ('processStdout' p)---     _ <- 'waitForProcess' p+--   p \<- 'startProcess' $ 'proc' "sed" ["-e", "s\/\\\\\>\/!\/g"]+--     \`'setStdin'\` 'CreatePipe'+--     \`'setStdout'\` 'CreatePipe'+--   B.hPut ('processStdin' p) "Lorem ipsum dolor sit amet"+--   hClose ('processStdin' p)+--   result <- B.hGetContents ('processStdout' p)+--   print result+--   -- "Lorem! ipsum! dolor! sit! amet!"+-- @ -----     print (result == "hello\\n")+-- That's it! You can totally skip the verbose explanation below.+--+-- == Verbose Explanation of the Example+--+-- We launch @sed@ as a child process. As we know, it is a regular expression+-- search and replacement tool. In the example, @sed@ is a simple Unix pipe+-- utility: Take some text from @stdin@ and output the processed text to+-- @stdout@.+--+-- In @sed@ regex, @\\\>@ means "the end of the word." So, @"s\/\\\\\>\/!\/g"@+-- means "substitute all ends of the words with an exclamation mark." Then, we+-- feed some text to its @stdin@, close @stdin@ (to send EOF to @sed@ EOF),+-- and read what it said to @stdout@.+--+-- The interesting part is 'proc'. It is a simple function that takes a+-- command and its arguments and returns a 'ProcessConf' which defines the+-- properties of the child process you want to create. You can use+-- functions like 'setStdin' or 'setStdout' to change those properties.+--+-- The advantage of this interface is type safety. Take @stdout@ for example.+-- There are four options: @Inherit@, @UseHandle@, @CreatePipe@, and+-- @NoStream@. If you want to read @stdout@ of the child process, you must set+-- it to @CreatePipe@. With the @process@ package, this is done by giving a+-- proper argument to @createProcess@. The trouble is, regardless of the+-- argument, @createProcess@ returns 'Maybe' 'Handle' as @stdout@. You may or+-- may not get a 'Handle'.+--+-- This is not what we want with Haskell. We want to ensure that (1) we use+-- 'CreatePipe' and certainly get the @stdout@ 'Handle' without the fear of+-- 'Nothing', and (2) if we don't use 'CreatePipe' but still request the+-- @stdout@ 'Handle', it is an error, detected at compile time.+--+-- So that's what @RawFilePath.Process@ does. In the above example, we use+-- functions like 'setStdout'. Later, you use the 'processStdout' family of+-- functions to get the process's standard stream handles. This requires that+-- the process was created with 'CreatePipe' appropriately set for that+-- stream.+--+-- It sounds all complicated, but all you really need to do is as simple as:+-- -- @+-- 'startProcess' $ 'proc' \"...\" [...] \`'setStdout'\` 'CreatePipe'+-- @+--+-- ... If you want to create a new pipe for the child process's @stdin@. Then+-- you can later use `processStdout` to get the 'Handle'. If you don't put the+-- @\`setStdout\` CreatePipe@ part or set it to something other than+-- @CreatePipe@, it will be a compile-time error to use 'processStdout' on+-- this process object.+--+-- In short, it makes the correct code easy and the wrong code impossible.+-- This approach was inspired by the @typed-process@ package. Then why not+-- just @typed-process@? @rawfilepath@ offers+--+-- 1. RawFilePath!+-- 2. A lot less dependency (only three packages)+-- 3. A lot more portability (doesn't require any language extension).+--+-- Enjoy. -- -----------------------------------------------------------------------------