with-utf8 (empty) → 1.0.0.0
raw patch · 15 files changed
+1248/−0 lines, 15 filesdep +HUnitdep +basedep +deepseq
Dependencies added: HUnit, base, deepseq, hedgehog, safe-exceptions, tasty, tasty-hedgehog, tasty-hunit, temporary, text, unix, with-utf8
Files
- CHANGELOG.md +27/−0
- LICENSES/MPL-2.0.txt +312/−0
- README.md +96/−0
- lib/Data/Text/IO/Utf8.hs +35/−0
- lib/Data/Text/Lazy/IO/Utf8.hs +35/−0
- lib/Main/Utf8.hs +82/−0
- lib/System/IO/Utf8.hs +170/−0
- lib/System/IO/Utf8/Internal.hs +86/−0
- test/Test.hs +22/−0
- test/Test/Utf8/Choice.hs +106/−0
- test/Test/Utf8/ReadWrite.hs +84/−0
- test/Test/Utf8/Set.hs +64/−0
- test/Test/Util.hs +37/−0
- test/Tree.hs +6/−0
- with-utf8.cabal +86/−0
+ CHANGELOG.md view
@@ -0,0 +1,27 @@+<!--+SPDX-FileCopyrightText: 2020 Serokell <https://serokell.io/>++SPDX-License-Identifier: MPL-2.0+-->++# Changelog++## Unreleased+++## 1.0.0.0++Initial release.++### Added++- `withUtf8`+- `withStdTerminalHandles`+- `setHandleEncoding`+- `withHandle`+- `setTerminalHandleEncoding`+- `withTerminalHandle`+- `openFile`+- `withFile`+- `readFile`+- `writeFile`
+ LICENSES/MPL-2.0.txt view
@@ -0,0 +1,312 @@+Mozilla Public License Version 2.0++ 1. Definitions++1.1. "Contributor" means each individual or legal entity that creates, contributes+to the creation of, or owns Covered Software.++1.2. "Contributor Version" means the combination of the Contributions of others+(if any) used by a Contributor and that particular Contributor's Contribution.++ 1.3. "Contribution" means Covered Software of a particular Contributor.++1.4. "Covered Software" means Source Code Form to which the initial Contributor+has attached the notice in Exhibit A, the Executable Form of such Source Code+Form, and Modifications of such Source Code Form, in each case including portions+thereof.++ 1.5. "Incompatible With Secondary Licenses" means++(a) that the initial Contributor has attached the notice described in Exhibit+B to the Covered Software; or++(b) that the Covered Software was made available under the terms of version+1.1 or earlier of the License, but not also under the terms of a Secondary+License.++1.6. "Executable Form" means any form of the work other than Source Code Form.++1.7. "Larger Work" means a work that combines Covered Software with other+material, in a separate file or files, that is not Covered Software.++ 1.8. "License" means this document.++1.9. "Licensable" means having the right to grant, to the maximum extent possible,+whether at the time of the initial grant or subsequently, any and all of the+rights conveyed by this License.++ 1.10. "Modifications" means any of the following:++(a) any file in Source Code Form that results from an addition to, deletion+from, or modification of the contents of Covered Software; or++(b) any new file in Source Code Form that contains any Covered Software.++1.11. "Patent Claims" of a Contributor means any patent claim(s), including+without limitation, method, process, and apparatus claims, in any patent Licensable+by such Contributor that would be infringed, but for the grant of the License,+by the making, using, selling, offering for sale, having made, import, or+transfer of either its Contributions or its Contributor Version.++1.12. "Secondary License" means either the GNU General Public License, Version+2.0, the GNU Lesser General Public License, Version 2.1, the GNU Affero General+Public License, Version 3.0, or any later versions of those licenses.++1.13. "Source Code Form" means the form of the work preferred for making modifications.++1.14. "You" (or "Your") means an individual or a legal entity exercising rights+under this License. For legal entities, "You" includes any entity that controls,+is controlled by, or is under common control with You. For purposes of this+definition, "control" means (a) the power, direct or indirect, to cause the+direction or management of such entity, whether by contract or otherwise,+or (b) ownership of more than fifty percent (50%) of the outstanding shares+or beneficial ownership of such entity.++ 2. License Grants and Conditions++ 2.1. Grants++Each Contributor hereby grants You a world-wide, royalty-free, non-exclusive+license:++(a) under intellectual property rights (other than patent or trademark) Licensable+by such Contributor to use, reproduce, make available, modify, display, perform,+distribute, and otherwise exploit its Contributions, either on an unmodified+basis, with Modifications, or as part of a Larger Work; and++(b) under Patent Claims of such Contributor to make, use, sell, offer for+sale, have made, import, and otherwise transfer either its Contributions or+its Contributor Version.++ 2.2. Effective Date++The licenses granted in Section 2.1 with respect to any Contribution become+effective for each Contribution on the date the Contributor first distributes+such Contribution.++ 2.3. Limitations on Grant Scope++The licenses granted in this Section 2 are the only rights granted under this+License. No additional rights or licenses will be implied from the distribution+or licensing of Covered Software under this License. Notwithstanding Section+2.1(b) above, no patent license is granted by a Contributor:++(a) for any code that a Contributor has removed from Covered Software; or++(b) for infringements caused by: (i) Your and any other third party's modifications+of Covered Software, or (ii) the combination of its Contributions with other+software (except as part of its Contributor Version); or++(c) under Patent Claims infringed by Covered Software in the absence of its+Contributions.++This License does not grant any rights in the trademarks, service marks, or+logos of any Contributor (except as may be necessary to comply with the notice+requirements in Section 3.4).++ 2.4. Subsequent Licenses++No Contributor makes additional grants as a result of Your choice to distribute+the Covered Software under a subsequent version of this License (see Section+10.2) or under the terms of a Secondary License (if permitted under the terms+of Section 3.3).++ 2.5. Representation++Each Contributor represents that the Contributor believes its Contributions+are its original creation(s) or it has sufficient rights to grant the rights+to its Contributions conveyed by this License.++ 2.6. Fair Use++This License is not intended to limit any rights You have under applicable+copyright doctrines of fair use, fair dealing, or other equivalents.++ 2.7. Conditions++Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted in+Section 2.1.++ 3. Responsibilities++ 3.1. Distribution of Source Form++All distribution of Covered Software in Source Code Form, including any Modifications+that You create or to which You contribute, must be under the terms of this+License. You must inform recipients that the Source Code Form of the Covered+Software is governed by the terms of this License, and how they can obtain+a copy of this License. You may not attempt to alter or restrict the recipients'+rights in the Source Code Form.++ 3.2. Distribution of Executable Form++ If You distribute Covered Software in Executable Form then:++(a) such Covered Software must also be made available in Source Code Form,+as described in Section 3.1, and You must inform recipients of the Executable+Form how they can obtain a copy of such Source Code Form by reasonable means+in a timely manner, at a charge no more than the cost of distribution to the+recipient; and++(b) You may distribute such Executable Form under the terms of this License,+or sublicense it under different terms, provided that the license for the+Executable Form does not attempt to limit or alter the recipients' rights+in the Source Code Form under this License.++ 3.3. Distribution of a Larger Work++You may create and distribute a Larger Work under terms of Your choice, provided+that You also comply with the requirements of this License for the Covered+Software. If the Larger Work is a combination of Covered Software with a work+governed by one or more Secondary Licenses, and the Covered Software is not+Incompatible With Secondary Licenses, this License permits You to additionally+distribute such Covered Software under the terms of such Secondary License(s),+so that the recipient of the Larger Work may, at their option, further distribute+the Covered Software under the terms of either this License or such Secondary+License(s).++ 3.4. Notices++You may not remove or alter the substance of any license notices (including+copyright notices, patent notices, disclaimers of warranty, or limitations+of liability) contained within the Source Code Form of the Covered Software,+except that You may alter any license notices to the extent required to remedy+known factual inaccuracies.++ 3.5. Application of Additional Terms++You may choose to offer, and to charge a fee for, warranty, support, indemnity+or liability obligations to one or more recipients of Covered Software. However,+You may do so only on Your own behalf, and not on behalf of any Contributor.+You must make it absolutely clear that any such warranty, support, indemnity,+or liability obligation is offered by You alone, and You hereby agree to indemnify+every Contributor for any liability incurred by such Contributor as a result+of warranty, support, indemnity or liability terms You offer. You may include+additional disclaimers of warranty and limitations of liability specific to+any jurisdiction.++ 4. Inability to Comply Due to Statute or Regulation++If it is impossible for You to comply with any of the terms of this License+with respect to some or all of the Covered Software due to statute, judicial+order, or regulation then You must: (a) comply with the terms of this License+to the maximum extent possible; and (b) describe the limitations and the code+they affect. Such description must be placed in a text file included with+all distributions of the Covered Software under this License. Except to the+extent prohibited by statute or regulation, such description must be sufficiently+detailed for a recipient of ordinary skill to be able to understand it.++ 5. Termination++5.1. The rights granted under this License will terminate automatically if+You fail to comply with any of its terms. However, if You become compliant,+then the rights granted under this License from a particular Contributor are+reinstated (a) provisionally, unless and until such Contributor explicitly+and finally terminates Your grants, and (b) on an ongoing basis, if such Contributor+fails to notify You of the non-compliance by some reasonable means prior to+60 days after You have come back into compliance. Moreover, Your grants from+a particular Contributor are reinstated on an ongoing basis if such Contributor+notifies You of the non-compliance by some reasonable means, this is the first+time You have received notice of non-compliance with this License from such+Contributor, and You become compliant prior to 30 days after Your receipt+of the notice.++5.2. If You initiate litigation against any entity by asserting a patent infringement+claim (excluding declaratory judgment actions, counter-claims, and cross-claims)+alleging that a Contributor Version directly or indirectly infringes any patent,+then the rights granted to You by any and all Contributors for the Covered+Software under Section 2.1 of this License shall terminate.++5.3. In the event of termination under Sections 5.1 or 5.2 above, all end+user license agreements (excluding distributors and resellers) which have+been validly granted by You or Your distributors under this License prior+to termination shall survive termination.++ 6. Disclaimer of Warranty++Covered Software is provided under this License on an "as is" basis, without+warranty of any kind, either expressed, implied, or statutory, including,+without limitation, warranties that the Covered Software is free of defects,+merchantable, fit for a particular purpose or non-infringing. The entire risk+as to the quality and performance of the Covered Software is with You. Should+any Covered Software prove defective in any respect, You (not any Contributor)+assume the cost of any necessary servicing, repair, or correction. This disclaimer+of warranty constitutes an essential part of this License. No use of any Covered+Software is authorized under this License except under this disclaimer.++ 7. Limitation of Liability++Under no circumstances and under no legal theory, whether tort (including+negligence), contract, or otherwise, shall any Contributor, or anyone who+distributes Covered Software as permitted above, be liable to You for any+direct, indirect, special, incidental, or consequential damages of any character+including, without limitation, damages for lost profits, loss of goodwill,+work stoppage, computer failure or malfunction, or any and all other commercial+damages or losses, even if such party shall have been informed of the possibility+of such damages. This limitation of liability shall not apply to liability+for death or personal injury resulting from such party's negligence to the+extent applicable law prohibits such limitation. Some jurisdictions do not+allow the exclusion or limitation of incidental or consequential damages,+so this exclusion and limitation may not apply to You.++ 8. Litigation++Any litigation relating to this License may be brought only in the courts+of a jurisdiction where the defendant maintains its principal place of business+and such litigation shall be governed by laws of that jurisdiction, without+reference to its conflict-of-law provisions. Nothing in this Section shall+prevent a party's ability to bring cross-claims or counter-claims.++ 9. Miscellaneous++This License represents the complete agreement concerning the subject matter+hereof. If any provision of this License is held to be unenforceable, such+provision shall be reformed only to the extent necessary to make it enforceable.+Any law or regulation which provides that the language of a contract shall+be construed against the drafter shall not be used to construe this License+against a Contributor.++ 10. Versions of the License++ 10.1. New Versions++Mozilla Foundation is the license steward. Except as provided in Section 10.3,+no one other than the license steward has the right to modify or publish new+versions of this License. Each version will be given a distinguishing version+number.++ 10.2. Effect of New Versions++You may distribute the Covered Software under the terms of the version of+the License under which You originally received the Covered Software, or under+the terms of any subsequent version published by the license steward.++ 10.3. Modified Versions++If you create software not governed by this License, and you want to create+a new license for such software, you may create and use a modified version+of this License if you rename the license and remove any references to the+name of the license steward (except to note that such modified license differs+from this License).++10.4. Distributing Source Code Form that is Incompatible With Secondary Licenses++If You choose to distribute Source Code Form that is Incompatible With Secondary+Licenses under the terms of this version of the License, the notice described+in Exhibit B of this License must be attached. Exhibit A - Source Code Form+License Notice++This Source Code Form is subject to the terms of the Mozilla Public License,+v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain+one at http://mozilla.org/MPL/2.0/.++If it is not possible or desirable to put the notice in a particular file,+then You may include the notice in a location (such as a LICENSE file in a+relevant directory) where a recipient would be likely to look for such a notice.++You may add additional accurate notices of copyright ownership.++Exhibit B - "Incompatible With Secondary Licenses" Notice++This Source Code Form is "Incompatible With Secondary Licenses", as defined+by the Mozilla Public License, v. 2.0.
+ README.md view
@@ -0,0 +1,96 @@+# `with-utf8`++Get your IO right on the first try.++Reading files in Haskell is trickier than it could be due to the non-obvious+interactions between file encodings and system locale. This library is meant+to make it easy once and for all by providing “defaults” that make more sense+in the modern world.++See [this blog post][blog:post] for more details on why this library needs to+exists and an explanation of some of the opinionated decisions it is based on.++[blog:post]: https://serokell.io/blog/haskell-with-utf8+++## Use++See the documentation on Hackage for details, this is a quick summary.++### Step 1: Get it++The [library is on Hackage][hackage:with-utf8],+go ahead and add it to the dependencies of your project.++[hackage:with-utf8]: https://hackage.haskell.org/package/with-utf8++### Step 2: Wrap your `main`++Import `withUtf8` from `Main.Utf8` and wrap it around your `main`:++```haskell+import Main.Utf8 (withUtf8)++main :: IO ()+main = withUtf8 $+ {- ... your main function ... -}+```++This will make sure that if your program reads something from `stdin` or+outputs something to `stdout`/`stderr`, it will not fail with a runtime+error due to encoding issues.++### Step 3: Read files using UTF-8++If you are going to read a text file (to be precise, if you are going to open+a file in text mode), you’ll probably use `withFile`, `openFile`, or `readFile`.+Grab the first two from `System.IO.Utf8` or the latter from `Data.Text.IO.Utf8`.++_Note: it is best to import these modules qualified._++_Note: there is no `System.IO.Utf8.readFile` because it’s 2020 and+you should not read `String`s from files._++All these functions will make sure that the content will be treated as if it+was encoded in UTF-8.++If, for some reason, you really need to use `withFile`/`openFile` from `base`,+or you got your file handle from somewhere else, wrap the code that works+with it in a call to `withHandle` from `System.IO.Utf8`:++```haskell+import qualified System.IO as IO+import qualified System.IO.Utf8 as Utf8++doSomethingWithAFile :: IO.Handle -> IO ()+doSomethingWithAFile h = Utf8.withhandle h $ do+ {- ... work with the file ... -}+```++### Step 4: Write files using UTF-8++When writing a file either open it using `withFile`/`openFile` from+`System.IO.Utf8` or write to it directly with `writeFile` from+`Data.Text.IO.Utf8`.++_Note: it is best to import these modules qualified._++_Note: there is no `System.IO.Utf8.writeFile`._++If, for some reason, you really need to use `withFile`/`openFile` from `base`,+do the same as in the previous step.+++## Contributing++If you encounter any issues when using this library or have improvement ideas,+please open report in issue on GitHub. You are also very welcome to submit+pull request, if you feel like doing so.+++## License++[MPL-2.0] © [Serokell]++[MPL-2.0]: https://spdx.org/licenses/MPL-2.0.html+[Serokell]: https://serokell.io/
+ lib/Data/Text/IO/Utf8.hs view
@@ -0,0 +1,35 @@+{- SPDX-FileCopyrightText: 2020 Serokell <https://serokell.io/>+ -+ - SPDX-License-Identifier: MPL-2.0+ -}++-- | "Data.Text.IO" for the modern world.+--+-- Wrappers around simple file reading/writing functions from the+-- @text@ package that reset the handle encoding to UTF-8.+module Data.Text.IO.Utf8+ ( readFile+ , writeFile+ ) where++import Prelude hiding (readFile, writeFile)++import Control.Exception.Safe (MonadMask)+import Control.Monad.IO.Class (MonadIO, liftIO)+import Data.Text (Text)++import qualified Data.Text.IO as T+import qualified System.IO as IO++import qualified System.IO.Utf8 as Utf8+++-- | Like 'T.readFile', but assumes the file is encoded in UTF-8, regardless+-- of the current locale.+readFile :: MonadIO m => IO.FilePath -> m Text+readFile path = Utf8.openFile path IO.ReadMode >>= liftIO . T.hGetContents++-- | Like 'T.writeFile', but encodes the data in UTF-8, regardless+-- of the current locale.+writeFile :: (MonadIO m, MonadMask m) => IO.FilePath -> Text -> m ()+writeFile path = Utf8.withFile path IO.WriteMode . (liftIO .) . flip T.hPutStr
+ lib/Data/Text/Lazy/IO/Utf8.hs view
@@ -0,0 +1,35 @@+{- SPDX-FileCopyrightText: 2020 Serokell <https://serokell.io/>+ -+ - SPDX-License-Identifier: MPL-2.0+ -}++-- | "Data.Text.Lazy.IO" for the modern world.+--+-- Wrappers around simple file reading/writing functions from the+-- @text@ package that reset the handle encoding to UTF-8.+module Data.Text.Lazy.IO.Utf8+ ( readFile+ , writeFile+ ) where++import Prelude hiding (readFile, writeFile)++import Control.Exception.Safe (MonadMask)+import Control.Monad.IO.Class (MonadIO, liftIO)+import Data.Text.Lazy (Text)++import qualified Data.Text.Lazy.IO as T+import qualified System.IO as IO++import qualified System.IO.Utf8 as Utf8+++-- | Like @readFile@, but assumes the file is encoded in UTF-8, regardless+-- of the current locale.+readFile :: MonadIO m => IO.FilePath -> m Text+readFile path = Utf8.openFile path IO.ReadMode >>= liftIO . T.hGetContents++-- | Like @writeFile@, but encodes the data in UTF-8, regardless+-- of the current locale.+writeFile :: (MonadIO m, MonadMask m) => IO.FilePath -> Text -> m ()+writeFile path = Utf8.withFile path IO.WriteMode . (liftIO .) . flip T.hPutStr
+ lib/Main/Utf8.hs view
@@ -0,0 +1,82 @@+{- SPDX-FileCopyrightText: 2020 Serokell <https://serokell.io/>+ -+ - SPDX-License-Identifier: MPL-2.0+ -}++-----------------------------------------------------------------------------+-- |+--+-- Functions in this module will help you make your /executable/ work+-- correctly with encodings of text files and standard handles.+--+-- /Note: if you are developing a library, see "System.IO.Utf8"./+--+-- = Quick start+--+-- Wrap a call to 'withUtf8' around your @main@:+--+-- @+-- import Main.Utf8 (withUtf8)+--+-- main :: IO ()+-- main = 'withUtf8' $ do+-- putStrLn "Hello, мир!"+-- @+--+-- Basically, this is all you have to do for a program that uses+-- @stdin@ and @stdout@ to interact with the user. However, some+-- programs read input from and write output to files and,+-- at the same time, allow the user to redirect @stdin@ and @stdout@+-- instead of providing explicit file names.+--+-- If this is the case for your executable, you should also wrap+-- @Utf8.@'System.IO.Utf8.withHandle' around the code that passes+-- the handle to a third-party library. It is not necessary to do+-- when passing it to your own library, assuming that it follows+-- the recommendations from the documentation of "System.IO.Utf8".+module Main.Utf8+ ( withUtf8+ , withStdTerminalHandles+ ) where++import Control.Exception.Safe (MonadMask, bracket)+import Control.Monad.IO.Class (MonadIO, liftIO)+import GHC.IO.Encoding (getLocaleEncoding, setLocaleEncoding, utf8)+import System.IO (stderr, stdin, stdout)++import System.IO.Utf8 (withTerminalHandle)+++-- | Make standard handles safe to write anything to them and change+-- program-global default file handle encoding to UTF-8.+--+-- This function will:+--+-- 1. Adjust the encoding of 'stdin', 'stdout', and 'stderr' to+-- enable transliteration, like 'withStdTerminalHandles' does.+-- 2. Call 'setLocaleEncoding' to change the program-global locale+-- encoding to UTF-8.+-- 3. Undo everything when the wrapped action finishes.+withUtf8 :: (MonadIO m, MonadMask m) => m r -> m r+withUtf8 act = withStdTerminalHandles $+ bracket+ (liftIO $ getLocaleEncoding <* setLocaleEncoding utf8)+ (liftIO . setLocaleEncoding)+ (\_ -> act)++-- | Make standard handles safe to write anything to them.+--+-- This function will for each of 'stdin', 'stdout', 'stderr' do:+--+-- 1. Tweak the existing encoding so that unrepresentable characters+-- will get approximated (aka transliterated) by visually similar+-- ones or question marks.+-- 2. Restore the original encoding when the wrapped action finishes.+--+-- Use this function only if you do not want to change the program-global+-- locale encoding. Otherwise prefer 'withUtf8'.+withStdTerminalHandles :: (MonadIO m, MonadMask m) => m r -> m r+withStdTerminalHandles+ = withTerminalHandle stdin+ . withTerminalHandle stdout+ . withTerminalHandle stderr
+ lib/System/IO/Utf8.hs view
@@ -0,0 +1,170 @@+{- SPDX-FileCopyrightText: 2020 Serokell <https://serokell.io/>+ -+ - SPDX-License-Identifier: MPL-2.0+ -}++{-# LANGUAGE LambdaCase #-}++-----------------------------------------------------------------------------+-- |+--+-- Standard IO functions assume that the character encoding of the data+-- they read or write is the same as the one used by current locale. In many+-- situtations this assumption is wrong, as tools work with files, and+-- files nowadays are mostly UTF-8 encoded, regardless of the locale.+-- Therefore, it is almost always a good idea to switch the encoding of+-- file handles to UTF-8.+--+-- The same applies to standard input, output, and error handles. However,+-- there is an edge-case: if they are attached to a terminal, and the+-- encoding is not UTF-8, using UTF-8 might actually be unsafe.+--+-- If you are developing an executable, in most cases, it is enough to+-- configure the environment accordingly on program start, see the+-- "Main.Utf8" for functions that help with this.+-- However, if you are a library author, you should avoid modifying the+-- global environment.+--+-- = Quick start+--+-- == Opening new files+--+-- If you need to open a text file, use @Utf8.@'withFile'+-- (or @Utf8.@'openFile'). These will not only open the file, but also+-- set the handle’s encoding to UTF-8, regardless of the user’s locale.+--+-- == Working with existing handles+--+-- Suppose you are creating a function which produces some text and writes+-- it to a file handle that is passed to it from the outside.+-- Ask yourself this question: do I want to encode this text in UTF-8+-- or using the encoding from the user’s locale?+--+-- In many cases this question is easy to answer. For example, if your+-- function produces Haskell code, then you always want it in UTF-8,+-- because that is what all other tools (including GHC) expect.+--+-- In some cases it is not that clear. What you can do then is consider+-- what the user is going to do with the data produced.+-- If it is, primarily, meant to be displayed on their screen and then+-- forgotten, you don’t need UTF-8. On the other hand, if it is meant+-- to be saved somewhere and then used or edited by other tools, then+-- you need UTF-8.+--+-- If you decided that your function needs to try to switch the handle+-- to UTF-8, it is very easy to achieve:+--+-- @+-- import qualified System.IO.Utf8 as Utf8+--+-- writeData :: 'IO.Handle' -> InputDataType -> IO ()+-- writeData hOut inData = Utf8.'withHandle' hOut $ do+-- {- ... write the data ... -}+-- @+--+-- If you decided that you don’t need to try to switch it to UTF-8,+-- replace @withHandle@ with 'withTerminalHandle' to only make the+-- handle safe to write to without runtime errors.+module System.IO.Utf8+ ( withHandle+ , withTerminalHandle++ , setHandleEncoding+ , setTerminalHandleEncoding++ , openFile+ , withFile+ ) where++import Control.Exception.Safe (MonadMask, bracket)+import Control.Monad.IO.Class (MonadIO, liftIO)+import Data.Functor (void)+import GHC.IO.Encoding (mkTextEncoding, utf8)++import qualified System.IO as IO++import System.IO.Utf8.Internal (EncodingAction (..), chooseBestEnc)+++type EncRestoreAction m = IO.Handle -> m ()++-- | Set the best available UTF-8-compatible encoding for the handle.+-- Returns the action that will restore the previous one.+--+-- If the handle is in binary mode, does nothing.+-- If the handle is not attached to a terminal, sets UTF-8.+-- Otherwise, keeps its current encoding, but augments it to transliterate+-- unsupported characters.+hSetBestUtf8Enc+ :: MonadIO m+ => (IO.Handle -> IO Bool)+ -> IO.Handle+ -> m (EncRestoreAction m)+hSetBestUtf8Enc hIsTerm h = liftIO $ do+ IO.hGetEncoding h >>= chooseBestEnc h hIsTerm >>= \case+ Keep -> pure (\_ -> pure ())+ ChangeFromTo enc newName -> do+ mkTextEncoding newName >>= IO.hSetEncoding h+ pure $ liftIO . flip IO.hSetEncoding enc+++-- | Set handle encoding to the best possible.+--+-- See 'withHandle' for description and prefer it, if possible.+setHandleEncoding :: MonadIO m => IO.Handle -> m ()+setHandleEncoding = liftIO . void . hSetBestUtf8Enc IO.hIsTerminalDevice++-- | Temporarily set handle encoding to the best possible.+--+-- “Best possible” means UTF-8, unless the handle points to a terminal+-- device, in which case the encoding will be left the same, but tweaked+-- to approximate unencodable characters.+--+-- This function is safe to call on handles open in binary mode and it will+-- do nothing on them.+--+-- To sum up:+--+-- * If the handle is in binary mode, do nothing.+-- * If the handle points to a terminal device, act like 'withTerminalHandle'.+-- * For regular files always choose UTF-8, of course.+withHandle :: (MonadIO m, MonadMask m) => IO.Handle -> m r -> m r+withHandle h = bracket (hSetBestUtf8Enc IO.hIsTerminalDevice h) ($ h) . const++-- | Make a handle safe to write any text to.+--+-- See 'withTerminalHandle' for description and prefer it, if possible.+setTerminalHandleEncoding :: MonadIO m => IO.Handle -> m ()+setTerminalHandleEncoding = liftIO . void . hSetBestUtf8Enc (const $ pure True)++-- | Temporarily make a handle safe to write any text to.+--+-- If the handle is not using UTF-8, adjust the encoding to remain the same+-- as before, but approximate unencodable characters. When the action is done,+-- restore it back to the previous one.+--+-- Use this function only if you are sure you want to treat this handle as+-- a terminal (that is, you will be using it to interact with the user+-- and to write user-visible messages, rather than something that can+-- be reasonable expected to go to a file).+--+-- This function is safe to call on handles open in binary mode and it will+-- do nothing on them.+withTerminalHandle :: (MonadIO m, MonadMask m) => IO.Handle -> m r -> m r+withTerminalHandle h = bracket (hSetBestUtf8Enc (const $ pure True) h) ($ h) . const+++-- | Like 'System.IO.openFile', but sets the file encoding to UTF-8, regardless+-- of the current locale.+openFile :: MonadIO m => IO.FilePath -> IO.IOMode -> m IO.Handle+openFile path mode = liftIO $ do+ h <- IO.openFile path mode+ IO.hSetEncoding h utf8+ pure h++-- | Like 'System.IO.withFile', but sets the file encoding to UTF-8, regardless+-- of the current locale.+withFile+ :: (MonadIO m, MonadMask m)+ => IO.FilePath -> IO.IOMode -> (IO.Handle -> m r) -> m r+withFile path mode = bracket (openFile path mode) (liftIO . IO.hClose)
+ lib/System/IO/Utf8/Internal.hs view
@@ -0,0 +1,86 @@+{- SPDX-FileCopyrightText: 2020 Serokell <https://serokell.io/>+ -+ - SPDX-License-Identifier: MPL-2.0+ -}++{-# LANGUAGE LambdaCase #-}+{-# LANGUAGE ViewPatterns #-}++-- | Internal functions that implement encoding selection logic.+module System.IO.Utf8.Internal+ ( EncodingAction (..)++ , chooseBestEncPure+ , chooseBestEnc+ ) where++import Data.List (isSuffixOf)+import GHC.IO.Encoding (TextEncoding, textEncodingName, utf8)++import qualified System.IO as IO+++-- | What to do with the encoding of the handle.+--+-- We return new encoding as string to simplify testing, because,+-- it turns out, when you make an encoding with some name, the+-- resulting encoding can have a different name.+--+-- The second constructor also contains the old encoding as a proof+-- that the encoding is set to a new one only if there was a previous+-- one in the first place. You probably think I am crazy, but I am not,+-- it is just the simplest way to make the @hSetBestUtf8Enc@ easy to write.+data EncodingAction+ = Keep+ -- ^ Do nothing.+ | ChangeFromTo TextEncoding String+ -- ^ Change the first encoding to the second.++-- | Pure version of 'chooseBestEnc'.+--+-- This function is not actually used in the library. It exists only+-- for documentation purposes to demonstrate the logic.+-- It is also used in tests to verify that the logic implemented is+-- indeed this.+chooseBestEncPure+ :: Bool -- ^ Is a terminal device?+ -> Maybe String -- ^ Previous encoding name.+ -> Maybe String+-- Never touch handles in binary mode.+chooseBestEncPure _ Nothing = Nothing+-- Already UTF-8, that's cool.+chooseBestEncPure _ (Just "UTF-8") = Nothing+-- Handle in text mode that is not a terminal.+chooseBestEncPure False _ = Just "UTF-8"+chooseBestEncPure True (Just name)+ -- Be idempotent.+ | "//TRANSLIT" `isSuffixOf` name = Nothing+ | otherwise = Just $ name ++ "//TRANSLIT"++-- | Choose the best encoding for a file handle.+--+-- This function implements the same logic as 'chooseBestEncPure',+-- but in a way that is more optimal in terms of IO queries. In+-- particular:+--+-- 1. It receives both a handle and its current encoding, because the+-- calling function will, most likely, need to know the current encoding+-- (e.g. to be able to restore it), so we avoid repeating the query.+-- 2. It first checks for the cases where it doesn't care whether the device+-- is a terminal or not, so the query will be made only if really necessary.+chooseBestEnc+ :: IO.Handle -- ^ Handle to choose encoding for.+ -> (IO.Handle -> IO Bool) -- ^ hIsTerminalDevice.+ -> Maybe TextEncoding -- ^ Current encoding.+ -> IO EncodingAction+chooseBestEnc _ _ Nothing = pure Keep+chooseBestEnc h hIsTerm (Just enc) = case textEncodingName enc of+ "UTF-8" -> pure Keep+ name+ -- XXX: The first branch is actually never used, because the encoding+ -- loses the @//TRANSLIT@ suffix after it is being created.+ -- TODO: Find a way to detect that the encoding is already trasliterating.+ | "//TRANSLIT" `isSuffixOf` name -> pure Keep+ | otherwise -> hIsTerm h >>= \case+ False -> pure $ ChangeFromTo enc (textEncodingName utf8)+ True -> pure $ ChangeFromTo enc (name ++ "//TRANSLIT")
+ test/Test.hs view
@@ -0,0 +1,22 @@+{- SPDX-FileCopyrightText: 2020 Serokell <https://serokell.io/>+ -+ - SPDX-License-Identifier: MPL-2.0+ -}++module Main+ ( main+ ) where++import GHC.IO.Encoding (mkTextEncoding, setLocaleEncoding)+import Main.Utf8 (withUtf8)+import Test.Tasty (defaultMain)++import Tree (tests)++main :: IO ()+main = withUtf8 $ do+ -- The issue only shows up when current locale encoding is ASCII.+ -- Realistically, very often when running this test this will not be+ -- the case, so we unset locale encoding manually.+ mkTextEncoding "ASCII" >>= setLocaleEncoding+ tests >>= defaultMain
+ test/Test/Utf8/Choice.hs view
@@ -0,0 +1,106 @@+{- SPDX-FileCopyrightText: 2020 Serokell <https://serokell.io/>+ -+ - SPDX-License-Identifier: MPL-2.0+ -}++module Test.Utf8.Choice where++import GHC.IO.Encoding (char8, latin1, textEncodingName, utf8)+import System.IO.Temp (withSystemTempFile)++import qualified System.IO as IO++import Test.HUnit (Assertion, assertFailure)+import Test.Tasty.HUnit ((@=?))++import System.IO (hIsTerminalDevice)+import System.IO.Utf8.Internal (EncodingAction (..), chooseBestEnc, chooseBestEncPure)++import qualified System.IO.Utf8 as Utf8++import Test.Util (withTerminalIn, withTmpFileIn)+++-- | Compare @chooseBestEnc@ with @chooseBestEncPure@ on the handle.+verifyOn :: IO.Handle -> Assertion+verifyOn h = do+ isTerm <- IO.hIsTerminalDevice h+ enc <- IO.hGetEncoding h++ let pureResult = chooseBestEncPure isTerm (textEncodingName <$> enc)+ realResult <- chooseBestEnc h hIsTerminalDevice enc++ case (pureResult, realResult) of+ (Nothing, Keep) -> pure ()+ (Just pureName, ChangeFromTo _ realName) -> do+ pureName @=? realName+ (_, _) -> assertFailure "Action mismatch"+++unit_pure_binary_file :: Assertion+unit_pure_binary_file =+ Nothing @=? chooseBestEncPure False Nothing++unit_pure_binary_term :: Assertion+unit_pure_binary_term =+ Nothing @=? chooseBestEncPure True Nothing++unit_pure_term_ascii :: Assertion+unit_pure_term_ascii =+ Just "ASCII//TRANSLIT" @=? chooseBestEncPure True (Just "ASCII")++unit_pure_term_utf8 :: Assertion+unit_pure_term_utf8 =+ Nothing @=? chooseBestEncPure True (Just "UTF-8")++unit_pure_file_ascii :: Assertion+unit_pure_file_ascii =+ Just "UTF-8" @=? chooseBestEncPure False (Just "ASCII")++unit_pure_file_utf8 :: Assertion+unit_pure_file_utf8 =+ Nothing @=? chooseBestEncPure False (Just "UTF-8")++unit_pure_idempotent :: Assertion+unit_pure_idempotent =+ Nothing @=? chooseBestEncPure True (Just "ASCII//TRANSLIT")+++unit_term_binary :: Assertion+unit_term_binary = withTerminalIn utf8 $ \h -> do+ IO.hSetBinaryMode h True+ verifyOn h++unit_term_utf8 :: Assertion+unit_term_utf8 = withTerminalIn utf8 verifyOn++unit_term_char8 :: Assertion+unit_term_char8 = withTerminalIn char8 verifyOn++unit_term_latin1 :: Assertion+unit_term_latin1 = withTerminalIn latin1 verifyOn+++unit_file_binary :: Assertion+unit_file_binary = withSystemTempFile "utf8.bin" $ \_ h -> do+ IO.hSetBinaryMode h True+ verifyOn h++unit_file_utf8 :: Assertion+unit_file_utf8 = withTmpFileIn utf8 verifyOn++unit_file_char8 :: Assertion+unit_file_char8 = withTmpFileIn char8 verifyOn++unit_file_latin1 :: Assertion+unit_file_latin1 = withTmpFileIn latin1 verifyOn++unit_term_idempotent :: Assertion+unit_term_idempotent = withTerminalIn char8 $ \h -> do+ Just enc <- IO.hGetEncoding h+ "ISO-8859-1" @=? textEncodingName enc -- sanity check+ Utf8.withHandle h $ do+ -- XXX: Actually not true, there is no suffix in the name+ -- Just enc' <- IO.hGetEncoding h+ -- "ISO-8859-1//TRANSLIT" @=? textEncodingName enc' -- sanity check+ verifyOn h
+ test/Test/Utf8/ReadWrite.hs view
@@ -0,0 +1,84 @@+{- SPDX-FileCopyrightText: 2020 Serokell <https://serokell.io/>+ -+ - SPDX-License-Identifier: MPL-2.0+ -}++{-# LANGUAGE LambdaCase #-}+{-# LANGUAGE OverloadedStrings #-}+{-# LANGUAGE ScopedTypeVariables #-}++module Test.Utf8.ReadWrite where++import Control.DeepSeq (force)+import Control.Exception (IOException, evaluate)+import Control.Exception.Safe (MonadMask, try)+import Control.Monad ((>=>))+import Control.Monad.IO.Class (MonadIO, liftIO)+import Data.Text (Text)+import GHC.IO.Encoding (utf8)+import System.IO (FilePath)+import System.IO.Temp (withSystemTempFile)++import Hedgehog (Property, (===), forAll, property)+import Test.HUnit (Assertion, assertFailure)++import qualified Data.Text.IO as T+import qualified Data.Text.IO.Utf8 as Utf8+import qualified System.IO as IO+import qualified System.IO.Utf8 as Utf8++import qualified Hedgehog.Gen as G+import qualified Hedgehog.Range as R+++-- | Helper that writes Text to a temp file.+withTestFile :: (MonadIO m, MonadMask m) => Text -> (FilePath -> m r) -> m r+withTestFile t act = withSystemTempFile "utf8.txt" $ \fp h -> do+ liftIO $ IO.hSetEncoding h utf8 *> T.hPutStr h t *> IO.hClose h+ act fp+++-- | Sanity check.+unit_standard_readFile_fails :: Assertion+unit_standard_readFile_fails = withTestFile str $ \fp ->+ try (T.readFile fp) >>= \case+ Right _ -> assertFailure "Standard `readFile` should fail."+ Left (_ :: IOException) -> pure ()+ where+ -- We use an escape here because otherwise _both_ tasty-discover+ -- and hedgehog fail when they are reading this file for their+ -- discovery or nice error reporting purposes... I know, right?+ str = "doma\285e"+++hprop_readFile :: Property+hprop_readFile = property $ do+ str <- forAll $ G.text (R.linear 0 1000) G.unicode+ str' <- liftIO $ withTestFile str (Utf8.readFile >=> evaluate . force)+ str === str'++hprop_writeFile :: Property+hprop_writeFile = property $ do+ str <- forAll $ G.text (R.linear 0 1000) G.unicode+ liftIO $ withTestFile str (flip Utf8.writeFile str)++hprop_openFile :: Property+hprop_openFile = property $ do+ str <- forAll $ G.text (R.linear 0 1000) G.unicode+ str' <- liftIO $ withTestFile str $ \fp -> do+ h <- Utf8.openFile fp IO.ReadMode+ res <- T.hGetContents h+ res' <- evaluate . force $ res+ IO.hClose h+ pure res'+ str === str'++hprop_withFile :: Property+hprop_withFile = property $ do+ str <- forAll $ G.text (R.linear 0 1000) G.unicode+ str' <- liftIO $ withTestFile str $ \fp ->+ Utf8.withFile fp IO.ReadMode $ \h -> do+ res <- T.hGetContents h+ res' <- evaluate . force $ res+ pure res'+ str === str'
+ test/Test/Utf8/Set.hs view
@@ -0,0 +1,64 @@+{- SPDX-FileCopyrightText: 2020 Serokell <https://serokell.io/>+ -+ - SPDX-License-Identifier: MPL-2.0+ -}++module Test.Utf8.Set where++import GHC.IO.Encoding (char8, latin1, textEncodingName, utf8)++import qualified System.IO as IO++import Test.HUnit (Assertion)+import Test.Tasty.HUnit ((@=?))++import System.IO (hIsTerminalDevice)+import System.IO.Utf8.Internal (EncodingAction (..), chooseBestEnc)++import qualified System.IO.Utf8 as Utf8++import Test.Util (withTmpFileIn)+++-- | Check that the encoding was actually set to the right one.+verifyOn :: IO.Handle -> Assertion+verifyOn h = do+ menc <- IO.hGetEncoding h+ act <- chooseBestEnc h hIsTerminalDevice menc++ Utf8.withHandle h $ do+ menc' <- IO.hGetEncoding h+ case act of+ Keep ->+ fmap textEncodingName menc @=? fmap textEncodingName menc'+ ChangeFromTo enc enc' -> do+ fmap textEncodingName menc @=? Just (textEncodingName enc)+ Just enc' @=? fmap textEncodingName menc'++ -- Check that it restores+ menc' <- IO.hGetEncoding h+ fmap textEncodingName menc @=? fmap textEncodingName menc'+++-- XXX: This group of tests does not actually work, because there is+-- no way to check that the newly set encoding is transliterating+-- (see the comment in @chooseBestEnc@).++-- unit_term_utf8 :: Assertion+-- unit_term_utf8 = withTerminalIn utf8 verifyOn+--+-- unit_term_char8 :: Assertion+-- unit_term_char8 = withTerminalIn char8 verifyOn+--+-- unit_term_latin1 :: Assertion+-- unit_term_latin1 = withTerminalIn latin1 verifyOn+++unit_file_utf8 :: Assertion+unit_file_utf8 = withTmpFileIn utf8 verifyOn++unit_file_char8 :: Assertion+unit_file_char8 = withTmpFileIn char8 verifyOn++unit_file_latin1 :: Assertion+unit_file_latin1 = withTmpFileIn latin1 verifyOn
+ test/Test/Util.hs view
@@ -0,0 +1,37 @@+{- SPDX-FileCopyrightText: 2020 Serokell <https://serokell.io/>+ -+ - SPDX-License-Identifier: MPL-2.0+ -}++-- | Testing helpers.+module Test.Util+ ( withTerminalIn+ , withTmpFileIn+ ) where++import Control.Exception.Safe (bracket, finally)+import Control.Monad (void)+import GHC.IO.Encoding (TextEncoding)+import System.IO.Temp (withSystemTempFile)+import System.Posix.IO (closeFd, fdToHandle, handleToFd)+import System.Posix.Terminal (openPseudoTerminal)++import qualified System.IO as IO+++-- | Make a new pseudo-terminal with the given encoding.+withTerminalIn :: TextEncoding -> (IO.Handle -> IO r) -> IO r+withTerminalIn enc act = bracket+ openPseudoTerminal+ (\(fd1, fd2) -> closeFd fd2 `finally` closeFd fd1)+ (\(fd, _) -> bracket+ (fdToHandle fd)+ (void . handleToFd) -- just release the fd from the handle+ (\h -> IO.hSetEncoding h enc *> act h)+ )++-- | Make a temp file with the given encoding.+withTmpFileIn :: TextEncoding -> (IO.Handle -> IO r) -> IO r+withTmpFileIn enc act = withSystemTempFile "utf8.txt" $ \_ h -> do+ IO.hSetEncoding h enc+ act h
+ test/Tree.hs view
@@ -0,0 +1,6 @@+{- SPDX-FileCopyrightText: 2020 Serokell <https://serokell.io/>+ -+ - SPDX-License-Identifier: MPL-2.0+ -}++{-# OPTIONS_GHC -F -pgmF tasty-discover -optF --tree-display -optF --generated-module -optF Tree #-}
+ with-utf8.cabal view
@@ -0,0 +1,86 @@+cabal-version: 1.18++-- This file has been generated from package.yaml by hpack version 0.33.0.+--+-- see: https://github.com/sol/hpack+--+-- hash: 9386db1f384343c819e3ea257cc30e49b04d6d410978363c12d47d6e027182a8++name: with-utf8+version: 1.0.0.0+synopsis: Get your IO right on the first try+description: This minimalistic library helps you navigate the world of text encodings+ avoiding @invalid argument (invalid byte sequence)@+ and @invalid argument (invalid character)@ in runtime.+ .+ The two most important modules are:+ .+ * "Main.Utf8"+ * "System.IO.Utf8"+ .+ See <https://serokell.io/blog/haskell-with-utf8 this blog post> for why this+ library exists and what exactly it does.+category: IO+homepage: https://github.com/serokell/haskell-with-utf8#readme+bug-reports: https://github.com/serokell/haskell-with-utf8/issues+author: Kirill Elagin <kirelagin@serokell.io>+maintainer: Kirill Elagin <kirelagin@serokell.io>+copyright: 2020 Serokell+license: MPL-2.0+license-file: LICENSES/MPL-2.0.txt+build-type: Simple+extra-doc-files:+ CHANGELOG.md+ README.md++source-repository head+ type: git+ location: https://github.com/serokell/haskell-with-utf8++library+ exposed-modules:+ Data.Text.IO.Utf8+ Data.Text.Lazy.IO.Utf8+ Main.Utf8+ System.IO.Utf8+ System.IO.Utf8.Internal+ other-modules:+ Paths_with_utf8+ hs-source-dirs:+ lib+ ghc-options: -Wall -Wcompat -Wincomplete-record-updates -Wincomplete-uni-patterns -Wredundant-constraints+ build-depends:+ base >=4.10 && <4.14+ , safe-exceptions >=0.1 && <0.2+ , text >=0.7 && <1.3+ default-language: Haskell2010++test-suite with-utf8-test+ type: exitcode-stdio-1.0+ main-is: Test.hs+ other-modules:+ Test.Utf8.Choice+ Test.Utf8.ReadWrite+ Test.Utf8.Set+ Test.Util+ Tree+ Paths_with_utf8+ hs-source-dirs:+ test+ ghc-options: -Wall -Wcompat -Wincomplete-record-updates -Wincomplete-uni-patterns -Wredundant-constraints+ build-tool-depends:+ tasty-discover:tasty-discover+ build-depends:+ HUnit+ , base+ , deepseq+ , hedgehog+ , safe-exceptions+ , tasty+ , tasty-hedgehog+ , tasty-hunit+ , temporary+ , text+ , unix+ , with-utf8+ default-language: Haskell2010