unicoder (empty) → 0.4.0

raw patch · 16 files changed

+716/−0 lines, 16 filesdep +attoparsecdep +basedep +directorysetup-changed

Dependencies added: attoparsec, base, directory, text

Files

• LICENSE +30/−0
• README.md +99/−0
• Setup.hs +2/−0
• Text/Unicoder.hs +132/−0
• changes.md +33/−0
• data/default.conf +199/−0
• test/di.in +2/−0
• test/di.out +2/−0
• test/mono.in +5/−0
• test/mono.out +5/−0
• test/passthrough.in +4/−0
• test/passthrough.out +4/−0
• test/test.config +4/−0
• test/test.hs +27/−0
• unicoder.cabal +65/−0
• unicoder.hs +103/−0

@@ -0,0 +1,30 @@+Copyright (c) 2014, Zankoku Okuno++All rights reserved.++Redistribution and use in source and binary forms, with or without+modification, are permitted provided that the following conditions are met:++    * Redistributions of source code must retain the above copyright+      notice, this list of conditions and the following disclaimer.++    * Redistributions in binary form must reproduce the above+      copyright notice, this list of conditions and the following+      disclaimer in the documentation and/or other materials provided+      with the distribution.++    * Neither the name of Zankoku Okuno nor the names of other+      contributors may be used to endorse or promote products derived+      from this software without specific prior written permission.++THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

@@ -0,0 +1,99 @@+Unicoder+========++The unicoder reads in a source file and makes replacements in-place. The goal+is to allow ascii interfaces to be able to insert unicode without taking your+hands off the keyboard. This can allow for unicode to be entered into source+code or any other text document you're editing.++Entering unicode is not as easy as typing a special string (default backslash)+followed by an identifier. There's also syntax for wrapping content inside a+pair of unicode strings.+For example, with the default configuration, unicoder turns+`\floor{x} \def \lambda x. (floor x)` into `⌊x⌋ ≡ λ x. (floor x)`.+Admitedly, this is not a great syntax for some kinds of documents (esp. XeLaTeX),+but that's why we've allowed for configuration of each of the special marks as+well as the identifier character set, so Unicoder can be relevant to any type of+text data.++There's a detailed explanation of the unicoder algorithm and configuration on our+[Viewdocs](http://zankoku-okuno.viewdocs.io/unicoder/specs.md). It's fairly likely+that the system can be deduced just from examples, though.++Examples+--------++Assuming a config file that looks like this:++```+\ . { } a-z++lambda λ+pi π+bag ⟅ ⟆+```++we can write this with a normal keyboard:++```+\lambda.x. x + \pi+```++and after unicodizing, we will get:++```+λx. x + π+```++and celebrate the nice, clean lambda-calculus.++Have no fear, however, code such as this:++```+id = \x -> x+newline_period = "\n."+```++Will remain unchanged, as `x` and `n` are not in the config file.++There are also two-part replacements. These take a single (non-nested) argument, transforming++```+\bag{black}+```++into++```+⟅black⟆+```++You can also use each half of a two-part replacement individually. This is especially+usefule for nesting, but also when you simply have argument-close marks in the argument:++```+\{bag {} \}bag+```++becomes++```+⟅ {} ⟆+```++Pitfalls+--------++Even in something as simple as this, you may want to be aware of a few facts:++ * Beware of adding names like `n` or `t` in your config file. If you are using +   a language that isn't esoteric, you will probably change the meaning of your +   code.+ * It _is_ still possible to mess up strings. For example, `"\neq"` → `"≠"` +   instead of being equivalent to `"\n" ++ "eq"`. I conjecture that there is +   no way to solve this problem without sacrificing idempotence.+ * I've made little attempt to ensure safety, other than using Haskell. Make +   backups if you are wary (and your editor doesn't already).++Thankfully, the pitfalls are realistically enumerable.+

@@ -0,0 +1,2 @@+import Distribution.Simple+main = defaultMain

@@ -0,0 +1,132 @@+{-# LANGUAGE OverloadedStrings #-}+module Text.Unicoder (+      unicodize+    , unicodizeStr+    , Config+    , parseConfigFile+    ) where++import System.IO++import Data.Text (Text)+import qualified Data.Text as T+import qualified Data.Text.IO as T+import Data.Attoparsec.Text+import Data.Attoparsec.Combinator++import Data.Maybe+import Data.Either+import Data.Monoid+import Control.Applicative+import Control.Monad+++unicodize :: Config -> Text -> Text+unicodize config input = case parseOnly (xform config) input of+    Left err -> error "unicoder: internal error"+    Right val -> val++unicodizeStr :: Config -> String -> String+unicodizeStr config = T.unpack . unicodize config . T.pack+++data Config = Config { _idChars      :: Char -> Bool+                     , _beginMark    :: Text+                     , _endMark      :: Maybe Text+                     , _betweenMarks :: Maybe (Text, Text)+                     , _macros0      :: [(Text, Text)]+                     , _macros1      :: [(Text, (Text, Text))]+                     }++parseConfigFile :: FilePath -> IO (Maybe Config)+parseConfigFile path = withFile path ReadMode $ \fp -> do+    xs <- filter (not . T.null) . T.lines <$> T.hGetContents fp+    return $ case xs of+        [] -> Nothing+        (lexer:raw_macros) -> do+            --TODO configure begin mark+            emptyConfig <- case filter (not . T.null) $ T.splitOn " " lexer of+                [idChars] -> return+                    Config { _idChars = inClass (T.unpack idChars)+                           , _beginMark = "\\"+                           , _endMark = Nothing+                           , _betweenMarks = Nothing+                           , _macros0 = [], _macros1 = []+                           }+                [begin, idChars] -> return+                    Config { _idChars = inClass (T.unpack idChars)+                           , _beginMark = begin+                           , _endMark = Nothing+                           , _betweenMarks = Nothing+                           , _macros0 = [], _macros1 = []+                           }+                [begin, end, idChars] -> return+                    Config { _idChars = inClass (T.unpack idChars)+                           , _beginMark = begin+                           , _endMark = Just end+                           , _betweenMarks = Nothing+                           , _macros0 = [], _macros1 = []+                           }+                [begin, end, open, close, idChars] -> return+                    Config { _idChars = inClass (T.unpack idChars)+                           , _beginMark = begin+                           , _endMark = Just end+                           , _betweenMarks = Just (open, close)+                           , _macros0 = [], _macros1 = []+                           }+                _ -> Nothing+            let (macros0, macros1) = partitionEithers . catMaybes $ parseMacro <$> raw_macros+            return $ emptyConfig { _macros0 = macros0, _macros1 = macros1 }++parseMacro :: T.Text -> Maybe (Either (Text, Text) (Text, (Text, Text)))+parseMacro input = case T.words input of+    [k, v] -> Just $ Left (k, v)+    [k, v1, v2] -> Just $ Right (k, (v1, v2))+    _ -> Nothing+++xform :: Config -> Parser Text+xform config = mconcat <$> many (passthrough <|> macro <|> strayBegin) <* endOfInput+    where+    (beginStr, beginChr) = (_beginMark config, T.head beginStr)+    passthrough = takeWhile1 (/= beginChr)+    macro = do+        string beginStr+        full <|> half+    strayBegin = T.singleton <$> char beginChr+    full = do+        name <- takeWhile1 (_idChars config)+        mono name <|> di name+        where+        mono name = do+            replace <- name `lookupM` _macros0 config+            endMark+            return replace+        di name = do+            (open, close) <- betweenMarks+            (rOpen, rClose) <- name `lookupM` _macros1 config+            string open+            inner <- T.pack <$> anyChar `manyTill` string close+            return $ rOpen <> inner <> rClose+    half = do+        (open, close) <- betweenMarks+        which <- (const fst <$> string open) <|> (const snd <$> string close)+        name <- takeWhile1 (_idChars config)+        replace <- which <$> name `lookupM` _macros1 config+        endMark+        return replace+    endMark = case _endMark config of+        Nothing -> return ()+        Just end -> option () $ void (string end)+    betweenMarks = case _betweenMarks config of+        Nothing -> fail "" +        Just x -> return x+    lookupM k v = maybe (fail "") return (k `lookup` v)++{-TODO+a config lint +    characters don't appear twice in the lexer+    open and end are distinguishable+    macros are defined only using idChars+    macro lines are word-length 2 or 3+-}

@@ -0,0 +1,33 @@+Changes+=======++v0.4.0+------+ * Cabalized.+ * Config file location determined by cabal.+ * Split major functions into a library.+ * Allow two-part replacements.+ * No more tmp file needed.++v0.3.1+------+ * Use config files from locations other than `/etc/zanoku-okuno/unicoder/`.+ * Use a `.unicoder.tmp` extension instead of `.tmp` so that there's less+   chance of name-collision with the tempfile.++v0.3.0+------+ * Each configuration file can specify separator string and valid name+   characters.+ * More than one configuration file can be stored and selected between on the +   command line.+ * The role of `symbols.conf` is now dealt with by `default.conf`.+ * No more extra newline at end of file++v0.2.0+------++ * There are no longer any failure modes of the parser. One could probably do a +   formal proof that given finite input and no OS errors, unicoder will +   complete without failure in a finite time.+ * No special handling of strings. Unicoder is therefore now language-agnostic.

@@ -0,0 +1,199 @@+\ . { } a-zA-Z~!%^&*_=+<>/?|-+++l           λ+L           Λ+A           ∀+E           ∃+def         ≡+is          ≡+qq          ⌜+uq          ⌞+uqs         ⌟+p           ′+hole        □+++prime       ′+ellipsis    …+ldots       …+proportion  ∝+ccw         ↺+cw          ↻++of          ∘+to          →+To          ⇒+from        ←+maps        ↦+unmaps      ↤+rharpoon    ⇀+lharpoon    ↼+up          ↑+down        ↓+Up          ⇑+Down        ⇓+hookto      ↪+hookfrom    ↩+->          →+=>          ⇒+<-          ←+|->         ↦+-/>         ↛+</-         ↚+~>          ↝+<~          ↜++Brack       +angle       ‹ ›+Angle       « »+chevron     ⟨ ⟩+shell       ❲ ❳+Shell       ⟬ ⟭+bag         ⟅ ⟆+floor       ⌊ ⌋+ceil        ⌈ ⌉+quine       ⌜ ⌝+lowquine    ⌞ ⌟+<           ⟨ ⟩++cross       ×+dot         ·+div         ÷+star        ⋆+sqrt        √+oplus       ⊕+ominus      ⊖+otimes      ⊗+odiv        ⊘++inf         ∞+nabla       ∇+del         ∂+integral    ∫++neq         ≠+geq         ≥+leq         ≤+approx      ≈+plusminus   ±+minusplus   ∓+!=          ≠+/=          ≠+>=          ≥+<=          ≤++null        ∅+elem        ∈+nelem       ∉+contains    ∋+ncontains   ∌+subset      ⊆+superset    ⊇+psubset     ⊂+psuperset   ⊃+union       ∪+intersect   ∩+Subset      ⊑+Superset    ⊒+pSubset     ⊏+pSuperset   ⊐+squnion     ⊓+sqinter     ⊔+join        ⋈+ljoin       ⋉+rjoin       ⋊++forall      ∀+exists      ∃+nexists     ∄+exists!     ∃!+not         ¬+and         ∧+or          ∨+xor         ⊻+nand        ⊼+nor         ⊽+implies     ⇒+iff         ⇔+equiv       ≡++bot         ⊥+rtack       ⊢+ltack       ⊣+rttack      ⊩+therefore   ∴+because     ∵++circle      ○+square      □+diamond     ◇+triangle    △+ltriangle   ◁+rtriangle   ▷+lozenge     ◊++nat         ℕ+int         ℤ+rational    ℚ+real        ℝ+complex     ℂ++alpha       α+beta        β+gamma       γ+delta       δ+epsilon     ε+zeta        ζ+eta         η+theta       θ+iota        ι+kappa       κ+lambda      λ+mu          μ+nu          ν+xi          ξ+pi          π+rho         ρ+sigma       σ+stigma      ϛ+tau         τ+upsilon     υ+phi         ϕ+chi         χ+psi         ψ+omega       ω+digamma     ϝ+koppa       ϟ+sampi       ϡ++Alpha       +Beta        +Gamma       Γ+Delta       Δ+Epsilon     +Zeta        +Eta         +Theta       Θ+Iota        +Kappa       +Lambda      Λ+Mu          +Nu          +Xi          Ξ+Pi          Π+Rho         +Sigma       Σ+Stigma      Ϛ+Tau         +Upsilon     +Phi         Φ+Chi         +Psi         Ψ+Omega       Ω+Digamma     +Qoppa       Ϟ+Sampi       Ϡ+++mizu        水

@@ -0,0 +1,2 @@+\bag{foo}+\{bag\bag{foo}\}bag.

@@ -0,0 +1,2 @@+⟅foo⟆+⟅⟅foo⟆⟆

@@ -0,0 +1,5 @@+\lambda+\lambda x+\lambda.x+\lambdax+\lambda{x}

@@ -0,0 +1,5 @@+λ+λ x+λx+\lambdax+λ{x}

@@ -0,0 +1,4 @@+foo+{}+\+.

@@ -0,0 +1,4 @@+foo+{}+\+.

@@ -0,0 +1,4 @@+\ . { } a-z++lambda λ+bag ⟅ ⟆

@@ -0,0 +1,27 @@+import System.IO+import System.Exit+import Control.Monad+import qualified Data.Text.IO as T++import Text.Unicoder++main :: IO ()+main = do+	m_config <- parseConfigFile "test/test.config"+	testFile <- case m_config of+		Nothing -> die "Could not parse config file."+		Just config -> return $ testFile config+	let files = ["passthrough", "mono", "di"]+	results <- mapM testFile files+	unless (all id results) $ do+		mapM_ putStrLn $ zipWith (\a b -> a ++ ": " ++ if b then "OK" else "FAILURE") files results+		exitFailure++testFile :: Config -> FilePath -> IO Bool+testFile config path = do+	input <- T.readFile $ "test/" ++ path ++ ".in"+	output <- T.readFile $ "test/" ++ path ++ ".out"+	return $ unicodize config input == output++die :: String -> IO a+die msg = putStrLn msg >> exitFailure

@@ -0,0 +1,65 @@+name:                unicoder+version:             0.4.0+stability:           beta+synopsis:            Make writing in unicode easy.+description:         Unicoder transforms text documents, replacing simple patterns with+                     unicode equivalents. The patterns can be easily configured by the user.+                     This package is especially meant to open the vast and expressive array+                     of unicode identifiers to programmers and language designers, but there's+                     nothing wrong with a technically savvy user putting unicoder to work+                     on documents for human consumption.+                     .+                     With the default settings,+                     .+                     @+                       \\E x. \\A y. \\\<x \\-> y\\\> \\ldots+                       \\l x,y. x \\of x \\of y+                     @+                     .+                     becomes+                     .+                     @+                       ∃x ∀y ⟨x → y⟩ …+                       λ x,y. x ∘ x ∘ y+                     @+                     .+                     Many more possibilities abound just in the default set of characters.+                     Any system of special characters can be made easy to type with a normal+                     keyboard as long as unicode supports it.+license:             BSD3+license-file:        LICENSE+author:              Zankoku Okuno+maintainer:          zankoku.okuno@gmail.com+copyright:           Copyright © 2013, 2014, Okuno Zankoku+category:            Text+build-type:          Simple+cabal-version:       >=1.9.2+data-dir:            data+data-files:          *.conf+extra-source-files:  README.md, changes.md,+                     test/*.in, test/*.out, test/test.config++executable unicoder+  main-is:             unicoder.hs+  -- other-modules:       +  build-depends:       base ==4.6.*,+                       text ==0.11.*,+                       attoparsec >=0.10.0.0,+                       directory ==1.2.*++library+  exposed-modules:     Text.Unicoder+  build-depends:       base ==4.6.*,+                       text ==0.11.*,+                       attoparsec >=0.10.0.0++Test-Suite test-unicoder+    type:              exitcode-stdio-1.0+    main-is:           test/test.hs+    build-depends:     base ==4.6.*,+                       text ==0.11.*,+                       attoparsec >=0.10.0.0++source-repository head+  type: git+  location: https://github.com/Zankoku-Okuno/unicoder.git

@@ -0,0 +1,103 @@+import System.Environment+import System.IO+import System.Directory+import System.Exit+import System.Console.GetOpt+import qualified Data.Text.IO as T+import Control.Monad++import Text.Unicoder++import Paths_unicoder+import Data.Version++symbolFile :: Options -> IO FilePath+symbolFile Options { optConfig = which } = +    if '/' `elem` which+        then return which+        else getDataFileName (which ++ ".conf")++main :: IO ()+main = do+        (opts, args) <- getOptions+        m_config <- parseConfigFile =<< symbolFile opts+        config <- case m_config of+            Nothing -> die "bad configuration file"+            Just config -> return config+        -- TODO -o/--output flag, but how for many files?+        mapM_ (mainLoop config) args+        exitSuccess++mainLoop :: Config -> FilePath -> IO ()    +mainLoop config filename = do+    source <- T.readFile filename+    let result = unicodize config source+    T.writeFile filename result+++die err = do+    putErrLn err+    exitFailure++makePair [a, b] = (a, b)+putErrLn = hPutStrLn stderr+++data Options = Options  { optConfig :: String+                        , optOutput :: Maybe FilePath+                        } deriving (Show)+startOptions = Options  { optConfig = "default"+                        , optOutput = Nothing+                        }++getOptions :: IO (Options, [FilePath])+getOptions = do+    (actions, args, errors) <- return . getOpt Permute options =<< getArgs+    if null errors+      then do+        opts <- foldl (>>=) (return startOptions) actions+        when (null args) $ die "no input files"+        return (opts, args)+      else do+        mapM_ putErrLn errors+        exitFailure++options :: [OptDescr (Options -> IO Options)]+options =+    [ Option "c" ["config"]+        (ReqArg +            (\arg opt -> return opt { optConfig = arg })+            "file")+        "Specify a configuration (usually a programming language)."++    , Option "V" ["version"]+        (NoArg+            (\_ -> do+                putStrLn (showVersion version)+                exitSuccess))+        "Print version."+    , Option "" ["show-config-dir"]+        (NoArg+            (\_ -> do+                putStrLn =<< getDataFileName ""+                exitSuccess))+        "Print directory where configuration files are stored."+    , Option "h" ["help"]+        (NoArg+            (\_ -> do+                prg <- getProgName+                putStrLn "Ease input of unicode glyphs."+                putStrLn "This program is lisenced under the 3-clause BSD lisence."+                putStrLn (usageInfo (prg ++ " [options] files...") options)+                putStrLn "Run on one or more files to replace special sequences of characters (as defined in a config file) with replacements."+                putStrLn "The idea is to put in useful unicode symbols and then use those symbols in source code."+                putStrLn " -- Config Files"+                putStrLn "    Configuration files are stored in `/etc/zankoku-okuno/unicoder`, and end in `.conf`."+                putStrLn "    Passing a file to `-c` that contains a slash will use exactly the file you specify, instead of looking one up."+                putStrLn "    Configuration files consist of a top line and a body. The body is simply a database with two whitespace-separated fields. \+                             \The first is the a name, and the second is the replacement. The top line contains one or two whitespace-separated fields. \+                             \The first (optional) is a separator, which is optionally consumed after matching a name in the database. The second holds all the characters than can be used to define and use a name."+                exitSuccess))+        "Show help"+    ]+