diff --git a/LICENSE b/LICENSE new file mode 100644 --- /dev/null +++ b/LICENSE @@ -0,0 +1,30 @@ +Copyright (c) 2014, Zankoku Okuno + +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above + copyright notice, this list of conditions and the following + disclaimer in the documentation and/or other materials provided + with the distribution. + + * Neither the name of Zankoku Okuno nor the names of other + contributors may be used to endorse or promote products derived + from this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. diff --git a/README.md b/README.md new file mode 100644 --- /dev/null +++ b/README.md @@ -0,0 +1,99 @@ +Unicoder +======== + +The unicoder reads in a source file and makes replacements in-place. The goal +is to allow ascii interfaces to be able to insert unicode without taking your +hands off the keyboard. This can allow for unicode to be entered into source +code or any other text document you're editing. + +Entering unicode is not as easy as typing a special string (default backslash) +followed by an identifier. There's also syntax for wrapping content inside a +pair of unicode strings. +For example, with the default configuration, unicoder turns +`\floor{x} \def \lambda x. (floor x)` into `⌊x⌋ ≡ λ x. (floor x)`. +Admitedly, this is not a great syntax for some kinds of documents (esp. XeLaTeX), +but that's why we've allowed for configuration of each of the special marks as +well as the identifier character set, so Unicoder can be relevant to any type of +text data. + +There's a detailed explanation of the unicoder algorithm and configuration on our +[Viewdocs](http://zankoku-okuno.viewdocs.io/unicoder/specs.md). It's fairly likely +that the system can be deduced just from examples, though. + +Examples +-------- + +Assuming a config file that looks like this: + +``` +\ . { } a-z + +lambda λ +pi π +bag ⟅ ⟆ +``` + +we can write this with a normal keyboard: + +``` +\lambda.x. x + \pi +``` + +and after unicodizing, we will get: + +``` +λx. x + π +``` + +and celebrate the nice, clean lambda-calculus. + +Have no fear, however, code such as this: + +``` +id = \x -> x +newline_period = "\n." +``` + +Will remain unchanged, as `x` and `n` are not in the config file. + +There are also two-part replacements. These take a single (non-nested) argument, transforming + +``` +\bag{black} +``` + +into + +``` +⟅black⟆ +``` + +You can also use each half of a two-part replacement individually. This is especially +usefule for nesting, but also when you simply have argument-close marks in the argument: + +``` +\{bag {} \}bag +``` + +becomes + +``` +⟅ {} ⟆ +``` + +Pitfalls +-------- + +Even in something as simple as this, you may want to be aware of a few facts: + + * Beware of adding names like `n` or `t` in your config file. If you are using + a language that isn't esoteric, you will probably change the meaning of your + code. + * It _is_ still possible to mess up strings. For example, `"\neq"` → `"≠"` + instead of being equivalent to `"\n" ++ "eq"`. I conjecture that there is + no way to solve this problem without sacrificing idempotence. + * I've made little attempt to ensure safety, other than using Haskell. Make + backups if you are wary (and your editor doesn't already). + +Thankfully, the pitfalls are realistically enumerable. + diff --git a/Setup.hs b/Setup.hs new file mode 100644 --- /dev/null +++ b/Setup.hs @@ -0,0 +1,2 @@ +import Distribution.Simple +main = defaultMain diff --git a/Text/Unicoder.hs b/Text/Unicoder.hs new file mode 100644 --- /dev/null +++ b/Text/Unicoder.hs @@ -0,0 +1,132 @@ +{-# LANGUAGE OverloadedStrings #-} +module Text.Unicoder ( + unicodize + , unicodizeStr + , Config + , parseConfigFile + ) where + +import System.IO + +import Data.Text (Text) +import qualified Data.Text as T +import qualified Data.Text.IO as T +import Data.Attoparsec.Text +import Data.Attoparsec.Combinator + +import Data.Maybe +import Data.Either +import Data.Monoid +import Control.Applicative +import Control.Monad + + +unicodize :: Config -> Text -> Text +unicodize config input = case parseOnly (xform config) input of + Left err -> error "unicoder: internal error" + Right val -> val + +unicodizeStr :: Config -> String -> String +unicodizeStr config = T.unpack . unicodize config . T.pack + + +data Config = Config { _idChars :: Char -> Bool + , _beginMark :: Text + , _endMark :: Maybe Text + , _betweenMarks :: Maybe (Text, Text) + , _macros0 :: [(Text, Text)] + , _macros1 :: [(Text, (Text, Text))] + } + +parseConfigFile :: FilePath -> IO (Maybe Config) +parseConfigFile path = withFile path ReadMode $ \fp -> do + xs <- filter (not . T.null) . T.lines <$> T.hGetContents fp + return $ case xs of + [] -> Nothing + (lexer:raw_macros) -> do + --TODO configure begin mark + emptyConfig <- case filter (not . T.null) $ T.splitOn " " lexer of + [idChars] -> return + Config { _idChars = inClass (T.unpack idChars) + , _beginMark = "\\" + , _endMark = Nothing + , _betweenMarks = Nothing + , _macros0 = [], _macros1 = [] + } + [begin, idChars] -> return + Config { _idChars = inClass (T.unpack idChars) + , _beginMark = begin + , _endMark = Nothing + , _betweenMarks = Nothing + , _macros0 = [], _macros1 = [] + } + [begin, end, idChars] -> return + Config { _idChars = inClass (T.unpack idChars) + , _beginMark = begin + , _endMark = Just end + , _betweenMarks = Nothing + , _macros0 = [], _macros1 = [] + } + [begin, end, open, close, idChars] -> return + Config { _idChars = inClass (T.unpack idChars) + , _beginMark = begin + , _endMark = Just end + , _betweenMarks = Just (open, close) + , _macros0 = [], _macros1 = [] + } + _ -> Nothing + let (macros0, macros1) = partitionEithers . catMaybes $ parseMacro <$> raw_macros + return $ emptyConfig { _macros0 = macros0, _macros1 = macros1 } + +parseMacro :: T.Text -> Maybe (Either (Text, Text) (Text, (Text, Text))) +parseMacro input = case T.words input of + [k, v] -> Just $ Left (k, v) + [k, v1, v2] -> Just $ Right (k, (v1, v2)) + _ -> Nothing + + +xform :: Config -> Parser Text +xform config = mconcat <$> many (passthrough <|> macro <|> strayBegin) <* endOfInput + where + (beginStr, beginChr) = (_beginMark config, T.head beginStr) + passthrough = takeWhile1 (/= beginChr) + macro = do + string beginStr + full <|> half + strayBegin = T.singleton <$> char beginChr + full = do + name <- takeWhile1 (_idChars config) + mono name <|> di name + where + mono name = do + replace <- name `lookupM` _macros0 config + endMark + return replace + di name = do + (open, close) <- betweenMarks + (rOpen, rClose) <- name `lookupM` _macros1 config + string open + inner <- T.pack <$> anyChar `manyTill` string close + return $ rOpen <> inner <> rClose + half = do + (open, close) <- betweenMarks + which <- (const fst <$> string open) <|> (const snd <$> string close) + name <- takeWhile1 (_idChars config) + replace <- which <$> name `lookupM` _macros1 config + endMark + return replace + endMark = case _endMark config of + Nothing -> return () + Just end -> option () $ void (string end) + betweenMarks = case _betweenMarks config of + Nothing -> fail "" + Just x -> return x + lookupM k v = maybe (fail "") return (k `lookup` v) + +{-TODO +a config lint + characters don't appear twice in the lexer + open and end are distinguishable + macros are defined only using idChars + macro lines are word-length 2 or 3 +-} diff --git a/changes.md b/changes.md new file mode 100644 --- /dev/null +++ b/changes.md @@ -0,0 +1,33 @@ +Changes +======= + +v0.4.0 +------ + * Cabalized. + * Config file location determined by cabal. + * Split major functions into a library. + * Allow two-part replacements. + * No more tmp file needed. + +v0.3.1 +------ + * Use config files from locations other than `/etc/zanoku-okuno/unicoder/`. + * Use a `.unicoder.tmp` extension instead of `.tmp` so that there's less + chance of name-collision with the tempfile. + +v0.3.0 +------ + * Each configuration file can specify separator string and valid name + characters. + * More than one configuration file can be stored and selected between on the + command line. + * The role of `symbols.conf` is now dealt with by `default.conf`. + * No more extra newline at end of file + +v0.2.0 +------ + + * There are no longer any failure modes of the parser. One could probably do a + formal proof that given finite input and no OS errors, unicoder will + complete without failure in a finite time. + * No special handling of strings. Unicoder is therefore now language-agnostic. diff --git a/data/default.conf b/data/default.conf new file mode 100644 --- /dev/null +++ b/data/default.conf @@ -0,0 +1,199 @@ +\ . { } a-zA-Z~!%^&*_=+<>/?|- + + +l λ +L Λ +A ∀ +E ∃ +def ≡ +is ≡ +qq ⌜ +uq ⌞ +uqs ⌟ +p ′ +hole □ + + +prime ′ +ellipsis … +ldots … +proportion ∝ +ccw ↺ +cw ↻ + +of ∘ +to → +To ⇒ +from ← +maps ↦ +unmaps ↤ +rharpoon ⇀ +lharpoon ↼ +up ↑ +down ↓ +Up ⇑ +Down ⇓ +hookto ↪ +hookfrom ↩ +-> → +=> ⇒ +<- ← +|-> ↦ +-/> ↛ + ↝ +<~ ↜ + +Brack +angle ‹ › +Angle « » +chevron ⟨ ⟩ +shell ❲ ❳ +Shell ⟬ ⟭ +bag ⟅ ⟆ +floor ⌊ ⌋ +ceil ⌈ ⌉ +quine ⌜ ⌝ +lowquine ⌞ ⌟ +< ⟨ ⟩ + +cross × +dot · +div ÷ +star ⋆ +sqrt √ +oplus ⊕ +ominus ⊖ +otimes ⊗ +odiv ⊘ + +inf ∞ +nabla ∇ +del ∂ +integral ∫ + +neq ≠ +geq ≥ +leq ≤ +approx ≈ +plusminus ± +minusplus ∓ +!= ≠ +/= ≠ +>= ≥ +<= ≤ + +null ∅ +elem ∈ +nelem ∉ +contains ∋ +ncontains ∌ +subset ⊆ +superset ⊇ +psubset ⊂ +psuperset ⊃ +union ∪ +intersect ∩ +Subset ⊑ +Superset ⊒ +pSubset ⊏ +pSuperset ⊐ +squnion ⊓ +sqinter ⊔ +join ⋈ +ljoin ⋉ +rjoin ⋊ + +forall ∀ +exists ∃ +nexists ∄ +exists! ∃! +not ¬ +and ∧ +or ∨ +xor ⊻ +nand ⊼ +nor ⊽ +implies ⇒ +iff ⇔ +equiv ≡ + +bot ⊥ +rtack ⊢ +ltack ⊣ +rttack ⊩ +therefore ∴ +because ∵ + +circle ○ +square □ +diamond ◇ +triangle △ +ltriangle ◁ +rtriangle ▷ +lozenge ◊ + +nat ℕ +int ℤ +rational ℚ +real ℝ +complex ℂ + +alpha α +beta β +gamma γ +delta δ +epsilon ε +zeta ζ +eta η +theta θ +iota ι +kappa κ +lambda λ +mu μ +nu ν +xi ξ +pi π +rho ρ +sigma σ +stigma ϛ +tau τ +upsilon υ +phi ϕ +chi χ +psi ψ +omega ω +digamma ϝ +koppa ϟ +sampi ϡ + +Alpha +Beta +Gamma Γ +Delta Δ +Epsilon +Zeta +Eta +Theta Θ +Iota +Kappa +Lambda Λ +Mu +Nu +Xi Ξ +Pi Π +Rho +Sigma Σ +Stigma Ϛ +Tau +Upsilon +Phi Φ +Chi +Psi Ψ +Omega Ω +Digamma +Qoppa Ϟ +Sampi Ϡ + + +mizu 水 diff --git a/test/di.in b/test/di.in new file mode 100644 --- /dev/null +++ b/test/di.in @@ -0,0 +1,2 @@ +\bag{foo} +\{bag\bag{foo}\}bag. diff --git a/test/di.out b/test/di.out new file mode 100644 --- /dev/null +++ b/test/di.out @@ -0,0 +1,2 @@ +⟅foo⟆ +⟅⟅foo⟆⟆ diff --git a/test/mono.in b/test/mono.in new file mode 100644 --- /dev/null +++ b/test/mono.in @@ -0,0 +1,5 @@ +\lambda +\lambda x +\lambda.x +\lambdax +\lambda{x} diff --git a/test/mono.out b/test/mono.out new file mode 100644 --- /dev/null +++ b/test/mono.out @@ -0,0 +1,5 @@ +λ +λ x +λx +\lambdax +λ{x} diff --git a/test/passthrough.in b/test/passthrough.in new file mode 100644 --- /dev/null +++ b/test/passthrough.in @@ -0,0 +1,4 @@ +foo +{} +\ +. diff --git a/test/passthrough.out b/test/passthrough.out new file mode 100644 --- /dev/null +++ b/test/passthrough.out @@ -0,0 +1,4 @@ +foo +{} +\ +. diff --git a/test/test.config b/test/test.config new file mode 100644 --- /dev/null +++ b/test/test.config @@ -0,0 +1,4 @@ +\ . { } a-z + +lambda λ +bag ⟅ ⟆ diff --git a/test/test.hs b/test/test.hs new file mode 100644 --- /dev/null +++ b/test/test.hs @@ -0,0 +1,27 @@ +import System.IO +import System.Exit +import Control.Monad +import qualified Data.Text.IO as T + +import Text.Unicoder + +main :: IO () +main = do + m_config <- parseConfigFile "test/test.config" + testFile <- case m_config of + Nothing -> die "Could not parse config file." + Just config -> return $ testFile config + let files = ["passthrough", "mono", "di"] + results <- mapM testFile files + unless (all id results) $ do + mapM_ putStrLn $ zipWith (\a b -> a ++ ": " ++ if b then "OK" else "FAILURE") files results + exitFailure + +testFile :: Config -> FilePath -> IO Bool +testFile config path = do + input <- T.readFile $ "test/" ++ path ++ ".in" + output <- T.readFile $ "test/" ++ path ++ ".out" + return $ unicodize config input == output + +die :: String -> IO a +die msg = putStrLn msg >> exitFailure diff --git a/unicoder.cabal b/unicoder.cabal new file mode 100644 --- /dev/null +++ b/unicoder.cabal @@ -0,0 +1,65 @@ +name: unicoder +version: 0.4.0 +stability: beta +synopsis: Make writing in unicode easy. +description: Unicoder transforms text documents, replacing simple patterns with + unicode equivalents. The patterns can be easily configured by the user. + This package is especially meant to open the vast and expressive array + of unicode identifiers to programmers and language designers, but there's + nothing wrong with a technically savvy user putting unicoder to work + on documents for human consumption. + . + With the default settings, + . + @ + \\E x. \\A y. \\\ y\\\> \\ldots + \\l x,y. x \\of x \\of y + @ + . + becomes + . + @ + ∃x ∀y ⟨x → y⟩ … + λ x,y. x ∘ x ∘ y + @ + . + Many more possibilities abound just in the default set of characters. + Any system of special characters can be made easy to type with a normal + keyboard as long as unicode supports it. +license: BSD3 +license-file: LICENSE +author: Zankoku Okuno +maintainer: zankoku.okuno@gmail.com +copyright: Copyright © 2013, 2014, Okuno Zankoku +category: Text +build-type: Simple +cabal-version: >=1.9.2 +data-dir: data +data-files: *.conf +extra-source-files: README.md, changes.md, + test/*.in, test/*.out, test/test.config + +executable unicoder + main-is: unicoder.hs + -- other-modules: + build-depends: base ==4.6.*, + text ==0.11.*, + attoparsec >=0.10.0.0, + directory ==1.2.* + +library + exposed-modules: Text.Unicoder + build-depends: base ==4.6.*, + text ==0.11.*, + attoparsec >=0.10.0.0 + +Test-Suite test-unicoder + type: exitcode-stdio-1.0 + main-is: test/test.hs + build-depends: base ==4.6.*, + text ==0.11.*, + attoparsec >=0.10.0.0 + +source-repository head + type: git + location: https://github.com/Zankoku-Okuno/unicoder.git diff --git a/unicoder.hs b/unicoder.hs new file mode 100644 --- /dev/null +++ b/unicoder.hs @@ -0,0 +1,103 @@ +import System.Environment +import System.IO +import System.Directory +import System.Exit +import System.Console.GetOpt +import qualified Data.Text.IO as T +import Control.Monad + +import Text.Unicoder + +import Paths_unicoder +import Data.Version + +symbolFile :: Options -> IO FilePath +symbolFile Options { optConfig = which } = + if '/' `elem` which + then return which + else getDataFileName (which ++ ".conf") + +main :: IO () +main = do + (opts, args) <- getOptions + m_config <- parseConfigFile =<< symbolFile opts + config <- case m_config of + Nothing -> die "bad configuration file" + Just config -> return config + -- TODO -o/--output flag, but how for many files? + mapM_ (mainLoop config) args + exitSuccess + +mainLoop :: Config -> FilePath -> IO () +mainLoop config filename = do + source <- T.readFile filename + let result = unicodize config source + T.writeFile filename result + + +die err = do + putErrLn err + exitFailure + +makePair [a, b] = (a, b) +putErrLn = hPutStrLn stderr + + +data Options = Options { optConfig :: String + , optOutput :: Maybe FilePath + } deriving (Show) +startOptions = Options { optConfig = "default" + , optOutput = Nothing + } + +getOptions :: IO (Options, [FilePath]) +getOptions = do + (actions, args, errors) <- return . getOpt Permute options =<< getArgs + if null errors + then do + opts <- foldl (>>=) (return startOptions) actions + when (null args) $ die "no input files" + return (opts, args) + else do + mapM_ putErrLn errors + exitFailure + +options :: [OptDescr (Options -> IO Options)] +options = + [ Option "c" ["config"] + (ReqArg + (\arg opt -> return opt { optConfig = arg }) + "file") + "Specify a configuration (usually a programming language)." + + , Option "V" ["version"] + (NoArg + (\_ -> do + putStrLn (showVersion version) + exitSuccess)) + "Print version." + , Option "" ["show-config-dir"] + (NoArg + (\_ -> do + putStrLn =<< getDataFileName "" + exitSuccess)) + "Print directory where configuration files are stored." + , Option "h" ["help"] + (NoArg + (\_ -> do + prg <- getProgName + putStrLn "Ease input of unicode glyphs." + putStrLn "This program is lisenced under the 3-clause BSD lisence." + putStrLn (usageInfo (prg ++ " [options] files...") options) + putStrLn "Run on one or more files to replace special sequences of characters (as defined in a config file) with replacements." + putStrLn "The idea is to put in useful unicode symbols and then use those symbols in source code." + putStrLn " -- Config Files" + putStrLn " Configuration files are stored in `/etc/zankoku-okuno/unicoder`, and end in `.conf`." + putStrLn " Passing a file to `-c` that contains a slash will use exactly the file you specify, instead of looking one up." + putStrLn " Configuration files consist of a top line and a body. The body is simply a database with two whitespace-separated fields. \ + \The first is the a name, and the second is the replacement. The top line contains one or two whitespace-separated fields. \ + \The first (optional) is a separator, which is optionally consumed after matching a name in the database. The second holds all the characters than can be used to define and use a name." + exitSuccess)) + "Show help" + ] +