packages feed

slugger (empty) → 0.1.0.0

raw patch · 8 files changed

+467/−0 lines, 8 filesdep +basedep +hspecdep +slugger

Dependencies added: base, hspec, slugger, text, text-icu

Files

+ CHANGELOG.md view
@@ -0,0 +1,5 @@+# Revision history for slugger++## 0.1.0.0 -- YYYY-mm-dd++* First version. Released on an unsuspecting world.
+ LICENSE view
@@ -0,0 +1,30 @@+Copyright (c) 2021, Robert W. Pearce++All rights reserved.++Redistribution and use in source and binary forms, with or without+modification, are permitted provided that the following conditions are met:++    * Redistributions of source code must retain the above copyright+      notice, this list of conditions and the following disclaimer.++    * Redistributions in binary form must reproduce the above+      copyright notice, this list of conditions and the following+      disclaimer in the documentation and/or other materials provided+      with the distribution.++    * Neither the name of Robert W. Pearce nor the names of other+      contributors may be used to endorse or promote products derived+      from this software without specific prior written permission.++THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ README.md view
@@ -0,0 +1,106 @@+# slugger++Clean URI slugs for Haskell++Convert multi-language text to a US-ASCII, lowercase, hyphenated, URI-friendly "slug".++[![built with nix](https://builtwithnix.org/badge.svg)](https://builtwithnix.org)++## Usage++### Library++There are `Data.Text` and `Data.String` library interfaces to `slugger` that+have plenty of examples in [`the test file`](./test/SluggerTest.hs), and here+are some simple examples.++Example of `Data.String.Slugger`:++```haskell+import qualified Data.String.Slugger as SluggerString++SluggerString.toSlug "Hey there,   world!"+-- "hey-there-world"++SluggerString.toSlug "GARÇON - déjà , Forêt — Zoë"+-- "garcon-deja-foret-zoe"+```++Example of `Data.Text.Slugger`:++```haskell+import qualified Data.Text as T+import qualified Data.Text.Slugger as SluggerText++SluggerText.toSlug (T.pack "Hey there,   world!")+-- "hey-there-world"++SluggerText.toSlug (T.pack "GARÇON - déjà , Forêt — Zoë")+-- "garcon-deja-foret-zoe"+```++### Executable++```sh+λ slugger "Hey there,   world!"+hey-there-world++λ slugger "Pijamalı hasta yağız şoföre çabucak güvendi"+pijamali-hasta-yagiz-sofore-cabucak-guvendi+```++## Language Support++These are the languages that are currently tested and therefore marked as+supported. Contributions are welcome for more extensive tests or tests for+additional languages.++- [x] Dansk     (Danish)+- [x] Deutsche  (German)+- [x] English+- [x] Español   (Spanish)+- [x] Français  (French)+- [x] Íslenskur (Icelandic)+- [x] Polskie   (Polish)+- [x] Svenska   (Swedish)+- [x] Türk      (Turkish)++## Development++Try the project executable via a nix flake app:++```sh+λ nix run . "Testing 1,2,3"+testing-1-2-3+```++Get into a nix dev environment:++```sh+λ nix develop+[nix]λ+```++Build the project:++```sh+[nix]λ nix build+```++Run the tests from the shell:++```sh+[nix]λ cabal test --test-show-details=streaming --test-option=--color+```++Run the tests from GHCi:++```sh+[nix]λ cabal repl slugger-test+[ghci]λ :main+# test output prints here++# make some changes, then...+[ghci]λ :reload+[ghci]λ :main+```
+ app/Main.hs view
@@ -0,0 +1,23 @@+module Main where+++import qualified Data.String.Slugger as Slugger+import qualified System.Environment as Env+++help :: String+help =+    "slugger: Clean URI slugs\n" +++    "usage: slugger \"<text>\"\n" +++    "  -h, --help   Read this help info\n"+++parse :: [String] -> String+parse ["-h"]     = help+parse ["--help"] = help+parse [str]      = Slugger.toSlug str+parse _          = help+++main :: IO ()+main = putStrLn . parse =<< Env.getArgs
+ lib/Data/String/Slugger.hs view
@@ -0,0 +1,23 @@+module Data.String.Slugger (toSlug) where+++--------------------------------------------------------------------------------+import qualified Data.Text as T+import qualified Data.Text.Slugger as Slugger+++--------------------------------------------------------------------------------+{- | Converts to a US-ASCII, lowercase, hyphenated, URI-friendly "slug"++__Examples:__++@+toSlug "Hey there,   world!"+-- "hey-there-world"++toSlug "GARÇON - déjà , Forêt — Zoë"+-- "garcon-deja-foret-zoe"+@+-}+toSlug :: String -> String+toSlug = T.unpack . Slugger.toSlug . T.pack
+ lib/Data/Text/Slugger.hs view
@@ -0,0 +1,108 @@+{-# LANGUAGE OverloadedStrings #-}++module Data.Text.Slugger (toSlug) where+++--------------------------------------------------------------------------------+import qualified Data.Char as Char+import qualified Data.Text as T+import qualified Data.Text.ICU.Char as ICUChar+import qualified Data.Text.ICU.Normalize as ICUN+++--------------------------------------------------------------------------------+{- | Converts to a US-ASCII, lowercase, hyphenated, URI-friendly "slug"++__Examples:__++@+toSlug (T.pack "Hey there,   world!")+-- "hey-there-world"++toSlug (T.pack "GARÇON - déjà , Forêt — Zoë")+-- "garcon-deja-foret-zoe"+@+-}+toSlug :: T.Text -> T.Text+toSlug = hyphenateWords . clean . normalize+++--------------------------------------------------------------------------------+normalize :: T.Text -> T.Text+normalize = ICUN.normalize ICUN.NFKD+++--------------------------------------------------------------------------------+clean :: T.Text -> T.Text+clean = T.foldr buildCleanText T.empty+++buildCleanText :: Char -> T.Text -> T.Text+buildCleanText x acc+    | isCharModifier x || isSingleQuote x = acc+    | otherwise = T.concat [adjustChar x, acc]+++isSingleQuote :: Char -> Bool+isSingleQuote = (== '\'')+++isCharModifier :: Char -> Bool+isCharModifier = ICUChar.property ICUChar.Diacritic+++adjustChar :: Char -> T.Text+adjustChar 'æ' = "ae"+adjustChar 'Æ' = "ae"+adjustChar 'ð' = "d"+adjustChar 'Ð' = "d"+adjustChar 'ƒ' = "f"+adjustChar 'Ƒ' = "f"+adjustChar 'ø' = "o"+adjustChar 'Ø' = "o"+adjustChar 'œ' = "oe"+adjustChar 'Œ' = "oe"+adjustChar 'ł' = "l"+adjustChar 'Ł' = "l"+adjustChar 'ß' = "ss"+adjustChar 'þ' = "th"+adjustChar 'Þ' = "th"+adjustChar 'ı' = "i"+-- See Note [Turkish I]+adjustChar x+  | isAsciiAlphaNum x = toLowerAsText x+  | otherwise = " "+++isAsciiAlphaNum :: Char -> Bool+isAsciiAlphaNum x = Char.isAscii x && Char.isAlphaNum x+++toLowerAsText :: Char -> T.Text+toLowerAsText = T.singleton . Char.toLower+++--------------------------------------------------------------------------------+hyphenateWords :: T.Text -> T.Text+hyphenateWords = T.intercalate (T.singleton '-') . T.words++{-+Note: [Turkish I]++Turkish has an unusual casing of the letter 'I'. In Turkish, we have+'i' and 'ı', two separate letters. They correspond to uppercase 'İ'+and 'I'. Notice that this is the opposite of most other languages,+where lowercase 'i' correspond to uppercase 'I' (losing the dot for no+good reason).++Unicode gets this correctly, so a Unicode-aware `toLower` function would+convert uppercase 'I' to 'ı' when on Turkish locale. This tend to break+functions like the one we are writing, if we incorrectly assume that every+ASCII uppercase letter would correspond to an ASCII lowercase letter.++The surprise is that; `Data.Char.toLower` function we use is not+locale-aware, `Data.Text.ICU.toLower` is. Only because of this fact `I`+becomes `i` even on Turkish locale on this function. This note is here+so that we do not start using `Data.Text.ICU.toLower` and break the+library on Turkish locale.+-}
+ slugger.cabal view
@@ -0,0 +1,82 @@+cabal-version:        3.0+name:                 slugger+version:              0.1.0.0+synopsis:             Clean URI slugs for Haskell+description:          Convert multi-language text to a US-ASCII, lowercase, hyphenated, URI-friendly "slug"+homepage:             https://github.com/rpearce/slugger+bug-reports:          https://github.com/rpearce/slugger/issues+license:              BSD-3-Clause+license-file:         LICENSE+author:               Robert W. Pearce <me@robertwpearce.com>+maintainer:           Robert W. Pearce <me@robertwpearce.com>+copyright:            2021 Robert W. Pearce+category:             Data, Text, Web+extra-source-files:   CHANGELOG.md+                    , README.md++source-repository head+  type:     git+  location: git://github.com/rpearce/slugger.git++flag ci+  description: Use CI settings+  default:     False+  manual:      True++flag dev+  description: Use development settings+  default:     False+  manual:      True++common lang+    default-language: Haskell2010++common deps+    build-depends:+        base     >= 4.8 && < 5+      , text     >= 1   && < 1.3+      , text-icu >= 0.7 && < 0.8++common self-dep+    build-depends: slugger++common flags+    if flag(ci)+        ghc-options:+            -Wall+            -fwarn-incomplete-record-updates+            -fwarn-incomplete-uni-patterns+    else+        ghc-options:+            -Wall+            -Wcompat+            -Widentities+            -Wincomplete-record-updates+            -Wincomplete-uni-patterns+            -Wpartial-fields+            -Wredundant-constraints++    if flag(dev)+        ghc-options:+            -Werror++library+    import:           lang, flags, deps+    exposed-modules:  Data.String.Slugger+                      Data.Text.Slugger+    hs-source-dirs:   lib++executable slugger+    import:           lang, flags, deps, self-dep+    main-is:          Main.hs+    hs-source-dirs:   app+    ghc-options:      -rtsopts+                      -threaded+                      -with-rtsopts=-N++test-suite slugger-test+    import:           lang, flags, deps, self-dep+    main-is:          SluggerTest.hs+    type:             exitcode-stdio-1.0+    hs-source-dirs:   test+    build-depends:    hspec >= 2 && < 3
+ test/SluggerTest.hs view
@@ -0,0 +1,90 @@+module Main (main) where++import Test.Hspec+import qualified Data.Text as T+import qualified Data.String.Slugger as SluggerString+import qualified Data.Text.Slugger as SluggerText++main :: IO ()+main = hspec $ do+    describe "Data.String.Slugger.toSlug" $ do++        it "works" $ do+            SluggerString.toSlug "HELLO, WORLD!!!" `shouldBe` "hello-world"+++    describe "Data.Text.Slugger.toSlug" $ do++        context "when an empty string" $ do+            it "returns an empty string" $ do+                SluggerText.toSlug (T.pack "") `shouldBe` T.pack ""++        context "when a string with empty spaces" $ do+            it "returns an empty string" $ do+                SluggerText.toSlug (T.pack "  ") `shouldBe` T.pack ""++        context "when uppercase words" $ do+            it "returns words lowercased and hyphenated" $ do+                SluggerText.toSlug (T.pack "HELLO GREAT WORLD") `shouldBe`+                    T.pack "hello-great-world"++        context "when uppercase words with symbols" $ do+            it "returns words lowercased and hyphenated" $ do+                SluggerText.toSlug (T.pack "HELLO, WORLD!!!") `shouldBe`+                    T.pack "hello-world"++        context "when contractions are used (single quotes)" $ do+            it "keeps words like can't and don't together" $ do+                SluggerText.toSlug (T.pack "I can't won't don't want to!")+                    `shouldBe` T.pack "i-cant-wont-dont-want-to"++        context "when leading & trailing whitespace" $ do+            it "does nothing with the extra whitespace" $ do+                SluggerText.toSlug (T.pack "  Hey world!   ") `shouldBe`+                    T.pack "hey-world"++        context "when repeated inner whitespace" $ do+            it "treats the repeated inner whitespace as a single space" $ do+                SluggerText.toSlug (T.pack "Hey there,  world!") `shouldBe`+                    T.pack "hey-there-world"++        context "when non US-ASCII letters provided" $ do+            it "handles Danish: Være så venlig... Øl" $ do+                SluggerText.toSlug (T.pack "Være så venlig... Øl")+                    `shouldBe` T.pack "vaere-sa-venlig-ol"++            it "handles French: GARÇON - déjà , Forêt — Zoë" $ do+                SluggerText.toSlug (T.pack "GARÇON - déjà , Forêt — Zoë")+                    `shouldBe` T.pack "garcon-deja-foret-zoe"++            it "handles German: Straße, müde, Äpfel und Ökologie" $ do+                SluggerText.toSlug (T.pack "Straße, müde, Äpfel und Ökologie")+                    `shouldBe` T.pack "strasse-mude-apfel-und-okologie"++            it "handles Icelandic: (Ég) er kominn aftur (á ný) Inn í þig (Það er) svo gott að vera (hér) En stoppa stutt við" $ do+                SluggerText.toSlug (T.pack "(Ég) er kominn aftur (á ný) Inn í þig (Það er) svo gott að vera (hér) En stoppa stutt við")+                    `shouldBe` T.pack "eg-er-kominn-aftur-a-ny-inn-i-thig-thad-er-svo-gott-ad-vera-her-en-stoppa-stutt-vid"++            it "handles Polish: Żółć, Szczęście, & Następstw" $ do+                SluggerText.toSlug (T.pack "Żółć, Szczęście, & Następstw")+                    `shouldBe` T.pack "zolc-szczescie-nastepstw"++            it "handles Spanish: ¿Qué pasó? Soy de España" $ do+                SluggerText.toSlug (T.pack "¿Qué pasó? Soy de España")+                    `shouldBe` T.pack "que-paso-soy-de-espana"++            it "handles Swedish: Varsågod ++ tack så mycket -- ölet" $ do+                SluggerText.toSlug (T.pack "Varsågod ++ tack så mycket -- ölet")+                    `shouldBe` T.pack "varsagod-tack-sa-mycket-olet"++            it "handles Turkish: Pijamalı hasta yağız şoföre çabucak güvendi" $ do+                SluggerText.toSlug (T.pack "Pijamalı hasta yağız şoföre çabucak güvendi")+                    `shouldBe` T.pack "pijamali-hasta-yagiz-sofore-cabucak-guvendi"++            it "handles Turkish (uppercase): PİJAMALI HASTA YAĞIZ ŞOFÖRE ÇABUCAK GÜVENDİ" $ do+                SluggerText.toSlug (T.pack "PİJAMALI HASTA YAĞIZ ŞOFÖRE ÇABUCAK GÜVENDİ")+                    `shouldBe` T.pack "pijamali-hasta-yagiz-sofore-cabucak-guvendi"++            it "handles a bunch we may have left out: æ Æ ð Ð ƒ Ƒ ø Ø œ Œ ł Ł ß þ Þ" $ do+                SluggerText.toSlug (T.pack "æ Æ ð Ð ƒ Ƒ ø Ø œ Œ ł Ł ß þ Þ")+                    `shouldBe` T.pack "ae-ae-d-d-f-f-o-o-oe-oe-l-l-ss-th-th"