unicode-data-scripts (empty) → 0.1.0
raw patch · 10 files changed
+739/−0 lines, 10 filesdep +basedep +deepseqdep +hspec
Dependencies added: base, deepseq, hspec, tasty, tasty-bench, unicode-data, unicode-data-scripts
Files
- Changelog.md +7/−0
- LICENSE +249/−0
- README.md +23/−0
- bench/Main.hs +51/−0
- lib/Unicode/Char/General/Scripts.hs +95/−0
- lib/Unicode/Internal/Char/ScriptExtensions.hs too large to diff
- lib/Unicode/Internal/Char/Scripts.hs too large to diff
- test/Main.hs +1/−0
- test/Unicode/Char/General/ScriptsSpec.hs +200/−0
- unicode-data-scripts.cabal +113/−0
+ Changelog.md view
@@ -0,0 +1,7 @@+# Changelog++## 0.1.0 (September 2022)++Initial release++- Added the module `Unicode.Char.General.Scripts`.
+ LICENSE view
@@ -0,0 +1,249 @@++ Apache License+ Version 2.0, January 2004+ http://www.apache.org/licenses/++ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION++ 1. Definitions.++ "License" shall mean the terms and conditions for use, reproduction,+ and distribution as defined by Sections 1 through 9 of this document.++ "Licensor" shall mean the copyright owner or entity authorized by+ the copyright owner that is granting the License.++ "Legal Entity" shall mean the union of the acting entity and all+ other entities that control, are controlled by, or are under common+ control with that entity. For the purposes of this definition,+ "control" means (i) the power, direct or indirect, to cause the+ direction or management of such entity, whether by contract or+ otherwise, or (ii) ownership of fifty percent (50%) or more of the+ outstanding shares, or (iii) beneficial ownership of such entity.++ "You" (or "Your") shall mean an individual or Legal Entity+ exercising permissions granted by this License.++ "Source" form shall mean the preferred form for making modifications,+ including but not limited to software source code, documentation+ source, and configuration files.++ "Object" form shall mean any form resulting from mechanical+ transformation or translation of a Source form, including but+ not limited to compiled object code, generated documentation,+ and conversions to other media types.++ "Work" shall mean the work of authorship, whether in Source or+ Object form, made available under the License, as indicated by a+ copyright notice that is included in or attached to the work.++ "Derivative Works" shall mean any work, whether in Source or Object+ form, that is based on (or derived from) the Work and for which the+ editorial revisions, annotations, elaborations, or other modifications+ represent, as a whole, an original work of authorship. For the purposes+ of this License, Derivative Works shall not include works that remain+ separable from, or merely link (or bind by name) to the interfaces of,+ the Work and Derivative Works thereof.++ "Contribution" shall mean any work of authorship, including+ the original version of the Work and any modifications or additions+ to that Work or Derivative Works thereof, that is intentionally+ submitted to Licensor for inclusion in the Work by the copyright owner+ or by an individual or Legal Entity authorized to submit on behalf of+ the copyright owner. For the purposes of this definition, "submitted"+ means any form of electronic, verbal, or written communication sent+ to the Licensor or its representatives, including but not limited to+ communication on electronic mailing lists, source code control systems,+ and issue tracking systems that are managed by, or on behalf of, the+ Licensor for the purpose of discussing and improving the Work, but+ excluding communication that is conspicuously marked or otherwise+ designated in writing by the copyright owner as "Not a Contribution."++ "Contributor" shall mean Licensor and any individual or Legal Entity+ on behalf of whom a Contribution has been received by Licensor and+ subsequently incorporated within the Work.++ 2. Grant of Copyright License. Subject to the terms and conditions of+ this License, each Contributor hereby grants to You a perpetual,+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable+ copyright license to reproduce, prepare Derivative Works of,+ publicly display, publicly perform, sublicense, and distribute the+ Work and such Derivative Works in Source or Object form.++ 3. Grant of Patent License. Subject to the terms and conditions of+ this License, each Contributor hereby grants to You a perpetual,+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable+ (except as stated in this section) patent license to make, have made,+ use, offer to sell, sell, import, and otherwise transfer the Work,+ where such license applies only to those patent claims licensable+ by such Contributor that are necessarily infringed by their+ Contribution(s) alone or by combination of their Contribution(s)+ with the Work to which such Contribution(s) was submitted. If You+ institute patent litigation against any entity (including a+ cross-claim or counterclaim in a lawsuit) alleging that the Work+ or a Contribution incorporated within the Work constitutes direct+ or contributory patent infringement, then any patent licenses+ granted to You under this License for that Work shall terminate+ as of the date such litigation is filed.++ 4. Redistribution. You may reproduce and distribute copies of the+ Work or Derivative Works thereof in any medium, with or without+ modifications, and in Source or Object form, provided that You+ meet the following conditions:++ (a) You must give any other recipients of the Work or+ Derivative Works a copy of this License; and++ (b) You must cause any modified files to carry prominent notices+ stating that You changed the files; and++ (c) You must retain, in the Source form of any Derivative Works+ that You distribute, all copyright, patent, trademark, and+ attribution notices from the Source form of the Work,+ excluding those notices that do not pertain to any part of+ the Derivative Works; and++ (d) If the Work includes a "NOTICE" text file as part of its+ distribution, then any Derivative Works that You distribute must+ include a readable copy of the attribution notices contained+ within such NOTICE file, excluding those notices that do not+ pertain to any part of the Derivative Works, in at least one+ of the following places: within a NOTICE text file distributed+ as part of the Derivative Works; within the Source form or+ documentation, if provided along with the Derivative Works; or,+ within a display generated by the Derivative Works, if and+ wherever such third-party notices normally appear. The contents+ of the NOTICE file are for informational purposes only and+ do not modify the License. You may add Your own attribution+ notices within Derivative Works that You distribute, alongside+ or as an addendum to the NOTICE text from the Work, provided+ that such additional attribution notices cannot be construed+ as modifying the License.++ You may add Your own copyright statement to Your modifications and+ may provide additional or different license terms and conditions+ for use, reproduction, or distribution of Your modifications, or+ for any such Derivative Works as a whole, provided Your use,+ reproduction, and distribution of the Work otherwise complies with+ the conditions stated in this License.++ 5. Submission of Contributions. Unless You explicitly state otherwise,+ any Contribution intentionally submitted for inclusion in the Work+ by You to the Licensor shall be under the terms and conditions of+ this License, without any additional terms or conditions.+ Notwithstanding the above, nothing herein shall supersede or modify+ the terms of any separate license agreement you may have executed+ with Licensor regarding such Contributions.++ 6. Trademarks. This License does not grant permission to use the trade+ names, trademarks, service marks, or product names of the Licensor,+ except as required for reasonable and customary use in describing the+ origin of the Work and reproducing the content of the NOTICE file.++ 7. Disclaimer of Warranty. Unless required by applicable law or+ agreed to in writing, Licensor provides the Work (and each+ Contributor provides its Contributions) on an "AS IS" BASIS,+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or+ implied, including, without limitation, any warranties or conditions+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A+ PARTICULAR PURPOSE. You are solely responsible for determining the+ appropriateness of using or redistributing the Work and assume any+ risks associated with Your exercise of permissions under this License.++ 8. Limitation of Liability. In no event and under no legal theory,+ whether in tort (including negligence), contract, or otherwise,+ unless required by applicable law (such as deliberate and grossly+ negligent acts) or agreed to in writing, shall any Contributor be+ liable to You for damages, including any direct, indirect, special,+ incidental, or consequential damages of any character arising as a+ result of this License or out of the use or inability to use the+ Work (including but not limited to damages for loss of goodwill,+ work stoppage, computer failure or malfunction, or any and all+ other commercial damages or losses), even if such Contributor+ has been advised of the possibility of such damages.++ 9. Accepting Warranty or Additional Liability. While redistributing+ the Work or Derivative Works thereof, You may choose to offer,+ and charge a fee for, acceptance of support, warranty, indemnity,+ or other liability obligations and/or rights consistent with this+ License. However, in accepting such obligations, You may act only+ on Your own behalf and on Your sole responsibility, not on behalf+ of any other Contributor, and only if You agree to indemnify,+ defend, and hold each Contributor harmless for any liability+ incurred by, or claims asserted against, such Contributor by reason+ of your accepting any such warranty or additional liability.++ END OF TERMS AND CONDITIONS++-------------------------------------------------------------------------------+This distribution includes portions of code from the "unicode-transforms"+package (https://github.com/composewell/unicode-transforms/) which is+available under BSD-3-Clause license as described below.+-------------------------------------------------------------------------------++Copyright (c) 2016, Harendra Kumar+All rights reserved.++Redistribution and use in source and binary forms, with or without+modification, are permitted provided that the following conditions are met:++1. Redistributions of source code must retain the above copyright notice, this+list of conditions and the following disclaimer.++2. Redistributions in binary form must reproduce the above copyright notice,+this list of conditions and the following disclaimer in the documentation+and/or other materials provided with the distribution.++3. Neither the name of the copyright holder nor the names of its contributors+may be used to endorse or promote products derived from this software without+specific prior written permission.++THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.++-------------------------------------------------------------------------------+This distribution includes portions of code from the "unicode-transforms"+package (https://github.com/composewell/unicode-transforms/)+which included portions of code from the "prose"+(https://github.com/llelf/prose) package available under BSD-3-Clause+license as described below.+-------------------------------------------------------------------------------++Copyright (c) 2014–2015, Antonio Nikishaev++All rights reserved.++Redistribution and use in source and binary forms, with or without+modification, are permitted provided that the following conditions are met:++ * Redistributions of source code must retain the above copyright+ notice, this list of conditions and the following disclaimer.++ * Redistributions in binary form must reproduce the above+ copyright notice, this list of conditions and the following+ disclaimer in the documentation and/or other materials provided+ with the distribution.++ * Neither the name of Antonio Nikishaev nor the names of other+ contributors may be used to endorse or promote products derived+ from this software without specific prior written permission.++THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ README.md view
@@ -0,0 +1,23 @@+# README++`unicode-data-scripts` provides Haskell APIs to efficiently access the Unicode+character [scripts](https://www.unicode.org/reports/tr24/) from the+[Unicode character database](https://www.unicode.org/ucd/).++The Haskell data structures are generated programmatically from the+Unicode character database (UCD) files. The latest Unicode version+supported by this library is+[`14.0.0`](https://www.unicode.org/versions/Unicode14.0.0/).++Please see the+[Haddock documentation](https://hackage.haskell.org/package/unicode-data-scripts)+for reference documentation.++## Licensing++`unicode-data-scripts` is an [open source](https://github.com/composewell/unicode-data)+project available under a liberal [Apache-2.0 license](LICENSE).++## Contributing++As an open project we welcome contributions.
+ bench/Main.hs view
@@ -0,0 +1,51 @@+import Control.DeepSeq (NFData, deepseq, force)+import Control.Exception (evaluate)+import Data.Ix (Ix(..))+import Test.Tasty.Bench+ (Benchmark, bgroup, bench, defaultMain, env, nf)++import qualified Unicode.Char.General as G+import qualified Unicode.Char.General.Scripts as S++main :: IO ()+main = defaultMain+ [ bgroup "Unicode.Char.General.Script"+ [ bgroup "script"+ [ benchChars "unicode-data" (show . S.script)+ ]+ , bgroup "scriptDefinition"+ [ benchNF "unicode-data" (show . S.scriptDefinition)+ ]+ , bgroup "scriptExtensions"+ [ benchChars "unicode-data" (show . S.scriptExtensions)+ ]+ ]+ ]+ where+ benchChars :: forall a. (NFData a) => String -> (Char -> a) -> Benchmark+ benchChars t f =+ -- Avoid side-effects with garbage collection (see tasty-bench doc)+ env+ (evaluate (force chars)) -- initialize+ (bench t . nf (foldString f)) -- benchmark+ where+ -- Filter out: Surrogates, Private Use Areas and unsassigned code points+ chars = filter isValid [minBound..maxBound]+ isValid c = G.generalCategory c < G.Surrogate++ foldString :: forall a. (NFData a) => (Char -> a) -> String -> ()+ foldString f = foldr (deepseq . f) ()++ benchNF+ :: forall a b. (Bounded a, Ix a, NFData b)+ => String+ -> (a -> b)+ -> Benchmark+ benchNF t f = bench t (nf (fold_ f) (minBound, maxBound))++ fold_+ :: forall a b. (Ix a, NFData b)+ => (a -> b)+ -> (a, a)+ -> ()+ fold_ f = foldr (deepseq . f) () . range
+ lib/Unicode/Char/General/Scripts.hs view
@@ -0,0 +1,95 @@+{-# LANGUAGE CPP #-}++-- |+-- Module : Unicode.Char.General.Scripts+-- Copyright : (c) 2020 Composewell Technologies and Contributors+-- License : Apache-2.0+-- Maintainer : streamly@composewell.com+-- Stability : experimental+--+-- Unicode [scripts](https://www.unicode.org/reports/tr24/) related functions.+--+-- @since 0.1.0+--++module Unicode.Char.General.Scripts+ ( S.Script(..)+ , script+ , scriptExtensions+ , scriptDefinition+ )+where++import Data.Char (chr)+import Data.List.NonEmpty (NonEmpty)+import GHC.Exts+ (Ptr(..), Char(..), Int(..),+ indexWord32OffAddr#, word2Int#,+ and#, isTrue#, neWord#, (-#), (<#), chr#)+#if MIN_VERSION_base(4,16,0)+import GHC.Exts (word32ToWord#)+#endif+#ifdef WORDS_BIGENDIAN+import GHC.Exts (byteSwap32#)+#endif++import qualified Unicode.Internal.Char.Scripts as S+import qualified Unicode.Internal.Char.ScriptExtensions as S++-- | Character [script](https://www.unicode.org/reports/tr24/).+--+-- @since 0.1.0+{-# INLINE script #-}+script :: Char -> S.Script+script = toEnum . S.script++{- HLINT ignore scriptDefinition "Eta reduce" -}+-- | Characters corresponding to a 'S.Script'.+--+-- @since 0.1.0+scriptDefinition :: S.Script -> String+scriptDefinition = unpack . S.scriptDefinition+ where+ -- [NOTE] Encoding:+ -- • A single char is encoded as an LE Word32.+ -- • A range is encoded as two LE Word32 (first is lower bound, second is+ -- upper bound), which correspond to the codepoints with the 32th bit set.++ scriptRangeMask# = 0x80000000## -- 1 << 31+ maskComplement# = 0x7fffffff## -- 1 << 31 ^ 0xffffffff++ unpack (Ptr addr#, I# n#) = let {+ getRawCodePoint k# =+#ifdef WORDS_BIGENDIAN+#if MIN_VERSION_base(4,16,0)+ byteSwap32# (word32ToWord# (indexWord32OffAddr# addr# k#));+#else+ byteSwap32# (indexWord32OffAddr# addr# k#);+#endif+#elif MIN_VERSION_base(4,16,0)+ word32ToWord# (indexWord32OffAddr# addr# k#);+#else+ indexWord32OffAddr# addr# k#;+#endif+ getCodePoint k# = word2Int# (and# maskComplement# k#);+ addRange k# acc = if isTrue# (k# <# 0#)+ then acc+ else let {+ r1# = getRawCodePoint k#;+ c1# = getCodePoint r1#;+ isRange = isTrue# (and# r1# scriptRangeMask# `neWord#` 0##)+ } in if isRange+ then let {+ c2# = getCodePoint (getRawCodePoint (k# -# 1#));+ acc' = foldr ((:) . chr) acc [I# c2# .. I# c1#]+ } in addRange (k# -# 2#) acc'+ else addRange (k# -# 1#) (C# (chr# c1#) : acc)+ } in addRange (n# -# 1#) mempty++-- | Character+-- [script extensions](https://www.unicode.org/reports/tr24/#Script_Extensions).+--+-- @since 0.1.0+{-# INLINE scriptExtensions #-}+scriptExtensions :: Char -> NonEmpty S.Script+scriptExtensions = S.decodeScriptExtensions . S.scriptExtensions
+ lib/Unicode/Internal/Char/ScriptExtensions.hs view
file too large to diff
+ lib/Unicode/Internal/Char/Scripts.hs view
file too large to diff
+ test/Main.hs view
@@ -0,0 +1,1 @@+{-# OPTIONS_GHC -F -pgmF hspec-discover #-}
+ test/Unicode/Char/General/ScriptsSpec.hs view
@@ -0,0 +1,200 @@+{-# LANGUAGE BlockArguments, CPP, OverloadedLists #-}++module Unicode.Char.General.ScriptsSpec+ ( spec+ ) where++import Data.Foldable (traverse_)+import Test.Hspec+import qualified Unicode.Char.General.Scripts as UScripts+import qualified Unicode.Internal.Char.Scripts as S++import GHC.Exts+ (Ptr(..), Char(..), Int(..),+ indexWord32OffAddr#, int2Word#,+ and#, isTrue#, eqWord#, leWord#, neWord#,+ andI#, (-#), (<#),+ ord#)+#if MIN_VERSION_base(4,16,0)+import GHC.Exts (word32ToWord#)+#endif+#ifdef WORDS_BIGENDIAN+import GHC.Exts (byteSwap32#)+#endif++{- [NOTE]+These tests may fail if the compiler’s Unicode version+does not match the version of this package.+++-------------+----------------+-----------------++| GHC version | @base@ version | Unicode version |++=============+================+=================++| 8.8 | 4.13 | 12.0 |+| 8.10.[1-4] | 4.14.{0,1} | 12.0 |+| 8.10.5+ | 4.14.2+ | 13.0 |+| 9.0.[1-2] | 4.15.0 | 12.1 |+| 9.2.[1-4] | 4.16.0 | 14.0 |+| 9.4.[1-2] | 4.17.0 | 14.0 |++-------------+----------------+-----------------++-}++spec :: Spec+spec = do+ describe "Unicode scripts" do+ describe "Examples" do+ it "script" do+ let check s = (== s) . UScripts.script+ minBound `shouldSatisfy` check UScripts.Common+ maxBound `shouldSatisfy` check UScripts.Unknown+ '.' `shouldSatisfy` check UScripts.Common+ '1' `shouldSatisfy` check UScripts.Common+ 'A' `shouldSatisfy` check UScripts.Latin+ 'Α' `shouldSatisfy` check UScripts.Greek -- Greek capital+ 'α' `shouldSatisfy` check UScripts.Greek+ '\x0300' `shouldSatisfy` check UScripts.Inherited+ '\x0485' `shouldSatisfy` check UScripts.Inherited+ '\x0600' `shouldSatisfy` check UScripts.Arabic+ '\x060c' `shouldSatisfy` check UScripts.Common+ '\x0965' `shouldSatisfy` check UScripts.Common+ '\x1100' `shouldSatisfy` check UScripts.Hangul+ '\x3000' `shouldSatisfy` check UScripts.Common+ '\x4E00' `shouldSatisfy` check UScripts.Han+ '\x11FD0' `shouldSatisfy` check UScripts.Tamil+ '\x1F600' `shouldSatisfy` check UScripts.Common+ '\x20000' `shouldSatisfy` check UScripts.Han+ -- BOM+ '\xFEFF' `shouldSatisfy` check UScripts.Common+ '\xFFFF' `shouldSatisfy` check UScripts.Unknown+ -- Private Use Areas+ '\xE000' `shouldSatisfy` check UScripts.Unknown+ '\xF0000' `shouldSatisfy` check UScripts.Unknown+ it "scriptExtensions" do+ let check s = (== s) . UScripts.scriptExtensions+ minBound `shouldSatisfy` check [ UScripts.Common]+ maxBound `shouldSatisfy` check [ UScripts.Unknown]+ '.' `shouldSatisfy` check [ UScripts.Common]+ '1' `shouldSatisfy` check [ UScripts.Common]+ 'A' `shouldSatisfy` check [ UScripts.Latin]+ 'Α' `shouldSatisfy` check [ UScripts.Greek]+ 'α' `shouldSatisfy` check [ UScripts.Greek]+ '\x0300' `shouldSatisfy` check [ UScripts.Inherited]+ '\x0485' `shouldSatisfy` check [ UScripts.Cyrillic, UScripts.Latin]+ '\x0600' `shouldSatisfy` check [ UScripts.Arabic]+ '\x060C' `shouldSatisfy` check [ UScripts.Arabic+ , UScripts.Nko+ , UScripts.HanifiRohingya+ , UScripts.Syriac+ , UScripts.Thaana+ , UScripts.Yezidi ]+ '\x0965' `shouldSatisfy` check [ UScripts.Bengali+ , UScripts.Devanagari+ , UScripts.Dogra+ , UScripts.GunjalaGondi+ , UScripts.MasaramGondi+ , UScripts.Grantha+ , UScripts.Gujarati+ , UScripts.Gurmukhi+ , UScripts.Kannada+ , UScripts.Limbu+ , UScripts.Mahajani+ , UScripts.Malayalam+ , UScripts.Nandinagari+ , UScripts.Oriya+ , UScripts.Khudawadi+ , UScripts.Sinhala+ , UScripts.SylotiNagri+ , UScripts.Takri+ , UScripts.Tamil+ , UScripts.Telugu+ , UScripts.Tirhuta ]+ '\x1100' `shouldSatisfy` check [ UScripts.Hangul]+ '\x3001' `shouldSatisfy` check [ UScripts.Bopomofo+ , UScripts.Hangul+ , UScripts.Han+ , UScripts.Hiragana+ , UScripts.Katakana+ , UScripts.Yi ]+ '\x4E00' `shouldSatisfy` check [ UScripts.Han]+ '\x11FD0' `shouldSatisfy` check [ UScripts.Grantha, UScripts.Tamil ]+ '\x1F600' `shouldSatisfy` check [ UScripts.Common]+ '\x20000' `shouldSatisfy` check [ UScripts.Han]+ -- BOM+ '\xFEFF' `shouldSatisfy` check [ UScripts.Common ]+ '\xFFFF' `shouldSatisfy` check [ UScripts.Unknown ]+ -- Private Use Areas+ '\xE000' `shouldSatisfy` check [ UScripts.Unknown ]+ '\xF0000' `shouldSatisfy` check [ UScripts.Unknown ]+ it "Characters are in the definition of their corresponding script"+ let {+ check c =+ let s = UScripts.script c+ in if s `inScript` c+ then pure ()+ else expectationFailure $ mconcat+ [ "Char “", show c, "” in not in the definition of “"+ , show s, "”." ]+ } in traverse_ check (enumFromTo minBound maxBound)+ it "Characters in a script definition have the corresponding script"+ let {+ checkChar s c = let s' = UScripts.script c in if s' == s+ then pure ()+ else expectationFailure $ mconcat+ [ "Script is different for “", show c, "”. Expected: “"+ , show s, "” but got: “", show s', "”." ];+ check s = let chars = UScripts.scriptDefinition s+ in traverse_ (checkChar s) chars+ } in traverse_ check (enumFromTo minBound maxBound)+ it "Characters in with a script extension different from its script"+ let {+ check c =+ let script = UScripts.script c+ exts = UScripts.scriptExtensions c+ in if exts == pure script+ || (isSpecialScript script && script `notElem` exts)+ || (script `elem` exts)+ then pure ()+ else expectationFailure (show (c, script, exts));+ isSpecialScript = \case+ UScripts.Common -> True+ UScripts.Inherited -> True+ _ -> False+ } in traverse_ check (enumFromTo minBound maxBound)++{- HLINT ignore inScript "Eta reduce" -}+-- Check if a character is in a 'S.Script'.+-- This is faster than testing the string from UScripts.scriptDefinition+inScript :: S.Script -> Char -> Bool+inScript s (C# c#) = check (S.scriptDefinition s)+ where+ -- [NOTE] see 'scriptDefinition' for the description of the encoding.++ scriptRangeMask# = 0x80000000## -- 1 << 31+ maskComplement# = 0x7fffffff## -- 1 << 31 ^ 0xffffffff+ cp# = int2Word# (ord# c#)++ check (Ptr addr#, I# n#) = let {+ getRawCodePoint k# =+#ifdef WORDS_BIGENDIAN+#if MIN_VERSION_base(4,16,0)+ byteSwap32# (word32ToWord# (indexWord32OffAddr# addr# k#));+#else+ byteSwap32# (indexWord32OffAddr# addr# k#);+#endif+#elif MIN_VERSION_base(4,16,0)+ word32ToWord# (indexWord32OffAddr# addr# k#);+#else+ indexWord32OffAddr# addr# k#;+#endif+ getCodePoint k# = and# maskComplement# k#;+ find k# = not (isTrue# (k# <# 0#)) &&+ let {+ r1# = getRawCodePoint k#;+ c1# = getCodePoint r1#;+ isRange = isTrue# (and# r1# scriptRangeMask# `neWord#` 0##)+ } in if isRange+ then let {+ c2# = getCodePoint (getRawCodePoint (k# -# 1#));+ found = isTrue# ((c2# `leWord#` cp#) `andI#` (cp# `leWord#` c1#))+ } in found || find (k# -# 2#)+ else isTrue# (c1# `eqWord#` cp#) || find (k# -# 1#)+ } in find (n# -# 1#)
+ unicode-data-scripts.cabal view
@@ -0,0 +1,113 @@+cabal-version: 2.2+name: unicode-data-scripts+version: 0.1.0+synopsis: Unicode characters scripts+description:+ @unicode-data-scripts@ provides Haskell APIs to access the Unicode+ character <https://www.unicode.org/reports/tr24/ scripts> from the+ <https://www.unicode.org/ucd/ Unicode character database> (UCD).+ .+ The Haskell data structures are generated programmatically from the UCD files.+ The latest Unicode version supported by this library is+ @<https://www.unicode.org/versions/Unicode14.0.0/ 14.0.0>@.+homepage: http://github.com/composewell/unicode-data+bug-reports: https://github.com/composewell/unicode-data/issues+license: Apache-2.0+license-file: LICENSE+author: Composewell Technologies and Contributors+maintainer: dev@wismill.eu+copyright: 2022 Composewell Technologies and Contributors+category: Data,Text,Unicode+stability: Experimental+build-type: Simple+tested-with: GHC==8.0.2+ , GHC==8.2.2+ , GHC==8.4.4+ , GHC==8.6.5+ , GHC==8.8.4+ , GHC==8.10.7+ , GHC==9.0.1+ , GHC==9.2.1+ , GHC==9.4.2++extra-source-files:+ Changelog.md+ README.md++source-repository head+ type: git+ location: https://github.com/composewell/unicode-data++common default-extensions+ default-extensions:+ BangPatterns+ DeriveGeneric+ MagicHash+ RecordWildCards+ ScopedTypeVariables+ TupleSections+ FlexibleContexts+ LambdaCase++ -- Experimental, may lead to issues+ DeriveAnyClass+ TemplateHaskell+ UnboxedTuples++common compile-options+ ghc-options: -Wall+ -fwarn-identities+ -fwarn-incomplete-record-updates+ -fwarn-incomplete-uni-patterns+ -fwarn-tabs+ default-language: Haskell2010++library+ import: default-extensions, compile-options+ default-language: Haskell2010+ exposed-modules:+ -- The module structure is derived from+ -- https://www.unicode.org/reports/tr44/#Property_Index_Table+ Unicode.Char.General.Scripts++ -- Generated files+ -- This module structure is largely based on the UCD file names from which+ -- the properties are generated.+ Unicode.Internal.Char.Scripts+ Unicode.Internal.Char.ScriptExtensions++ hs-source-dirs: lib+ build-depends:+ base >= 4.7 && < 4.18+ , unicode-data >= 0.3 && < 0.4++test-suite test+ import: default-extensions, compile-options+ type: exitcode-stdio-1.0+ main-is: Main.hs+ hs-source-dirs:+ test+ other-modules:+ Unicode.Char.General.ScriptsSpec+ build-depends:+ base >= 4.7 && < 4.18+ , hspec >= 2.0 && < 2.11+ , unicode-data+ , unicode-data-scripts+ build-tool-depends:+ hspec-discover:hspec-discover >= 2.0 && < 2.11+ default-language: Haskell2010++benchmark bench+ import: default-extensions, compile-options+ type: exitcode-stdio-1.0+ hs-source-dirs: bench+ main-is: Main.hs+ build-depends:+ base >= 4.7 && < 4.18,+ deepseq >= 1.1 && < 1.5,+ tasty-bench >= 0.2.5 && < 0.4,+ tasty >= 1.4.1,+ unicode-data,+ unicode-data-scripts+ ghc-options: -O2 -fdicts-strict -rtsopts