hxt-8.0.0: doc/hdoc/src/Data-Char-UTF8.html
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
<head>
<!-- Generated by HsColour, http://www.cs.york.ac.uk/fp/darcs/hscolour/ -->
<title>Haskell Code by HsColour</title>
<link type='text/css' rel='stylesheet' href='hscolour.css' />
</head>
<body>
<pre><a name="line-1"></a><span class='comment'>{-
<a name="line-2"></a>
<a name="line-3"></a>Copyright (c) 2002, members of the Haskell Internationalisation Working
<a name="line-4"></a>Group All rights reserved.
<a name="line-5"></a>
<a name="line-6"></a>Redistribution and use in source and binary forms, with or without
<a name="line-7"></a>modification, are permitted provided that the following conditions are met:
<a name="line-8"></a>
<a name="line-9"></a>* Redistributions of source code must retain the above copyright notice,
<a name="line-10"></a> this list of conditions and the following disclaimer.
<a name="line-11"></a>* Redistributions in binary form must reproduce the above copyright notice,
<a name="line-12"></a> this list of conditions and the following disclaimer in the
<a name="line-13"></a> documentation and/or other materials provided with the distribution.
<a name="line-14"></a>* Neither the name of the Haskell Internationalisation Working Group nor
<a name="line-15"></a> the names of its contributors may be used to endorse or promote products
<a name="line-16"></a> derived from this software without specific prior written permission.
<a name="line-17"></a>
<a name="line-18"></a>THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
<a name="line-19"></a>AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
<a name="line-20"></a>IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
<a name="line-21"></a>ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
<a name="line-22"></a>LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
<a name="line-23"></a>CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
<a name="line-24"></a>SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
<a name="line-25"></a>INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
<a name="line-26"></a>CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
<a name="line-27"></a>ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
<a name="line-28"></a>POSSIBILITY OF SUCH DAMAGE.
<a name="line-29"></a>
<a name="line-30"></a>This module provides lazy stream encoding/decoding facilities for UTF-8,
<a name="line-31"></a>the Unicode Transformation Format with 8-bit words.
<a name="line-32"></a>
<a name="line-33"></a>2002-09-02 Sven Moritz Hallberg <pesco@gmx.de>
<a name="line-34"></a>
<a name="line-35"></a>-}</span>
<a name="line-36"></a>
<a name="line-37"></a><span class='comment'>{-
<a name="line-38"></a>
<a name="line-39"></a>2007-04-30 Henning Thielemann:
<a name="line-40"></a>Slight changes to make decode lazy.
<a name="line-41"></a>The calls of 'reverse' in the original version have broken laziness
<a name="line-42"></a>and thus had memory leaks.
<a name="line-43"></a>
<a name="line-44"></a>-}</span>
<a name="line-45"></a>
<a name="line-46"></a><span class='keyword'>module</span> <span class='conid'>Data</span><span class='varop'>.</span><span class='conid'>Char</span><span class='varop'>.</span><span class='conid'>UTF8</span>
<a name="line-47"></a> <span class='layout'>(</span> <span class='varid'>encode</span><span class='layout'>,</span> <span class='varid'>decode</span><span class='layout'>,</span>
<a name="line-48"></a> <span class='varid'>decodeEmbedErrors</span><span class='layout'>,</span>
<a name="line-49"></a> <span class='varid'>encodeOne</span><span class='layout'>,</span> <span class='varid'>decodeOne</span><span class='layout'>,</span>
<a name="line-50"></a> <span class='conid'>Error</span><span class='layout'>,</span> <span class='comment'>-- Haddock does not want to document signatures with private types</span>
<a name="line-51"></a> <span class='comment'>-- these functions should be moved to a utility module</span>
<a name="line-52"></a> <span class='layout'>)</span> <span class='keyword'>where</span>
<a name="line-53"></a>
<a name="line-54"></a><span class='keyword'>import</span> <span class='conid'>Data</span><span class='varop'>.</span><span class='conid'>Char</span> <span class='layout'>(</span><span class='varid'>ord</span><span class='layout'>,</span> <span class='varid'>chr</span><span class='layout'>)</span>
<a name="line-55"></a><span class='keyword'>import</span> <span class='conid'>Data</span><span class='varop'>.</span><span class='conid'>Word</span> <span class='layout'>(</span><span class='conid'>Word8</span><span class='layout'>,</span> <span class='conid'>Word16</span><span class='layout'>,</span> <span class='conid'>Word32</span><span class='layout'>)</span>
<a name="line-56"></a><span class='keyword'>import</span> <span class='conid'>Data</span><span class='varop'>.</span><span class='conid'>Bits</span> <span class='layout'>(</span><span class='conid'>Bits</span><span class='layout'>,</span> <span class='varid'>shiftL</span><span class='layout'>,</span> <span class='varid'>shiftR</span><span class='layout'>,</span> <span class='layout'>(</span><span class='varop'>.&.</span><span class='layout'>)</span><span class='layout'>,</span> <span class='layout'>(</span><span class='varop'>.|.</span><span class='layout'>)</span><span class='layout'>)</span>
<a name="line-57"></a>
<a name="line-58"></a><span class='keyword'>import</span> <span class='conid'>Data</span><span class='varop'>.</span><span class='conid'>List</span> <span class='layout'>(</span><span class='varid'>unfoldr</span><span class='layout'>)</span>
<a name="line-59"></a>
<a name="line-60"></a><span class='keyword'>import</span> <span class='conid'>Text</span><span class='varop'>.</span><span class='conid'>XML</span><span class='varop'>.</span><span class='conid'>HXT</span><span class='varop'>.</span><span class='conid'>DOM</span><span class='varop'>.</span><span class='conid'>Util</span> <span class='layout'>(</span><span class='varid'>partitionEither</span><span class='layout'>,</span> <span class='varid'>swap</span><span class='layout'>,</span> <span class='varid'>toMaybe</span><span class='layout'>)</span>
<a name="line-61"></a>
<a name="line-62"></a>
<a name="line-63"></a>
<a name="line-64"></a><span class='comment'>-- - UTF-8 in General -</span>
<a name="line-65"></a>
<a name="line-66"></a><span class='comment'>-- Adapted from the Unicode standard, version 3.2,</span>
<a name="line-67"></a><span class='comment'>-- Table 3.1 "UTF-8 Bit Distribution" (excluded are UTF-16 encodings):</span>
<a name="line-68"></a>
<a name="line-69"></a><span class='comment'>-- Scalar 1st Byte 2nd Byte 3rd Byte 4th Byte</span>
<a name="line-70"></a><span class='comment'>-- 000000000xxxxxxx 0xxxxxxx</span>
<a name="line-71"></a><span class='comment'>-- 00000yyyyyxxxxxx 110yyyyy 10xxxxxx</span>
<a name="line-72"></a><span class='comment'>-- zzzzyyyyyyxxxxxx 1110zzzz 10yyyyyy 10xxxxxx</span>
<a name="line-73"></a><span class='comment'>-- 000uuuzzzzzzyyyyyyxxxxxx 11110uuu 10zzzzzz 10yyyyyy 10xxxxxx</span>
<a name="line-74"></a>
<a name="line-75"></a><span class='comment'>-- Also from the Unicode standard, version 3.2,</span>
<a name="line-76"></a><span class='comment'>-- Table 3.1B "Legal UTF-8 Byte Sequences":</span>
<a name="line-77"></a>
<a name="line-78"></a><span class='comment'>-- Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte</span>
<a name="line-79"></a><span class='comment'>-- U+0000..U+007F 00..7F</span>
<a name="line-80"></a><span class='comment'>-- U+0080..U+07FF C2..DF 80..BF</span>
<a name="line-81"></a><span class='comment'>-- U+0800..U+0FFF E0 A0..BF 80..BF</span>
<a name="line-82"></a><span class='comment'>-- U+1000..U+CFFF E1..EC 80..BF 80..BF</span>
<a name="line-83"></a><span class='comment'>-- U+D000..U+D7FF ED 80..9F 80..BF</span>
<a name="line-84"></a><span class='comment'>-- U+D800..U+DFFF ill-formed</span>
<a name="line-85"></a><span class='comment'>-- U+E000..U+FFFF EE..EF 80..BF 80..BF</span>
<a name="line-86"></a><span class='comment'>-- U+10000..U+3FFFF F0 90..BF 80..BF 80..BF</span>
<a name="line-87"></a><span class='comment'>-- U+40000..U+FFFFF F1..F3 80..BF 80..BF 80..BF</span>
<a name="line-88"></a><span class='comment'>-- U+100000..U+10FFFF F4 80..8F 80..BF 80..BF</span>
<a name="line-89"></a>
<a name="line-90"></a>
<a name="line-91"></a>
<a name="line-92"></a><span class='comment'>-- - Encoding Functions -</span>
<a name="line-93"></a>
<a name="line-94"></a><span class='comment'>-- Must the encoder ensure that no illegal byte sequences are output or</span>
<a name="line-95"></a><span class='comment'>-- can we trust the Haskell system to supply only legal values?</span>
<a name="line-96"></a><span class='comment'>-- For now I include error case for the surrogate values U+D800..U+DFFF and</span>
<a name="line-97"></a><span class='comment'>-- out-of-range scalars.</span>
<a name="line-98"></a>
<a name="line-99"></a><span class='comment'>-- The function is pretty much a transscript of table 3.1B with error checks.</span>
<a name="line-100"></a><span class='comment'>-- It dispatches the actual encoding to functions specific to the number of</span>
<a name="line-101"></a><span class='comment'>-- required bytes.</span>
<a name="line-102"></a>
<a name="line-103"></a><a name="encodeOne"></a><span class='definition'>encodeOne</span> <span class='keyglyph'>::</span> <span class='conid'>Char</span> <span class='keyglyph'>-></span> <span class='keyglyph'>[</span><span class='conid'>Word8</span><span class='keyglyph'>]</span>
<a name="line-104"></a><span class='definition'>encodeOne</span> <span class='varid'>c</span>
<a name="line-105"></a> <span class='comment'>-- The report guarantees in (6.1.2) that this won't happen:</span>
<a name="line-106"></a> <span class='comment'>-- | n < 0 = error "encodeUTF8: ord returned a negative value"</span>
<a name="line-107"></a> <span class='keyglyph'>|</span> <span class='varid'>n</span> <span class='varop'><</span> <span class='num'>0x0080</span> <span class='keyglyph'>=</span> <span class='varid'>encodeOne_onebyte</span> <span class='varid'>n8</span>
<a name="line-108"></a> <span class='keyglyph'>|</span> <span class='varid'>n</span> <span class='varop'><</span> <span class='num'>0x0800</span> <span class='keyglyph'>=</span> <span class='varid'>encodeOne_twobyte</span> <span class='varid'>n16</span>
<a name="line-109"></a> <span class='keyglyph'>|</span> <span class='varid'>n</span> <span class='varop'><</span> <span class='num'>0xD800</span> <span class='keyglyph'>=</span> <span class='varid'>encodeOne_threebyte</span> <span class='varid'>n16</span>
<a name="line-110"></a> <span class='keyglyph'>|</span> <span class='varid'>n</span> <span class='varop'><</span> <span class='num'>0xE000</span> <span class='keyglyph'>=</span> <span class='varid'>error</span> <span class='str'>"encodeUTF8: ord returned a surrogate value"</span>
<a name="line-111"></a> <span class='keyglyph'>|</span> <span class='varid'>n</span> <span class='varop'><</span> <span class='num'>0x10000</span> <span class='keyglyph'>=</span> <span class='varid'>encodeOne_threebyte</span> <span class='varid'>n16</span>
<a name="line-112"></a> <span class='comment'>-- Haskell 98 only talks about 16 bit characters, but ghc handles 20.1.</span>
<a name="line-113"></a> <span class='keyglyph'>|</span> <span class='varid'>n</span> <span class='varop'><</span> <span class='num'>0x10FFFF</span> <span class='keyglyph'>=</span> <span class='varid'>encodeOne_fourbyte</span> <span class='varid'>n32</span>
<a name="line-114"></a> <span class='keyglyph'>|</span> <span class='varid'>otherwise</span> <span class='keyglyph'>=</span> <span class='varid'>error</span> <span class='str'>"encodeUTF8: ord returned a value above 0x10FFFF"</span>
<a name="line-115"></a> <span class='keyword'>where</span>
<a name="line-116"></a> <span class='varid'>n</span> <span class='keyglyph'>=</span> <span class='varid'>ord</span> <span class='varid'>c</span> <span class='keyglyph'>::</span> <span class='conid'>Int</span>
<a name="line-117"></a> <span class='varid'>n8</span> <span class='keyglyph'>=</span> <span class='varid'>fromIntegral</span> <span class='varid'>n</span> <span class='keyglyph'>::</span> <span class='conid'>Word8</span>
<a name="line-118"></a> <span class='varid'>n16</span> <span class='keyglyph'>=</span> <span class='varid'>fromIntegral</span> <span class='varid'>n</span> <span class='keyglyph'>::</span> <span class='conid'>Word16</span>
<a name="line-119"></a> <span class='varid'>n32</span> <span class='keyglyph'>=</span> <span class='varid'>fromIntegral</span> <span class='varid'>n</span> <span class='keyglyph'>::</span> <span class='conid'>Word32</span>
<a name="line-120"></a>
<a name="line-121"></a>
<a name="line-122"></a><span class='comment'>-- With the above, a stream decoder is trivial:</span>
<a name="line-123"></a>
<a name="line-124"></a><a name="encode"></a><span class='definition'>encode</span> <span class='keyglyph'>::</span> <span class='keyglyph'>[</span><span class='conid'>Char</span><span class='keyglyph'>]</span> <span class='keyglyph'>-></span> <span class='keyglyph'>[</span><span class='conid'>Word8</span><span class='keyglyph'>]</span>
<a name="line-125"></a><span class='definition'>encode</span> <span class='keyglyph'>=</span> <span class='varid'>concatMap</span> <span class='varid'>encodeOne</span>
<a name="line-126"></a>
<a name="line-127"></a>
<a name="line-128"></a><span class='comment'>-- Now follow the individual encoders for certain numbers of bytes...</span>
<a name="line-129"></a><span class='comment'>-- _</span>
<a name="line-130"></a><span class='comment'>-- / | __ ___ __ __</span>
<a name="line-131"></a><span class='comment'>-- / ^| // /__/ // //</span>
<a name="line-132"></a><span class='comment'>-- /.==| \\ //_ // //</span>
<a name="line-133"></a><span class='comment'>-- It's // || // \_/_//_//_ and it's here to stay!</span>
<a name="line-134"></a>
<a name="line-135"></a><a name="encodeOne_onebyte"></a><span class='definition'>encodeOne_onebyte</span> <span class='keyglyph'>::</span> <span class='conid'>Word8</span> <span class='keyglyph'>-></span> <span class='keyglyph'>[</span><span class='conid'>Word8</span><span class='keyglyph'>]</span>
<a name="line-136"></a><span class='definition'>encodeOne_onebyte</span> <span class='varid'>cp</span> <span class='keyglyph'>=</span> <span class='keyglyph'>[</span><span class='varid'>cp</span><span class='keyglyph'>]</span>
<a name="line-137"></a>
<a name="line-138"></a>
<a name="line-139"></a><span class='comment'>-- 00000yyyyyxxxxxx -> 110yyyyy 10xxxxxx</span>
<a name="line-140"></a>
<a name="line-141"></a><a name="encodeOne_twobyte"></a><span class='definition'>encodeOne_twobyte</span> <span class='keyglyph'>::</span> <span class='conid'>Word16</span> <span class='keyglyph'>-></span> <span class='keyglyph'>[</span><span class='conid'>Word8</span><span class='keyglyph'>]</span>
<a name="line-142"></a><span class='definition'>encodeOne_twobyte</span> <span class='varid'>cp</span> <span class='keyglyph'>=</span> <span class='keyglyph'>[</span><span class='layout'>(</span><span class='num'>0xC0</span><span class='varop'>.|.</span><span class='varid'>ys</span><span class='layout'>)</span><span class='layout'>,</span> <span class='layout'>(</span><span class='num'>0x80</span><span class='varop'>.|.</span><span class='varid'>xs</span><span class='layout'>)</span><span class='keyglyph'>]</span>
<a name="line-143"></a> <span class='keyword'>where</span>
<a name="line-144"></a> <span class='varid'>xs</span><span class='layout'>,</span> <span class='varid'>ys</span> <span class='keyglyph'>::</span> <span class='conid'>Word8</span>
<a name="line-145"></a> <span class='varid'>ys</span> <span class='keyglyph'>=</span> <span class='varid'>fromIntegral</span> <span class='layout'>(</span><span class='varid'>shiftR</span> <span class='varid'>cp</span> <span class='num'>6</span><span class='layout'>)</span>
<a name="line-146"></a> <span class='varid'>xs</span> <span class='keyglyph'>=</span> <span class='layout'>(</span><span class='varid'>fromIntegral</span> <span class='varid'>cp</span><span class='layout'>)</span> <span class='varop'>.&.</span> <span class='num'>0x3F</span>
<a name="line-147"></a>
<a name="line-148"></a>
<a name="line-149"></a><span class='comment'>-- zzzzyyyyyyxxxxxx -> 1110zzzz 10yyyyyy 10xxxxxx</span>
<a name="line-150"></a>
<a name="line-151"></a><a name="encodeOne_threebyte"></a><span class='definition'>encodeOne_threebyte</span> <span class='keyglyph'>::</span> <span class='conid'>Word16</span> <span class='keyglyph'>-></span> <span class='keyglyph'>[</span><span class='conid'>Word8</span><span class='keyglyph'>]</span>
<a name="line-152"></a><span class='definition'>encodeOne_threebyte</span> <span class='varid'>cp</span> <span class='keyglyph'>=</span> <span class='keyglyph'>[</span><span class='layout'>(</span><span class='num'>0xE0</span><span class='varop'>.|.</span><span class='varid'>zs</span><span class='layout'>)</span><span class='layout'>,</span> <span class='layout'>(</span><span class='num'>0x80</span><span class='varop'>.|.</span><span class='varid'>ys</span><span class='layout'>)</span><span class='layout'>,</span> <span class='layout'>(</span><span class='num'>0x80</span><span class='varop'>.|.</span><span class='varid'>xs</span><span class='layout'>)</span><span class='keyglyph'>]</span>
<a name="line-153"></a> <span class='keyword'>where</span>
<a name="line-154"></a> <span class='varid'>xs</span><span class='layout'>,</span> <span class='varid'>ys</span><span class='layout'>,</span> <span class='varid'>zs</span> <span class='keyglyph'>::</span> <span class='conid'>Word8</span>
<a name="line-155"></a> <span class='varid'>xs</span> <span class='keyglyph'>=</span> <span class='layout'>(</span><span class='varid'>fromIntegral</span> <span class='varid'>cp</span><span class='layout'>)</span> <span class='varop'>.&.</span> <span class='num'>0x3F</span>
<a name="line-156"></a> <span class='varid'>ys</span> <span class='keyglyph'>=</span> <span class='layout'>(</span><span class='varid'>fromIntegral</span> <span class='layout'>(</span><span class='varid'>shiftR</span> <span class='varid'>cp</span> <span class='num'>6</span><span class='layout'>)</span><span class='layout'>)</span> <span class='varop'>.&.</span> <span class='num'>0x3F</span>
<a name="line-157"></a> <span class='varid'>zs</span> <span class='keyglyph'>=</span> <span class='varid'>fromIntegral</span> <span class='layout'>(</span><span class='varid'>shiftR</span> <span class='varid'>cp</span> <span class='num'>12</span><span class='layout'>)</span>
<a name="line-158"></a>
<a name="line-159"></a>
<a name="line-160"></a><span class='comment'>-- 000uuuzzzzzzyyyyyyxxxxxx -> 11110uuu 10zzzzzz 10yyyyyy 10xxxxxx</span>
<a name="line-161"></a>
<a name="line-162"></a><a name="encodeOne_fourbyte"></a><span class='definition'>encodeOne_fourbyte</span> <span class='keyglyph'>::</span> <span class='conid'>Word32</span> <span class='keyglyph'>-></span> <span class='keyglyph'>[</span><span class='conid'>Word8</span><span class='keyglyph'>]</span>
<a name="line-163"></a><span class='definition'>encodeOne_fourbyte</span> <span class='varid'>cp</span> <span class='keyglyph'>=</span> <span class='keyglyph'>[</span><span class='num'>0xF0</span><span class='varop'>.|.</span><span class='varid'>us</span><span class='layout'>,</span> <span class='num'>0x80</span><span class='varop'>.|.</span><span class='varid'>zs</span><span class='layout'>,</span> <span class='num'>0x80</span><span class='varop'>.|.</span><span class='varid'>ys</span><span class='layout'>,</span> <span class='num'>0x80</span><span class='varop'>.|.</span><span class='varid'>xs</span><span class='keyglyph'>]</span>
<a name="line-164"></a> <span class='keyword'>where</span>
<a name="line-165"></a> <span class='varid'>xs</span><span class='layout'>,</span> <span class='varid'>ys</span><span class='layout'>,</span> <span class='varid'>zs</span><span class='layout'>,</span> <span class='varid'>us</span> <span class='keyglyph'>::</span> <span class='conid'>Word8</span>
<a name="line-166"></a> <span class='varid'>xs</span> <span class='keyglyph'>=</span> <span class='layout'>(</span><span class='varid'>fromIntegral</span> <span class='varid'>cp</span><span class='layout'>)</span> <span class='varop'>.&.</span> <span class='num'>0x3F</span>
<a name="line-167"></a> <span class='varid'>ys</span> <span class='keyglyph'>=</span> <span class='layout'>(</span><span class='varid'>fromIntegral</span> <span class='layout'>(</span><span class='varid'>shiftR</span> <span class='varid'>cp</span> <span class='num'>6</span><span class='layout'>)</span><span class='layout'>)</span> <span class='varop'>.&.</span> <span class='num'>0x3F</span>
<a name="line-168"></a> <span class='varid'>zs</span> <span class='keyglyph'>=</span> <span class='layout'>(</span><span class='varid'>fromIntegral</span> <span class='layout'>(</span><span class='varid'>shiftR</span> <span class='varid'>cp</span> <span class='num'>12</span><span class='layout'>)</span><span class='layout'>)</span> <span class='varop'>.&.</span> <span class='num'>0x3F</span>
<a name="line-169"></a> <span class='varid'>us</span> <span class='keyglyph'>=</span> <span class='varid'>fromIntegral</span> <span class='layout'>(</span><span class='varid'>shiftR</span> <span class='varid'>cp</span> <span class='num'>18</span><span class='layout'>)</span>
<a name="line-170"></a>
<a name="line-171"></a>
<a name="line-172"></a>
<a name="line-173"></a><span class='comment'>-- - Decoding -</span>
<a name="line-174"></a>
<a name="line-175"></a><span class='comment'>-- The decoding is a bit more involved. The byte sequence could contain all</span>
<a name="line-176"></a><span class='comment'>-- sorts of corruptions. The user must be able to either notice or ignore these</span>
<a name="line-177"></a><span class='comment'>-- errors.</span>
<a name="line-178"></a>
<a name="line-179"></a><span class='comment'>-- I will first look at the decoding of a single character. The process</span>
<a name="line-180"></a><span class='comment'>-- consumes a certain number of bytes from the input. It returns the</span>
<a name="line-181"></a><span class='comment'>-- remaining input and either an error and the index of its occurance in the</span>
<a name="line-182"></a><span class='comment'>-- byte sequence or the decoded character.</span>
<a name="line-183"></a>
<a name="line-184"></a><a name="Error"></a><span class='keyword'>data</span> <span class='conid'>Error</span>
<a name="line-185"></a>
<a name="line-186"></a><span class='comment'>-- The first byte in a sequence starts with either zero, two, three, or four</span>
<a name="line-187"></a><span class='comment'>-- ones and one zero to indicate the length of the sequence. If it doesn't,</span>
<a name="line-188"></a><span class='comment'>-- it is invalid. It is dropped and the next byte interpreted as the start</span>
<a name="line-189"></a><span class='comment'>-- of a new sequence.</span>
<a name="line-190"></a>
<a name="line-191"></a> <span class='keyglyph'>=</span> <span class='conid'>InvalidFirstByte</span>
<a name="line-192"></a>
<a name="line-193"></a><span class='comment'>-- All bytes in the sequence except the first match the bit pattern 10xxxxxx.</span>
<a name="line-194"></a><span class='comment'>-- If one doesn't, it is invalid. The sequence up to that point is dropped</span>
<a name="line-195"></a><span class='comment'>-- and the "invalid" byte interpreted as the start of a new sequence. The error</span>
<a name="line-196"></a><span class='comment'>-- includes the length of the partial sequence and the number of expected bytes.</span>
<a name="line-197"></a>
<a name="line-198"></a> <span class='keyglyph'>|</span> <span class='conid'>InvalidLaterByte</span> <span class='conid'>Int</span> <span class='comment'>-- the byte at relative index n was invalid</span>
<a name="line-199"></a>
<a name="line-200"></a><span class='comment'>-- If a sequence ends prematurely, it has been truncated. It dropped and</span>
<a name="line-201"></a><span class='comment'>-- decoding stops. The error reports the actual and expected lengths of the</span>
<a name="line-202"></a><span class='comment'>-- sequence.</span>
<a name="line-203"></a>
<a name="line-204"></a> <span class='keyglyph'>|</span> <span class='conid'>Truncated</span> <span class='conid'>Int</span> <span class='conid'>Int</span> <span class='comment'>-- only n of m expected bytes were present</span>
<a name="line-205"></a>
<a name="line-206"></a><span class='comment'>-- Some sequences would represent code points which would be encoded as a</span>
<a name="line-207"></a><span class='comment'>-- shorter sequence by a conformant encoder. Such non-shortest sequences are</span>
<a name="line-208"></a><span class='comment'>-- considered erroneous and dropped. The error reports the actual and</span>
<a name="line-209"></a><span class='comment'>-- expected number of bytes used.</span>
<a name="line-210"></a>
<a name="line-211"></a> <span class='keyglyph'>|</span> <span class='conid'>NonShortest</span> <span class='conid'>Int</span> <span class='conid'>Int</span> <span class='comment'>-- n instead of m bytes were used</span>
<a name="line-212"></a>
<a name="line-213"></a><span class='comment'>-- Unicode code points are in the range of [0..0x10FFFF]. Any values outside</span>
<a name="line-214"></a><span class='comment'>-- of those bounds are simply invalid.</span>
<a name="line-215"></a>
<a name="line-216"></a> <span class='keyglyph'>|</span> <span class='conid'>ValueOutOfBounds</span>
<a name="line-217"></a>
<a name="line-218"></a><span class='comment'>-- There is no such thing as "surrogate pairs" any more in UTF-8. The</span>
<a name="line-219"></a><span class='comment'>-- corresponding code points now form illegal byte sequences.</span>
<a name="line-220"></a>
<a name="line-221"></a> <span class='keyglyph'>|</span> <span class='conid'>Surrogate</span>
<a name="line-222"></a> <span class='keyword'>deriving</span> <span class='layout'>(</span><span class='conid'>Show</span><span class='layout'>,</span> <span class='conid'>Eq</span><span class='layout'>)</span>
<a name="line-223"></a>
<a name="line-224"></a>
<a name="line-225"></a><span class='comment'>-- Second, third, and fourth bytes share the common requirement to start</span>
<a name="line-226"></a><span class='comment'>-- with the bit sequence 10. So, here's the function to check that property.</span>
<a name="line-227"></a>
<a name="line-228"></a><a name="first_bits_not_10"></a><span class='definition'>first_bits_not_10</span> <span class='keyglyph'>::</span> <span class='conid'>Word8</span> <span class='keyglyph'>-></span> <span class='conid'>Bool</span>
<a name="line-229"></a><span class='definition'>first_bits_not_10</span> <span class='varid'>b</span>
<a name="line-230"></a> <span class='keyglyph'>|</span> <span class='layout'>(</span><span class='varid'>b</span><span class='varop'>.&.</span><span class='num'>0xC0</span><span class='layout'>)</span> <span class='varop'>/=</span> <span class='num'>0x80</span> <span class='keyglyph'>=</span> <span class='conid'>True</span>
<a name="line-231"></a> <span class='keyglyph'>|</span> <span class='varid'>otherwise</span> <span class='keyglyph'>=</span> <span class='conid'>False</span>
<a name="line-232"></a>
<a name="line-233"></a>
<a name="line-234"></a><span class='comment'>-- Erm, OK, the single-character decoding function's return type is a bit</span>
<a name="line-235"></a><span class='comment'>-- longish. It is a tripel:</span>
<a name="line-236"></a>
<a name="line-237"></a><span class='comment'>-- - The first component contains the decoded character or an error</span>
<a name="line-238"></a><span class='comment'>-- if the byte sequence was erroneous.</span>
<a name="line-239"></a><span class='comment'>-- - The second component contains the number of bytes that were consumed</span>
<a name="line-240"></a><span class='comment'>-- from the input.</span>
<a name="line-241"></a><span class='comment'>-- - The third component contains the remaining bytes of input.</span>
<a name="line-242"></a>
<a name="line-243"></a><a name="decodeOne"></a><span class='definition'>decodeOne</span> <span class='keyglyph'>::</span> <span class='keyglyph'>[</span><span class='conid'>Word8</span><span class='keyglyph'>]</span> <span class='keyglyph'>-></span> <span class='layout'>(</span><span class='conid'>Either</span> <span class='conid'>Error</span> <span class='conid'>Char</span><span class='layout'>,</span> <span class='conid'>Int</span><span class='layout'>,</span> <span class='keyglyph'>[</span><span class='conid'>Word8</span><span class='keyglyph'>]</span><span class='layout'>)</span>
<a name="line-244"></a><span class='definition'>decodeOne</span> <span class='varid'>bs</span><span class='keyglyph'>@</span><span class='layout'>(</span><span class='varid'>b1</span><span class='conop'>:</span><span class='varid'>rest</span><span class='layout'>)</span>
<a name="line-245"></a> <span class='keyglyph'>|</span> <span class='varid'>b1</span> <span class='varop'><</span> <span class='num'>0x80</span> <span class='keyglyph'>=</span> <span class='varid'>decodeOne_onebyte</span> <span class='varid'>bs</span>
<a name="line-246"></a> <span class='keyglyph'>|</span> <span class='varid'>b1</span> <span class='varop'><</span> <span class='num'>0xC0</span> <span class='keyglyph'>=</span> <span class='layout'>(</span><span class='conid'>Left</span> <span class='conid'>InvalidFirstByte</span><span class='layout'>,</span> <span class='num'>1</span><span class='layout'>,</span> <span class='varid'>rest</span><span class='layout'>)</span>
<a name="line-247"></a> <span class='keyglyph'>|</span> <span class='varid'>b1</span> <span class='varop'><</span> <span class='num'>0xE0</span> <span class='keyglyph'>=</span> <span class='varid'>decodeOne_twobyte</span> <span class='varid'>bs</span>
<a name="line-248"></a> <span class='keyglyph'>|</span> <span class='varid'>b1</span> <span class='varop'><</span> <span class='num'>0xF0</span> <span class='keyglyph'>=</span> <span class='varid'>decodeOne_threebyte</span> <span class='varid'>bs</span>
<a name="line-249"></a> <span class='keyglyph'>|</span> <span class='varid'>b1</span> <span class='varop'><</span> <span class='num'>0xF5</span> <span class='keyglyph'>=</span> <span class='varid'>decodeOne_fourbyte</span> <span class='varid'>bs</span>
<a name="line-250"></a> <span class='keyglyph'>|</span> <span class='varid'>otherwise</span> <span class='keyglyph'>=</span> <span class='layout'>(</span><span class='conid'>Left</span> <span class='conid'>ValueOutOfBounds</span><span class='layout'>,</span> <span class='num'>1</span><span class='layout'>,</span> <span class='varid'>rest</span><span class='layout'>)</span>
<a name="line-251"></a><span class='definition'>decodeOne</span> <span class='conid'>[]</span> <span class='keyglyph'>=</span> <span class='varid'>error</span> <span class='str'>"UTF8.decodeOne: No input"</span>
<a name="line-252"></a>
<a name="line-253"></a>
<a name="line-254"></a><span class='comment'>-- 0xxxxxxx -> 000000000xxxxxxx</span>
<a name="line-255"></a>
<a name="line-256"></a><a name="decodeOne_onebyte"></a><span class='definition'>decodeOne_onebyte</span> <span class='keyglyph'>::</span> <span class='keyglyph'>[</span><span class='conid'>Word8</span><span class='keyglyph'>]</span> <span class='keyglyph'>-></span> <span class='layout'>(</span><span class='conid'>Either</span> <span class='conid'>Error</span> <span class='conid'>Char</span><span class='layout'>,</span> <span class='conid'>Int</span><span class='layout'>,</span> <span class='keyglyph'>[</span><span class='conid'>Word8</span><span class='keyglyph'>]</span><span class='layout'>)</span>
<a name="line-257"></a><span class='definition'>decodeOne_onebyte</span> <span class='layout'>(</span><span class='varid'>b</span><span class='conop'>:</span><span class='varid'>bs</span><span class='layout'>)</span> <span class='keyglyph'>=</span> <span class='layout'>(</span><span class='conid'>Right</span> <span class='layout'>(</span><span class='varid'>cpToChar</span> <span class='varid'>b</span><span class='layout'>)</span><span class='layout'>,</span> <span class='num'>1</span><span class='layout'>,</span> <span class='varid'>bs</span><span class='layout'>)</span>
<a name="line-258"></a><span class='definition'>decodeOne_onebyte</span><span class='conid'>[]</span> <span class='keyglyph'>=</span> <span class='varid'>error</span> <span class='str'>"UTF8.decodeOne_onebyte: No input (can't happen)"</span>
<a name="line-259"></a>
<a name="line-260"></a><a name="cpToChar"></a><span class='definition'>cpToChar</span> <span class='keyglyph'>::</span> <span class='conid'>Integral</span> <span class='varid'>a</span> <span class='keyglyph'>=></span> <span class='varid'>a</span> <span class='keyglyph'>-></span> <span class='conid'>Char</span>
<a name="line-261"></a><span class='definition'>cpToChar</span> <span class='keyglyph'>=</span> <span class='varid'>chr</span> <span class='varop'>.</span> <span class='varid'>fromIntegral</span>
<a name="line-262"></a>
<a name="line-263"></a>
<a name="line-264"></a><span class='comment'>-- 110yyyyy 10xxxxxx -> 00000yyyyyxxxxxx</span>
<a name="line-265"></a>
<a name="line-266"></a><a name="decodeOne_twobyte"></a><span class='definition'>decodeOne_twobyte</span> <span class='keyglyph'>::</span> <span class='keyglyph'>[</span><span class='conid'>Word8</span><span class='keyglyph'>]</span> <span class='keyglyph'>-></span> <span class='layout'>(</span><span class='conid'>Either</span> <span class='conid'>Error</span> <span class='conid'>Char</span><span class='layout'>,</span> <span class='conid'>Int</span><span class='layout'>,</span> <span class='keyglyph'>[</span><span class='conid'>Word8</span><span class='keyglyph'>]</span><span class='layout'>)</span>
<a name="line-267"></a><span class='definition'>decodeOne_twobyte</span> <span class='layout'>(</span><span class='keyword'>_</span><span class='conop'>:</span><span class='conid'>[]</span><span class='layout'>)</span>
<a name="line-268"></a> <span class='keyglyph'>=</span> <span class='layout'>(</span><span class='conid'>Left</span> <span class='layout'>(</span><span class='conid'>Truncated</span> <span class='num'>1</span> <span class='num'>2</span><span class='layout'>)</span><span class='layout'>,</span> <span class='num'>1</span><span class='layout'>,</span> <span class='conid'>[]</span><span class='layout'>)</span>
<a name="line-269"></a><span class='definition'>decodeOne_twobyte</span> <span class='layout'>(</span><span class='varid'>b1</span><span class='conop'>:</span><span class='varid'>b2</span><span class='conop'>:</span><span class='varid'>bs</span><span class='layout'>)</span>
<a name="line-270"></a> <span class='keyglyph'>|</span> <span class='varid'>b1</span> <span class='varop'><</span> <span class='num'>0xC2</span> <span class='keyglyph'>=</span> <span class='layout'>(</span><span class='conid'>Left</span> <span class='layout'>(</span><span class='conid'>NonShortest</span> <span class='num'>2</span> <span class='num'>1</span><span class='layout'>)</span><span class='layout'>,</span> <span class='num'>2</span><span class='layout'>,</span> <span class='varid'>bs</span><span class='layout'>)</span>
<a name="line-271"></a> <span class='keyglyph'>|</span> <span class='varid'>first_bits_not_10</span> <span class='varid'>b2</span> <span class='keyglyph'>=</span> <span class='layout'>(</span><span class='conid'>Left</span> <span class='layout'>(</span><span class='conid'>InvalidLaterByte</span> <span class='num'>1</span><span class='layout'>)</span><span class='layout'>,</span> <span class='num'>1</span><span class='layout'>,</span> <span class='layout'>(</span><span class='varid'>b2</span><span class='conop'>:</span><span class='varid'>bs</span><span class='layout'>)</span><span class='layout'>)</span>
<a name="line-272"></a> <span class='keyglyph'>|</span> <span class='varid'>otherwise</span> <span class='keyglyph'>=</span> <span class='layout'>(</span><span class='conid'>Right</span> <span class='layout'>(</span><span class='varid'>cpToChar</span> <span class='varid'>result</span><span class='layout'>)</span><span class='layout'>,</span> <span class='num'>2</span><span class='layout'>,</span> <span class='varid'>bs</span><span class='layout'>)</span>
<a name="line-273"></a> <span class='keyword'>where</span>
<a name="line-274"></a> <span class='varid'>xs</span><span class='layout'>,</span> <span class='varid'>ys</span><span class='layout'>,</span> <span class='varid'>result</span> <span class='keyglyph'>::</span> <span class='conid'>Word32</span>
<a name="line-275"></a> <span class='varid'>xs</span> <span class='keyglyph'>=</span> <span class='varid'>fromIntegral</span> <span class='layout'>(</span><span class='varid'>b2</span><span class='varop'>.&.</span><span class='num'>0x3F</span><span class='layout'>)</span>
<a name="line-276"></a> <span class='varid'>ys</span> <span class='keyglyph'>=</span> <span class='varid'>fromIntegral</span> <span class='layout'>(</span><span class='varid'>b1</span><span class='varop'>.&.</span><span class='num'>0x1F</span><span class='layout'>)</span>
<a name="line-277"></a> <span class='varid'>result</span> <span class='keyglyph'>=</span> <span class='varid'>shiftL</span> <span class='varid'>ys</span> <span class='num'>6</span> <span class='varop'>.|.</span> <span class='varid'>xs</span>
<a name="line-278"></a><span class='definition'>decodeOne_twobyte</span><span class='conid'>[]</span> <span class='keyglyph'>=</span> <span class='varid'>error</span> <span class='str'>"UTF8.decodeOne_twobyte: No input (can't happen)"</span>
<a name="line-279"></a>
<a name="line-280"></a>
<a name="line-281"></a><span class='comment'>-- 1110zzzz 10yyyyyy 10xxxxxx -> zzzzyyyyyyxxxxxx</span>
<a name="line-282"></a>
<a name="line-283"></a><a name="decodeOne_threebyte"></a><span class='definition'>decodeOne_threebyte</span> <span class='keyglyph'>::</span> <span class='keyglyph'>[</span><span class='conid'>Word8</span><span class='keyglyph'>]</span> <span class='keyglyph'>-></span> <span class='layout'>(</span><span class='conid'>Either</span> <span class='conid'>Error</span> <span class='conid'>Char</span><span class='layout'>,</span> <span class='conid'>Int</span><span class='layout'>,</span> <span class='keyglyph'>[</span><span class='conid'>Word8</span><span class='keyglyph'>]</span><span class='layout'>)</span>
<a name="line-284"></a><span class='definition'>decodeOne_threebyte</span> <span class='layout'>(</span><span class='keyword'>_</span><span class='conop'>:</span><span class='conid'>[]</span><span class='layout'>)</span> <span class='keyglyph'>=</span> <span class='varid'>threebyte_truncated</span> <span class='num'>1</span>
<a name="line-285"></a><span class='definition'>decodeOne_threebyte</span> <span class='layout'>(</span><span class='keyword'>_</span><span class='conop'>:</span><span class='keyword'>_</span><span class='conop'>:</span><span class='conid'>[]</span><span class='layout'>)</span> <span class='keyglyph'>=</span> <span class='varid'>threebyte_truncated</span> <span class='num'>2</span>
<a name="line-286"></a><span class='definition'>decodeOne_threebyte</span> <span class='varid'>bs</span><span class='keyglyph'>@</span><span class='layout'>(</span><span class='varid'>b1</span><span class='conop'>:</span><span class='varid'>b2</span><span class='conop'>:</span><span class='varid'>b3</span><span class='conop'>:</span><span class='varid'>rest</span><span class='layout'>)</span>
<a name="line-287"></a> <span class='keyglyph'>|</span> <span class='varid'>first_bits_not_10</span> <span class='varid'>b2</span>
<a name="line-288"></a> <span class='keyglyph'>=</span> <span class='layout'>(</span><span class='conid'>Left</span> <span class='layout'>(</span><span class='conid'>InvalidLaterByte</span> <span class='num'>1</span><span class='layout'>)</span><span class='layout'>,</span> <span class='num'>1</span><span class='layout'>,</span> <span class='varid'>drop</span> <span class='num'>1</span> <span class='varid'>bs</span><span class='layout'>)</span>
<a name="line-289"></a> <span class='keyglyph'>|</span> <span class='varid'>first_bits_not_10</span> <span class='varid'>b3</span>
<a name="line-290"></a> <span class='keyglyph'>=</span> <span class='layout'>(</span><span class='conid'>Left</span> <span class='layout'>(</span><span class='conid'>InvalidLaterByte</span> <span class='num'>2</span><span class='layout'>)</span><span class='layout'>,</span> <span class='num'>2</span><span class='layout'>,</span> <span class='varid'>drop</span> <span class='num'>2</span> <span class='varid'>bs</span><span class='layout'>)</span>
<a name="line-291"></a> <span class='keyglyph'>|</span> <span class='varid'>result</span> <span class='varop'><</span> <span class='num'>0x0080</span>
<a name="line-292"></a> <span class='keyglyph'>=</span> <span class='layout'>(</span><span class='conid'>Left</span> <span class='layout'>(</span><span class='conid'>NonShortest</span> <span class='num'>3</span> <span class='num'>1</span><span class='layout'>)</span><span class='layout'>,</span> <span class='num'>3</span><span class='layout'>,</span> <span class='varid'>rest</span><span class='layout'>)</span>
<a name="line-293"></a> <span class='keyglyph'>|</span> <span class='varid'>result</span> <span class='varop'><</span> <span class='num'>0x0800</span>
<a name="line-294"></a> <span class='keyglyph'>=</span> <span class='layout'>(</span><span class='conid'>Left</span> <span class='layout'>(</span><span class='conid'>NonShortest</span> <span class='num'>3</span> <span class='num'>2</span><span class='layout'>)</span><span class='layout'>,</span> <span class='num'>3</span><span class='layout'>,</span> <span class='varid'>rest</span><span class='layout'>)</span>
<a name="line-295"></a> <span class='keyglyph'>|</span> <span class='varid'>result</span> <span class='varop'>>=</span> <span class='num'>0xD800</span> <span class='varop'>&&</span> <span class='varid'>result</span> <span class='varop'><</span> <span class='num'>0xE000</span>
<a name="line-296"></a> <span class='keyglyph'>=</span> <span class='layout'>(</span><span class='conid'>Left</span> <span class='conid'>Surrogate</span><span class='layout'>,</span> <span class='num'>3</span><span class='layout'>,</span> <span class='varid'>rest</span><span class='layout'>)</span>
<a name="line-297"></a> <span class='keyglyph'>|</span> <span class='varid'>otherwise</span>
<a name="line-298"></a> <span class='keyglyph'>=</span> <span class='layout'>(</span><span class='conid'>Right</span> <span class='layout'>(</span><span class='varid'>cpToChar</span> <span class='varid'>result</span><span class='layout'>)</span><span class='layout'>,</span> <span class='num'>3</span><span class='layout'>,</span> <span class='varid'>rest</span><span class='layout'>)</span>
<a name="line-299"></a> <span class='keyword'>where</span>
<a name="line-300"></a> <span class='varid'>xs</span><span class='layout'>,</span> <span class='varid'>ys</span><span class='layout'>,</span> <span class='varid'>zs</span><span class='layout'>,</span> <span class='varid'>result</span> <span class='keyglyph'>::</span> <span class='conid'>Word32</span>
<a name="line-301"></a> <span class='varid'>xs</span> <span class='keyglyph'>=</span> <span class='varid'>fromIntegral</span> <span class='layout'>(</span><span class='varid'>b3</span><span class='varop'>.&.</span><span class='num'>0x3F</span><span class='layout'>)</span>
<a name="line-302"></a> <span class='varid'>ys</span> <span class='keyglyph'>=</span> <span class='varid'>fromIntegral</span> <span class='layout'>(</span><span class='varid'>b2</span><span class='varop'>.&.</span><span class='num'>0x3F</span><span class='layout'>)</span>
<a name="line-303"></a> <span class='varid'>zs</span> <span class='keyglyph'>=</span> <span class='varid'>fromIntegral</span> <span class='layout'>(</span><span class='varid'>b1</span><span class='varop'>.&.</span><span class='num'>0x0F</span><span class='layout'>)</span>
<a name="line-304"></a> <span class='varid'>result</span> <span class='keyglyph'>=</span> <span class='varid'>shiftL</span> <span class='varid'>zs</span> <span class='num'>12</span> <span class='varop'>.|.</span> <span class='varid'>shiftL</span> <span class='varid'>ys</span> <span class='num'>6</span> <span class='varop'>.|.</span> <span class='varid'>xs</span>
<a name="line-305"></a><span class='definition'>decodeOne_threebyte</span><span class='conid'>[]</span>
<a name="line-306"></a> <span class='keyglyph'>=</span> <span class='varid'>error</span> <span class='str'>"UTF8.decodeOne_threebyte: No input (can't happen)"</span>
<a name="line-307"></a>
<a name="line-308"></a><a name="threebyte_truncated"></a><span class='definition'>threebyte_truncated</span> <span class='keyglyph'>::</span> <span class='conid'>Int</span> <span class='keyglyph'>-></span> <span class='layout'>(</span><span class='conid'>Either</span> <span class='conid'>Error</span> <span class='conid'>Char</span><span class='layout'>,</span> <span class='conid'>Int</span><span class='layout'>,</span> <span class='keyglyph'>[</span><span class='conid'>Word8</span><span class='keyglyph'>]</span><span class='layout'>)</span>
<a name="line-309"></a><span class='definition'>threebyte_truncated</span> <span class='varid'>n</span> <span class='keyglyph'>=</span> <span class='layout'>(</span><span class='conid'>Left</span> <span class='layout'>(</span><span class='conid'>Truncated</span> <span class='varid'>n</span> <span class='num'>3</span><span class='layout'>)</span><span class='layout'>,</span> <span class='varid'>n</span><span class='layout'>,</span> <span class='conid'>[]</span><span class='layout'>)</span>
<a name="line-310"></a>
<a name="line-311"></a>
<a name="line-312"></a><span class='comment'>-- 11110uuu 10zzzzzz 10yyyyyy 10xxxxxx -> 000uuuzzzzzzyyyyyyxxxxxx</span>
<a name="line-313"></a>
<a name="line-314"></a><a name="decodeOne_fourbyte"></a><span class='definition'>decodeOne_fourbyte</span> <span class='keyglyph'>::</span> <span class='keyglyph'>[</span><span class='conid'>Word8</span><span class='keyglyph'>]</span> <span class='keyglyph'>-></span> <span class='layout'>(</span><span class='conid'>Either</span> <span class='conid'>Error</span> <span class='conid'>Char</span><span class='layout'>,</span> <span class='conid'>Int</span><span class='layout'>,</span> <span class='keyglyph'>[</span><span class='conid'>Word8</span><span class='keyglyph'>]</span><span class='layout'>)</span>
<a name="line-315"></a><span class='definition'>decodeOne_fourbyte</span> <span class='layout'>(</span><span class='keyword'>_</span><span class='conop'>:</span><span class='conid'>[]</span><span class='layout'>)</span> <span class='keyglyph'>=</span> <span class='varid'>fourbyte_truncated</span> <span class='num'>1</span>
<a name="line-316"></a><span class='definition'>decodeOne_fourbyte</span> <span class='layout'>(</span><span class='keyword'>_</span><span class='conop'>:</span><span class='keyword'>_</span><span class='conop'>:</span><span class='conid'>[]</span><span class='layout'>)</span> <span class='keyglyph'>=</span> <span class='varid'>fourbyte_truncated</span> <span class='num'>2</span>
<a name="line-317"></a><span class='definition'>decodeOne_fourbyte</span> <span class='layout'>(</span><span class='keyword'>_</span><span class='conop'>:</span><span class='keyword'>_</span><span class='conop'>:</span><span class='keyword'>_</span><span class='conop'>:</span><span class='conid'>[]</span><span class='layout'>)</span> <span class='keyglyph'>=</span> <span class='varid'>fourbyte_truncated</span> <span class='num'>3</span>
<a name="line-318"></a><span class='definition'>decodeOne_fourbyte</span> <span class='varid'>bs</span><span class='keyglyph'>@</span><span class='layout'>(</span><span class='varid'>b1</span><span class='conop'>:</span><span class='varid'>b2</span><span class='conop'>:</span><span class='varid'>b3</span><span class='conop'>:</span><span class='varid'>b4</span><span class='conop'>:</span><span class='varid'>rest</span><span class='layout'>)</span>
<a name="line-319"></a> <span class='keyglyph'>|</span> <span class='varid'>first_bits_not_10</span> <span class='varid'>b2</span>
<a name="line-320"></a> <span class='keyglyph'>=</span> <span class='layout'>(</span><span class='conid'>Left</span> <span class='layout'>(</span><span class='conid'>InvalidLaterByte</span> <span class='num'>1</span><span class='layout'>)</span><span class='layout'>,</span> <span class='num'>1</span><span class='layout'>,</span> <span class='varid'>drop</span> <span class='num'>1</span> <span class='varid'>bs</span><span class='layout'>)</span>
<a name="line-321"></a> <span class='keyglyph'>|</span> <span class='varid'>first_bits_not_10</span> <span class='varid'>b3</span>
<a name="line-322"></a> <span class='keyglyph'>=</span> <span class='layout'>(</span><span class='conid'>Left</span> <span class='layout'>(</span><span class='conid'>InvalidLaterByte</span> <span class='num'>2</span><span class='layout'>)</span><span class='layout'>,</span> <span class='num'>2</span><span class='layout'>,</span> <span class='varid'>drop</span> <span class='num'>2</span> <span class='varid'>bs</span><span class='layout'>)</span>
<a name="line-323"></a> <span class='keyglyph'>|</span> <span class='varid'>first_bits_not_10</span> <span class='varid'>b4</span>
<a name="line-324"></a> <span class='keyglyph'>=</span> <span class='layout'>(</span><span class='conid'>Left</span> <span class='layout'>(</span><span class='conid'>InvalidLaterByte</span> <span class='num'>3</span><span class='layout'>)</span><span class='layout'>,</span> <span class='num'>3</span><span class='layout'>,</span> <span class='varid'>drop</span> <span class='num'>3</span> <span class='varid'>bs</span><span class='layout'>)</span>
<a name="line-325"></a> <span class='keyglyph'>|</span> <span class='varid'>result</span> <span class='varop'><</span> <span class='num'>0x0080</span>
<a name="line-326"></a> <span class='keyglyph'>=</span> <span class='layout'>(</span><span class='conid'>Left</span> <span class='layout'>(</span><span class='conid'>NonShortest</span> <span class='num'>4</span> <span class='num'>1</span><span class='layout'>)</span><span class='layout'>,</span> <span class='num'>4</span><span class='layout'>,</span> <span class='varid'>rest</span><span class='layout'>)</span>
<a name="line-327"></a> <span class='keyglyph'>|</span> <span class='varid'>result</span> <span class='varop'><</span> <span class='num'>0x0800</span>
<a name="line-328"></a> <span class='keyglyph'>=</span> <span class='layout'>(</span><span class='conid'>Left</span> <span class='layout'>(</span><span class='conid'>NonShortest</span> <span class='num'>4</span> <span class='num'>2</span><span class='layout'>)</span><span class='layout'>,</span> <span class='num'>4</span><span class='layout'>,</span> <span class='varid'>rest</span><span class='layout'>)</span>
<a name="line-329"></a> <span class='keyglyph'>|</span> <span class='varid'>result</span> <span class='varop'><</span> <span class='num'>0x10000</span>
<a name="line-330"></a> <span class='keyglyph'>=</span> <span class='layout'>(</span><span class='conid'>Left</span> <span class='layout'>(</span><span class='conid'>NonShortest</span> <span class='num'>4</span> <span class='num'>3</span><span class='layout'>)</span><span class='layout'>,</span> <span class='num'>4</span><span class='layout'>,</span> <span class='varid'>rest</span><span class='layout'>)</span>
<a name="line-331"></a> <span class='keyglyph'>|</span> <span class='varid'>result</span> <span class='varop'>></span> <span class='num'>0x10FFFF</span>
<a name="line-332"></a> <span class='keyglyph'>=</span> <span class='layout'>(</span><span class='conid'>Left</span> <span class='conid'>ValueOutOfBounds</span><span class='layout'>,</span> <span class='num'>4</span><span class='layout'>,</span> <span class='varid'>rest</span><span class='layout'>)</span>
<a name="line-333"></a> <span class='keyglyph'>|</span> <span class='varid'>otherwise</span>
<a name="line-334"></a> <span class='keyglyph'>=</span> <span class='layout'>(</span><span class='conid'>Right</span> <span class='layout'>(</span><span class='varid'>cpToChar</span> <span class='varid'>result</span><span class='layout'>)</span><span class='layout'>,</span> <span class='num'>4</span><span class='layout'>,</span> <span class='varid'>rest</span><span class='layout'>)</span>
<a name="line-335"></a> <span class='keyword'>where</span>
<a name="line-336"></a> <span class='varid'>xs</span><span class='layout'>,</span> <span class='varid'>ys</span><span class='layout'>,</span> <span class='varid'>zs</span><span class='layout'>,</span> <span class='varid'>us</span><span class='layout'>,</span> <span class='varid'>result</span> <span class='keyglyph'>::</span> <span class='conid'>Word32</span>
<a name="line-337"></a> <span class='varid'>xs</span> <span class='keyglyph'>=</span> <span class='varid'>fromIntegral</span> <span class='layout'>(</span><span class='varid'>b4</span> <span class='varop'>.&.</span> <span class='num'>0x3F</span><span class='layout'>)</span>
<a name="line-338"></a> <span class='varid'>ys</span> <span class='keyglyph'>=</span> <span class='varid'>fromIntegral</span> <span class='layout'>(</span><span class='varid'>b3</span> <span class='varop'>.&.</span> <span class='num'>0x3F</span><span class='layout'>)</span>
<a name="line-339"></a> <span class='varid'>zs</span> <span class='keyglyph'>=</span> <span class='varid'>fromIntegral</span> <span class='layout'>(</span><span class='varid'>b2</span> <span class='varop'>.&.</span> <span class='num'>0x3F</span><span class='layout'>)</span>
<a name="line-340"></a> <span class='varid'>us</span> <span class='keyglyph'>=</span> <span class='varid'>fromIntegral</span> <span class='layout'>(</span><span class='varid'>b1</span> <span class='varop'>.&.</span> <span class='num'>0x07</span><span class='layout'>)</span>
<a name="line-341"></a> <span class='varid'>result</span> <span class='keyglyph'>=</span> <span class='varid'>xs</span> <span class='varop'>.|.</span> <span class='varid'>shiftL</span> <span class='varid'>ys</span> <span class='num'>6</span> <span class='varop'>.|.</span> <span class='varid'>shiftL</span> <span class='varid'>zs</span> <span class='num'>12</span> <span class='varop'>.|.</span> <span class='varid'>shiftL</span> <span class='varid'>us</span> <span class='num'>18</span>
<a name="line-342"></a><span class='definition'>decodeOne_fourbyte</span><span class='conid'>[]</span>
<a name="line-343"></a> <span class='keyglyph'>=</span> <span class='varid'>error</span> <span class='str'>"UTF8.decodeOne_fourbyte: No input (can't happen)"</span>
<a name="line-344"></a>
<a name="line-345"></a><a name="fourbyte_truncated"></a><span class='definition'>fourbyte_truncated</span> <span class='keyglyph'>::</span> <span class='conid'>Int</span> <span class='keyglyph'>-></span> <span class='layout'>(</span><span class='conid'>Either</span> <span class='conid'>Error</span> <span class='conid'>Char</span><span class='layout'>,</span> <span class='conid'>Int</span><span class='layout'>,</span> <span class='keyglyph'>[</span><span class='conid'>Word8</span><span class='keyglyph'>]</span><span class='layout'>)</span>
<a name="line-346"></a><span class='definition'>fourbyte_truncated</span> <span class='varid'>n</span> <span class='keyglyph'>=</span> <span class='layout'>(</span><span class='conid'>Left</span> <span class='layout'>(</span><span class='conid'>Truncated</span> <span class='varid'>n</span> <span class='num'>4</span><span class='layout'>)</span><span class='layout'>,</span> <span class='varid'>n</span><span class='layout'>,</span> <span class='conid'>[]</span><span class='layout'>)</span>
<a name="line-347"></a>
<a name="line-348"></a>
<a name="line-349"></a><span class='comment'>-- The decoder examines all input, recording decoded characters as well as</span>
<a name="line-350"></a><span class='comment'>-- error-index pairs along the way.</span>
<a name="line-351"></a>
<a name="line-352"></a><a name="decode"></a><span class='definition'>decode</span> <span class='keyglyph'>::</span> <span class='keyglyph'>[</span><span class='conid'>Word8</span><span class='keyglyph'>]</span> <span class='keyglyph'>-></span> <span class='layout'>(</span><span class='keyglyph'>[</span><span class='conid'>Char</span><span class='keyglyph'>]</span><span class='layout'>,</span> <span class='keyglyph'>[</span><span class='layout'>(</span><span class='conid'>Error</span><span class='layout'>,</span><span class='conid'>Int</span><span class='layout'>)</span><span class='keyglyph'>]</span><span class='layout'>)</span>
<a name="line-353"></a><span class='definition'>decode</span> <span class='keyglyph'>=</span> <span class='varid'>swap</span> <span class='varop'>.</span> <span class='varid'>partitionEither</span> <span class='varop'>.</span> <span class='varid'>decodeEmbedErrors</span>
<a name="line-354"></a>
<a name="line-355"></a><a name="decodeEmbedErrors"></a><span class='definition'>decodeEmbedErrors</span> <span class='keyglyph'>::</span> <span class='keyglyph'>[</span><span class='conid'>Word8</span><span class='keyglyph'>]</span> <span class='keyglyph'>-></span> <span class='keyglyph'>[</span><span class='conid'>Either</span> <span class='layout'>(</span><span class='conid'>Error</span><span class='layout'>,</span><span class='conid'>Int</span><span class='layout'>)</span> <span class='conid'>Char</span><span class='keyglyph'>]</span>
<a name="line-356"></a><span class='definition'>decodeEmbedErrors</span> <span class='keyglyph'>=</span>
<a name="line-357"></a> <span class='varid'>unfoldr</span> <span class='layout'>(</span><span class='keyglyph'>\</span><span class='layout'>(</span><span class='varid'>pos</span><span class='layout'>,</span><span class='varid'>xs</span><span class='layout'>)</span> <span class='keyglyph'>-></span>
<a name="line-358"></a> <span class='varid'>toMaybe</span>
<a name="line-359"></a> <span class='layout'>(</span><span class='varid'>not</span> <span class='varop'>$</span> <span class='varid'>null</span> <span class='varid'>xs</span><span class='layout'>)</span>
<a name="line-360"></a> <span class='layout'>(</span><span class='keyword'>let</span> <span class='layout'>(</span><span class='varid'>c</span><span class='layout'>,</span><span class='varid'>n</span><span class='layout'>,</span><span class='varid'>rest</span><span class='layout'>)</span> <span class='keyglyph'>=</span> <span class='varid'>decodeOne</span> <span class='varid'>xs</span>
<a name="line-361"></a> <span class='keyword'>in</span> <span class='layout'>(</span><span class='varid'>either</span> <span class='layout'>(</span><span class='keyglyph'>\</span><span class='varid'>err</span> <span class='keyglyph'>-></span> <span class='conid'>Left</span> <span class='layout'>(</span><span class='varid'>err</span><span class='layout'>,</span><span class='varid'>pos</span><span class='layout'>)</span><span class='layout'>)</span> <span class='conid'>Right</span> <span class='varid'>c</span><span class='layout'>,</span>
<a name="line-362"></a> <span class='layout'>(</span><span class='varid'>pos</span><span class='varop'>+</span><span class='varid'>n</span><span class='layout'>,</span><span class='varid'>rest</span><span class='layout'>)</span><span class='layout'>)</span><span class='layout'>)</span><span class='layout'>)</span> <span class='varop'>.</span>
<a name="line-363"></a> <span class='conid'>(,)</span> <span class='num'>0</span>
</pre></body>
</html>