packages feed

mangrove (empty) → 0.1.0.0

raw patch · 66 files changed

+13076/−0 lines, 66 filesdep +HUnitdep +aesondep +base

Dependencies added: HUnit, aeson, base, bytestring, containers, filepath, mangrove, text, transformers, unordered-containers, utility-ht, vector, willow

Files

+ CHANGELOG.md view
@@ -0,0 +1,20 @@+# Revision history++Version numbering for 'mangrove' follows the [standard Haskell versioning +policy](https://pvp.haskell.org), where the first *two* numbers together +indicate the major version of the package, the third the minor, and the fourth +the patch; a fifth number is optional and indicates a revision purely to the +documentation and/or packaging.+++## v/html/0.1.0.0. -- 2021-04-16++### Added++* f/1.1.  Parse HTML(5) documents into a very basic tree (see however the known +  issues), or the intermediate character/token streams++### Known issues++* i/1.  Handling for some cases of mis-nested tags has not yet been +  implemented, throwing an error to terminate parsing
+ LICENSE view
@@ -0,0 +1,373 @@+Mozilla Public License Version 2.0+==================================++1. Definitions+--------------++1.1. "Contributor"+    means each individual or legal entity that creates, contributes to+    the creation of, or owns Covered Software.++1.2. "Contributor Version"+    means the combination of the Contributions of others (if any) used+    by a Contributor and that particular Contributor's Contribution.++1.3. "Contribution"+    means Covered Software of a particular Contributor.++1.4. "Covered Software"+    means Source Code Form to which the initial Contributor has attached+    the notice in Exhibit A, the Executable Form of such Source Code+    Form, and Modifications of such Source Code Form, in each case+    including portions thereof.++1.5. "Incompatible With Secondary Licenses"+    means++    (a) that the initial Contributor has attached the notice described+        in Exhibit B to the Covered Software; or++    (b) that the Covered Software was made available under the terms of+        version 1.1 or earlier of the License, but not also under the+        terms of a Secondary License.++1.6. "Executable Form"+    means any form of the work other than Source Code Form.++1.7. "Larger Work"+    means a work that combines Covered Software with other material, in+    a separate file or files, that is not Covered Software.++1.8. "License"+    means this document.++1.9. "Licensable"+    means having the right to grant, to the maximum extent possible,+    whether at the time of the initial grant or subsequently, any and+    all of the rights conveyed by this License.++1.10. "Modifications"+    means any of the following:++    (a) any file in Source Code Form that results from an addition to,+        deletion from, or modification of the contents of Covered+        Software; or++    (b) any new file in Source Code Form that contains any Covered+        Software.++1.11. "Patent Claims" of a Contributor+    means any patent claim(s), including without limitation, method,+    process, and apparatus claims, in any patent Licensable by such+    Contributor that would be infringed, but for the grant of the+    License, by the making, using, selling, offering for sale, having+    made, import, or transfer of either its Contributions or its+    Contributor Version.++1.12. "Secondary License"+    means either the GNU General Public License, Version 2.0, the GNU+    Lesser General Public License, Version 2.1, the GNU Affero General+    Public License, Version 3.0, or any later versions of those+    licenses.++1.13. "Source Code Form"+    means the form of the work preferred for making modifications.++1.14. "You" (or "Your")+    means an individual or a legal entity exercising rights under this+    License. For legal entities, "You" includes any entity that+    controls, is controlled by, or is under common control with You. For+    purposes of this definition, "control" means (a) the power, direct+    or indirect, to cause the direction or management of such entity,+    whether by contract or otherwise, or (b) ownership of more than+    fifty percent (50%) of the outstanding shares or beneficial+    ownership of such entity.++2. License Grants and Conditions+--------------------------------++2.1. Grants++Each Contributor hereby grants You a world-wide, royalty-free,+non-exclusive license:++(a) under intellectual property rights (other than patent or trademark)+    Licensable by such Contributor to use, reproduce, make available,+    modify, display, perform, distribute, and otherwise exploit its+    Contributions, either on an unmodified basis, with Modifications, or+    as part of a Larger Work; and++(b) under Patent Claims of such Contributor to make, use, sell, offer+    for sale, have made, import, and otherwise transfer either its+    Contributions or its Contributor Version.++2.2. Effective Date++The licenses granted in Section 2.1 with respect to any Contribution+become effective for each Contribution on the date the Contributor first+distributes such Contribution.++2.3. Limitations on Grant Scope++The licenses granted in this Section 2 are the only rights granted under+this License. No additional rights or licenses will be implied from the+distribution or licensing of Covered Software under this License.+Notwithstanding Section 2.1(b) above, no patent license is granted by a+Contributor:++(a) for any code that a Contributor has removed from Covered Software;+    or++(b) for infringements caused by: (i) Your and any other third party's+    modifications of Covered Software, or (ii) the combination of its+    Contributions with other software (except as part of its Contributor+    Version); or++(c) under Patent Claims infringed by Covered Software in the absence of+    its Contributions.++This License does not grant any rights in the trademarks, service marks,+or logos of any Contributor (except as may be necessary to comply with+the notice requirements in Section 3.4).++2.4. Subsequent Licenses++No Contributor makes additional grants as a result of Your choice to+distribute the Covered Software under a subsequent version of this+License (see Section 10.2) or under the terms of a Secondary License (if+permitted under the terms of Section 3.3).++2.5. Representation++Each Contributor represents that the Contributor believes its+Contributions are its original creation(s) or it has sufficient rights+to grant the rights to its Contributions conveyed by this License.++2.6. Fair Use++This License is not intended to limit any rights You have under+applicable copyright doctrines of fair use, fair dealing, or other+equivalents.++2.7. Conditions++Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted+in Section 2.1.++3. Responsibilities+-------------------++3.1. Distribution of Source Form++All distribution of Covered Software in Source Code Form, including any+Modifications that You create or to which You contribute, must be under+the terms of this License. You must inform recipients that the Source+Code Form of the Covered Software is governed by the terms of this+License, and how they can obtain a copy of this License. You may not+attempt to alter or restrict the recipients' rights in the Source Code+Form.++3.2. Distribution of Executable Form++If You distribute Covered Software in Executable Form then:++(a) such Covered Software must also be made available in Source Code+    Form, as described in Section 3.1, and You must inform recipients of+    the Executable Form how they can obtain a copy of such Source Code+    Form by reasonable means in a timely manner, at a charge no more+    than the cost of distribution to the recipient; and++(b) You may distribute such Executable Form under the terms of this+    License, or sublicense it under different terms, provided that the+    license for the Executable Form does not attempt to limit or alter+    the recipients' rights in the Source Code Form under this License.++3.3. Distribution of a Larger Work++You may create and distribute a Larger Work under terms of Your choice,+provided that You also comply with the requirements of this License for+the Covered Software. If the Larger Work is a combination of Covered+Software with a work governed by one or more Secondary Licenses, and the+Covered Software is not Incompatible With Secondary Licenses, this+License permits You to additionally distribute such Covered Software+under the terms of such Secondary License(s), so that the recipient of+the Larger Work may, at their option, further distribute the Covered+Software under the terms of either this License or such Secondary+License(s).++3.4. Notices++You may not remove or alter the substance of any license notices+(including copyright notices, patent notices, disclaimers of warranty,+or limitations of liability) contained within the Source Code Form of+the Covered Software, except that You may alter any license notices to+the extent required to remedy known factual inaccuracies.++3.5. Application of Additional Terms++You may choose to offer, and to charge a fee for, warranty, support,+indemnity or liability obligations to one or more recipients of Covered+Software. However, You may do so only on Your own behalf, and not on+behalf of any Contributor. You must make it absolutely clear that any+such warranty, support, indemnity, or liability obligation is offered by+You alone, and You hereby agree to indemnify every Contributor for any+liability incurred by such Contributor as a result of warranty, support,+indemnity or liability terms You offer. You may include additional+disclaimers of warranty and limitations of liability specific to any+jurisdiction.++4. Inability to Comply Due to Statute or Regulation+---------------------------------------------------++If it is impossible for You to comply with any of the terms of this+License with respect to some or all of the Covered Software due to+statute, judicial order, or regulation then You must: (a) comply with+the terms of this License to the maximum extent possible; and (b)+describe the limitations and the code they affect. Such description must+be placed in a text file included with all distributions of the Covered+Software under this License. Except to the extent prohibited by statute+or regulation, such description must be sufficiently detailed for a+recipient of ordinary skill to be able to understand it.++5. Termination+--------------++5.1. The rights granted under this License will terminate automatically+if You fail to comply with any of its terms. However, if You become+compliant, then the rights granted under this License from a particular+Contributor are reinstated (a) provisionally, unless and until such+Contributor explicitly and finally terminates Your grants, and (b) on an+ongoing basis, if such Contributor fails to notify You of the+non-compliance by some reasonable means prior to 60 days after You have+come back into compliance. Moreover, Your grants from a particular+Contributor are reinstated on an ongoing basis if such Contributor+notifies You of the non-compliance by some reasonable means, this is the+first time You have received notice of non-compliance with this License+from such Contributor, and You become compliant prior to 30 days after+Your receipt of the notice.++5.2. If You initiate litigation against any entity by asserting a patent+infringement claim (excluding declaratory judgment actions,+counter-claims, and cross-claims) alleging that a Contributor Version+directly or indirectly infringes any patent, then the rights granted to+You by any and all Contributors for the Covered Software under Section+2.1 of this License shall terminate.++5.3. In the event of termination under Sections 5.1 or 5.2 above, all+end user license agreements (excluding distributors and resellers) which+have been validly granted by You or Your distributors under this License+prior to termination shall survive termination.++************************************************************************+*                                                                      *+*  6. Disclaimer of Warranty                                           *+*  -------------------------                                           *+*                                                                      *+*  Covered Software is provided under this License on an "as is"       *+*  basis, without warranty of any kind, either expressed, implied, or  *+*  statutory, including, without limitation, warranties that the       *+*  Covered Software is free of defects, merchantable, fit for a        *+*  particular purpose or non-infringing. The entire risk as to the     *+*  quality and performance of the Covered Software is with You.        *+*  Should any Covered Software prove defective in any respect, You     *+*  (not any Contributor) assume the cost of any necessary servicing,   *+*  repair, or correction. This disclaimer of warranty constitutes an   *+*  essential part of this License. No use of any Covered Software is   *+*  authorized under this License except under this disclaimer.         *+*                                                                      *+************************************************************************++************************************************************************+*                                                                      *+*  7. Limitation of Liability                                          *+*  --------------------------                                          *+*                                                                      *+*  Under no circumstances and under no legal theory, whether tort      *+*  (including negligence), contract, or otherwise, shall any           *+*  Contributor, or anyone who distributes Covered Software as          *+*  permitted above, be liable to You for any direct, indirect,         *+*  special, incidental, or consequential damages of any character      *+*  including, without limitation, damages for lost profits, loss of    *+*  goodwill, work stoppage, computer failure or malfunction, or any    *+*  and all other commercial damages or losses, even if such party      *+*  shall have been informed of the possibility of such damages. This   *+*  limitation of liability shall not apply to liability for death or   *+*  personal injury resulting from such party's negligence to the       *+*  extent applicable law prohibits such limitation. Some               *+*  jurisdictions do not allow the exclusion or limitation of           *+*  incidental or consequential damages, so this exclusion and          *+*  limitation may not apply to You.                                    *+*                                                                      *+************************************************************************++8. Litigation+-------------++Any litigation relating to this License may be brought only in the+courts of a jurisdiction where the defendant maintains its principal+place of business and such litigation shall be governed by laws of that+jurisdiction, without reference to its conflict-of-law provisions.+Nothing in this Section shall prevent a party's ability to bring+cross-claims or counter-claims.++9. Miscellaneous+----------------++This License represents the complete agreement concerning the subject+matter hereof. If any provision of this License is held to be+unenforceable, such provision shall be reformed only to the extent+necessary to make it enforceable. Any law or regulation which provides+that the language of a contract shall be construed against the drafter+shall not be used to construe this License against a Contributor.++10. Versions of the License+---------------------------++10.1. New Versions++Mozilla Foundation is the license steward. Except as provided in Section+10.3, no one other than the license steward has the right to modify or+publish new versions of this License. Each version will be given a+distinguishing version number.++10.2. Effect of New Versions++You may distribute the Covered Software under the terms of the version+of the License under which You originally received the Covered Software,+or under the terms of any subsequent version published by the license+steward.++10.3. Modified Versions++If you create software not governed by this License, and you want to+create a new license for such software, you may create and use a+modified version of this License if you rename the license and remove+any references to the name of the license steward (except to note that+such modified license differs from this License).++10.4. Distributing Source Code Form that is Incompatible With Secondary+Licenses++If You choose to distribute Source Code Form that is Incompatible With+Secondary Licenses under the terms of this version of the License, the+notice described in Exhibit B of this License must be attached.++Exhibit A - Source Code Form License Notice+-------------------------------------------++  This Source Code Form is subject to the terms of the Mozilla Public+  License, v. 2.0. If a copy of the MPL was not distributed with this+  file, You can obtain one at http://mozilla.org/MPL/2.0/.++If it is not possible or desirable to put the notice in a particular+file, then You may include the notice in a location (such as a LICENSE+file in a relevant directory) where a recipient would be likely to look+for such a notice.++You may add additional accurate notices of copyright ownership.++Exhibit B - "Incompatible With Secondary Licenses" Notice+---------------------------------------------------------++  This Source Code Form is "Incompatible With Secondary Licenses", as+  defined by the Mozilla Public License, v. 2.0.
+ README.md view
@@ -0,0 +1,21 @@+# About++'mangrove' provides an HTML5-compatible parser for web documents, implemented +in Haskell.  In keeping with the immutable data paradigms, an emphasis has been +placed on avoiding side effects and mutable structures rather than strictly +following the official algorithms.  The resulting document tree can be returned +to `willow` to be styled and rendered.++This readme is rather sparse, as it has been written for a subfolder of the +complete repository; for full info on the project, see the primary readme in +either [this directory](README.project.md), [its parent](../README.md), or the +[online host](https://code.eitilt.life/src/willow/README.md), whichever of +those links may work.+++# Coverage reporting++Unfortunately, the invocation of hpc by cabal-install <= 3.4.0.0 doesn't work +properly when multiple packages are developed as part of the same project.  +Until the next version is released, I recommend that you don't enable coverage +reports for `mangrove`, in order for the tests themselves to run correctly.
+ README.project.md view
@@ -0,0 +1,127 @@+# About++I enjoy working in the terminal, but none of the existing pure-text web +browsers quite provide the full-featured experience I'm looking for; quite good +for retro computing, not as comfortable as current standards of development.  +On the graphical side, there are a number of extensions which tack keyboard +controls onto existing browsers, but the resulting join is necessarily a bit +ugly (there *are* a few browsers which have been specifically designed around +the keyboard -- [qutebrowser](https://qutebrowser.org) stands out among them -- +but they all still have their own pain points in turn).  So, I'm making my own. +There's no way this will ever be able to compete with the big names[1], but I +hope to gradually grow, one stable step at a time, and eventually reach +something I find comfortable to use.++Instructions for writing other packages using these libraries are generated as +Hackage pages; those can be found in the standard locations (Hackage, the +`hoogle` CLI, `~/.cabal/store/*/share/doc/`, etc.), and are mirrored to the +project web page[2].+++# Cloning++In order to ensure the modules provide all relevant information, this file has +been added to the `extra-source-files` list in addition to the README specific +to that module, but to avoid duplicate patches, the copy/link has not been +added to Darcs.  If you have obtained this via `cabal get` or a single-module +tarball, everything should be good.  If instead you have cloned this through +Darcs or otherwise downloaded the entire repository, you won't have the +duplicate in the place Cabal expects it to be.  That shouldn't be an issue +unless you try packaging it up for some reason, but if you do run into trouble:++	ln README.md mangrove/README.project.md+	ln README.md willow/README.project.md+++# Contribution++Warnings are only useful to developers; while an end user seeing a warning +*might* feel inclined to fix the program, chances are it's just going to be +ignored as the list of built files scrolls by.  Reflecting that, a build flag +`dev` has been added to all packages to toggle the printing of warnings, and +potentially other developer aids in the future.  If you decide to help improve +this code, please enable it:++	cabal configure -fdev++Depending on the size of your hack, I welcome either a diff file, or you can +bundle your complete darcs patches with `darcs send`.  Either way, attach the +changes to an email addressed to ag@eitilt.life and I'll see about adding it to +the codebase.++## Patch format++If you do send the latter, every patch should have a comment with a (at least +mildly) descriptive name prefixed with a tag indicating the general category +addressed by the patch: for fixing issues, that is an "i/" followed by the +issue number (without leading zeros, *unlike* the reference file); for general +development, "f/" followed by the best topic in the `FEATURES` key; for +completeness, version tags are "v/" followed by the package reference followed +by the version.  All are terminated with a final period.++Please do not squash patches.  Each patch should represent a minimal but +complete change: certainly enough context to successfully compile, hopefully +enough context to not break any previously-passing tests, and potentially +combining closely-related changes at your discretion, but if you wind up trying +to decide between multiple tags or you're adding all of your work over the +entire day, you should probably look closely at whether you can break the patch +apart any farther.++Additionally, make good use of the `--ask-deps` flag.  Until I get a CI +integration together, it's easy to miss a dependency, but do your best to +select anything which your patch may require to build successfully.+++## External test suites++Several test suites have already been written against the HTML specification +(including some spillover into other web technology), and I have taken +advantage that previous work to ensure the correctness of the modules here.  +Rather than duplicate the storage and have to worry about keeping my mirror +up-to-date, I've instead just set up the tests to look for the repositories in +specific locations, and rely on the programmer to clone them manually.++If possible, these should be saved in the primary data folders; however, as the +test data is shared between multiple packages, a naïve setup will take up a lot +of unnecessary space.  If you're only building a single package, then you're +already good and can just use the `data/` folder in this same directory.  If +you do have the full repo, Haskell can easily handle symlinks, so I recommend +cloning the data to a single location and then linking to it from the other +package(s); the top-level directory has been used in the examples, but you can +also just choose, e.g., `willow/data/` and link the other packages to that.++### html5lib++The supplemental html5lib conformance suite may be enabled via the Cabal flag +`html5lib`.  Before doing so, however, the test data must be downloaded to +`data/test/html5lib-tests`:++	git clone 'https://github.com/html5lib/html5lib-tests' data/test/html5lib-tests++And, unless you only downloaded a single module:++	ln -s data/test/html5lib-tests mangrove/data/test+	ln -s data/test/html5lib-tests willow/data/test++Followed either way by:++	cabal configure --enable-tests -fhtml5lib+	cabal v2-test mangrove:html5lib willow:html5lib++### wpt++Likewise, most of the web-platform-tests conformance suite (Cabal flag `wpt`) +requires the directory `data/test/wpt` to be populated:++	git clone 'https://github.com/web-platform-tests/wpt' data/test/wpt++	ln -s data/test/wpt willow/data/test++	cabal configure --enable-tests -fwpt+	cabal v2-test willow:wpt+++# Footnotes++[1]: https://drewdevault.com/2020/03/18/Reckless-limitless-scope.html+[2]: https://ag.eitilt.life/willow/
+ data/entities.json view
@@ -0,0 +1,2233 @@+{+  "&AElig": { "codepoints": [198], "characters": "\u00C6" },+  "&AElig;": { "codepoints": [198], "characters": "\u00C6" },+  "&AMP": { "codepoints": [38], "characters": "\u0026" },+  "&AMP;": { "codepoints": [38], "characters": "\u0026" },+  "&Aacute": { "codepoints": [193], "characters": "\u00C1" },+  "&Aacute;": { "codepoints": [193], "characters": "\u00C1" },+  "&Abreve;": { "codepoints": [258], "characters": "\u0102" },+  "&Acirc": { "codepoints": [194], "characters": "\u00C2" },+  "&Acirc;": { "codepoints": [194], "characters": "\u00C2" },+  "&Acy;": { "codepoints": [1040], "characters": "\u0410" },+  "&Afr;": { "codepoints": [120068], "characters": "\uD835\uDD04" },+  "&Agrave": { "codepoints": [192], "characters": "\u00C0" },+  "&Agrave;": { "codepoints": [192], "characters": "\u00C0" },+  "&Alpha;": { "codepoints": [913], "characters": "\u0391" },+  "&Amacr;": { "codepoints": [256], "characters": "\u0100" },+  "&And;": { "codepoints": [10835], "characters": "\u2A53" },+  "&Aogon;": { "codepoints": [260], "characters": "\u0104" },+  "&Aopf;": { "codepoints": [120120], "characters": "\uD835\uDD38" },+  "&ApplyFunction;": { "codepoints": [8289], "characters": "\u2061" },+  "&Aring": { "codepoints": [197], "characters": "\u00C5" },+  "&Aring;": { "codepoints": [197], "characters": "\u00C5" },+  "&Ascr;": { "codepoints": [119964], "characters": "\uD835\uDC9C" },+  "&Assign;": { "codepoints": [8788], "characters": "\u2254" },+  "&Atilde": { "codepoints": [195], "characters": "\u00C3" },+  "&Atilde;": { "codepoints": [195], "characters": "\u00C3" },+  "&Auml": { "codepoints": [196], "characters": "\u00C4" },+  "&Auml;": { "codepoints": [196], "characters": "\u00C4" },+  "&Backslash;": { "codepoints": [8726], "characters": "\u2216" },+  "&Barv;": { "codepoints": [10983], "characters": "\u2AE7" },+  "&Barwed;": { "codepoints": [8966], "characters": "\u2306" },+  "&Bcy;": { "codepoints": [1041], "characters": "\u0411" },+  "&Because;": { "codepoints": [8757], "characters": "\u2235" },+  "&Bernoullis;": { "codepoints": [8492], "characters": "\u212C" },+  "&Beta;": { "codepoints": [914], "characters": "\u0392" },+  "&Bfr;": { "codepoints": [120069], "characters": "\uD835\uDD05" },+  "&Bopf;": { "codepoints": [120121], "characters": "\uD835\uDD39" },+  "&Breve;": { "codepoints": [728], "characters": "\u02D8" },+  "&Bscr;": { "codepoints": [8492], "characters": "\u212C" },+  "&Bumpeq;": { "codepoints": [8782], "characters": "\u224E" },+  "&CHcy;": { "codepoints": [1063], "characters": "\u0427" },+  "&COPY": { "codepoints": [169], "characters": "\u00A9" },+  "&COPY;": { "codepoints": [169], "characters": "\u00A9" },+  "&Cacute;": { "codepoints": [262], "characters": "\u0106" },+  "&Cap;": { "codepoints": [8914], "characters": "\u22D2" },+  "&CapitalDifferentialD;": { "codepoints": [8517], "characters": "\u2145" },+  "&Cayleys;": { "codepoints": [8493], "characters": "\u212D" },+  "&Ccaron;": { "codepoints": [268], "characters": "\u010C" },+  "&Ccedil": { "codepoints": [199], "characters": "\u00C7" },+  "&Ccedil;": { "codepoints": [199], "characters": "\u00C7" },+  "&Ccirc;": { "codepoints": [264], "characters": "\u0108" },+  "&Cconint;": { "codepoints": [8752], "characters": "\u2230" },+  "&Cdot;": { "codepoints": [266], "characters": "\u010A" },+  "&Cedilla;": { "codepoints": [184], "characters": "\u00B8" },+  "&CenterDot;": { "codepoints": [183], "characters": "\u00B7" },+  "&Cfr;": { "codepoints": [8493], "characters": "\u212D" },+  "&Chi;": { "codepoints": [935], "characters": "\u03A7" },+  "&CircleDot;": { "codepoints": [8857], "characters": "\u2299" },+  "&CircleMinus;": { "codepoints": [8854], "characters": "\u2296" },+  "&CirclePlus;": { "codepoints": [8853], "characters": "\u2295" },+  "&CircleTimes;": { "codepoints": [8855], "characters": "\u2297" },+  "&ClockwiseContourIntegral;": { "codepoints": [8754], "characters": "\u2232" },+  "&CloseCurlyDoubleQuote;": { "codepoints": [8221], "characters": "\u201D" },+  "&CloseCurlyQuote;": { "codepoints": [8217], "characters": "\u2019" },+  "&Colon;": { "codepoints": [8759], "characters": "\u2237" },+  "&Colone;": { "codepoints": [10868], "characters": "\u2A74" },+  "&Congruent;": { "codepoints": [8801], "characters": "\u2261" },+  "&Conint;": { "codepoints": [8751], "characters": "\u222F" },+  "&ContourIntegral;": { "codepoints": [8750], "characters": "\u222E" },+  "&Copf;": { "codepoints": [8450], "characters": "\u2102" },+  "&Coproduct;": { "codepoints": [8720], "characters": "\u2210" },+  "&CounterClockwiseContourIntegral;": { "codepoints": [8755], "characters": "\u2233" },+  "&Cross;": { "codepoints": [10799], "characters": "\u2A2F" },+  "&Cscr;": { "codepoints": [119966], "characters": "\uD835\uDC9E" },+  "&Cup;": { "codepoints": [8915], "characters": "\u22D3" },+  "&CupCap;": { "codepoints": [8781], "characters": "\u224D" },+  "&DD;": { "codepoints": [8517], "characters": "\u2145" },+  "&DDotrahd;": { "codepoints": [10513], "characters": "\u2911" },+  "&DJcy;": { "codepoints": [1026], "characters": "\u0402" },+  "&DScy;": { "codepoints": [1029], "characters": "\u0405" },+  "&DZcy;": { "codepoints": [1039], "characters": "\u040F" },+  "&Dagger;": { "codepoints": [8225], "characters": "\u2021" },+  "&Darr;": { "codepoints": [8609], "characters": "\u21A1" },+  "&Dashv;": { "codepoints": [10980], "characters": "\u2AE4" },+  "&Dcaron;": { "codepoints": [270], "characters": "\u010E" },+  "&Dcy;": { "codepoints": [1044], "characters": "\u0414" },+  "&Del;": { "codepoints": [8711], "characters": "\u2207" },+  "&Delta;": { "codepoints": [916], "characters": "\u0394" },+  "&Dfr;": { "codepoints": [120071], "characters": "\uD835\uDD07" },+  "&DiacriticalAcute;": { "codepoints": [180], "characters": "\u00B4" },+  "&DiacriticalDot;": { "codepoints": [729], "characters": "\u02D9" },+  "&DiacriticalDoubleAcute;": { "codepoints": [733], "characters": "\u02DD" },+  "&DiacriticalGrave;": { "codepoints": [96], "characters": "\u0060" },+  "&DiacriticalTilde;": { "codepoints": [732], "characters": "\u02DC" },+  "&Diamond;": { "codepoints": [8900], "characters": "\u22C4" },+  "&DifferentialD;": { "codepoints": [8518], "characters": "\u2146" },+  "&Dopf;": { "codepoints": [120123], "characters": "\uD835\uDD3B" },+  "&Dot;": { "codepoints": [168], "characters": "\u00A8" },+  "&DotDot;": { "codepoints": [8412], "characters": "\u20DC" },+  "&DotEqual;": { "codepoints": [8784], "characters": "\u2250" },+  "&DoubleContourIntegral;": { "codepoints": [8751], "characters": "\u222F" },+  "&DoubleDot;": { "codepoints": [168], "characters": "\u00A8" },+  "&DoubleDownArrow;": { "codepoints": [8659], "characters": "\u21D3" },+  "&DoubleLeftArrow;": { "codepoints": [8656], "characters": "\u21D0" },+  "&DoubleLeftRightArrow;": { "codepoints": [8660], "characters": "\u21D4" },+  "&DoubleLeftTee;": { "codepoints": [10980], "characters": "\u2AE4" },+  "&DoubleLongLeftArrow;": { "codepoints": [10232], "characters": "\u27F8" },+  "&DoubleLongLeftRightArrow;": { "codepoints": [10234], "characters": "\u27FA" },+  "&DoubleLongRightArrow;": { "codepoints": [10233], "characters": "\u27F9" },+  "&DoubleRightArrow;": { "codepoints": [8658], "characters": "\u21D2" },+  "&DoubleRightTee;": { "codepoints": [8872], "characters": "\u22A8" },+  "&DoubleUpArrow;": { "codepoints": [8657], "characters": "\u21D1" },+  "&DoubleUpDownArrow;": { "codepoints": [8661], "characters": "\u21D5" },+  "&DoubleVerticalBar;": { "codepoints": [8741], "characters": "\u2225" },+  "&DownArrow;": { "codepoints": [8595], "characters": "\u2193" },+  "&DownArrowBar;": { "codepoints": [10515], "characters": "\u2913" },+  "&DownArrowUpArrow;": { "codepoints": [8693], "characters": "\u21F5" },+  "&DownBreve;": { "codepoints": [785], "characters": "\u0311" },+  "&DownLeftRightVector;": { "codepoints": [10576], "characters": "\u2950" },+  "&DownLeftTeeVector;": { "codepoints": [10590], "characters": "\u295E" },+  "&DownLeftVector;": { "codepoints": [8637], "characters": "\u21BD" },+  "&DownLeftVectorBar;": { "codepoints": [10582], "characters": "\u2956" },+  "&DownRightTeeVector;": { "codepoints": [10591], "characters": "\u295F" },+  "&DownRightVector;": { "codepoints": [8641], "characters": "\u21C1" },+  "&DownRightVectorBar;": { "codepoints": [10583], "characters": "\u2957" },+  "&DownTee;": { "codepoints": [8868], "characters": "\u22A4" },+  "&DownTeeArrow;": { "codepoints": [8615], "characters": "\u21A7" },+  "&Downarrow;": { "codepoints": [8659], "characters": "\u21D3" },+  "&Dscr;": { "codepoints": [119967], "characters": "\uD835\uDC9F" },+  "&Dstrok;": { "codepoints": [272], "characters": "\u0110" },+  "&ENG;": { "codepoints": [330], "characters": "\u014A" },+  "&ETH": { "codepoints": [208], "characters": "\u00D0" },+  "&ETH;": { "codepoints": [208], "characters": "\u00D0" },+  "&Eacute": { "codepoints": [201], "characters": "\u00C9" },+  "&Eacute;": { "codepoints": [201], "characters": "\u00C9" },+  "&Ecaron;": { "codepoints": [282], "characters": "\u011A" },+  "&Ecirc": { "codepoints": [202], "characters": "\u00CA" },+  "&Ecirc;": { "codepoints": [202], "characters": "\u00CA" },+  "&Ecy;": { "codepoints": [1069], "characters": "\u042D" },+  "&Edot;": { "codepoints": [278], "characters": "\u0116" },+  "&Efr;": { "codepoints": [120072], "characters": "\uD835\uDD08" },+  "&Egrave": { "codepoints": [200], "characters": "\u00C8" },+  "&Egrave;": { "codepoints": [200], "characters": "\u00C8" },+  "&Element;": { "codepoints": [8712], "characters": "\u2208" },+  "&Emacr;": { "codepoints": [274], "characters": "\u0112" },+  "&EmptySmallSquare;": { "codepoints": [9723], "characters": "\u25FB" },+  "&EmptyVerySmallSquare;": { "codepoints": [9643], "characters": "\u25AB" },+  "&Eogon;": { "codepoints": [280], "characters": "\u0118" },+  "&Eopf;": { "codepoints": [120124], "characters": "\uD835\uDD3C" },+  "&Epsilon;": { "codepoints": [917], "characters": "\u0395" },+  "&Equal;": { "codepoints": [10869], "characters": "\u2A75" },+  "&EqualTilde;": { "codepoints": [8770], "characters": "\u2242" },+  "&Equilibrium;": { "codepoints": [8652], "characters": "\u21CC" },+  "&Escr;": { "codepoints": [8496], "characters": "\u2130" },+  "&Esim;": { "codepoints": [10867], "characters": "\u2A73" },+  "&Eta;": { "codepoints": [919], "characters": "\u0397" },+  "&Euml": { "codepoints": [203], "characters": "\u00CB" },+  "&Euml;": { "codepoints": [203], "characters": "\u00CB" },+  "&Exists;": { "codepoints": [8707], "characters": "\u2203" },+  "&ExponentialE;": { "codepoints": [8519], "characters": "\u2147" },+  "&Fcy;": { "codepoints": [1060], "characters": "\u0424" },+  "&Ffr;": { "codepoints": [120073], "characters": "\uD835\uDD09" },+  "&FilledSmallSquare;": { "codepoints": [9724], "characters": "\u25FC" },+  "&FilledVerySmallSquare;": { "codepoints": [9642], "characters": "\u25AA" },+  "&Fopf;": { "codepoints": [120125], "characters": "\uD835\uDD3D" },+  "&ForAll;": { "codepoints": [8704], "characters": "\u2200" },+  "&Fouriertrf;": { "codepoints": [8497], "characters": "\u2131" },+  "&Fscr;": { "codepoints": [8497], "characters": "\u2131" },+  "&GJcy;": { "codepoints": [1027], "characters": "\u0403" },+  "&GT": { "codepoints": [62], "characters": "\u003E" },+  "&GT;": { "codepoints": [62], "characters": "\u003E" },+  "&Gamma;": { "codepoints": [915], "characters": "\u0393" },+  "&Gammad;": { "codepoints": [988], "characters": "\u03DC" },+  "&Gbreve;": { "codepoints": [286], "characters": "\u011E" },+  "&Gcedil;": { "codepoints": [290], "characters": "\u0122" },+  "&Gcirc;": { "codepoints": [284], "characters": "\u011C" },+  "&Gcy;": { "codepoints": [1043], "characters": "\u0413" },+  "&Gdot;": { "codepoints": [288], "characters": "\u0120" },+  "&Gfr;": { "codepoints": [120074], "characters": "\uD835\uDD0A" },+  "&Gg;": { "codepoints": [8921], "characters": "\u22D9" },+  "&Gopf;": { "codepoints": [120126], "characters": "\uD835\uDD3E" },+  "&GreaterEqual;": { "codepoints": [8805], "characters": "\u2265" },+  "&GreaterEqualLess;": { "codepoints": [8923], "characters": "\u22DB" },+  "&GreaterFullEqual;": { "codepoints": [8807], "characters": "\u2267" },+  "&GreaterGreater;": { "codepoints": [10914], "characters": "\u2AA2" },+  "&GreaterLess;": { "codepoints": [8823], "characters": "\u2277" },+  "&GreaterSlantEqual;": { "codepoints": [10878], "characters": "\u2A7E" },+  "&GreaterTilde;": { "codepoints": [8819], "characters": "\u2273" },+  "&Gscr;": { "codepoints": [119970], "characters": "\uD835\uDCA2" },+  "&Gt;": { "codepoints": [8811], "characters": "\u226B" },+  "&HARDcy;": { "codepoints": [1066], "characters": "\u042A" },+  "&Hacek;": { "codepoints": [711], "characters": "\u02C7" },+  "&Hat;": { "codepoints": [94], "characters": "\u005E" },+  "&Hcirc;": { "codepoints": [292], "characters": "\u0124" },+  "&Hfr;": { "codepoints": [8460], "characters": "\u210C" },+  "&HilbertSpace;": { "codepoints": [8459], "characters": "\u210B" },+  "&Hopf;": { "codepoints": [8461], "characters": "\u210D" },+  "&HorizontalLine;": { "codepoints": [9472], "characters": "\u2500" },+  "&Hscr;": { "codepoints": [8459], "characters": "\u210B" },+  "&Hstrok;": { "codepoints": [294], "characters": "\u0126" },+  "&HumpDownHump;": { "codepoints": [8782], "characters": "\u224E" },+  "&HumpEqual;": { "codepoints": [8783], "characters": "\u224F" },+  "&IEcy;": { "codepoints": [1045], "characters": "\u0415" },+  "&IJlig;": { "codepoints": [306], "characters": "\u0132" },+  "&IOcy;": { "codepoints": [1025], "characters": "\u0401" },+  "&Iacute": { "codepoints": [205], "characters": "\u00CD" },+  "&Iacute;": { "codepoints": [205], "characters": "\u00CD" },+  "&Icirc": { "codepoints": [206], "characters": "\u00CE" },+  "&Icirc;": { "codepoints": [206], "characters": "\u00CE" },+  "&Icy;": { "codepoints": [1048], "characters": "\u0418" },+  "&Idot;": { "codepoints": [304], "characters": "\u0130" },+  "&Ifr;": { "codepoints": [8465], "characters": "\u2111" },+  "&Igrave": { "codepoints": [204], "characters": "\u00CC" },+  "&Igrave;": { "codepoints": [204], "characters": "\u00CC" },+  "&Im;": { "codepoints": [8465], "characters": "\u2111" },+  "&Imacr;": { "codepoints": [298], "characters": "\u012A" },+  "&ImaginaryI;": { "codepoints": [8520], "characters": "\u2148" },+  "&Implies;": { "codepoints": [8658], "characters": "\u21D2" },+  "&Int;": { "codepoints": [8748], "characters": "\u222C" },+  "&Integral;": { "codepoints": [8747], "characters": "\u222B" },+  "&Intersection;": { "codepoints": [8898], "characters": "\u22C2" },+  "&InvisibleComma;": { "codepoints": [8291], "characters": "\u2063" },+  "&InvisibleTimes;": { "codepoints": [8290], "characters": "\u2062" },+  "&Iogon;": { "codepoints": [302], "characters": "\u012E" },+  "&Iopf;": { "codepoints": [120128], "characters": "\uD835\uDD40" },+  "&Iota;": { "codepoints": [921], "characters": "\u0399" },+  "&Iscr;": { "codepoints": [8464], "characters": "\u2110" },+  "&Itilde;": { "codepoints": [296], "characters": "\u0128" },+  "&Iukcy;": { "codepoints": [1030], "characters": "\u0406" },+  "&Iuml": { "codepoints": [207], "characters": "\u00CF" },+  "&Iuml;": { "codepoints": [207], "characters": "\u00CF" },+  "&Jcirc;": { "codepoints": [308], "characters": "\u0134" },+  "&Jcy;": { "codepoints": [1049], "characters": "\u0419" },+  "&Jfr;": { "codepoints": [120077], "characters": "\uD835\uDD0D" },+  "&Jopf;": { "codepoints": [120129], "characters": "\uD835\uDD41" },+  "&Jscr;": { "codepoints": [119973], "characters": "\uD835\uDCA5" },+  "&Jsercy;": { "codepoints": [1032], "characters": "\u0408" },+  "&Jukcy;": { "codepoints": [1028], "characters": "\u0404" },+  "&KHcy;": { "codepoints": [1061], "characters": "\u0425" },+  "&KJcy;": { "codepoints": [1036], "characters": "\u040C" },+  "&Kappa;": { "codepoints": [922], "characters": "\u039A" },+  "&Kcedil;": { "codepoints": [310], "characters": "\u0136" },+  "&Kcy;": { "codepoints": [1050], "characters": "\u041A" },+  "&Kfr;": { "codepoints": [120078], "characters": "\uD835\uDD0E" },+  "&Kopf;": { "codepoints": [120130], "characters": "\uD835\uDD42" },+  "&Kscr;": { "codepoints": [119974], "characters": "\uD835\uDCA6" },+  "&LJcy;": { "codepoints": [1033], "characters": "\u0409" },+  "&LT": { "codepoints": [60], "characters": "\u003C" },+  "&LT;": { "codepoints": [60], "characters": "\u003C" },+  "&Lacute;": { "codepoints": [313], "characters": "\u0139" },+  "&Lambda;": { "codepoints": [923], "characters": "\u039B" },+  "&Lang;": { "codepoints": [10218], "characters": "\u27EA" },+  "&Laplacetrf;": { "codepoints": [8466], "characters": "\u2112" },+  "&Larr;": { "codepoints": [8606], "characters": "\u219E" },+  "&Lcaron;": { "codepoints": [317], "characters": "\u013D" },+  "&Lcedil;": { "codepoints": [315], "characters": "\u013B" },+  "&Lcy;": { "codepoints": [1051], "characters": "\u041B" },+  "&LeftAngleBracket;": { "codepoints": [10216], "characters": "\u27E8" },+  "&LeftArrow;": { "codepoints": [8592], "characters": "\u2190" },+  "&LeftArrowBar;": { "codepoints": [8676], "characters": "\u21E4" },+  "&LeftArrowRightArrow;": { "codepoints": [8646], "characters": "\u21C6" },+  "&LeftCeiling;": { "codepoints": [8968], "characters": "\u2308" },+  "&LeftDoubleBracket;": { "codepoints": [10214], "characters": "\u27E6" },+  "&LeftDownTeeVector;": { "codepoints": [10593], "characters": "\u2961" },+  "&LeftDownVector;": { "codepoints": [8643], "characters": "\u21C3" },+  "&LeftDownVectorBar;": { "codepoints": [10585], "characters": "\u2959" },+  "&LeftFloor;": { "codepoints": [8970], "characters": "\u230A" },+  "&LeftRightArrow;": { "codepoints": [8596], "characters": "\u2194" },+  "&LeftRightVector;": { "codepoints": [10574], "characters": "\u294E" },+  "&LeftTee;": { "codepoints": [8867], "characters": "\u22A3" },+  "&LeftTeeArrow;": { "codepoints": [8612], "characters": "\u21A4" },+  "&LeftTeeVector;": { "codepoints": [10586], "characters": "\u295A" },+  "&LeftTriangle;": { "codepoints": [8882], "characters": "\u22B2" },+  "&LeftTriangleBar;": { "codepoints": [10703], "characters": "\u29CF" },+  "&LeftTriangleEqual;": { "codepoints": [8884], "characters": "\u22B4" },+  "&LeftUpDownVector;": { "codepoints": [10577], "characters": "\u2951" },+  "&LeftUpTeeVector;": { "codepoints": [10592], "characters": "\u2960" },+  "&LeftUpVector;": { "codepoints": [8639], "characters": "\u21BF" },+  "&LeftUpVectorBar;": { "codepoints": [10584], "characters": "\u2958" },+  "&LeftVector;": { "codepoints": [8636], "characters": "\u21BC" },+  "&LeftVectorBar;": { "codepoints": [10578], "characters": "\u2952" },+  "&Leftarrow;": { "codepoints": [8656], "characters": "\u21D0" },+  "&Leftrightarrow;": { "codepoints": [8660], "characters": "\u21D4" },+  "&LessEqualGreater;": { "codepoints": [8922], "characters": "\u22DA" },+  "&LessFullEqual;": { "codepoints": [8806], "characters": "\u2266" },+  "&LessGreater;": { "codepoints": [8822], "characters": "\u2276" },+  "&LessLess;": { "codepoints": [10913], "characters": "\u2AA1" },+  "&LessSlantEqual;": { "codepoints": [10877], "characters": "\u2A7D" },+  "&LessTilde;": { "codepoints": [8818], "characters": "\u2272" },+  "&Lfr;": { "codepoints": [120079], "characters": "\uD835\uDD0F" },+  "&Ll;": { "codepoints": [8920], "characters": "\u22D8" },+  "&Lleftarrow;": { "codepoints": [8666], "characters": "\u21DA" },+  "&Lmidot;": { "codepoints": [319], "characters": "\u013F" },+  "&LongLeftArrow;": { "codepoints": [10229], "characters": "\u27F5" },+  "&LongLeftRightArrow;": { "codepoints": [10231], "characters": "\u27F7" },+  "&LongRightArrow;": { "codepoints": [10230], "characters": "\u27F6" },+  "&Longleftarrow;": { "codepoints": [10232], "characters": "\u27F8" },+  "&Longleftrightarrow;": { "codepoints": [10234], "characters": "\u27FA" },+  "&Longrightarrow;": { "codepoints": [10233], "characters": "\u27F9" },+  "&Lopf;": { "codepoints": [120131], "characters": "\uD835\uDD43" },+  "&LowerLeftArrow;": { "codepoints": [8601], "characters": "\u2199" },+  "&LowerRightArrow;": { "codepoints": [8600], "characters": "\u2198" },+  "&Lscr;": { "codepoints": [8466], "characters": "\u2112" },+  "&Lsh;": { "codepoints": [8624], "characters": "\u21B0" },+  "&Lstrok;": { "codepoints": [321], "characters": "\u0141" },+  "&Lt;": { "codepoints": [8810], "characters": "\u226A" },+  "&Map;": { "codepoints": [10501], "characters": "\u2905" },+  "&Mcy;": { "codepoints": [1052], "characters": "\u041C" },+  "&MediumSpace;": { "codepoints": [8287], "characters": "\u205F" },+  "&Mellintrf;": { "codepoints": [8499], "characters": "\u2133" },+  "&Mfr;": { "codepoints": [120080], "characters": "\uD835\uDD10" },+  "&MinusPlus;": { "codepoints": [8723], "characters": "\u2213" },+  "&Mopf;": { "codepoints": [120132], "characters": "\uD835\uDD44" },+  "&Mscr;": { "codepoints": [8499], "characters": "\u2133" },+  "&Mu;": { "codepoints": [924], "characters": "\u039C" },+  "&NJcy;": { "codepoints": [1034], "characters": "\u040A" },+  "&Nacute;": { "codepoints": [323], "characters": "\u0143" },+  "&Ncaron;": { "codepoints": [327], "characters": "\u0147" },+  "&Ncedil;": { "codepoints": [325], "characters": "\u0145" },+  "&Ncy;": { "codepoints": [1053], "characters": "\u041D" },+  "&NegativeMediumSpace;": { "codepoints": [8203], "characters": "\u200B" },+  "&NegativeThickSpace;": { "codepoints": [8203], "characters": "\u200B" },+  "&NegativeThinSpace;": { "codepoints": [8203], "characters": "\u200B" },+  "&NegativeVeryThinSpace;": { "codepoints": [8203], "characters": "\u200B" },+  "&NestedGreaterGreater;": { "codepoints": [8811], "characters": "\u226B" },+  "&NestedLessLess;": { "codepoints": [8810], "characters": "\u226A" },+  "&NewLine;": { "codepoints": [10], "characters": "\u000A" },+  "&Nfr;": { "codepoints": [120081], "characters": "\uD835\uDD11" },+  "&NoBreak;": { "codepoints": [8288], "characters": "\u2060" },+  "&NonBreakingSpace;": { "codepoints": [160], "characters": "\u00A0" },+  "&Nopf;": { "codepoints": [8469], "characters": "\u2115" },+  "&Not;": { "codepoints": [10988], "characters": "\u2AEC" },+  "&NotCongruent;": { "codepoints": [8802], "characters": "\u2262" },+  "&NotCupCap;": { "codepoints": [8813], "characters": "\u226D" },+  "&NotDoubleVerticalBar;": { "codepoints": [8742], "characters": "\u2226" },+  "&NotElement;": { "codepoints": [8713], "characters": "\u2209" },+  "&NotEqual;": { "codepoints": [8800], "characters": "\u2260" },+  "&NotEqualTilde;": { "codepoints": [8770, 824], "characters": "\u2242\u0338" },+  "&NotExists;": { "codepoints": [8708], "characters": "\u2204" },+  "&NotGreater;": { "codepoints": [8815], "characters": "\u226F" },+  "&NotGreaterEqual;": { "codepoints": [8817], "characters": "\u2271" },+  "&NotGreaterFullEqual;": { "codepoints": [8807, 824], "characters": "\u2267\u0338" },+  "&NotGreaterGreater;": { "codepoints": [8811, 824], "characters": "\u226B\u0338" },+  "&NotGreaterLess;": { "codepoints": [8825], "characters": "\u2279" },+  "&NotGreaterSlantEqual;": { "codepoints": [10878, 824], "characters": "\u2A7E\u0338" },+  "&NotGreaterTilde;": { "codepoints": [8821], "characters": "\u2275" },+  "&NotHumpDownHump;": { "codepoints": [8782, 824], "characters": "\u224E\u0338" },+  "&NotHumpEqual;": { "codepoints": [8783, 824], "characters": "\u224F\u0338" },+  "&NotLeftTriangle;": { "codepoints": [8938], "characters": "\u22EA" },+  "&NotLeftTriangleBar;": { "codepoints": [10703, 824], "characters": "\u29CF\u0338" },+  "&NotLeftTriangleEqual;": { "codepoints": [8940], "characters": "\u22EC" },+  "&NotLess;": { "codepoints": [8814], "characters": "\u226E" },+  "&NotLessEqual;": { "codepoints": [8816], "characters": "\u2270" },+  "&NotLessGreater;": { "codepoints": [8824], "characters": "\u2278" },+  "&NotLessLess;": { "codepoints": [8810, 824], "characters": "\u226A\u0338" },+  "&NotLessSlantEqual;": { "codepoints": [10877, 824], "characters": "\u2A7D\u0338" },+  "&NotLessTilde;": { "codepoints": [8820], "characters": "\u2274" },+  "&NotNestedGreaterGreater;": { "codepoints": [10914, 824], "characters": "\u2AA2\u0338" },+  "&NotNestedLessLess;": { "codepoints": [10913, 824], "characters": "\u2AA1\u0338" },+  "&NotPrecedes;": { "codepoints": [8832], "characters": "\u2280" },+  "&NotPrecedesEqual;": { "codepoints": [10927, 824], "characters": "\u2AAF\u0338" },+  "&NotPrecedesSlantEqual;": { "codepoints": [8928], "characters": "\u22E0" },+  "&NotReverseElement;": { "codepoints": [8716], "characters": "\u220C" },+  "&NotRightTriangle;": { "codepoints": [8939], "characters": "\u22EB" },+  "&NotRightTriangleBar;": { "codepoints": [10704, 824], "characters": "\u29D0\u0338" },+  "&NotRightTriangleEqual;": { "codepoints": [8941], "characters": "\u22ED" },+  "&NotSquareSubset;": { "codepoints": [8847, 824], "characters": "\u228F\u0338" },+  "&NotSquareSubsetEqual;": { "codepoints": [8930], "characters": "\u22E2" },+  "&NotSquareSuperset;": { "codepoints": [8848, 824], "characters": "\u2290\u0338" },+  "&NotSquareSupersetEqual;": { "codepoints": [8931], "characters": "\u22E3" },+  "&NotSubset;": { "codepoints": [8834, 8402], "characters": "\u2282\u20D2" },+  "&NotSubsetEqual;": { "codepoints": [8840], "characters": "\u2288" },+  "&NotSucceeds;": { "codepoints": [8833], "characters": "\u2281" },+  "&NotSucceedsEqual;": { "codepoints": [10928, 824], "characters": "\u2AB0\u0338" },+  "&NotSucceedsSlantEqual;": { "codepoints": [8929], "characters": "\u22E1" },+  "&NotSucceedsTilde;": { "codepoints": [8831, 824], "characters": "\u227F\u0338" },+  "&NotSuperset;": { "codepoints": [8835, 8402], "characters": "\u2283\u20D2" },+  "&NotSupersetEqual;": { "codepoints": [8841], "characters": "\u2289" },+  "&NotTilde;": { "codepoints": [8769], "characters": "\u2241" },+  "&NotTildeEqual;": { "codepoints": [8772], "characters": "\u2244" },+  "&NotTildeFullEqual;": { "codepoints": [8775], "characters": "\u2247" },+  "&NotTildeTilde;": { "codepoints": [8777], "characters": "\u2249" },+  "&NotVerticalBar;": { "codepoints": [8740], "characters": "\u2224" },+  "&Nscr;": { "codepoints": [119977], "characters": "\uD835\uDCA9" },+  "&Ntilde": { "codepoints": [209], "characters": "\u00D1" },+  "&Ntilde;": { "codepoints": [209], "characters": "\u00D1" },+  "&Nu;": { "codepoints": [925], "characters": "\u039D" },+  "&OElig;": { "codepoints": [338], "characters": "\u0152" },+  "&Oacute": { "codepoints": [211], "characters": "\u00D3" },+  "&Oacute;": { "codepoints": [211], "characters": "\u00D3" },+  "&Ocirc": { "codepoints": [212], "characters": "\u00D4" },+  "&Ocirc;": { "codepoints": [212], "characters": "\u00D4" },+  "&Ocy;": { "codepoints": [1054], "characters": "\u041E" },+  "&Odblac;": { "codepoints": [336], "characters": "\u0150" },+  "&Ofr;": { "codepoints": [120082], "characters": "\uD835\uDD12" },+  "&Ograve": { "codepoints": [210], "characters": "\u00D2" },+  "&Ograve;": { "codepoints": [210], "characters": "\u00D2" },+  "&Omacr;": { "codepoints": [332], "characters": "\u014C" },+  "&Omega;": { "codepoints": [937], "characters": "\u03A9" },+  "&Omicron;": { "codepoints": [927], "characters": "\u039F" },+  "&Oopf;": { "codepoints": [120134], "characters": "\uD835\uDD46" },+  "&OpenCurlyDoubleQuote;": { "codepoints": [8220], "characters": "\u201C" },+  "&OpenCurlyQuote;": { "codepoints": [8216], "characters": "\u2018" },+  "&Or;": { "codepoints": [10836], "characters": "\u2A54" },+  "&Oscr;": { "codepoints": [119978], "characters": "\uD835\uDCAA" },+  "&Oslash": { "codepoints": [216], "characters": "\u00D8" },+  "&Oslash;": { "codepoints": [216], "characters": "\u00D8" },+  "&Otilde": { "codepoints": [213], "characters": "\u00D5" },+  "&Otilde;": { "codepoints": [213], "characters": "\u00D5" },+  "&Otimes;": { "codepoints": [10807], "characters": "\u2A37" },+  "&Ouml": { "codepoints": [214], "characters": "\u00D6" },+  "&Ouml;": { "codepoints": [214], "characters": "\u00D6" },+  "&OverBar;": { "codepoints": [8254], "characters": "\u203E" },+  "&OverBrace;": { "codepoints": [9182], "characters": "\u23DE" },+  "&OverBracket;": { "codepoints": [9140], "characters": "\u23B4" },+  "&OverParenthesis;": { "codepoints": [9180], "characters": "\u23DC" },+  "&PartialD;": { "codepoints": [8706], "characters": "\u2202" },+  "&Pcy;": { "codepoints": [1055], "characters": "\u041F" },+  "&Pfr;": { "codepoints": [120083], "characters": "\uD835\uDD13" },+  "&Phi;": { "codepoints": [934], "characters": "\u03A6" },+  "&Pi;": { "codepoints": [928], "characters": "\u03A0" },+  "&PlusMinus;": { "codepoints": [177], "characters": "\u00B1" },+  "&Poincareplane;": { "codepoints": [8460], "characters": "\u210C" },+  "&Popf;": { "codepoints": [8473], "characters": "\u2119" },+  "&Pr;": { "codepoints": [10939], "characters": "\u2ABB" },+  "&Precedes;": { "codepoints": [8826], "characters": "\u227A" },+  "&PrecedesEqual;": { "codepoints": [10927], "characters": "\u2AAF" },+  "&PrecedesSlantEqual;": { "codepoints": [8828], "characters": "\u227C" },+  "&PrecedesTilde;": { "codepoints": [8830], "characters": "\u227E" },+  "&Prime;": { "codepoints": [8243], "characters": "\u2033" },+  "&Product;": { "codepoints": [8719], "characters": "\u220F" },+  "&Proportion;": { "codepoints": [8759], "characters": "\u2237" },+  "&Proportional;": { "codepoints": [8733], "characters": "\u221D" },+  "&Pscr;": { "codepoints": [119979], "characters": "\uD835\uDCAB" },+  "&Psi;": { "codepoints": [936], "characters": "\u03A8" },+  "&QUOT": { "codepoints": [34], "characters": "\u0022" },+  "&QUOT;": { "codepoints": [34], "characters": "\u0022" },+  "&Qfr;": { "codepoints": [120084], "characters": "\uD835\uDD14" },+  "&Qopf;": { "codepoints": [8474], "characters": "\u211A" },+  "&Qscr;": { "codepoints": [119980], "characters": "\uD835\uDCAC" },+  "&RBarr;": { "codepoints": [10512], "characters": "\u2910" },+  "&REG": { "codepoints": [174], "characters": "\u00AE" },+  "&REG;": { "codepoints": [174], "characters": "\u00AE" },+  "&Racute;": { "codepoints": [340], "characters": "\u0154" },+  "&Rang;": { "codepoints": [10219], "characters": "\u27EB" },+  "&Rarr;": { "codepoints": [8608], "characters": "\u21A0" },+  "&Rarrtl;": { "codepoints": [10518], "characters": "\u2916" },+  "&Rcaron;": { "codepoints": [344], "characters": "\u0158" },+  "&Rcedil;": { "codepoints": [342], "characters": "\u0156" },+  "&Rcy;": { "codepoints": [1056], "characters": "\u0420" },+  "&Re;": { "codepoints": [8476], "characters": "\u211C" },+  "&ReverseElement;": { "codepoints": [8715], "characters": "\u220B" },+  "&ReverseEquilibrium;": { "codepoints": [8651], "characters": "\u21CB" },+  "&ReverseUpEquilibrium;": { "codepoints": [10607], "characters": "\u296F" },+  "&Rfr;": { "codepoints": [8476], "characters": "\u211C" },+  "&Rho;": { "codepoints": [929], "characters": "\u03A1" },+  "&RightAngleBracket;": { "codepoints": [10217], "characters": "\u27E9" },+  "&RightArrow;": { "codepoints": [8594], "characters": "\u2192" },+  "&RightArrowBar;": { "codepoints": [8677], "characters": "\u21E5" },+  "&RightArrowLeftArrow;": { "codepoints": [8644], "characters": "\u21C4" },+  "&RightCeiling;": { "codepoints": [8969], "characters": "\u2309" },+  "&RightDoubleBracket;": { "codepoints": [10215], "characters": "\u27E7" },+  "&RightDownTeeVector;": { "codepoints": [10589], "characters": "\u295D" },+  "&RightDownVector;": { "codepoints": [8642], "characters": "\u21C2" },+  "&RightDownVectorBar;": { "codepoints": [10581], "characters": "\u2955" },+  "&RightFloor;": { "codepoints": [8971], "characters": "\u230B" },+  "&RightTee;": { "codepoints": [8866], "characters": "\u22A2" },+  "&RightTeeArrow;": { "codepoints": [8614], "characters": "\u21A6" },+  "&RightTeeVector;": { "codepoints": [10587], "characters": "\u295B" },+  "&RightTriangle;": { "codepoints": [8883], "characters": "\u22B3" },+  "&RightTriangleBar;": { "codepoints": [10704], "characters": "\u29D0" },+  "&RightTriangleEqual;": { "codepoints": [8885], "characters": "\u22B5" },+  "&RightUpDownVector;": { "codepoints": [10575], "characters": "\u294F" },+  "&RightUpTeeVector;": { "codepoints": [10588], "characters": "\u295C" },+  "&RightUpVector;": { "codepoints": [8638], "characters": "\u21BE" },+  "&RightUpVectorBar;": { "codepoints": [10580], "characters": "\u2954" },+  "&RightVector;": { "codepoints": [8640], "characters": "\u21C0" },+  "&RightVectorBar;": { "codepoints": [10579], "characters": "\u2953" },+  "&Rightarrow;": { "codepoints": [8658], "characters": "\u21D2" },+  "&Ropf;": { "codepoints": [8477], "characters": "\u211D" },+  "&RoundImplies;": { "codepoints": [10608], "characters": "\u2970" },+  "&Rrightarrow;": { "codepoints": [8667], "characters": "\u21DB" },+  "&Rscr;": { "codepoints": [8475], "characters": "\u211B" },+  "&Rsh;": { "codepoints": [8625], "characters": "\u21B1" },+  "&RuleDelayed;": { "codepoints": [10740], "characters": "\u29F4" },+  "&SHCHcy;": { "codepoints": [1065], "characters": "\u0429" },+  "&SHcy;": { "codepoints": [1064], "characters": "\u0428" },+  "&SOFTcy;": { "codepoints": [1068], "characters": "\u042C" },+  "&Sacute;": { "codepoints": [346], "characters": "\u015A" },+  "&Sc;": { "codepoints": [10940], "characters": "\u2ABC" },+  "&Scaron;": { "codepoints": [352], "characters": "\u0160" },+  "&Scedil;": { "codepoints": [350], "characters": "\u015E" },+  "&Scirc;": { "codepoints": [348], "characters": "\u015C" },+  "&Scy;": { "codepoints": [1057], "characters": "\u0421" },+  "&Sfr;": { "codepoints": [120086], "characters": "\uD835\uDD16" },+  "&ShortDownArrow;": { "codepoints": [8595], "characters": "\u2193" },+  "&ShortLeftArrow;": { "codepoints": [8592], "characters": "\u2190" },+  "&ShortRightArrow;": { "codepoints": [8594], "characters": "\u2192" },+  "&ShortUpArrow;": { "codepoints": [8593], "characters": "\u2191" },+  "&Sigma;": { "codepoints": [931], "characters": "\u03A3" },+  "&SmallCircle;": { "codepoints": [8728], "characters": "\u2218" },+  "&Sopf;": { "codepoints": [120138], "characters": "\uD835\uDD4A" },+  "&Sqrt;": { "codepoints": [8730], "characters": "\u221A" },+  "&Square;": { "codepoints": [9633], "characters": "\u25A1" },+  "&SquareIntersection;": { "codepoints": [8851], "characters": "\u2293" },+  "&SquareSubset;": { "codepoints": [8847], "characters": "\u228F" },+  "&SquareSubsetEqual;": { "codepoints": [8849], "characters": "\u2291" },+  "&SquareSuperset;": { "codepoints": [8848], "characters": "\u2290" },+  "&SquareSupersetEqual;": { "codepoints": [8850], "characters": "\u2292" },+  "&SquareUnion;": { "codepoints": [8852], "characters": "\u2294" },+  "&Sscr;": { "codepoints": [119982], "characters": "\uD835\uDCAE" },+  "&Star;": { "codepoints": [8902], "characters": "\u22C6" },+  "&Sub;": { "codepoints": [8912], "characters": "\u22D0" },+  "&Subset;": { "codepoints": [8912], "characters": "\u22D0" },+  "&SubsetEqual;": { "codepoints": [8838], "characters": "\u2286" },+  "&Succeeds;": { "codepoints": [8827], "characters": "\u227B" },+  "&SucceedsEqual;": { "codepoints": [10928], "characters": "\u2AB0" },+  "&SucceedsSlantEqual;": { "codepoints": [8829], "characters": "\u227D" },+  "&SucceedsTilde;": { "codepoints": [8831], "characters": "\u227F" },+  "&SuchThat;": { "codepoints": [8715], "characters": "\u220B" },+  "&Sum;": { "codepoints": [8721], "characters": "\u2211" },+  "&Sup;": { "codepoints": [8913], "characters": "\u22D1" },+  "&Superset;": { "codepoints": [8835], "characters": "\u2283" },+  "&SupersetEqual;": { "codepoints": [8839], "characters": "\u2287" },+  "&Supset;": { "codepoints": [8913], "characters": "\u22D1" },+  "&THORN": { "codepoints": [222], "characters": "\u00DE" },+  "&THORN;": { "codepoints": [222], "characters": "\u00DE" },+  "&TRADE;": { "codepoints": [8482], "characters": "\u2122" },+  "&TSHcy;": { "codepoints": [1035], "characters": "\u040B" },+  "&TScy;": { "codepoints": [1062], "characters": "\u0426" },+  "&Tab;": { "codepoints": [9], "characters": "\u0009" },+  "&Tau;": { "codepoints": [932], "characters": "\u03A4" },+  "&Tcaron;": { "codepoints": [356], "characters": "\u0164" },+  "&Tcedil;": { "codepoints": [354], "characters": "\u0162" },+  "&Tcy;": { "codepoints": [1058], "characters": "\u0422" },+  "&Tfr;": { "codepoints": [120087], "characters": "\uD835\uDD17" },+  "&Therefore;": { "codepoints": [8756], "characters": "\u2234" },+  "&Theta;": { "codepoints": [920], "characters": "\u0398" },+  "&ThickSpace;": { "codepoints": [8287, 8202], "characters": "\u205F\u200A" },+  "&ThinSpace;": { "codepoints": [8201], "characters": "\u2009" },+  "&Tilde;": { "codepoints": [8764], "characters": "\u223C" },+  "&TildeEqual;": { "codepoints": [8771], "characters": "\u2243" },+  "&TildeFullEqual;": { "codepoints": [8773], "characters": "\u2245" },+  "&TildeTilde;": { "codepoints": [8776], "characters": "\u2248" },+  "&Topf;": { "codepoints": [120139], "characters": "\uD835\uDD4B" },+  "&TripleDot;": { "codepoints": [8411], "characters": "\u20DB" },+  "&Tscr;": { "codepoints": [119983], "characters": "\uD835\uDCAF" },+  "&Tstrok;": { "codepoints": [358], "characters": "\u0166" },+  "&Uacute": { "codepoints": [218], "characters": "\u00DA" },+  "&Uacute;": { "codepoints": [218], "characters": "\u00DA" },+  "&Uarr;": { "codepoints": [8607], "characters": "\u219F" },+  "&Uarrocir;": { "codepoints": [10569], "characters": "\u2949" },+  "&Ubrcy;": { "codepoints": [1038], "characters": "\u040E" },+  "&Ubreve;": { "codepoints": [364], "characters": "\u016C" },+  "&Ucirc": { "codepoints": [219], "characters": "\u00DB" },+  "&Ucirc;": { "codepoints": [219], "characters": "\u00DB" },+  "&Ucy;": { "codepoints": [1059], "characters": "\u0423" },+  "&Udblac;": { "codepoints": [368], "characters": "\u0170" },+  "&Ufr;": { "codepoints": [120088], "characters": "\uD835\uDD18" },+  "&Ugrave": { "codepoints": [217], "characters": "\u00D9" },+  "&Ugrave;": { "codepoints": [217], "characters": "\u00D9" },+  "&Umacr;": { "codepoints": [362], "characters": "\u016A" },+  "&UnderBar;": { "codepoints": [95], "characters": "\u005F" },+  "&UnderBrace;": { "codepoints": [9183], "characters": "\u23DF" },+  "&UnderBracket;": { "codepoints": [9141], "characters": "\u23B5" },+  "&UnderParenthesis;": { "codepoints": [9181], "characters": "\u23DD" },+  "&Union;": { "codepoints": [8899], "characters": "\u22C3" },+  "&UnionPlus;": { "codepoints": [8846], "characters": "\u228E" },+  "&Uogon;": { "codepoints": [370], "characters": "\u0172" },+  "&Uopf;": { "codepoints": [120140], "characters": "\uD835\uDD4C" },+  "&UpArrow;": { "codepoints": [8593], "characters": "\u2191" },+  "&UpArrowBar;": { "codepoints": [10514], "characters": "\u2912" },+  "&UpArrowDownArrow;": { "codepoints": [8645], "characters": "\u21C5" },+  "&UpDownArrow;": { "codepoints": [8597], "characters": "\u2195" },+  "&UpEquilibrium;": { "codepoints": [10606], "characters": "\u296E" },+  "&UpTee;": { "codepoints": [8869], "characters": "\u22A5" },+  "&UpTeeArrow;": { "codepoints": [8613], "characters": "\u21A5" },+  "&Uparrow;": { "codepoints": [8657], "characters": "\u21D1" },+  "&Updownarrow;": { "codepoints": [8661], "characters": "\u21D5" },+  "&UpperLeftArrow;": { "codepoints": [8598], "characters": "\u2196" },+  "&UpperRightArrow;": { "codepoints": [8599], "characters": "\u2197" },+  "&Upsi;": { "codepoints": [978], "characters": "\u03D2" },+  "&Upsilon;": { "codepoints": [933], "characters": "\u03A5" },+  "&Uring;": { "codepoints": [366], "characters": "\u016E" },+  "&Uscr;": { "codepoints": [119984], "characters": "\uD835\uDCB0" },+  "&Utilde;": { "codepoints": [360], "characters": "\u0168" },+  "&Uuml": { "codepoints": [220], "characters": "\u00DC" },+  "&Uuml;": { "codepoints": [220], "characters": "\u00DC" },+  "&VDash;": { "codepoints": [8875], "characters": "\u22AB" },+  "&Vbar;": { "codepoints": [10987], "characters": "\u2AEB" },+  "&Vcy;": { "codepoints": [1042], "characters": "\u0412" },+  "&Vdash;": { "codepoints": [8873], "characters": "\u22A9" },+  "&Vdashl;": { "codepoints": [10982], "characters": "\u2AE6" },+  "&Vee;": { "codepoints": [8897], "characters": "\u22C1" },+  "&Verbar;": { "codepoints": [8214], "characters": "\u2016" },+  "&Vert;": { "codepoints": [8214], "characters": "\u2016" },+  "&VerticalBar;": { "codepoints": [8739], "characters": "\u2223" },+  "&VerticalLine;": { "codepoints": [124], "characters": "\u007C" },+  "&VerticalSeparator;": { "codepoints": [10072], "characters": "\u2758" },+  "&VerticalTilde;": { "codepoints": [8768], "characters": "\u2240" },+  "&VeryThinSpace;": { "codepoints": [8202], "characters": "\u200A" },+  "&Vfr;": { "codepoints": [120089], "characters": "\uD835\uDD19" },+  "&Vopf;": { "codepoints": [120141], "characters": "\uD835\uDD4D" },+  "&Vscr;": { "codepoints": [119985], "characters": "\uD835\uDCB1" },+  "&Vvdash;": { "codepoints": [8874], "characters": "\u22AA" },+  "&Wcirc;": { "codepoints": [372], "characters": "\u0174" },+  "&Wedge;": { "codepoints": [8896], "characters": "\u22C0" },+  "&Wfr;": { "codepoints": [120090], "characters": "\uD835\uDD1A" },+  "&Wopf;": { "codepoints": [120142], "characters": "\uD835\uDD4E" },+  "&Wscr;": { "codepoints": [119986], "characters": "\uD835\uDCB2" },+  "&Xfr;": { "codepoints": [120091], "characters": "\uD835\uDD1B" },+  "&Xi;": { "codepoints": [926], "characters": "\u039E" },+  "&Xopf;": { "codepoints": [120143], "characters": "\uD835\uDD4F" },+  "&Xscr;": { "codepoints": [119987], "characters": "\uD835\uDCB3" },+  "&YAcy;": { "codepoints": [1071], "characters": "\u042F" },+  "&YIcy;": { "codepoints": [1031], "characters": "\u0407" },+  "&YUcy;": { "codepoints": [1070], "characters": "\u042E" },+  "&Yacute": { "codepoints": [221], "characters": "\u00DD" },+  "&Yacute;": { "codepoints": [221], "characters": "\u00DD" },+  "&Ycirc;": { "codepoints": [374], "characters": "\u0176" },+  "&Ycy;": { "codepoints": [1067], "characters": "\u042B" },+  "&Yfr;": { "codepoints": [120092], "characters": "\uD835\uDD1C" },+  "&Yopf;": { "codepoints": [120144], "characters": "\uD835\uDD50" },+  "&Yscr;": { "codepoints": [119988], "characters": "\uD835\uDCB4" },+  "&Yuml;": { "codepoints": [376], "characters": "\u0178" },+  "&ZHcy;": { "codepoints": [1046], "characters": "\u0416" },+  "&Zacute;": { "codepoints": [377], "characters": "\u0179" },+  "&Zcaron;": { "codepoints": [381], "characters": "\u017D" },+  "&Zcy;": { "codepoints": [1047], "characters": "\u0417" },+  "&Zdot;": { "codepoints": [379], "characters": "\u017B" },+  "&ZeroWidthSpace;": { "codepoints": [8203], "characters": "\u200B" },+  "&Zeta;": { "codepoints": [918], "characters": "\u0396" },+  "&Zfr;": { "codepoints": [8488], "characters": "\u2128" },+  "&Zopf;": { "codepoints": [8484], "characters": "\u2124" },+  "&Zscr;": { "codepoints": [119989], "characters": "\uD835\uDCB5" },+  "&aacute": { "codepoints": [225], "characters": "\u00E1" },+  "&aacute;": { "codepoints": [225], "characters": "\u00E1" },+  "&abreve;": { "codepoints": [259], "characters": "\u0103" },+  "&ac;": { "codepoints": [8766], "characters": "\u223E" },+  "&acE;": { "codepoints": [8766, 819], "characters": "\u223E\u0333" },+  "&acd;": { "codepoints": [8767], "characters": "\u223F" },+  "&acirc": { "codepoints": [226], "characters": "\u00E2" },+  "&acirc;": { "codepoints": [226], "characters": "\u00E2" },+  "&acute": { "codepoints": [180], "characters": "\u00B4" },+  "&acute;": { "codepoints": [180], "characters": "\u00B4" },+  "&acy;": { "codepoints": [1072], "characters": "\u0430" },+  "&aelig": { "codepoints": [230], "characters": "\u00E6" },+  "&aelig;": { "codepoints": [230], "characters": "\u00E6" },+  "&af;": { "codepoints": [8289], "characters": "\u2061" },+  "&afr;": { "codepoints": [120094], "characters": "\uD835\uDD1E" },+  "&agrave": { "codepoints": [224], "characters": "\u00E0" },+  "&agrave;": { "codepoints": [224], "characters": "\u00E0" },+  "&alefsym;": { "codepoints": [8501], "characters": "\u2135" },+  "&aleph;": { "codepoints": [8501], "characters": "\u2135" },+  "&alpha;": { "codepoints": [945], "characters": "\u03B1" },+  "&amacr;": { "codepoints": [257], "characters": "\u0101" },+  "&amalg;": { "codepoints": [10815], "characters": "\u2A3F" },+  "&amp": { "codepoints": [38], "characters": "\u0026" },+  "&amp;": { "codepoints": [38], "characters": "\u0026" },+  "&and;": { "codepoints": [8743], "characters": "\u2227" },+  "&andand;": { "codepoints": [10837], "characters": "\u2A55" },+  "&andd;": { "codepoints": [10844], "characters": "\u2A5C" },+  "&andslope;": { "codepoints": [10840], "characters": "\u2A58" },+  "&andv;": { "codepoints": [10842], "characters": "\u2A5A" },+  "&ang;": { "codepoints": [8736], "characters": "\u2220" },+  "&ange;": { "codepoints": [10660], "characters": "\u29A4" },+  "&angle;": { "codepoints": [8736], "characters": "\u2220" },+  "&angmsd;": { "codepoints": [8737], "characters": "\u2221" },+  "&angmsdaa;": { "codepoints": [10664], "characters": "\u29A8" },+  "&angmsdab;": { "codepoints": [10665], "characters": "\u29A9" },+  "&angmsdac;": { "codepoints": [10666], "characters": "\u29AA" },+  "&angmsdad;": { "codepoints": [10667], "characters": "\u29AB" },+  "&angmsdae;": { "codepoints": [10668], "characters": "\u29AC" },+  "&angmsdaf;": { "codepoints": [10669], "characters": "\u29AD" },+  "&angmsdag;": { "codepoints": [10670], "characters": "\u29AE" },+  "&angmsdah;": { "codepoints": [10671], "characters": "\u29AF" },+  "&angrt;": { "codepoints": [8735], "characters": "\u221F" },+  "&angrtvb;": { "codepoints": [8894], "characters": "\u22BE" },+  "&angrtvbd;": { "codepoints": [10653], "characters": "\u299D" },+  "&angsph;": { "codepoints": [8738], "characters": "\u2222" },+  "&angst;": { "codepoints": [197], "characters": "\u00C5" },+  "&angzarr;": { "codepoints": [9084], "characters": "\u237C" },+  "&aogon;": { "codepoints": [261], "characters": "\u0105" },+  "&aopf;": { "codepoints": [120146], "characters": "\uD835\uDD52" },+  "&ap;": { "codepoints": [8776], "characters": "\u2248" },+  "&apE;": { "codepoints": [10864], "characters": "\u2A70" },+  "&apacir;": { "codepoints": [10863], "characters": "\u2A6F" },+  "&ape;": { "codepoints": [8778], "characters": "\u224A" },+  "&apid;": { "codepoints": [8779], "characters": "\u224B" },+  "&apos;": { "codepoints": [39], "characters": "\u0027" },+  "&approx;": { "codepoints": [8776], "characters": "\u2248" },+  "&approxeq;": { "codepoints": [8778], "characters": "\u224A" },+  "&aring": { "codepoints": [229], "characters": "\u00E5" },+  "&aring;": { "codepoints": [229], "characters": "\u00E5" },+  "&ascr;": { "codepoints": [119990], "characters": "\uD835\uDCB6" },+  "&ast;": { "codepoints": [42], "characters": "\u002A" },+  "&asymp;": { "codepoints": [8776], "characters": "\u2248" },+  "&asympeq;": { "codepoints": [8781], "characters": "\u224D" },+  "&atilde": { "codepoints": [227], "characters": "\u00E3" },+  "&atilde;": { "codepoints": [227], "characters": "\u00E3" },+  "&auml": { "codepoints": [228], "characters": "\u00E4" },+  "&auml;": { "codepoints": [228], "characters": "\u00E4" },+  "&awconint;": { "codepoints": [8755], "characters": "\u2233" },+  "&awint;": { "codepoints": [10769], "characters": "\u2A11" },+  "&bNot;": { "codepoints": [10989], "characters": "\u2AED" },+  "&backcong;": { "codepoints": [8780], "characters": "\u224C" },+  "&backepsilon;": { "codepoints": [1014], "characters": "\u03F6" },+  "&backprime;": { "codepoints": [8245], "characters": "\u2035" },+  "&backsim;": { "codepoints": [8765], "characters": "\u223D" },+  "&backsimeq;": { "codepoints": [8909], "characters": "\u22CD" },+  "&barvee;": { "codepoints": [8893], "characters": "\u22BD" },+  "&barwed;": { "codepoints": [8965], "characters": "\u2305" },+  "&barwedge;": { "codepoints": [8965], "characters": "\u2305" },+  "&bbrk;": { "codepoints": [9141], "characters": "\u23B5" },+  "&bbrktbrk;": { "codepoints": [9142], "characters": "\u23B6" },+  "&bcong;": { "codepoints": [8780], "characters": "\u224C" },+  "&bcy;": { "codepoints": [1073], "characters": "\u0431" },+  "&bdquo;": { "codepoints": [8222], "characters": "\u201E" },+  "&becaus;": { "codepoints": [8757], "characters": "\u2235" },+  "&because;": { "codepoints": [8757], "characters": "\u2235" },+  "&bemptyv;": { "codepoints": [10672], "characters": "\u29B0" },+  "&bepsi;": { "codepoints": [1014], "characters": "\u03F6" },+  "&bernou;": { "codepoints": [8492], "characters": "\u212C" },+  "&beta;": { "codepoints": [946], "characters": "\u03B2" },+  "&beth;": { "codepoints": [8502], "characters": "\u2136" },+  "&between;": { "codepoints": [8812], "characters": "\u226C" },+  "&bfr;": { "codepoints": [120095], "characters": "\uD835\uDD1F" },+  "&bigcap;": { "codepoints": [8898], "characters": "\u22C2" },+  "&bigcirc;": { "codepoints": [9711], "characters": "\u25EF" },+  "&bigcup;": { "codepoints": [8899], "characters": "\u22C3" },+  "&bigodot;": { "codepoints": [10752], "characters": "\u2A00" },+  "&bigoplus;": { "codepoints": [10753], "characters": "\u2A01" },+  "&bigotimes;": { "codepoints": [10754], "characters": "\u2A02" },+  "&bigsqcup;": { "codepoints": [10758], "characters": "\u2A06" },+  "&bigstar;": { "codepoints": [9733], "characters": "\u2605" },+  "&bigtriangledown;": { "codepoints": [9661], "characters": "\u25BD" },+  "&bigtriangleup;": { "codepoints": [9651], "characters": "\u25B3" },+  "&biguplus;": { "codepoints": [10756], "characters": "\u2A04" },+  "&bigvee;": { "codepoints": [8897], "characters": "\u22C1" },+  "&bigwedge;": { "codepoints": [8896], "characters": "\u22C0" },+  "&bkarow;": { "codepoints": [10509], "characters": "\u290D" },+  "&blacklozenge;": { "codepoints": [10731], "characters": "\u29EB" },+  "&blacksquare;": { "codepoints": [9642], "characters": "\u25AA" },+  "&blacktriangle;": { "codepoints": [9652], "characters": "\u25B4" },+  "&blacktriangledown;": { "codepoints": [9662], "characters": "\u25BE" },+  "&blacktriangleleft;": { "codepoints": [9666], "characters": "\u25C2" },+  "&blacktriangleright;": { "codepoints": [9656], "characters": "\u25B8" },+  "&blank;": { "codepoints": [9251], "characters": "\u2423" },+  "&blk12;": { "codepoints": [9618], "characters": "\u2592" },+  "&blk14;": { "codepoints": [9617], "characters": "\u2591" },+  "&blk34;": { "codepoints": [9619], "characters": "\u2593" },+  "&block;": { "codepoints": [9608], "characters": "\u2588" },+  "&bne;": { "codepoints": [61, 8421], "characters": "\u003D\u20E5" },+  "&bnequiv;": { "codepoints": [8801, 8421], "characters": "\u2261\u20E5" },+  "&bnot;": { "codepoints": [8976], "characters": "\u2310" },+  "&bopf;": { "codepoints": [120147], "characters": "\uD835\uDD53" },+  "&bot;": { "codepoints": [8869], "characters": "\u22A5" },+  "&bottom;": { "codepoints": [8869], "characters": "\u22A5" },+  "&bowtie;": { "codepoints": [8904], "characters": "\u22C8" },+  "&boxDL;": { "codepoints": [9559], "characters": "\u2557" },+  "&boxDR;": { "codepoints": [9556], "characters": "\u2554" },+  "&boxDl;": { "codepoints": [9558], "characters": "\u2556" },+  "&boxDr;": { "codepoints": [9555], "characters": "\u2553" },+  "&boxH;": { "codepoints": [9552], "characters": "\u2550" },+  "&boxHD;": { "codepoints": [9574], "characters": "\u2566" },+  "&boxHU;": { "codepoints": [9577], "characters": "\u2569" },+  "&boxHd;": { "codepoints": [9572], "characters": "\u2564" },+  "&boxHu;": { "codepoints": [9575], "characters": "\u2567" },+  "&boxUL;": { "codepoints": [9565], "characters": "\u255D" },+  "&boxUR;": { "codepoints": [9562], "characters": "\u255A" },+  "&boxUl;": { "codepoints": [9564], "characters": "\u255C" },+  "&boxUr;": { "codepoints": [9561], "characters": "\u2559" },+  "&boxV;": { "codepoints": [9553], "characters": "\u2551" },+  "&boxVH;": { "codepoints": [9580], "characters": "\u256C" },+  "&boxVL;": { "codepoints": [9571], "characters": "\u2563" },+  "&boxVR;": { "codepoints": [9568], "characters": "\u2560" },+  "&boxVh;": { "codepoints": [9579], "characters": "\u256B" },+  "&boxVl;": { "codepoints": [9570], "characters": "\u2562" },+  "&boxVr;": { "codepoints": [9567], "characters": "\u255F" },+  "&boxbox;": { "codepoints": [10697], "characters": "\u29C9" },+  "&boxdL;": { "codepoints": [9557], "characters": "\u2555" },+  "&boxdR;": { "codepoints": [9554], "characters": "\u2552" },+  "&boxdl;": { "codepoints": [9488], "characters": "\u2510" },+  "&boxdr;": { "codepoints": [9484], "characters": "\u250C" },+  "&boxh;": { "codepoints": [9472], "characters": "\u2500" },+  "&boxhD;": { "codepoints": [9573], "characters": "\u2565" },+  "&boxhU;": { "codepoints": [9576], "characters": "\u2568" },+  "&boxhd;": { "codepoints": [9516], "characters": "\u252C" },+  "&boxhu;": { "codepoints": [9524], "characters": "\u2534" },+  "&boxminus;": { "codepoints": [8863], "characters": "\u229F" },+  "&boxplus;": { "codepoints": [8862], "characters": "\u229E" },+  "&boxtimes;": { "codepoints": [8864], "characters": "\u22A0" },+  "&boxuL;": { "codepoints": [9563], "characters": "\u255B" },+  "&boxuR;": { "codepoints": [9560], "characters": "\u2558" },+  "&boxul;": { "codepoints": [9496], "characters": "\u2518" },+  "&boxur;": { "codepoints": [9492], "characters": "\u2514" },+  "&boxv;": { "codepoints": [9474], "characters": "\u2502" },+  "&boxvH;": { "codepoints": [9578], "characters": "\u256A" },+  "&boxvL;": { "codepoints": [9569], "characters": "\u2561" },+  "&boxvR;": { "codepoints": [9566], "characters": "\u255E" },+  "&boxvh;": { "codepoints": [9532], "characters": "\u253C" },+  "&boxvl;": { "codepoints": [9508], "characters": "\u2524" },+  "&boxvr;": { "codepoints": [9500], "characters": "\u251C" },+  "&bprime;": { "codepoints": [8245], "characters": "\u2035" },+  "&breve;": { "codepoints": [728], "characters": "\u02D8" },+  "&brvbar": { "codepoints": [166], "characters": "\u00A6" },+  "&brvbar;": { "codepoints": [166], "characters": "\u00A6" },+  "&bscr;": { "codepoints": [119991], "characters": "\uD835\uDCB7" },+  "&bsemi;": { "codepoints": [8271], "characters": "\u204F" },+  "&bsim;": { "codepoints": [8765], "characters": "\u223D" },+  "&bsime;": { "codepoints": [8909], "characters": "\u22CD" },+  "&bsol;": { "codepoints": [92], "characters": "\u005C" },+  "&bsolb;": { "codepoints": [10693], "characters": "\u29C5" },+  "&bsolhsub;": { "codepoints": [10184], "characters": "\u27C8" },+  "&bull;": { "codepoints": [8226], "characters": "\u2022" },+  "&bullet;": { "codepoints": [8226], "characters": "\u2022" },+  "&bump;": { "codepoints": [8782], "characters": "\u224E" },+  "&bumpE;": { "codepoints": [10926], "characters": "\u2AAE" },+  "&bumpe;": { "codepoints": [8783], "characters": "\u224F" },+  "&bumpeq;": { "codepoints": [8783], "characters": "\u224F" },+  "&cacute;": { "codepoints": [263], "characters": "\u0107" },+  "&cap;": { "codepoints": [8745], "characters": "\u2229" },+  "&capand;": { "codepoints": [10820], "characters": "\u2A44" },+  "&capbrcup;": { "codepoints": [10825], "characters": "\u2A49" },+  "&capcap;": { "codepoints": [10827], "characters": "\u2A4B" },+  "&capcup;": { "codepoints": [10823], "characters": "\u2A47" },+  "&capdot;": { "codepoints": [10816], "characters": "\u2A40" },+  "&caps;": { "codepoints": [8745, 65024], "characters": "\u2229\uFE00" },+  "&caret;": { "codepoints": [8257], "characters": "\u2041" },+  "&caron;": { "codepoints": [711], "characters": "\u02C7" },+  "&ccaps;": { "codepoints": [10829], "characters": "\u2A4D" },+  "&ccaron;": { "codepoints": [269], "characters": "\u010D" },+  "&ccedil": { "codepoints": [231], "characters": "\u00E7" },+  "&ccedil;": { "codepoints": [231], "characters": "\u00E7" },+  "&ccirc;": { "codepoints": [265], "characters": "\u0109" },+  "&ccups;": { "codepoints": [10828], "characters": "\u2A4C" },+  "&ccupssm;": { "codepoints": [10832], "characters": "\u2A50" },+  "&cdot;": { "codepoints": [267], "characters": "\u010B" },+  "&cedil": { "codepoints": [184], "characters": "\u00B8" },+  "&cedil;": { "codepoints": [184], "characters": "\u00B8" },+  "&cemptyv;": { "codepoints": [10674], "characters": "\u29B2" },+  "&cent": { "codepoints": [162], "characters": "\u00A2" },+  "&cent;": { "codepoints": [162], "characters": "\u00A2" },+  "&centerdot;": { "codepoints": [183], "characters": "\u00B7" },+  "&cfr;": { "codepoints": [120096], "characters": "\uD835\uDD20" },+  "&chcy;": { "codepoints": [1095], "characters": "\u0447" },+  "&check;": { "codepoints": [10003], "characters": "\u2713" },+  "&checkmark;": { "codepoints": [10003], "characters": "\u2713" },+  "&chi;": { "codepoints": [967], "characters": "\u03C7" },+  "&cir;": { "codepoints": [9675], "characters": "\u25CB" },+  "&cirE;": { "codepoints": [10691], "characters": "\u29C3" },+  "&circ;": { "codepoints": [710], "characters": "\u02C6" },+  "&circeq;": { "codepoints": [8791], "characters": "\u2257" },+  "&circlearrowleft;": { "codepoints": [8634], "characters": "\u21BA" },+  "&circlearrowright;": { "codepoints": [8635], "characters": "\u21BB" },+  "&circledR;": { "codepoints": [174], "characters": "\u00AE" },+  "&circledS;": { "codepoints": [9416], "characters": "\u24C8" },+  "&circledast;": { "codepoints": [8859], "characters": "\u229B" },+  "&circledcirc;": { "codepoints": [8858], "characters": "\u229A" },+  "&circleddash;": { "codepoints": [8861], "characters": "\u229D" },+  "&cire;": { "codepoints": [8791], "characters": "\u2257" },+  "&cirfnint;": { "codepoints": [10768], "characters": "\u2A10" },+  "&cirmid;": { "codepoints": [10991], "characters": "\u2AEF" },+  "&cirscir;": { "codepoints": [10690], "characters": "\u29C2" },+  "&clubs;": { "codepoints": [9827], "characters": "\u2663" },+  "&clubsuit;": { "codepoints": [9827], "characters": "\u2663" },+  "&colon;": { "codepoints": [58], "characters": "\u003A" },+  "&colone;": { "codepoints": [8788], "characters": "\u2254" },+  "&coloneq;": { "codepoints": [8788], "characters": "\u2254" },+  "&comma;": { "codepoints": [44], "characters": "\u002C" },+  "&commat;": { "codepoints": [64], "characters": "\u0040" },+  "&comp;": { "codepoints": [8705], "characters": "\u2201" },+  "&compfn;": { "codepoints": [8728], "characters": "\u2218" },+  "&complement;": { "codepoints": [8705], "characters": "\u2201" },+  "&complexes;": { "codepoints": [8450], "characters": "\u2102" },+  "&cong;": { "codepoints": [8773], "characters": "\u2245" },+  "&congdot;": { "codepoints": [10861], "characters": "\u2A6D" },+  "&conint;": { "codepoints": [8750], "characters": "\u222E" },+  "&copf;": { "codepoints": [120148], "characters": "\uD835\uDD54" },+  "&coprod;": { "codepoints": [8720], "characters": "\u2210" },+  "&copy": { "codepoints": [169], "characters": "\u00A9" },+  "&copy;": { "codepoints": [169], "characters": "\u00A9" },+  "&copysr;": { "codepoints": [8471], "characters": "\u2117" },+  "&crarr;": { "codepoints": [8629], "characters": "\u21B5" },+  "&cross;": { "codepoints": [10007], "characters": "\u2717" },+  "&cscr;": { "codepoints": [119992], "characters": "\uD835\uDCB8" },+  "&csub;": { "codepoints": [10959], "characters": "\u2ACF" },+  "&csube;": { "codepoints": [10961], "characters": "\u2AD1" },+  "&csup;": { "codepoints": [10960], "characters": "\u2AD0" },+  "&csupe;": { "codepoints": [10962], "characters": "\u2AD2" },+  "&ctdot;": { "codepoints": [8943], "characters": "\u22EF" },+  "&cudarrl;": { "codepoints": [10552], "characters": "\u2938" },+  "&cudarrr;": { "codepoints": [10549], "characters": "\u2935" },+  "&cuepr;": { "codepoints": [8926], "characters": "\u22DE" },+  "&cuesc;": { "codepoints": [8927], "characters": "\u22DF" },+  "&cularr;": { "codepoints": [8630], "characters": "\u21B6" },+  "&cularrp;": { "codepoints": [10557], "characters": "\u293D" },+  "&cup;": { "codepoints": [8746], "characters": "\u222A" },+  "&cupbrcap;": { "codepoints": [10824], "characters": "\u2A48" },+  "&cupcap;": { "codepoints": [10822], "characters": "\u2A46" },+  "&cupcup;": { "codepoints": [10826], "characters": "\u2A4A" },+  "&cupdot;": { "codepoints": [8845], "characters": "\u228D" },+  "&cupor;": { "codepoints": [10821], "characters": "\u2A45" },+  "&cups;": { "codepoints": [8746, 65024], "characters": "\u222A\uFE00" },+  "&curarr;": { "codepoints": [8631], "characters": "\u21B7" },+  "&curarrm;": { "codepoints": [10556], "characters": "\u293C" },+  "&curlyeqprec;": { "codepoints": [8926], "characters": "\u22DE" },+  "&curlyeqsucc;": { "codepoints": [8927], "characters": "\u22DF" },+  "&curlyvee;": { "codepoints": [8910], "characters": "\u22CE" },+  "&curlywedge;": { "codepoints": [8911], "characters": "\u22CF" },+  "&curren": { "codepoints": [164], "characters": "\u00A4" },+  "&curren;": { "codepoints": [164], "characters": "\u00A4" },+  "&curvearrowleft;": { "codepoints": [8630], "characters": "\u21B6" },+  "&curvearrowright;": { "codepoints": [8631], "characters": "\u21B7" },+  "&cuvee;": { "codepoints": [8910], "characters": "\u22CE" },+  "&cuwed;": { "codepoints": [8911], "characters": "\u22CF" },+  "&cwconint;": { "codepoints": [8754], "characters": "\u2232" },+  "&cwint;": { "codepoints": [8753], "characters": "\u2231" },+  "&cylcty;": { "codepoints": [9005], "characters": "\u232D" },+  "&dArr;": { "codepoints": [8659], "characters": "\u21D3" },+  "&dHar;": { "codepoints": [10597], "characters": "\u2965" },+  "&dagger;": { "codepoints": [8224], "characters": "\u2020" },+  "&daleth;": { "codepoints": [8504], "characters": "\u2138" },+  "&darr;": { "codepoints": [8595], "characters": "\u2193" },+  "&dash;": { "codepoints": [8208], "characters": "\u2010" },+  "&dashv;": { "codepoints": [8867], "characters": "\u22A3" },+  "&dbkarow;": { "codepoints": [10511], "characters": "\u290F" },+  "&dblac;": { "codepoints": [733], "characters": "\u02DD" },+  "&dcaron;": { "codepoints": [271], "characters": "\u010F" },+  "&dcy;": { "codepoints": [1076], "characters": "\u0434" },+  "&dd;": { "codepoints": [8518], "characters": "\u2146" },+  "&ddagger;": { "codepoints": [8225], "characters": "\u2021" },+  "&ddarr;": { "codepoints": [8650], "characters": "\u21CA" },+  "&ddotseq;": { "codepoints": [10871], "characters": "\u2A77" },+  "&deg": { "codepoints": [176], "characters": "\u00B0" },+  "&deg;": { "codepoints": [176], "characters": "\u00B0" },+  "&delta;": { "codepoints": [948], "characters": "\u03B4" },+  "&demptyv;": { "codepoints": [10673], "characters": "\u29B1" },+  "&dfisht;": { "codepoints": [10623], "characters": "\u297F" },+  "&dfr;": { "codepoints": [120097], "characters": "\uD835\uDD21" },+  "&dharl;": { "codepoints": [8643], "characters": "\u21C3" },+  "&dharr;": { "codepoints": [8642], "characters": "\u21C2" },+  "&diam;": { "codepoints": [8900], "characters": "\u22C4" },+  "&diamond;": { "codepoints": [8900], "characters": "\u22C4" },+  "&diamondsuit;": { "codepoints": [9830], "characters": "\u2666" },+  "&diams;": { "codepoints": [9830], "characters": "\u2666" },+  "&die;": { "codepoints": [168], "characters": "\u00A8" },+  "&digamma;": { "codepoints": [989], "characters": "\u03DD" },+  "&disin;": { "codepoints": [8946], "characters": "\u22F2" },+  "&div;": { "codepoints": [247], "characters": "\u00F7" },+  "&divide": { "codepoints": [247], "characters": "\u00F7" },+  "&divide;": { "codepoints": [247], "characters": "\u00F7" },+  "&divideontimes;": { "codepoints": [8903], "characters": "\u22C7" },+  "&divonx;": { "codepoints": [8903], "characters": "\u22C7" },+  "&djcy;": { "codepoints": [1106], "characters": "\u0452" },+  "&dlcorn;": { "codepoints": [8990], "characters": "\u231E" },+  "&dlcrop;": { "codepoints": [8973], "characters": "\u230D" },+  "&dollar;": { "codepoints": [36], "characters": "\u0024" },+  "&dopf;": { "codepoints": [120149], "characters": "\uD835\uDD55" },+  "&dot;": { "codepoints": [729], "characters": "\u02D9" },+  "&doteq;": { "codepoints": [8784], "characters": "\u2250" },+  "&doteqdot;": { "codepoints": [8785], "characters": "\u2251" },+  "&dotminus;": { "codepoints": [8760], "characters": "\u2238" },+  "&dotplus;": { "codepoints": [8724], "characters": "\u2214" },+  "&dotsquare;": { "codepoints": [8865], "characters": "\u22A1" },+  "&doublebarwedge;": { "codepoints": [8966], "characters": "\u2306" },+  "&downarrow;": { "codepoints": [8595], "characters": "\u2193" },+  "&downdownarrows;": { "codepoints": [8650], "characters": "\u21CA" },+  "&downharpoonleft;": { "codepoints": [8643], "characters": "\u21C3" },+  "&downharpoonright;": { "codepoints": [8642], "characters": "\u21C2" },+  "&drbkarow;": { "codepoints": [10512], "characters": "\u2910" },+  "&drcorn;": { "codepoints": [8991], "characters": "\u231F" },+  "&drcrop;": { "codepoints": [8972], "characters": "\u230C" },+  "&dscr;": { "codepoints": [119993], "characters": "\uD835\uDCB9" },+  "&dscy;": { "codepoints": [1109], "characters": "\u0455" },+  "&dsol;": { "codepoints": [10742], "characters": "\u29F6" },+  "&dstrok;": { "codepoints": [273], "characters": "\u0111" },+  "&dtdot;": { "codepoints": [8945], "characters": "\u22F1" },+  "&dtri;": { "codepoints": [9663], "characters": "\u25BF" },+  "&dtrif;": { "codepoints": [9662], "characters": "\u25BE" },+  "&duarr;": { "codepoints": [8693], "characters": "\u21F5" },+  "&duhar;": { "codepoints": [10607], "characters": "\u296F" },+  "&dwangle;": { "codepoints": [10662], "characters": "\u29A6" },+  "&dzcy;": { "codepoints": [1119], "characters": "\u045F" },+  "&dzigrarr;": { "codepoints": [10239], "characters": "\u27FF" },+  "&eDDot;": { "codepoints": [10871], "characters": "\u2A77" },+  "&eDot;": { "codepoints": [8785], "characters": "\u2251" },+  "&eacute": { "codepoints": [233], "characters": "\u00E9" },+  "&eacute;": { "codepoints": [233], "characters": "\u00E9" },+  "&easter;": { "codepoints": [10862], "characters": "\u2A6E" },+  "&ecaron;": { "codepoints": [283], "characters": "\u011B" },+  "&ecir;": { "codepoints": [8790], "characters": "\u2256" },+  "&ecirc": { "codepoints": [234], "characters": "\u00EA" },+  "&ecirc;": { "codepoints": [234], "characters": "\u00EA" },+  "&ecolon;": { "codepoints": [8789], "characters": "\u2255" },+  "&ecy;": { "codepoints": [1101], "characters": "\u044D" },+  "&edot;": { "codepoints": [279], "characters": "\u0117" },+  "&ee;": { "codepoints": [8519], "characters": "\u2147" },+  "&efDot;": { "codepoints": [8786], "characters": "\u2252" },+  "&efr;": { "codepoints": [120098], "characters": "\uD835\uDD22" },+  "&eg;": { "codepoints": [10906], "characters": "\u2A9A" },+  "&egrave": { "codepoints": [232], "characters": "\u00E8" },+  "&egrave;": { "codepoints": [232], "characters": "\u00E8" },+  "&egs;": { "codepoints": [10902], "characters": "\u2A96" },+  "&egsdot;": { "codepoints": [10904], "characters": "\u2A98" },+  "&el;": { "codepoints": [10905], "characters": "\u2A99" },+  "&elinters;": { "codepoints": [9191], "characters": "\u23E7" },+  "&ell;": { "codepoints": [8467], "characters": "\u2113" },+  "&els;": { "codepoints": [10901], "characters": "\u2A95" },+  "&elsdot;": { "codepoints": [10903], "characters": "\u2A97" },+  "&emacr;": { "codepoints": [275], "characters": "\u0113" },+  "&empty;": { "codepoints": [8709], "characters": "\u2205" },+  "&emptyset;": { "codepoints": [8709], "characters": "\u2205" },+  "&emptyv;": { "codepoints": [8709], "characters": "\u2205" },+  "&emsp13;": { "codepoints": [8196], "characters": "\u2004" },+  "&emsp14;": { "codepoints": [8197], "characters": "\u2005" },+  "&emsp;": { "codepoints": [8195], "characters": "\u2003" },+  "&eng;": { "codepoints": [331], "characters": "\u014B" },+  "&ensp;": { "codepoints": [8194], "characters": "\u2002" },+  "&eogon;": { "codepoints": [281], "characters": "\u0119" },+  "&eopf;": { "codepoints": [120150], "characters": "\uD835\uDD56" },+  "&epar;": { "codepoints": [8917], "characters": "\u22D5" },+  "&eparsl;": { "codepoints": [10723], "characters": "\u29E3" },+  "&eplus;": { "codepoints": [10865], "characters": "\u2A71" },+  "&epsi;": { "codepoints": [949], "characters": "\u03B5" },+  "&epsilon;": { "codepoints": [949], "characters": "\u03B5" },+  "&epsiv;": { "codepoints": [1013], "characters": "\u03F5" },+  "&eqcirc;": { "codepoints": [8790], "characters": "\u2256" },+  "&eqcolon;": { "codepoints": [8789], "characters": "\u2255" },+  "&eqsim;": { "codepoints": [8770], "characters": "\u2242" },+  "&eqslantgtr;": { "codepoints": [10902], "characters": "\u2A96" },+  "&eqslantless;": { "codepoints": [10901], "characters": "\u2A95" },+  "&equals;": { "codepoints": [61], "characters": "\u003D" },+  "&equest;": { "codepoints": [8799], "characters": "\u225F" },+  "&equiv;": { "codepoints": [8801], "characters": "\u2261" },+  "&equivDD;": { "codepoints": [10872], "characters": "\u2A78" },+  "&eqvparsl;": { "codepoints": [10725], "characters": "\u29E5" },+  "&erDot;": { "codepoints": [8787], "characters": "\u2253" },+  "&erarr;": { "codepoints": [10609], "characters": "\u2971" },+  "&escr;": { "codepoints": [8495], "characters": "\u212F" },+  "&esdot;": { "codepoints": [8784], "characters": "\u2250" },+  "&esim;": { "codepoints": [8770], "characters": "\u2242" },+  "&eta;": { "codepoints": [951], "characters": "\u03B7" },+  "&eth": { "codepoints": [240], "characters": "\u00F0" },+  "&eth;": { "codepoints": [240], "characters": "\u00F0" },+  "&euml": { "codepoints": [235], "characters": "\u00EB" },+  "&euml;": { "codepoints": [235], "characters": "\u00EB" },+  "&euro;": { "codepoints": [8364], "characters": "\u20AC" },+  "&excl;": { "codepoints": [33], "characters": "\u0021" },+  "&exist;": { "codepoints": [8707], "characters": "\u2203" },+  "&expectation;": { "codepoints": [8496], "characters": "\u2130" },+  "&exponentiale;": { "codepoints": [8519], "characters": "\u2147" },+  "&fallingdotseq;": { "codepoints": [8786], "characters": "\u2252" },+  "&fcy;": { "codepoints": [1092], "characters": "\u0444" },+  "&female;": { "codepoints": [9792], "characters": "\u2640" },+  "&ffilig;": { "codepoints": [64259], "characters": "\uFB03" },+  "&fflig;": { "codepoints": [64256], "characters": "\uFB00" },+  "&ffllig;": { "codepoints": [64260], "characters": "\uFB04" },+  "&ffr;": { "codepoints": [120099], "characters": "\uD835\uDD23" },+  "&filig;": { "codepoints": [64257], "characters": "\uFB01" },+  "&fjlig;": { "codepoints": [102, 106], "characters": "\u0066\u006A" },+  "&flat;": { "codepoints": [9837], "characters": "\u266D" },+  "&fllig;": { "codepoints": [64258], "characters": "\uFB02" },+  "&fltns;": { "codepoints": [9649], "characters": "\u25B1" },+  "&fnof;": { "codepoints": [402], "characters": "\u0192" },+  "&fopf;": { "codepoints": [120151], "characters": "\uD835\uDD57" },+  "&forall;": { "codepoints": [8704], "characters": "\u2200" },+  "&fork;": { "codepoints": [8916], "characters": "\u22D4" },+  "&forkv;": { "codepoints": [10969], "characters": "\u2AD9" },+  "&fpartint;": { "codepoints": [10765], "characters": "\u2A0D" },+  "&frac12": { "codepoints": [189], "characters": "\u00BD" },+  "&frac12;": { "codepoints": [189], "characters": "\u00BD" },+  "&frac13;": { "codepoints": [8531], "characters": "\u2153" },+  "&frac14": { "codepoints": [188], "characters": "\u00BC" },+  "&frac14;": { "codepoints": [188], "characters": "\u00BC" },+  "&frac15;": { "codepoints": [8533], "characters": "\u2155" },+  "&frac16;": { "codepoints": [8537], "characters": "\u2159" },+  "&frac18;": { "codepoints": [8539], "characters": "\u215B" },+  "&frac23;": { "codepoints": [8532], "characters": "\u2154" },+  "&frac25;": { "codepoints": [8534], "characters": "\u2156" },+  "&frac34": { "codepoints": [190], "characters": "\u00BE" },+  "&frac34;": { "codepoints": [190], "characters": "\u00BE" },+  "&frac35;": { "codepoints": [8535], "characters": "\u2157" },+  "&frac38;": { "codepoints": [8540], "characters": "\u215C" },+  "&frac45;": { "codepoints": [8536], "characters": "\u2158" },+  "&frac56;": { "codepoints": [8538], "characters": "\u215A" },+  "&frac58;": { "codepoints": [8541], "characters": "\u215D" },+  "&frac78;": { "codepoints": [8542], "characters": "\u215E" },+  "&frasl;": { "codepoints": [8260], "characters": "\u2044" },+  "&frown;": { "codepoints": [8994], "characters": "\u2322" },+  "&fscr;": { "codepoints": [119995], "characters": "\uD835\uDCBB" },+  "&gE;": { "codepoints": [8807], "characters": "\u2267" },+  "&gEl;": { "codepoints": [10892], "characters": "\u2A8C" },+  "&gacute;": { "codepoints": [501], "characters": "\u01F5" },+  "&gamma;": { "codepoints": [947], "characters": "\u03B3" },+  "&gammad;": { "codepoints": [989], "characters": "\u03DD" },+  "&gap;": { "codepoints": [10886], "characters": "\u2A86" },+  "&gbreve;": { "codepoints": [287], "characters": "\u011F" },+  "&gcirc;": { "codepoints": [285], "characters": "\u011D" },+  "&gcy;": { "codepoints": [1075], "characters": "\u0433" },+  "&gdot;": { "codepoints": [289], "characters": "\u0121" },+  "&ge;": { "codepoints": [8805], "characters": "\u2265" },+  "&gel;": { "codepoints": [8923], "characters": "\u22DB" },+  "&geq;": { "codepoints": [8805], "characters": "\u2265" },+  "&geqq;": { "codepoints": [8807], "characters": "\u2267" },+  "&geqslant;": { "codepoints": [10878], "characters": "\u2A7E" },+  "&ges;": { "codepoints": [10878], "characters": "\u2A7E" },+  "&gescc;": { "codepoints": [10921], "characters": "\u2AA9" },+  "&gesdot;": { "codepoints": [10880], "characters": "\u2A80" },+  "&gesdoto;": { "codepoints": [10882], "characters": "\u2A82" },+  "&gesdotol;": { "codepoints": [10884], "characters": "\u2A84" },+  "&gesl;": { "codepoints": [8923, 65024], "characters": "\u22DB\uFE00" },+  "&gesles;": { "codepoints": [10900], "characters": "\u2A94" },+  "&gfr;": { "codepoints": [120100], "characters": "\uD835\uDD24" },+  "&gg;": { "codepoints": [8811], "characters": "\u226B" },+  "&ggg;": { "codepoints": [8921], "characters": "\u22D9" },+  "&gimel;": { "codepoints": [8503], "characters": "\u2137" },+  "&gjcy;": { "codepoints": [1107], "characters": "\u0453" },+  "&gl;": { "codepoints": [8823], "characters": "\u2277" },+  "&glE;": { "codepoints": [10898], "characters": "\u2A92" },+  "&gla;": { "codepoints": [10917], "characters": "\u2AA5" },+  "&glj;": { "codepoints": [10916], "characters": "\u2AA4" },+  "&gnE;": { "codepoints": [8809], "characters": "\u2269" },+  "&gnap;": { "codepoints": [10890], "characters": "\u2A8A" },+  "&gnapprox;": { "codepoints": [10890], "characters": "\u2A8A" },+  "&gne;": { "codepoints": [10888], "characters": "\u2A88" },+  "&gneq;": { "codepoints": [10888], "characters": "\u2A88" },+  "&gneqq;": { "codepoints": [8809], "characters": "\u2269" },+  "&gnsim;": { "codepoints": [8935], "characters": "\u22E7" },+  "&gopf;": { "codepoints": [120152], "characters": "\uD835\uDD58" },+  "&grave;": { "codepoints": [96], "characters": "\u0060" },+  "&gscr;": { "codepoints": [8458], "characters": "\u210A" },+  "&gsim;": { "codepoints": [8819], "characters": "\u2273" },+  "&gsime;": { "codepoints": [10894], "characters": "\u2A8E" },+  "&gsiml;": { "codepoints": [10896], "characters": "\u2A90" },+  "&gt": { "codepoints": [62], "characters": "\u003E" },+  "&gt;": { "codepoints": [62], "characters": "\u003E" },+  "&gtcc;": { "codepoints": [10919], "characters": "\u2AA7" },+  "&gtcir;": { "codepoints": [10874], "characters": "\u2A7A" },+  "&gtdot;": { "codepoints": [8919], "characters": "\u22D7" },+  "&gtlPar;": { "codepoints": [10645], "characters": "\u2995" },+  "&gtquest;": { "codepoints": [10876], "characters": "\u2A7C" },+  "&gtrapprox;": { "codepoints": [10886], "characters": "\u2A86" },+  "&gtrarr;": { "codepoints": [10616], "characters": "\u2978" },+  "&gtrdot;": { "codepoints": [8919], "characters": "\u22D7" },+  "&gtreqless;": { "codepoints": [8923], "characters": "\u22DB" },+  "&gtreqqless;": { "codepoints": [10892], "characters": "\u2A8C" },+  "&gtrless;": { "codepoints": [8823], "characters": "\u2277" },+  "&gtrsim;": { "codepoints": [8819], "characters": "\u2273" },+  "&gvertneqq;": { "codepoints": [8809, 65024], "characters": "\u2269\uFE00" },+  "&gvnE;": { "codepoints": [8809, 65024], "characters": "\u2269\uFE00" },+  "&hArr;": { "codepoints": [8660], "characters": "\u21D4" },+  "&hairsp;": { "codepoints": [8202], "characters": "\u200A" },+  "&half;": { "codepoints": [189], "characters": "\u00BD" },+  "&hamilt;": { "codepoints": [8459], "characters": "\u210B" },+  "&hardcy;": { "codepoints": [1098], "characters": "\u044A" },+  "&harr;": { "codepoints": [8596], "characters": "\u2194" },+  "&harrcir;": { "codepoints": [10568], "characters": "\u2948" },+  "&harrw;": { "codepoints": [8621], "characters": "\u21AD" },+  "&hbar;": { "codepoints": [8463], "characters": "\u210F" },+  "&hcirc;": { "codepoints": [293], "characters": "\u0125" },+  "&hearts;": { "codepoints": [9829], "characters": "\u2665" },+  "&heartsuit;": { "codepoints": [9829], "characters": "\u2665" },+  "&hellip;": { "codepoints": [8230], "characters": "\u2026" },+  "&hercon;": { "codepoints": [8889], "characters": "\u22B9" },+  "&hfr;": { "codepoints": [120101], "characters": "\uD835\uDD25" },+  "&hksearow;": { "codepoints": [10533], "characters": "\u2925" },+  "&hkswarow;": { "codepoints": [10534], "characters": "\u2926" },+  "&hoarr;": { "codepoints": [8703], "characters": "\u21FF" },+  "&homtht;": { "codepoints": [8763], "characters": "\u223B" },+  "&hookleftarrow;": { "codepoints": [8617], "characters": "\u21A9" },+  "&hookrightarrow;": { "codepoints": [8618], "characters": "\u21AA" },+  "&hopf;": { "codepoints": [120153], "characters": "\uD835\uDD59" },+  "&horbar;": { "codepoints": [8213], "characters": "\u2015" },+  "&hscr;": { "codepoints": [119997], "characters": "\uD835\uDCBD" },+  "&hslash;": { "codepoints": [8463], "characters": "\u210F" },+  "&hstrok;": { "codepoints": [295], "characters": "\u0127" },+  "&hybull;": { "codepoints": [8259], "characters": "\u2043" },+  "&hyphen;": { "codepoints": [8208], "characters": "\u2010" },+  "&iacute": { "codepoints": [237], "characters": "\u00ED" },+  "&iacute;": { "codepoints": [237], "characters": "\u00ED" },+  "&ic;": { "codepoints": [8291], "characters": "\u2063" },+  "&icirc": { "codepoints": [238], "characters": "\u00EE" },+  "&icirc;": { "codepoints": [238], "characters": "\u00EE" },+  "&icy;": { "codepoints": [1080], "characters": "\u0438" },+  "&iecy;": { "codepoints": [1077], "characters": "\u0435" },+  "&iexcl": { "codepoints": [161], "characters": "\u00A1" },+  "&iexcl;": { "codepoints": [161], "characters": "\u00A1" },+  "&iff;": { "codepoints": [8660], "characters": "\u21D4" },+  "&ifr;": { "codepoints": [120102], "characters": "\uD835\uDD26" },+  "&igrave": { "codepoints": [236], "characters": "\u00EC" },+  "&igrave;": { "codepoints": [236], "characters": "\u00EC" },+  "&ii;": { "codepoints": [8520], "characters": "\u2148" },+  "&iiiint;": { "codepoints": [10764], "characters": "\u2A0C" },+  "&iiint;": { "codepoints": [8749], "characters": "\u222D" },+  "&iinfin;": { "codepoints": [10716], "characters": "\u29DC" },+  "&iiota;": { "codepoints": [8489], "characters": "\u2129" },+  "&ijlig;": { "codepoints": [307], "characters": "\u0133" },+  "&imacr;": { "codepoints": [299], "characters": "\u012B" },+  "&image;": { "codepoints": [8465], "characters": "\u2111" },+  "&imagline;": { "codepoints": [8464], "characters": "\u2110" },+  "&imagpart;": { "codepoints": [8465], "characters": "\u2111" },+  "&imath;": { "codepoints": [305], "characters": "\u0131" },+  "&imof;": { "codepoints": [8887], "characters": "\u22B7" },+  "&imped;": { "codepoints": [437], "characters": "\u01B5" },+  "&in;": { "codepoints": [8712], "characters": "\u2208" },+  "&incare;": { "codepoints": [8453], "characters": "\u2105" },+  "&infin;": { "codepoints": [8734], "characters": "\u221E" },+  "&infintie;": { "codepoints": [10717], "characters": "\u29DD" },+  "&inodot;": { "codepoints": [305], "characters": "\u0131" },+  "&int;": { "codepoints": [8747], "characters": "\u222B" },+  "&intcal;": { "codepoints": [8890], "characters": "\u22BA" },+  "&integers;": { "codepoints": [8484], "characters": "\u2124" },+  "&intercal;": { "codepoints": [8890], "characters": "\u22BA" },+  "&intlarhk;": { "codepoints": [10775], "characters": "\u2A17" },+  "&intprod;": { "codepoints": [10812], "characters": "\u2A3C" },+  "&iocy;": { "codepoints": [1105], "characters": "\u0451" },+  "&iogon;": { "codepoints": [303], "characters": "\u012F" },+  "&iopf;": { "codepoints": [120154], "characters": "\uD835\uDD5A" },+  "&iota;": { "codepoints": [953], "characters": "\u03B9" },+  "&iprod;": { "codepoints": [10812], "characters": "\u2A3C" },+  "&iquest": { "codepoints": [191], "characters": "\u00BF" },+  "&iquest;": { "codepoints": [191], "characters": "\u00BF" },+  "&iscr;": { "codepoints": [119998], "characters": "\uD835\uDCBE" },+  "&isin;": { "codepoints": [8712], "characters": "\u2208" },+  "&isinE;": { "codepoints": [8953], "characters": "\u22F9" },+  "&isindot;": { "codepoints": [8949], "characters": "\u22F5" },+  "&isins;": { "codepoints": [8948], "characters": "\u22F4" },+  "&isinsv;": { "codepoints": [8947], "characters": "\u22F3" },+  "&isinv;": { "codepoints": [8712], "characters": "\u2208" },+  "&it;": { "codepoints": [8290], "characters": "\u2062" },+  "&itilde;": { "codepoints": [297], "characters": "\u0129" },+  "&iukcy;": { "codepoints": [1110], "characters": "\u0456" },+  "&iuml": { "codepoints": [239], "characters": "\u00EF" },+  "&iuml;": { "codepoints": [239], "characters": "\u00EF" },+  "&jcirc;": { "codepoints": [309], "characters": "\u0135" },+  "&jcy;": { "codepoints": [1081], "characters": "\u0439" },+  "&jfr;": { "codepoints": [120103], "characters": "\uD835\uDD27" },+  "&jmath;": { "codepoints": [567], "characters": "\u0237" },+  "&jopf;": { "codepoints": [120155], "characters": "\uD835\uDD5B" },+  "&jscr;": { "codepoints": [119999], "characters": "\uD835\uDCBF" },+  "&jsercy;": { "codepoints": [1112], "characters": "\u0458" },+  "&jukcy;": { "codepoints": [1108], "characters": "\u0454" },+  "&kappa;": { "codepoints": [954], "characters": "\u03BA" },+  "&kappav;": { "codepoints": [1008], "characters": "\u03F0" },+  "&kcedil;": { "codepoints": [311], "characters": "\u0137" },+  "&kcy;": { "codepoints": [1082], "characters": "\u043A" },+  "&kfr;": { "codepoints": [120104], "characters": "\uD835\uDD28" },+  "&kgreen;": { "codepoints": [312], "characters": "\u0138" },+  "&khcy;": { "codepoints": [1093], "characters": "\u0445" },+  "&kjcy;": { "codepoints": [1116], "characters": "\u045C" },+  "&kopf;": { "codepoints": [120156], "characters": "\uD835\uDD5C" },+  "&kscr;": { "codepoints": [120000], "characters": "\uD835\uDCC0" },+  "&lAarr;": { "codepoints": [8666], "characters": "\u21DA" },+  "&lArr;": { "codepoints": [8656], "characters": "\u21D0" },+  "&lAtail;": { "codepoints": [10523], "characters": "\u291B" },+  "&lBarr;": { "codepoints": [10510], "characters": "\u290E" },+  "&lE;": { "codepoints": [8806], "characters": "\u2266" },+  "&lEg;": { "codepoints": [10891], "characters": "\u2A8B" },+  "&lHar;": { "codepoints": [10594], "characters": "\u2962" },+  "&lacute;": { "codepoints": [314], "characters": "\u013A" },+  "&laemptyv;": { "codepoints": [10676], "characters": "\u29B4" },+  "&lagran;": { "codepoints": [8466], "characters": "\u2112" },+  "&lambda;": { "codepoints": [955], "characters": "\u03BB" },+  "&lang;": { "codepoints": [10216], "characters": "\u27E8" },+  "&langd;": { "codepoints": [10641], "characters": "\u2991" },+  "&langle;": { "codepoints": [10216], "characters": "\u27E8" },+  "&lap;": { "codepoints": [10885], "characters": "\u2A85" },+  "&laquo": { "codepoints": [171], "characters": "\u00AB" },+  "&laquo;": { "codepoints": [171], "characters": "\u00AB" },+  "&larr;": { "codepoints": [8592], "characters": "\u2190" },+  "&larrb;": { "codepoints": [8676], "characters": "\u21E4" },+  "&larrbfs;": { "codepoints": [10527], "characters": "\u291F" },+  "&larrfs;": { "codepoints": [10525], "characters": "\u291D" },+  "&larrhk;": { "codepoints": [8617], "characters": "\u21A9" },+  "&larrlp;": { "codepoints": [8619], "characters": "\u21AB" },+  "&larrpl;": { "codepoints": [10553], "characters": "\u2939" },+  "&larrsim;": { "codepoints": [10611], "characters": "\u2973" },+  "&larrtl;": { "codepoints": [8610], "characters": "\u21A2" },+  "&lat;": { "codepoints": [10923], "characters": "\u2AAB" },+  "&latail;": { "codepoints": [10521], "characters": "\u2919" },+  "&late;": { "codepoints": [10925], "characters": "\u2AAD" },+  "&lates;": { "codepoints": [10925, 65024], "characters": "\u2AAD\uFE00" },+  "&lbarr;": { "codepoints": [10508], "characters": "\u290C" },+  "&lbbrk;": { "codepoints": [10098], "characters": "\u2772" },+  "&lbrace;": { "codepoints": [123], "characters": "\u007B" },+  "&lbrack;": { "codepoints": [91], "characters": "\u005B" },+  "&lbrke;": { "codepoints": [10635], "characters": "\u298B" },+  "&lbrksld;": { "codepoints": [10639], "characters": "\u298F" },+  "&lbrkslu;": { "codepoints": [10637], "characters": "\u298D" },+  "&lcaron;": { "codepoints": [318], "characters": "\u013E" },+  "&lcedil;": { "codepoints": [316], "characters": "\u013C" },+  "&lceil;": { "codepoints": [8968], "characters": "\u2308" },+  "&lcub;": { "codepoints": [123], "characters": "\u007B" },+  "&lcy;": { "codepoints": [1083], "characters": "\u043B" },+  "&ldca;": { "codepoints": [10550], "characters": "\u2936" },+  "&ldquo;": { "codepoints": [8220], "characters": "\u201C" },+  "&ldquor;": { "codepoints": [8222], "characters": "\u201E" },+  "&ldrdhar;": { "codepoints": [10599], "characters": "\u2967" },+  "&ldrushar;": { "codepoints": [10571], "characters": "\u294B" },+  "&ldsh;": { "codepoints": [8626], "characters": "\u21B2" },+  "&le;": { "codepoints": [8804], "characters": "\u2264" },+  "&leftarrow;": { "codepoints": [8592], "characters": "\u2190" },+  "&leftarrowtail;": { "codepoints": [8610], "characters": "\u21A2" },+  "&leftharpoondown;": { "codepoints": [8637], "characters": "\u21BD" },+  "&leftharpoonup;": { "codepoints": [8636], "characters": "\u21BC" },+  "&leftleftarrows;": { "codepoints": [8647], "characters": "\u21C7" },+  "&leftrightarrow;": { "codepoints": [8596], "characters": "\u2194" },+  "&leftrightarrows;": { "codepoints": [8646], "characters": "\u21C6" },+  "&leftrightharpoons;": { "codepoints": [8651], "characters": "\u21CB" },+  "&leftrightsquigarrow;": { "codepoints": [8621], "characters": "\u21AD" },+  "&leftthreetimes;": { "codepoints": [8907], "characters": "\u22CB" },+  "&leg;": { "codepoints": [8922], "characters": "\u22DA" },+  "&leq;": { "codepoints": [8804], "characters": "\u2264" },+  "&leqq;": { "codepoints": [8806], "characters": "\u2266" },+  "&leqslant;": { "codepoints": [10877], "characters": "\u2A7D" },+  "&les;": { "codepoints": [10877], "characters": "\u2A7D" },+  "&lescc;": { "codepoints": [10920], "characters": "\u2AA8" },+  "&lesdot;": { "codepoints": [10879], "characters": "\u2A7F" },+  "&lesdoto;": { "codepoints": [10881], "characters": "\u2A81" },+  "&lesdotor;": { "codepoints": [10883], "characters": "\u2A83" },+  "&lesg;": { "codepoints": [8922, 65024], "characters": "\u22DA\uFE00" },+  "&lesges;": { "codepoints": [10899], "characters": "\u2A93" },+  "&lessapprox;": { "codepoints": [10885], "characters": "\u2A85" },+  "&lessdot;": { "codepoints": [8918], "characters": "\u22D6" },+  "&lesseqgtr;": { "codepoints": [8922], "characters": "\u22DA" },+  "&lesseqqgtr;": { "codepoints": [10891], "characters": "\u2A8B" },+  "&lessgtr;": { "codepoints": [8822], "characters": "\u2276" },+  "&lesssim;": { "codepoints": [8818], "characters": "\u2272" },+  "&lfisht;": { "codepoints": [10620], "characters": "\u297C" },+  "&lfloor;": { "codepoints": [8970], "characters": "\u230A" },+  "&lfr;": { "codepoints": [120105], "characters": "\uD835\uDD29" },+  "&lg;": { "codepoints": [8822], "characters": "\u2276" },+  "&lgE;": { "codepoints": [10897], "characters": "\u2A91" },+  "&lhard;": { "codepoints": [8637], "characters": "\u21BD" },+  "&lharu;": { "codepoints": [8636], "characters": "\u21BC" },+  "&lharul;": { "codepoints": [10602], "characters": "\u296A" },+  "&lhblk;": { "codepoints": [9604], "characters": "\u2584" },+  "&ljcy;": { "codepoints": [1113], "characters": "\u0459" },+  "&ll;": { "codepoints": [8810], "characters": "\u226A" },+  "&llarr;": { "codepoints": [8647], "characters": "\u21C7" },+  "&llcorner;": { "codepoints": [8990], "characters": "\u231E" },+  "&llhard;": { "codepoints": [10603], "characters": "\u296B" },+  "&lltri;": { "codepoints": [9722], "characters": "\u25FA" },+  "&lmidot;": { "codepoints": [320], "characters": "\u0140" },+  "&lmoust;": { "codepoints": [9136], "characters": "\u23B0" },+  "&lmoustache;": { "codepoints": [9136], "characters": "\u23B0" },+  "&lnE;": { "codepoints": [8808], "characters": "\u2268" },+  "&lnap;": { "codepoints": [10889], "characters": "\u2A89" },+  "&lnapprox;": { "codepoints": [10889], "characters": "\u2A89" },+  "&lne;": { "codepoints": [10887], "characters": "\u2A87" },+  "&lneq;": { "codepoints": [10887], "characters": "\u2A87" },+  "&lneqq;": { "codepoints": [8808], "characters": "\u2268" },+  "&lnsim;": { "codepoints": [8934], "characters": "\u22E6" },+  "&loang;": { "codepoints": [10220], "characters": "\u27EC" },+  "&loarr;": { "codepoints": [8701], "characters": "\u21FD" },+  "&lobrk;": { "codepoints": [10214], "characters": "\u27E6" },+  "&longleftarrow;": { "codepoints": [10229], "characters": "\u27F5" },+  "&longleftrightarrow;": { "codepoints": [10231], "characters": "\u27F7" },+  "&longmapsto;": { "codepoints": [10236], "characters": "\u27FC" },+  "&longrightarrow;": { "codepoints": [10230], "characters": "\u27F6" },+  "&looparrowleft;": { "codepoints": [8619], "characters": "\u21AB" },+  "&looparrowright;": { "codepoints": [8620], "characters": "\u21AC" },+  "&lopar;": { "codepoints": [10629], "characters": "\u2985" },+  "&lopf;": { "codepoints": [120157], "characters": "\uD835\uDD5D" },+  "&loplus;": { "codepoints": [10797], "characters": "\u2A2D" },+  "&lotimes;": { "codepoints": [10804], "characters": "\u2A34" },+  "&lowast;": { "codepoints": [8727], "characters": "\u2217" },+  "&lowbar;": { "codepoints": [95], "characters": "\u005F" },+  "&loz;": { "codepoints": [9674], "characters": "\u25CA" },+  "&lozenge;": { "codepoints": [9674], "characters": "\u25CA" },+  "&lozf;": { "codepoints": [10731], "characters": "\u29EB" },+  "&lpar;": { "codepoints": [40], "characters": "\u0028" },+  "&lparlt;": { "codepoints": [10643], "characters": "\u2993" },+  "&lrarr;": { "codepoints": [8646], "characters": "\u21C6" },+  "&lrcorner;": { "codepoints": [8991], "characters": "\u231F" },+  "&lrhar;": { "codepoints": [8651], "characters": "\u21CB" },+  "&lrhard;": { "codepoints": [10605], "characters": "\u296D" },+  "&lrm;": { "codepoints": [8206], "characters": "\u200E" },+  "&lrtri;": { "codepoints": [8895], "characters": "\u22BF" },+  "&lsaquo;": { "codepoints": [8249], "characters": "\u2039" },+  "&lscr;": { "codepoints": [120001], "characters": "\uD835\uDCC1" },+  "&lsh;": { "codepoints": [8624], "characters": "\u21B0" },+  "&lsim;": { "codepoints": [8818], "characters": "\u2272" },+  "&lsime;": { "codepoints": [10893], "characters": "\u2A8D" },+  "&lsimg;": { "codepoints": [10895], "characters": "\u2A8F" },+  "&lsqb;": { "codepoints": [91], "characters": "\u005B" },+  "&lsquo;": { "codepoints": [8216], "characters": "\u2018" },+  "&lsquor;": { "codepoints": [8218], "characters": "\u201A" },+  "&lstrok;": { "codepoints": [322], "characters": "\u0142" },+  "&lt": { "codepoints": [60], "characters": "\u003C" },+  "&lt;": { "codepoints": [60], "characters": "\u003C" },+  "&ltcc;": { "codepoints": [10918], "characters": "\u2AA6" },+  "&ltcir;": { "codepoints": [10873], "characters": "\u2A79" },+  "&ltdot;": { "codepoints": [8918], "characters": "\u22D6" },+  "&lthree;": { "codepoints": [8907], "characters": "\u22CB" },+  "&ltimes;": { "codepoints": [8905], "characters": "\u22C9" },+  "&ltlarr;": { "codepoints": [10614], "characters": "\u2976" },+  "&ltquest;": { "codepoints": [10875], "characters": "\u2A7B" },+  "&ltrPar;": { "codepoints": [10646], "characters": "\u2996" },+  "&ltri;": { "codepoints": [9667], "characters": "\u25C3" },+  "&ltrie;": { "codepoints": [8884], "characters": "\u22B4" },+  "&ltrif;": { "codepoints": [9666], "characters": "\u25C2" },+  "&lurdshar;": { "codepoints": [10570], "characters": "\u294A" },+  "&luruhar;": { "codepoints": [10598], "characters": "\u2966" },+  "&lvertneqq;": { "codepoints": [8808, 65024], "characters": "\u2268\uFE00" },+  "&lvnE;": { "codepoints": [8808, 65024], "characters": "\u2268\uFE00" },+  "&mDDot;": { "codepoints": [8762], "characters": "\u223A" },+  "&macr": { "codepoints": [175], "characters": "\u00AF" },+  "&macr;": { "codepoints": [175], "characters": "\u00AF" },+  "&male;": { "codepoints": [9794], "characters": "\u2642" },+  "&malt;": { "codepoints": [10016], "characters": "\u2720" },+  "&maltese;": { "codepoints": [10016], "characters": "\u2720" },+  "&map;": { "codepoints": [8614], "characters": "\u21A6" },+  "&mapsto;": { "codepoints": [8614], "characters": "\u21A6" },+  "&mapstodown;": { "codepoints": [8615], "characters": "\u21A7" },+  "&mapstoleft;": { "codepoints": [8612], "characters": "\u21A4" },+  "&mapstoup;": { "codepoints": [8613], "characters": "\u21A5" },+  "&marker;": { "codepoints": [9646], "characters": "\u25AE" },+  "&mcomma;": { "codepoints": [10793], "characters": "\u2A29" },+  "&mcy;": { "codepoints": [1084], "characters": "\u043C" },+  "&mdash;": { "codepoints": [8212], "characters": "\u2014" },+  "&measuredangle;": { "codepoints": [8737], "characters": "\u2221" },+  "&mfr;": { "codepoints": [120106], "characters": "\uD835\uDD2A" },+  "&mho;": { "codepoints": [8487], "characters": "\u2127" },+  "&micro": { "codepoints": [181], "characters": "\u00B5" },+  "&micro;": { "codepoints": [181], "characters": "\u00B5" },+  "&mid;": { "codepoints": [8739], "characters": "\u2223" },+  "&midast;": { "codepoints": [42], "characters": "\u002A" },+  "&midcir;": { "codepoints": [10992], "characters": "\u2AF0" },+  "&middot": { "codepoints": [183], "characters": "\u00B7" },+  "&middot;": { "codepoints": [183], "characters": "\u00B7" },+  "&minus;": { "codepoints": [8722], "characters": "\u2212" },+  "&minusb;": { "codepoints": [8863], "characters": "\u229F" },+  "&minusd;": { "codepoints": [8760], "characters": "\u2238" },+  "&minusdu;": { "codepoints": [10794], "characters": "\u2A2A" },+  "&mlcp;": { "codepoints": [10971], "characters": "\u2ADB" },+  "&mldr;": { "codepoints": [8230], "characters": "\u2026" },+  "&mnplus;": { "codepoints": [8723], "characters": "\u2213" },+  "&models;": { "codepoints": [8871], "characters": "\u22A7" },+  "&mopf;": { "codepoints": [120158], "characters": "\uD835\uDD5E" },+  "&mp;": { "codepoints": [8723], "characters": "\u2213" },+  "&mscr;": { "codepoints": [120002], "characters": "\uD835\uDCC2" },+  "&mstpos;": { "codepoints": [8766], "characters": "\u223E" },+  "&mu;": { "codepoints": [956], "characters": "\u03BC" },+  "&multimap;": { "codepoints": [8888], "characters": "\u22B8" },+  "&mumap;": { "codepoints": [8888], "characters": "\u22B8" },+  "&nGg;": { "codepoints": [8921, 824], "characters": "\u22D9\u0338" },+  "&nGt;": { "codepoints": [8811, 8402], "characters": "\u226B\u20D2" },+  "&nGtv;": { "codepoints": [8811, 824], "characters": "\u226B\u0338" },+  "&nLeftarrow;": { "codepoints": [8653], "characters": "\u21CD" },+  "&nLeftrightarrow;": { "codepoints": [8654], "characters": "\u21CE" },+  "&nLl;": { "codepoints": [8920, 824], "characters": "\u22D8\u0338" },+  "&nLt;": { "codepoints": [8810, 8402], "characters": "\u226A\u20D2" },+  "&nLtv;": { "codepoints": [8810, 824], "characters": "\u226A\u0338" },+  "&nRightarrow;": { "codepoints": [8655], "characters": "\u21CF" },+  "&nVDash;": { "codepoints": [8879], "characters": "\u22AF" },+  "&nVdash;": { "codepoints": [8878], "characters": "\u22AE" },+  "&nabla;": { "codepoints": [8711], "characters": "\u2207" },+  "&nacute;": { "codepoints": [324], "characters": "\u0144" },+  "&nang;": { "codepoints": [8736, 8402], "characters": "\u2220\u20D2" },+  "&nap;": { "codepoints": [8777], "characters": "\u2249" },+  "&napE;": { "codepoints": [10864, 824], "characters": "\u2A70\u0338" },+  "&napid;": { "codepoints": [8779, 824], "characters": "\u224B\u0338" },+  "&napos;": { "codepoints": [329], "characters": "\u0149" },+  "&napprox;": { "codepoints": [8777], "characters": "\u2249" },+  "&natur;": { "codepoints": [9838], "characters": "\u266E" },+  "&natural;": { "codepoints": [9838], "characters": "\u266E" },+  "&naturals;": { "codepoints": [8469], "characters": "\u2115" },+  "&nbsp": { "codepoints": [160], "characters": "\u00A0" },+  "&nbsp;": { "codepoints": [160], "characters": "\u00A0" },+  "&nbump;": { "codepoints": [8782, 824], "characters": "\u224E\u0338" },+  "&nbumpe;": { "codepoints": [8783, 824], "characters": "\u224F\u0338" },+  "&ncap;": { "codepoints": [10819], "characters": "\u2A43" },+  "&ncaron;": { "codepoints": [328], "characters": "\u0148" },+  "&ncedil;": { "codepoints": [326], "characters": "\u0146" },+  "&ncong;": { "codepoints": [8775], "characters": "\u2247" },+  "&ncongdot;": { "codepoints": [10861, 824], "characters": "\u2A6D\u0338" },+  "&ncup;": { "codepoints": [10818], "characters": "\u2A42" },+  "&ncy;": { "codepoints": [1085], "characters": "\u043D" },+  "&ndash;": { "codepoints": [8211], "characters": "\u2013" },+  "&ne;": { "codepoints": [8800], "characters": "\u2260" },+  "&neArr;": { "codepoints": [8663], "characters": "\u21D7" },+  "&nearhk;": { "codepoints": [10532], "characters": "\u2924" },+  "&nearr;": { "codepoints": [8599], "characters": "\u2197" },+  "&nearrow;": { "codepoints": [8599], "characters": "\u2197" },+  "&nedot;": { "codepoints": [8784, 824], "characters": "\u2250\u0338" },+  "&nequiv;": { "codepoints": [8802], "characters": "\u2262" },+  "&nesear;": { "codepoints": [10536], "characters": "\u2928" },+  "&nesim;": { "codepoints": [8770, 824], "characters": "\u2242\u0338" },+  "&nexist;": { "codepoints": [8708], "characters": "\u2204" },+  "&nexists;": { "codepoints": [8708], "characters": "\u2204" },+  "&nfr;": { "codepoints": [120107], "characters": "\uD835\uDD2B" },+  "&ngE;": { "codepoints": [8807, 824], "characters": "\u2267\u0338" },+  "&nge;": { "codepoints": [8817], "characters": "\u2271" },+  "&ngeq;": { "codepoints": [8817], "characters": "\u2271" },+  "&ngeqq;": { "codepoints": [8807, 824], "characters": "\u2267\u0338" },+  "&ngeqslant;": { "codepoints": [10878, 824], "characters": "\u2A7E\u0338" },+  "&nges;": { "codepoints": [10878, 824], "characters": "\u2A7E\u0338" },+  "&ngsim;": { "codepoints": [8821], "characters": "\u2275" },+  "&ngt;": { "codepoints": [8815], "characters": "\u226F" },+  "&ngtr;": { "codepoints": [8815], "characters": "\u226F" },+  "&nhArr;": { "codepoints": [8654], "characters": "\u21CE" },+  "&nharr;": { "codepoints": [8622], "characters": "\u21AE" },+  "&nhpar;": { "codepoints": [10994], "characters": "\u2AF2" },+  "&ni;": { "codepoints": [8715], "characters": "\u220B" },+  "&nis;": { "codepoints": [8956], "characters": "\u22FC" },+  "&nisd;": { "codepoints": [8954], "characters": "\u22FA" },+  "&niv;": { "codepoints": [8715], "characters": "\u220B" },+  "&njcy;": { "codepoints": [1114], "characters": "\u045A" },+  "&nlArr;": { "codepoints": [8653], "characters": "\u21CD" },+  "&nlE;": { "codepoints": [8806, 824], "characters": "\u2266\u0338" },+  "&nlarr;": { "codepoints": [8602], "characters": "\u219A" },+  "&nldr;": { "codepoints": [8229], "characters": "\u2025" },+  "&nle;": { "codepoints": [8816], "characters": "\u2270" },+  "&nleftarrow;": { "codepoints": [8602], "characters": "\u219A" },+  "&nleftrightarrow;": { "codepoints": [8622], "characters": "\u21AE" },+  "&nleq;": { "codepoints": [8816], "characters": "\u2270" },+  "&nleqq;": { "codepoints": [8806, 824], "characters": "\u2266\u0338" },+  "&nleqslant;": { "codepoints": [10877, 824], "characters": "\u2A7D\u0338" },+  "&nles;": { "codepoints": [10877, 824], "characters": "\u2A7D\u0338" },+  "&nless;": { "codepoints": [8814], "characters": "\u226E" },+  "&nlsim;": { "codepoints": [8820], "characters": "\u2274" },+  "&nlt;": { "codepoints": [8814], "characters": "\u226E" },+  "&nltri;": { "codepoints": [8938], "characters": "\u22EA" },+  "&nltrie;": { "codepoints": [8940], "characters": "\u22EC" },+  "&nmid;": { "codepoints": [8740], "characters": "\u2224" },+  "&nopf;": { "codepoints": [120159], "characters": "\uD835\uDD5F" },+  "&not": { "codepoints": [172], "characters": "\u00AC" },+  "&not;": { "codepoints": [172], "characters": "\u00AC" },+  "&notin;": { "codepoints": [8713], "characters": "\u2209" },+  "&notinE;": { "codepoints": [8953, 824], "characters": "\u22F9\u0338" },+  "&notindot;": { "codepoints": [8949, 824], "characters": "\u22F5\u0338" },+  "&notinva;": { "codepoints": [8713], "characters": "\u2209" },+  "&notinvb;": { "codepoints": [8951], "characters": "\u22F7" },+  "&notinvc;": { "codepoints": [8950], "characters": "\u22F6" },+  "&notni;": { "codepoints": [8716], "characters": "\u220C" },+  "&notniva;": { "codepoints": [8716], "characters": "\u220C" },+  "&notnivb;": { "codepoints": [8958], "characters": "\u22FE" },+  "&notnivc;": { "codepoints": [8957], "characters": "\u22FD" },+  "&npar;": { "codepoints": [8742], "characters": "\u2226" },+  "&nparallel;": { "codepoints": [8742], "characters": "\u2226" },+  "&nparsl;": { "codepoints": [11005, 8421], "characters": "\u2AFD\u20E5" },+  "&npart;": { "codepoints": [8706, 824], "characters": "\u2202\u0338" },+  "&npolint;": { "codepoints": [10772], "characters": "\u2A14" },+  "&npr;": { "codepoints": [8832], "characters": "\u2280" },+  "&nprcue;": { "codepoints": [8928], "characters": "\u22E0" },+  "&npre;": { "codepoints": [10927, 824], "characters": "\u2AAF\u0338" },+  "&nprec;": { "codepoints": [8832], "characters": "\u2280" },+  "&npreceq;": { "codepoints": [10927, 824], "characters": "\u2AAF\u0338" },+  "&nrArr;": { "codepoints": [8655], "characters": "\u21CF" },+  "&nrarr;": { "codepoints": [8603], "characters": "\u219B" },+  "&nrarrc;": { "codepoints": [10547, 824], "characters": "\u2933\u0338" },+  "&nrarrw;": { "codepoints": [8605, 824], "characters": "\u219D\u0338" },+  "&nrightarrow;": { "codepoints": [8603], "characters": "\u219B" },+  "&nrtri;": { "codepoints": [8939], "characters": "\u22EB" },+  "&nrtrie;": { "codepoints": [8941], "characters": "\u22ED" },+  "&nsc;": { "codepoints": [8833], "characters": "\u2281" },+  "&nsccue;": { "codepoints": [8929], "characters": "\u22E1" },+  "&nsce;": { "codepoints": [10928, 824], "characters": "\u2AB0\u0338" },+  "&nscr;": { "codepoints": [120003], "characters": "\uD835\uDCC3" },+  "&nshortmid;": { "codepoints": [8740], "characters": "\u2224" },+  "&nshortparallel;": { "codepoints": [8742], "characters": "\u2226" },+  "&nsim;": { "codepoints": [8769], "characters": "\u2241" },+  "&nsime;": { "codepoints": [8772], "characters": "\u2244" },+  "&nsimeq;": { "codepoints": [8772], "characters": "\u2244" },+  "&nsmid;": { "codepoints": [8740], "characters": "\u2224" },+  "&nspar;": { "codepoints": [8742], "characters": "\u2226" },+  "&nsqsube;": { "codepoints": [8930], "characters": "\u22E2" },+  "&nsqsupe;": { "codepoints": [8931], "characters": "\u22E3" },+  "&nsub;": { "codepoints": [8836], "characters": "\u2284" },+  "&nsubE;": { "codepoints": [10949, 824], "characters": "\u2AC5\u0338" },+  "&nsube;": { "codepoints": [8840], "characters": "\u2288" },+  "&nsubset;": { "codepoints": [8834, 8402], "characters": "\u2282\u20D2" },+  "&nsubseteq;": { "codepoints": [8840], "characters": "\u2288" },+  "&nsubseteqq;": { "codepoints": [10949, 824], "characters": "\u2AC5\u0338" },+  "&nsucc;": { "codepoints": [8833], "characters": "\u2281" },+  "&nsucceq;": { "codepoints": [10928, 824], "characters": "\u2AB0\u0338" },+  "&nsup;": { "codepoints": [8837], "characters": "\u2285" },+  "&nsupE;": { "codepoints": [10950, 824], "characters": "\u2AC6\u0338" },+  "&nsupe;": { "codepoints": [8841], "characters": "\u2289" },+  "&nsupset;": { "codepoints": [8835, 8402], "characters": "\u2283\u20D2" },+  "&nsupseteq;": { "codepoints": [8841], "characters": "\u2289" },+  "&nsupseteqq;": { "codepoints": [10950, 824], "characters": "\u2AC6\u0338" },+  "&ntgl;": { "codepoints": [8825], "characters": "\u2279" },+  "&ntilde": { "codepoints": [241], "characters": "\u00F1" },+  "&ntilde;": { "codepoints": [241], "characters": "\u00F1" },+  "&ntlg;": { "codepoints": [8824], "characters": "\u2278" },+  "&ntriangleleft;": { "codepoints": [8938], "characters": "\u22EA" },+  "&ntrianglelefteq;": { "codepoints": [8940], "characters": "\u22EC" },+  "&ntriangleright;": { "codepoints": [8939], "characters": "\u22EB" },+  "&ntrianglerighteq;": { "codepoints": [8941], "characters": "\u22ED" },+  "&nu;": { "codepoints": [957], "characters": "\u03BD" },+  "&num;": { "codepoints": [35], "characters": "\u0023" },+  "&numero;": { "codepoints": [8470], "characters": "\u2116" },+  "&numsp;": { "codepoints": [8199], "characters": "\u2007" },+  "&nvDash;": { "codepoints": [8877], "characters": "\u22AD" },+  "&nvHarr;": { "codepoints": [10500], "characters": "\u2904" },+  "&nvap;": { "codepoints": [8781, 8402], "characters": "\u224D\u20D2" },+  "&nvdash;": { "codepoints": [8876], "characters": "\u22AC" },+  "&nvge;": { "codepoints": [8805, 8402], "characters": "\u2265\u20D2" },+  "&nvgt;": { "codepoints": [62, 8402], "characters": "\u003E\u20D2" },+  "&nvinfin;": { "codepoints": [10718], "characters": "\u29DE" },+  "&nvlArr;": { "codepoints": [10498], "characters": "\u2902" },+  "&nvle;": { "codepoints": [8804, 8402], "characters": "\u2264\u20D2" },+  "&nvlt;": { "codepoints": [60, 8402], "characters": "\u003C\u20D2" },+  "&nvltrie;": { "codepoints": [8884, 8402], "characters": "\u22B4\u20D2" },+  "&nvrArr;": { "codepoints": [10499], "characters": "\u2903" },+  "&nvrtrie;": { "codepoints": [8885, 8402], "characters": "\u22B5\u20D2" },+  "&nvsim;": { "codepoints": [8764, 8402], "characters": "\u223C\u20D2" },+  "&nwArr;": { "codepoints": [8662], "characters": "\u21D6" },+  "&nwarhk;": { "codepoints": [10531], "characters": "\u2923" },+  "&nwarr;": { "codepoints": [8598], "characters": "\u2196" },+  "&nwarrow;": { "codepoints": [8598], "characters": "\u2196" },+  "&nwnear;": { "codepoints": [10535], "characters": "\u2927" },+  "&oS;": { "codepoints": [9416], "characters": "\u24C8" },+  "&oacute": { "codepoints": [243], "characters": "\u00F3" },+  "&oacute;": { "codepoints": [243], "characters": "\u00F3" },+  "&oast;": { "codepoints": [8859], "characters": "\u229B" },+  "&ocir;": { "codepoints": [8858], "characters": "\u229A" },+  "&ocirc": { "codepoints": [244], "characters": "\u00F4" },+  "&ocirc;": { "codepoints": [244], "characters": "\u00F4" },+  "&ocy;": { "codepoints": [1086], "characters": "\u043E" },+  "&odash;": { "codepoints": [8861], "characters": "\u229D" },+  "&odblac;": { "codepoints": [337], "characters": "\u0151" },+  "&odiv;": { "codepoints": [10808], "characters": "\u2A38" },+  "&odot;": { "codepoints": [8857], "characters": "\u2299" },+  "&odsold;": { "codepoints": [10684], "characters": "\u29BC" },+  "&oelig;": { "codepoints": [339], "characters": "\u0153" },+  "&ofcir;": { "codepoints": [10687], "characters": "\u29BF" },+  "&ofr;": { "codepoints": [120108], "characters": "\uD835\uDD2C" },+  "&ogon;": { "codepoints": [731], "characters": "\u02DB" },+  "&ograve": { "codepoints": [242], "characters": "\u00F2" },+  "&ograve;": { "codepoints": [242], "characters": "\u00F2" },+  "&ogt;": { "codepoints": [10689], "characters": "\u29C1" },+  "&ohbar;": { "codepoints": [10677], "characters": "\u29B5" },+  "&ohm;": { "codepoints": [937], "characters": "\u03A9" },+  "&oint;": { "codepoints": [8750], "characters": "\u222E" },+  "&olarr;": { "codepoints": [8634], "characters": "\u21BA" },+  "&olcir;": { "codepoints": [10686], "characters": "\u29BE" },+  "&olcross;": { "codepoints": [10683], "characters": "\u29BB" },+  "&oline;": { "codepoints": [8254], "characters": "\u203E" },+  "&olt;": { "codepoints": [10688], "characters": "\u29C0" },+  "&omacr;": { "codepoints": [333], "characters": "\u014D" },+  "&omega;": { "codepoints": [969], "characters": "\u03C9" },+  "&omicron;": { "codepoints": [959], "characters": "\u03BF" },+  "&omid;": { "codepoints": [10678], "characters": "\u29B6" },+  "&ominus;": { "codepoints": [8854], "characters": "\u2296" },+  "&oopf;": { "codepoints": [120160], "characters": "\uD835\uDD60" },+  "&opar;": { "codepoints": [10679], "characters": "\u29B7" },+  "&operp;": { "codepoints": [10681], "characters": "\u29B9" },+  "&oplus;": { "codepoints": [8853], "characters": "\u2295" },+  "&or;": { "codepoints": [8744], "characters": "\u2228" },+  "&orarr;": { "codepoints": [8635], "characters": "\u21BB" },+  "&ord;": { "codepoints": [10845], "characters": "\u2A5D" },+  "&order;": { "codepoints": [8500], "characters": "\u2134" },+  "&orderof;": { "codepoints": [8500], "characters": "\u2134" },+  "&ordf": { "codepoints": [170], "characters": "\u00AA" },+  "&ordf;": { "codepoints": [170], "characters": "\u00AA" },+  "&ordm": { "codepoints": [186], "characters": "\u00BA" },+  "&ordm;": { "codepoints": [186], "characters": "\u00BA" },+  "&origof;": { "codepoints": [8886], "characters": "\u22B6" },+  "&oror;": { "codepoints": [10838], "characters": "\u2A56" },+  "&orslope;": { "codepoints": [10839], "characters": "\u2A57" },+  "&orv;": { "codepoints": [10843], "characters": "\u2A5B" },+  "&oscr;": { "codepoints": [8500], "characters": "\u2134" },+  "&oslash": { "codepoints": [248], "characters": "\u00F8" },+  "&oslash;": { "codepoints": [248], "characters": "\u00F8" },+  "&osol;": { "codepoints": [8856], "characters": "\u2298" },+  "&otilde": { "codepoints": [245], "characters": "\u00F5" },+  "&otilde;": { "codepoints": [245], "characters": "\u00F5" },+  "&otimes;": { "codepoints": [8855], "characters": "\u2297" },+  "&otimesas;": { "codepoints": [10806], "characters": "\u2A36" },+  "&ouml": { "codepoints": [246], "characters": "\u00F6" },+  "&ouml;": { "codepoints": [246], "characters": "\u00F6" },+  "&ovbar;": { "codepoints": [9021], "characters": "\u233D" },+  "&par;": { "codepoints": [8741], "characters": "\u2225" },+  "&para": { "codepoints": [182], "characters": "\u00B6" },+  "&para;": { "codepoints": [182], "characters": "\u00B6" },+  "&parallel;": { "codepoints": [8741], "characters": "\u2225" },+  "&parsim;": { "codepoints": [10995], "characters": "\u2AF3" },+  "&parsl;": { "codepoints": [11005], "characters": "\u2AFD" },+  "&part;": { "codepoints": [8706], "characters": "\u2202" },+  "&pcy;": { "codepoints": [1087], "characters": "\u043F" },+  "&percnt;": { "codepoints": [37], "characters": "\u0025" },+  "&period;": { "codepoints": [46], "characters": "\u002E" },+  "&permil;": { "codepoints": [8240], "characters": "\u2030" },+  "&perp;": { "codepoints": [8869], "characters": "\u22A5" },+  "&pertenk;": { "codepoints": [8241], "characters": "\u2031" },+  "&pfr;": { "codepoints": [120109], "characters": "\uD835\uDD2D" },+  "&phi;": { "codepoints": [966], "characters": "\u03C6" },+  "&phiv;": { "codepoints": [981], "characters": "\u03D5" },+  "&phmmat;": { "codepoints": [8499], "characters": "\u2133" },+  "&phone;": { "codepoints": [9742], "characters": "\u260E" },+  "&pi;": { "codepoints": [960], "characters": "\u03C0" },+  "&pitchfork;": { "codepoints": [8916], "characters": "\u22D4" },+  "&piv;": { "codepoints": [982], "characters": "\u03D6" },+  "&planck;": { "codepoints": [8463], "characters": "\u210F" },+  "&planckh;": { "codepoints": [8462], "characters": "\u210E" },+  "&plankv;": { "codepoints": [8463], "characters": "\u210F" },+  "&plus;": { "codepoints": [43], "characters": "\u002B" },+  "&plusacir;": { "codepoints": [10787], "characters": "\u2A23" },+  "&plusb;": { "codepoints": [8862], "characters": "\u229E" },+  "&pluscir;": { "codepoints": [10786], "characters": "\u2A22" },+  "&plusdo;": { "codepoints": [8724], "characters": "\u2214" },+  "&plusdu;": { "codepoints": [10789], "characters": "\u2A25" },+  "&pluse;": { "codepoints": [10866], "characters": "\u2A72" },+  "&plusmn": { "codepoints": [177], "characters": "\u00B1" },+  "&plusmn;": { "codepoints": [177], "characters": "\u00B1" },+  "&plussim;": { "codepoints": [10790], "characters": "\u2A26" },+  "&plustwo;": { "codepoints": [10791], "characters": "\u2A27" },+  "&pm;": { "codepoints": [177], "characters": "\u00B1" },+  "&pointint;": { "codepoints": [10773], "characters": "\u2A15" },+  "&popf;": { "codepoints": [120161], "characters": "\uD835\uDD61" },+  "&pound": { "codepoints": [163], "characters": "\u00A3" },+  "&pound;": { "codepoints": [163], "characters": "\u00A3" },+  "&pr;": { "codepoints": [8826], "characters": "\u227A" },+  "&prE;": { "codepoints": [10931], "characters": "\u2AB3" },+  "&prap;": { "codepoints": [10935], "characters": "\u2AB7" },+  "&prcue;": { "codepoints": [8828], "characters": "\u227C" },+  "&pre;": { "codepoints": [10927], "characters": "\u2AAF" },+  "&prec;": { "codepoints": [8826], "characters": "\u227A" },+  "&precapprox;": { "codepoints": [10935], "characters": "\u2AB7" },+  "&preccurlyeq;": { "codepoints": [8828], "characters": "\u227C" },+  "&preceq;": { "codepoints": [10927], "characters": "\u2AAF" },+  "&precnapprox;": { "codepoints": [10937], "characters": "\u2AB9" },+  "&precneqq;": { "codepoints": [10933], "characters": "\u2AB5" },+  "&precnsim;": { "codepoints": [8936], "characters": "\u22E8" },+  "&precsim;": { "codepoints": [8830], "characters": "\u227E" },+  "&prime;": { "codepoints": [8242], "characters": "\u2032" },+  "&primes;": { "codepoints": [8473], "characters": "\u2119" },+  "&prnE;": { "codepoints": [10933], "characters": "\u2AB5" },+  "&prnap;": { "codepoints": [10937], "characters": "\u2AB9" },+  "&prnsim;": { "codepoints": [8936], "characters": "\u22E8" },+  "&prod;": { "codepoints": [8719], "characters": "\u220F" },+  "&profalar;": { "codepoints": [9006], "characters": "\u232E" },+  "&profline;": { "codepoints": [8978], "characters": "\u2312" },+  "&profsurf;": { "codepoints": [8979], "characters": "\u2313" },+  "&prop;": { "codepoints": [8733], "characters": "\u221D" },+  "&propto;": { "codepoints": [8733], "characters": "\u221D" },+  "&prsim;": { "codepoints": [8830], "characters": "\u227E" },+  "&prurel;": { "codepoints": [8880], "characters": "\u22B0" },+  "&pscr;": { "codepoints": [120005], "characters": "\uD835\uDCC5" },+  "&psi;": { "codepoints": [968], "characters": "\u03C8" },+  "&puncsp;": { "codepoints": [8200], "characters": "\u2008" },+  "&qfr;": { "codepoints": [120110], "characters": "\uD835\uDD2E" },+  "&qint;": { "codepoints": [10764], "characters": "\u2A0C" },+  "&qopf;": { "codepoints": [120162], "characters": "\uD835\uDD62" },+  "&qprime;": { "codepoints": [8279], "characters": "\u2057" },+  "&qscr;": { "codepoints": [120006], "characters": "\uD835\uDCC6" },+  "&quaternions;": { "codepoints": [8461], "characters": "\u210D" },+  "&quatint;": { "codepoints": [10774], "characters": "\u2A16" },+  "&quest;": { "codepoints": [63], "characters": "\u003F" },+  "&questeq;": { "codepoints": [8799], "characters": "\u225F" },+  "&quot": { "codepoints": [34], "characters": "\u0022" },+  "&quot;": { "codepoints": [34], "characters": "\u0022" },+  "&rAarr;": { "codepoints": [8667], "characters": "\u21DB" },+  "&rArr;": { "codepoints": [8658], "characters": "\u21D2" },+  "&rAtail;": { "codepoints": [10524], "characters": "\u291C" },+  "&rBarr;": { "codepoints": [10511], "characters": "\u290F" },+  "&rHar;": { "codepoints": [10596], "characters": "\u2964" },+  "&race;": { "codepoints": [8765, 817], "characters": "\u223D\u0331" },+  "&racute;": { "codepoints": [341], "characters": "\u0155" },+  "&radic;": { "codepoints": [8730], "characters": "\u221A" },+  "&raemptyv;": { "codepoints": [10675], "characters": "\u29B3" },+  "&rang;": { "codepoints": [10217], "characters": "\u27E9" },+  "&rangd;": { "codepoints": [10642], "characters": "\u2992" },+  "&range;": { "codepoints": [10661], "characters": "\u29A5" },+  "&rangle;": { "codepoints": [10217], "characters": "\u27E9" },+  "&raquo": { "codepoints": [187], "characters": "\u00BB" },+  "&raquo;": { "codepoints": [187], "characters": "\u00BB" },+  "&rarr;": { "codepoints": [8594], "characters": "\u2192" },+  "&rarrap;": { "codepoints": [10613], "characters": "\u2975" },+  "&rarrb;": { "codepoints": [8677], "characters": "\u21E5" },+  "&rarrbfs;": { "codepoints": [10528], "characters": "\u2920" },+  "&rarrc;": { "codepoints": [10547], "characters": "\u2933" },+  "&rarrfs;": { "codepoints": [10526], "characters": "\u291E" },+  "&rarrhk;": { "codepoints": [8618], "characters": "\u21AA" },+  "&rarrlp;": { "codepoints": [8620], "characters": "\u21AC" },+  "&rarrpl;": { "codepoints": [10565], "characters": "\u2945" },+  "&rarrsim;": { "codepoints": [10612], "characters": "\u2974" },+  "&rarrtl;": { "codepoints": [8611], "characters": "\u21A3" },+  "&rarrw;": { "codepoints": [8605], "characters": "\u219D" },+  "&ratail;": { "codepoints": [10522], "characters": "\u291A" },+  "&ratio;": { "codepoints": [8758], "characters": "\u2236" },+  "&rationals;": { "codepoints": [8474], "characters": "\u211A" },+  "&rbarr;": { "codepoints": [10509], "characters": "\u290D" },+  "&rbbrk;": { "codepoints": [10099], "characters": "\u2773" },+  "&rbrace;": { "codepoints": [125], "characters": "\u007D" },+  "&rbrack;": { "codepoints": [93], "characters": "\u005D" },+  "&rbrke;": { "codepoints": [10636], "characters": "\u298C" },+  "&rbrksld;": { "codepoints": [10638], "characters": "\u298E" },+  "&rbrkslu;": { "codepoints": [10640], "characters": "\u2990" },+  "&rcaron;": { "codepoints": [345], "characters": "\u0159" },+  "&rcedil;": { "codepoints": [343], "characters": "\u0157" },+  "&rceil;": { "codepoints": [8969], "characters": "\u2309" },+  "&rcub;": { "codepoints": [125], "characters": "\u007D" },+  "&rcy;": { "codepoints": [1088], "characters": "\u0440" },+  "&rdca;": { "codepoints": [10551], "characters": "\u2937" },+  "&rdldhar;": { "codepoints": [10601], "characters": "\u2969" },+  "&rdquo;": { "codepoints": [8221], "characters": "\u201D" },+  "&rdquor;": { "codepoints": [8221], "characters": "\u201D" },+  "&rdsh;": { "codepoints": [8627], "characters": "\u21B3" },+  "&real;": { "codepoints": [8476], "characters": "\u211C" },+  "&realine;": { "codepoints": [8475], "characters": "\u211B" },+  "&realpart;": { "codepoints": [8476], "characters": "\u211C" },+  "&reals;": { "codepoints": [8477], "characters": "\u211D" },+  "&rect;": { "codepoints": [9645], "characters": "\u25AD" },+  "&reg": { "codepoints": [174], "characters": "\u00AE" },+  "&reg;": { "codepoints": [174], "characters": "\u00AE" },+  "&rfisht;": { "codepoints": [10621], "characters": "\u297D" },+  "&rfloor;": { "codepoints": [8971], "characters": "\u230B" },+  "&rfr;": { "codepoints": [120111], "characters": "\uD835\uDD2F" },+  "&rhard;": { "codepoints": [8641], "characters": "\u21C1" },+  "&rharu;": { "codepoints": [8640], "characters": "\u21C0" },+  "&rharul;": { "codepoints": [10604], "characters": "\u296C" },+  "&rho;": { "codepoints": [961], "characters": "\u03C1" },+  "&rhov;": { "codepoints": [1009], "characters": "\u03F1" },+  "&rightarrow;": { "codepoints": [8594], "characters": "\u2192" },+  "&rightarrowtail;": { "codepoints": [8611], "characters": "\u21A3" },+  "&rightharpoondown;": { "codepoints": [8641], "characters": "\u21C1" },+  "&rightharpoonup;": { "codepoints": [8640], "characters": "\u21C0" },+  "&rightleftarrows;": { "codepoints": [8644], "characters": "\u21C4" },+  "&rightleftharpoons;": { "codepoints": [8652], "characters": "\u21CC" },+  "&rightrightarrows;": { "codepoints": [8649], "characters": "\u21C9" },+  "&rightsquigarrow;": { "codepoints": [8605], "characters": "\u219D" },+  "&rightthreetimes;": { "codepoints": [8908], "characters": "\u22CC" },+  "&ring;": { "codepoints": [730], "characters": "\u02DA" },+  "&risingdotseq;": { "codepoints": [8787], "characters": "\u2253" },+  "&rlarr;": { "codepoints": [8644], "characters": "\u21C4" },+  "&rlhar;": { "codepoints": [8652], "characters": "\u21CC" },+  "&rlm;": { "codepoints": [8207], "characters": "\u200F" },+  "&rmoust;": { "codepoints": [9137], "characters": "\u23B1" },+  "&rmoustache;": { "codepoints": [9137], "characters": "\u23B1" },+  "&rnmid;": { "codepoints": [10990], "characters": "\u2AEE" },+  "&roang;": { "codepoints": [10221], "characters": "\u27ED" },+  "&roarr;": { "codepoints": [8702], "characters": "\u21FE" },+  "&robrk;": { "codepoints": [10215], "characters": "\u27E7" },+  "&ropar;": { "codepoints": [10630], "characters": "\u2986" },+  "&ropf;": { "codepoints": [120163], "characters": "\uD835\uDD63" },+  "&roplus;": { "codepoints": [10798], "characters": "\u2A2E" },+  "&rotimes;": { "codepoints": [10805], "characters": "\u2A35" },+  "&rpar;": { "codepoints": [41], "characters": "\u0029" },+  "&rpargt;": { "codepoints": [10644], "characters": "\u2994" },+  "&rppolint;": { "codepoints": [10770], "characters": "\u2A12" },+  "&rrarr;": { "codepoints": [8649], "characters": "\u21C9" },+  "&rsaquo;": { "codepoints": [8250], "characters": "\u203A" },+  "&rscr;": { "codepoints": [120007], "characters": "\uD835\uDCC7" },+  "&rsh;": { "codepoints": [8625], "characters": "\u21B1" },+  "&rsqb;": { "codepoints": [93], "characters": "\u005D" },+  "&rsquo;": { "codepoints": [8217], "characters": "\u2019" },+  "&rsquor;": { "codepoints": [8217], "characters": "\u2019" },+  "&rthree;": { "codepoints": [8908], "characters": "\u22CC" },+  "&rtimes;": { "codepoints": [8906], "characters": "\u22CA" },+  "&rtri;": { "codepoints": [9657], "characters": "\u25B9" },+  "&rtrie;": { "codepoints": [8885], "characters": "\u22B5" },+  "&rtrif;": { "codepoints": [9656], "characters": "\u25B8" },+  "&rtriltri;": { "codepoints": [10702], "characters": "\u29CE" },+  "&ruluhar;": { "codepoints": [10600], "characters": "\u2968" },+  "&rx;": { "codepoints": [8478], "characters": "\u211E" },+  "&sacute;": { "codepoints": [347], "characters": "\u015B" },+  "&sbquo;": { "codepoints": [8218], "characters": "\u201A" },+  "&sc;": { "codepoints": [8827], "characters": "\u227B" },+  "&scE;": { "codepoints": [10932], "characters": "\u2AB4" },+  "&scap;": { "codepoints": [10936], "characters": "\u2AB8" },+  "&scaron;": { "codepoints": [353], "characters": "\u0161" },+  "&sccue;": { "codepoints": [8829], "characters": "\u227D" },+  "&sce;": { "codepoints": [10928], "characters": "\u2AB0" },+  "&scedil;": { "codepoints": [351], "characters": "\u015F" },+  "&scirc;": { "codepoints": [349], "characters": "\u015D" },+  "&scnE;": { "codepoints": [10934], "characters": "\u2AB6" },+  "&scnap;": { "codepoints": [10938], "characters": "\u2ABA" },+  "&scnsim;": { "codepoints": [8937], "characters": "\u22E9" },+  "&scpolint;": { "codepoints": [10771], "characters": "\u2A13" },+  "&scsim;": { "codepoints": [8831], "characters": "\u227F" },+  "&scy;": { "codepoints": [1089], "characters": "\u0441" },+  "&sdot;": { "codepoints": [8901], "characters": "\u22C5" },+  "&sdotb;": { "codepoints": [8865], "characters": "\u22A1" },+  "&sdote;": { "codepoints": [10854], "characters": "\u2A66" },+  "&seArr;": { "codepoints": [8664], "characters": "\u21D8" },+  "&searhk;": { "codepoints": [10533], "characters": "\u2925" },+  "&searr;": { "codepoints": [8600], "characters": "\u2198" },+  "&searrow;": { "codepoints": [8600], "characters": "\u2198" },+  "&sect": { "codepoints": [167], "characters": "\u00A7" },+  "&sect;": { "codepoints": [167], "characters": "\u00A7" },+  "&semi;": { "codepoints": [59], "characters": "\u003B" },+  "&seswar;": { "codepoints": [10537], "characters": "\u2929" },+  "&setminus;": { "codepoints": [8726], "characters": "\u2216" },+  "&setmn;": { "codepoints": [8726], "characters": "\u2216" },+  "&sext;": { "codepoints": [10038], "characters": "\u2736" },+  "&sfr;": { "codepoints": [120112], "characters": "\uD835\uDD30" },+  "&sfrown;": { "codepoints": [8994], "characters": "\u2322" },+  "&sharp;": { "codepoints": [9839], "characters": "\u266F" },+  "&shchcy;": { "codepoints": [1097], "characters": "\u0449" },+  "&shcy;": { "codepoints": [1096], "characters": "\u0448" },+  "&shortmid;": { "codepoints": [8739], "characters": "\u2223" },+  "&shortparallel;": { "codepoints": [8741], "characters": "\u2225" },+  "&shy": { "codepoints": [173], "characters": "\u00AD" },+  "&shy;": { "codepoints": [173], "characters": "\u00AD" },+  "&sigma;": { "codepoints": [963], "characters": "\u03C3" },+  "&sigmaf;": { "codepoints": [962], "characters": "\u03C2" },+  "&sigmav;": { "codepoints": [962], "characters": "\u03C2" },+  "&sim;": { "codepoints": [8764], "characters": "\u223C" },+  "&simdot;": { "codepoints": [10858], "characters": "\u2A6A" },+  "&sime;": { "codepoints": [8771], "characters": "\u2243" },+  "&simeq;": { "codepoints": [8771], "characters": "\u2243" },+  "&simg;": { "codepoints": [10910], "characters": "\u2A9E" },+  "&simgE;": { "codepoints": [10912], "characters": "\u2AA0" },+  "&siml;": { "codepoints": [10909], "characters": "\u2A9D" },+  "&simlE;": { "codepoints": [10911], "characters": "\u2A9F" },+  "&simne;": { "codepoints": [8774], "characters": "\u2246" },+  "&simplus;": { "codepoints": [10788], "characters": "\u2A24" },+  "&simrarr;": { "codepoints": [10610], "characters": "\u2972" },+  "&slarr;": { "codepoints": [8592], "characters": "\u2190" },+  "&smallsetminus;": { "codepoints": [8726], "characters": "\u2216" },+  "&smashp;": { "codepoints": [10803], "characters": "\u2A33" },+  "&smeparsl;": { "codepoints": [10724], "characters": "\u29E4" },+  "&smid;": { "codepoints": [8739], "characters": "\u2223" },+  "&smile;": { "codepoints": [8995], "characters": "\u2323" },+  "&smt;": { "codepoints": [10922], "characters": "\u2AAA" },+  "&smte;": { "codepoints": [10924], "characters": "\u2AAC" },+  "&smtes;": { "codepoints": [10924, 65024], "characters": "\u2AAC\uFE00" },+  "&softcy;": { "codepoints": [1100], "characters": "\u044C" },+  "&sol;": { "codepoints": [47], "characters": "\u002F" },+  "&solb;": { "codepoints": [10692], "characters": "\u29C4" },+  "&solbar;": { "codepoints": [9023], "characters": "\u233F" },+  "&sopf;": { "codepoints": [120164], "characters": "\uD835\uDD64" },+  "&spades;": { "codepoints": [9824], "characters": "\u2660" },+  "&spadesuit;": { "codepoints": [9824], "characters": "\u2660" },+  "&spar;": { "codepoints": [8741], "characters": "\u2225" },+  "&sqcap;": { "codepoints": [8851], "characters": "\u2293" },+  "&sqcaps;": { "codepoints": [8851, 65024], "characters": "\u2293\uFE00" },+  "&sqcup;": { "codepoints": [8852], "characters": "\u2294" },+  "&sqcups;": { "codepoints": [8852, 65024], "characters": "\u2294\uFE00" },+  "&sqsub;": { "codepoints": [8847], "characters": "\u228F" },+  "&sqsube;": { "codepoints": [8849], "characters": "\u2291" },+  "&sqsubset;": { "codepoints": [8847], "characters": "\u228F" },+  "&sqsubseteq;": { "codepoints": [8849], "characters": "\u2291" },+  "&sqsup;": { "codepoints": [8848], "characters": "\u2290" },+  "&sqsupe;": { "codepoints": [8850], "characters": "\u2292" },+  "&sqsupset;": { "codepoints": [8848], "characters": "\u2290" },+  "&sqsupseteq;": { "codepoints": [8850], "characters": "\u2292" },+  "&squ;": { "codepoints": [9633], "characters": "\u25A1" },+  "&square;": { "codepoints": [9633], "characters": "\u25A1" },+  "&squarf;": { "codepoints": [9642], "characters": "\u25AA" },+  "&squf;": { "codepoints": [9642], "characters": "\u25AA" },+  "&srarr;": { "codepoints": [8594], "characters": "\u2192" },+  "&sscr;": { "codepoints": [120008], "characters": "\uD835\uDCC8" },+  "&ssetmn;": { "codepoints": [8726], "characters": "\u2216" },+  "&ssmile;": { "codepoints": [8995], "characters": "\u2323" },+  "&sstarf;": { "codepoints": [8902], "characters": "\u22C6" },+  "&star;": { "codepoints": [9734], "characters": "\u2606" },+  "&starf;": { "codepoints": [9733], "characters": "\u2605" },+  "&straightepsilon;": { "codepoints": [1013], "characters": "\u03F5" },+  "&straightphi;": { "codepoints": [981], "characters": "\u03D5" },+  "&strns;": { "codepoints": [175], "characters": "\u00AF" },+  "&sub;": { "codepoints": [8834], "characters": "\u2282" },+  "&subE;": { "codepoints": [10949], "characters": "\u2AC5" },+  "&subdot;": { "codepoints": [10941], "characters": "\u2ABD" },+  "&sube;": { "codepoints": [8838], "characters": "\u2286" },+  "&subedot;": { "codepoints": [10947], "characters": "\u2AC3" },+  "&submult;": { "codepoints": [10945], "characters": "\u2AC1" },+  "&subnE;": { "codepoints": [10955], "characters": "\u2ACB" },+  "&subne;": { "codepoints": [8842], "characters": "\u228A" },+  "&subplus;": { "codepoints": [10943], "characters": "\u2ABF" },+  "&subrarr;": { "codepoints": [10617], "characters": "\u2979" },+  "&subset;": { "codepoints": [8834], "characters": "\u2282" },+  "&subseteq;": { "codepoints": [8838], "characters": "\u2286" },+  "&subseteqq;": { "codepoints": [10949], "characters": "\u2AC5" },+  "&subsetneq;": { "codepoints": [8842], "characters": "\u228A" },+  "&subsetneqq;": { "codepoints": [10955], "characters": "\u2ACB" },+  "&subsim;": { "codepoints": [10951], "characters": "\u2AC7" },+  "&subsub;": { "codepoints": [10965], "characters": "\u2AD5" },+  "&subsup;": { "codepoints": [10963], "characters": "\u2AD3" },+  "&succ;": { "codepoints": [8827], "characters": "\u227B" },+  "&succapprox;": { "codepoints": [10936], "characters": "\u2AB8" },+  "&succcurlyeq;": { "codepoints": [8829], "characters": "\u227D" },+  "&succeq;": { "codepoints": [10928], "characters": "\u2AB0" },+  "&succnapprox;": { "codepoints": [10938], "characters": "\u2ABA" },+  "&succneqq;": { "codepoints": [10934], "characters": "\u2AB6" },+  "&succnsim;": { "codepoints": [8937], "characters": "\u22E9" },+  "&succsim;": { "codepoints": [8831], "characters": "\u227F" },+  "&sum;": { "codepoints": [8721], "characters": "\u2211" },+  "&sung;": { "codepoints": [9834], "characters": "\u266A" },+  "&sup1": { "codepoints": [185], "characters": "\u00B9" },+  "&sup1;": { "codepoints": [185], "characters": "\u00B9" },+  "&sup2": { "codepoints": [178], "characters": "\u00B2" },+  "&sup2;": { "codepoints": [178], "characters": "\u00B2" },+  "&sup3": { "codepoints": [179], "characters": "\u00B3" },+  "&sup3;": { "codepoints": [179], "characters": "\u00B3" },+  "&sup;": { "codepoints": [8835], "characters": "\u2283" },+  "&supE;": { "codepoints": [10950], "characters": "\u2AC6" },+  "&supdot;": { "codepoints": [10942], "characters": "\u2ABE" },+  "&supdsub;": { "codepoints": [10968], "characters": "\u2AD8" },+  "&supe;": { "codepoints": [8839], "characters": "\u2287" },+  "&supedot;": { "codepoints": [10948], "characters": "\u2AC4" },+  "&suphsol;": { "codepoints": [10185], "characters": "\u27C9" },+  "&suphsub;": { "codepoints": [10967], "characters": "\u2AD7" },+  "&suplarr;": { "codepoints": [10619], "characters": "\u297B" },+  "&supmult;": { "codepoints": [10946], "characters": "\u2AC2" },+  "&supnE;": { "codepoints": [10956], "characters": "\u2ACC" },+  "&supne;": { "codepoints": [8843], "characters": "\u228B" },+  "&supplus;": { "codepoints": [10944], "characters": "\u2AC0" },+  "&supset;": { "codepoints": [8835], "characters": "\u2283" },+  "&supseteq;": { "codepoints": [8839], "characters": "\u2287" },+  "&supseteqq;": { "codepoints": [10950], "characters": "\u2AC6" },+  "&supsetneq;": { "codepoints": [8843], "characters": "\u228B" },+  "&supsetneqq;": { "codepoints": [10956], "characters": "\u2ACC" },+  "&supsim;": { "codepoints": [10952], "characters": "\u2AC8" },+  "&supsub;": { "codepoints": [10964], "characters": "\u2AD4" },+  "&supsup;": { "codepoints": [10966], "characters": "\u2AD6" },+  "&swArr;": { "codepoints": [8665], "characters": "\u21D9" },+  "&swarhk;": { "codepoints": [10534], "characters": "\u2926" },+  "&swarr;": { "codepoints": [8601], "characters": "\u2199" },+  "&swarrow;": { "codepoints": [8601], "characters": "\u2199" },+  "&swnwar;": { "codepoints": [10538], "characters": "\u292A" },+  "&szlig": { "codepoints": [223], "characters": "\u00DF" },+  "&szlig;": { "codepoints": [223], "characters": "\u00DF" },+  "&target;": { "codepoints": [8982], "characters": "\u2316" },+  "&tau;": { "codepoints": [964], "characters": "\u03C4" },+  "&tbrk;": { "codepoints": [9140], "characters": "\u23B4" },+  "&tcaron;": { "codepoints": [357], "characters": "\u0165" },+  "&tcedil;": { "codepoints": [355], "characters": "\u0163" },+  "&tcy;": { "codepoints": [1090], "characters": "\u0442" },+  "&tdot;": { "codepoints": [8411], "characters": "\u20DB" },+  "&telrec;": { "codepoints": [8981], "characters": "\u2315" },+  "&tfr;": { "codepoints": [120113], "characters": "\uD835\uDD31" },+  "&there4;": { "codepoints": [8756], "characters": "\u2234" },+  "&therefore;": { "codepoints": [8756], "characters": "\u2234" },+  "&theta;": { "codepoints": [952], "characters": "\u03B8" },+  "&thetasym;": { "codepoints": [977], "characters": "\u03D1" },+  "&thetav;": { "codepoints": [977], "characters": "\u03D1" },+  "&thickapprox;": { "codepoints": [8776], "characters": "\u2248" },+  "&thicksim;": { "codepoints": [8764], "characters": "\u223C" },+  "&thinsp;": { "codepoints": [8201], "characters": "\u2009" },+  "&thkap;": { "codepoints": [8776], "characters": "\u2248" },+  "&thksim;": { "codepoints": [8764], "characters": "\u223C" },+  "&thorn": { "codepoints": [254], "characters": "\u00FE" },+  "&thorn;": { "codepoints": [254], "characters": "\u00FE" },+  "&tilde;": { "codepoints": [732], "characters": "\u02DC" },+  "&times": { "codepoints": [215], "characters": "\u00D7" },+  "&times;": { "codepoints": [215], "characters": "\u00D7" },+  "&timesb;": { "codepoints": [8864], "characters": "\u22A0" },+  "&timesbar;": { "codepoints": [10801], "characters": "\u2A31" },+  "&timesd;": { "codepoints": [10800], "characters": "\u2A30" },+  "&tint;": { "codepoints": [8749], "characters": "\u222D" },+  "&toea;": { "codepoints": [10536], "characters": "\u2928" },+  "&top;": { "codepoints": [8868], "characters": "\u22A4" },+  "&topbot;": { "codepoints": [9014], "characters": "\u2336" },+  "&topcir;": { "codepoints": [10993], "characters": "\u2AF1" },+  "&topf;": { "codepoints": [120165], "characters": "\uD835\uDD65" },+  "&topfork;": { "codepoints": [10970], "characters": "\u2ADA" },+  "&tosa;": { "codepoints": [10537], "characters": "\u2929" },+  "&tprime;": { "codepoints": [8244], "characters": "\u2034" },+  "&trade;": { "codepoints": [8482], "characters": "\u2122" },+  "&triangle;": { "codepoints": [9653], "characters": "\u25B5" },+  "&triangledown;": { "codepoints": [9663], "characters": "\u25BF" },+  "&triangleleft;": { "codepoints": [9667], "characters": "\u25C3" },+  "&trianglelefteq;": { "codepoints": [8884], "characters": "\u22B4" },+  "&triangleq;": { "codepoints": [8796], "characters": "\u225C" },+  "&triangleright;": { "codepoints": [9657], "characters": "\u25B9" },+  "&trianglerighteq;": { "codepoints": [8885], "characters": "\u22B5" },+  "&tridot;": { "codepoints": [9708], "characters": "\u25EC" },+  "&trie;": { "codepoints": [8796], "characters": "\u225C" },+  "&triminus;": { "codepoints": [10810], "characters": "\u2A3A" },+  "&triplus;": { "codepoints": [10809], "characters": "\u2A39" },+  "&trisb;": { "codepoints": [10701], "characters": "\u29CD" },+  "&tritime;": { "codepoints": [10811], "characters": "\u2A3B" },+  "&trpezium;": { "codepoints": [9186], "characters": "\u23E2" },+  "&tscr;": { "codepoints": [120009], "characters": "\uD835\uDCC9" },+  "&tscy;": { "codepoints": [1094], "characters": "\u0446" },+  "&tshcy;": { "codepoints": [1115], "characters": "\u045B" },+  "&tstrok;": { "codepoints": [359], "characters": "\u0167" },+  "&twixt;": { "codepoints": [8812], "characters": "\u226C" },+  "&twoheadleftarrow;": { "codepoints": [8606], "characters": "\u219E" },+  "&twoheadrightarrow;": { "codepoints": [8608], "characters": "\u21A0" },+  "&uArr;": { "codepoints": [8657], "characters": "\u21D1" },+  "&uHar;": { "codepoints": [10595], "characters": "\u2963" },+  "&uacute": { "codepoints": [250], "characters": "\u00FA" },+  "&uacute;": { "codepoints": [250], "characters": "\u00FA" },+  "&uarr;": { "codepoints": [8593], "characters": "\u2191" },+  "&ubrcy;": { "codepoints": [1118], "characters": "\u045E" },+  "&ubreve;": { "codepoints": [365], "characters": "\u016D" },+  "&ucirc": { "codepoints": [251], "characters": "\u00FB" },+  "&ucirc;": { "codepoints": [251], "characters": "\u00FB" },+  "&ucy;": { "codepoints": [1091], "characters": "\u0443" },+  "&udarr;": { "codepoints": [8645], "characters": "\u21C5" },+  "&udblac;": { "codepoints": [369], "characters": "\u0171" },+  "&udhar;": { "codepoints": [10606], "characters": "\u296E" },+  "&ufisht;": { "codepoints": [10622], "characters": "\u297E" },+  "&ufr;": { "codepoints": [120114], "characters": "\uD835\uDD32" },+  "&ugrave": { "codepoints": [249], "characters": "\u00F9" },+  "&ugrave;": { "codepoints": [249], "characters": "\u00F9" },+  "&uharl;": { "codepoints": [8639], "characters": "\u21BF" },+  "&uharr;": { "codepoints": [8638], "characters": "\u21BE" },+  "&uhblk;": { "codepoints": [9600], "characters": "\u2580" },+  "&ulcorn;": { "codepoints": [8988], "characters": "\u231C" },+  "&ulcorner;": { "codepoints": [8988], "characters": "\u231C" },+  "&ulcrop;": { "codepoints": [8975], "characters": "\u230F" },+  "&ultri;": { "codepoints": [9720], "characters": "\u25F8" },+  "&umacr;": { "codepoints": [363], "characters": "\u016B" },+  "&uml": { "codepoints": [168], "characters": "\u00A8" },+  "&uml;": { "codepoints": [168], "characters": "\u00A8" },+  "&uogon;": { "codepoints": [371], "characters": "\u0173" },+  "&uopf;": { "codepoints": [120166], "characters": "\uD835\uDD66" },+  "&uparrow;": { "codepoints": [8593], "characters": "\u2191" },+  "&updownarrow;": { "codepoints": [8597], "characters": "\u2195" },+  "&upharpoonleft;": { "codepoints": [8639], "characters": "\u21BF" },+  "&upharpoonright;": { "codepoints": [8638], "characters": "\u21BE" },+  "&uplus;": { "codepoints": [8846], "characters": "\u228E" },+  "&upsi;": { "codepoints": [965], "characters": "\u03C5" },+  "&upsih;": { "codepoints": [978], "characters": "\u03D2" },+  "&upsilon;": { "codepoints": [965], "characters": "\u03C5" },+  "&upuparrows;": { "codepoints": [8648], "characters": "\u21C8" },+  "&urcorn;": { "codepoints": [8989], "characters": "\u231D" },+  "&urcorner;": { "codepoints": [8989], "characters": "\u231D" },+  "&urcrop;": { "codepoints": [8974], "characters": "\u230E" },+  "&uring;": { "codepoints": [367], "characters": "\u016F" },+  "&urtri;": { "codepoints": [9721], "characters": "\u25F9" },+  "&uscr;": { "codepoints": [120010], "characters": "\uD835\uDCCA" },+  "&utdot;": { "codepoints": [8944], "characters": "\u22F0" },+  "&utilde;": { "codepoints": [361], "characters": "\u0169" },+  "&utri;": { "codepoints": [9653], "characters": "\u25B5" },+  "&utrif;": { "codepoints": [9652], "characters": "\u25B4" },+  "&uuarr;": { "codepoints": [8648], "characters": "\u21C8" },+  "&uuml": { "codepoints": [252], "characters": "\u00FC" },+  "&uuml;": { "codepoints": [252], "characters": "\u00FC" },+  "&uwangle;": { "codepoints": [10663], "characters": "\u29A7" },+  "&vArr;": { "codepoints": [8661], "characters": "\u21D5" },+  "&vBar;": { "codepoints": [10984], "characters": "\u2AE8" },+  "&vBarv;": { "codepoints": [10985], "characters": "\u2AE9" },+  "&vDash;": { "codepoints": [8872], "characters": "\u22A8" },+  "&vangrt;": { "codepoints": [10652], "characters": "\u299C" },+  "&varepsilon;": { "codepoints": [1013], "characters": "\u03F5" },+  "&varkappa;": { "codepoints": [1008], "characters": "\u03F0" },+  "&varnothing;": { "codepoints": [8709], "characters": "\u2205" },+  "&varphi;": { "codepoints": [981], "characters": "\u03D5" },+  "&varpi;": { "codepoints": [982], "characters": "\u03D6" },+  "&varpropto;": { "codepoints": [8733], "characters": "\u221D" },+  "&varr;": { "codepoints": [8597], "characters": "\u2195" },+  "&varrho;": { "codepoints": [1009], "characters": "\u03F1" },+  "&varsigma;": { "codepoints": [962], "characters": "\u03C2" },+  "&varsubsetneq;": { "codepoints": [8842, 65024], "characters": "\u228A\uFE00" },+  "&varsubsetneqq;": { "codepoints": [10955, 65024], "characters": "\u2ACB\uFE00" },+  "&varsupsetneq;": { "codepoints": [8843, 65024], "characters": "\u228B\uFE00" },+  "&varsupsetneqq;": { "codepoints": [10956, 65024], "characters": "\u2ACC\uFE00" },+  "&vartheta;": { "codepoints": [977], "characters": "\u03D1" },+  "&vartriangleleft;": { "codepoints": [8882], "characters": "\u22B2" },+  "&vartriangleright;": { "codepoints": [8883], "characters": "\u22B3" },+  "&vcy;": { "codepoints": [1074], "characters": "\u0432" },+  "&vdash;": { "codepoints": [8866], "characters": "\u22A2" },+  "&vee;": { "codepoints": [8744], "characters": "\u2228" },+  "&veebar;": { "codepoints": [8891], "characters": "\u22BB" },+  "&veeeq;": { "codepoints": [8794], "characters": "\u225A" },+  "&vellip;": { "codepoints": [8942], "characters": "\u22EE" },+  "&verbar;": { "codepoints": [124], "characters": "\u007C" },+  "&vert;": { "codepoints": [124], "characters": "\u007C" },+  "&vfr;": { "codepoints": [120115], "characters": "\uD835\uDD33" },+  "&vltri;": { "codepoints": [8882], "characters": "\u22B2" },+  "&vnsub;": { "codepoints": [8834, 8402], "characters": "\u2282\u20D2" },+  "&vnsup;": { "codepoints": [8835, 8402], "characters": "\u2283\u20D2" },+  "&vopf;": { "codepoints": [120167], "characters": "\uD835\uDD67" },+  "&vprop;": { "codepoints": [8733], "characters": "\u221D" },+  "&vrtri;": { "codepoints": [8883], "characters": "\u22B3" },+  "&vscr;": { "codepoints": [120011], "characters": "\uD835\uDCCB" },+  "&vsubnE;": { "codepoints": [10955, 65024], "characters": "\u2ACB\uFE00" },+  "&vsubne;": { "codepoints": [8842, 65024], "characters": "\u228A\uFE00" },+  "&vsupnE;": { "codepoints": [10956, 65024], "characters": "\u2ACC\uFE00" },+  "&vsupne;": { "codepoints": [8843, 65024], "characters": "\u228B\uFE00" },+  "&vzigzag;": { "codepoints": [10650], "characters": "\u299A" },+  "&wcirc;": { "codepoints": [373], "characters": "\u0175" },+  "&wedbar;": { "codepoints": [10847], "characters": "\u2A5F" },+  "&wedge;": { "codepoints": [8743], "characters": "\u2227" },+  "&wedgeq;": { "codepoints": [8793], "characters": "\u2259" },+  "&weierp;": { "codepoints": [8472], "characters": "\u2118" },+  "&wfr;": { "codepoints": [120116], "characters": "\uD835\uDD34" },+  "&wopf;": { "codepoints": [120168], "characters": "\uD835\uDD68" },+  "&wp;": { "codepoints": [8472], "characters": "\u2118" },+  "&wr;": { "codepoints": [8768], "characters": "\u2240" },+  "&wreath;": { "codepoints": [8768], "characters": "\u2240" },+  "&wscr;": { "codepoints": [120012], "characters": "\uD835\uDCCC" },+  "&xcap;": { "codepoints": [8898], "characters": "\u22C2" },+  "&xcirc;": { "codepoints": [9711], "characters": "\u25EF" },+  "&xcup;": { "codepoints": [8899], "characters": "\u22C3" },+  "&xdtri;": { "codepoints": [9661], "characters": "\u25BD" },+  "&xfr;": { "codepoints": [120117], "characters": "\uD835\uDD35" },+  "&xhArr;": { "codepoints": [10234], "characters": "\u27FA" },+  "&xharr;": { "codepoints": [10231], "characters": "\u27F7" },+  "&xi;": { "codepoints": [958], "characters": "\u03BE" },+  "&xlArr;": { "codepoints": [10232], "characters": "\u27F8" },+  "&xlarr;": { "codepoints": [10229], "characters": "\u27F5" },+  "&xmap;": { "codepoints": [10236], "characters": "\u27FC" },+  "&xnis;": { "codepoints": [8955], "characters": "\u22FB" },+  "&xodot;": { "codepoints": [10752], "characters": "\u2A00" },+  "&xopf;": { "codepoints": [120169], "characters": "\uD835\uDD69" },+  "&xoplus;": { "codepoints": [10753], "characters": "\u2A01" },+  "&xotime;": { "codepoints": [10754], "characters": "\u2A02" },+  "&xrArr;": { "codepoints": [10233], "characters": "\u27F9" },+  "&xrarr;": { "codepoints": [10230], "characters": "\u27F6" },+  "&xscr;": { "codepoints": [120013], "characters": "\uD835\uDCCD" },+  "&xsqcup;": { "codepoints": [10758], "characters": "\u2A06" },+  "&xuplus;": { "codepoints": [10756], "characters": "\u2A04" },+  "&xutri;": { "codepoints": [9651], "characters": "\u25B3" },+  "&xvee;": { "codepoints": [8897], "characters": "\u22C1" },+  "&xwedge;": { "codepoints": [8896], "characters": "\u22C0" },+  "&yacute": { "codepoints": [253], "characters": "\u00FD" },+  "&yacute;": { "codepoints": [253], "characters": "\u00FD" },+  "&yacy;": { "codepoints": [1103], "characters": "\u044F" },+  "&ycirc;": { "codepoints": [375], "characters": "\u0177" },+  "&ycy;": { "codepoints": [1099], "characters": "\u044B" },+  "&yen": { "codepoints": [165], "characters": "\u00A5" },+  "&yen;": { "codepoints": [165], "characters": "\u00A5" },+  "&yfr;": { "codepoints": [120118], "characters": "\uD835\uDD36" },+  "&yicy;": { "codepoints": [1111], "characters": "\u0457" },+  "&yopf;": { "codepoints": [120170], "characters": "\uD835\uDD6A" },+  "&yscr;": { "codepoints": [120014], "characters": "\uD835\uDCCE" },+  "&yucy;": { "codepoints": [1102], "characters": "\u044E" },+  "&yuml": { "codepoints": [255], "characters": "\u00FF" },+  "&yuml;": { "codepoints": [255], "characters": "\u00FF" },+  "&zacute;": { "codepoints": [378], "characters": "\u017A" },+  "&zcaron;": { "codepoints": [382], "characters": "\u017E" },+  "&zcy;": { "codepoints": [1079], "characters": "\u0437" },+  "&zdot;": { "codepoints": [380], "characters": "\u017C" },+  "&zeetrf;": { "codepoints": [8488], "characters": "\u2128" },+  "&zeta;": { "codepoints": [950], "characters": "\u03B6" },+  "&zfr;": { "codepoints": [120119], "characters": "\uD835\uDD37" },+  "&zhcy;": { "codepoints": [1078], "characters": "\u0436" },+  "&zigrarr;": { "codepoints": [8669], "characters": "\u21DD" },+  "&zopf;": { "codepoints": [120171], "characters": "\uD835\uDD6B" },+  "&zscr;": { "codepoints": [120015], "characters": "\uD835\uDCCF" },+  "&zwj;": { "codepoints": [8205], "characters": "\u200D" },+  "&zwnj;": { "codepoints": [8204], "characters": "\u200C" }+}
+ mangrove.cabal view
@@ -0,0 +1,223 @@+cabal-version:      2.2++name:               mangrove+author:             Sam May+category:           Web+synopsis:           A parser for web documents according to the HTML5 specification.++version:            0.1.0.0+stability:          provisional+maintainer:         sam@eitilt.life++copyright:          © 2020-2021 Sam May+license:            MPL-2.0+license-file:       LICENSE++extra-source-files: CHANGELOG.md+                  , README.md+                  , README.project.md+data-dir:           data+data-files:         entities.json++homepage:           https://ag.eitilt.life/willow+bug-reports:        mailto:ag@eitilt.life++description:        `mangrove` provides HTML parsing for the Willow web+                    browser suite.  As such, it has not necessarily been+                    written with a broader audience in mind, but the resulting+                    data structures should still be generic enough to serve as+                    a general parsing library should you need HTML5+                    compatibility (most likely, its codified error recovery+                    algorithms); if you do use this for other projects, please+                    do share any issues —or even just discomforts— that+                    broader usage reveals.  Notably, however, `mangrove` makes+                    no attempt to parse CSS, JavaScript, or to access linked+                    files, leaving those tasks to other parts of the suite and+                    merely generates a simple document tree from the markup.++build-type:         Simple+tested-with:        GHC ==8.2.2, GHC ==8.4.4, GHC ==8.6.5, GHC ==8.8.4, GHC ==8.10.3+++source-repository head+  type:             darcs+  location:         https://darcs.eitilt.life/willow+  subdir:           mangrove++source-repository this+  type:             darcs+  location:         https://darcs.eitilt.life/willow+  subdir:           mangrove+  tag:              v/html/0.1.0.0.+++source-repository head+  type:             darcs+  location:         https://hub.darcs.net/ag.eitilt/willow+  subdir:           mangrove++source-repository this+  type:             darcs+  location:         https://hub.darcs.net/ag.eitilt/willow+  subdir:           mangrove+  tag:              v/html/0.1.0.0.+++flag dev+  description:      Trigger stricter behaviour for development+  default:          False+  manual:           True++flag html5lib+  default:          False+  description:      Enable the html5lib tests, which require manually+                    downloading the test data.++flag wpt+  default:          False+  description:      Enable integration with web-platform-tests suite, greatly+                    expanding the test surface at the expense of requiring the+                    manual download of the (large) test data repository.+++common flags-ghc+  if flag(dev)+    ghc-options:    -Weverything+                    -Wno-unrecognised-pragmas+                    -Wno-implicit-prelude -Wno-missing-import-lists+                    -Wno-missing-exported-signatures -Wno-missing-local-signatures+                    -Wno-missed-specialisations -Wno-all-missed-specialisations+                    -Wno-safe -Wno-unsafe+                    -Wno-monomorphism-restriction+    if impl(ghc >= 8.4.1)+      ghc-options:  -Wno-missing-export-lists+    if impl(ghc >= 8.8.1)+      ghc-options:  -Wno-missing-deriving-strategies+    if impl(ghc >= 8.10.1)+      ghc-options:  -Wno-missing-safe-haskell-mode+                    -Wno-prepositive-qualified-module+  else+    ghc-options:    -Wno-unrecognised-pragmas+++library+  import:           flags-ghc+  hs-source-dirs:   src+  exposed-modules:  Web.Mangrove.Parse.Common.Error+                  , Web.Mangrove.Parse.Encoding.Preprocess+                  , Web.Mangrove.Parse.Tokenize+                  , Web.Mangrove.Parse.Tree+  other-modules:    Paths_mangrove+                  , Web.Mangrove.Parse.Common.Character+                  , Web.Mangrove.Parse.Tokenize.CData+                  , Web.Mangrove.Parse.Tokenize.Character+                  , Web.Mangrove.Parse.Tokenize.Common+                  , Web.Mangrove.Parse.Tokenize.Comment+                  , Web.Mangrove.Parse.Tokenize.Data+                  , Web.Mangrove.Parse.Tokenize.Dispatcher+                  , Web.Mangrove.Parse.Tokenize.Doctype+                  , Web.Mangrove.Parse.Tokenize.RawText+                  , Web.Mangrove.Parse.Tokenize.RCData+                  , Web.Mangrove.Parse.Tokenize.PlainText+                  , Web.Mangrove.Parse.Tokenize.ScriptData+                  , Web.Mangrove.Parse.Tokenize.ScriptDataEscaped+                  , Web.Mangrove.Parse.Tokenize.ScriptDataDoubleEscaped+                  , Web.Mangrove.Parse.Tokenize.Tag+                  , Web.Mangrove.Parse.Tree.AfterAfterBody+                  , Web.Mangrove.Parse.Tree.AfterAfterFrameset+                  , Web.Mangrove.Parse.Tree.AfterBody+                  , Web.Mangrove.Parse.Tree.AfterFrameset+                  , Web.Mangrove.Parse.Tree.AfterHead+                  , Web.Mangrove.Parse.Tree.BeforeHead+                  , Web.Mangrove.Parse.Tree.BeforeHtml+                  , Web.Mangrove.Parse.Tree.Common+                  , Web.Mangrove.Parse.Tree.Dispatcher+                  , Web.Mangrove.Parse.Tree.Foreign+                  , Web.Mangrove.Parse.Tree.InBody+                  , Web.Mangrove.Parse.Tree.InCaption+                  , Web.Mangrove.Parse.Tree.InCell+                  , Web.Mangrove.Parse.Tree.InColumnGroup+                  , Web.Mangrove.Parse.Tree.InFrameset+                  , Web.Mangrove.Parse.Tree.InHead+                  , Web.Mangrove.Parse.Tree.InHeadNoscript+                  , Web.Mangrove.Parse.Tree.Initial+                  , Web.Mangrove.Parse.Tree.InRow+                  , Web.Mangrove.Parse.Tree.InSelect+                  , Web.Mangrove.Parse.Tree.InSelectInTable+                  , Web.Mangrove.Parse.Tree.InTable+                  , Web.Mangrove.Parse.Tree.InTableBody+                  , Web.Mangrove.Parse.Tree.InTableText+                  , Web.Mangrove.Parse.Tree.InTemplate+                  , Web.Mangrove.Parse.Tree.InText+                  , Web.Mangrove.Parse.Tree.Patch+                  , Web.Mangrove.Parse.Tree.Patch.Fold+  autogen-modules:  Paths_mangrove+  build-depends:    base >=4.10 && <4.15+                  , willow+                  , aeson <1.6+                  , bytestring <0.11+                  , containers <0.7+                  , filepath <1.5+                  , text <1.3+                  , transformers <0.6+                  , unordered-containers <0.3+                  , utility-ht <0.1+                  , vector <0.13+  default-language: Haskell98+  other-extensions: ApplicativeDo+                  , CPP+                  , FlexibleInstances+                  , FunctionalDependencies+                  , KindSignatures+                  , MultiParamTypeClasses+                  , OverloadedStrings+                  , Trustworthy+                  , UndecidableInstances++test-suite html5lib+  import:           flags-ghc+  type:             exitcode-stdio-1.0+  hs-source-dirs:   test+  if !flag(html5lib)+    buildable:      False+  main-is:          Test/Mangrove/Html5Lib.hs+  other-modules:    Paths_mangrove+                  , Test.Mangrove.Html5Lib.Common+                  , Test.Mangrove.Html5Lib.Tokenizer+                  , Test.Mangrove.Html5Lib.Tokenizer.JSON+                  , Test.Mangrove.Html5Lib.TreeConstruction+                  , Test.Mangrove.Html5Lib.TreeConstruction.Parser+  autogen-modules:  Paths_mangrove+  build-depends:    base+                  , mangrove+                  , willow+                  , aeson+                  , bytestring+                  , filepath+                  , text+                  , unordered-containers+                  , vector+                  , HUnit <1.7+  default-language: Haskell98+  other-extensions: CPP+                  , OverloadedStrings++test-suite unit+  import:           flags-ghc+  type:             exitcode-stdio-1.0+  hs-source-dirs:   test+  main-is:          Test/Mangrove/Unit.hs+  other-modules:    Paths_mangrove+                    Test.Mangrove.Unit.Common+                  , Test.Mangrove.Unit.Tokenize+                  , Test.Mangrove.Unit.Tokenize.Tag+  autogen-modules:  Paths_mangrove+  build-depends:    base+                  , mangrove+                  , bytestring+                  , filepath+                  , text+                  , utility-ht+                  , HUnit <1.7+  default-language: Haskell98+  other-extensions: OverloadedStrings
+ src/Web/Mangrove/Parse/Common/Character.hs view
@@ -0,0 +1,202 @@+{-# LANGUAGE Trustworthy #-}++{-|+Description:    HTML named character reference definitions and lookup.++Copyright:      (c) 2020 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      provisional+Portability:    portable++The __[HTML](https://html.spec.whatwg.org/)__ standard defines a large number+of iconic character names for accessing commonly-used characters outside of the+file encoding, or just the capabilities of the keyboard used.  A simple+implementation would be a @'M.HashMap' 'String' 'String'@ (as a few names map+to multiple Unicode characters), but unfortunately the compatibility+restrictions of the parsing algorithm mean that type would be+less-than-performant; as the parser needs to check for a valid reference on+every character, the full reference pool would need to be searched each time+for @O(n*log m)@ with a large @m@.  (While @m@ can be reduced by filtering the+map at each step, amortized for @O(n*m*log m)@, there's no guarantee the+smaller @m@ is ultimately an improvement.)++This module instead organizes the character references into a search tree+indexed by 'Char', allowing each test step to operate over a much smaller+search space for @O(n*log 62)@, at the expense of slightly greater space+overhead.+-}+module Web.Mangrove.Parse.Common.Character+    ( CharacterReferenceTree ( .. )+    , ReferenceValue ( .. )+    , lookupCharacterReference+    , characterReferences+    ) where+++import qualified Data.Aeson as J+import qualified Data.Bifunctor as F.B+import qualified Data.Either as E+import qualified Data.HashMap.Strict as M+import qualified Data.Text as T++import qualified System.IO.Unsafe as IO.Unsafe++import Paths_mangrove++import Control.Applicative ( (<|>) )+import Data.Aeson ( (.:) )+import System.FilePath ( (<.>) )+++-- | __HTML:__+--      @[named character references]+--      (https://html.spec.whatwg.org/multipage/named-characters.html#named-character-references)@+-- +-- The decomposition of the HTML named character reference table into a+-- search-optimized form.  The value type consists of the character reference+-- 'String' whose name terminates with the key 'Char', if one exists, alongside+-- any character references whose names are prefixed appropriately.+-- +-- The ampersand and semicolon delimiting the character reference are not+-- considered part of the name for storage; the former is silently dropped,+-- while the latter is indicated by the value of 'isSemicolonOptional'.+-- +-- For example, a minimal tree defining only the reference names @"&cent"@,+-- @"&cent;"@, and @"&centerdot;"@ would have the structure:+-- +-- > [ ( 'c'+-- >   , Nothing+-- >   , [ ( 'e'+-- >       , Nothing+-- >       , [ ( 'n'+-- >           , Nothing+-- >           , [ ( 't'+-- >               , Just $ ReferenceValue True '\xA2'+-- >               , [ ( 'e'+-- >                   , Nothing+-- >                   , [ ( 'r'+-- >                       , Nothing+-- >                       , [ ( 'd'+-- >                           , Nothing+-- >                           , [ ( 'o'+-- >                               , Nothing+-- >                               , [ ( 't'+-- >                                   , Just $ ReferenceValue False '\xB7'+-- >                                   , []+-- >   ) ] ) ] ) ] ) ] ) ] ) ] ) ] ) ] ) ]+newtype CharacterReferenceTree =+    CharacterReferenceTree (M.HashMap Char (Maybe ReferenceValue, CharacterReferenceTree))+  deriving ( Eq, Show, Read )+++-- | A collection of data describing how to replace some named character+-- reference with a Unicode character sequence.+data ReferenceValue = ReferenceValue+    { isSemicolonOptional :: Bool+        -- ^ Whether the reference allows a compatibility form without a+        -- terminating semicolon.+    , referenceValue :: String+        -- ^ The 'Char'(s) to insert into the document in place of the+        -- reference.+    }+  deriving ( Eq, Show, Read )+++-- | User-friendly access into 'characterReferences', if the full name of the+-- potential character reference is already known.  Note that the underlying+-- map isn't structured as a traditional 'M.HashMap', and so lookup is @O(n)@+-- over the length of the name rather than @O(log m)@ over the size of the map.+-- +-- This doesn't perform the longest-match calculations described by the HTML+-- standard, just a simple "does this string match a reference name" as if the+-- underlying structure were a flat @'M.HashMap' 'String' ref@.  The leading+-- ampersand and trailing semicolon may be present, but neither is required.+lookupCharacterReference :: String -> Maybe ReferenceValue+lookupCharacterReference = lookupCharacterReference' characterReferences . dropAmpersand+  where dropAmpersand ('&':cs) = cs+        dropAmpersand cs = cs++-- | Iterate through the reference tree according to the remainder of the+-- reference name.+lookupCharacterReference' :: CharacterReferenceTree -> String -> Maybe ReferenceValue+lookupCharacterReference' _ [] = Nothing+lookupCharacterReference' (CharacterReferenceTree refs) [c] = M.lookup c refs >>= fst+lookupCharacterReference' (CharacterReferenceTree refs) [c, ';'] = M.lookup c refs >>= fst+lookupCharacterReference' (CharacterReferenceTree refs) (c:cs) = do+    (_, refs') <- M.lookup c refs+    lookupCharacterReference' refs' cs+++-- | The full set of named character references defined by the HTML standard,+-- in a search-optimized form.  Unless the potential reference name isn't+-- completely and unambiguously known (e.g., during the resolution algorithm+-- described by the HTML standard), 'lookupCharacterReference' is the better+-- interface to use.+-- +-- Uses 'IO.Unsafe.unsafePerformIO' internally, as the underlying file should+-- never change at runtime, and so every evaluation would be pure.+characterReferences :: CharacterReferenceTree+characterReferences = IO.Unsafe.unsafePerformIO $ do+    entities <- getDataFileName $ "entities" <.> "json"+    maybe (CharacterReferenceTree M.empty) repackReferences <$> J.decodeFileStrict entities+{-# NOINLINE characterReferences #-}+++-- | Given a naïve map of character reference names to Unicode values, optimize+-- it for searching character-by-character.+repackReferences :: M.HashMap String CharacterData -> CharacterReferenceTree+repackReferences = foldr (repackReferences' . F.B.bimap (drop 1) (map toEnum . codepoints))+    (CharacterReferenceTree M.empty) . M.toList++-- | Add a single key-value character reference pair to the growing search tree.+repackReferences' :: (String, String) -> CharacterReferenceTree -> CharacterReferenceTree+repackReferences' ref (CharacterReferenceTree refs) = CharacterReferenceTree $+    M.unionWith joinTree (singletonReferenceTree ref) refs++-- | 'M.unionWith' any character references sharing a prefix.  Note that if+-- multiple references have the same name (modulo semicolons) but different+-- values, the resulting value isn't necessarily predictable, nor guaranteed to+-- be stable between even minor library versions.+joinTree+    :: (Maybe ReferenceValue, CharacterReferenceTree)+        -- ^ The reference value to be added at some character.+    -> (Maybe ReferenceValue, CharacterReferenceTree)+        -- ^ The reference value already existing in the map.+    -> (Maybe ReferenceValue, CharacterReferenceTree)+joinTree (l, CharacterReferenceTree ls) (r, CharacterReferenceTree rs) =+    (fmap checkOptional $ l <|> r, CharacterReferenceTree $ M.unionWith joinTree ls rs)+  where checkOptional ref = ref+            { isSemicolonOptional =+                maybe False isSemicolonOptional l || maybe False isSemicolonOptional r+            }++-- | Generate a search tree containing the single key-value character reference+-- pair, to be merged into a larger accumulation.+singletonReferenceTree+    :: (String, String)+    -> M.HashMap Char (Maybe ReferenceValue, CharacterReferenceTree)+singletonReferenceTree (key, ref) = E.fromRight M.empty $+    foldr singletonReferenceTree' (Left True) key+  where singletonReferenceTree' ';' _ = Left False+        singletonReferenceTree' c (Left semicolon) = Right $+            M.singleton c (Just $ ReferenceValue semicolon ref, CharacterReferenceTree M.empty)+        singletonReferenceTree' c (Right ref') = Right $+            M.singleton c (Nothing, CharacterReferenceTree ref')+++-- | Internal representation of the character reference definitions in the+-- @entities.json@ file; the names are provided by JSON dictionary keys.+newtype CharacterData = CharacterData+    { codepoints :: [Int]+        -- ^ The Unicode code points represented by a given name.+ -- , characters :: String+ --     Uses surrogate character points rather than high-Unicode characters,+ --     and so isn't as desirable to build from.+    }+  deriving ( Eq, Show, Read )+instance J.FromJSON CharacterData where+    parseJSON = J.withObject "reference" $ \v -> CharacterData+        <$> v .: T.pack "codepoints"+     -- <*> v .: T.pack "characters"
+ src/Web/Mangrove/Parse/Common/Error.hs view
@@ -0,0 +1,128 @@+{-|+Description:    Well-typed errors which may be raised by the HTML parsing algorithm.++Copyright:      (c) 2020 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      experimental+Portability:    portable+-}+module Web.Mangrove.Parse.Common.Error+    ( ParseError ( .. )+    ) where+++import qualified Data.ByteString.Short as BS.SH+import qualified Data.Text as T++import qualified Numeric.Natural as Z++import Web.Willow.DOM+++-- | __HTML:__+--      @[parse errors]+--      (https://html.spec.whatwg.org/multipage/parsing.html#parse-errors)@+--      (constructor names directly reflect the identifiers in the spec)+-- +-- Indicators that something syntactically unexpected was found in an HTML+-- file.  Given the focus on recovery in the parse algorithm, these are more+-- properly warnings than errors, but the name follows that in the+-- specification.+-- +-- This datatype consists of the union of the errors raised by any of the+-- various stages of the algorithm; no distinction is made between an "encoding+-- error" and a "tree-building error".+-- +-- Several constructors wrap a payload which would otherwise be unrecoverable+-- from the DOM, to enable richer error reporting.+data ParseError+    = AbruptTemplateFragment+    | AbruptClosingOfEmptyComment+    | AbruptDoctypePublicIdentifier+    | AbruptDoctypeSystemIdentifier+    | AbsenceOfDigitsInNumericCharacterReference+    | BREndTag+    | CDataInHtmlContent+    | CharacterReferenceOutsideUnicodeRange Z.Natural+    | ControlCharacterInInputStream+    | ControlCharacterReference+    | DuplicateAttribute BasicAttribute+    | DuplicateSingletonElement ElementParams+    | EndTagWithAttributes+    | EndTagWithTrailingSolidus+    | EOFBeforeTagName+    | EOFInCData+    | EOFInComment+    | EOFInDoctype+    | EOFInScriptHtmlCommentLikeText+    | EOFInTag -- Payload?+    | EOFInTemplate+    | EOFInText+    | FramesetInBody ElementParams+    | IncompletelyClosedFormattingElement+    | IncorrectlyClosedComment+    | IncorrectlyOpenedComment+    | InvalidByteSequence BS.SH.ShortByteString+    | InvalidCharacterSequenceAfterDoctypeName+    | InvalidFirstCharacterOfTagName+    | LegacyDoctype+    | MalformedTableStructure ElementParams+    | MissingAttributeValue+    | MissingDoctype+    | MissingDoctypeName+    | MissingDoctypePublicIdentifier+    | MissingDoctypeSystemIdentifier+    | MissingEndTagName+    | MissingQuoteBeforeDoctypePublicIdentifier+    | MissingQuoteBeforeDoctypeSystemIdentifier+    | MissingSemicolonAfterCharacterReference+    | MissingWhitespaceAfterDoctypePublicKeyword+    | MissingWhitespaceAfterDoctypeSystemKeyword+    | MissingWhitespaceBeforeDoctypeName+    | MissingWhitespaceBetweenAttributes+    | MissingWhitespaceBetweenDoctypePublicAndSystemIdentifiers+    | NestedComment+    | NestedElementForAdoptionAgency+    | NestedNonRecursiveElement+    | NestedSingletonElement+    | NoncharacterCharacterReference Char+    | NoncharacterInInputStream+    | NonVoidHtmlElementStartTagWithTrailingSolidus+    | NullCharacterReference+    | ObsoleteTagName T.Text+    | OverlappingFormattingElements+    | OverlappingHeaderElements+    | OverlappingInputElements+    | SurrogateCharacterReference Char+    | SurrogateInInputStream+    | UnexpectedCharacterAfterDoctypeSystemIdentifier Char+    | UnexpectedCharacterInAttributeName+    | UnexpectedCharacterInUnquotedAttributeValue+    | UnexpectedContentAfterBody+    | UnexpectedDescendantElement ElementParams+    | UnexpectedDoctype DocumentTypeParams+    | UnexpectedElementInTableStructure+    | UnexpectedElementWithImpliedEndTag+    | UnexpectedEqualsSignBeforeAttributeName+    | UnexpectedEndColInColumnGroup+    | UnexpectedEndOfBody+    | UnexpectedEndTag ElementParams+    | UnexpectedFormattingElementOutOfScope+    | UnexpectedHtmlElementInForeignContent+    | UnexpectedMetadataOutsideOfHead+    | UnexpectedNodeAfterFrameset -- Payload?+    | UnexpectedNodeInFrameset -- Payload?+    | UnexpectedNodeInHeadNoscript+    | UnexpectedNodeInTableStructure+    | UnexpectedNullCharacter+    | UnexpectedQuestionMarkInsteadOfTagName+    | UnexpectedSolidusInTag+    | UnexpectedTableCellOutsideOfRow+    | UnexpectedTextInTableStructure+    | UnknownNamedCharacterReference+    | UnmatchedEndTag ElementParams+    | UnmatchedEndPTag+    | XmlnsAttribtuteWithMismatchedValue+  deriving ( Eq, Show, Read )
+ src/Web/Mangrove/Parse/Encoding/Preprocess.hs view
@@ -0,0 +1,118 @@+{-# LANGUAGE CPP #-}++{-|+Description:    Clean a 'Char' stream by normalizing newlines and warning about control characters.++Copyright:      (c) 2020 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      experimental+Portability:    portable++To simplify the tokenization parsers, the many representations of line breaks+are unified into a single, Unix-style @\\n@.  While we're iterating over the+input, and before some of the special characters are replaced, it's also a good+time to trigger the warnings for unexpected characters+('ControlCharacterInInputStream', 'SurrogateInInputStream', and+'NoncharacterInInputStream').+-}+module Web.Mangrove.Parse.Encoding.Preprocess+    ( preprocess+    , preprocessStep+      -- * Initialization+    , Encoding ( .. )+    , DecoderState+    , initialDecoderState+    ) where+++import qualified Data.Bifunctor as F.B+import qualified Data.ByteString as BS+import qualified Data.ByteString.Short as BS.SH+import qualified Data.Tuple.HT as U.HT++#if MIN_VERSION_base(4,11,0)+#else+import Data.Semigroup ( (<>) )+#endif++import Web.Mangrove.Parse.Common.Error+import Web.Willow.Common.Encoding+import Web.Willow.Common.Encoding.Character+import Web.Willow.Common.Parser.Util+++-- | __Encoding:__+--      @[preprocessing the input stream]+--      (https://encoding.spec.whatwg.org/#preprocessing-the-input-stream)@+-- +-- Given a character encoding scheme, transform a dependant 'BS.ByteString'+-- into portable 'Char's.  If any byte sequences are meaningless or illegal,+-- they are replaced with the Unicode replacement character @\\xFFFD@.  All+-- newlines are normallized to a single @\\n@ 'Char', and Unicode control+-- characters, surrogate characters, and non-characters are marked with the+-- proper errors.+-- +-- See 'preprocessStep' to operate over only a minimal section.+preprocess :: DecoderState -> BS.ByteString -> ([([ParseError], Char)], DecoderState)+preprocess state = F.B.first (normalize . map charError . concatMap flatten) . decode state+  where normalize ((err1, '\r'):(err2, '\n'):cs) = (err1 <> err2, '\n') : normalize cs+        normalize ((err, '\r'):cs) = (err, '\n') : normalize cs+        normalize (c:cs) = c : normalize cs+        normalize [] = []++-- | __Encoding:__+--      @[preprocessing the input stream]+--      (https://encoding.spec.whatwg.org/#preprocessing-the-input-stream)@+-- +-- Read the smallest number of bytes from the head of the 'BS.ByteString'+-- which would leave the decoder in a re-enterable state.  Any byte+-- sequences which are meaningless or illegal are replaced with the Unicode+-- replacement character @\\xFFFD@.  All newlines are normallized to a single+-- @\\n@ 'Char', and Unicode control characters, surrogate characters, and+-- non-characters are marked with the proper errors.+-- +-- See 'preprocess' to operate over the entire string at once.+preprocessStep+    :: DecoderState+    -> BS.ByteString+    -> ([([ParseError], Char)], DecoderState, BS.ByteString)+preprocessStep state stream = normalize' . flatten' $ decodeStep state stream+  where flatten' = U.HT.mapFst3 (maybe [] $ map charError . flatten)+        normalize' c'@([], _, _) = c'+        normalize' ([(errs, '\r')], state', stream') = case flatten' $ decodeStep state' stream' of+            ((errs', '\n'):cs, state'', stream'') ->+                normalize' ((errs ++ errs', '\n') : cs, state'', stream'')+            _ -> ([(errs, '\n')], state', stream')+        normalize' ((errs, '\r'):(errs', '\n'):cs, state', stream') =+            U.HT.mapFst3 ((errs ++ errs', '\n') :) $ normalize' (cs, state', stream')+        normalize' ((errs, '\r'):cs, state', stream') =+            U.HT.mapFst3 ((errs, '\n') :) $ normalize' (cs, state', stream')+        normalize' (c:cs, state', stream') =+            U.HT.mapFst3 (c :) $ normalize' (cs, state', stream')+++-- | Add a 'ControlCharacterInInputStream', 'SurrogateInInputStream', or+-- 'NoncharacterInInputStream' error to the relevant characters.+charError :: ([ParseError], Char) -> ([ParseError], Char)+charError c'@(_, c)+    | range '\xD800' '\xDFFF' c = addErr SurrogateInInputStream+    | range '\xFDD0' '\xFDEF' c = addErr NoncharacterInInputStream+    | noncharacter = addErr NoncharacterInInputStream+    | range '\SOH' '\US' c && notElem c "\t\n\f\r" = addErr ControlCharacterInInputStream+    | range '\DEL' '\x9F' c = addErr ControlCharacterInInputStream+    | otherwise = c'+  where noncharacter = case mod (fromEnum c) 0x10000 of+            0xFFFE -> True+            0xFFFF -> True+            _ -> False+        -- If this function could ever be applied twice, any errors would be+        -- duplicated.  As it's not exported from the module, that shouldn't be+        -- an issue.+        addErr err = F.B.first (err :) c'++-- | Rewrap the split type containing a list into a list at the top level.  Any+-- 'Left' errors are replaced by @[('InvalidByteSequence' /bytes/, '\\xFFFD')]@.+flatten :: Either BS.SH.ShortByteString String -> [([ParseError], Char)]+flatten = either (\err -> [([InvalidByteSequence err], replacementChar)]) (map $ \c -> ([], c))
+ src/Web/Mangrove/Parse/Tokenize.hs view
@@ -0,0 +1,189 @@+{-|+Description:    Extract basic semantic categories from a simple textual stream.++Copyright:      (c) 2020 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      provisional+Portability:    portable++This module and the internal branch it heads implement the "Tokenization"+section of the+__[HTML](https://html.spec.whatwg.org/multipage/parsing.html#tokenization)__+document parsing specification, processing a stream of text to add information+on, or group it by, semantic category.  This then allows the following stage to+base its logic on such higher-level concepts as "markup tag" or "comment"+without worrying about the (sometimes complex) escaping behaviour required to+parse them.+-}+module Web.Mangrove.Parse.Tokenize+    ( -- * Types+      -- ** Final+      Token ( .. )+    , BasicAttribute+    , TagParams ( .. )+    , emptyTagParams+    , DoctypeParams ( .. )+    , emptyDoctypeParams+      -- ** Intermediate+    , TokenizerState+    , CurrentTokenizerState ( .. )+    , Encoding ( .. )+      -- * Initialization+    , defaultTokenizerState+    , tokenizerMode+    , tokenizerStartTag+    , tokenizerEncoding+      -- * Transformations+    , tokenize+    , tokenizeStep+    , finalizeTokenizer+    ) where+++import qualified Control.Applicative as A+import qualified Control.Monad.Trans.State as N.S++import qualified Data.Bifunctor as F.B+import qualified Data.ByteString as BS+import qualified Data.ByteString.Short as BS.SH+import qualified Data.Maybe as Y++import Web.Willow.DOM++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Encoding.Preprocess+import Web.Mangrove.Parse.Tokenize.Common hiding ( setRemainder )+import Web.Mangrove.Parse.Tokenize.Dispatcher+import Web.Willow.Common.Encoding+import Web.Willow.Common.Encoding.Sniffer+import Web.Willow.Common.Parser+++-- | __HTML:__+--      @[tokenization]+--      (https://html.spec.whatwg.org/multipage/parsing.html#tokenization)@+-- +-- Given a starting environment, transform a binary document stream into a+-- stream of semantic atoms.  If the parse fails, returns all tokens before the+-- one which caused the error, but any trailing bytes are silently dropped.+tokenize+    :: TokenizerState+    -> BS.ByteString+    -> ([([ParseError], Token)], TokenizerState)+tokenize state stream = loop $ tokenizeStep state stream+  where loop (ts, state', output)+            | BS.null output = (ts, state')+            | otherwise = F.B.first (ts ++) . loop $ tokenizeStep state' output+++-- | Parse a minimal number of bytes from an input stream, into a sequence of+-- semantic tokens.  Returns all data required to seamlessly resume parsing.+tokenizeStep+    :: TokenizerState+    -> BS.ByteString+    -> ([([ParseError], Token)], TokenizerState, BS.ByteString)+tokenizeStep state stream = case runParserT (N.S.runStateT dispatcher' state) stream of+    Just ((out, state'), stream') -> (out, state', stream')+    Nothing -> ([], stateEof, BS.empty)+  where stateEof = state+            { decoderState_ = Right $ setRemainder (BS.SH.toShort stream) <$>+                decoderDefaultState state stream+            }++-- | Explicitly indicate that the input stream will not contain any further+-- bytes, and perform any finalization processing based on that.+finalizeTokenizer :: TokenizerState -> [([ParseError], Token)]+finalizeTokenizer state = fst $ tokenize state' BS.empty+  where state' = state+            { tokenParserState = (tokenParserState state)+                { atEndOfStream = True+                }+            }+++-- | Given a string as emitted by the decoder and the final state of that+-- parser, repack it into a single list with that final state encapsulated in a+-- 'Just'; the 'init' of the string is given 'Nothing's.+repackStream :: ([([ParseError], Char)], DecoderState, BS.ByteString) -> [TokenizerInput]+repackStream ([], _, _) = []+repackStream ([(errs, c)], dState, bs) =+    [TokenizerInput errs c $ Just (Just dState, bs)]+repackStream ((errs, c):cs, dState, bs) =+    TokenizerInput errs c Nothing : repackStream (cs, dState, bs)+++-- | Wrap the standard dispatcher to operate over a raw 'BS.ByteString'+-- rather than the "Web.Mangrove.Parse.Encoding" output.+dispatcher' :: StateParser TokenizerState BS.ByteString [([ParseError], Token)]+dispatcher' = do+    state <- N.S.get+    stream <- abridge+    let dState = Y.fromMaybe (initialDecoderState Utf8) $ decoderDefaultState state stream+    recurse state $ preprocessStep' dState stream+  where preprocessStep' dState input = case preprocessStep dState input of+            ([], _, _) -> []+            cs'@(_, dState', input') -> repackStream cs' ++ preprocessStep' dState' input'++-- | Loop the tokenization dispatcher until it returns a set of tokens which+-- happens to coincide with a decoder breakpoint.  Relies on lazy evaluation in+-- the stream generation to avoid forcing the entire thing at once, while still+-- retaining the capability to consume as much input as necessary to get the+-- parsers to line up.+recurse+    :: TokenizerState+    -> [TokenizerInput]+    -> StateParser TokenizerState BS.ByteString [([ParseError], Token)]+recurse state stream = case runParserT (N.S.runStateT dispatcher $ tokenParserState state) stream of+    Nothing -> A.empty+    Just ((out, tokState'), stream') -> case Y.listToMaybe (reverse out) >>= tokenizedState of+        Nothing -> do+            let state' = state+                    { tokenParserState = tokState'+                    }+            N.S.put state'+            out' <- recurse state' stream'+            return $ map repackOut out ++ out'+        Just (dState, dStream) -> do+            pushChunk dStream+            N.S.put $ state+                { decoderState_ = Right dState+                , tokenParserState = tokState'+                }+            return $ map repackOut out+  where repackOut t' = (tokenizedErrs t', tokenizedOut t')+++-- | Specify which section of the finite state machine describing the+-- tokenization algorithm should be active.+tokenizerMode :: CurrentTokenizerState -> TokenizerState -> TokenizerState+tokenizerMode mode state = state+    { tokenParserState = (tokenParserState state)+        { currentState = mode+        }+    }++-- | Specify the data to use as the previous tag which had been emitted by the+-- tokenizer.  This only has to be called when required for external algorithms+-- or constructions; the parser automatically updates as required for generated+-- 'StartTag' tokens.+tokenizerStartTag :: Maybe Namespace -> ElementName -> TokenizerState -> TokenizerState+tokenizerStartTag ns name state = state+    { tokenParserState = (tokenParserState state)+        { prevStartTag = Just name+        , currentNodeNamespace = ns+        }+    }++-- | Specify the encoding scheme used by a given parse environment to read from+-- the binary input stream.  Note that this will always use the initial state+-- for the respective decoder; intermediate states as returned by 'decodeStep'+-- are not supported.+tokenizerEncoding :: Either SnifferEnvironment (Maybe Encoding) -> TokenizerState -> TokenizerState+tokenizerEncoding enc' state = state+    { decoderState_ = case enc' of+        Right Nothing -> Right Nothing+        Right (Just enc) -> Left $ Right enc+        Left env -> Left $ Left env+    }
+ src/Web/Mangrove/Parse/Tokenize/CData.hs view
@@ -0,0 +1,65 @@+{-|+Description:    Tokenization rules for characters within @\<[CDATA[@ ... @]]\>@ sections.++Copyright:      (c) 2020-2021 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      stable+Portability:    portable+-}+module Web.Mangrove.Parse.Tokenize.CData+    ( tokenCDataSection+    ) where+++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tokenize.Common+++-- | __HTML:__+--      @[CDATA section state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#cdata-section-state)@+-- +-- The parsing instructions rooted in the 'CDataState' section of the state+-- machine.+tokenCDataSection :: Tokenizer [TokenizerOutput Token]+tokenCDataSection = tokenizers (Just [([EOFInCData], EndOfStream)])+    [ ifs_ (== ']') tokenCDataSectionBracket+    , elsesChar $ \c -> emit' ([], Character c)+    ]++-- | __HTML:__+--      @[CDATA section bracket state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#cdata-section-bracket-state)@+-- +-- The parsing instructions for after reading @"]"@ in the 'CDataState' section+-- of the state machine.+tokenCDataSectionBracket :: Tokenizer [TokenizerOutput Token]+tokenCDataSectionBracket = tokenizers (Just [([], Character ']'), ([EOFInCData], EndOfStream)])+    [ ifs_ (== ']') tokenCDataSectionEnd+    , elsePush_ $ emit' ([], Character ']')+    ]++-- | __HTML:__+--      @[CDATA section end state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#cdata-section-end-state)@+-- +-- The parsing instructions for after reading @"]]"@ in the 'CDataState'+-- section of the state machine.+tokenCDataSectionEnd :: Tokenizer [TokenizerOutput Token]+tokenCDataSectionEnd = tokenCDataSectionEnd' >>= \ts -> case tokenizedOut ts of+    [] -> return []+    (t':ts') -> emits (tokenizedState ts) $+        (tokenizedErrs ts, Character t') : [([], Character t) | t <- ts']+  where emits state ts = finalStateList state <$> mapM emit ts++-- | Loop within the __HTML__ @[CDATA section end state]+-- (https://html.spec.whatwg.org/multipage/parsing.html#cdata-section-end-state)@+-- to read and re-emit any number of @']'@ 'Char's before the closing @"]]\>"@.+tokenCDataSectionEnd' :: Tokenizer (TokenizerOutput String)+tokenCDataSectionEnd' = tokenizer (Just ([EOFInCData], "]]"))+    [ if_ (== ']') $ consOut ']' <$> tokenCDataSectionEnd'+    , if_ (== '>') $ changeState DataState *> packToken ([], "")+    , elsePush_ $ packToken ([], "]]")+    ]
+ src/Web/Mangrove/Parse/Tokenize/Character.hs view
@@ -0,0 +1,396 @@+{-# LANGUAGE Trustworthy #-}++{-|+Description:    Tokenization rules for characters comprising character references.++Copyright:      (c) 2020-2021 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      stable+Portability:    portable++The monadic parser construction and prefix-preferring 'String' concatenation+mean that character references are resolved opposite the algorithm described in+the __[HTML](https://html.spec.whatwg.org/multipage/parsing.html)__+specification.  That is most notable in the numeric references, where the spec+recommends multiplying an accumulator by the numeric base before adding each+digit, while this implementation multiplies each digit by the positional value+before adding the accumulator.  The reversal in named references is more+conceptual and subtle, but the spec is worded to read a name from the input+while the implementation reads input 'Char's according to the list of reference+names; this is likely how most implementations accomplish it.+-}+module Web.Mangrove.Parse.Tokenize.Character+    ( flushCharRef+    , tokenCharacterReference+    ) where+++import qualified Control.Applicative as A+import qualified Control.Monad as N++import qualified Data.Bifunctor as F.B+import qualified Data.Char as C+import qualified Data.HashMap.Strict as M+import qualified Data.Maybe as Y+import qualified Data.Vector as V+import qualified Data.Word as W++import qualified Numeric.Natural as Z++import Data.Functor ( ($>) )+import Data.Vector ( (!?) )++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Common.Character+import Web.Mangrove.Parse.Tokenize.Common+import Web.Willow.Common.Encoding.Character+import Web.Willow.Common.Parser+import Web.Willow.Common.Parser.Util++import {-# SOURCE #-} Web.Mangrove.Parse.Tokenize.Dispatcher+++-- | __HTML:__+--      @[flush code points consumed as a character reference]+--      (https://html.spec.whatwg.org/multipage/parsing.html#flush-code-points-consumed-as-a-character-reference)@+-- +-- Transform a wrapped 'Char' sequence into a sequence of wrapped 'Character'+-- tokens.+flushCharRef :: TokenizerOutput String -> Tokenizer [TokenizerOutput Token]+flushCharRef out = case tokenizedOut out of+    "" -> case tokenizedErrs out of+        [] -> return []+        errs -> consTokenErrorsList errs . Y.fromMaybe [] <$> A.optional dispatcher+    [c] -> return [out { tokenizedOut = Character c }]+    (c:_) -> do+        let c' = out+                { tokenizedOut = Character c+                , tokenizedState = Nothing+                }+        cs' <- flushCharRef out'+        return $ c' : cs'+  where out' = out+            { tokenizedErrs = []+            , tokenizedOut = drop 1 $ tokenizedOut out+            }+++-- | __HTML:__+--      @[character reference state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#character-reference-state)@+-- +-- The parsing instructions for after reading @"&"@ in a section of the state+-- machine which allows character references.+tokenCharacterReference+    :: Bool+        -- ^ Whether this parser was called from within a markup tag's attribute.+    -> Tokenizer (TokenizerOutput String)+tokenCharacterReference inAttribute = tokenizer (Just ([], "&"))+    [ ifPush_ isAsciiAlphaNum $ do+        ref' <- tokenNamedCharacterReference inAttribute characterReferences+        case tokenizedOut ref' of+            NotFound -> consTokenErrors (tokenizedErrs ref') . consOut '&' <$> tokenAmbiguousAmpersand+            Flush name -> return ref'+                { tokenizedOut = '&' : name+                }+            Found ref -> return ref'+                { tokenizedOut = ref+                }+    , if_ (== '#') tokenNumericCharacterReference+    , elsePush_ $ packToken ([], "&")+    ]++-- | The result of looking up a named character reference.+data CharacterReference+    = NotFound+        -- ^ No matching name found.+    | Flush String+        -- ^ A named reference was found, but historical reasons require+        -- emitting the name as characters anyway.+    | Found String+        -- ^ A named reference was found and successfully resolved.++-- | __HTML:__+--      @[named character reference state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#named-character-reference-state)@+-- +-- The parsing instructions for after reading @"&"@ followed by letters and/or+-- numbers in a section of the state machine which allows character references.+tokenNamedCharacterReference+    :: Bool+        -- ^ Whether this parser was called from within a markup tag's attribute.+    -> CharacterReferenceTree+        -- ^ The list of reference names, filtered according to what prefix has+        -- been already encountered.+    -> Tokenizer (TokenizerOutput CharacterReference)+tokenNamedCharacterReference inAttribute (CharacterReferenceTree refs) = do+    cm' <- A.optional next+    let state1 = endState+        state2 = do+            cm <- cm'+            decodedState cm+    case cm' of+        Nothing -> packState ([], NotFound) state1+        Just c -> case M.lookup (decodedOut c) refs of+            -- No reference found, and no further references possible.+            Nothing -> push c *> packState ([], NotFound) state2+            -- No reference found, but longer potential reference names exist.+            Just (Nothing, refs') -> do+                ref' <- tokenNamedCharacterReference inAttribute refs'+                let errs' = decodedErrs c ++ tokenizedErrs ref'+                case tokenizedOut ref' of+                    -- None of those longer names match.+                    NotFound -> push c *> packState ([], NotFound) state2+                    -- Fall back on the longer name.+                    Flush name -> return ref'+                        { tokenizedErrs = errs'+                        , tokenizedOut = Flush $ decodedOut c : name+                        }+                    Found ref -> return ref'+                        { tokenizedErrs = errs'+                        , tokenizedOut = Found ref+                        }+            -- Reference found at the current name.+            Just (Just found, refs') -> do+                semicolon <- A.optional $ next >>= satisfying (\d -> decodedOut d == ';')+                let state3 = maybe state2 decodedState semicolon+                if not (isSemicolonOptional found) && Y.isNothing semicolon+                    then tokenNamedCharacterReference inAttribute refs' >>= \ref ->+                        case tokenizedOut ref of+                            NotFound -> push c $> ref+                            _ -> return ref+                    else tokenNamedCharacterReference inAttribute refs' >>=+                        foundNamedCharacterReference+                            inAttribute+                            (Y.isJust semicolon)+                            (decodedOut c)+                            state3+                            (referenceValue found)++-- | Perform the logistics around determining what 'Character'(s) should be+-- emitted, based on the current environment and any longer match.+foundNamedCharacterReference+    :: Bool+        -- ^ Whether the reference is being evaluated as part of an attribute value.+    -> Bool+        -- ^ Whether the last character matched is a semicolon.+    -> Char+        -- ^ The last character of the reference, /excluding/ the semicolon.+    -> DecoderOutputState+        -- ^ Remainder of the binary document stream after the last character matched.+    -> String+        -- ^ The evaluated sequence represented by the name.+    -> TokenizerOutput CharacterReference+        -- ^ The value returned by continuing the lookup on a longer string.+    -> Tokenizer (TokenizerOutput CharacterReference)+foundNamedCharacterReference attribute semicolon char state found ref = case tokenizedOut ref of+    NotFound -> if attribute && not semicolon+        then do+            c' <- A.optional $ next >>= satisfying+                (\c -> decodedOut c == '=' || isAsciiAlphaNum (decodedOut c))+            case c' of+                Nothing -> packState ([MissingSemicolonAfterCharacterReference], Found found) state+                Just c -> push c *> packState ([], Flush [char]) state+        else packState (errs', Found found) state+    Flush name -> return $ ref+        { tokenizedOut = Flush $ char : name+        }+    _ -> return ref+  where errs'+            | semicolon = []+            | otherwise = [MissingSemicolonAfterCharacterReference]++-- | __HTML:__+--      @[ambiguous ampersand state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#ambiguous-ampersand-state)@+-- +-- The parsing instructions for after reading @"&"@ followed by a string which+-- does not correspond to any known reference name, in a section of the state+-- machine which allows character references.+tokenAmbiguousAmpersand :: Tokenizer (TokenizerOutput String)+tokenAmbiguousAmpersand = tokenizer (Just ([], ""))+    [ ifChar isAsciiAlphaNum $ \c -> consOut c <$> tokenAmbiguousAmpersand+    , ifPush_ (== ';') $ packToken ([UnknownNamedCharacterReference], "")+    , elsePush_ $ packToken ([], "")+    ]++-- | __HTML:__+--      @[numeric character reference state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#numeric-character-reference-state)@+-- +-- The parsing instructions for after reading @"&#"@ in a section of the state+-- machine which allows character references.+tokenNumericCharacterReference :: Tokenizer (TokenizerOutput String)+tokenNumericCharacterReference = fmap (either ("&#" ++) id) <$> tokenizer+        (Just ([AbsenceOfDigitsInNumericCharacterReference], Left ""))+    [ if_ (== 'x') $ flatten . fmap (F.B.bimap ('x' :) tokenNumericCharacterReferenceEnd) <$>+        tokenHexadecimalCharacterReferenceStart+    , if_ (== 'X') $ flatten . fmap (F.B.bimap ('X' :) tokenNumericCharacterReferenceEnd) <$>+        tokenHexadecimalCharacterReferenceStart+    , elsePush_ $ flatten . fmap (F.B.second tokenNumericCharacterReferenceEnd) <$>+        tokenDecimalCharacterReferenceStart+    ]+  where flatten tok = case tokenizedOut tok of+            Left str -> tok { tokenizedOut = Left str }+            Right (errs, c) -> consTokenErrors errs $ tok { tokenizedOut = Right [c] }++-- | __HTML:__+--      @[hexadecimal character reference start state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#hexadecimal-character-reference-start-state)@+-- +-- The parsing instructions for after reading @"&#x"@ or @"&#X"@ in a section+-- of the state machine which allows character references.+tokenHexadecimalCharacterReferenceStart+    :: Tokenizer (TokenizerOutput (Either String Z.Natural))+        -- ^ The inner data contains a theoretically-Unicode code point+        -- ('Right', though it may exceed the upper bound) or the characters+        -- consumed in reading something invalid ('Left'), as relevant.+tokenHexadecimalCharacterReferenceStart = tokenizer+        (Just ([AbsenceOfDigitsInNumericCharacterReference], Left ""))+    [ ifPush_ C.isHexDigit $ fmap packReference tokenHexadecimalCharacterReference+    , elsePush_ $ packToken ([AbsenceOfDigitsInNumericCharacterReference], Left "")+    ]++-- | __HTML:__+--      @[decimal character reference start state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#decimal-character-reference-start-state)@+-- +-- The parsing instructions for after reading @"&#"@ followed by digit from @0@+-- to @9@ in a section of the state machine which allows character references.+tokenDecimalCharacterReferenceStart+    :: Tokenizer (TokenizerOutput (Either String Z.Natural))+        -- ^ The inner data contains a theoretically-Unicode code point+        -- ('Right', though it may exceed the upper bound) or the characters+        -- consumed in reading something invalid ('Left'), as relevant.+tokenDecimalCharacterReferenceStart = tokenizer+        (Just ([AbsenceOfDigitsInNumericCharacterReference], Left ""))+    [ ifPush_ C.isDigit $ fmap packReference tokenDecimalCharacterReference+    , elsePush_ $ packToken ([AbsenceOfDigitsInNumericCharacterReference], Left "")+    ]++-- | Extract the Unicode-ish code point calculated by the numeric character+-- reference loops, and repack it as output by the respective initializer.+packReference :: TokenizerOutput (Z.Natural, Z.Natural) -> TokenizerOutput (Either String Z.Natural)+packReference = fmap $ Right . snd++-- | __HTML:__+--      @[hexadecimal character reference state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#hexadecimal-character-reference-state)@+-- +-- The parsing instructions for after reading @"&#x"@ or @"&#X"@ followed by a+-- hexadecimal digit in a section of the state machine which allows character+-- references.+tokenHexadecimalCharacterReference+    :: Tokenizer (TokenizerOutput (Z.Natural, Z.Natural))+        -- ^ The inner data contains the number of valid numeric digits to the+        -- right ('fst') and the number composed from those digits ('snd').+        -- +        -- The datatype has been chosen so that even extremely long references+        -- (over a gigabyte of digits /at minimum/ with 'Word') have no chance+        -- of causing system-dependant behaviour; @&#x20;@ and @&#x20..0;@ being+        -- equivalent when the second has 2^30 @0@s (among other alignments)+        -- could potentially allow some obscure attack.+tokenHexadecimalCharacterReference = tokenizer+        (Just ([MissingSemicolonAfterCharacterReference], (0, 0)))+    [ ifChar C.isDigit $ \c -> increment 0x30 c <$> tokenHexadecimalCharacterReference+    , ifChar (range 'A' 'F') $ \c -> increment 0x37 c <$> tokenHexadecimalCharacterReference+    , ifChar (range 'a' 'f') $ \c -> increment 0x57 c <$> tokenHexadecimalCharacterReference+    , if_ (== ';') $ packToken ([], (0, 0))+    , elsePush_ $ packToken ([MissingSemicolonAfterCharacterReference], (0, 0))+    ]+  where increment offset c = fmap $ \(pos, accum) ->+            (pos + 1, fromIntegral (fromEnum c - offset) * (16 ^ pos) + accum)++-- | __HTML:__+--      @[decimal character reference state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#decimal-character-reference-state)@+-- +-- The parsing instructions for after reading @"&#"@ followed by a digit from+-- @0@ through @9@ in a section of the state machine which allows character+-- references.+tokenDecimalCharacterReference+    :: Tokenizer (TokenizerOutput (Z.Natural, Z.Natural))+        -- ^ The inner data contains the number of valid numeric digits to the+        -- right ('fst') and the number composed from those digits ('snd').+        -- +        -- The datatype has been chosen so that even extremely long references+        -- (over a gigabyte of digits /at minimum/ with 'Word') have no chance+        -- of causing system-dependant behaviour; @&#10;@ and @&#10..0;@ being+        -- equivalent when the second has 2^30 @0@s (among other alignments)+        -- could potentially allow some obscure attack.+tokenDecimalCharacterReference = tokenizer+        (Just ([MissingSemicolonAfterCharacterReference], (0, 0)))+    [ ifChar C.isDigit $ \c -> increment c <$> tokenDecimalCharacterReference+    , if_ (== ';') $ packToken ([], (0, 0))+    , elsePush_ $ packToken ([MissingSemicolonAfterCharacterReference], (0, 0))+    ]+  where increment c = fmap $ \(pos, accum) ->+            (pos + 1, fromIntegral (fromEnum c - 0x30) * (10 ^ pos) + accum)++-- | __HTML:__+--      @[numeric character reference end state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#numeric-character-reference-end-state)@+-- +-- The instructions for processing a theoretically-Unicode code point into an+-- actually-legal 'Char', after reading a non-digit character following a+-- @"&#"@ in a section of the state machine which allows character references.+tokenNumericCharacterReferenceEnd :: Z.Natural -> ([ParseError], Char)+tokenNumericCharacterReferenceEnd 0x00 = ([NullCharacterReference], replacementChar)+tokenNumericCharacterReferenceEnd code+    | code > 0x10FFFF =+        ([CharacterReferenceOutsideUnicodeRange code], replacementChar)+    | range 0xD800 0xDFFF code =+        ([SurrogateCharacterReference . toEnum $ fromIntegral code], replacementChar)+    | range 0xFDD0 0xFDEF code =+        ([NoncharacterCharacterReference . toEnum $ fromIntegral code], toEnum $ fromIntegral code)+    | cMod == 0xFFFE || cMod == 0xFFFF =+        ([NoncharacterCharacterReference . toEnum $ fromIntegral code], toEnum $ fromIntegral code)+    | range 0x00 0x1F code && notElem code [0x09, 0x0A, 0x0C] =+        ([ControlCharacterReference], toEnum $ fromIntegral code)+    | range 0x7F 0x9F code = ([ControlCharacterReference],+        Y.fromMaybe (toEnum $ fromIntegral code) . N.join $+        controlReplacement !? (fromIntegral code - 0x7F))+    | otherwise = ([], toEnum $ fromIntegral code)+  where cMod = fromIntegral code :: W.Word16++-- | The specification-defined replacements for the C1 control characters, plus+-- @0x7F@ (DEL).  'Nothing' placeholders are used for controls without a+-- replacement, to allow indexing by the code point minus 0x7F.+controlReplacement :: V.Vector (Maybe Char)+controlReplacement = V.fromList+    [ Nothing       -- 0x7F+    , Just '\x20AC' -- 0x80+    , Nothing       -- 0x81+    , Just '\x201A' -- 0x82+    , Just '\x0192' -- 0x83+    , Just '\x201E' -- 0x84+    , Just '\x2026' -- 0x85+    , Just '\x2020' -- 0x86+    , Just '\x2021' -- 0x87+    , Just '\x02C6' -- 0x88+    , Just '\x2030' -- 0x89+    , Just '\x0160' -- 0x8A+    , Just '\x2039' -- 0x8B+    , Just '\x0152' -- 0x8C+    , Nothing       -- 0x8D+    , Just '\x017D' -- 0x8E+    , Nothing       -- 0x8F+    , Nothing       -- 0x90+    , Just '\x2018' -- 0x91+    , Just '\x2019' -- 0x92+    , Just '\x201C' -- 0x93+    , Just '\x201D' -- 0x94+    , Just '\x2022' -- 0x95+    , Just '\x2013' -- 0x96+    , Just '\x2014' -- 0x97+    , Just '\x02DC' -- 0x98+    , Just '\x2122' -- 0x99+    , Just '\x0161' -- 0x9A+    , Just '\x203A' -- 0x9B+    , Just '\x0153' -- 0x9C+    , Nothing       -- 0x9D+    , Just '\x017E' -- 0x9E+    , Just '\x0178' -- 0x9F+    ]
+ src/Web/Mangrove/Parse/Tokenize/Comment.hs view
@@ -0,0 +1,155 @@+{-|+Description:    Tokenization rules for characters within @\<!--@ ... @--\>@ sections.++Copyright:      (c) 2020 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      stable+Portability:    portable+-}+module Web.Mangrove.Parse.Tokenize.Comment+    ( tokenCommentStart+    , packCommentToken+    ) where+++import qualified Data.Text as T++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tokenize.Common+import Web.Willow.Common.Encoding.Character+++-- | Wrap a given span of 'Char's into the payload of a 'Comment'.+packCommentToken :: TokenizerOutput String -> TokenizerOutput Token+packCommentToken = fmap $ Comment . T.pack+++-- | __HTML:__+--      @[comment start state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#comment-start-state)@+-- +-- The parsing instructions for after reading @"\<!--"@ in a section of the+-- state machine which allows markup declarations.+tokenCommentStart :: Tokenizer (TokenizerOutput String)+tokenCommentStart = tokenizer (Just ([EOFInComment], ""))+    [ if_ (== '-') tokenCommentStartDash+    , if_ (== '>') $ changeState DataState *> packToken ([AbruptClosingOfEmptyComment], "")+    , elsePush_ tokenComment+    ]++-- | __HTML:__+--      @[comment start dash state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#comment-start-state)@+-- +-- The parsing instructions for after reading @"\<!---"@ in a section of the+-- state machine which allows markup declarations.+tokenCommentStartDash :: Tokenizer (TokenizerOutput String)+tokenCommentStartDash = tokenizer (Just ([EOFInComment], ""))+    [ if_ (== '-') tokenCommentEnd+    , if_ (== '>') $ changeState DataState *> packToken ([AbruptClosingOfEmptyComment], "")+    , elsePush_ $ consOut '-' <$> tokenComment+    ]++-- | __HTML:__+--      @[comment state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#comment-state)@+-- +-- The parsing instructions rooted in the HTML comment section of the state+-- machine.+tokenComment :: Tokenizer (TokenizerOutput String)+tokenComment = tokenizer (Just ([EOFInComment], ""))+    [ ifChar (== '<') $ \c -> consOut c <$> tokenCommentLessThanSign+    , if_ (== '-') tokenCommentEndDash+    , if_ (== '\NUL') $ consTokenError UnexpectedNullCharacter . consOut replacementChar <$> tokenComment+    , elseChar $ \c -> consOut c <$> tokenComment+    ]++-- | __HTML:__+--      @[comment less-than sign state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#comment-less-than-sign-state)@+-- +-- The parsing instructions for after reading @"\<"@ in the HTML comment+-- section of the state machine.+tokenCommentLessThanSign :: Tokenizer (TokenizerOutput String)+tokenCommentLessThanSign = tokenizer (Just ([EOFInComment], ""))+    [ ifChar (== '!') $ \c -> consOut c <$> tokenCommentLessThanSignBang+    , ifChar (== '<') $ \c -> consOut c <$> tokenCommentLessThanSign+    , elsePush_ tokenComment+    ]++-- | __HTML:__+--      @[comment less-than sign bang state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#comment-less-than-sign-bang-state)@+-- +-- The parsing instructions for after reading @"\<!"@ in the HTML comment+-- section of the state machine.+tokenCommentLessThanSignBang :: Tokenizer (TokenizerOutput String)+tokenCommentLessThanSignBang = tokenizer (Just ([EOFInComment], ""))+    [ if_ (== '-') tokenCommentLessThanSignBangDash+    , elsePush_ tokenComment+    ]++-- | __HTML:__+--      @[comment less-than sign bang dash state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#comment-less-than-sign-bang-dash-state)@+-- +-- The parsing instructions for after reading @"\<!-"@ in the HTML comment+-- section of the state machine.+tokenCommentLessThanSignBangDash :: Tokenizer (TokenizerOutput String)+tokenCommentLessThanSignBangDash = tokenizer (Just ([EOFInComment], ""))+    [ if_ (== '-') tokenCommentLessThanSignBangDashDash+    , elsePush_ tokenCommentEndDash+    ]++-- | __HTML:__+--      @[comment less-than sign bang dash dash state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#comment-less-than-sign-bang-dash-dash-state)@+-- +-- The parsing instructions for after reading @"\<!--"@ in the HTML comment+-- section of the state machine.+tokenCommentLessThanSignBangDashDash :: Tokenizer (TokenizerOutput String)+tokenCommentLessThanSignBangDashDash = tokenizer (Just ([EOFInComment], ""))+    [ ifPush_ (== '>') tokenCommentEnd+    , elsePush_ $ consTokenError NestedComment <$> tokenCommentEnd+    ]++-- | __HTML:__+--      @[comment end dash state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#comment-end-dash-state)@+-- +-- The parsing instructions for after reading @"-"@ in the HTML comment section+-- of the state machine.+tokenCommentEndDash :: Tokenizer (TokenizerOutput String)+tokenCommentEndDash = tokenizer (Just ([EOFInComment], ""))+    [ if_ (== '-') tokenCommentEnd+    , elsePush_ $ consOut '-' <$> tokenComment+    ]++-- | __HTML:__+--      @[comment end state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#comment-end-state)@+-- +-- The parsing instructions for after reading @"--"@ in the HTML comment+-- section of the state machine.+tokenCommentEnd :: Tokenizer (TokenizerOutput String)+tokenCommentEnd = tokenizer (Just ([EOFInComment], ""))+    [ if_ (== '>') $ changeState DataState *> packToken ([], "")+    , if_ (== '!') tokenCommentEndBang+    , ifChar (== '-') $ \c -> consOut c <$> tokenCommentEnd+    , elsePush_ $ consOuts "--" <$> tokenComment+    ]++-- | __HTML:__+--      @[comment end bang state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#comment-end-bang-state)@+-- +-- The parsing instructions for after reading @"--!"@ in the HTML comment+-- section of the state machine.+tokenCommentEndBang :: Tokenizer (TokenizerOutput String)+tokenCommentEndBang = tokenizer (Just ([EOFInComment], ""))+    [ if_ (== '-') $ consOuts "--!" <$> tokenCommentEndDash+    , if_ (== '>') $ packToken ([IncorrectlyClosedComment], "")+    , elsePush_ $ consOuts "--!" <$> tokenComment+    ]
+ src/Web/Mangrove/Parse/Tokenize/Common.hs view
@@ -0,0 +1,778 @@+{-|+Description:    Functions and objects used to build the tokenizer.++Copyright:      (c) 2020-2021 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      provisional+Portability:    portable++The tokenizer stage is in a uniquely challenging position where the standard's+instructions are given as 'switch' statements (unlike+'Web.Mangrove.Parse.Encoding.Preprocess'), it operates over the output+---including errors and re-entrant state--- of a previous stage (unlike+'Web.Mangrove.Parse.Encoding'), and the conceptual input is of a type, 'Char',+with many constructors and a lot of standard library support (unlike+'Web.Mangrove.Parse.Tree').  That intersection means that it has proven best to+abstract the parser constructors themselves, rather than to abstract the+predicate tests and/or the combinator functions as in the other stages.+-}+module Web.Mangrove.Parse.Tokenize.Common+    ( -- * Types+      -- ** Parser+      Tokenizer+    , TokenizerState ( .. )+    , TokenParserState ( .. )+    , decoderState+    , decoderDefaultState+    , CurrentTokenizerState ( .. )+    , defaultTokenizerState+      -- *** Output+      -- **** Tokenizer+    , TokenizerOutput ( .. )+    , mapErrs+    , continueState+    , endState+    , finalStateList+    , Wrapped+    , WrappedOutput+    , WrappedOutputs+      -- **** Decoder+    , TokenizerInput ( .. )+    , DecoderOutputState+    , decodedRemainder+    , setRemainder+      -- ** Data+    , Token ( .. )+    , DoctypeParams ( .. )+    , emptyDoctypeParams+    , TagParams ( .. )+    , emptyTagParams+    , BasicAttribute+      -- * Parser building+      -- ** Input+      -- *** Single token+    , tokenizer+    , if_+    , ifChar+    , else_+    , elseChar+      -- *** Token list+    , tokenizers+    , ifs_+    , ifsChar+    , elses_+    , elsesChar+      -- *** Reconsume input+    , ifPush_+    , ifPushChar+    , elsePush_+    , elsePushChar+      -- ** Output+    , packToken+    , packState+    , emit+    , emit'+    , consEmit+    , consTokenError+    , consTokenErrors+    , consTokenErrorsList+    , consOut+    , consOuts+      -- ** Combinators+    , appropriateEndTag+    , changeState+    , chunk'+    ) where+++import qualified Control.Applicative as A+import qualified Control.Monad.Trans.State as N.S++import qualified Data.ByteString as BS+import qualified Data.ByteString.Short as BS.SH+import qualified Data.Either as E+import qualified Data.HashMap.Strict as M+import qualified Data.Maybe as Y+import qualified Data.Text as T++import Web.Willow.DOM++import Web.Mangrove.Parse.Common.Error+import Web.Willow.Common.Encoding hiding ( setRemainder )+import Web.Willow.Common.Encoding.Sniffer+import Web.Willow.Common.Parser+import Web.Willow.Common.Parser.Switch++import qualified Web.Willow.Common.Encoding as Willow++import Control.Applicative ( (<|>) )+++-- | The smallest segment of data which carries semantic meaning.+data Token+    = Doctype DoctypeParams+        -- ^ __HTML:__+        --      @DOCTYPE token@+        -- +        -- 'Web.Mangrove.DOM.DocumentType', describing the language used in the+        -- document.+    | StartTag TagParams+        -- ^ __HTML:__+        --      @start tag token@+        -- +        -- 'Web.Mangrove.DOM.Element', marking the start of a markup section,+        -- or a point of markup which (per the specification) doesn't contain+        -- any content.+    | EndTag TagParams+        -- ^ __HTML:__+        --      @end tag token@+        -- +        -- 'Web.Mangrove.DOM.Element' with a @'/'@ character before the name,+        -- marking the end of a section opened by 'StartTag'.+    | Comment T.Text+        -- ^ __HTML:__+        --      @comment token@+        -- +        -- 'Web.Mangrove.DOM.Comment', marking author's notes or other text+        -- about the document itself, rather than being part of the content.+    | Character Char+        -- ^ __HTML:__+        --      @character token@+        -- +        -- 'Web.Mangrove.DOM.Character', usually containing (a small portion+        -- of) text which should rendered for the user or included in the+        -- header metadata, but occasionally subject to further processing+        -- (i.e. the content of @\<script\>@ or @\<style\>@ sections).+    | EndOfStream+        -- ^ __HTML:__+        --      @end-of-file token@+        -- +        -- Represents both an explicit mark of the end of the stream, when a+        -- simple @[]@ doesn't suffice, and provides a seat to carry+        -- 'ParseError's if no other token is emitted at the same time.+        -- +        -- Note: the former role doesn't have any guarantees; a stream can end+        -- without an 'EndOfStream' termination, and 'EndOfStream' tokens can occur in+        -- places other than the end of the file.+  deriving ( Eq, Show, Read )+++-- | __HTML:__+--      the data associated with a @doctype token@+-- +-- All data comprising a document type declaration which may be obtained+-- directly from the raw document stream.  Values may be easily instantiated as+-- updates to 'emptyDoctypeParams'.+data DoctypeParams = DoctypeParams+    { doctypeName :: Maybe T.Text+        -- ^ The root element of the document, which may also identify the+        -- primary language used.+    , doctypePublicId :: Maybe T.Text+        -- ^ A globally-unique reference to the definition of the language.+    , doctypeSystemId :: Maybe T.Text+        -- ^ A system-dependant (but perhaps easier to access) reference to the+        -- definition of the language.+    , doctypeQuirks :: Bool+        -- ^ Whether the document should be read and rendered in a+        -- backwards-compatible manner, even if the other data in the token+        -- would match that expected by the specification.  Note that 'False'+        -- value is still subject to those expectations; this just provides an+        -- override in the case of, for example, a malformed declaration.+    }+  deriving ( Eq, Show, Read )++-- | A sane default collection for easy record initialization; namely,+-- 'Nothing's and 'False'.+emptyDoctypeParams :: DoctypeParams+emptyDoctypeParams = DoctypeParams+    { doctypeName = Nothing+    , doctypePublicId = Nothing+    , doctypeSystemId = Nothing+    , doctypeQuirks = False+    }+++-- | __HTML:__+--      the data associated with a @start tag@ or an @end tag token@+-- +-- All data comprising a markup tag which may be obtained directly from the raw+-- document stream.  Values may be easily instantiated as updates to+-- 'emptyTagParams'.+data TagParams = TagParams+    { tagName :: ElementName+        -- ^ The primary identifier of the markup tag, defining its behaviour+        -- during rendering, and providing a means of matching opening tags+        -- with closing ones.+    , tagIsSelfClosing :: Bool+        -- ^ Whether the tag was closed at the same point it was opened,+        -- according to the XML-style "@/>@" syntax.  HTML null elements are+        -- handled in the tree construction stage instead.+    , tagAttributes :: M.HashMap T.Text T.Text+        -- ^ Finer-grained metadata attached to the markup tag.+    }+  deriving ( Eq, Show, Read )++-- | A sane default collection for easy record initialization.+emptyTagParams :: TagParams+emptyTagParams = TagParams+    { tagName = T.empty+    , tagIsSelfClosing = False+    , tagAttributes = M.empty+    }+++-- | Parser combinators written over the output of the+-- 'Web.Mangrove.Parse.Encoding.decoder' stage, segmenting the raw 'Char'+-- strings into semantic atoms.+type Tokenizer = StateParser TokenParserState [TokenizerInput]+++-- | The collection of data required to extract a list of semantic atoms from a+-- binary document stream.  Values may be easily instantiated as updates to+-- 'defaultTokenizerState'.+data TokenizerState = TokenizerState+    { tokenParserState :: TokenParserState+        -- ^ The state of the current 'Web.Mangrove.Parse.Tokenize.tokenize'+        -- stage.+    , decoderState_ :: Either (Either SnifferEnvironment Encoding) (Maybe DecoderState)+        -- ^ The state of the previous 'Web.Mangrove.Parse.Encoding.decoder'+        -- stage, or the data used to initialize it.  For easy access to the+        -- 'DecoderState' itself, see 'decoderState'.+    }+  deriving ( Eq, Show, Read )++-- | All the data which needs to be tracked for correct behaviour in the+-- tokenization stage.+data TokenParserState = TokenParserState+    { prevStartTag :: Maybe ElementName+        -- ^ __HTML:__+        --      @[appropriate end tag token]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#appropriate-end-tag-token)@+        -- +        -- Certain states in the parser, which only emit 'Character' tokens,+        -- are able to collapse multiple tags which result in that behaviour by+        -- comparing a potential closing markup tag to the name on the+        -- 'StartTag' token which triggered the state.+    , currentState :: CurrentTokenizerState+        -- ^ The set of rules currently active in the state machine.+    , currentNodeNamespace :: Maybe Namespace+        -- ^ Certain states in the parser change behaviour if the @adjusted+        -- current node@ is not an HTML element.  Given the direction of+        -- visibility in the parser stack, this stage can't directly access+        -- that (tree construction level) datum, and so that needs to be+        -- tracked redundently.+    , atEndOfStream :: Bool+        -- ^ Whether the current input stream is known to be the final part of+        -- the document stream ('True') or whether additional input may still+        -- follow ('False') and thus any finalization should not be performed.+    }+  deriving ( Eq, Show, Read )++-- | The various fixed points in the tokenization algorithm, where the parser+-- may break and re-enter seamlessly.+data CurrentTokenizerState+    = DataState+        -- ^ __HTML:__+        --      @[data state]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#data-state)@+        -- +        -- The core rules, providing the most common tokenization behaviour.+    | RCDataState+        -- ^ __HTML:__+        --      @[RCDATA state]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#rcdata-state)@+        -- +        -- 'Character'-focused production while, unlike 'RawTextState',+        -- resolving character reference values.+    | RawTextState+        -- ^ __HTML:__+        --      @[RAWTEXT state]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#rawtext-state)@+        -- +        -- 'Character'-focused production which, unlike 'RCDataState', passes+        -- character reference sequences unchanged.+    | PlainTextState+        -- ^ __HTML:__+        --      @[PLAINTEXT state]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#plaintext-state)@+        -- +        -- Blind conversion of the entire document stream into 'Character'+        -- tokens.+    | ScriptDataState+        -- ^ __HTML:__+        --      @[script data state]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#script-data-state)@+        -- +        -- 'Character'-focused production according to the (occasionally+        -- complex) rules governing the handling of @\<script\>@ contents.+    | ScriptDataEscapedState+        -- ^ __HTML:__+        --      @[script data escaped state]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#script-data-escaped-state)@+        -- +        -- 'Character'-focused production for data within a @\<!--@ / @--\>@+        -- section within 'ScriptDataState'.+    | ScriptDataDoubleEscapedState+        -- ^ __HTML:__+        --      @[script data double escaped state]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#script-data-escaped-state)@+        -- +        -- 'Character'-focused production for data within a @\<script\>@+        -- section within 'ScriptDataEscapedState'.+    | CDataState+        -- ^ __HTML:__+        --      @[CDATA section state]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#cdata-section-state)@+        -- +        -- 'Character'-focused production for data within a foreign @\<[CDATA[@+        -- / @]]\>@ escape section.+  deriving ( Eq, Ord, Bounded, Enum, Show, Read )++-- | A sane default collection for easy record initialization; namely,+-- interpret the binary stream as 'Utf8' in the primary 'DataState'.+defaultTokenizerState :: TokenizerState+defaultTokenizerState = TokenizerState+    { decoderState_ = Left $ Right Utf8+    , tokenParserState = TokenParserState+        { prevStartTag = Nothing+        , currentState = DataState+        , currentNodeNamespace = Nothing+        , atEndOfStream = False+        }+    }+++-- | The state of the previous 'Web.Mangrove.Parse.Encoding.decoder' stage.+-- Note that the high-level conceptual view of the parser stack is of each+-- stage moving along the 'BS.ByteString' as a (more or less) unified front,+-- rather than each stage independently running over the output of the+-- previous.+decoderState :: TokenizerState -> Maybe DecoderState+decoderState = E.fromRight Nothing . decoderState_++-- | As 'decoderState', but generating a default initial state if it doesn't+-- yet exist.+decoderDefaultState :: TokenizerState -> BS.ByteString -> Maybe DecoderState+decoderDefaultState state stream = case decoderState_ state of+    Right dState -> dState+    Left initialize -> Just $ either (flip sniffDecoderState stream) initialDecoderState initialize+++-- | __HTML:__+--      @[appropriate end tag token]+--      (https://html.spec.whatwg.org/multipage/parsing.html#appropriate-end-tag-token)@+-- +-- Check whether the last 'StartTag' token emitted by the tokenizer has a name+-- that matches the given 'String'.+appropriateEndTag :: String -> Tokenizer Bool+appropriateEndTag testName = do+    prevName <- prevStartTag <$> N.S.get+    return $ prevName == Just (T.pack testName)+++-- | Prepare the tokenizer to process the /next/ input 'Char' according to the+-- given set of instructions.+changeState :: CurrentTokenizerState -> Tokenizer ()+changeState newState = do+    state <- N.S.get+    N.S.put $ state+        { currentState = newState+        }+++-- | All data with which to initialize the decoder, to resume as if the state+-- machine transition hadn't been interrupted.  If 'Nothing', the associated+-- 'Char' was emitted in the 'init' of several at once; in this case, the+-- decoder can't be re-entered in exactly the same place with the data wrapped+-- in this type, and so the stream must continue to be processed until the+-- first 'Just' value.+type DecoderOutputState = Maybe (Maybe DecoderState, BS.ByteString)+++-- | A smart wrapper over 'switch' to provide an end-of-stream recovery.  See+-- 'tokenizers' if multiple objects may be returned at once.+tokenizer+    :: Maybe ([ParseError], out)+        -- ^ The return value to use when the stream is empty.+    -> [SwitchCase TokenizerInput Tokenizer (WrappedOutput out)]+        -- ^ If the return value is paired with 'True', the final state of the+        -- input stream is allowed as a fallback; if 'False' (i.e., something+        -- has been 'push'ed back for reconsumption) the returned state is used+        -- as-is, even if that doesn't result in a valid fixed breakpoint.+    -> Tokenizer (TokenizerOutput out)+tokenizer Nothing cases = next >>= switch' cases+tokenizer (Just (errs, out)) cases = (next >>= switch' cases) <|> do+    eos <- atEndOfStream <$> N.S.get+    if eos+        then end *> return recovery+        else A.empty+  where recovery = TokenizerOutput+            { tokenizedErrs = errs+            , tokenizedOut = out+            , tokenizedState = endState+            }++-- | As 'switch', but with an internal marker to indicate whether the final+-- state of the input stream may be retrieved from the given value, or whether+-- it must only reflect the generated output.  See @switches'@ if multiple+-- objects may be returned at once.+switch'+    :: [SwitchCase TokenizerInput Tokenizer (WrappedOutput out)]+    -> TokenizerInput+    -> Tokenizer (TokenizerOutput out)+switch' cases t' = mapState' <$> switch cases t'+  where mapState' (True, out) = mapState (<|> decodedState t') out+        mapState' (False, out) = out+++-- | A smart wrapper over 'switch' to provide an end-of-stream recovery.  See+-- 'tokenizer' if only a single object may be returned.+tokenizers+    :: Maybe [([ParseError], out)]+        -- ^ The return values to use when the stream is empty.+    -> [SwitchCase TokenizerInput Tokenizer (WrappedOutputs out)]+        -- ^ If the return value is paired with 'True', the final state of the+        -- input stream is allowed as a fallback; if 'False' (i.e., something+        -- has been 'push'ed back for reconsumption) the returned state is used+        -- as-is, even if that doesn't result in a valid fixed breakpoint.+    -> Tokenizer [TokenizerOutput out]+tokenizers Nothing cases = next >>= switches' cases+tokenizers (Just recovery) cases = (next >>= switches' cases) <|> do+    eos <- atEndOfStream <$> N.S.get+    if eos+        then end *> return recovery'+        else A.empty+  where recovery' = finalStateList endState . flip map recovery $ \(errs, out) ->+            TokenizerOutput errs out Nothing++-- | As 'switch', but with an internal marker to indicate whether the final+-- state of the input stream may be retrieved from the given value, or whether+-- it must only reflect the generated output.  See @switch'@ if only a single+-- object may be returned.+switches'+    :: [SwitchCase TokenizerInput Tokenizer (WrappedOutputs out)]+    -> TokenizerInput+    -> Tokenizer [TokenizerOutput out]+switches' cases t' = repack <$> switch cases t'+  where repack (_, []) = []+        repack (True, [out']) = [mapState (<|> decodedState t') out']+        repack (False, [out']) = [out']+        repack (useState, t:ts) = t : repack (useState, ts)+++-- | Perform some operation on the list of errors associated with a wrapped+-- output value.+mapErrs :: ([ParseError] -> [ParseError]) -> TokenizerOutput out -> TokenizerOutput out+mapErrs f out = out { tokenizedErrs = f $ tokenizedErrs out }++-- | Perform some operation on the stream state associated with a wrapped+-- output value.+mapState+    :: (DecoderOutputState -> DecoderOutputState)+    -> TokenizerOutput out+    -> TokenizerOutput out+mapState f out = out { tokenizedState = f $ tokenizedState out }++-- | Indicate that the given wrapped value should never be used as a breakpoint+-- for future stages of the parser.+continueState :: TokenizerOutput out -> TokenizerOutput out+continueState = mapState $ const Nothing++-- | Mark the list of wrapped values such that any in the 'init' may not be+-- used as a breakpoint for future stages of the parser, and the 'last' has the+-- given 'tokenizedState'.+finalStateList :: DecoderOutputState -> [TokenizerOutput out] -> [TokenizerOutput out]+finalStateList _ [] = []+finalStateList state [t'] = [t' { tokenizedState = state }]+finalStateList state (t':ts') = continueState t' : finalStateList state ts'+++-- | Wrap an output value with a placeholder state indicating that the value+-- may not be used as a tokenizer re-entry point.  If the value is a 'Token',+-- use 'emit' instead.+packToken :: ([ParseError], out) -> Tokenizer (TokenizerOutput out)+packToken = flip packState Nothing++-- | Wrap an output value such that parsing will continue with the given binary+-- document remainder if 'Just', or such that it can not serve as a breakpoint+-- with 'Nothing'.+packState :: ([ParseError], out) -> DecoderOutputState -> Tokenizer (TokenizerOutput out)+packState (errs, out) dState = return $ TokenizerOutput+    { tokenizedErrs = errs+    , tokenizedOut = out+    , tokenizedState = dState+    }+++-- | Wrap a semantic atom and associated errors into the form the tokenizing+-- parsers expect to output.  See 'packToken' if the 'snd' item in the tuple is+-- not a 'Token', or 'emit'' to easily wrap the output as a singleton list.+emit :: ([ParseError], Token) -> Tokenizer (TokenizerOutput Token)+emit t'@(_, StartTag d) = do+    state <- N.S.get+    N.S.put $ state+        { prevStartTag = Just $ tagName d+        }+    packToken t'+emit t'@(_, EndTag d) = consTokenErrors errs <$> packToken t'+  where errs = Y.catMaybes [attrError, closeError]+        attrError+            | null (tagAttributes d) = Nothing+            | otherwise = Just EndTagWithAttributes+        closeError+            | tagIsSelfClosing d = Just EndTagWithTrailingSolidus+            | otherwise = Nothing+emit t' = packToken t'++-- | Wrap a semantic atom and associated errors into the form the tokenizing+-- parsers expect to output, and pack it as a singleton list.  See 'packToken'+-- if the 'snd' item in the tuple is not a 'Token', or 'emit' if the outer list+-- is unnecessary.+emit' :: ([ParseError], Token) -> Tokenizer [TokenizerOutput Token]+emit' = fmap (: []) . emit++-- | Wrap a semantic atom and associated errors into the form the tokenizing+-- parsers expect to output, and add it at the head of a list of the same+-- generated by a separate parser function.+consEmit+    :: ([ParseError], Token)+    -> Tokenizer [TokenizerOutput Token]+    -> Tokenizer [TokenizerOutput Token]+consEmit tok p = do+    t <- emit tok+    ts <- p+    return $ t : ts+++-- | The collection of data returned by the "Web.Mangrove.Parse.Decode"+-- stage, and so comprising the input to the tokenizer.+data TokenizerInput = TokenizerInput+    { decodedErrs :: [ParseError]+        -- ^ Any authoring errors detected during decoding.+    , decodedOut :: Char+        -- ^ The decoded character itself.+    , decodedState :: DecoderOutputState+        -- ^ The data required to resume decoding immediately following the+        -- value, if possible.  See also 'decodedRemainder'.+    }+  deriving ( Eq, Show, Read )++-- | The unparsed portion of the binary stream, /after/ parsing the associated+-- 'Char'.  See also 'decodedState'.+decodedRemainder :: TokenizerInput -> Maybe BS.ByteString+decodedRemainder = fmap snd . decodedState+++-- | Store the given binary sequence as unparsable without further input, to be+-- prepended to the beginning of stream on the next call to+-- 'Web.Mangrove.Parse.Tokenize.tokenize'.+setRemainder :: BS.SH.ShortByteString -> TokenizerState -> TokenizerState+setRemainder bs state = state+    { decoderState_ = Right $ Willow.setRemainder bs <$> decoderDefaultState state BS.empty+    }+++-- | The standard output of parsers used in the tokenization stage.+-- Specifically, it contains the final state of the decoder stage in addition+-- to the generated value, to enable the recursion loop to detect the end of+-- multi-'Char' outputs and properly update the resume state.  May be easily+-- instantiated through calls to 'packToken' and 'packState'.+data TokenizerOutput out = TokenizerOutput+    { tokenizedErrs :: [ParseError]+        -- ^ Any authoring errors detected during decoding or tokenization.+    , tokenizedOut :: out+        -- ^ The point of data specifically generated by the parser.+    , tokenizedState :: DecoderOutputState+        -- ^ The data required to resume tokenization immediately following the+        -- value, if possible.+    }+  deriving ( Eq, Show, Read )++instance Functor TokenizerOutput where+    fmap f out = out { tokenizedOut = f $ tokenizedOut out }+++-- | Type-level abstraction for values output by any tokenize-stage parser,+-- along with the flag required to determine the proper remainder of the input+-- stream to return.+type Wrapped out = (Bool, out)++-- | Type-level abstraction for single values output by a tokenizer, along with+-- the flag required to determine the proper remainder of the input stream to+-- return.+type WrappedOutput out = Wrapped (TokenizerOutput out)++-- | Type-level abstraction for multiple values output by a tokenizer, along+-- with the flag required to determine the proper remainder of the input stream+-- to return.+type WrappedOutputs out = Wrapped [TokenizerOutput out]+++-- | Wrap an 'If_' clause returning a single value, with a more user-friendly+-- predicate type than would otherwise be required by a raw 'SwitchCase' in the+-- tokenizer stage.+if_+    :: (Char -> Bool)+    -> Tokenizer (TokenizerOutput out)+    -> SwitchCase TokenizerInput Tokenizer (WrappedOutput out)+if_ f t = If (f . decodedOut) $ \c' ->+    (,) True . mapErrs (decodedErrs c' ++) <$> t++-- | Wrap an 'If_' clause returning a list of semantic atoms, with a more+-- user-friendly predicate type than would otherwise be required by a raw+-- 'SwitchCase' in the tokenizer stage.+ifs_+    :: (Char -> Bool)+    -> Tokenizer [TokenizerOutput Token]+    -> SwitchCase TokenizerInput Tokenizer (WrappedOutputs Token)+ifs_ f t = If (f . decodedOut) $ \c' ->+    (,) True . consTokenErrorsList (decodedErrs c') <$> t+++-- | Wrap an 'If' clause returning a single value, with more user-friendly+-- predicate and monad types than would otherwise be required by a raw+-- 'SwitchCase' in the tokenizer stage.+ifChar+    :: (Char -> Bool)+    -> (Char -> Tokenizer (TokenizerOutput out))+    -> SwitchCase TokenizerInput Tokenizer (WrappedOutput out)+ifChar f t = If (f . decodedOut) $ \c' ->+    (,) True . mapErrs (decodedErrs c' ++) <$> t (decodedOut c')++-- | Wrap an 'If' clause returning a list of semantic atoms, with more+-- user-friendly predicate and monad types than would otherwise be required by+-- a raw 'SwitchCase' in the tokenizer stage.+ifsChar+    :: (Char -> Bool)+    -> (Char -> Tokenizer [TokenizerOutput Token])+    -> SwitchCase TokenizerInput Tokenizer (WrappedOutputs Token)+ifsChar f t = If (f . decodedOut) $ \c' ->+    (,) True . consTokenErrorsList (decodedErrs c') <$> t (decodedOut c')+++-- | Wrap an 'If_' clause returning a single value, with a more user-friendly+-- predicate type, and return the input character to the stream for+-- reconsumption.+ifPush_+    :: (Char -> Bool)+    -> Tokenizer out+    -> SwitchCase TokenizerInput Tokenizer (Wrapped out)+ifPush_ f t = If (f . decodedOut) $ \c' ->+    push c' *> fmap ((,) False) t++-- | Wrap an 'If' clause returning a single value, with more user-friendly+-- predicate and monad types, and return the input character to the stream for+-- reconsumption.+ifPushChar+    :: (Char -> Bool)+    -> (Char -> Tokenizer out)+    -> SwitchCase TokenizerInput Tokenizer (Wrapped out)+ifPushChar f t = If (f . decodedOut) $ \c' ->+    push c' *> fmap ((,) False) (t $ decodedOut c')+++-- | Wrap an 'Else_' clause returning a single value, performing any relevant+-- back-end processing.+else_+    :: Tokenizer (TokenizerOutput out)+    -> SwitchCase TokenizerInput Tokenizer (WrappedOutput out)+else_ t = Else $ \c' ->+    (,) True . mapErrs (decodedErrs c' ++) <$> t++-- | Wrap an 'Else_' clause returning a list of semantic atoms, performing any+-- relevant back-end processing.+elses_+    :: Tokenizer [TokenizerOutput Token]+    -> SwitchCase TokenizerInput Tokenizer (WrappedOutputs Token)+elses_ t = Else $ \c' ->+    (,) True . consTokenErrorsList (decodedErrs c') <$> t+++-- | Wrap 'Else' clause returning a single value, with a more user-friendly+-- monad input type than would otherwise be required by a raw 'SwitchCase' in+-- the tokenizer stage.+elseChar+    :: (Char -> Tokenizer (TokenizerOutput out))+    -> SwitchCase TokenizerInput Tokenizer (WrappedOutput out)+elseChar t = Else $ \c' ->+    (,) True . mapErrs (decodedErrs c' ++) <$> t (decodedOut c')++-- | Wrap an 'Else' clause returning a list of semantic atoms, with a more+-- user-friendly monad input type than would otherwise be required by a raw+-- 'SwitchCase' in the tokenizer stage.+elsesChar+    :: (Char -> Tokenizer [TokenizerOutput Token])+    -> SwitchCase TokenizerInput Tokenizer (WrappedOutputs Token)+elsesChar t = Else $ \c' ->+    (,) True . consTokenErrorsList (decodedErrs c') <$> t (decodedOut c')+++-- | Wrap an 'Else_' clause returning a single value, performing any relevant+-- back-end processing, and return the input character to the stream for+-- reconsumption.+elsePush_+    :: Tokenizer out+    -> SwitchCase TokenizerInput Tokenizer (Wrapped out)+elsePush_ t = Else $ \c' ->+    push c' *> fmap ((,) False) t++-- | Wrap an 'Else' clause returning a single value, with a more user-friendly+-- monad type, and return the input character to the stream for reconsumption.+elsePushChar+    :: (Char -> Tokenizer out)+    -> SwitchCase TokenizerInput Tokenizer (Wrapped out)+elsePushChar t = Else $ \c' ->+    push c' *> fmap ((,) False) (t $ decodedOut c')+++-- | Add the given 'ParseError' to the collection of warnings associated with+-- the wrapped value.+consTokenError :: ParseError -> TokenizerOutput out -> TokenizerOutput out+consTokenError err = mapErrs (err :)++-- | Add the given 'ParseError's to the collection of warnings associated with+-- the wrapped value.+consTokenErrors :: [ParseError] -> TokenizerOutput out -> TokenizerOutput out+consTokenErrors errs = mapErrs (errs ++)++-- | Add the given 'ParseError's to the collection of warnings associated with+-- the first wrapped value in the list, or create a placeholder if it does not+-- yet contain any 'Token's.+consTokenErrorsList :: [ParseError] -> [TokenizerOutput Token] -> [TokenizerOutput Token]+consTokenErrorsList [] [] = []+consTokenErrorsList errs [] = [TokenizerOutput errs EndOfStream Nothing]+consTokenErrorsList errs (t:ts) = consTokenErrors errs t : ts+++-- | Push the given value to the head of the wrapped collection.+-- +-- @+-- consOut x == fmap (x :)+-- @+consOut :: out -> TokenizerOutput [out] -> TokenizerOutput [out]+consOut x = fmap (x :)++-- | Push the given values to the head of the wrapped collection.+-- +-- @+-- consOut xs == fmap (xs ++)+-- @+consOuts :: [out] -> TokenizerOutput [out] -> TokenizerOutput [out]+consOuts xs = fmap (xs ++)+++-- | Lift a call to 'chunk' over the wrapped 'Char's which make up the input to+-- the tokenization stage.  As an additional quality-of-life, perform some+-- function on the input 'String' before comparing it to the desired value; for+-- behaviour according to the base 'chunk' function, use @'chunk'' 'id'@.+chunk' :: (Char -> Char) -> String -> Tokenizer [TokenizerInput]+chunk' f test = nextChunk (fromIntegral $ length test) >>=+    satisfying (\str -> map (f . decodedOut) str == test)+++-- | Produce a placeholder value for use in generating a 'TokenizerOutput'+-- value located at the end of the byte stream.+endState :: DecoderOutputState+endState = Just (Nothing, BS.empty)
+ src/Web/Mangrove/Parse/Tokenize/Data.hs view
@@ -0,0 +1,34 @@+{-|+Description:    Basic tokenization rules for the character stream.++Copyright:      (c) 2020-2021 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      stable+Portability:    portable+-}+module Web.Mangrove.Parse.Tokenize.Data+    ( tokenData+    ) where+++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tokenize.Common+import Web.Mangrove.Parse.Tokenize.Character+import Web.Mangrove.Parse.Tokenize.Tag+++-- | __HTML:__+--      @[data state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#data-state)@+-- +-- The parsing instructions rooted in the 'DataState' section of the state+-- machine.+tokenData :: Tokenizer [TokenizerOutput Token]+tokenData = tokenizers (Just [([], EndOfStream)])+    [ ifs_ (== '&') $ tokenCharacterReference False >>= flushCharRef+    , ifs_ (== '<') tokenTagOpen+    , ifs_ (== '\NUL') $ emit' ([UnexpectedNullCharacter], Character '\NUL')+    , elsesChar $ \c -> emit' ([], Character c)+    ]
+ src/Web/Mangrove/Parse/Tokenize/Dispatcher.hs view
@@ -0,0 +1,42 @@+{-|+Description:    Central directory for tokenization rules.++Copyright:      (c) 2020 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      stable+Portability:    portable+-}+module Web.Mangrove.Parse.Tokenize.Dispatcher+    ( dispatcher+    ) where+++import qualified Control.Monad.Trans.State as N.S++import Web.Mangrove.Parse.Tokenize.CData+import Web.Mangrove.Parse.Tokenize.Common+import Web.Mangrove.Parse.Tokenize.Data+import Web.Mangrove.Parse.Tokenize.RawText+import Web.Mangrove.Parse.Tokenize.RCData+import Web.Mangrove.Parse.Tokenize.PlainText+import Web.Mangrove.Parse.Tokenize.ScriptData+import Web.Mangrove.Parse.Tokenize.ScriptDataEscaped+import Web.Mangrove.Parse.Tokenize.ScriptDataDoubleEscaped+++-- | Delegate parsing the binary stream to the appropriate part of the finite+-- state machine, according to the top-level 'CurrentTokenizerState'.+dispatcher :: Tokenizer [TokenizerOutput Token]+dispatcher = do+    state <- N.S.get+    case currentState state of+        DataState -> tokenData+        RCDataState -> tokenRCData+        RawTextState -> tokenRawText+        PlainTextState -> fmap (: []) tokenPlainText+        ScriptDataState -> tokenScriptData+        ScriptDataEscapedState -> tokenScriptDataEscaped+        ScriptDataDoubleEscapedState -> tokenScriptDataDoubleEscaped+        CDataState -> tokenCDataSection
+ src/Web/Mangrove/Parse/Tokenize/Dispatcher.hs-boot view
@@ -0,0 +1,14 @@+{-|+Copyright:      (c) 2020 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com+-}+module Web.Mangrove.Parse.Tokenize.Dispatcher+    ( dispatcher+    ) where+++import Web.Mangrove.Parse.Tokenize.Common+++dispatcher :: Tokenizer [TokenizerOutput Token]
+ src/Web/Mangrove/Parse/Tokenize/Doctype.hs view
@@ -0,0 +1,383 @@+{-|+Description:    Tokenization rules for characters comprising a @\<!DOCTYPE\>@ declaration.++Copyright:      (c) 2020 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      stable+Portability:    portable+-}+module Web.Mangrove.Parse.Tokenize.Doctype+    ( tokenDoctype+    ) where+++import qualified Data.Char as C+import qualified Data.Text as T++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tokenize.Common+import Web.Willow.Common.Encoding.Character+import Web.Willow.Common.Parser.Util++import Control.Applicative ( (<|>) )+++-- | An intermediate form of 'DoctypeParams' with unpacked parameter types to+-- facilitate construction.  Specifically, 'Char'-by-'Char' construction of a+-- 'String' followed by a single 'T.pack' is much faster than repeated @O(n)@+-- calls to 'T.cons'.  Values may be easily instantiated as updates to+-- @emptyDoctypeData@ or @quirksDoctype@.+data DoctypeParams' = DoctypeParams'+    { doctypeName' :: Maybe String+        -- ^ See 'doctypeName'.+    , doctypePublicId' :: Maybe String+        -- ^ See 'doctypePublicId'.+    , doctypeSystemId' :: Maybe String+        -- ^ See 'doctypeSystemId'.+    , doctypeQuirks' :: Bool+        -- ^ See 'doctypeQuirks'.+    }+  deriving ( Eq, Show, Read )++-- | A sane default collection for easy record initialization; namely,+-- 'Nothing's and 'False'.  See also @quirksDoctype@.+emptyDoctypeData :: DoctypeParams'+emptyDoctypeData = DoctypeParams'+    { doctypeName' = Nothing+    , doctypePublicId' = Nothing+    , doctypeSystemId' = Nothing+    , doctypeQuirks' = False+    }++-- | As like @emptyDoctypeData@, but indicating that the document should be+-- forced to render in the quirks mode.+quirksDoctype :: DoctypeParams'+quirksDoctype = emptyDoctypeData+    { doctypeQuirks' = True+    }+++-- | Wrap a given collection of data into the payload of a 'Doctype'.+packDoctypeToken :: TokenizerOutput DoctypeParams' -> TokenizerOutput Token+packDoctypeToken d' = flip fmap d' $ \d -> Doctype $ emptyDoctypeParams+    { doctypeName = T.pack <$> doctypeName' d+    , doctypePublicId = T.pack <$> doctypePublicId' d+    , doctypeSystemId = T.pack <$> doctypeSystemId' d+    , doctypeQuirks = doctypeQuirks' d+    }+++-- | Mark a collection of data as containing a doctype public identifier,+-- without overwriting any data which may already be set.+ensurePublicId :: TokenizerOutput DoctypeParams' -> TokenizerOutput DoctypeParams'+ensurePublicId = fmap $ \d -> d+    { doctypePublicId' = doctypePublicId' d <|> Just ""+    }++-- | Mark a collection of data as containing a doctype system identifier,+-- without overwriting any data which may already be set.+ensureSystemId :: TokenizerOutput DoctypeParams' -> TokenizerOutput DoctypeParams'+ensureSystemId = fmap $ \d -> d+    { doctypeSystemId' = doctypeSystemId' d <|> Just ""+    }++-- | Mark a collection of data as forcing a document to render in quirks mode.+ensureQuirks :: TokenizerOutput DoctypeParams' -> TokenizerOutput DoctypeParams'+ensureQuirks = fmap $ \d -> d+    { doctypeQuirks' = True+    }+++-- | Prepend a character to the unpacked name of the root element -- or, in+-- other words, the markup language used.+consDoctypeName :: Char -> TokenizerOutput DoctypeParams' -> TokenizerOutput DoctypeParams'+consDoctypeName c = fmap $ \d -> d+    { doctypeName' = fmap (c :) $ doctypeName' d <|> Just []+    }++-- | Prepend a character to the unpacked public identifier of a doctype.+consPublicId :: Char -> TokenizerOutput DoctypeParams' -> TokenizerOutput DoctypeParams'+consPublicId c = fmap $ \d -> d+    { doctypePublicId' = fmap (c :) $ doctypePublicId' d <|> Just []+    }++-- | Prepend a character to the unpacked system identifier of a doctype.+consSystemId :: Char -> TokenizerOutput DoctypeParams' -> TokenizerOutput DoctypeParams'+consSystemId c = fmap $ \d -> d+    { doctypeSystemId' = fmap (c :) $ doctypeSystemId' d <|> Just []+    }+++-- | __HTML:__+--      @[DOCTYPE state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#doctype-state)@+-- +-- The parsing instructions for after reading @"\<!DOCTYPE"@ in a section of the+-- state machine which allows markup declarations.+tokenDoctype :: Tokenizer (TokenizerOutput Token)+tokenDoctype = tokenizer (Just ([EOFInDoctype], quirksDoctype'))+    [ if_ isAsciiWhitespace $ fmap packDoctypeToken tokenBeforeDoctypeName+    , ifPush_ (== '>') $ fmap packDoctypeToken tokenBeforeDoctypeName+    , elsePush_ $ fmap (packDoctypeToken . consTokenError MissingWhitespaceBeforeDoctypeName)+        tokenBeforeDoctypeName+    ]+  where quirksDoctype' = Doctype $ emptyDoctypeParams+            { doctypeQuirks = True+            }++-- | __HTML:__+--      @[before DOCTYPE name state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#before-doctype-name-state)@+-- +-- The parsing instructions for after reading @"\<!DOCTYPE "@ in the doctype+-- declaration section of the state machine.+tokenBeforeDoctypeName :: Tokenizer (TokenizerOutput DoctypeParams')+tokenBeforeDoctypeName = tokenizer (Just ([EOFInDoctype], quirksDoctype))+    [ if_ isAsciiWhitespace tokenBeforeDoctypeName+    , if_ (== '\NUL') $ consTokenError UnexpectedNullCharacter . consDoctypeName replacementChar <$>+        tokenDoctypeName+    , if_ (== '>') $ packToken ([MissingDoctypeName], quirksDoctype)+    , elseChar $ \c -> consDoctypeName (toAsciiLower c) <$> tokenDoctypeName+    ]++-- | __HTML:__+--      @[DOCTYPE name state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#doctype-name-state)@+-- +-- The parsing instructions at the initial segment of the doctype declaration+-- section of the state machine.+tokenDoctypeName :: Tokenizer (TokenizerOutput DoctypeParams')+tokenDoctypeName = tokenizer (Just ([EOFInDoctype], quirksDoctype))+    [ if_ isAsciiWhitespace tokenAfterDoctypeName+    , if_ (== '>') $ changeState DataState *> packToken ([], emptyDoctypeData)+    , if_ (== '\NUL') $ consTokenError UnexpectedNullCharacter . consDoctypeName replacementChar <$>+        tokenDoctypeName+    , elseChar $ \c -> consDoctypeName (toAsciiLower c) <$> tokenDoctypeName+    ]++-- | __HTML:__+--      @[after DOCTYPE name state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#after-doctype-name-state)@+-- +-- The parsing instructions for after reading the root element in the doctype+-- declaration section of the state machine.+tokenAfterDoctypeName :: Tokenizer (TokenizerOutput DoctypeParams')+tokenAfterDoctypeName = tokenizer (Just ([EOFInDoctype], quirksDoctype))+    [ if_ isAsciiWhitespace tokenAfterDoctypeName+    , if_ (== '>') $ changeState DataState *> packToken ([], emptyDoctypeData)+    , elsePush_ tokenAfterDoctypeName'+    ]++-- | Dispatch the parser according to a @"PUBLIC"@ or @"SYSTEM"@ keyword in the+-- doctype declaration.+tokenAfterDoctypeName' :: Tokenizer (TokenizerOutput DoctypeParams')+tokenAfterDoctypeName' = choice+    [ chunk' C.toUpper "PUBLIC" *> tokenAfterDoctypePublicKeyword+    , chunk' C.toUpper "SYSTEM" *> tokenAfterDoctypeSystemKeyword+    , consTokenError InvalidCharacterSequenceAfterDoctypeName . ensureQuirks <$> tokenBogusDoctype+    ]++-- | __HTML:__+--      @[after DOCTYPE public keyword state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#after-doctype-public-keyword-state)@+-- +-- The parsing instructions for after reading @"PUBLIC"@ in the doctype+-- declaration section of the state machine.+tokenAfterDoctypePublicKeyword :: Tokenizer (TokenizerOutput DoctypeParams')+tokenAfterDoctypePublicKeyword = tokenizer (Just ([EOFInDoctype], quirksDoctype))+    [ if_ isAsciiWhitespace tokenBeforeDoctypePublicIdentifier+    , if_ (== '"') $ consTokenError MissingWhitespaceAfterDoctypePublicKeyword . ensurePublicId <$>+        tokenDoctypePublicIdentifierDoubleQuoted+    , if_ (== '\'') $ consTokenError MissingWhitespaceAfterDoctypePublicKeyword . ensurePublicId <$>+        tokenDoctypePublicIdentifierSingleQuoted+    , if_ (== '>') $ changeState DataState *>+        packToken ([MissingDoctypePublicIdentifier], quirksDoctype)+    , elsePush_ $ consTokenError MissingQuoteBeforeDoctypePublicIdentifier . ensureQuirks <$>+        tokenBogusDoctype+    ]++-- | __HTML:__+--      @[before DOCTYPE public identifier state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#before-doctype-public-identifier-state)@+-- +-- The parsing instructions for after reading @"PUBLIC "@ in the doctype+-- declaration section of the state machine.+tokenBeforeDoctypePublicIdentifier :: Tokenizer (TokenizerOutput DoctypeParams')+tokenBeforeDoctypePublicIdentifier = tokenizer (Just ([EOFInDoctype], quirksDoctype))+    [ if_ isAsciiWhitespace tokenBeforeDoctypePublicIdentifier+    , if_ (== '"') $ fmap ensurePublicId tokenDoctypePublicIdentifierDoubleQuoted+    , if_ (== '\'') $ fmap ensurePublicId tokenDoctypePublicIdentifierSingleQuoted+    , if_ (== '>') $ changeState DataState *>+        packToken ([MissingDoctypePublicIdentifier], quirksDoctype)+    , elsePush_ $ consTokenError MissingQuoteBeforeDoctypePublicIdentifier . ensureQuirks <$>+        tokenBogusDoctype+    ]++-- | __HTML:__+--      @[DOCTYPE public identifier (double-quoted) state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#doctype-public-identifier-(double-quoted\)-state)@+-- +-- The parsing instructions for after reading @"PUBLIC \\""@ in the doctype+-- declaration section of the state machine.+tokenDoctypePublicIdentifierDoubleQuoted :: Tokenizer (TokenizerOutput DoctypeParams')+tokenDoctypePublicIdentifierDoubleQuoted = tokenizer (Just ([EOFInDoctype], quirksDoctype))+    [ if_ (== '"') tokenAfterDoctypePublicIdentifier+    , if_ (== '\NUL') $ consTokenError UnexpectedNullCharacter . consPublicId replacementChar <$>+        tokenDoctypePublicIdentifierDoubleQuoted+    , if_ (== '>') $ changeState DataState *> +        packToken ([AbruptDoctypePublicIdentifier], quirksDoctype)+    , elseChar $ \c -> consPublicId c <$> tokenDoctypePublicIdentifierDoubleQuoted+    ]++-- | __HTML:__+--      @[DOCTYPE public identifier (single-quoted) state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#doctype-public-identifier-(single-quoted\)-state)@+-- +-- The parsing instructions for after reading @"PUBLIC '"@ in the doctype+-- declaration section of the state machine.+tokenDoctypePublicIdentifierSingleQuoted :: Tokenizer (TokenizerOutput DoctypeParams')+tokenDoctypePublicIdentifierSingleQuoted = tokenizer (Just ([EOFInDoctype], quirksDoctype))+    [ if_ (== '\'') tokenAfterDoctypePublicIdentifier+    , if_ (== '\NUL') $ consTokenError UnexpectedNullCharacter . consPublicId replacementChar <$>+        tokenDoctypePublicIdentifierSingleQuoted+    , if_ (== '>') $ changeState DataState *> +        packToken ([AbruptDoctypePublicIdentifier], quirksDoctype)+    , elseChar $ \c -> consPublicId c <$> tokenDoctypePublicIdentifierSingleQuoted+    ]++-- | __HTML:__+--      @[after DOCTYPE public identifier state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#after-doctype-public-identifier-state)@+-- +-- The parsing instructions for after reading the closing quote of a public+-- identifier in the doctype declaration section of the state machine.+tokenAfterDoctypePublicIdentifier :: Tokenizer (TokenizerOutput DoctypeParams')+tokenAfterDoctypePublicIdentifier = tokenizer (Just ([EOFInDoctype], quirksDoctype))+    [ if_ isAsciiWhitespace tokenBetweenDoctypePublicAndSystemIdentifiers+    , if_ (== '>') $ changeState DataState *> packToken ([], emptyDoctypeData)+    , if_ (== '"') $+        consTokenError MissingWhitespaceBetweenDoctypePublicAndSystemIdentifiers . ensureSystemId <$>+        tokenDoctypeSystemIdentifierDoubleQuoted+    , if_ (== '\'') $+        consTokenError MissingWhitespaceBetweenDoctypePublicAndSystemIdentifiers . ensureSystemId <$>+        tokenDoctypeSystemIdentifierSingleQuoted+    , elsePush_ $+        consTokenError MissingQuoteBeforeDoctypeSystemIdentifier . ensureQuirks <$>+        tokenBogusDoctype+    ]++-- | __HTML:__+--      @[between DOCTYPE public and system identifier state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#between-doctype-public-and-system-identifiers-state)@+-- +-- The parsing instructions for after reading a whitespace character after the+-- closing quote of a public identifier in the doctype declaration section of+-- the state machine.+tokenBetweenDoctypePublicAndSystemIdentifiers :: Tokenizer (TokenizerOutput DoctypeParams')+tokenBetweenDoctypePublicAndSystemIdentifiers = tokenizer (Just ([EOFInDoctype], quirksDoctype))+    [ if_ isAsciiWhitespace tokenBetweenDoctypePublicAndSystemIdentifiers+    , if_ (== '>') $ changeState DataState *> packToken ([], emptyDoctypeData)+    , if_ (== '"') $ fmap ensureSystemId tokenDoctypeSystemIdentifierDoubleQuoted+    , if_ (== '\'') $ fmap ensureSystemId tokenDoctypeSystemIdentifierSingleQuoted+    , elsePush_ $ consTokenError MissingQuoteBeforeDoctypeSystemIdentifier . ensureQuirks <$>+        tokenBogusDoctype+    ]++-- | __HTML:__+--      @[after DOCTYPE system keyword state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#after-doctype-system-keyword-state)@+-- +-- The parsing instructions for after reading @"SYSTEM"@ in the doctype+-- declaration section of the state machine.+tokenAfterDoctypeSystemKeyword :: Tokenizer (TokenizerOutput DoctypeParams')+tokenAfterDoctypeSystemKeyword = tokenizer (Just ([EOFInDoctype], quirksDoctype))+    [ if_ isAsciiWhitespace tokenBeforeDoctypeSystemIdentifier+    , if_ (== '"') $ consTokenError MissingWhitespaceAfterDoctypeSystemKeyword . ensureSystemId <$>+        tokenDoctypeSystemIdentifierDoubleQuoted+    , if_ (== '\'') $ consTokenError MissingWhitespaceAfterDoctypeSystemKeyword . ensureSystemId <$>+        tokenDoctypeSystemIdentifierSingleQuoted+    , if_ (== '>') $ changeState DataState *>+        packToken ([MissingDoctypeSystemIdentifier], quirksDoctype)+    , elsePush_ $ consTokenError MissingQuoteBeforeDoctypeSystemIdentifier . ensureQuirks <$>+        tokenBogusDoctype+    ]++-- | __HTML:__+--      @[before DOCTYPE system identifier state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#before-doctype-system-identifier-state)@+-- +-- The parsing instructions for after reading @"SYSTEM "@ in the doctype+-- declaration section of the state machine.+tokenBeforeDoctypeSystemIdentifier :: Tokenizer (TokenizerOutput DoctypeParams')+tokenBeforeDoctypeSystemIdentifier = tokenizer (Just ([EOFInDoctype], quirksDoctype))+    [ if_ isAsciiWhitespace tokenBeforeDoctypeSystemIdentifier+    , if_ (== '"') $ fmap ensureSystemId tokenDoctypeSystemIdentifierDoubleQuoted+    , if_ (== '\'') $ fmap ensureSystemId tokenDoctypeSystemIdentifierSingleQuoted+    , if_ (== '>') $ changeState DataState *>+        packToken ([MissingDoctypeSystemIdentifier], quirksDoctype)+    , elsePush_ $ consTokenError MissingQuoteBeforeDoctypeSystemIdentifier . ensureQuirks <$>+        tokenBogusDoctype+    ]++-- | __HTML:__+--      @[DOCTYPE system identifier (double-quoted) state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#doctype-system-identifier-(double-quoted\)-state)@+-- +-- The parsing instructions for after reading @"SYSTEM \\""@, or a @'"'@ after+-- a public identifier, in the doctype declaration section of the state+-- machine.+tokenDoctypeSystemIdentifierDoubleQuoted :: Tokenizer (TokenizerOutput DoctypeParams')+tokenDoctypeSystemIdentifierDoubleQuoted = tokenizer (Just ([EOFInDoctype], quirksDoctype))+    [ if_ (== '"') tokenAfterDoctypeSystemIdentifier+    , if_ (== '\NUL') $ consTokenError UnexpectedNullCharacter . consSystemId replacementChar <$>+        tokenDoctypeSystemIdentifierDoubleQuoted+    , if_ (== '>') $ changeState DataState *> +        packToken ([AbruptDoctypeSystemIdentifier], quirksDoctype)+    , elseChar $ \c -> consSystemId c <$> tokenDoctypeSystemIdentifierDoubleQuoted+    ]++-- | __HTML:__+--      @[DOCTYPE system identifier (single-quoted) state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#doctype-system-identifier-(single-quoted\)-state)@+-- +-- The parsing instructions for after reading @"SYSTEM '"@, or a @'\''@ after+-- a public identifier, in the doctype declaration section of the state+-- machine.+tokenDoctypeSystemIdentifierSingleQuoted :: Tokenizer (TokenizerOutput DoctypeParams')+tokenDoctypeSystemIdentifierSingleQuoted = tokenizer (Just ([EOFInDoctype], quirksDoctype))+    [ if_ (== '\'') tokenAfterDoctypeSystemIdentifier+    , if_ (== '\NUL') $ consTokenError UnexpectedNullCharacter . consSystemId replacementChar <$>+        tokenDoctypeSystemIdentifierSingleQuoted+    , if_ (== '>') $ changeState DataState *> +        packToken ([AbruptDoctypeSystemIdentifier], quirksDoctype)+    , elseChar $ \c -> consSystemId c <$> tokenDoctypeSystemIdentifierSingleQuoted+    ]++-- | __HTML:__+--      @[after DOCTYPE system identifier state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#after-doctype-system-identifier-state)@+-- +-- The parsing instructions for after reading the closing quote of a system+-- identifier in the doctype declaration section of the state machine.+tokenAfterDoctypeSystemIdentifier :: Tokenizer (TokenizerOutput DoctypeParams')+tokenAfterDoctypeSystemIdentifier = tokenizer (Just ([EOFInDoctype], quirksDoctype))+    [ if_ isAsciiWhitespace tokenAfterDoctypeSystemIdentifier+    , if_ (== '>') $ changeState DataState *> packToken ([], emptyDoctypeData)+    , elsePushChar $ \c -> consTokenError (UnexpectedCharacterAfterDoctypeSystemIdentifier c) <$>+        tokenBogusDoctype+    ]++-- | __HTML:__+--      @[bogus DOCTYPE state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#bogus-doctype-state)@+-- +-- The parsing instructions for after reading a disallowed character sequence+-- in the doctype declaration section of the state machine.+tokenBogusDoctype :: Tokenizer (TokenizerOutput DoctypeParams')+tokenBogusDoctype = tokenizer (Just ([], emptyDoctypeData))+    [ if_ (== '>') $ changeState DataState *> packToken ([], emptyDoctypeData)+    , if_ (== '\NUL') $ consTokenError UnexpectedNullCharacter <$> tokenBogusDoctype+    , else_ tokenBogusDoctype+    ]
+ src/Web/Mangrove/Parse/Tokenize/PlainText.hs view
@@ -0,0 +1,31 @@+{-|+Description:    Tokenization rules for characters which should not be processed further, without an exit condition.++Copyright:      (c) 2020 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      stable+Portability:    portable+-}+module Web.Mangrove.Parse.Tokenize.PlainText+    ( tokenPlainText+    ) where+++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tokenize.Common+import Web.Willow.Common.Encoding.Character+++-- | __HTML:__+--      @[PLAINTEXT state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#plaintext-state)@+-- +-- The parsing instructions rooted in the 'PlainTextState' section of the state+-- machine.+tokenPlainText :: Tokenizer (TokenizerOutput Token)+tokenPlainText = tokenizer Nothing+    [ if_ (== '\NUL') $ emit ([UnexpectedNullCharacter], Character replacementChar)+    , elseChar $ \c -> emit ([], Character c)+    ]
+ src/Web/Mangrove/Parse/Tokenize/RCData.hs view
@@ -0,0 +1,35 @@+{-|+Description:    Tokenization rules for characters which should be minimally processed.++Copyright:      (c) 2020-2021 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      stable+Portability:    portable+-}+module Web.Mangrove.Parse.Tokenize.RCData+    ( tokenRCData+    ) where+++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tokenize.Common+import Web.Mangrove.Parse.Tokenize.Character+import Web.Mangrove.Parse.Tokenize.Tag+import Web.Willow.Common.Encoding.Character+++-- | __HTML:__+--      @[RCDATA state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#rcdata-state)@+-- +-- The parsing instructions rooted in the 'RCDataState' section of the state+-- machine.+tokenRCData :: Tokenizer [TokenizerOutput Token]+tokenRCData = tokenizers Nothing+    [ ifs_ (== '&') $ tokenCharacterReference False >>= flushCharRef+    , ifs_ (== '<') tokenAppropriateEndTagLessThanSign+    , ifs_ (== '\NUL') $ emit' ([UnexpectedNullCharacter], Character replacementChar)+    , elsesChar $ \c -> emit' ([], Character c)+    ]
+ src/Web/Mangrove/Parse/Tokenize/RawText.hs view
@@ -0,0 +1,33 @@+{-|+Description:    Tokenization rules for characters which should not be processed further.++Copyright:      (c) 2020 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      stable+Portability:    portable+-}+module Web.Mangrove.Parse.Tokenize.RawText+    ( tokenRawText+    ) where+++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tokenize.Common+import Web.Mangrove.Parse.Tokenize.Tag+import Web.Willow.Common.Encoding.Character+++-- | __HTML:__+--      @[RAWTEXT state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#rawtext-state)@+-- +-- The parsing instructions rooted in the 'RawTextState' section of the state+-- machine.+tokenRawText :: Tokenizer [TokenizerOutput Token]+tokenRawText = tokenizers Nothing+    [ ifs_ (== '<') tokenAppropriateEndTagLessThanSign+    , ifs_ (== '\NUL') $ emit' ([UnexpectedNullCharacter], Character replacementChar)+    , elsesChar $ \c -> emit' ([], Character c)+    ]
+ src/Web/Mangrove/Parse/Tokenize/ScriptData.hs view
@@ -0,0 +1,51 @@+{-# HLINT ignore "Use ." #-}++{-|+Description:    Tokenization rules for characters within @\<script\>@ ... @\</script\>@+                sections.++Copyright:      (c) 2020 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      stable+Portability:    portable+-}+module Web.Mangrove.Parse.Tokenize.ScriptData+    ( tokenScriptData+    ) where+++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tokenize.Common+import Web.Mangrove.Parse.Tokenize.ScriptDataEscaped+import Web.Mangrove.Parse.Tokenize.Tag+import Web.Willow.Common.Encoding.Character+++-- | __HTML:__+--      @[script data state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#script-data-state)@+-- +-- The parsing instructions rooted in the 'ScriptDataState' section of the+-- state machine.+tokenScriptData :: Tokenizer [TokenizerOutput Token]+tokenScriptData = tokenizers Nothing+    [ ifs_ (== '<') tokenScriptDataLessThanSign+    , ifs_ (== '\NUL') $ emit' ([UnexpectedNullCharacter], Character replacementChar)+    , elsesChar $ \c -> emit' ([], Character c)+    ]++-- | __HTML:__+--      @[script data less-than sign state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#script-data-less-than-sign-state)@+-- +-- The parsing instructions for after reading @"\<"@ in the 'ScriptDataState'+-- section of the state machine.+tokenScriptDataLessThanSign :: Tokenizer [TokenizerOutput Token]+tokenScriptDataLessThanSign = tokenizers (Just [([], Character '<')])+    [ ifs_ (== '/') tokenAppropriateEndTagOpen+    , ifs_ (== '!') $ consEmit ([], Character '<') . consEmit ([], Character '!') $+        tokenScriptDataEscapeStart+    , elsePush_ $ emit' ([], Character '<')+    ]
+ src/Web/Mangrove/Parse/Tokenize/ScriptDataDoubleEscaped.hs view
@@ -0,0 +1,172 @@+{-|+Description:    Tokenization rules for characters within @\<script\>@ ... @\</script\>@+                sections in HTML comments in script data.++Copyright:      (c) 2020-2021 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      stable+Portability:    portable+-}+module Web.Mangrove.Parse.Tokenize.ScriptDataDoubleEscaped+    ( tokenScriptDataDoubleEscaped+    , tokenScriptDataDoubleEscapeStart+    ) where+++import qualified Control.Applicative as A++import qualified Data.Maybe as Y++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tokenize.Common+import Web.Willow.Common.Encoding.Character+import Web.Willow.Common.Parser++import {-# SOURCE #-} Web.Mangrove.Parse.Tokenize.Dispatcher+++-- | __HTML:__+--      @[script data escape start state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#script-data-escape-start-state)@+-- +-- The parsing instructions for after reading @"\<"@ when the next character is+-- a letter in the 'ScriptDataEscapedState' section of the state machine.+tokenScriptDataDoubleEscapeStart :: Tokenizer [TokenizerOutput Token]+tokenScriptDataDoubleEscapeStart = tokenScriptDataDoubleEscapeStart' >>= \t' ->+    case map toAsciiLower $ tokenizedOut t' of+        "script" -> do+            recovery <- map unpackToken <$> output t'+            tokenizers (Just $ recovery ++ [([EOFInScriptHtmlCommentLikeText], EndOfStream)])+                [ ifsChar isAsciiWhitespace $ escape t'+                , ifsChar (== '/') $ escape t'+                , ifsChar (== '>') $ escape t'+                , elsePush_ $ output t'+                ]+        _ -> output t'+  where escape t' c = do+            changeState ScriptDataDoubleEscapedState+            cs <- finalStateList Nothing <$> output t'+            c' <- emit' ([], Character c)+            cs' <- tokenScriptDataDoubleEscaped+            return $ cs ++ c' ++ cs'+        output t' = case tokenizedOut t' of+            [] -> case tokenizedErrs t' of+                [] -> return []+                errs -> consTokenErrorsList errs <$> dispatcher+            (c:cs) -> emits (tokenizedState t') $+                (tokenizedErrs t', Character c) : [([], Character c') | c' <- cs]+        emits state ts = finalStateList state <$> mapM emit ts+        unpackToken t' = (tokenizedErrs t', tokenizedOut t')++-- | Loop within the __HTML__ @[script data double escape start state]+-- (https://html.spec.whatwg.org/multipage/parsing.html#script-data-double-escape-start-state)@+-- to read the name of the tag.+tokenScriptDataDoubleEscapeStart' :: Tokenizer (TokenizerOutput String)+tokenScriptDataDoubleEscapeStart' = tokenizer (Just ([], ""))+    [ ifChar isAsciiAlpha $ \c -> consOut c <$> tokenScriptDataDoubleEscapeStart'+    , elsePush_ $ packToken ([], "")+    ]++-- | __HTML:__+--      @[script data double escaped state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#script-data-double-escaped-state)@+-- +-- The parsing instructions rooted in the 'ScriptDataDoubleEscapedState'+-- section of the state machine.+tokenScriptDataDoubleEscaped :: Tokenizer [TokenizerOutput Token]+tokenScriptDataDoubleEscaped = tokenizers (Just [([EOFInScriptHtmlCommentLikeText], EndOfStream)])+    [ ifs_ (== '-') $ consEmit ([], Character '-') tokenScriptDataDoubleEscapedDash+    , ifs_ (== '<') $ consEmit ([], Character '<') tokenScriptDataDoubleEscapedLessThanSign+    , ifs_ (== '\NUL') $ emit' ([UnexpectedNullCharacter], Character replacementChar)+    , elsesChar $ \c -> do+        e <- A.optional end+        consEmit ([], Character c) $ if Y.isJust e+            then emit' ([EOFInScriptHtmlCommentLikeText], EndOfStream)+            else return []+    ]++-- | __HTML:__+--      @[script data double escaped dash state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#script-data-escaped-double-dash-state)@+-- +-- The parsing instructions for after reading @"-"@ in the+-- 'ScriptDataDoubleEscapedState' section of the state machine.+tokenScriptDataDoubleEscapedDash :: Tokenizer [TokenizerOutput Token]+tokenScriptDataDoubleEscapedDash = tokenizers (Just [([EOFInScriptHtmlCommentLikeText], EndOfStream)])+    [ ifs_ (== '-') $ consEmit ([], Character '-') tokenScriptDataDoubleEscapedDashDash+    , ifs_ (== '<') $ consEmit ([], Character '<') tokenScriptDataDoubleEscapedLessThanSign+    , ifs_ (== '\NUL') $ emit' ([UnexpectedNullCharacter], Character replacementChar)+    , elsesChar $ \c -> emit' ([], Character c)+    ]++-- | __HTML:__+--      @[script data double escaped dash dash state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#script-data-escaped-double-dash-dash-state)@+-- +-- The parsing instructions for after reading @"--"@ in the+-- 'ScriptDataDoubleEscapedState' section of the state machine.+tokenScriptDataDoubleEscapedDashDash :: Tokenizer [TokenizerOutput Token]+tokenScriptDataDoubleEscapedDashDash = tokenizers (Just [([EOFInScriptHtmlCommentLikeText], EndOfStream)])+    [ ifs_ (== '-') $ consEmit ([], Character '-') tokenScriptDataDoubleEscapedDashDash+    , ifs_ (== '<') $ consEmit ([], Character '<') tokenScriptDataDoubleEscapedLessThanSign+    , ifs_ (== '>') $ changeState ScriptDataState *> emit' ([], Character '>')+    , ifs_ (== '\NUL') $ emit' ([UnexpectedNullCharacter], Character replacementChar)+    , elsesChar $ \c -> emit' ([], Character c)+    ]++-- | __HTML:__+--      @[script data double escaped less-than state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#script-data-double-escaped-less-than-state)@+-- +-- The parsing instructions for after reading @"\<"@ in the+-- 'ScriptDataDoubleEscapedState' section of the state machine.+tokenScriptDataDoubleEscapedLessThanSign :: Tokenizer [TokenizerOutput Token]+tokenScriptDataDoubleEscapedLessThanSign = tokenizers (Just [([EOFInScriptHtmlCommentLikeText], EndOfStream)])+    [ ifs_ (== '/') $ consEmit ([], Character '/') tokenScriptDataDoubleEscapeEnd+    , elsePush_ $ return []+    ]++-- | __HTML:__+--      @[script data double escape end state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#script-data-double-escape-end-state)@+-- +-- The parsing instructions for after reading @"\</"@ in the+-- 'ScriptDataDoubleEscapedState' section of the state machine.+tokenScriptDataDoubleEscapeEnd :: Tokenizer [TokenizerOutput Token]+tokenScriptDataDoubleEscapeEnd = tokenScriptDataDoubleEscapeEnd' >>= \t' -> do+    case map toAsciiLower $ tokenizedOut t' of+        "script" -> do+            recovery <- map unpackToken <$> output t'+            tokenizers (Just $ recovery ++ [([EOFInScriptHtmlCommentLikeText], EndOfStream)])+                [ ifsChar isAsciiWhitespace $ escape t'+                , ifsChar (== '/') $ escape t'+                , ifsChar (== '>') $ escape t'+                , elsePush_ $ output t'+                ]+        _ -> output t'+  where escape t' c = do+            changeState ScriptDataEscapedState+            cs <- finalStateList Nothing <$> output t'+            c' <- emit' ([], Character c)+            return $ cs ++ c'+        output t' = do+            buffer <- emits (tokenizedState t') [([], Character c) | c <- tokenizedOut t']+            case tokenizedErrs t' of+                [] -> return buffer+                errs -> consTokenErrorsList errs <$> case buffer of+                    [] -> dispatcher+                    _ -> return buffer+        emits state ts = finalStateList state <$> mapM emit ts+        unpackToken t' = (tokenizedErrs t', tokenizedOut t')+++-- | Loop within the __HTML__ @[script data double escape end state]+-- (https://html.spec.whatwg.org/multipage/parsing.html#script-data-double-escape-end-state)@+-- to read the name of the tag.+tokenScriptDataDoubleEscapeEnd' :: Tokenizer (TokenizerOutput String)+tokenScriptDataDoubleEscapeEnd' = tokenizer (Just ([], ""))+    [ ifChar isAsciiAlpha $ \c -> consOut c <$> tokenScriptDataDoubleEscapeEnd'+    , elsePush_ $ packToken ([], "")+    ]
+ src/Web/Mangrove/Parse/Tokenize/ScriptDataEscaped.hs view
@@ -0,0 +1,103 @@+{-|+Description:    Tokenization rules for characters within @\<!--@ ... @--\>@+                sections in script data.++Copyright:      (c) 2020-2021 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      stable+Portability:    portable+-}+module Web.Mangrove.Parse.Tokenize.ScriptDataEscaped+    ( tokenScriptDataEscaped+    , tokenScriptDataEscapeStart+    ) where+++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tokenize.Common+import Web.Mangrove.Parse.Tokenize.ScriptDataDoubleEscaped+import Web.Mangrove.Parse.Tokenize.Tag+import Web.Willow.Common.Encoding.Character+++-- | __HTML:__+--      @[script data escape start state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#script-data-escape-start-state)@+-- +-- The parsing instructions for after reading @"\<!"@ in the 'ScriptDataState'+-- section of the state machine.+tokenScriptDataEscapeStart :: Tokenizer [TokenizerOutput Token]+tokenScriptDataEscapeStart = tokenizers (Just [([], EndOfStream)])+    [ ifs_ (== '-') $ consEmit ([], Character '-') tokenScriptDataEscapeStartDash+    , elsePush_ $ return []+    ]++-- | __HTML:__+--      @[script data escape start dash state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#script-data-escape-start-state)@+-- +-- The parsing instructions for after reading @"\<!-"@ in the 'ScriptDataState'+-- section of the state machine.+tokenScriptDataEscapeStartDash :: Tokenizer [TokenizerOutput Token]+tokenScriptDataEscapeStartDash = tokenizers (Just [([], EndOfStream)])+    [ ifs_ (== '-') $ changeState ScriptDataEscapedState *> emit' ([], Character '-')+    , elsePush_ $ return []+    ]++-- | __HTML:__+--      @[script data escaped state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#script-data-escaped-state)@+-- +-- The parsing instructions rooted in the 'ScriptDataEscapedState' section of+-- the state machine.+tokenScriptDataEscaped :: Tokenizer [TokenizerOutput Token]+tokenScriptDataEscaped = tokenizers (Just [([EOFInScriptHtmlCommentLikeText], EndOfStream)])+    [ ifs_ (== '-') $ consEmit ([], Character '-') tokenScriptDataEscapedDash+    , ifs_ (== '<') tokenScriptDataEscapedLessThanSign+    , ifs_ (== '\NUL') $ emit' ([UnexpectedNullCharacter], Character replacementChar)+    , elsesChar $ \c -> emit' ([], Character c)+    ]++-- | __HTML:__+--      @[script data escaped dash state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#script-data-escaped-dash-state)@+-- +-- The parsing instructions for after reading @"-"@ in the+-- 'ScriptDataEscapedState' section of the state machine.+tokenScriptDataEscapedDash :: Tokenizer [TokenizerOutput Token]+tokenScriptDataEscapedDash = tokenizers (Just [([EOFInScriptHtmlCommentLikeText], EndOfStream)])+    [ ifs_ (== '-') $ consEmit ([], Character '-') tokenScriptDataEscapedDashDash+    , ifPush_ (== '<') $ return []+    , ifPush_ (== '\NUL') $ return []+    , elsePush_ $ return []+    ]++-- | __HTML:__+--      @[script data escaped dash dash state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#script-data-escaped-dash-dash-state)@+-- +-- The parsing instructions for after reading @"--"@ in the+-- 'ScriptDataEscapedState' section of the state machine.+tokenScriptDataEscapedDashDash :: Tokenizer [TokenizerOutput Token]+tokenScriptDataEscapedDashDash = tokenizers (Just [([EOFInScriptHtmlCommentLikeText], EndOfStream)])+    [ ifs_ (== '-') $ consEmit ([], Character '-') tokenScriptDataEscapedDashDash+    , ifPush_ (== '<') $ return []+    , ifs_ (== '>') $ changeState ScriptDataState *> emit' ([], Character '>')+    , ifPush_ (== '\NUL') $ return []+    , elsePush_ $ return []+    ]++-- | __HTML:__+--      @[script data escaped less-than state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#script-data-escaped-less-than-state)@+-- +-- The parsing instructions for after reading @"\<"@ in the+-- 'ScriptDataEscapedState' section of the state machine.+tokenScriptDataEscapedLessThanSign :: Tokenizer [TokenizerOutput Token]+tokenScriptDataEscapedLessThanSign = tokenizers (Just [([], Character '<')])+    [ ifs_ (== '/') tokenAppropriateEndTagOpen+    , ifPush_ isAsciiAlpha $ consEmit ([], Character '<') tokenScriptDataDoubleEscapeStart+    , elsePush_ $ emit' ([], Character '<')+    ]
+ src/Web/Mangrove/Parse/Tokenize/Tag.hs view
@@ -0,0 +1,505 @@+{-|+Description:    Tokenization rules for characters comprising a markup element.++Copyright:      (c) 2020-2021 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      stable+Portability:    portable+-}+module Web.Mangrove.Parse.Tokenize.Tag+    ( tokenTagOpen+    , tokenBeforeAttributeName+    , tokenSelfClosingStartTag+    , tokenAppropriateEndTagLessThanSign+    , tokenAppropriateEndTagOpen+      -- * Types+    , TagParams' ( .. )+    , emptyTagData+    , packTagToken+    ) where+++import qualified Control.Applicative as A+import qualified Control.Monad.Trans.State as N.S++import qualified Data.Bifunctor as F.B+import qualified Data.List as L+import qualified Data.HashMap.Strict as M+import qualified Data.Maybe as Y+import qualified Data.Text as T++import Data.Functor ( ($>) )++import Web.Willow.DOM++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tokenize.Common+import Web.Mangrove.Parse.Tokenize.Character+import Web.Mangrove.Parse.Tokenize.Comment+import Web.Mangrove.Parse.Tokenize.Doctype+import Web.Willow.Common.Encoding.Character+import Web.Willow.Common.Parser.Util++import {-# SOURCE #-} Web.Mangrove.Parse.Tokenize.Dispatcher+++-- | An intermediate form of 'TagParams' with unpacked parameter types to+-- facilitate construction.  Specifically, 'Char'-by-'Char' construction of a+-- 'String' followed by a single 'T.pack' is much faster than repeated @O(n)@+-- calls to 'T.cons'.  Values may be easily instantiated as updates to+-- @emptyTagData@.+data TagParams' = TagParams'+    { tagName' :: String+        -- ^ See 'tagName'.+    , tagIsSelfClosing' :: Bool+        -- ^ See 'tagIsSelfClosing'.+    , tagAttributes' :: [(String, String)]+        -- ^ See 'tagAttributes'.+    }+  deriving ( Eq, Show, Read )++-- | A sane default collection for easy record initialization; namely,+-- 'null's and 'False'.+emptyTagData :: TagParams'+emptyTagData = TagParams'+    { tagName' = ""+    , tagIsSelfClosing' = False+    , tagAttributes' = []+    }+++-- | Wrap a given collection of data into the payload of a 'StartTag' or+-- 'EndTag', according to the given constructor.+packTagToken+    :: (TagParams -> Token)+    -> TokenizerOutput (Maybe TagParams')+    -> Tokenizer [TokenizerOutput Token]+packTagToken toToken d' = case tokenizedOut d' of+    Just tagData -> do+        let tok = toToken $ emptyTagParams+                { tagName = T.pack $ tagName' tagData+                , tagIsSelfClosing = tagIsSelfClosing' tagData+                , tagAttributes = attrs+                }+            (attrErrs, attrs) = packAttributes $ tagAttributes' tagData+        tok' <- emit (tokenizedErrs d' ++ attrErrs, tok)+        return [tok' { tokenizedState = tokenizedState d' }]+    Nothing -> case tokenizedErrs d' of+        [] -> return []+        errs -> do+            atEOS <- atEndOfStream <$> N.S.get+            consTokenErrorsList errs <$> if atEOS+                then Y.fromMaybe [] <$> A.optional dispatcher+                else dispatcher++-- | Wrap a loose collection of unpacked 'String's into a form suitable for+-- representing the 'tagAttributes' of a markup element, removing all but the+-- first occurrence of any single name.+-- +-- Note that any nameless attributes at the head of the list are dropped, to+-- work around the edge case where @newAttr@ is called immediately following+-- the tag name; this shouldn't cause any issue as the specification doesn't+-- seem to output any such attributes in the first place.+packAttributes+    :: [(String, String)]+    -> ([ParseError], M.HashMap T.Text T.Text)+packAttributes = foldr filterAttr ([], M.empty) . L.sortOn fst . dropWhile (null . fst)++-- | Add a single attribute to the processed collection, potentially throwing a+-- 'DuplicateAttribute' warning (with the old value) if one with the same name+-- already exists.+filterAttr+    :: (String, String)+    -> ([ParseError], M.HashMap T.Text T.Text)+    -> ([ParseError], M.HashMap T.Text T.Text)+filterAttr (n, v) (errs, as)+    | M.null as = (errs, M.singleton name value)+    | otherwise = case M.lookup name as of+        Nothing -> (errs, M.insert name value as)+        Just value' -> (DuplicateAttribute (name, value') : errs, M.insert name value as)+  where name = T.pack n+        value = T.pack v+++-- | Finalize the currently "active" attribute, so that any further calls to,+-- e.g., @consAttrValue@ will be applied to a different (initially empty) pair+-- instead.+newAttr :: TokenizerOutput (Maybe TagParams') -> TokenizerOutput (Maybe TagParams')+newAttr = fmap . fmap $ \d -> d+    { tagAttributes' = case tagAttributes' d of+        [] -> []+        as@(("", ""):_) -> as+        as -> ("", "") : as+    }++-- | Prepend a character to the unpacked key of the currently "active" metadata+-- pair in a markup element.+consAttrName :: Char -> TokenizerOutput (Maybe TagParams') -> TokenizerOutput (Maybe TagParams')+consAttrName c = fmap . fmap $ \d -> d+    { tagAttributes' = consAttrName' $ tagAttributes' d+    }+  where consAttrName' [] = [([c], "")]+        consAttrName' (a:as) = F.B.first (c :) a : as++-- | Prepend a character to the unpacked value of the currently "active"+-- metadata pair in a markup element.+consAttrValue :: Char -> TokenizerOutput (Maybe TagParams') -> TokenizerOutput (Maybe TagParams')+consAttrValue c = consAttrValueString [c]++-- | Prepend a character sequence to the unpacked value of the currently+-- "active" metadata pair in a markup element.+consAttrValueString :: String -> TokenizerOutput (Maybe TagParams') -> TokenizerOutput (Maybe TagParams')+consAttrValueString str = fmap . fmap $ \d -> d+    { tagAttributes' = consAttrValueString' $ tagAttributes' d+    }+  where consAttrValueString' [] = [("", str)]+        consAttrValueString' (a:as) = fmap (str ++) a : as+++-- | __HTML:__+--      @[tag open state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#tag-open-state)@+-- +-- The parsing instructions for after reading @"\<"@ in a section of the state+-- machine which allows markup declarations.+tokenTagOpen :: Tokenizer [TokenizerOutput Token]+tokenTagOpen = tokenizers (Just [([EOFBeforeTagName], Character '<')])+    [ ifs_ (== '!') tokenMarkupDeclarationOpen+    , ifs_ (== '/') tokenEndTagOpen+    , ifPush_ isAsciiAlpha $ tokenTagName >>= packTagToken StartTag+    , ifPush_ (== '?') $ (: []) . packCommentToken .+        consTokenError UnexpectedQuestionMarkInsteadOfTagName <$>+        tokenBogusComment+    , elsePush_ $ changeState DataState *>+        emit' ([InvalidFirstCharacterOfTagName], Character '<')+    ]++-- | __HTML:__+--      @[end tag open state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#end-tag-open-state)@+-- +-- The parsing instructions for after reading @"\</"@ in a section of the state+-- machine which allows markup declarations.+tokenEndTagOpen :: Tokenizer [TokenizerOutput Token]+tokenEndTagOpen = tokenizers (Just recovery)+    [ ifPush_ isAsciiAlpha $ tokenTagName >>= packTagToken EndTag+    , ifs_ (== '>') $ do+        changeState DataState+        consTokenErrorsList [MissingEndTagName] <$> dispatcher+    , elsePush_ $ (: []) . packCommentToken . consTokenError InvalidFirstCharacterOfTagName <$>+        tokenBogusComment+    ]+  where recovery =+            [ ([], Character '<')+            , ([], Character '/')+            , ([EOFBeforeTagName], EndOfStream)+            ]++-- | __HTML:__+--      @[tag name state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#tag-name-state)@+-- +-- The parsing instructions for after reading @"\<"@ followed by a letter in a+-- section of the state machine which allows markup declarations.+tokenTagName :: Tokenizer (TokenizerOutput (Maybe TagParams'))+tokenTagName = tokenizer (Just ([EOFInTag], Nothing))+    [ if_ isAsciiWhitespace tokenBeforeAttributeName+    , if_ (== '/') tokenSelfClosingStartTag+    , if_ (== '>') $ packToken ([], Just emptyTagData)+    , if_ (== '\NUL') $ consTokenError UnexpectedNullCharacter . consTagName replacementChar <$>+        tokenTagName+    , elseChar $ \c -> consTagName (toAsciiLower c) <$> tokenTagName+    ]+  where consTagName char = fmap . fmap $ \d -> d+            { tagName' = char : tagName' d+            }++-- | __HTML:__+--      @[before attribute name state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#before-attribute-name-state)@+-- +-- The parsing instructions for reading whitespace interspersed with key-value+-- metadata pairs in the markup tag section of the state machine.+tokenBeforeAttributeName :: Tokenizer (TokenizerOutput (Maybe TagParams'))+tokenBeforeAttributeName = tokenizer (Just ([EOFInTag], Nothing))+    [ if_ isAsciiWhitespace tokenBeforeAttributeName+    , ifPush_ (`elem` "/>") tokenAfterAttributeName+    , ifChar (== '=') $ \c -> consTokenError UnexpectedEqualsSignBeforeAttributeName . consAttrName c <$>+        tokenAttributeName+    , elsePush_ tokenAttributeName+    ]++-- | __HTML:__+--      @[attribute name state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#attribute-name-state)@+-- +-- The parsing instructions for reading the initial (key) part of a key-value+-- metadata pair in the markup tag section of the state machine.+tokenAttributeName :: Tokenizer (TokenizerOutput (Maybe TagParams'))+tokenAttributeName = tokenizer (Just ([EOFInTag], Nothing))+    [ ifPush_ (`elem` "\t\n\f />") tokenAfterAttributeName+    , if_ (== '=') tokenBeforeAttributeValue+    , if_ (== '\NUL') $ consTokenError UnexpectedNullCharacter . consAttrName replacementChar <$>+        tokenAttributeName+    , ifChar (`elem` "\"'<") $ \c -> consTokenError UnexpectedCharacterInAttributeName . consAttrName c <$>+        tokenAttributeName+    , elseChar $ \c -> consAttrName (toAsciiLower c) <$> tokenAttributeName+    ]++-- | __HTML:__+--      @[after attribute name state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#after-attribute-name-state)@+-- +-- The parsing instructions for reading whitespace after the initial (key) part+-- of a key-value metadata pair in the markup tag section of the state machine.+tokenAfterAttributeName :: Tokenizer (TokenizerOutput (Maybe TagParams'))+tokenAfterAttributeName = tokenizer (Just ([EOFInTag], Nothing))+    [ if_ isAsciiWhitespace $ fmap newAttr tokenAfterAttributeName+    , if_ (== '/') $ fmap newAttr tokenSelfClosingStartTag+    , if_ (== '=') tokenBeforeAttributeValue+    , if_ (== '>') $ changeState DataState *> packToken ([], Just emptyTagData)+    , elsePush_ $ fmap newAttr tokenAttributeName+    ]++-- | __HTML:__+--      @[before attribute value state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#before-attribute-value-state)@+-- +-- The parsing instructions for reading whitespace after @"="@ following the+-- initial (key) part of a key-value metadata pair in the markup tag section of+-- the state machine.+tokenBeforeAttributeValue :: Tokenizer (TokenizerOutput (Maybe TagParams'))+tokenBeforeAttributeValue = tokenizer (Just ([EOFInTag], Nothing))+    [ if_ isAsciiWhitespace tokenBeforeAttributeValue+    , if_ (== '"') tokenAttributeValueDoubleQuoted+    , if_ (== '\'') tokenAttributeValueSingleQuoted+    , if_ (== '>') $ changeState DataState *>+        packToken ([MissingAttributeValue], Just emptyTagData)+    , elsePush_ tokenAttributeValueUnquoted+    ]++-- | __HTML:__+--      @[attribute value (double-quoted) state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#attribute-value-(double-quoted\)-state)@+-- +-- The parsing instructions for reading the second (value) part of a key-value+-- metadata pair following a @'"'@ in the markup tag section of the state+-- machine.+tokenAttributeValueDoubleQuoted :: Tokenizer (TokenizerOutput (Maybe TagParams'))+tokenAttributeValueDoubleQuoted = tokenizer (Just ([EOFInTag], Nothing))+    [ if_ (== '"') tokenAfterAttributeValueQuoted+    , if_ (== '&') $ tokenAttributeCharacterReference tokenAttributeValueDoubleQuoted+    , if_ (== '\NUL') $ consTokenError UnexpectedNullCharacter . consAttrValue replacementChar <$>+        tokenAttributeValueDoubleQuoted+    , elseChar $ \c -> consAttrValue c <$> tokenAttributeValueDoubleQuoted+    ]++-- | __HTML:__+--      @[attribute value (single-quoted) state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#attribute-value-(single-quoted\)-state)@+-- +-- The parsing instructions for reading the second (value) part of a key-value+-- metadata pair following a @'\''@ in the markup tag section of the state+-- machine.+tokenAttributeValueSingleQuoted :: Tokenizer (TokenizerOutput (Maybe TagParams'))+tokenAttributeValueSingleQuoted = tokenizer (Just ([EOFInTag], Nothing))+    [ if_ (== '\'') tokenAfterAttributeValueQuoted+    , if_ (== '&') $ tokenAttributeCharacterReference tokenAttributeValueSingleQuoted+    , if_ (== '\NUL') $ consTokenError UnexpectedNullCharacter . consAttrValue replacementChar <$>+        tokenAttributeValueSingleQuoted+    , elseChar $ \c -> consAttrValue c <$> tokenAttributeValueSingleQuoted+    ]++-- | __HTML:__+--      @[attribute value (unquoted) state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#attribute-value-(unquoted\)-state)@+-- +-- The parsing instructions for reading the second (value) part of a key-value+-- metadata pair directly following a @'='@ in the markup tag section of the+-- state machine.+tokenAttributeValueUnquoted :: Tokenizer (TokenizerOutput (Maybe TagParams'))+tokenAttributeValueUnquoted = tokenizer (Just ([EOFInTag], Nothing))+    [ if_ isAsciiWhitespace $ fmap newAttr tokenBeforeAttributeName+    , if_ (== '&') $ tokenAttributeCharacterReference tokenAttributeValueUnquoted+    , if_ (== '>') $ changeState DataState *> packToken ([], Just emptyTagData)+    , if_ (== '\NUL') $ consTokenError UnexpectedNullCharacter . consAttrValue replacementChar <$>+        tokenAttributeValueUnquoted+    , ifChar (`elem` "\"'<=`") $ \c ->+        consTokenError UnexpectedCharacterInUnquotedAttributeValue . consAttrValue c <$>+        tokenAttributeValueUnquoted+    , elseChar $ \c -> consAttrValue c <$> tokenAttributeValueUnquoted+    ]++-- | Resolve a character reference occurring within the second (value) part of+-- a key-value metadata pair, and continue parsing using the given tokenizer.+tokenAttributeCharacterReference+    :: Tokenizer (TokenizerOutput (Maybe TagParams'))+    -> Tokenizer (TokenizerOutput (Maybe TagParams'))+tokenAttributeCharacterReference resume = do+    ref <- tokenCharacterReference True+    tag <- resume+    return . consTokenErrors (tokenizedErrs ref) $ consAttrValueString (tokenizedOut ref) tag++-- | __HTML:__+--      @[after attribute value (quoted) state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#after-attribute-value-(quoted\)-state)@+-- +-- The parsing instructions for after reading a @'"'@ or @'\''@ matching the+-- opening mark of the second (value) part of a key-value metadata pair in the+-- markup tag section of the state machine.+tokenAfterAttributeValueQuoted :: Tokenizer (TokenizerOutput (Maybe TagParams'))+tokenAfterAttributeValueQuoted = tokenizer (Just ([EOFInTag], Nothing))+    [ if_ isAsciiWhitespace $ fmap newAttr tokenBeforeAttributeName+    , if_ (== '/') tokenSelfClosingStartTag+    , if_ (== '>') $ changeState DataState *> packToken ([], Just emptyTagData)+    , elsePush_ $ consTokenError MissingWhitespaceBetweenAttributes . newAttr <$>+        tokenBeforeAttributeName+    ]++-- | __HTML:__+--      @[self closing start tag state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#self-closing-start-tag-state)@+-- +-- The parsing instructions for after reading @"/"@ in the markup tag section+-- of the state machine.+tokenSelfClosingStartTag :: Tokenizer (TokenizerOutput (Maybe TagParams'))+tokenSelfClosingStartTag = tokenizer (Just ([EOFInTag], Nothing))+    [ if_ (== '>') $ changeState DataState *> packToken ([], Just $ emptyTagData+        { tagIsSelfClosing' = True+        })+    , elsePush_ $ consTokenError UnexpectedSolidusInTag <$> tokenBeforeAttributeName+    ]++-- | __HTML:__+--      @[bogus comment state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#bogus-comment-state)@+-- +-- The parsing instructions for after reading @"\<?"@ in a section of the state+-- machine which allows markup declarations.+tokenBogusComment :: Tokenizer (TokenizerOutput String)+tokenBogusComment = tokenizer (Just ([], ""))+    [ if_ (== '>') $ changeState DataState *> packToken ([], "")+    , if_ (== '\NUL') $ consTokenError UnexpectedNullCharacter . consOut replacementChar <$>+        tokenBogusComment+    , elseChar $ \c -> consOut c <$> tokenBogusComment+    ]++-- | __HTML:__+--      @[markup declaration open state]+--      (https://html.spec.whatwg.org/multipage/parsing.html#markup-declaration-open-state)@+-- +-- The parsing instructions for after reading @"\<!"@ in a section of the state+-- machine which allows markup declarations.+tokenMarkupDeclarationOpen :: Tokenizer [TokenizerOutput Token]+tokenMarkupDeclarationOpen = do+    parserChoice <- choice+        [ chunk' id "--" *> return commentParser+        , chunk' toAsciiUpper "DOCTYPE" *> return doctypeParser+        , chunk' id "[CDATA[" *> return cdataParser+        , return failureParser+        ]+    parserChoice+  where commentParser = fmap (pure . packCommentToken) tokenCommentStart+        doctypeParser = fmap pure tokenDoctype+        cdataParser = do+            currentNamespace <- currentNodeNamespace <$> N.S.get+            case currentNamespace of+                Just ns | ns /= htmlNamespace ->+                    changeState CDataState $> []+                _ -> pure . packCommentToken .+                    consTokenError CDataInHtmlContent . consOuts "[CDATA[" <$>+                    tokenBogusComment+        failureParser = pure . packCommentToken .+            consTokenError IncorrectlyOpenedComment <$> tokenBogusComment+++-- | __HTML:__+-- +--    * [@RAWTEXT less-than sign state@]+--      (https://html.spec.whatwg.org/multipage/parsing.html#rawtext-less-than-sign-state)+--    * [@RCDATA less-than sign state@]+--      (https://html.spec.whatwg.org/multipage/parsing.html#rcdata-less-than-sign-state)+-- +-- The parsing instructions for after reading @"\<"@ in sections of the state+-- machine which may be closed by tag, and which emit only 'Character' tokens.+tokenAppropriateEndTagLessThanSign :: Tokenizer [TokenizerOutput Token]+tokenAppropriateEndTagLessThanSign = tokenizers (Just [([], Character '<')])+    [ ifs_ (== '/') tokenAppropriateEndTagOpen+    , elsePush_ $ emit' ([], Character '<')+    ]++-- | __HTML:__+-- +--    * [@RAWTEXT end tag open state@]+--      (https://html.spec.whatwg.org/multipage/parsing.html#rawtext-end-tag-open-state)+--    * [@RCDATA end tag open state@]+--      (https://html.spec.whatwg.org/multipage/parsing.html#rcdata-end-tag-open-state)+--    * [@script data end tag open state@]+--      (https://html.spec.whatwg.org/multipage/parsing.html#script-data-end-tag-open-state)+--    * [@script data escaped end tag open state@]+--      (https://html.spec.whatwg.org/multipage/parsing.html#script-data-escaped-end-tag-open-state)+-- +-- The parsing instructions for after reading @"\</"@ in sections of the state+-- machine which may be closed by tag, and which emit only 'Character' tokens.+tokenAppropriateEndTagOpen :: Tokenizer [TokenizerOutput Token]+tokenAppropriateEndTagOpen = tokenizers (Just [([], Character '<'), ([], Character '/')])+    [ ifPush_ isAsciiAlpha tokenAppropriateEndTag+    , elsePush_ $ mapM emit+        [ ([], Character '<')+        , ([], Character '/')+        ]+    ]++-- | __HTML:__+-- +--    * [@RAWTEXT end tag name state@]+--      (https://html.spec.whatwg.org/multipage/parsing.html#rawtext-end-tag-name-state)+--    * [@RCDATA end tag name state@]+--      (https://html.spec.whatwg.org/multipage/parsing.html#rcdata-end-tag-name-state)+--    * [@script data end tag name state@]+--      (https://html.spec.whatwg.org/multipage/parsing.html#script-data-end-tag-name-state)+--    * [@script data escaped end tag name state@]+--      (https://html.spec.whatwg.org/multipage/parsing.html#script-data-escaped-end-tag-name-state)+-- +-- The parsing instructions for after reading @"\</"@ in sections of the state+-- machine which may be closed by tag, and which emit only 'Character' tokens.+tokenAppropriateEndTag :: Tokenizer [TokenizerOutput Token]+tokenAppropriateEndTag = tokenAppropriateEndTagName >>= \t' -> do+    close <- appropriateEndTag (map toAsciiLower $ tokenizedOut t')+    if close+        then tokenizers (Just $ fallback t')+            [ ifs_ isAsciiWhitespace $+                tokenBeforeAttributeName >>= packName (tokenizedOut t')+            , ifs_ (== '/') $+                tokenSelfClosingStartTag >>= packName (tokenizedOut t')+            , ifs_ (== '>') $ do+                changeState DataState+                out <- packName (tokenizedOut t') $ t'+                    { tokenizedOut = Just emptyTagData+                    }+                -- 'packName' keeps the state of the input (i.e. /before/ the+                -- closing @>@), and so that needs to be cleared so 'ifs_' will+                -- assign the state of the input token (/after/ the closing+                -- @>@).+                return $ map continueState out+            , elsePush_ $ emits (tokenizedState t') (anythingElse (tokenizedErrs t') (tokenizedOut t'))+            ]+        else emits (tokenizedState t') (fallback t')+  where fallback t' = anythingElse (tokenizedErrs t') (tokenizedOut t')+        packName buffer = packTagToken $ EndTag . packName' (T.pack buffer)+        packName' buffer tagData = tagData+            { tagName = T.map toAsciiLower buffer+            }+        anythingElse errs buffer =+            [ ([], Character '<')+            , ([], Character '/')+            ] ++ case buffer of+                [] -> []+                (c:cs) -> (errs, Character c) : [([], Character c') | c' <- cs]+        emits state ts = finalStateList state <$> mapM emit ts+        tokenAppropriateEndTagName = tokenizer (Just ([], ""))+            [ ifChar isAsciiAlpha $ \c -> consOut c <$> tokenAppropriateEndTagName+            , elsePush_ $ packToken ([], "")+            ]
+ src/Web/Mangrove/Parse/Tree.hs view
@@ -0,0 +1,288 @@+{-# HLINT ignore "Redundant flip" #-}++{-|+Description:    Fold a linear, semantic stream into a tree structure.++Copyright:      (c) 2020-2021 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      provisional+Portability:    portable++This module and the internal branch it heads implement the "Tree Construction"+section of the+__[HTML](https://html.spec.whatwg.org/multipage/parsing.html#tree-construction)__+document parsing specification, operating over the output of the+"Web.Mangrove.Parse.Tokenize" stage to produce a DOM tree representation of a+web page.  As this library is still in the early stages of development, the+representation produced here is not actually a proper DOM implementation, but+instead only stores basic parameters in an equivalent (but less-featured)+structure.  Nonetheless, it is still enough for basic evaluation and unstyled+rendering.+-}+module Web.Mangrove.Parse.Tree+    ( -- * Types+      -- ** Final+      Tree ( .. )+    , Node ( .. )+    , QuirksMode ( .. )+      -- ** Intermediate+    , Patch+    , TreeState+    , Encoding ( .. )+    , NodeIndex+    , ElementParams ( .. )+    , emptyElementParams+      -- * Initialization+    , defaultTreeState+    , treeEncoding+    , treeFragment+    , treeInIFrame+      -- * Transformations+    , tree+    , treeStep+    , finalizeTree+    ) where+++import qualified Control.Monad.Trans.State as N.S++import qualified Data.ByteString as BS+import qualified Data.ByteString.Short as BS.SH+import qualified Data.List as L+import qualified Data.Maybe as Y+import qualified Data.Text as T++import Web.Willow.DOM++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tokenize+import Web.Mangrove.Parse.Tokenize.Common+import Web.Mangrove.Parse.Tree.Common+import Web.Mangrove.Parse.Tree.Dispatcher+import Web.Mangrove.Parse.Tree.Patch+import Web.Mangrove.Parse.Tree.Patch.Fold+import Web.Willow.Common.Encoding hiding ( setRemainder )+import Web.Willow.Common.Encoding.Sniffer+import Web.Willow.Common.Parser+++-- | __HTML:__+--      @[tree construction]+--      (https://html.spec.whatwg.org/multipage/parsing.html#tree-construction)@+-- +-- Given a starting environment, transform a binary document stream into a+-- hierarchical markup tree.  If the parse fails, returns an empty tree (a+-- 'Document' node with no children).+tree :: TreeState -> BS.ByteString -> ([Patch], TreeState)+tree state stream = L.foldl' treeFold ([], state) $ L.unfoldr treeUnfold (state, stream)+  where treeUnfold = unfoldLoop treeStep tokenizerState+        treeFold (ps, _) (ps', state', _) = (ps ++ ps', state')++-- | Wrap a parser in a signature appropriate to pass to 'L.unfoldr'.+unfoldLoop+    :: Eq state+    => (state -> BS.ByteString -> ([out], state, BS.ByteString))+        -- ^ The single-step parser function.+    -> (state -> TokenizerState)+        -- ^ Repack the state used by the step parser into a standardized form.+    -> (state, BS.ByteString)+        -- ^ The initial inputs to the parser.+    -> Maybe (([out], state, BS.ByteString), (state, BS.ByteString))+unfoldLoop step toTokState (state, stream)+    | BS.null stream = Nothing+    | otherwise = case step state stream of+        out@(_, state', _) | state /= state' || hasRemainder state' -> continueUnfold out+        ([], _, stream') | BS.null stream' -> Nothing+        out -> continueUnfold out+  where continueUnfold (ps, state', stream') = Just ((ps, state', stream'), (state', stream'))+        hasRemainder = maybe False (not . BS.SH.null . decoderRemainder) . decoderState . toTokState++-- | Parse a minimal number of tokens from a binary document stream, into a+-- state-independent sequence of folding instructions.  Returns all data+-- required to seamlessly resume parsing.+treeStep :: TreeState -> BS.ByteString -> ([Patch], TreeState, BS.ByteString)+treeStep state stream = treeStep' stream' stateRemainder state+  where stateRemainder state' = state'+            { tokenizerState = setRemainder (BS.SH.toShort stream) $ tokenizerState state'+            }+        stream' = L.unfoldr tokenUnfold (tokenizerState state, stream)+        tokenUnfold = unfoldLoop tokenizeStep id++-- | Parse a minimal number of tokens from a binary document stream, performing+-- any backend processing required to correctly generate the document tree.+treeStep'+    :: [([([ParseError], Token)], TokenizerState, BS.ByteString)]+    -> (TreeState -> TreeState)+    -> TreeState+    -> ([Patch], TreeState, BS.ByteString)+treeStep' input fallback state =+    case runParserT (N.S.runStateT recurse $ treeParserState state) stream of+        Just (((ps, tokState, stream'), parserState), _) -> (ps', state', stream')+          where state' = TreeState+                    { tokenizerState = tokState+                    , treeParserState = parserState+                    }+                ps' = map redirectPatches ps+        Nothing -> ([], fallback state, BS.empty)+  where stream = L.foldr repackStream [] input+        redirectPatches p = case fragmentContext $ treeParserState state of+            -- A fragment with an @\<html\>@ context won't create that node in+            -- the folded tree, and so any patches sent there won't find an+            -- anchor.+            Just ctx | nodeIsElement (T.pack "html") $ fst ctx -> case p of+                InsertComment errs InDocument _ -> ErrorList errs+                InsertComment errs InHtmlElement txt -> InsertComment errs InDocument txt+                AddAttribute InDocument _ -> ErrorList []+                AddAttribute InHtmlElement attr -> AddAttribute InDocument attr+                _ -> p+            -- The fragment parsing algorithm returns the children of the+            -- context node, and so any patches destined for locations above+            -- that shouldn't show up in the resulting tree.+            Just _ -> case p of+                InsertComment errs InDocument _ -> ErrorList errs+                InsertComment errs InHtmlElement _ -> ErrorList errs+                AddAttribute InDocument _ -> ErrorList []+                AddAttribute InHtmlElement _ -> ErrorList []+                _ -> p+            _ -> p++-- | Explicitly indicate that the input stream will not contain any further+-- bytes, and perform any finalization processing based on that.+finalizeTree :: [Patch] -> TreeState -> Tree+finalizeTree ps state = buildTree $ ps ++ ps'+  where (ps', _, _) = treeStep' [(ts, setRemainder BS.SH.empty tokState, BS.empty)] id state+        tokState = tokenizerState state+        ts = finalizeTokenizer tokState ++ [([], EndOfStream)]+++-- | Specify the encoding scheme a given parse environment should use to read+-- from the binary document stream.  Note that this will always use the initial+-- state for the respective decoder; intermediate states as returned by+-- 'decodeStep' are not supported.+treeEncoding :: Either SnifferEnvironment (Maybe Encoding) -> TreeState -> TreeState+treeEncoding enc state = state+    { tokenizerState = tokenizerEncoding enc $ tokenizerState state+    }++-- | __HTML:__+--      @[fragment parsing algorithm]+--      (https://html.spec.whatwg.org/multipage/parsing.html#html-fragment-parsing-algorithm)@+-- +-- Transform a given parse environment by adding context for an embedded but+-- separate document fragment.  Calling this with an intermediate state+-- returned by 'treeStep' (as opposed to an initial state from+-- 'defaultTreeState') may result in an unexpected tree structure.+treeFragment+    :: ElementParams+        -- ^ __HTML:__+        --      @[context element]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#concept-frag-parse-context)@+        -- +        -- The node wrapping -- in one way or another -- the embedded document fragment.+    -> [(NodeIndex, ElementParams)]+        -- ^ The ancestors of the context element, most immediate first.+    -> Maybe QuirksMode+        -- ^ The degree of backwards compatibility used in the node document of+        -- the context element, if it can be determined.+    -> Maybe Bool+        -- ^ Whether the node document of the context element has been parsed+        -- in a way which would require scripting to be enabled (@'Just'+        -- 'True'@) or disabled (@'Just' 'False'@).+    -> TreeState+    -> TreeState+treeFragment ctxNode ctxTree ctxQuirks ctxScript state = state+    { tokenizerState =+        let mode = if elementNamespace ctxNode == Just htmlNamespace+                then case T.unpack $ elementName ctxNode of+                    "title" -> RCDataState+                    "textarea" -> RCDataState+                    "style" -> RawTextState+                    "xmp" -> RawTextState+                    "iframe" -> RawTextState+                    "noembed" -> RawTextState+                    "noframes" -> RawTextState+                    "script" -> ScriptDataState+                    "noscript" -> if ctxScript == Just True+                        then RawTextState+                        else DataState+                    "plaintext" -> PlainTextState+                    _ -> DataState+                else DataState+        in  tokenizerStartTag (Just htmlNamespace) (T.pack "html") .+            tokenizerMode mode $ tokenizerState state+    , treeParserState = resetInsertionMode' $ (treeParserState state)+        { openElements = [(0, htmlElement)]+        , elementIndex = ctxIndex + 1+        , quirksMode = Y.fromMaybe NoQuirks ctxQuirks+        , templateInsertionModes = if nodeIsElement (T.pack "template") ctxNode+            then [InTemplate]+            else []+        , fragmentContext = Just (ctxNode, ctxTree)+        , scriptingEnabled = Y.fromMaybe False ctxScript+        , formElementPointer = fmap fst . L.find (nodeIsElement (T.pack "form") . snd) $+            (ctxIndex, ctxNode) : ctxTree+        }+    }+  where ctxIndex = foldr (max . fst) 0 ctxTree + 1+        htmlElement = emptyElementParams+            { elementName = T.pack "html"+            , elementNamespace = Just htmlNamespace+            }++-- | Specify whether the given parse environment should be treated as if the+-- document were contained within the @srcdoc@ attribute of an @\<iframe\>@+-- element ('False' by default).+treeInIFrame :: Bool -> TreeState -> TreeState+treeInIFrame b state = state+    { treeParserState = (treeParserState state)+        { isInIFrameSrcDoc = b+        }+    }+++-- | Given the output of 'tokenizeStep', rewrap the token list and single state+-- into a single uniform stream.  The final returned tuple will have a 'Just'+-- value containing the single state, while every other element has 'Nothing'+-- (indicating a point where reentry is impossible).+repackStream+    :: ([([ParseError], Token)], TokenizerState, BS.ByteString)+    -> [TreeInput]+    -> [TreeInput]+repackStream ([], _, _) is = is+repackStream (ts, state, stream) is = case reverse ts of+    -- Be sure any 'EndOfStream' tokens encountered mid-stream don't break parsing.+    ((errs, EndOfStream):ts') -> case is of+        [] -> repackStream' ts+        (i:is') -> repackStream' (reverse ts') ++ consErrors errs i : is'+    -- Any other token ending the segment, however, can simply be repacked.+    _ -> repackStream' ts ++ is+  where repackStream' = flip foldr [] $ \(errs, t) ts' -> TreeInput+            { tokenErrs = errs+            , tokenOut = t+            , tokenState = if null ts'+                then Just (state, stream)+                else Nothing+            } : ts'+        consErrors errs i = i+            { tokenErrs = errs ++ tokenErrs i+            }+++-- | Loop the tree dispatcher until it returns a patchset which happens to+-- coincide with a tokenizer breakpoint.  Relies on lazy evaluation in the+-- stream generation to avoid forcing the entire thing at once, while still+-- retaining the capability to consume as much input as necessary to get the+-- parsers to line up.+recurse :: TreeBuilder ([Patch], TokenizerState, BS.ByteString)+recurse = do+    out <- dispatcher+    case treeState out of+        -- The tokenizer can't provide a breakpoint at the current token.+        Nothing -> do+            (out', tokState', stream') <- recurse+            return (treePatches out ++ out', tokState', stream')+        -- We have a re-entrant state to seamlessly resume the tokenizer.+        Just (tokState, stream) ->+            return (treePatches out, tokState, stream)
+ src/Web/Mangrove/Parse/Tree/AfterAfterBody.hs view
@@ -0,0 +1,47 @@+{-|+Description:    Token processing rules between a closing @\</html\>@ tag containing standard body content.++Copyright:      (c) 2020 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      stable+Portability:    portable+-}+module Web.Mangrove.Parse.Tree.AfterAfterBody+    ( treeAfterAfterBody+    ) where+++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tree.Common+import Web.Mangrove.Parse.Tree.InBody+import Web.Mangrove.Parse.Tree.Patch+import Web.Willow.Common.Parser+import Web.Willow.Common.Parser.Switch+++-- | __HTML:__+--      @[the "after after body" insertion mode]+--      (https://html.spec.whatwg.org/multipage/parsing.html#the-after-after-body-insertion-mode)@+-- +-- The parsing instructions corresponding to the 'AfterAfterBody' section of the+-- state machine.+treeAfterAfterBody :: TreeBuilder TreeOutput+treeAfterAfterBody = next >>= switch+    [ If isComment $ insertComment' InDocument+    , If isDoctype $ \t' -> do+        push t'+        treeInBody+    , If isWhitespace $ \t' -> do+        push t'+        treeInBody+    , If (isStartTag ["html"]) $ \t' -> do+        push t'+        treeInBody+    , If isEOF stopParsing+    , Else $ \t' -> do+        push t'+        switchMode InBody+        packTreeErrors_ [UnexpectedContentAfterBody]+    ]
+ src/Web/Mangrove/Parse/Tree/AfterAfterFrameset.hs view
@@ -0,0 +1,40 @@+{-|+Description:    Token processing rules between a closing @\</html\>@ tag containing frames of embedded external content.++Copyright:      (c) 2020 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      stable+Portability:    portable+-}+module Web.Mangrove.Parse.Tree.AfterAfterFrameset+    ( treeAfterAfterFrameset+    ) where+++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tree.Common+import Web.Mangrove.Parse.Tree.InBody+import Web.Mangrove.Parse.Tree.InHead+import Web.Mangrove.Parse.Tree.Patch+import Web.Willow.Common.Parser+import Web.Willow.Common.Parser.Switch+++-- | __HTML:__+--      @[the "after after frameset" insertion mode]+--      (https://html.spec.whatwg.org/multipage/parsing.html#the-after-after-frameset-insertion-mode)@+-- +-- The parsing instructions corresponding to the 'AfterAfterFrameset' section+-- of the state machine.+treeAfterAfterFrameset :: TreeBuilder TreeOutput+treeAfterAfterFrameset = next >>= switch+    [ If isComment $ insertComment' InDocument+    , If isDoctype $ \t' -> push t' *> treeInBody+    , If isWhitespace $ \t' -> push t' *> treeInBody+    , If (isStartTag ["html"]) $ \t' -> push t' *> treeInBody+    , If isEOF stopParsing+    , If (isStartTag ["noframes"]) $ \t' -> push t' *> treeInHead+    , Else $ packTreeErrors [UnexpectedNodeAfterFrameset]+    ]
+ src/Web/Mangrove/Parse/Tree/AfterBody.hs view
@@ -0,0 +1,54 @@+{-|+Description:    Token processing rules between the closing @\</body\>@ and @\</html\>@ tags.++Copyright:      (c) 2020 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      stable+Portability:    portable+-}+module Web.Mangrove.Parse.Tree.AfterBody+    ( treeAfterBody+    ) where+++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tree.Common+import Web.Mangrove.Parse.Tree.InBody+import Web.Mangrove.Parse.Tree.Patch+import Web.Willow.Common.Parser+import Web.Willow.Common.Parser.Switch+++-- | __HTML:__+--      @[the "after body" insertion mode]+--      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-afterbody)@+-- +-- The parsing instructions corresponding to the 'AfterBody' section of the+-- state machine.+treeAfterBody :: TreeBuilder TreeOutput+treeAfterBody = next >>= switch+    [ If isWhitespace $ \t' -> do+        push t'+        treeInBody+    , If isComment $ insertComment' InHtmlElement+        -- Even document fragments will always have an @\<html\>@ element as+        -- the root, so this 'InsertAt' will always target "the first element+        -- in the stack of open elements".+    , If isDoctype $ \t' ->+        packTreeErrors [UnexpectedDoctype $ tokenDocumentType t'] t'+    , If (isStartTag ["html"]) $ \t' -> do+        push t'+        treeInBody+    , If (isEndTag ["html"]) $ \t' -> do+        isFragment <- inFragment+        if isFragment+            then packTreeErrors [AbruptTemplateFragment] t'+            else switchMode AfterAfterBody *> packTreeErrors [] t'+    , If isEOF stopParsing+    , Else $ \t' -> do+        push t'+        switchMode InBody+        packTreeErrors_ [UnexpectedContentAfterBody]+    ]
+ src/Web/Mangrove/Parse/Tree/AfterFrameset.hs view
@@ -0,0 +1,46 @@+{-|+Description:    Token processing rules between the closing @\</frameset\>@ and @\</html\>@ tags.++Copyright:      (c) 2020 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      stable+Portability:    portable+-}+module Web.Mangrove.Parse.Tree.AfterFrameset+    ( treeAfterFrameset+    ) where+++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tree.Common+import Web.Mangrove.Parse.Tree.InBody+import Web.Mangrove.Parse.Tree.InHead+import Web.Mangrove.Parse.Tree.Patch+import Web.Willow.Common.Parser+import Web.Willow.Common.Parser.Switch+++-- | __HTML:__+--      @[the "after frameset" insertion mode]+--      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-afterframeset)@+-- +-- The parsing instructions corresponding to the 'AfterFrameset' section of the+-- state machine.+treeAfterFrameset :: TreeBuilder TreeOutput+treeAfterFrameset = next >>= switch+    [ If isWhitespace insertCharacter+    , If isComment insertComment+    , If isDoctype $ \t' ->+        packTreeErrors [UnexpectedDoctype $ tokenDocumentType t'] t'+    , If (isStartTag ["html"]) $ \t' -> do+        push t'+        treeInBody+    , If (isEndTag ["html"]) $ \t' -> switchMode AfterAfterFrameset *> packTreeErrors [] t'+    , If (isStartTag ["noframes"]) $ \t' -> do+        push t'+        treeInHead+    , If isEOF stopParsing+    , Else $ packTreeErrors [UnexpectedNodeAfterFrameset]+    ]
+ src/Web/Mangrove/Parse/Tree/AfterHead.hs view
@@ -0,0 +1,85 @@+{-|+Description:    Token processing rules between the @\</head\>@ and @\<body\>@ tags.++Copyright:      (c) 2020 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      stable+Portability:    portable+-}+module Web.Mangrove.Parse.Tree.AfterHead+    ( treeAfterHead+    ) where+++import qualified Data.Text as T++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tokenize.Common+import Web.Mangrove.Parse.Tree.Common+import Web.Mangrove.Parse.Tree.InBody+import Web.Mangrove.Parse.Tree.InHead+import Web.Mangrove.Parse.Tree.Patch+import Web.Willow.Common.Parser+import Web.Willow.Common.Parser.Switch+++-- | __HTML:__+--      @[the "after head" insertion mode]+--      (https://html.spec.whatwg.org/multipage/parsing.html#the-after-head-insertion-mode)@+-- +-- The parsing instructions corresponding to the 'AfterHead' section of the+-- state machine.+treeAfterHead :: TreeBuilder TreeOutput+treeAfterHead = next >>= switch+    [ If isWhitespace insertCharacter+    , If isComment insertComment+    , If isDoctype $ \t' ->+        packTreeErrors [UnexpectedDoctype $ tokenDocumentType t'] t'+    , If (isStartTag ["html"]) $ \t' -> do+        push t'+        treeInBody+    , If (isStartTag ["body"]) $ \t' -> do+        switchMode InBody+        setFramesetNotOk+        close <- closeCurrentNode_+        insert <- insertElement t'+        return $ close ++| insert+    , If (isStartTag ["frameset"]) $ \t' -> do+        switchMode InFrameset+        close <- closeCurrentNode_+        insert <- insertElement t'+        return $ close ++| insert+    , If (isStartTag+        [ "base"+        , "basefont"+        , "bgsound"+        , "link"+        , "meta"+        , "noframes"+        , "script"+        , "style"+        , "template"+        , "title"+        ]) $ \t' -> do+            push t'+            consTreeError UnexpectedMetadataOutsideOfHead <$> treeInHead+    , If (isEndTag ["template"]) $ \t' -> do+        push t'+        treeInHead+    , If (isEndTag ["body", "html", "br"]) anythingElse+    , If (isStartTag ["head"]) $ \t' ->+        packTreeErrors [DuplicateSingletonElement $ tokenElement t'] t'+    , If isAnyEndTag $ \t' ->+        packTreeErrors [UnmatchedEndTag $ tokenElement t'] t'+    , Else anythingElse+    ]+  where anythingElse t' = do+            push t'+            switchMode InBody+            close <- closeCurrentNode_+            insert <- insertElement_ $ emptyTagParams+                { tagName = T.pack "body"+                }+            packTree_ $ close ++ insert
+ src/Web/Mangrove/Parse/Tree/BeforeHead.hs view
@@ -0,0 +1,56 @@+{-|+Description:    Token processing rules between the opening @\<html\>@ and @\<head\>@ tags.++Copyright:      (c) 2020 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      stable+Portability:    portable+-}+module Web.Mangrove.Parse.Tree.BeforeHead+    ( treeBeforeHead+    ) where+++import qualified Data.Text as T++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tokenize.Common+import Web.Mangrove.Parse.Tree.Common+import Web.Mangrove.Parse.Tree.InBody+import Web.Mangrove.Parse.Tree.Patch+import Web.Willow.Common.Parser+import Web.Willow.Common.Parser.Switch+++-- | __HTML:__+--      @[the "before head" insertion mode]+--      (https://html.spec.whatwg.org/multipage/parsing.html#the-before-head-insertion-mode)@+-- +-- The parsing instructions corresponding to the 'BeforeHead' section of the+-- state machine.+treeBeforeHead :: TreeBuilder TreeOutput+treeBeforeHead = next >>= switch+    [ If isWhitespace $ packTreeErrors []+    , If isComment insertComment+    , If isDoctype $ \t' ->+        packTreeErrors [UnexpectedDoctype $ tokenDocumentType t'] t'+    , If (isStartTag ["html"]) $ \t' -> do+        push t'+        treeInBody+    , If (isStartTag ["head"]) $ \t' -> do+        switchMode InHead+        insertHeadElement t'+    , If (isEndTag ["head", "body", "html", "br"]) anythingElse+    , If isAnyEndTag $ \t' ->+        packTreeErrors [UnmatchedEndTag $ tokenElement t'] t'+    , Else anythingElse+    ]+  where anythingElse t' = do+            push t'+            switchMode InHead+            ps <- insertHeadElement_ $ emptyTagParams+                { tagName = T.pack "head"+                }+            packTree_ ps
+ src/Web/Mangrove/Parse/Tree/BeforeHtml.hs view
@@ -0,0 +1,52 @@+{-|+Description:    Token processing rules before the root element is opened.++Copyright:      (c) 2020 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      stable+Portability:    portable+-}+module Web.Mangrove.Parse.Tree.BeforeHtml+    ( treeBeforeHtml+    ) where+++import qualified Data.Text as T++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tokenize.Common+import Web.Mangrove.Parse.Tree.Common+import Web.Mangrove.Parse.Tree.Patch+import Web.Willow.Common.Parser+import Web.Willow.Common.Parser.Switch+++-- | __HTML:__+--      @[the "before html" insertion mode]+--      (https://html.spec.whatwg.org/multipage/parsing.html#the-before-html-insertion-mode)@+-- +-- The parsing instructions corresponding to the 'BeforeHtml' section of the+-- state machine.+treeBeforeHtml :: TreeBuilder TreeOutput+treeBeforeHtml = next >>= switch+    [ If isDoctype $ \t' ->+        packTreeErrors [UnexpectedDoctype $ tokenDocumentType t'] t'+    , If isComment $ insertComment' InDocument+    , If isWhitespace $ packTreeErrors []+    , If (isStartTag ["html"]) $ \t' -> do+        switchMode BeforeHead+        insertElement t'+    , If (isEndTag ["head", "body", "html", "br"]) anythingElse+    , If isAnyEndTag $ \t' ->+        packTreeErrors [UnmatchedEndTag $ tokenElement t'] t'+    , Else anythingElse+    ]+  where anythingElse t' = do+            push t'+            switchMode BeforeHead+            insert <- insertElement_ emptyTagParams+                { tagName = T.pack "html"+                }+            packTree_ insert
+ src/Web/Mangrove/Parse/Tree/Common.hs view
@@ -0,0 +1,1258 @@+{-# LANGUAGE OverloadedStrings #-}++{-|+Description:    Functions and objects used to build the tree construction parser.++Copyright:      (c) 2020-2021 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      provisional+Portability:    portable++This module provides the data structures and common functions used in the+first half of this implementation's split+__[HTML](https://html.spec.whatwg.org/multipage/parsing.html#tree-construction)__+tree construction algorithm; for more discussion on the structure, see the+documentation on "Web.Mangrove.Parse.Tree.Patch".  In particular, these objects+handle the state transitions and token processing which form the foundation of+that parser on which the 'Web.Mangrove.Parse.Tree.Patch.Patch' generation will+be able to build.+-}+module Web.Mangrove.Parse.Tree.Common+    ( -- * Types+      -- ** Tree construction+      -- *** Parser+      TreeBuilder+    , TreeState ( .. )+    , TreeParserState ( .. )+    , defaultTreeState+    , NodeIndex+    , InsertionMode ( .. )+      -- *** Data+      -- **** Elements+    , nodeIsElement+    , nodeIsSpecial+    , ElementParams ( .. )+    , emptyElementParams+    , packNodeData+    , scopeElements+    , specialElements+      -- **** Attributes+    , AttributeParams ( .. )+    , emptyAttributeParams+    , adjustForeignAttributes+    , adjustMathMLAttributes+    , adjustSvgAttributes+      -- **** Document type declarations+    , DocumentTypeParams ( .. )+    , emptyDocumentTypeParams+    , QuirksMode ( .. )+      -- ** Tokenization+      -- *** Parser+    , TokenizerState ( .. )+    , CurrentTokenizerState ( .. )+      -- *** Data+    , Token ( .. )+    , TreeInput ( .. )+    , TokenizerOutputState+    , tokenRemainder+    , dummyToken+    , dummyStateToken+    , mapTokenState+    , mapTokenState'+    , TagParams ( .. )+    , emptyTagParams+      -- *** Extraction+    , tokenCharacter+    , tokenDoctype+    , tokenDocumentType+    , tokenTag+    , tokenElement+      -- * Parser building+      -- ** Token matching+    , isEOF+    , isCharacter+    , isNull+    , isWhitespace+    , isComment+    , isDoctype+    , isAnyStartTag+    , isAnyEndTag+    , isStartTag+    , isEndTag+      -- ** State modification+    , switchMode+    , resetMode+    , setFramesetNotOk+    , insertFormattingMarker+    , clearFormattingElements+    , pushTemplateMode+    , popTemplateMode+    , resetInsertionMode+    , resetInsertionMode'+      -- ** Current node+      -- *** Retrieval+    , currentNode+    , currentNodeIndex+    , adjustedCurrentNode+      -- *** Namespace boundaries+    , atHtmlIntegration+    , atMathMLIntegration+    , isMathMLAnnotationXml+      -- ** Ancestor testing+    , inFragment+    , inIFrameSrcDoc+    , hasOpenElement+    , hasOpenElementExcept+      -- *** Scope+    , hasInScope+    , hasIndexInScope+    , hasInButtonScope+    , hasInListItemScope+    , hasInTableScope+    , hasInSelectScope+    ) where+++import qualified Control.Monad.Trans.State as N.S++import qualified Data.Bifunctor as F.B+import qualified Data.ByteString as BS+import qualified Data.HashMap.Strict as M+import qualified Data.Maybe as Y+import qualified Data.Text as T++import qualified Numeric.Natural as Z++import Web.Willow.DOM hiding+    ( Tree ( .. )+    , Node ( .. )+    )++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tokenize.Common+import Web.Willow.Common.Encoding.Character+import Web.Willow.Common.Parser+++-- | Parser combinators written over the output of the+-- "Web.Mangrove.Parse.Tokenize" stage, transforming it into a sequence of+-- folding instruction 'Web.Mangrove.Parse.Tree.Patch.Patch'es.+type TreeBuilder out = StateParser TreeParserState [TreeInput] out++-- | The collection of data returned by the "Web.Mangrove.Parse.Tokenize"+-- stage, and so comprising the input to the tree construction parser.+-- Values may be easily instantiated through 'dummyToken' or dummyStateToken'.+data TreeInput = TreeInput+    { tokenErrs :: [ParseError]+        -- ^ Any authoring errors detected during decoding and tokenization.+    , tokenOut :: Token+        -- ^ The token itself.+    , tokenState :: TokenizerOutputState+        -- ^ The data required to resume decoding immediately following the+        -- value, if possible.  See also 'decodedRemainder'.+    }+  deriving ( Eq, Show, Read )++-- | The unparsed portion of the binary stream, /after/ parsing the associated+-- token.  See also 'tokenState'.+tokenRemainder :: TreeInput -> Maybe BS.ByteString+tokenRemainder = fmap snd . tokenState++-- | All data with which to re-initialize the tokenizer, to resume as if the+-- state machine transition hadn't been interrupted.  If 'Nothing', the+-- associated 'Token' was emitted in the 'init' of several at once; in this+-- case, the tokenizer can't be re-entered in exactly the same place with the+-- data wrapped in this type, and so the stream must continue to be processed+-- until the first 'Just' value.+type TokenizerOutputState = Maybe (TokenizerState, BS.ByteString)+++-- | Generate a token /a priori/ in the tree construction stage to pass to a+-- function expecting the raw tokenizer output.  See 'dummyStateToken' if a+-- known 'TokenizerOutputState' must be associated as well.+dummyToken :: [ParseError] -> Token -> TreeInput+dummyToken errs t = dummyStateToken errs t Nothing++-- | Generate a token /a priori/ in the tree construction stage to pass to a+-- function expecting the raw tokenizer output, attaching a specific resume+-- state.  For most of the standard parsers, 'dummyToken' should be used+-- instead.+dummyStateToken :: [ParseError] -> Token -> TokenizerOutputState -> TreeInput+dummyStateToken errs t state = TreeInput+    { tokenErrs = errs+    , tokenOut = t+    , tokenState = state+    }++-- | Extract a single 'Char' from an input value.  Returns Nothing if the inner+-- 'Token' is not a 'Character'.+tokenCharacter :: TreeInput -> Maybe Char+tokenCharacter t' = case tokenOut t' of+    Character c -> Just c+    _ -> Nothing++-- | Extract the data comprising a DOCTYPE declaration from an input value.+-- Returns an empty collection if the inner 'Token' is not a 'Doctype'.+tokenDoctype :: TreeInput -> DoctypeParams+tokenDoctype t' = case tokenOut t' of+    Doctype d -> d+    _ -> emptyDoctypeParams++-- | Extract the data comprising a DOCTYPE declaration from an input value, in+-- a form suitable for 'ParseError's.  Returns an empty collection if the inner+-- 'Token' is not a 'Doctype'.+tokenDocumentType :: TreeInput -> DocumentTypeParams+tokenDocumentType t' = emptyDocumentTypeParams+    { documentTypeName = Y.fromMaybe T.empty $ doctypeName d+    , documentTypePublicId = Y.fromMaybe T.empty $ doctypePublicId d+    , documentTypeSystemId = Y.fromMaybe T.empty $ doctypeSystemId d+    }+  where d = tokenDoctype t'++-- | Extract the data comprising a markup element from an input value.  Returns+-- 'emptyTagParams' if the inner 'Token' is neither a 'StartTag' nor an+-- 'EndTag'.+tokenTag :: TreeInput -> TagParams+tokenTag t' = case tokenOut t' of+    StartTag d -> d+    EndTag d -> d+    _ -> emptyTagParams++-- | Extract the data comprising a markup element from an input value, in a+-- form suitable for 'ParseError's.  Returns an empty collection in the+-- 'htmlNamespace' if the inner 'Token' is neither a 'StartTag' nor an+-- 'EndTag'.+tokenElement :: TreeInput -> ElementParams+tokenElement = packNodeData (Just htmlNamespace) . tokenTag+++-- | Modify the state of the tokenizer as given by one of the wrapped tokens+-- used as input to the tree construction parsers.+mapTokenState :: TreeInput -> (TokenParserState -> TokenParserState) -> TreeInput+mapTokenState t' f = t'+    { tokenState = F.B.first f' <$> tokenState t'+    }+  where f' state = state+            { tokenParserState = f $ tokenParserState state+            }++-- | Modify the semi-opaque wrapped state of the tokenizer, within one of the+-- wrapped tokens used as input to the tree construction parsers.+mapTokenState' :: TreeInput -> (TokenizerOutputState -> TokenizerOutputState) -> TreeInput+mapTokenState' t' f = t'+    { tokenState = f $ tokenState t'+    }+++-- | Lift the tag data collected by the tokenizer into the namespaced datatype+-- of the DOM tree.+packNodeData :: Maybe Namespace -> TagParams -> ElementParams+packNodeData ns d = emptyElementParams+    { elementName = tagName d+    , elementNamespace = ns+    , elementAttributes = M.fromList . map packAttr . M.toList $ tagAttributes d+    }+  where packAttr (n, v) = ((Nothing, n), (Nothing, v))+++-- | Type-level clarification for an identifier uniquely identifying each+-- element in the document tree, assigned in a rough tree order.+-- +-- The specification assumes a reference-based and mutable memory model, where+-- each element in, e.g., the stack of open elements not only describes the+-- shape of the node, but also is distinct from all other nodes with the same+-- data.  Haskell's memory is much less accessible and more likely to be+-- shared, so an extra datapoint needs to be carried along.+type NodeIndex = Z.Natural+++-- | The collection of data required to extract a list of semantic atoms from a+-- binary document stream.  Values may be easily instantiated as updates to+-- 'defaultTreeState'.+data TreeState = TreeState+    { treeParserState :: TreeParserState+        -- ^ The state of the current 'Web.Mangrove.Parse.Tree.tree' stage.+    , tokenizerState :: TokenizerState+        -- ^ The state of the previous tokenization stage.  Note that the+        -- high-level conceptual view of the parser stack is of each stage+        -- moving along the 'BS.ByteString' as a (more or less) unified+        -- front, rather than each stage independently running over the output+        -- of the previous.+    }+  deriving ( Eq, Show, Read )++-- | All the data which needs to be tracked for correct behaviour in the tree+-- construction stage.+data TreeParserState = TreeParserState+    { insertionMode :: InsertionMode+        -- ^ __HTML:__+        --      @[insertion mode]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#insertion-mode)@+        -- +        -- The set of rules currently active in the state machine.+    , originalInsertionMode :: Maybe InsertionMode+        -- ^ __HTML:__+        --      @[original insertion mode]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#original-insertion-mode)@+        -- +        -- Which part of the state machine should be returned to after exiting+        -- the 'InText' insertion mode.+    , templateInsertionModes :: [InsertionMode]+        -- ^ __HTML:__+        --      @[stack of template insertion modes]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#stack-of-template-insertion-modes)@+        -- +        -- Which part(s) of the state machine should be returned to after+        -- exiting the 'InTemplate' insertion mode.+    , openElements :: [(NodeIndex, ElementParams)]+        -- ^ __HTML:__+        --      @[stack of open elements]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#stack-of-open-elements)@+        -- +        -- The ancestors of/path to the currently "active" node, closest parent+        -- first.+    , activeFormattingElements :: [[(NodeIndex, TagParams)]]+        -- ^ __HTML:__+        --      @[list of active formatting elements]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#list-of-active-formatting-elements)@+        -- +        -- The elements which should be reconstructed at the beginning of a tag+        -- closed unexpectedly early (overlapping tags).  These are divided+        -- into groups at particularly independent sections of the tree.+    , elementIndex :: NodeIndex+        -- ^ The unique ID for the /next/ node which will be generated.+    , isInIFrameSrcDoc :: Bool+        -- ^ __HTML:__+        --      @[an iframe srcdoc document]+        --      (https://html.spec.whatwg.org/multipage/iframe-embed-object.html#an-iframe-srcdoc-document)@+        -- +        -- Test whether the current document is being loaded via an @\<iframe\>@.+    , fragmentContext :: Maybe (ElementParams, [(NodeIndex, ElementParams)])+        -- ^ __HTML:__+        --      @[context]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#concept-frag-parse-context)@+        -- +        -- The element passed to the [HTML fragment parsing algorithm]+        -- (https://html.spec.whatwg.org/multipage/parsing.html#html-fragment-parsing-algorithm)+        -- to provide its immediate (conceptual) parent.  The first element of+        -- the tuple is a description of the context node itself, while the+        -- second is a list of its ancestors, most recent parent first.+    , quirksMode :: QuirksMode+        -- ^ __DOM:__+        --      @[Document mode]+        --      (https://dom.spec.whatwg.org/#concept-document-mode)@+        -- +        -- The degree to which the document should emulate weirdness in+        -- historic browsers' rendering.+    , fosteringEnabled :: Bool+        -- ^ __HTML:__+        --      @[foster parenting]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#foster-parent)@+        -- +        -- Whether the [appropriate place for inserting a node]+        -- (https://html.spec.whatwg.org/multipage/parsing.html#appropriate-place-for-inserting-a-node)+        -- should direct new nodes to be created outside the current @\<table\>@.+    , scriptingEnabled :: Bool+        -- ^ __HTML:__+        --      @[scripting flag]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#scripting-flag)@+        -- +        -- Whether the document is required to be parsed with JavaScript+        -- enabled or disabled.  Currently a binary choice, but the type may+        -- change in the future if non-deterministic parsing is implemented.+    , framesetOk :: Bool+        -- ^ __HTML:__+        --      @[frameset-ok flag]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#frameset-ok-flag)@+        -- +        -- Whether a particular class of content, which disallows the+        -- introduction of HTML page inclusion via @\<frameset\>@, has already+        -- been added to the tree.+    , headElementPointer :: Maybe NodeIndex+        -- ^ __HTML:__+        --      @[head element pointer]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#head-element-pointer)@+        -- +        -- The unique ID of the document's @\<head\>@ element, if one has been+        -- added to the tree.+    , formElementPointer :: Maybe NodeIndex+        -- ^ __HTML:__+        --      @[form element pointer]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#form-element-pointer)@+        -- +        -- The unique ID of the most recent @\<form\>@ element, if one exists.+    }+  deriving ( Eq, Show, Read )++-- | The collection of data which results in behaviour according to the+-- "initially" instructions in the HTML tree construction algorithm.+defaultTreeState :: TreeState+defaultTreeState = TreeState+    { tokenizerState = defaultTokenizerState+    , treeParserState = TreeParserState+        { insertionMode = Initial+        , originalInsertionMode = Nothing+        , templateInsertionModes = []+        , openElements = []+        , activeFormattingElements = []+        , elementIndex = 0+        , isInIFrameSrcDoc = False+        , fragmentContext = Nothing+        , quirksMode = NoQuirks+        , fosteringEnabled = False+        , scriptingEnabled = False+        , framesetOk = True+        , headElementPointer = Nothing+        , formElementPointer = Nothing+        }+    }+++-- | The various fixed points in the tree construction algorithm, where the+-- parser may break and re-enter seamlessly.+data InsertionMode+    = Initial+        -- ^ __HTML:__+        --      @[initial]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#the-initial-insertion-mode)@+        -- +        -- Before the doctype declaration, if one exists.+    | BeforeHtml+        -- ^ __HTML:__+        --      @[before html]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#the-before-html-insertion-mode)@+        -- +        -- Before the first markup tag in the document.+    | BeforeHead+        -- ^ __HTML:__+        --      @[before head]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#the-before-head-insertion-mode)@+        -- +        -- Between the root @\<html\>@ tag and the opening @\<head\>@.+    | InHead+        -- ^ __HTML:__+        --      @[in head]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-inhead)@+        -- +        -- Within the @\<head\>@ section, describing document metadata.+    | InHeadNoscript+        -- ^ __HTML:__+        --      @[in head noscript]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-inheadnoscript)@+        -- +        -- Within a @\<noscript\>@ section of the @\<head\>@, describing+        -- alternate document metadata to be used if script support has been+        -- disabled.+    | AfterHead+        -- ^ __HTML:__+        --      @[after head]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#the-after-head-insertion-mode)@+        -- +        -- Between the closing @\</html\>@ and opening @\<body\>@ markup tags.+    | InBody+        -- ^ __HTML:__+        --      @[in body]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-inbody)@+        -- +        -- Within the @\<body\>@ section, describing the primary, renderable+        -- document content.+    | InText+        -- ^ __HTML:__+        --      @[text]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-incdata)@+        -- +        -- Raw text which should be inserted into the document tree without+        -- translation to markup tags.  Corresponds to the+        -- "Web.Mangrove.Parse.Tokenize" state 'RCDataState', and as such it is+        -- critical that the 'currentState' and 'prevStartTag' items be set+        -- appropriately to allow the parser to continue to the next+        -- 'InsertionMode'.+    | InTable+        -- ^ __HTML:__+        --      @[in table]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-intable)@+        -- +        -- Within a @\<table\>@ markup section, describing content laid out in+        -- a rectangular grid.+    | InTableText+        -- ^ __HTML:__+        --      @[in table text]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-intabletext)@+        -- +        -- Special processing for content misnested within a @\<table\>@ but+        -- outside any @\<td\>@ or @\<th\>@.+    | InCaption+        -- ^ __HTML:__+        --      @[in caption]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-incaption)@+        -- +        -- Within a @\<caption\>@ markup section, describing the data presented+        -- by the enclosing @\<table\>@.+    | InColumnGroup+        -- ^ __HTML:__+        --      @[in column group]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-incolgroup)@+        -- +        -- Within a @\<colgroup\>@ markup section, describing how the vertical+        -- columns in the enclosing @\<table\>@ should be rendered.+    | InTableBody+        -- ^ __HTML:__+        --      @[in table body]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-intbody)@+        -- +        -- Within a @\<tbody\>@ markup section, describing the actual data+        -- presented by the enclosing @\<table\>@.+    | InRow+        -- ^ __HTML:__+        --      @[in row]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-intr)@+        -- +        -- Within a @\<tr\>@ markup section, describing a line of+        -- mutually-associated data presented by the enclosing @\<table\>@.+    | InCell+        -- ^ __HTML:__+        --      @[in cell]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-intd)@+        -- +        -- Within a @\<td\>@ markup section, describing a single point of data+        -- presented by the enclosing @\<table\>@.+    | InSelect+        -- ^ __HTML:__+        --      @[in select]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-inselect)@+        -- +        -- Within a @\<select\>@ markup section, presenting several predefined+        -- options for the user's input.+    | InSelectInTable+        -- ^ __HTML:__+        --      @[in select in table]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-inselectintable)@+        -- +        -- As 'InSelect', while providing extra cleanup logic for misnested+        -- table structure elements.+    | InTemplate+        -- ^ __HTML:__+        --      @[in template]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-intemplate)@+        -- +        -- Within a @\<template\>@ section, providing a+        -- 'Web.Mangrove.DOM.DocumentFragment' for simpler script-driven+        -- dynamic generation.+    | AfterBody+        -- ^ __HTML:__+        --      @[after body]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-afterbody)@+        -- +        -- Between the closing @\</body\>@ and @\</html\>@ markup tags.+    | InFrameset+        -- ^ __HTML:__+        --      @[in frameset]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-inframeset)@+        -- +        -- Within the @\<frameset\>@ section, listing several external+        -- documents to display in lieu of local content.+    | AfterFrameset+        -- ^ __HTML:__+        --      @[after frameset]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-afterframeset)@+        -- +        -- Between the closing @\</frameset\>@ and @\</html\>@ markup tags.+    | AfterAfterBody+        -- ^ __HTML:__+        --      @[after after body]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#the-after-after-body-insertion-mode)@+        -- +        -- After the final @\</html\>@ markup tag, when the primary content was+        -- described by a @\<body\>@ section.+    | AfterAfterFrameset+        -- ^ __HTML:__+        --      @[after after frameset]+        --      (https://html.spec.whatwg.org/multipage/parsing.html#the-after-after-frameset-insertion-mode)@+        -- +        -- After the final @\</html\>@ markup tag, when the primary content was+        -- linked via a @\<frameset\>@ section.+  deriving ( Eq, Ord, Bounded, Enum, Show, Read )+++-- | "Switch the insertion mode as specified."+switchMode :: InsertionMode -> TreeBuilder ()+switchMode mode = N.S.modify $ \state -> state+    { insertionMode = mode+    }++-- | "Switch the insertion mode to the original insertion mode."+resetMode :: TreeBuilder ()+resetMode = N.S.modify $ \state -> case originalInsertionMode state of+    Just mode -> state+        { insertionMode = mode+        , originalInsertionMode = Nothing+        }+    Nothing -> state+++-- | Indicate that the content which is or will be added to the document tree,+-- disallows the later use of @\<frameset\>@ transclusion.+setFramesetNotOk :: TreeBuilder ()+setFramesetNotOk = N.S.modify $ \state -> state+    { framesetOk = False+    }+++-- | "Insert a marker at the end of the list of active formatting elements."+insertFormattingMarker :: TreeBuilder ()+insertFormattingMarker = N.S.modify $ \state -> state+    { activeFormattingElements = [] : activeFormattingElements state+    }++-- | "Clear the list of active formatting elements up to the last marker."+clearFormattingElements :: TreeBuilder ()+clearFormattingElements = N.S.modify $ \state -> state+    { activeFormattingElements = drop 1 $ activeFormattingElements state+    }+++-- | "Push the specified insertion mode onto the stack of template insertion+-- modes."+pushTemplateMode :: InsertionMode -> TreeBuilder ()+pushTemplateMode mode = N.S.modify $ \state -> state+    { templateInsertionModes = mode : templateInsertionModes state+    }++-- | "Pop the current template insertion mode off the stack of template+-- insertion modes."+popTemplateMode :: TreeBuilder ()+popTemplateMode = N.S.modify $ \state -> state+    { templateInsertionModes = drop 1 $ templateInsertionModes state+    }+++-- | __HTML:__+--      @[an iframe srcdoc document]+--      (https://html.spec.whatwg.org/multipage/iframe-embed-object.html#an-iframe-srcdoc-document)@+-- +-- Test whether the current document is being loaded via an @\<iframe\>@.+inIFrameSrcDoc :: TreeBuilder Bool+inIFrameSrcDoc = isInIFrameSrcDoc <$> N.S.get+++-- | __HTML:__+--      @[current node]+--      (https://html.spec.whatwg.org/multipage/parsing.html#current-node)@+-- +-- The most-recently opened markup tag which has not yet been closed.+currentNode :: TreeBuilder (Maybe ElementParams)+currentNode = fmap snd . Y.listToMaybe . openElements <$> N.S.get++-- | The unique ID of the element described by 'currentNode'.+currentNodeIndex :: TreeBuilder (Maybe NodeIndex)+currentNodeIndex = fmap fst . Y.listToMaybe . openElements <$> N.S.get++-- | __HTML:__+--      @[adjusted current node]+--      (https://html.spec.whatwg.org/multipage/parsing.html#adjusted-current-node)@+-- +-- The most-recently opened markup tag which has not yet been closed, or the+-- context element if the document is being read as a document fragment and+-- everything else has been closed.+adjustedCurrentNode :: TreeBuilder (Maybe ElementParams)+adjustedCurrentNode = do+    state <- N.S.get+    if length (openElements state) <= 1 && Y.isJust (fragmentContext state)+        then return . fmap fst $ fragmentContext state+        else currentNode++-- | "Whether the parser was created as part of the HTML fragment parsing+-- algorithm."+inFragment :: TreeBuilder Bool+inFragment = do+    state <- N.S.get+    return . Y.isJust $ fragmentContext state+++-- | Map the given function over all attributes in the token data.+adjustAttributes :: (AttributeName -> AttributeName) -> TagParams -> TagParams+adjustAttributes adjust d = d+    { tagAttributes = M.fromList . map (F.B.first adjust) . M.toList $ tagAttributes d+    }++-- | __HTML:__+--      @[adjust MathML attributes]+--      (https://html.spec.whatwg.org/multipage/parsing.html#adjust-mathml-attributes)@+-- +-- Some attributes on [MathML](https://www.w3.org/TR/MathML/) elements are+-- defined in mixed case; restore that distinction to the case-folded token+-- data.+adjustMathMLAttributes :: TagParams -> TagParams+adjustMathMLAttributes = adjustAttributes adjust+  where adjust :: T.Text -> T.Text+        adjust "definitionurl" = "definitionURL"+        adjust name = name++-- | __HTML:__+--      @[adjust SVG attributes]+--      (https://html.spec.whatwg.org/multipage/parsing.html#adjust-svg-attributes)@+-- +-- Some attributes on [SVG](https://www.w3.org/TR/SVG/) elements are defined in+-- mixed case; restore that distinction to the case-folded token data.+adjustSvgAttributes :: TagParams -> TagParams+adjustSvgAttributes = adjustAttributes adjust+  where adjust :: T.Text -> T.Text+        adjust "attributename" = "attributeName"+        adjust "attributetype" = "attributeType"+        adjust "basefrequency" = "baseFrequency"+        adjust "baseprofile" = "baseProfile"+        adjust "calcmode" = "calcMode"+        adjust "clippathunits" = "clipPathUnits"+        adjust "diffuseconstant" = "diffuseConstant"+        adjust "edgemode" = "edgeMode"+        adjust "filterunits" = "filterUnits"+        adjust "glyphref" = "glyphRef"+        adjust "gradienttransform" = "gradientTransform"+        adjust "gradientunits" = "gradientUnits"+        adjust "kernelmatrix" = "kernelMatrix"+        adjust "kernelunitlength" = "kernelUnitLength"+        adjust "keypoints" = "keyPoints"+        adjust "keysplines" = "keySplines"+        adjust "keytimes" = "keyTimes"+        adjust "lengthadjust" = "lengthAdjust"+        adjust "limitingconeangle" = "limitingConeAngle"+        adjust "markerheight" = "markerHeight"+        adjust "markerunits" = "markerUnits"+        adjust "markerwidth" = "markerWidth"+        adjust "maskcontentunits" = "maskContentUnits"+        adjust "maskunits" = "maskUnits"+        adjust "numoctaves" = "numOctaves"+        adjust "pathlength" = "pathLength"+        adjust "patterncontentunits" = "patternContentUnits"+        adjust "patterntransform" = "patternTransform"+        adjust "patternunits" = "patternUnits"+        adjust "pointsatx" = "pointsAtX"+        adjust "pointsaty" = "pointsAtY"+        adjust "pointsatz" = "pointsAtZ"+        adjust "preservealpha" = "preserveAlpha"+        adjust "preserveaspectratio" = "preserveAspectRatio"+        adjust "primitiveunits" = "primitiveUnits"+        adjust "refx" = "refX"+        adjust "refy" = "refY"+        adjust "repeatcount" = "repeatCount"+        adjust "repeatdur" = "repeatDur"+        adjust "requiredextensions" = "requiredExtensions"+        adjust "requiredfeatures" = "requiredFeatures"+        adjust "specularconstant" = "specularConstant"+        adjust "specularexponent" = "specularExponent"+        adjust "spreadmethod" = "spreadMethod"+        adjust "startoffset" = "startOffset"+        adjust "stddeviation" = "stdDeviation"+        adjust "stitchtiles" = "stitchTiles"+        adjust "surfacescale" = "surfaceScale"+        adjust "systemlanguage" = "systemLanguage"+        adjust "tablevalues" = "tableValues"+        adjust "targetx" = "targetX"+        adjust "targety" = "targetY"+        adjust "textlength" = "textLength"+        adjust "viewbox" = "viewBox"+        adjust "viewtarget" = "viewTarget"+        adjust "xchannelselector" = "xChannelSelector"+        adjust "ychannelselector" = "yChannelSelector"+        adjust "zoomandpan" = "zoomAndPan"+        adjust name = name+++-- | __HTML:__+--      @[adjust foreign attributes]+--      (https://html.spec.whatwg.org/multipage/parsing.html#adjust-foreign-attributes)@+-- +-- The HTML specification expects most names to not carry XML-style prefixes+-- (e.g. @xlink:role@), and handles most which do anyway without issue.+-- However, that assumption proves disruptive for a few attributes, and those+-- exceptions should therefore be fixed.+-- +-- Note that this is a very simple algorithm in simply assuming a standard+-- prefix<->namespace assignment, and doesn't perform any scope test.  As+-- perfect representation of the structured data isn't actually the goal, that+-- winds up being enough /in this case/.  Do not expect actual XML to play+-- nicely.+adjustForeignAttributes :: ElementParams -> ElementParams+adjustForeignAttributes tag = case M.intersection attrs foreignNames of+    foreignAttrs | M.null foreignAttrs -> tag+    foreignAttrs ->+        let foreignAttrs' = fromAttrList . map adjustForeignAttribute $ toAttrList foreignAttrs+        in  tag+                { elementAttributes = M.union foreignAttrs' $ M.difference attrs foreignNames+                }+  where attrs = elementAttributes tag+        foreignNames = M.fromList+            [ ((Nothing, n), ())+            | n <-+                [ "xlink:actuate"+                , "xlink:arcrole"+                , "xlink:href"+                , "xlink:role"+                , "xlink:show"+                , "xlink:title"+                , "xlink:type"+                , "xml:lang"+                , "xml:space"+                , "xmlns"+                , "xmlns:xlink"+                ]+            ]++-- | Compare the name of a single attribute to the known exceptions described+-- in 'adjustForeignAttributes', and update the values if it matches.+adjustForeignAttribute :: AttributeParams -> AttributeParams+adjustForeignAttribute attr = case T.splitOn ":" $ attrName attr of+    [p@"xlink", n@"actuate"] -> updateAttr p n xlinkNamespace+    [p@"xlink", n@"arcrole"] -> updateAttr p n xlinkNamespace+    [p@"xlink", n@"href"] -> updateAttr p n xlinkNamespace+    [p@"xlink", n@"role"] -> updateAttr p n xlinkNamespace+    [p@"xlink", n@"show"] -> updateAttr p n xlinkNamespace+    [p@"xlink", n@"title"] -> updateAttr p n xlinkNamespace+    [p@"xlink", n@"type"] -> updateAttr p n xlinkNamespace+    [p@"xml", n@"lang"] -> updateAttr p n xmlNamespace+    [p@"xml", n@"space"] -> updateAttr p n xmlNamespace+    [n@"xmlns"] -> attr+        { attrPrefix = Nothing+        , attrName = n+        , attrNamespace = Just xmlnsNamespace+        }+    [p@"xmlns", n@"xlink"] -> updateAttr p n xmlnsNamespace+    _ -> attr+  where updateAttr p n ns = attr+            { attrPrefix = Just p+            , attrName = n+            , attrNamespace = Just ns+            }+++-- | 'Web.Mangrove.Parse.Common.Parser.switch' case guard testing whether the+-- wrapped token indicates the end of the stream.+isEOF :: TreeInput -> Bool+isEOF t' = case tokenOut t' of+    -- 'EndOfStream' only works as an explicit EOF because of the manipulations in+    -- the parser runner ('Web.Mangrove.Parse.Tree.repackStream').+    EndOfStream -> True+    _ -> False++-- | 'Web.Mangrove.Parse.Common.Parser.switch' case guard testing whether the+-- wrapped token is a whitespace 'Character' in the ASCII range.+isWhitespace :: TreeInput -> Bool+isWhitespace t' = case tokenOut t' of+    Character c | isAsciiWhitespace c -> True+    _ -> False++-- | 'Web.Mangrove.Parse.Common.Parser.switch' case guard testing whether the+-- wrapped token is any 'Character'.+isCharacter :: TreeInput -> Bool+isCharacter t' = case tokenOut t' of+    Character _ -> True+    _ -> False++-- | 'Web.Mangrove.Parse.Common.Parser.switch' case guard testing whether the+-- wrapped token is specifically a @U+0000@ null 'Character'.+isNull :: TreeInput -> Bool+isNull t' = case tokenOut t' of+    Character '\NUL' -> True+    _ -> False++-- | 'Web.Mangrove.Parse.Common.Parser.switch' case guard testing whether the+-- wrapped token is any 'Comment'.+isComment :: TreeInput -> Bool+isComment t' = case tokenOut t' of+    Comment _ -> True+    _ -> False++-- | 'Web.Mangrove.Parse.Common.Parser.switch' case guard testing whether the+-- wrapped token is any 'Doctype'.+isDoctype :: TreeInput -> Bool+isDoctype t' = case tokenOut t' of+    Doctype _ -> True+    _ -> False++-- | 'Web.Mangrove.Parse.Common.Parser.switch' case guard testing whether the+-- wrapped token is any 'StartTag'.  If the behaviour further requires the+-- 'StartTag' to have a specific name, use 'isStartTag' instead.+isAnyStartTag :: TreeInput -> Bool+isAnyStartTag t' = case tokenOut t' of+    StartTag _ -> True+    _ -> False++-- | 'Web.Mangrove.Parse.Common.Parser.switch' case guard testing whether the+-- wrapped token is a 'StartTag' with any of the given names; the list should+-- be entirely in lower-case.  If the exact name doesn't actually matter, use+-- 'isAnyStartTag' instead.+isStartTag :: [String] -> TreeInput -> Bool+isStartTag names t' = case tokenOut t' of+    StartTag d | elem (T.unpack $ tagName d) names -> True+    _ -> False++-- | 'Web.Mangrove.Parse.Common.Parser.switch' case guard testing whether the+-- wrapped token is any 'EndTag'.  If the behaviour further requires the+-- 'EndTag' to have a specific name, use 'isEndTag' instead.+isAnyEndTag :: TreeInput -> Bool+isAnyEndTag t' = case tokenOut t' of+    EndTag _ -> True+    _ -> False++-- | 'Web.Mangrove.Parse.Common.Parser.switch' case guard testing whether the+-- wrapped token is a 'EndTag' with any of the given names; the list should be+-- entirely in lower-case.  If the exact name doesn't actually matter, use+-- 'isAnyEndTag' instead.+isEndTag :: [String] -> TreeInput -> Bool+isEndTag names t' = case tokenOut t' of+    EndTag d | elem (T.unpack $ tagName d) names -> True+    _ -> False+++-- | Check if the described node has the given name, and is in the HTML+-- namespace.+nodeIsElement :: T.Text -> ElementParams -> Bool+nodeIsElement name node = elementNamespace node == Just htmlNamespace && elementName node == name++-- | Check if the described node is one of the many which have specific behaviour+-- governing their addition to the document tree.  See 'specialElements' for+-- the nodes themselves.+nodeIsSpecial :: ElementParams -> Bool+nodeIsSpecial node = case elementNamespace node of+    Just ns -> elem (ns, elementName node) specialElements+    Nothing -> False+++-- | Check whether any of the given tags, in the HTML namespace, are in the+-- stack of open elements.+hasOpenElement :: [ElementName] -> TreeBuilder Bool+hasOpenElement names = any isElement . filter isHtml . map snd . openElements <$> N.S.get+  where isElement tag = elem (elementName tag) names+        isHtml tag = elementNamespace tag == Just htmlNamespace++-- | Check if there are any tags on the stack of open elements, which are not+-- one of the given list, or are in a non-HTML namespace.+hasOpenElementExcept :: [ElementName] -> TreeBuilder Bool+hasOpenElementExcept names = any isNotElement . map snd . openElements <$> N.S.get+  where isNotElement tag = isNotHtml tag || notElem (elementName tag) names+        isNotHtml tag = elementNamespace tag /= Just htmlNamespace+++-- | __HTML:__+--      the elements listed for @[has a particular element in scope]+--      (https://html.spec.whatwg.org/multipage/parsing.html#has-an-element-in-scope)@+-- +-- Several elements provide a breakpoint which limits how far up the tree some+-- searches for open tags will propagate; this list defines those.  See+-- 'hasInScope' for a test using this list.+scopeElements :: [(Namespace, ElementName)]+scopeElements =+    [ (htmlNamespace, n)+    | n <-+        [ "applet"+        , "caption"+        , "html"+        , "table"+        , "td"+        , "th"+        , "marquee"+        , "object"+        , "template"+        ]+    ] +++    [ (mathMLNamespace, n)+    | n <-+        [ "mi"+        , "mo"+        , "mn"+        , "ms"+        , "mtext"+        , "annotation-xml"+        ]+    ] +++    [ (svgNamespace, n)+    | n <-+        [ "foreignObject"+        , "desc"+        , "title"+        ]+    ]++-- | __HTML:__+--      @[special category]+--      (https://html.spec.whatwg.org/multipage/parsing.html#special)@+-- +-- The specification describes specific behaviour for many tags, and refers+-- back to those same tags at a few points in the parse algorithm.  This then+-- collects everything in that category; see 'nodeIsSpecial' to test against+-- it.+specialElements :: [(Namespace, ElementName)]+specialElements =+    [ (htmlNamespace, n)+    | n <-+        [ "address"+        , "area"+        , "article"+        , "aside"+        , "base"+        , "basefont"+        , "bgsound"+        , "blockquote"+        , "body"+        , "br"+        , "button"+        , "center"+        , "col"+        , "colgroup"+        , "dd"+        , "details"+        , "dir"+        , "div"+        , "dl"+        , "dt"+        , "embed"+        , "fieldset"+        , "figcaption"+        , "figure"+        , "footer"+        , "form"+        , "frame"+        , "frameset"+        , "h1"+        , "h2"+        , "h3"+        , "h4"+        , "h5"+        , "h6"+        , "head"+        , "header"+        , "hgroup"+        , "hr"+        , "iframe"+        , "img"+        , "input"+        , "keygen"+        , "li"+        , "link"+        , "listing"+        , "main"+        , "menu"+        , "meta"+        , "nav"+        , "noembed"+        , "noframes"+        , "noscript"+        , "ol"+        , "p"+        , "param"+        , "plaintext"+        , "pre"+        , "script"+        , "section"+        , "select"+        , "style"+        , "source"+        , "summary"+        , "tbody"+        , "textarea"+        , "tfoot"+        , "thead"+        , "title"+        , "tr"+        , "track"+        , "ul"+        , "wbr"+        , "xmp"+        ]+    ] ++ scopeElements+++-- | __HTML:__+--      @[has a particular element in a specific scope]+--      (https://html.spec.whatwg.org/multipage/parsing.html#has-an-element-in-the-specific-scope)@+-- +-- Check if a tag with any of the given names, in the HTML namespace, is in the+-- stack of open elements more recently than the closest boundary element.  See+-- 'scopeElements' for the typical list, but generally 'hasInScope',+-- 'hasInButtonScope', 'hasInListItemScope', 'hasInSelectScope', or+-- 'hasInTableScope' should be used instead.+inScope :: [(Namespace, ElementName)] -> [T.Text] -> TreeBuilder Bool+inScope bound names = recurse . openElements <$> N.S.get+  where recurse [] = False+        recurse ((_, e):es)+            | elem (elementName e) names = True+            | elem (elementNamespace e, elementName e) (map (F.B.first Just) bound) = False+            | otherwise = recurse es++-- | Given a node's unique ID, check if it is in the stack of open elements,+-- more recently than the closest boundary node from 'scopeElements'.  See+-- 'hasInScope' if any node with a given name will suffice.+hasIndexInScope :: NodeIndex -> TreeBuilder Bool+hasIndexInScope index = any ((== index) . fst) . takeWhile (not . isScopeElement) .+    openElements <$> N.S.get+  where isScopeElement (_, e) = case elementNamespace e of+            Just ns -> elem (ns, elementName e) scopeElements+            Nothing -> False++-- | __HTML:__+--      @[has a particular element in scope]+--      (https://html.spec.whatwg.org/multipage/parsing.html#has-an-element-in-scope)@+-- +-- Check if a tag with any of the given names, in the HTML namespace, is in the+-- stack of open elements more recently than the closest boundary node from+-- 'scopeElements'.  See 'hasIndexInScope' if a specific existing element is+-- required.+hasInScope :: [T.Text] -> TreeBuilder Bool+hasInScope = inScope scopeElements++-- | __HTML:__+--      @[has a particular element in list item scope]+--      (https://html.spec.whatwg.org/multipage/parsing.html#has-an-element-in-list-item-scope)@+-- +-- Check if a tag with any of the given names, in the HTML namespace, is in the+-- stack of open elements more recently than the closest boundary node of+-- either @\<ol\>@, @\<ul\>@, or any from 'scopeElements'.+hasInListItemScope :: [T.Text] -> TreeBuilder Bool+hasInListItemScope = inScope $+    [ (htmlNamespace, n)+    | n <-+        [ "ol"+        , "ul"+        ]+    ] ++ scopeElements++-- | __HTML:__+--      @[has a particular element in button scope]+--      (https://html.spec.whatwg.org/multipage/parsing.html#has-an-element-in-button-scope)@+-- +-- Check if a tag with any of the given names, in the HTML namespace, is in the+-- stack of open elements more recently than the closest boundary node of+-- either @\<button\>@ or any from 'scopeElements'.+hasInButtonScope :: [T.Text] -> TreeBuilder Bool+hasInButtonScope = inScope $ (htmlNamespace, "button") : scopeElements++-- | __HTML:__+--      @[has a particular element in table scope]+--      (https://html.spec.whatwg.org/multipage/parsing.html#has-an-element-in-table-scope)@+-- +-- Check if a tag with any of the given names, in the HTML namespace, is in the+-- stack of open elements more recently than the closest boundary node of+-- either @\<table\>@ or a top-level element.+hasInTableScope :: [T.Text] -> TreeBuilder Bool+hasInTableScope = inScope+    [ (htmlNamespace, n)+    | n <-+        [ "html"+        , "table"+        , "template"+        ]+    ]++-- | __HTML:__+--      @[has a particular element in select scope]+--      (https://html.spec.whatwg.org/multipage/parsing.html#has-an-element-in-select-scope)@+-- +-- Check if a tag with any of the given names, in the HTML namespace, is in the+-- stack of open elements more recently than /any/ node except an+-- @\<optgroup\>@ or @\<option\>@ element.+hasInSelectScope :: [T.Text] -> TreeBuilder Bool+hasInSelectScope names = recurse . openElements <$> N.S.get+  where recurse [] = False+        recurse ((_, e):es)+            | elem (elementName e) names = True+            | elem (elementName e) ["optgroup", "option"]+                && elementNamespace e == Just htmlNamespace = recurse es+            | otherwise = False+++-- | __HTML:__+--      @[reset the insertion mode appropriately]+--      (https://html.spec.whatwg.org/multipage/parsing.html#reset-the-insertion-mode-appropriately)@+-- +-- Guess what part of the state machine should be active, according to the+-- stack of open elements.  See 'resetInsertionMode'' for use outside a parser+-- monad.+resetInsertionMode :: TreeBuilder ()+resetInsertionMode = N.S.modify resetInsertionMode'++-- | __HTML:__+--      @[reset the insertion mode appropriately]+--      (https://html.spec.whatwg.org/multipage/parsing.html#reset-the-insertion-mode-appropriately)@+-- +-- Guess what part of the state machine should be active, according to the+-- stack of open elements.  See 'resetInsertionMode' if the calculation is+-- being made as part of the tree construction parsing algorithm.+resetInsertionMode' :: TreeParserState -> TreeParserState+resetInsertionMode' state = resetInsertionMode'' $ openElements state+  where resetInsertionMode'' [] = switchMode' InBody+        resetInsertionMode'' ((_, e):es)+            | isLast && nodeIsElement "select" e' = switchMode' InSelect+            | nodeIsElement "select" e' = loopSelect es+            | not isLast && nodeIsElement "td" e' = switchMode' InCell+            | not isLast && nodeIsElement "th" e' = switchMode' InCell+            | nodeIsElement "tr" e' = switchMode' InRow+            | nodeIsElement "tbody" e' = switchMode' InTableBody+            | nodeIsElement "thead" e' = switchMode' InTableBody+            | nodeIsElement "tfoot" e' = switchMode' InTableBody+            | nodeIsElement "caption" e' = switchMode' InCaption+            | nodeIsElement "colgroup" e' = switchMode' InColumnGroup+            | nodeIsElement "table" e' = switchMode' InTable+            | nodeIsElement "template" e' = maybe state switchMode' . Y.listToMaybe $ templateInsertionModes state+            | not isLast && nodeIsElement "head" e' = switchMode' InHead+            | nodeIsElement "body" e' = switchMode' InBody+            | nodeIsElement "frameset" e' = switchMode' InFrameset+            | nodeIsElement "html" e' =+                switchMode' $ if Y.isNothing $ headElementPointer state+                    then BeforeHead+                    else AfterHead+            | isLast = switchMode' InBody+            | otherwise = resetInsertionMode'' es+          where isLast = null es+                e' = case fragmentContext state of+                    Nothing -> e+                    Just context -> fst context+                loopSelect [] = switchMode' InSelect+                loopSelect [_] = switchMode' InSelect+                loopSelect ((_, n):ns)+                    | nodeIsElement "template" n = switchMode' InSelect+                    | nodeIsElement "table" n = switchMode' InSelectInTable+                    | otherwise = loopSelect ns+        switchMode' mode = state+            { insertionMode = mode+            }+++-- | __HTML:__+--      @[MathML text integration point]+--      (https://html.spec.whatwg.org/multipage/parsing.html#mathml-text-integration-point)@+-- +-- Whether the markup tag introduces a section of less-structured text content+-- embedded within a MathML object.+atMathMLIntegration :: ElementParams -> Bool+atMathMLIntegration current =+    elementNamespace current == Just mathMLNamespace+    && elem (elementName current) ["mi", "mo", "mn", "ms", "mtext"]++-- | Whether the markup tag is specifically an @\<annotation-xml\>@ tag within+-- the MathML namespace.+isMathMLAnnotationXml :: ElementParams -> Bool+isMathMLAnnotationXml current+    | elementNamespace current == Just mathMLNamespace+        && elementName current == "annotation-xml" = True+    | otherwise = False++-- | __HTML:__+--      @[HTML integration point]+--      (https://html.spec.whatwg.org/multipage/parsing.html#html-integration-point)@+-- +-- Whether the markup tag introduces a section of HTML content embedded within+-- a MathML or SVG object.+atHtmlIntegration :: ElementParams -> Bool+atHtmlIntegration current+    | isMathMLAnnotationXml current+        && isIntegrationAttribute (M.lookup (Nothing, "encoding") $ elementAttributes current) = True+    | elementNamespace current == Just svgNamespace+        && elem (elementName current) ["foreignObject", "desc", "title"] = True+    | otherwise = False+  where isIntegrationAttribute (Just (_, value)) = case T.map toAsciiLower value of+            "text/html" -> True+            "application/xhtml+xml" -> True+            _ -> False+        isIntegrationAttribute _ = False
+ src/Web/Mangrove/Parse/Tree/Dispatcher.hs view
@@ -0,0 +1,123 @@+{-|+Description:    Token processing rules within non-HTML content.++Copyright:      (c) 2020 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      stable+Portability:    portable+-}+module Web.Mangrove.Parse.Tree.Dispatcher+    ( dispatcher+    , dispatchHtml+    ) where+++import qualified Control.Monad.Trans.State as N.S++import qualified Data.Text as T++import Web.Willow.DOM++import Web.Mangrove.Parse.Tree.Common+import Web.Mangrove.Parse.Tree.Patch+import Web.Willow.Common.Parser+import Web.Willow.Common.Parser.Switch++import Web.Mangrove.Parse.Tree.Initial+import Web.Mangrove.Parse.Tree.AfterAfterBody+import Web.Mangrove.Parse.Tree.AfterAfterFrameset+import Web.Mangrove.Parse.Tree.AfterBody+import Web.Mangrove.Parse.Tree.AfterFrameset+import Web.Mangrove.Parse.Tree.AfterHead+import Web.Mangrove.Parse.Tree.BeforeHead+import Web.Mangrove.Parse.Tree.BeforeHtml+import Web.Mangrove.Parse.Tree.Foreign+import Web.Mangrove.Parse.Tree.InBody+import Web.Mangrove.Parse.Tree.InCaption+import Web.Mangrove.Parse.Tree.InCell+import Web.Mangrove.Parse.Tree.InColumnGroup+import Web.Mangrove.Parse.Tree.InFrameset+import Web.Mangrove.Parse.Tree.InHead+import Web.Mangrove.Parse.Tree.InHeadNoscript+import Web.Mangrove.Parse.Tree.InRow+import Web.Mangrove.Parse.Tree.InSelect+import Web.Mangrove.Parse.Tree.InSelectInTable+import Web.Mangrove.Parse.Tree.InTable+import Web.Mangrove.Parse.Tree.InTableBody+import Web.Mangrove.Parse.Tree.InTableText+import Web.Mangrove.Parse.Tree.InTemplate+import Web.Mangrove.Parse.Tree.InText+++-- | __HTML:__+--      @[tree construction dispatcher]+--      (https://html.spec.whatwg.org/multipage/parsing.html#tree-construction-dispatcher)@+-- +-- Delegate parsing the binary stream to the appropriate content class: lenient+-- HTML (via insertion mode) or embedded, more-structured MathML/SVG content.+dispatcher :: TreeBuilder TreeOutput+dispatcher = dispatchToken+  where dispatchToken = do+            state <- N.S.get+            adjusted <- adjustedCurrentNode+            flip switch adjusted+                [ If_ (const $ null (openElements state)) dispatchHtml+                , If_ (\n -> (n >>= elementNamespace) == Just htmlNamespace) dispatchHtml+                , If_ (maybe False atMathMLIntegration) $ lookAhead next >>= switch+                    [ If_ (hasStartTagName $ T.pack "mglyph") treeForeign+                    , If_ (hasStartTagName $ T.pack "malignmark") treeForeign+                    , If_ isAnyStartTag dispatchHtml+                    , If_ isCharacter dispatchHtml+                    , Else_ treeForeign+                    ]+                , If_ (maybe False atHtmlIntegration) $ lookAhead next >>= switch+                    [ If_ isAnyStartTag dispatchHtml+                    , If_ isCharacter dispatchHtml+                    , Else_ treeForeign+                    ]+                -- 'isMathMLAnnotationXml' is less specific than+                -- 'atHtmlIntegration' and so needs to appear after it.+                , If_ (maybe False isMathMLAnnotationXml) $ lookAhead next >>= switch+                    [ If_ (hasStartTagName $ T.pack "svg") dispatchHtml+                    , Else_ treeForeign+                    ]+                , Else_ treeForeign+                ]+        hasStartTagName name t' = case tokenOut t' of+            StartTag d -> tagName d == name+            _ -> False+++-- | __HTML:__+--      @[the rules for parsing tokens in HTML content]+--      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-inhtml)@+-- +-- Defer processing of the current token to the instructions defined by the+-- active 'insertionMode'.+dispatchHtml :: TreeBuilder TreeOutput+dispatchHtml = N.S.get >>= \state -> case insertionMode state of+    Initial -> treeInitial+    BeforeHtml -> treeBeforeHtml+    BeforeHead -> treeBeforeHead+    InHead -> treeInHead+    InHeadNoscript -> treeInHeadNoscript+    AfterHead -> treeAfterHead+    InBody -> treeInBody+    InText -> treeInText+    InTable -> treeInTable+    InTableText -> treeInTableText+    InCaption -> treeInCaption+    InColumnGroup -> treeInColumnGroup+    InTableBody -> treeInTableBody+    InRow -> treeInRow+    InCell -> treeInCell+    InSelect -> treeInSelect+    InSelectInTable -> treeInSelectInTable+    InTemplate -> treeInTemplate+    AfterBody -> treeAfterBody+    InFrameset -> treeInFrameset+    AfterFrameset -> treeAfterFrameset+    AfterAfterBody -> treeAfterAfterBody+    AfterAfterFrameset -> treeAfterAfterFrameset
+ src/Web/Mangrove/Parse/Tree/Dispatcher.hs-boot view
@@ -0,0 +1,17 @@+{-|+Copyright:      (c) 2020 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com+-}+module Web.Mangrove.Parse.Tree.Dispatcher+    ( dispatcher+    , dispatchHtml+    ) where+++import Web.Mangrove.Parse.Tree.Common+import Web.Mangrove.Parse.Tree.Patch+++dispatcher :: TreeBuilder TreeOutput+dispatchHtml :: TreeBuilder TreeOutput
+ src/Web/Mangrove/Parse/Tree/Foreign.hs view
@@ -0,0 +1,212 @@+{-# LANGUAGE OverloadedStrings #-}++{-|+Description:    Token processing rules within non-HTML content.++Copyright:      (c) 2020 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      stable+Portability:    portable+-}+module Web.Mangrove.Parse.Tree.Foreign+    ( treeForeign+    ) where+++import qualified Control.Monad.Trans.State as N.S++import qualified Data.HashMap.Strict as M+import qualified Data.Maybe as Y+import qualified Data.Text as T++import Web.Willow.DOM++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tokenize.Common+import Web.Mangrove.Parse.Tree.Common+import Web.Mangrove.Parse.Tree.InBody+import Web.Mangrove.Parse.Tree.Patch+import Web.Willow.Common.Encoding.Character+import Web.Willow.Common.Parser+import Web.Willow.Common.Parser.Switch++import {-# SOURCE #-} Web.Mangrove.Parse.Tree.Dispatcher+++-- | __HTML:__+--      @[the rules for parsing tokens in foreign content]+--      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-inforeign)@+-- +-- The parsing instructions for non-HTML content, which follows a simpler+-- recovery model more in line with HTML.+treeForeign :: TreeBuilder TreeOutput+treeForeign = next >>= switch+    [ If isNull $ \t' ->+        consTreeError UnexpectedNullCharacter <$> insertCharacter (mapTokenOut (const $ Character replacementChar) t')+    , If isWhitespace insertCharacter+    , If isCharacter $ \t' -> setFramesetNotOk *> insertCharacter t'+    , If isComment insertComment+    , If isDoctype $ \t' ->+        packTreeErrors [UnexpectedDoctype $ tokenDocumentType t'] t'+    , If isEOF $ \t' -> push t' *> treeInBody+    , If (isStartTag+        [ "b"+        , "big"+        , "blockquote"+        , "body"+        , "br"+        , "center"+        , "code"+        , "dd"+        , "div"+        , "dl"+        , "dt"+        , "em"+        , "embed"+        , "h1"+        , "h2"+        , "h3"+        , "h4"+        , "h5"+        , "h6"+        , "head"+        , "hr"+        , "i"+        , "img"+        , "li"+        , "listing"+        , "menu"+        , "meta"+        , "nobr"+        , "ol"+        , "p"+        , "pre"+        , "ruby"+        , "s"+        , "small"+        , "span"+        , "strong"+        , "strike"+        , "sub"+        , "sup"+        , "table"+        , "tt"+        , "u"+        , "ul"+        , "var"+        ]) htmlStartTag+    , If (isStartTag ["font"]) $ \t' ->+        if any (`elem` ["color", "face", "size"]) (map fst . M.toList . tokenAttributes $ tokenOut t')+            then htmlStartTag t'+            else anyOtherStartTag t'+    , If isAnyStartTag anyOtherStartTag+    , If (isEndTag ["script"]) $ \t' -> do+        current <- currentNode+        state <- N.S.get+        if (current >>= elementNamespace) == Just mathMLNamespace+            then closeCurrentNode t' <* closeSvgScript+            else anyOtherEndTag (openElements state) t'+    , If isAnyEndTag $ \t' -> do+        state <- N.S.get+        anyOtherEndTag (openElements state) t'+    ]+  where anyOtherStartTag t' = case tokenOut t' of+            StartTag d -> do+                current <- adjustedCurrentNode+                let ns = Y.fromMaybe htmlNamespace $ current >>= elementNamespace+                    d' = case ns of+                        ns' | mathMLNamespace == ns' -> adjustMathMLAttributes d+                        ns' | svgNamespace == ns' -> adjustSvgAttributes $ case tagName d of+                            "altglyph" -> d { tagName = "altGlyph" }+                            "altglyphdef" -> d { tagName = "altGlyphDef" }+                            "altglyphitem" -> d { tagName = "altGlyphItem" }+                            "animatecolor" -> d { tagName = "animateColor" }+                            "animatemotion" -> d { tagName = "animateMotion" }+                            "animatetransform" -> d { tagName = "animateTransform" }+                            "clippath" -> d { tagName = "clipPath" }+                            "feblend" -> d { tagName = "feBlend" }+                            "fecolormatrix" -> d { tagName = "feColorMatrix" }+                            "fecomponenttransfer" -> d { tagName = "feComponentTransfer" }+                            "fecomposite" -> d { tagName = "feComposite" }+                            "feconvolvematrix" -> d { tagName = "feConvolveMatrix" }+                            "fediffuselighting" -> d { tagName = "feDiffuseLighting" }+                            "fedisplacementmap" -> d { tagName = "feDisplacementMap" }+                            "fedistantlight" -> d { tagName = "feDistantLight" }+                            "fedropshadow" -> d { tagName = "feDropShadow" }+                            "feflood" -> d { tagName = "feFlood" }+                            "fefunca" -> d { tagName = "feFuncA" }+                            "fefuncb" -> d { tagName = "feFuncB" }+                            "fefuncg" -> d { tagName = "feFuncG" }+                            "fefuncr" -> d { tagName = "feFuncR" }+                            "fegaussianblur" -> d { tagName = "feGaussianBlur" }+                            "feimage" -> d { tagName = "feImage" }+                            "femerge" -> d { tagName = "feMerge" }+                            "femergenode" -> d { tagName = "feMergeNode" }+                            "femorphology" -> d { tagName = "feMorphology" }+                            "feoffset" -> d { tagName = "feOffset" }+                            "fepointlight" -> d { tagName = "fePointLight" }+                            "fespecularlighting" -> d { tagName = "feSpecularLighting" }+                            "fespotlight" -> d { tagName = "feSpotLight" }+                            "fetile" -> d { tagName = "feTile" }+                            "feturbulence" -> d { tagName = "feTurbulence" }+                            "foreignobject" -> d { tagName = "foreignObject" }+                            "glyphref" -> d { tagName = "glyphRef" }+                            "lineargradient" -> d { tagName = "linearGradient" }+                            "radialgradient" -> d { tagName = "radialGradient" }+                            "textpath" -> d { tagName = "textPath" }+                            _ -> d+                        _ -> d+                    t'' = mapTokenOut (const $ StartTag d') t'+                if tagIsSelfClosing d'+                    then if tagName d' == "script" && ns == svgNamespace+                            then insertForeignNullElement ns t'' <* closeSvgScript+                            else insertForeignNullElement ns t''+                    else insertForeignElement ns t''+            _ -> packTreeErrors [] t'+        anyOtherEndTag [] t' = packTreeErrors [] t'+        anyOtherEndTag es@(e:_) t' = case tokenOut t' of+            EndTag d -> if T.map toAsciiLower (elementName $ snd e) == tagName d+                then loopEndTag d es t'+                else consTreeError UnexpectedElementWithImpliedEndTag <$> loopEndTag d es t'+            _ -> packTreeErrors [] t'+        loopEndTag d es t' = either id id <$> loopEndTag' d es t'+        loopEndTag' _ [] t' = Left <$> packTreeErrors [] t'+        loopEndTag' _ [_] t' = Left <$> packTreeErrors [] t'+        loopEndTag' d (e:es) t'+            | T.map toAsciiLower (elementName $ snd e) == tagName d =+                Right <$> closeCurrentNode t'+            | otherwise = case es of+                (e':_) | elementNamespace (snd e') == Just htmlNamespace -> push t' *> fmap Left dispatchHtml+                _ -> do+                    recurse <- loopEndTag' d es t'+                    case recurse of+                        l@(Left _) -> return l+                        Right clear -> do+                            close <- closeCurrentNode_+                            return . Right $ close ++| clear+        tokenAttributes (StartTag d) = tagAttributes d+        tokenAttributes (EndTag d) = tagAttributes d+        tokenAttributes _ = M.empty+        htmlStartTag t' = do+            isFragment <- inFragment+            consTreeError UnexpectedHtmlElementInForeignContent <$> if isFragment+                then anyOtherStartTag t'+                else do+                    push t'+                    close <- closeCurrentNode_+                    clear <- clearToIntegration+                    packTree_ $ close ++ clear+        clearToIntegration = do+            current <- currentNode+            let mathML = maybe False atMathMLIntegration current+                html = maybe False atHtmlIntegration current+                ns = Y.fromMaybe htmlNamespace $ current >>= elementNamespace+            if mathML || html || ns == htmlNamespace+                then return []+                else do+                    close <- closeCurrentNode_+                    clear <- clearToIntegration+                    return $ close ++ clear+        closeSvgScript = undefined
+ src/Web/Mangrove/Parse/Tree/InBody.hs view
@@ -0,0 +1,971 @@+{-# LANGUAGE OverloadedStrings #-}++{-|+Description:    Token processing rules within the @\<body\>@ section.++Copyright:      (c) 2020-2021 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      provisional+Portability:    portable+-}+module Web.Mangrove.Parse.Tree.InBody+    ( treeInBody+    ) where+++import qualified Control.Monad as N+import qualified Control.Monad.Trans.State as N.S++import qualified Data.HashMap.Strict as M+import qualified Data.List as L+import qualified Data.Maybe as Y+import qualified Data.Text as T++import Web.Willow.DOM++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tokenize.Common+import Web.Mangrove.Parse.Tree.Common+import Web.Mangrove.Parse.Tree.InHead+import Web.Mangrove.Parse.Tree.InTemplate+import Web.Mangrove.Parse.Tree.InText+import Web.Mangrove.Parse.Tree.Patch+import Web.Willow.Common.Parser+import Web.Willow.Common.Parser.Switch++import {-# SOURCE #-} Web.Mangrove.Parse.Tree.Dispatcher++import Control.Applicative ( (<|>) )+import Data.Functor ( ($>) )+++-- | __HTML:__+--      @[the "in body" insertion mode]+--      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-inbody)@+-- +-- The parsing instructions corresponding to the 'InBody' section of the state+-- machine.+treeInBody :: TreeBuilder TreeOutput+treeInBody = next >>= switch+    [ If isNull $ packTreeErrors [UnexpectedNullCharacter]+    , If isWhitespace $ \t' -> do+        format <- reconstructFormattingElements+        insert <- insertCharacter t'+        return $ format ++| insert+    , If isCharacter $ \t' -> do+        setFramesetNotOk+        format <- reconstructFormattingElements+        insert <- insertCharacter t'+        return $ format ++| insert+    , If isComment insertComment+    , If isDoctype $ \t' ->+        packTreeErrors [UnexpectedDoctype $ tokenDocumentType t'] t'+    , If (isStartTag ["html"]) $ \t' -> do+        errs <- packTreeErrors [NestedSingletonElement] t'+        elements <- openElements <$> N.S.get+        if Y.isJust $ L.find (nodeIsElement "template" . snd) elements+            then return errs+            else case reverse elements of+                [] -> return errs+                ((i, _):_) -> do+                    add <- mapM (addAttribute InHtmlElement i) .+                        M.toList . tagAttributes $ tokenTag t'+                    return $ errs |++ concat add+    , If (isStartTag+        [ "base"+        , "basefont"+        , "bgsound"+        , "link"+        , "meta"+        , "noframes"+        , "script"+        , "style"+        , "template"+        , "title"+        ]) $ \t' -> do+            push t'+            treeInHead+    , If (isEndTag ["template"]) $ \t' -> do+        push t'+        treeInHead+    , If (isStartTag ["body"]) $ \t' -> do+        errs <- packTreeErrors [NestedSingletonElement] t'+        elements <- openElements <$> N.S.get+        case reverse elements of+            [] -> return errs+            [_] -> return errs+            es | Y.isJust $ L.find (nodeIsElement "template" . snd) es -> return errs+            (_:(i, e):es)+                | not (nodeIsElement "body" e) -> return errs+                | otherwise -> do+                    setFramesetNotOk+                    add <- mapM+                        (addAttribute (RelativeLocation . fromIntegral $ length es) i) .+                        M.toList . tagAttributes $ tokenTag t'+                    return $ errs |++ concat add+    , If (isStartTag ["frameset"]) $ \t' -> do+        state <- N.S.get+        let elements = map snd $ openElements state+            err = FramesetInBody $ tokenElement t'+        errs <- packTreeErrors [err] t'+        case reverse elements of+            [] -> return errs+            [_] -> return errs+            (_:e:_) | not (nodeIsElement "body" e) -> return errs+            _ | not (framesetOk state) -> return errs+            _ -> do+                clear <- N.replicateM (length elements - 1) dropCurrentNode+                insert <- insertElement t'+                switchMode InFrameset+                return . consTreeError err $ concat clear ++| insert+    , If isEOF $ \t' -> do+        modes <- templateInsertionModes <$> N.S.get+        if null modes+            then do+                hasUnexpected <- hasUnexpectedOpenElement+                if hasUnexpected+                    then consTreeError UnexpectedElementWithImpliedEndTag <$> stopParsing t'+                    else stopParsing t'+            else do+                push t'+                treeInTemplate+    , If (isEndTag ["body"]) $ \t' -> do+        switchMode AfterBody+        hasBody <- hasInScope ["body"]+        hasUnexpected <- hasUnexpectedOpenElement+        if hasBody+            then if hasUnexpected+                then packTreeErrors [UnexpectedElementWithImpliedEndTag] t'+                else packTreeErrors [] t'+            else packTreeErrors [UnmatchedEndTag $ tokenElement t'] t'+    , If (isEndTag ["html"]) $ \t' -> do+        push t'+        switchMode AfterBody+        hasBody <- hasInScope ["body"]+        hasUnexpected <- hasUnexpectedOpenElement+        if hasBody+            then if hasUnexpected+                then consTreeError UnexpectedElementWithImpliedEndTag <$> dispatchHtml+                else packTreeErrors_ []+            else consTreeError (UnmatchedEndTag $ tokenElement t') <$> dispatchHtml+    , If (isStartTag+        [ "address"+        , "article"+        , "aside"+        , "blockquote"+        , "center"+        , "details"+        , "dialog"+        , "dir"+        , "div"+        , "dl"+        , "fieldset"+        , "figcaption"+        , "figure"+        , "footer"+        , "header"+        , "hgroup"+        , "main"+        , "menu"+        , "nav"+        , "ol"+        , "p"+        , "section"+        , "summary"+        , "ul"+        ]) $ \t' -> do+            hasP <- hasInButtonScope ["p"]+            if hasP+                then do+                    close <- closePElement+                    insert <- insertElement t'+                    return $ close ++| insert+                else insertElement t'+    , If (isStartTag $ map T.unpack headerNames) $ \t' -> do+        current <- currentNode+        hasP <- hasInButtonScope ["p"]+        close <- if hasP+            then closePElement+            else return []+        nest <- case fmap elementName current of+            Just h | elem h headerNames -> consTreeError_ OverlappingHeaderElements <$> closeCurrentNode_+            _ -> return []+        insert <- insertElement t'+        return $ close ++ nest ++| insert+    , If (isStartTag ["pre", "listing"]) $ \t' -> do+        lineFeed <- next+        lineFeedState <- if tokenOut lineFeed == Character '\n'+            then return $ tokenState lineFeed+            else push lineFeed $> Nothing+        setFramesetNotOk+        hasP <- hasInButtonScope ["p"]+        close <- if hasP+            then closePElement+            else return []+        insert <- insertElement $ mapTokenState' t' (lineFeedState <|>)+        return $ close ++| insert+    , If (isStartTag ["form"]) $ \t' -> do+        state <- N.S.get+        let hasTemplate = any (nodeIsElement "template" . snd) $ openElements state+        if hasTemplate || Y.isJust (formElementPointer state)+            then packTreeErrors [] t'+            else do+                hasP <- hasInButtonScope ["p"]+                close <- if hasP+                    then closePElement+                    else return []+                N.S.modify $ \state' -> state'+                    { formElementPointer = Just $ elementIndex state'+                    }+                insert <- insertElement t'+                return $ close ++| insert+    , If (isStartTag ["li"]) $ \t' -> do+        setFramesetNotOk+        state <- N.S.get+        clear <- liLoop $ openElements state+        hasP <- hasInButtonScope ["p"]+        close <- if hasP+            then closePElement+            else return []+        insert <- insertElement t'+        return $ clear ++ close ++| insert+    , If (isStartTag ["dd", "dt"]) $ \t' -> do+        setFramesetNotOk+        state <- N.S.get+        clear <- ddLoop $ openElements state+        hasP <- hasInButtonScope ["p"]+        close <- if hasP+            then closePElement+            else return []+        insert <- insertElement t'+        return $ clear ++ close ++| insert+    , If (isStartTag ["plaintext"]) $ \t' -> do+        hasP <- hasInButtonScope ["p"]+        close <- if hasP+            then closePElement+            else return []+        insert <- insertElement . mapTokenState t' $ \state -> state+            { currentState = PlainTextState+            }+        return $ close ++| insert+    , If (isStartTag ["button"]) $ \t' -> do+        hasButton <- hasInScope ["button"]+        nested <- if hasButton+            then do+                generate <- generateEndTags impliedEndTags+                clear <- closeElement "button"+                return . consTreeError_ NestedNonRecursiveElement $ generate ++ clear+            else return []+        format <- reconstructFormattingElements+        insert <- insertElement t'+        setFramesetNotOk+        return $ nested ++ format ++| insert+    , If (isEndTag+        [ "address"+        , "article"+        , "aside"+        , "blockquote"+        , "button"+        , "center"+        , "details"+        , "dialog"+        , "dir"+        , "div"+        , "dl"+        , "fieldset"+        , "figcaption"+        , "figure"+        , "footer"+        , "header"+        , "hgroup"+        , "listing"+        , "main"+        , "menu"+        , "nav"+        , "ol"+        , "pre"+        , "section"+        , "summary"+        , "ul"+        ]) $ \t' -> do+            let d = tokenTag t'+            hasMatch <- hasInScope [tagName d]+            if hasMatch+                then do+                    generate <- generateEndTags impliedEndTags+                    current <- currentNode+                    let errF = if maybe True (nodeIsElement $ tagName d) current+                            then id+                            else consTreeError_ UnexpectedElementWithImpliedEndTag+                    clear <- closeElement $ tagName d+                    packTree t' $ generate ++ errF clear+                else packTreeErrors [UnmatchedEndTag $ tokenElement t'] t'+    , If (isEndTag ["form"]) $ \t' -> do+        state <- N.S.get+        if Y.isJust . L.find (nodeIsElement "template" . snd) $ openElements state+            then do+                hasTemplate <- hasInScope ["template"]+                if hasTemplate+                    then do+                        generate <- generateEndTags impliedEndTags+                        current <- currentNode+                        let errF = if maybe True (nodeIsElement "form") current+                                then id+                                else consTreeError_ UnexpectedElementWithImpliedEndTag+                        clear <- closeElement "form"+                        packTree t' $ generate ++ errF clear+                    else packTreeErrors [] t'+            else do+                let formElement = formElementPointer state+                N.S.put $ state+                    { formElementPointer = Nothing+                    }+                hasFormElement <- maybe (return False) hasIndexInScope formElement+                if hasFormElement+                    then do+                        generate <- generateEndTags impliedEndTags+                        let current = maybe 0 fst . Y.listToMaybe $ openElements state+                            es = takeWhile ((/=) (Y.fromMaybe current formElement) . fst) $+                                openElements state+                            errF = if null es+                                then id+                                else consTreeError_ UnexpectedElementWithImpliedEndTag+                        close <- closeAncestorNode_ . fromIntegral $ length es+                        packTree t' $ generate ++ errF close+                    else packTreeErrors [UnmatchedEndTag $ tokenElement t'] t'+    , If (isEndTag ["p"]) $ \t' -> do+        hasP <- hasInButtonScope ["p"]+        ps <- if hasP+            then closePElement+            else fmap (consTreeError_ . UnmatchedEndTag $ tokenElement t') . insertNullElement_ $ emptyTagParams+                { tagName = "p"+                }+        packTree t' ps+    , If (isEndTag ["li"]) $ \t' -> do+        hasLi <- hasInListItemScope ["li"]+        if hasLi+            then do+                generate <- generateEndTags $ L.delete "li" impliedEndTags+                current <- currentNode+                let errF = if maybe True (nodeIsElement "li") current+                        then id+                        else consTreeError_ UnexpectedElementWithImpliedEndTag+                clear <- closeElement "li"+                packTree t' $ generate ++ errF clear+            else packTreeErrors [UnmatchedEndTag $ tokenElement t'] t'+    , If (isEndTag ["dd", "dt"]) $ \t' -> do+        let d = tokenTag t'+        hasMatch <- hasInScope [tagName d]+        if hasMatch+            then do+                generate <- generateEndTags $ L.delete (tagName d) impliedEndTags+                current <- currentNode+                let errF = if maybe True (nodeIsElement $ tagName d) current+                        then id+                        else consTreeError_ UnexpectedElementWithImpliedEndTag+                clear <- closeElement $ tagName d+                packTree t' $ generate ++ errF clear+            else packTreeErrors [UnmatchedEndTag $ tokenElement t'] t'+    , If (isEndTag $ map T.unpack headerNames) $ \t' -> do+        let d = tokenTag t'+        hasMatch <- hasInScope headerNames+        if hasMatch+            then do+                generate <- generateEndTags $ L.delete (tagName d) impliedEndTags+                current <- currentNode+                let errF = if maybe True (nodeIsElement $ tagName d) current+                        then id+                        else consTreeError_ UnexpectedElementWithImpliedEndTag+                clear <- closeElements headerNames+                packTree t' $ generate ++ errF clear+            else packTreeErrors [UnmatchedEndTag $ tokenElement t'] t'+    , If (isStartTag ["a"]) $ \t' -> do+        state <- N.S.get+        let formatting = activeFormattingElements state+            active = Y.fromMaybe [] $ Y.listToMaybe formatting+        nested <- case L.find (\f -> tagName (snd f) == "a") active of+            Just (i, _) -> do+                adopt <- consTreeError NestedElementForAdoptionAgency <$> runAdoptionAgency t'+                open' <- openElements <$> N.S.get+                close <- case break ((==) i . fst) open' of+                    (_, []) -> return []+                    (es1, _) -> do+                        N.S.modify $ \state' -> state'+                            { activeFormattingElements = case activeFormattingElements state' of+                                [] -> []+                                (es:ess) -> filter ((/=) i . fst) es : ess+                            }+                        closeAncestorNode_ . fromIntegral $ length es1+                return $ adopt |++ close+            Nothing -> packTreeErrors [] t'+        format <- reconstructFormattingElements+        insert <- insertFormattingElement t'+        return $ nested |++| format ++| insert+    , If (isStartTag+        [ "b"+        , "big"+        , "code"+        , "em"+        , "font"+        , "i"+        , "s"+        , "small"+        , "strike"+        , "strong"+        , "tt"+        , "u"+        ]) $ \t' -> do+            format <- reconstructFormattingElements+            insert <- insertFormattingElement t'+            return $ format ++| insert+    , If (isStartTag ["nobr"]) $ \t' -> do+        format <- reconstructFormattingElements+        hasNobr <- hasInScope ["nobr"]+        nested <- if hasNobr+            then do+                --BUG: Adoption agency + insertion may duplicate the errors?+                adopt <- consTreeError NestedElementForAdoptionAgency <$> runAdoptionAgency t'+                format' <- reconstructFormattingElements+                return $ adopt |++ format'+            else packTreeErrors [] t'+        insert <- insertFormattingElement t'+        return $ format ++| nested |++| insert+    , If (isEndTag+        [ "a"+        , "b"+        , "big"+        , "code"+        , "em"+        , "font"+        , "i"+        , "nobr"+        , "s"+        , "small"+        , "strike"+        , "strong"+        , "tt"+        , "u"+        ]) runAdoptionAgency+    , If (isStartTag ["applet", "marquee", "object"]) $ \t' -> do+        setFramesetNotOk+        format <- reconstructFormattingElements+        insertFormattingMarker+        insert <- insertElement t'+        return $ format ++| insert+    , If (isEndTag ["applet", "marquee", "object"]) $ \t' -> do+        let d = tokenTag t'+        hasMatch <- hasInScope [tagName d]+        if hasMatch+            then do+                generate <- generateEndTags impliedEndTags+                current <- currentNode+                let errF = if maybe True (nodeIsElement $ tagName d) current+                        then id+                        else consTreeError_ UnexpectedElementWithImpliedEndTag+                clear <- closeElement $ tagName d+                clearFormattingElements+                packTree t' $ generate ++ errF clear+            else packTreeErrors [UnmatchedEndTag $ tokenElement t'] t'+    , If (isStartTag ["table"]) $ \t' -> do+        hasP <- hasInButtonScope ["p"]+        quirks <- (== FullQuirks) . quirksMode <$> N.S.get+        clear <- if hasP && not quirks+            then closePElement+            else return []+        insert <- insertElement t'+        setFramesetNotOk+        switchMode InTable+        return $ clear ++| insert+    , If (isEndTag ["br"]) $ \t' -> do+        push . mapTokenErrs (BREndTag :) . flip mapTokenOut t' $+            const (StartTag $ (tokenTag t') { tagAttributes = M.empty })+        treeInBody+    , If (isStartTag ["area", "br", "embed", "img", "keygen", "wbr"]) $ \t' -> do+        setFramesetNotOk+        format <- reconstructFormattingElements+        insert <- insertNullElement t'+        return $ format ++| insert+    , If (isStartTag ["input"]) $ \t' -> do+        case fmap snd . L.find (\a -> fst a == "type") . M.toList . tagAttributes $ tokenTag t' of+            Nothing -> setFramesetNotOk+            Just "hidden" -> return ()+            Just _ -> setFramesetNotOk+        format <- reconstructFormattingElements+        insert <- insertNullElement t'+        return $ format ++| insert+    , If (isStartTag ["param", "source", "track"]) insertNullElement+    , If (isStartTag ["hr"]) $ \t' -> do+        setFramesetNotOk+        hasP <- hasInButtonScope ["p"]+        close <- if hasP+            then closePElement+            else return []+        insert <- insertNullElement t'+        return $ close ++| insert+    , If (isStartTag ["image"]) $ \t' -> do+        push . mapTokenErrs (ObsoleteTagName "image" :) . flip mapTokenOut t' $+            const (StartTag $ (tokenTag t') { tagName = "img" })+        treeInBody+    , If (isStartTag ["textarea"]) $ \t' -> do+        lineFeed <- next+        lineFeedState <- if tokenOut lineFeed == Character '\n'+            then return $ tokenState lineFeed+            else push lineFeed $> Nothing+        setFramesetNotOk+        genericRCDataElement $ mapTokenState' t' (lineFeedState <|>)+    , If (isStartTag ["xmp"]) $ \t' -> do+        setFramesetNotOk+        hasP <- hasInButtonScope ["p"]+        close <- if hasP+            then closePElement+            else return []+        format <- reconstructFormattingElements+        text <- genericRawTextElement t'+        return $ close ++ format ++| text+    , If (isStartTag ["iframe"]) $ \t' -> do+        setFramesetNotOk+        genericRawTextElement t'+    , If (isStartTag ["noembed"]) genericRawTextElement+    , If (isStartTag ["noscript"]) $ \t' -> do+        state <- N.S.get+        if scriptingEnabled state+            then genericRawTextElement t'+            else do -- "any other start tag"+                format <- reconstructFormattingElements+                insert <- insertElement t'+                return $ format ++| insert+    , If (isStartTag ["select"]) $ \t' -> do+        setFramesetNotOk+        state <- N.S.get+        switchMode $ if elem (insertionMode state)+                [ InTable+                , InCaption+                , InTableBody+                , InRow+                , InCell+                ]+            then InSelectInTable+            else InSelect+        format <- reconstructFormattingElements+        insert <- insertElement t'+        return $ format ++| insert+    , If (isStartTag ["optgroup", "option"]) $ \t' -> do+        current <- currentNode+        format <- case fmap elementName current of+                Just "option" -> do+                    close <- closeCurrentNode_+                    format' <- reconstructFormattingElements+                    return $ close ++ format'+                _ -> reconstructFormattingElements+        insert <- insertElement t'+        return $ format ++| insert+    , If (isStartTag ["rb", "rtc"]) $ \t' -> do+        hasRuby <- hasInScope ["ruby"]+        generate <- if hasRuby+            then do+                generate' <- generateEndTags impliedEndTags+                current <- currentNode+                let errF = if maybe True (nodeIsElement "ruby") current+                        then id+                        else consTreeError_ UnexpectedElementWithImpliedEndTag+                return $ errF generate'+            else return []+        insert <- insertElement t'+        return $ generate ++| insert+    , If (isStartTag ["rp", "rt"]) $ \t' -> do+        hasRuby <- hasInScope ["ruby"]+        generate <- if hasRuby+            then do+                generate' <- generateEndTags $ L.delete "rtc" impliedEndTags+                current <- currentNode+                let errF = if maybe True (\e -> nodeIsElement "ruby" e || nodeIsElement "rtc" e) current+                        then id+                        else consTreeError_ UnexpectedElementWithImpliedEndTag+                return $ errF generate'+            else return []+        insert <- insertElement t'+        return $ generate ++| insert+    , If (isStartTag ["math"]) $ \t' -> do+        let d = tokenTag t'+            insertF+                | tagIsSelfClosing d = insertForeignNullElement+                | otherwise = insertForeignElement+        format <- reconstructFormattingElements+        insert <- insertF mathMLNamespace $ mapTokenOut (const . StartTag $ adjustMathMLAttributes d) t'+        return $ format ++| insert+    , If (isStartTag ["svg"]) $ \t' -> do+        let d = tokenTag t'+            insertF+                | tagIsSelfClosing d = insertForeignNullElement+                | otherwise = insertForeignElement+        format <- reconstructFormattingElements+        insert <- insertF svgNamespace $ mapTokenOut (const . StartTag $ adjustSvgAttributes d) t'+        return $ format ++| insert+    , If (isStartTag+        [ "caption"+        , "col"+        , "colgroup"+        , "frame"+        , "head"+        , "tbody"+        , "td"+        , "tfoot"+        , "th"+        , "thead"+        , "tr"+        ]) $ \t' ->+            packTreeErrors [UnexpectedDescendantElement $ tokenElement t'] t'+    , If isAnyStartTag $ \t' -> do+        format <- reconstructFormattingElements+        insert <- insertElement t'+        return $ format ++| insert+    , If isAnyEndTag $ \t' -> do+        state <- N.S.get+        anyOtherEndTag (openElements state) t'+    ]+  where headerNames = [ T.snoc "h" i | i <- ['1'..'6'] ]+        hasUnexpectedOpenElement =+            hasOpenElementExcept+                [ "dd"+                , "dt"+                , "li"+                , "optgroup"+                , "option"+                , "p"+                , "rb"+                , "rp"+                , "rt"+                , "rtc"+                , "tbody"+                , "td"+                , "tfoot"+                , "th"+                , "thead"+                , "tr"+                , "body"+                , "html"+                ]++-- | __HTML:__+--      the "any other end tag" entry in @[the "in body" insertion mode]+--      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-inbody)@+-- +-- Delegate a token to to the 'InBody' section of the state machine, but skip+-- the token-dependent behaviour and instead simply treat it according to the+-- fallback case for 'EndTag' tokens.+anyOtherEndTag+    :: [(NodeIndex, ElementParams)]+        -- ^ The stack of open elements.+    -> TreeInput+        -- ^ The token to process.+    -> TreeBuilder TreeOutput+anyOtherEndTag [] t' = packTreeErrors [] t'+anyOtherEndTag ((i, e):es) t' = case tokenOut t' of+    EndTag d | nodeIsElement (tagName d) e -> do+        generate <- generateEndTags $ L.delete (tagName d) impliedEndTags+        elementIndices <- map fst . openElements <$> N.S.get+        let errF = if Y.listToMaybe elementIndices == Just i+                then id+                else consTreeError_ UnexpectedElementWithImpliedEndTag+            count = length (takeWhile (/= i) elementIndices) + 1+        clear <- clearCount $ fromIntegral count+        packTree t' $ generate ++ errF clear+    EndTag _ -> if nodeIsSpecial e+        then packTreeErrors [UnexpectedElementWithImpliedEndTag] t'+        else anyOtherEndTag es t'+    _ -> packTreeErrors [] t'+++-- | Check whether a node is in 'scopeElements' -- the tags which break+-- open-tag searches.+isScopeElement :: ElementParams -> Bool+isScopeElement d' = elem (Y.fromMaybe T.empty $ elementNamespace d', elementName d') scopeElements++-- | Check whether a node is in 'specialElements' -- the tags which are subject+-- to custom handling.+isSpecialElement :: ElementParams -> Bool+isSpecialElement d' = elem (Y.fromMaybe T.empty $ elementNamespace d', elementName d') specialElements+++-- | __HTML:__+--      the substeps spanned by the /"Loop"/ for processing @\<dd\>@ and+--      @\<dt\>@ tags in @[the "in body" insertion mode]+--      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-inbody)@+-- +-- @\<dd\>@ and @\<dt\>@ elements should close most opened elements if the+-- relevant end tag is left implied.+ddLoop+    :: [(NodeIndex, ElementParams)]+        -- ^ The stack of open elements.+    -> TreeBuilder [Patch]+ddLoop [] = return []+ddLoop ((_, e):es)+    | nodeIsElement "dd" e = do+        generate <- generateEndTags $ L.delete "dd" impliedEndTags+        current <- currentNode+        let err = case current of+                Just e' | not $ nodeIsElement "dd" e' -> consTreeError_ UnexpectedElementWithImpliedEndTag+                _ -> id+        clear <- closeElement "dd"+        return . err $ generate ++ clear+    | nodeIsElement "dt" e = do+        generate <- generateEndTags $ L.delete "dt" impliedEndTags+        current <- currentNode+        let err = case current of+                Just e' | not $ nodeIsElement "dt" e' -> consTreeError_ UnexpectedElementWithImpliedEndTag+                _ -> id+        clear <- closeElement "dt"+        return . err $ generate ++ clear+    | nodeIsElement "address" e = ddLoop es+    | nodeIsElement "div" e = ddLoop es+    | nodeIsElement "p" e = ddLoop es+    | isSpecialElement e = return []+    | otherwise = ddLoop es++-- | __HTML:__+--      the substeps spanned by the /"Loop"/ for processing @\<li\>@ tags in+--      @[the "in body" insertion mode]+--      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-inbody)@+-- +-- @\<li\>@ elements should close most opened elements if the relevant end tag+-- is left implied.+liLoop+    :: [(NodeIndex, ElementParams)]+        -- ^ The stack of open elements.+    -> TreeBuilder [Patch]+liLoop [] = return []+liLoop ((_, e):es)+    | nodeIsElement "li" e = do+        generate <- generateEndTags $ L.delete "li" impliedEndTags+        current <- currentNode+        let err = case current of+                Just e' | not $ nodeIsElement "li" e' -> consTreeError_ UnexpectedElementWithImpliedEndTag+                _ -> id+        clear <- closeElement "li"+        return . err $ generate ++ clear+    | nodeIsElement "address" e = liLoop es+    | nodeIsElement "div" e = liLoop es+    | nodeIsElement "p" e = liLoop es+    | isSpecialElement e = return []+    | otherwise = liLoop es+++-- | __HTML:__+--      @[adoption agency algorithm]+--      (https://html.spec.whatwg.org/multipage/parsing.html#adoption-agency-algorithm)@+-- +-- Handle a misnested formatting element by closing its ancestors up to a+-- stable branch point, and then reconstructing the tree in a more logical+-- order.+runAdoptionAgency :: TreeInput -> TreeBuilder TreeOutput+runAdoptionAgency t' = case tokenOut t' of+    StartTag d -> runAdoptionAgency' d+    EndTag d -> runAdoptionAgency' d+    _ -> packTreeErrors [] t'+  where runAdoptionAgency' d = do+            current <- currentNode+            index <- currentNodeIndex+            active <- concat . activeFormattingElements <$> N.S.get+            let isCurrent = maybe False (nodeIsElement $ tagName d) current+                inActive = Y.isJust $ L.find (\(i, _) -> Just i == index) active+            if isCurrent && not inActive+                then closeCurrentNode_ >>= packTree t'+                else runAdoptionAgencyOuterLoop 8 t' d++-- | __HTML:__+--      the substeps spanned by the /"Outer loop"/ within the+--      @[adoption agency algorithm]+--      (https://html.spec.whatwg.org/multipage/parsing.html#adoption-agency-algorithm)@+-- +-- Determine whether the current formatting element close tag is mis-nested,+-- and if so, close the element and reconstruct an equivalent formatting tree.+runAdoptionAgencyOuterLoop+    :: Word+        -- ^ The maximum number of times the loop should be run, to avoid+        -- overly-costly input.  Note that this implementation counts down+        -- rather than up, and so this function should generally be passed @8@+        -- when it is originally called to implement the official behaviour.+    -> TreeInput+        -- ^ The token to process.+    -> TagParams+        -- ^ The inner element data extracted from the second parameter.+    -> TreeBuilder TreeOutput+-- 4.+runAdoptionAgencyOuterLoop 0 t' _ = packTreeErrors [] t'+runAdoptionAgencyOuterLoop i t' d = do+    _ <- error "Adoption agency not yet implemented"+    -- 6.+    state <- N.S.get+    let formatting = activeFormattingElements state+        open = openElements state+    case L.find (\f -> tagName (snd $ snd f) == tagName d) . zip [0..] $ concat formatting of+        Nothing -> anyOtherEndTag open t'+        Just (indexFormat, formattingElement) -> do+            let removeFormattingElement = N.S.put $ state+                    { activeFormattingElements = map (L.delete formattingElement) formatting+                    }+            -- 7.+            case L.elemIndex (fst formattingElement) (map fst open) of+                Nothing -> do+                    removeFormattingElement+                    packTreeErrors [IncompletelyClosedFormattingElement] t'+                Just indexOpen -> do+                    -- 8.  Make use of the fact we can get an index rather than+                    -- just a 'Bool' when checking if the formatting element is+                    -- in the stack of open elements, to check if it's in+                    -- scope ('hasInScope' only checks the tag name, not UID).+                    let descendants = take indexOpen open+                        formattingNode = head $ drop indexOpen open+                    if any (isScopeElement . snd) descendants+                        then packTreeErrors [UnexpectedFormattingElementOutOfScope] t'+                        else do+                            -- 9.+                            let overlap = if maybe False (fst formattingElement ==) . Y.listToMaybe $ map fst open+                                    then consTreeError_ OverlappingFormattingElements+                                    else id+                            -- 10.+                            case L.find (isSpecialElement . snd) $ reverse descendants of+                                Nothing -> do+                                    -- 11.+                                    clear <- clearCount $ fromIntegral indexOpen+                                    close <- closeCurrentNode_+                                    removeFormattingElement+                                    packTree t' $ overlap clear ++ close+                                Just furthestBlock -> do+                                    -- 14.+                                    (bookmark, lastTag) <-+                                        runAdoptionAgencyInnerLoop 3 indexFormat formattingElement furthestBlock furthestBlock .+                                        drop 1 $ dropWhile (/= furthestBlock) open+                                    -- 15.+                                    reparent <- case L.elemIndex (fst lastTag) $ map fst open of+                                            Nothing -> return []+                                            Just indexInner -> if indexInner < indexOpen+                                                then closeAncestorNodes_+                                                    (fromIntegral indexInner)+                                                    (fromIntegral $ indexOpen - indexInner)+                                                else return []+                                    -- 16.+                                    newElement <- createElement $ snd formattingNode+                                    let newTag = unpackNodeData <$> newElement+                                    N.S.modify $ \state' -> state'+                                        -- 19.+                                        { activeFormattingElements =+                                            let findBookmark _ [] = []+                                                findBookmark bookmark' (es:ess)+                                                    | bookmark' > l = es : findBookmark (bookmark' - l) ess+                                                    | otherwise = (ds ++ newTag : drop 1 as) : ess+                                                  where l = length es+                                                        (ds, as) = splitAt bookmark' es+                                            in  findBookmark bookmark formatting+                                        -- 20.+                                        , openElements =+                                            let es = L.delete formattingNode open+                                                (ds, as) = break (== furthestBlock) es+                                            in  ds ++ newElement : as+                                        }+                                    -- 21.+                                    recurse <- runAdoptionAgencyOuterLoop (i - 1) t' d+                                    packTree t' $ overlap reparent ++ treePatches recurse++-- | __HTML:__+--      the substeps spanned by the /"Inner loop"/ within the+--      @[adoption agency algorithm]+--      (https://html.spec.whatwg.org/multipage/parsing.html#adoption-agency-algorithm)@+-- +-- Close ancestor elements until reaching a stable node for the new tree, and+-- then reconstruct an equivalent tree.+runAdoptionAgencyInnerLoop+    :: Word+        -- ^ __HTML:__+        --      @inner loop counter@+        -- +        -- The maximum number of times the loop should be run, to avoid+        -- overly-costly input.  Note that this implementation counts down+        -- rather than up, and so this function should generally be passed @3@.+    -> Int+        -- ^ __HTML:__+        --      the bookmark noting the position of @formatting element@+    -> (NodeIndex, TagParams)+        -- ^ __HTML:__+        --      @formatting element@+        -- +        -- The node selected as the base of the reparenting process.+    -> (NodeIndex, ElementParams)+        -- ^ __HTML:__+        --      @last node@+        -- +        -- The child node actively being reparented.+    -> (NodeIndex, ElementParams)+        -- ^ __HTML:__+        --      @furthest block@+        -- +        -- The most senior descendant of @formatting element@ which will serve+        -- as a secondary fixed point.+    -> [(NodeIndex, ElementParams)]+        -- ^ The stack of open elements; @inner node@ is derived from this.+    -> TreeBuilder (Int, (NodeIndex, TagParams))+runAdoptionAgencyInnerLoop _ bookmark formattingElement _ _ [] = return (bookmark, formattingElement)+runAdoptionAgencyInnerLoop i bookmark formattingElement lastNode furthestBlock (innerNode:ns)+    -- 4.+    | fst formattingElement == fst innerNode = return (bookmark, formattingElement)+    | otherwise = do+        state <- N.S.get+        let formatting = activeFormattingElements state+        case L.elemIndex innerTag $ concat formatting of+            Just innerIndex -> if i == 0+                -- 5. / 6.+                then do+                    N.S.modify $ \state' -> state'+                        --BUG: Not reflected in the patch list.+                        { openElements = L.delete innerNode $ openElements state'+                        , activeFormattingElements = map (L.delete innerTag) formatting+                        }+                    let bookmark'+                            | innerIndex <= bookmark = pred bookmark+                            | otherwise = bookmark+                    runAdoptionAgencyInnerLoop i' bookmark' formattingElement lastNode furthestBlock ns+                else do+                    -- 7.+                    newElement <- createElement $ snd innerNode+                    let newTag = unpackNodeData <$> newElement+                    N.S.modify $ \state' -> state'+                        { openElements = replaceNode newElement innerNode $ openElements state'+                        , activeFormattingElements = map (replaceNode newTag innerTag) formatting+                        }+                    -- 8.+                    let bookmark'+                            | lastNode == furthestBlock =+                                maybe bookmark succ . L.elemIndex newTag $ concat formatting+                            | otherwise = bookmark+                    -- 10. / 11.+                    runAdoptionAgencyInnerLoop i' bookmark' formattingElement innerNode furthestBlock ns+            -- 6.+            Nothing -> do+                N.S.modify $ \state' -> state'+                    --BUG: Not reflected in the patch list.+                    { openElements = L.delete innerNode $ openElements state'+                    }+                runAdoptionAgencyInnerLoop i' bookmark formattingElement lastNode furthestBlock ns+  where i' = if i == 0 then 0 else i - 1+        replaceNode newElement innerElement es = ds ++ newElement : drop 1 as+          where (ds, as) = break (== innerElement) es+        innerTag = unpackNodeData <$> innerNode++-- | Retrieve the data from the finalized form of the collection, and repack it+-- in the form expected by some of the parser combinators used by the+-- @runAdoptionAgency@ algorithm.+unpackNodeData :: ElementParams -> TagParams+unpackNodeData d = emptyTagParams+    { tagName = case elementPrefix d of+        Just prefix -> prefix <> ":" <> elementName d+        Nothing -> elementName d+    , tagAttributes = M.fromList . map unpackAttribute . toAttrList $ elementAttributes d+    -- Don't have to worry about self-closing tags, as if they're taking part+    -- in the adoption agency algorithm, they must have been able to have+    -- children.+    }+  where unpackAttribute d' = case attrPrefix d' of+            Just prefix -> (prefix <> ":" <> attrName d', attrValue d')+            Nothing -> (attrName d', attrValue d')
+ src/Web/Mangrove/Parse/Tree/InBody.hs-boot view
@@ -0,0 +1,15 @@+{-|+Copyright:      (c) 2020 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com+-}+module Web.Mangrove.Parse.Tree.InBody+    ( treeInBody+    ) where+++import Web.Mangrove.Parse.Tree.Common+import Web.Mangrove.Parse.Tree.Patch+++treeInBody :: TreeBuilder TreeOutput
+ src/Web/Mangrove/Parse/Tree/InCaption.hs view
@@ -0,0 +1,104 @@+{-# LANGUAGE OverloadedStrings #-}++{-|+Description:    Token processing rules within a @\<caption\>@ section describing a table.++Copyright:      (c) 2020 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      stable+Portability:    portable+-}+module Web.Mangrove.Parse.Tree.InCaption+    ( treeInCaption+    ) where+++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tree.Common+import Web.Mangrove.Parse.Tree.InBody+import Web.Mangrove.Parse.Tree.Patch+import Web.Willow.Common.Parser+import Web.Willow.Common.Parser.Switch+++-- | __HTML:__+--      @[the "in caption" insertion mode]+--      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-incaption)@+-- +-- The parsing instructions corresponding to the 'InCaption' section of the+-- state machine.+treeInCaption :: TreeBuilder TreeOutput+treeInCaption = next >>= switch+    [ If (isEndTag ["caption"]) $ \t' -> do+        hasCaption <- hasInTableScope ["caption"]+        if hasCaption+            then do+                generate <- generateEndTags impliedEndTags+                current <- currentNode+                unexpected <- if maybe True (nodeIsElement "caption") current+                        then packTree t' []+                        else packTreeErrors [UnexpectedElementWithImpliedEndTag] t'+                close <- closeElement "caption"+                clearFormattingElements+                switchMode InTable+                return $ generate ++| unexpected |++ close+            else packTreeErrors [UnmatchedEndTag $ tokenElement t'] t'+    , If (isStartTag+        [ "caption"+        , "col"+        , "colgroup"+        , "tbody"+        , "td"+        , "tfoot"+        , "th"+        , "thead"+        , "tr"+        ]) $ \t' -> do+            hasCaption <- hasInTableScope ["caption"]+            if hasCaption+                then do+                    push t'+                    generate <- generateEndTags impliedEndTags+                    current <- currentNode+                    let unexpected = if maybe True (nodeIsElement "caption") current+                            then id+                            else consTreeError_ UnexpectedElementWithImpliedEndTag+                    close <- closeElement "caption"+                    clearFormattingElements+                    switchMode InTable+                    packTree_ $ generate ++ unexpected close+                else packTreeErrors [UnexpectedDescendantElement $ tokenElement t'] t'+    , If (isEndTag ["table"]) $ \t' -> do+        hasCaption <- hasInTableScope ["caption"]+        if hasCaption+            then do+                push t'+                generate <- generateEndTags impliedEndTags+                current <- currentNode+                let unexpected = if maybe True (nodeIsElement "caption") current+                        then id+                        else consTreeError_ UnexpectedElementWithImpliedEndTag+                close <- closeElement "caption"+                clearFormattingElements+                switchMode InTable+                packTree_ $ generate ++ unexpected close+            else packTreeErrors [UnexpectedDescendantElement $ tokenElement t'] t'+    , If (isEndTag+        [ "body"+        , "col"+        , "colgroup"+        , "html"+        , "tbody"+        , "td"+        , "tfoot"+        , "th"+        , "thead"+        , "tr"+        ]) $ \t' ->+            packTreeErrors [UnmatchedEndTag $ tokenElement t'] t'+    , Else $ \t' -> do+        push t'+        treeInBody+    ]
+ src/Web/Mangrove/Parse/Tree/InCell.hs view
@@ -0,0 +1,100 @@+{-# LANGUAGE OverloadedStrings #-}++{-|+Description:    Token processing rules for a @\<td\>@ data point in a table.++Copyright:      (c) 2020 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      stable+Portability:    portable+-}+module Web.Mangrove.Parse.Tree.InCell+    ( treeInCell+    ) where+++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tokenize.Common+import Web.Mangrove.Parse.Tree.Common+import Web.Mangrove.Parse.Tree.InBody+import Web.Mangrove.Parse.Tree.Patch+import Web.Willow.Common.Parser+import Web.Willow.Common.Parser.Switch+++-- | __HTML:__+--      @[the "in cell" insertion mode]+--      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-intd)@+-- +-- The parsing instructions corresponding to the 'InCell' section of the state+-- machine.+treeInCell :: TreeBuilder TreeOutput+treeInCell = next >>= switch+    [ If (isEndTag ["td", "th"]) $ \t' -> do+        let d = tokenTag t'+        hasMatch <- hasInTableScope [tagName d]+        if hasMatch+            then do+                generate <- generateEndTags impliedEndTags+                current <- currentNode+                let unexpected = if maybe True (nodeIsElement $ tagName d) current+                        then id+                        else consTreeError_ UnexpectedElementWithImpliedEndTag+                close <- closeElement $ tagName d+                clearFormattingElements+                switchMode InRow+                packTree t' $ generate ++ unexpected close+            else packTreeErrors [UnmatchedEndTag $ tokenElement t'] t'+    , If (isStartTag+        [ "caption"+        , "col"+        , "colgroup"+        , "tbody"+        , "td"+        , "tfoot"+        , "th"+        , "thead"+        , "tr"]) $ \t' -> do+            hasMatch <- hasInTableScope ["td", "th"]+            if hasMatch+                then do+                    push t'+                    close <- closeCell+                    packTree_ close+                else packTreeErrors [MalformedTableStructure $ tokenElement t'] t'+    , If (isEndTag+        [ "body"+        , "caption"+        , "col"+        , "colgroup"+        , "html"+        ]) $ \t' ->+            packTreeErrors [UnexpectedEndTag $ tokenElement t'] t'+    , If (isEndTag+        [ "table"+        , "tbody"+        , "tfoot"+        , "thead"+        , "tr"+        ]) $ \t' -> do+            hasMatch <- hasInTableScope [tagName $ tokenTag t']+            if hasMatch+                then do+                    push t'+                    close <- closeCell+                    packTree_ close+                else packTreeErrors [UnmatchedEndTag $ tokenElement t'] t'+    , Else $ \t' -> push t' *> treeInBody+    ]+  where closeCell = do+            generate <- generateEndTags impliedEndTags+            current <- currentNode+            let errF = if elem (fmap elementName current) [Just "td", Just "th"]+                    then consTreeError_ UnexpectedElementWithImpliedEndTag+                    else id+            close <- closeElements ["td", "th"]+            clearFormattingElements+            switchMode InRow+            return $ generate ++ errF close
+ src/Web/Mangrove/Parse/Tree/InColumnGroup.hs view
@@ -0,0 +1,69 @@+{-# LANGUAGE OverloadedStrings #-}++{-|+Description:    Token processing rules within a @\<colgroup\>@ markup section.++Copyright:      (c) 2020 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      stable+Portability:    portable+-}+module Web.Mangrove.Parse.Tree.InColumnGroup+    ( treeInColumnGroup+    ) where+++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tree.Common+import Web.Mangrove.Parse.Tree.InBody+import Web.Mangrove.Parse.Tree.InHead+import Web.Mangrove.Parse.Tree.Patch+import Web.Willow.Common.Parser+import Web.Willow.Common.Parser.Switch+++-- | __HTML:__+--      @[the "in column group" insertion mode]+--      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-incolgroup)@+-- +-- The parsing instructions corresponding to the 'InColumnGroup' section of the+-- state machine.+treeInColumnGroup :: TreeBuilder TreeOutput+treeInColumnGroup = next >>= switch+    [ If isWhitespace insertCharacter+    , If isComment insertComment+    , If isDoctype $ \t' ->+        packTreeErrors [UnexpectedDoctype $ tokenDocumentType t'] t'+    , If (isStartTag ["html"]) $ \t' -> do+        push t'+        treeInBody+    , If (isStartTag ["col"]) insertNullElement+    , If (isEndTag ["colgroup"]) $ \t' -> do+        current <- currentNode+        if maybe False (nodeIsElement "colgroup") current+            then do+                switchMode InTable+                closeCurrentNode t'+            else packTreeErrors [UnmatchedEndTag $ tokenElement t'] t'+    , If (isEndTag ["col"]) $ packTreeErrors [UnexpectedEndColInColumnGroup]+    , If (isStartTag ["template"]) $ \t' -> do+        push t'+        treeInHead+    , If (isEndTag ["template"]) $ \t' -> do+        push t'+        treeInHead+    , If isEOF $ \t' -> do+        push t'+        treeInBody+    , Else $ \t' -> do+        current <- currentNode+        if maybe False (nodeIsElement "colgroup") current+            then do+                push t'+                switchMode InTable+                close <- closeCurrentNode_+                packTree_ close+            else packTreeErrors [UnexpectedElementWithImpliedEndTag] t'+    ]
+ src/Web/Mangrove/Parse/Tree/InFrameset.hs view
@@ -0,0 +1,71 @@+{-# LANGUAGE OverloadedStrings #-}++{-|+Description:    Token processing rules within a @\<frameset\>@ section.++Copyright:      (c) 2020 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      stable+Portability:    portable+-}+module Web.Mangrove.Parse.Tree.InFrameset+    ( treeInFrameset+    ) where+++import qualified Control.Monad as N+import qualified Control.Monad.Trans.State as N.S++import qualified Data.Maybe as Y++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tree.Common+import Web.Mangrove.Parse.Tree.InBody+import Web.Mangrove.Parse.Tree.InHead+import Web.Mangrove.Parse.Tree.Patch+import Web.Willow.Common.Parser+import Web.Willow.Common.Parser.Switch+++-- | __HTML:__+--      @[the "in frameset" insertion mode]+--      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-inframeset)@+-- +-- The parsing instructions corresponding to the 'InFrameset' section of the+-- state machine.+treeInFrameset :: TreeBuilder TreeOutput+treeInFrameset = next >>= switch+    [ If isWhitespace insertCharacter+    , If isComment insertComment+    , If isDoctype $ \t' ->+        packTreeErrors [UnexpectedDoctype $ tokenDocumentType t'] t'+    , If (isStartTag ["html"]) $ \t' -> do+        push t'+        treeInBody+    , If (isStartTag ["frameset"]) insertElement+    , If (isEndTag ["frameset"]) $ \t' -> do+        state <- N.S.get+        current <- currentNode+        close <- if length (openElements state) <= 1 && maybe True (nodeIsElement "html") current+            then packTreeErrors [UnmatchedEndTag $ tokenElement t'] t'+            else closeCurrentNode t'+        current' <- currentNode+        let notFragment = Y.isNothing $ fragmentContext state+            currentFrameset = maybe False (nodeIsElement "frameset") current'+        N.when (notFragment && not currentFrameset) $ switchMode AfterFrameset+        return close+    , If (isStartTag ["frame"]) insertNullElement+    , If (isStartTag ["noframes"]) $ \t' -> do+        push t'+        treeInHead+    , If isEOF $ \t' -> do+        elements <- openElements <$> N.S.get+        let errF = if length elements > 1+                then consTreeError UnexpectedElementWithImpliedEndTag+                else id+        eof <- stopParsing t'+        return $ errF eof+    , Else $ packTreeErrors [UnexpectedNodeInFrameset]+    ]
+ src/Web/Mangrove/Parse/Tree/InHead.hs view
@@ -0,0 +1,151 @@+{-# LANGUAGE OverloadedStrings #-}++{-|+Description:    Token processing rules within the @\<head\>@ section.++Copyright:      (c) 2020 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      stable+Portability:    portable+-}+module Web.Mangrove.Parse.Tree.InHead+    ( treeInHead+    ) where+++import qualified Control.Applicative as A+import qualified Control.Monad as N+import qualified Control.Monad.Trans.State as N.S++import qualified Data.Bifunctor as F.B+import qualified Data.ByteString.Short as BS.SH+import qualified Data.HashMap.Strict as M+import qualified Data.Text as T+import qualified Data.Text.Encoding as T++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tokenize.Common+import Web.Mangrove.Parse.Tree.Common+import Web.Mangrove.Parse.Tree.InText+import Web.Mangrove.Parse.Tree.Patch+import Web.Willow.Common.Encoding+import Web.Willow.Common.Encoding.Character+import Web.Willow.Common.Encoding.Labels+import Web.Willow.Common.Encoding.Sniffer+import Web.Willow.Common.Parser+import Web.Willow.Common.Parser.Switch++import {-# SOURCE #-} Web.Mangrove.Parse.Tree.InBody++import Control.Applicative ( (<|>) )+++-- | __HTML:__+--      @[the "in head" insertion mode]+--      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-inhead)@+-- +-- The parsing instructions corresponding to the 'InHead' section of the state+-- machine.+treeInHead :: TreeBuilder TreeOutput+treeInHead = next >>= switch+    [ If isWhitespace insertCharacter+    , If isComment insertComment+    , If isDoctype $ \t' ->+        packTreeErrors [UnexpectedDoctype $ tokenDocumentType t'] t'+    , If (isStartTag ["html"]) $ \t' -> do+        push t'+        treeInBody+    , If (isStartTag ["base", "basefont", "bgsound", "link"]) insertNullElement+    , If (isStartTag ["meta"]) $ \t' -> do+        insert <- insertNullElement t'+        change' <- A.optional $ changeEncoding t'+        return $ case change' of+            Just change -> insert |++| change+            Nothing -> insert+    , If (isStartTag ["title"]) genericRCDataElement+    , If (isStartTag ["noscript"]) $ \t' -> do+        state <- N.S.get+        if scriptingEnabled state+            then genericRawTextElement t'+            else do+                switchMode InHeadNoscript+                insertElement t'+    , If (isStartTag ["noframes", "style"]) genericRawTextElement+    , If (isStartTag ["script"]) $ \t' -> do+        N.S.modify $ \state -> state+            { originalInsertionMode = Just $ insertionMode state+            }+        switchMode InText+        insert <- insertElement . mapTokenState t' $ \state -> state+            { currentState = ScriptDataState+            }+        return insert+    , If (isEndTag ["head"]) $ \t' -> do+        switchMode AfterHead+        packTree t' softCloseCurrentNode_+    , If (isEndTag ["body", "html", "br"]) anythingElse+    , If (isStartTag ["template"]) $ \t' -> do+        insert <- insertElement t'+        insertFormattingMarker+        setFramesetNotOk+        switchMode InTemplate+        pushTemplateMode InTemplate+        return insert+    , If (isEndTag ["template"]) $ \t' -> do+        generate <- generateEndTags thoroughlyImpliedEndTags+        current <- currentNode+        let errF = case nodeIsElement "template" <$> current of+                Just True -> id+                _ -> consTreeError_ UnexpectedElementWithImpliedEndTag+        clear <- closeElement "template"+        clearFormattingElements+        popTemplateMode+        resetInsertionMode+        packTree t' $ errF generate ++ clear+    , If (isStartTag ["head"]) $ packTreeErrors [NestedSingletonElement]+    , If isAnyEndTag $ \t' ->+        packTreeErrors [UnmatchedEndTag $ tokenElement t'] t'+    , Else anythingElse+    ]+  where anythingElse t' = do+            push t'+            switchMode AfterHead+            packTree_ softCloseCurrentNode_+        changeEncoding t' = do+            let d = tokenTag t'+                state = tokenState t' >>= decoderState . fst+            enc' <- maybe A.empty return $ do+                e <- changeEncodingCharset d <|> changeEncodingContentType d+                return $ case e of+                    Utf16be -> Utf8+                    Utf16le -> Utf8+                    UserDefined -> Windows1252+                    enc -> enc+            case state of+                    Just s -> case decoderEncoding s of+                        Utf16be -> putDecoderState t' $ setEncodingCertain Utf16be s+                        Utf16le -> putDecoderState t' $ setEncodingCertain Utf16be s+                        enc | enc == enc' -> putDecoderState t' $ setEncodingCertain enc' s+                        _ -> case decoderConfidence s of+                            Tentative _ rec | not $ encodingEquivalent rec enc' -> do+                                _ <- putDecoderState t' $ initialDecoderState enc'+                                restartParsing $ streamStart rec+                            _ -> putDecoderState t' $ initialDecoderState enc'+                    Nothing -> putDecoderState t' $ initialDecoderState enc'+        changeEncodingCharset d = do+            charset <- M.lookup "charset" $ tagAttributes d+            lookupEncoding charset+        changeEncodingContentType d = do+            httpEquiv <- M.lookup "http-equiv" $ tagAttributes d+            N.unless (T.map toAsciiLower httpEquiv == "content-type") A.empty+            content <- M.lookup "content" $ tagAttributes d+            extractEncoding $ T.encodeUtf8 content+        encodingEquivalent rec enc = uncurry (==) .+            F.B.bimap (map $ parseChar enc) (map T.singleton) . unzip . M.toList $ parsedChars rec+        parseChar enc = fst . decode' (initialDecoderState enc) . BS.SH.fromShort+        putDecoderState t' decState =+            flip packTree [] . mapTokenState' t' . fmap . F.B.first $ \state -> state+                { decoderState_ = Right $ Just decState+                }
+ src/Web/Mangrove/Parse/Tree/InHeadNoscript.hs view
@@ -0,0 +1,67 @@+{-|+Description:    Token processing rules within a @\<noscript\>@ section of the @\<head\>@.++Copyright:      (c) 2020 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      stable+Portability:    portable+-}+module Web.Mangrove.Parse.Tree.InHeadNoscript+    ( treeInHeadNoscript+    ) where+++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tree.Common+import Web.Mangrove.Parse.Tree.InBody+import Web.Mangrove.Parse.Tree.InHead+import Web.Mangrove.Parse.Tree.Patch+import Web.Willow.Common.Parser+import Web.Willow.Common.Parser.Switch+++-- | __HTML:__+--      @[the "in head noscript" insertion mode]+--      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-inheadnoscript)@+-- +-- The parsing instructions corresponding to the 'InHeadNoscript' section of+-- the state machine.+treeInHeadNoscript :: TreeBuilder TreeOutput+treeInHeadNoscript = next >>= switch+    [ If isDoctype $ \t' ->+        packTreeErrors [UnexpectedDoctype $ tokenDocumentType t'] t'+    , If (isStartTag ["html"]) $ \t' -> do+        push t'+        treeInBody+    , If (isEndTag ["noscript"]) $ \t' -> do+        switchMode InHead+        closeCurrentNode t'+    , If isWhitespace $ \t' -> do+        push t'+        treeInHead+    , If isComment $ \t' -> do+        push t'+        treeInHead+    , If (isStartTag+        [ "basefont"+        , "bgsound"+        , "link"+        , "meta"+        , "noframes"+        , "style"+        ]) $ \t' -> do+            push t'+            treeInHead+    , If (isEndTag ["br"]) anythingElse+    , If (isStartTag ["head", "noscript"]) $ packTreeErrors [NestedSingletonElement]+    , If isAnyEndTag $ \t' ->+        packTreeErrors [UnmatchedEndTag $ tokenElement t'] t'+    , Else anythingElse+    ]+  where anythingElse t' = do+            push t'+            switchMode InHead+            close <- closeCurrentNode_+            packTree_ $ consTreeError_ UnexpectedNodeInHeadNoscript close
+ src/Web/Mangrove/Parse/Tree/InRow.hs view
@@ -0,0 +1,101 @@+{-# LANGUAGE OverloadedStrings #-}++{-|+Description:    Token processing rules for a @\<tr\>@ data segment in a table.++Copyright:      (c) 2020 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      stable+Portability:    portable+-}+module Web.Mangrove.Parse.Tree.InRow+    ( treeInRow+    ) where+++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tokenize.Common+import Web.Mangrove.Parse.Tree.Common+import Web.Mangrove.Parse.Tree.InTable+import Web.Mangrove.Parse.Tree.Patch+import Web.Willow.Common.Parser+import Web.Willow.Common.Parser.Switch+++-- | __HTML:__+--      @[the "in row" insertion mode]+--      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-intr)@+-- +-- The parsing instructions corresponding to the 'InRow' section of the state+-- machine.+treeInRow :: TreeBuilder TreeOutput+treeInRow = next >>= switch+    [ If (isStartTag ["th", "td"]) $ \t' -> do+        switchMode InCell+        clear <- clearToContext tableRowContext+        insertFormattingMarker+        insert <- insertElement t'+        return $ clear ++| insert+    , If (isEndTag ["tr"]) $ \t' -> do+        hasTr <- hasInTableScope ["tr"]+        if hasTr+            then do+                switchMode InTableBody+                close <- closeTableRow+                packTree t' close+            else packTreeErrors [UnmatchedEndTag $ tokenElement t'] t'+    , If (isStartTag+        [ "caption"+        , "col"+        , "colgroup"+        , "tbody"+        , "tfoot"+        , "thead"+        , "tr"+        ]) $ \t' -> do+            hasTr <- hasInTableScope ["tr"]+            if hasTr+                then do+                    push t'+                    switchMode InTableBody+                    close <- closeTableRow+                    packTree_ close+                else packTreeErrors [MalformedTableStructure $ tokenElement t'] t'+    , If (isEndTag ["table"]) $ \t' -> do+        hasTr <- hasInTableScope ["tr"]+        if hasTr+            then do+                push t'+                switchMode InTableBody+                close <- closeTableRow+                packTree_ close+            else packTreeErrors [MalformedTableStructure $ tokenElement t'] t'+    , If (isEndTag ["tbody", "tfoot", "thead"]) $ \t' -> do+        hasMatch <- hasInTableScope [tagName $ tokenTag t']+        if hasMatch+            then do+                push t'+                switchMode InTableBody+                close <- closeTableRow+                packTree_ close+            else packTreeErrors [UnmatchedEndTag $ tokenElement t'] t'+    , If (isEndTag+        [ "body"+        , "caption"+        , "col"+        , "colgroup"+        , "html"+        , "td"+        , "th"+        ]) $ \t' ->+            packTreeErrors [UnexpectedEndTag $ tokenElement t'] t'+    , Else $ \t' -> do+        push t'+        treeInTable+    ]+  where closeTableRow = do+            clear <- clearToContext tableRowContext+            close <- closeCurrentNode_+            return $ clear ++ close
+ src/Web/Mangrove/Parse/Tree/InSelect.hs view
@@ -0,0 +1,119 @@+{-# LANGUAGE OverloadedStrings #-}++{-|+Description:    Token processing rules within a @\<select\>@ set of choices for user input.++Copyright:      (c) 2020 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      stable+Portability:    portable+-}+module Web.Mangrove.Parse.Tree.InSelect+    ( treeInSelect+    ) where+++import qualified Control.Monad.Trans.State as N.S++import qualified Data.Maybe as Y++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tree.Common+import Web.Mangrove.Parse.Tree.InBody+import Web.Mangrove.Parse.Tree.InHead+import Web.Mangrove.Parse.Tree.Patch+import Web.Willow.Common.Parser+import Web.Willow.Common.Parser.Switch+++-- | __HTML:__+--      @[the "in select" insertion mode]+--      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-inselect)@+-- +-- The parsing instructions corresponding to the 'InSelect' section of the+-- state machine.+treeInSelect :: TreeBuilder TreeOutput+treeInSelect = next >>= switch+    [ If isNull $ packTreeErrors [UnexpectedNullCharacter]+    , If isCharacter insertCharacter+    , If isComment insertComment+    , If isDoctype $ \t' ->+        packTreeErrors [UnexpectedDoctype $ tokenDocumentType t'] t'+    , If (isStartTag ["html"]) $ \t' -> do+        push t'+        treeInBody+    , If (isStartTag ["option"]) $ \t' -> do+        current <- currentNode+        close <- if maybe False (nodeIsElement "option") current+            then closeCurrentNode_+            else return []+        insert <- insertElement t'+        return $ close ++| insert+    , If (isStartTag ["optgroup"]) $ \t' -> do+        current <- currentNode+        option <- if maybe False (nodeIsElement "option") current+            then closeCurrentNode_+            else return []+        current' <- currentNode+        optgroup <- if maybe False (nodeIsElement "optgroup") current'+            then closeCurrentNode_+            else return []+        insert <- insertElement t'+        return $ option ++ optgroup ++| insert+    , If (isEndTag ["optgroup"]) $ \t' -> do+        current <- currentNode+        second <- fmap snd . Y.listToMaybe . drop 1 . openElements <$> N.S.get+        let firstIsOption = maybe False (nodeIsElement "option") current+            secondIsOptgroup = maybe False (nodeIsElement "optgroup") second+        option <- if firstIsOption && secondIsOptgroup+            then closeCurrentNode_+            else return []+        current' <- currentNode+        optgroup <- if maybe False (nodeIsElement "optgroup") current'+            then closeCurrentNode t'+            else packTreeErrors [UnmatchedEndTag $ tokenElement t'] t'+        return $ option ++| optgroup+    , If (isEndTag ["option"]) $ \t' -> do+        current <- currentNode+        if maybe False (nodeIsElement "option") current+            then closeCurrentNode t'+            else packTreeErrors [UnmatchedEndTag $ tokenElement t'] t'+    , If (isEndTag ["select"]) $ \t' -> do+        hasSelect <- hasInSelectScope ["select"]+        if hasSelect+            then do+                close <- closeElement "select"+                resetInsertionMode+                packTree t' close+            else packTreeErrors [UnmatchedEndTag $ tokenElement t'] t'+    , If (isStartTag ["select"]) $ \t' -> do+        hasSelect <- hasInSelectScope ["select"]+        consTreeError NestedNonRecursiveElement <$> if hasSelect+            then do+                close <- closeElement "select"+                resetInsertionMode+                packTree t' close+            else packTreeErrors [] t'+    , If (isStartTag ["input", "keygen", "textarea"]) $ \t' -> do+        hasSelect <- hasInSelectScope ["select"]+        consTreeError OverlappingInputElements <$> if hasSelect+            then do+                push t'+                close <- closeElement "select"+                resetInsertionMode+                packTree_ close+            else packTreeErrors [] t'+    , If (isStartTag ["script", "template"]) $ \t' -> do+        push t'+        treeInHead+    , If (isEndTag ["template"]) $ \t' -> do+        push t'+        treeInHead+    , If isEOF $ \t' -> do+        push t'+        treeInBody+    , Else $ \t' ->+        packTreeErrors [UnexpectedDescendantElement $ tokenElement t'] t'+    ]
+ src/Web/Mangrove/Parse/Tree/InSelectInTable.hs view
@@ -0,0 +1,68 @@+{-# LANGUAGE OverloadedStrings #-}++{-|+Description:    Token processing rules within a @\<select\>@ input, which is contained within a table.++Copyright:      (c) 2020 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      stable+Portability:    portable+-}+module Web.Mangrove.Parse.Tree.InSelectInTable+    ( treeInSelectInTable+    ) where+++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tokenize.Common+import Web.Mangrove.Parse.Tree.Common+import Web.Mangrove.Parse.Tree.Patch+import Web.Mangrove.Parse.Tree.InSelect+import Web.Willow.Common.Parser+import Web.Willow.Common.Parser.Switch+++-- | __HTML:__+--      @[the "in select in table" insertion mode]+--      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-inselectintable)@+-- +-- The parsing instructions corresponding to the 'InSelectInTable' section of the+-- state machine.+treeInSelectInTable :: TreeBuilder TreeOutput+treeInSelectInTable = next >>= switch+    [ If (isStartTag+        [ "caption"+        , "table"+        , "tbody"+        , "tfoot"+        , "thead"+        , "tr"+        , "td"+        , "th"+        ]) $ \t' -> do+            push t'+            close <- closeElement "select"+            resetInsertionMode+            packTree_ . consTreeError_ (MalformedTableStructure $ tokenElement t') $ close+    , If (isEndTag+        [ "caption"+        , "table"+        , "tbody"+        , "tfoot"+        , "thead"+        , "tr"+        , "td"+        , "th"+        ]) $ \t' -> do+            hasMatch <- hasInTableScope [tagName $ tokenTag t']+            if hasMatch+                then do+                    push t'+                    close <- closeElement "select"+                    resetInsertionMode+                    packTree_ close+                else packTreeErrors [UnexpectedElementWithImpliedEndTag] t'+    , Else $ \t' -> push t' *> treeInSelect+    ]
+ src/Web/Mangrove/Parse/Tree/InTable.hs view
@@ -0,0 +1,167 @@+{-# LANGUAGE OverloadedStrings #-}++{-|+Description:    Token processing rules within a @\<table\>@ markup section.++Copyright:      (c) 2020 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      stable+Portability:    portable+-}+module Web.Mangrove.Parse.Tree.InTable+    ( treeInTable+    , anythingElse+    ) where+++import qualified Control.Monad.Trans.State as N.S++import qualified Data.HashMap.Strict as M+import qualified Data.List as L+import qualified Data.Maybe as Y++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tree.Common+import Web.Mangrove.Parse.Tree.InBody+import Web.Mangrove.Parse.Tree.InHead+import Web.Mangrove.Parse.Tree.Patch+import Web.Willow.Common.Parser+import Web.Willow.Common.Parser.Switch+++-- | __HTML:__+--      @[the "in table" insertion mode]+--      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-intable)@+-- +-- The parsing instructions corresponding to the 'InTable' section of the state+-- machine.+treeInTable :: TreeBuilder TreeOutput+treeInTable = next >>= switch+    [ If isCharacter $ \t' -> do+        current <- currentNode+        let toTableText = do+                N.S.modify $ \state -> state+                    { insertionMode = InTableText+                    , originalInsertionMode = Just $ insertionMode state+                    }+                push t'+                switchMode InTableText+                packTree_ []+        case fmap elementName current of+            Just "table" -> toTableText+            Just "tbody" -> toTableText+            Just "tfoot" -> toTableText+            Just "thead" -> toTableText+            Just "tr" -> toTableText+            _ -> anythingElse t'+    , If isComment insertComment+    , If isDoctype $ \t' ->+        packTreeErrors [UnexpectedDoctype $ tokenDocumentType t'] t'+    , If (isStartTag ["caption"]) $ \t' -> do+        switchMode InCaption+        clear <- clearToContext tableContext+        insertFormattingMarker+        insert <- insertElement t'+        return $ clear ++| insert+    , If (isStartTag ["colgroup"]) $ \t' -> do+        switchMode InColumnGroup+        clear <- clearToContext tableContext+        insert <- insertElement t'+        return $ clear ++| insert+    , If (isStartTag ["col"]) $ \t' -> do+        push t'+        switchMode InColumnGroup+        clear <- clearToContext tableContext+        insert <- insertElement_ $ emptyTagParams+            { tagName = "colgroup"+            }+        packTree_ $ clear ++ insert+    , If (isStartTag ["tbody", "tfoot", "thead"]) $ \t' -> do+        switchMode InTableBody+        clear <- clearToContext tableContext+        insert <- insertElement t'+        return $ clear ++| insert+    , If (isStartTag ["td", "th", "tr"]) $ \t' -> do+        push t'+        switchMode InTableBody+        clear <- clearToContext tableContext+        insert <- insertElement_ $ emptyTagParams+            { tagName = "tbody"+            }+        packTree_ $ clear ++ insert+    , If (isStartTag ["table"]) $ \t' -> do+        hasTable <- hasInTableScope ["table"]+        if hasTable+            then do+                push t'+                close <- closeTable+                packTree_ close+            else packTreeErrors [NestedNonRecursiveElement] t'+    , If (isEndTag ["table"]) $ \t' -> do+        hasTable <- hasInTableScope ["table"]+        if hasTable+            then fmap (|++) (packTreeErrors [] t') <*> closeTable+            else packTreeErrors [UnmatchedEndTag $ tokenElement t'] t'+    , If (isEndTag+        [ "body"+        , "caption"+        , "col"+        , "colgroup"+        , "html"+        , "tbody"+        , "td"+        , "tfoot"+        , "th"+        , "thead"+        , "tr"+        ]) $ \t' ->+            packTreeErrors [UnexpectedDescendantElement $ tokenElement t'] t'+    , If (isStartTag ["style", "script", "template"]) $ \t' -> do+        push t'+        treeInHead+    , If (isEndTag ["template"]) $ \t' -> do+        push t'+        treeInHead+    , If (isStartTag ["input"]) $ \t' ->+        case L.find (== "type") . map fst . M.toList . tagAttributes $ tokenTag t' of+            Just "hidden" -> consTreeError UnexpectedElementInTableStructure <$> insertNullElement t'+            _ -> anythingElse t'+    , If (isStartTag ["form"]) $ \t' -> do+        state <- N.S.get+        let hasTemplate = any (nodeIsElement "template" . snd) $ openElements state+        if hasTemplate || Y.isJust (formElementPointer state)+            then packTreeErrors [UnexpectedElementInTableStructure] t'+            else do+                N.S.modify $ \state' -> state'+                    { formElementPointer = Just $ elementIndex state'+                    }+                insertNullElement t'+    , If isEOF $ \t' -> do+        push t'+        treeInBody+    , Else anythingElse+    ]+  where closeTable = closeElement "table" <* resetInsertionMode+++-- | __HTML:__+--      the "anything else" entry in @[the "in table" insertion mode]+--      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-intable)@+-- +-- Delegate a token to to the 'InTable' section of the state machine, but skip+-- the token-dependent behaviour and instead simply treat it according to the+-- fallback case.+anythingElse :: TreeInput -> TreeBuilder TreeOutput+anythingElse t' = do+    _ <- error "Foster parenting not yet implemented"+    N.S.modify $ \state -> state+        { fosteringEnabled = True+        }+    push t'+    out <- treeInBody+    N.S.modify $ \state -> state+        { fosteringEnabled = False+        }+    return $ consTreeError UnexpectedNodeInTableStructure out
+ src/Web/Mangrove/Parse/Tree/InTableBody.hs view
@@ -0,0 +1,99 @@+{-# LANGUAGE OverloadedStrings #-}++{-|+Description:    Token processing rules for the primary content of a @\<table\>@.++Copyright:      (c) 2020 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      stable+Portability:    portable+-}+module Web.Mangrove.Parse.Tree.InTableBody+    ( treeInTableBody+    ) where+++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tokenize.Common+import Web.Mangrove.Parse.Tree.Common+import Web.Mangrove.Parse.Tree.InTable+import Web.Mangrove.Parse.Tree.Patch+import Web.Willow.Common.Parser+import Web.Willow.Common.Parser.Switch+++-- | __HTML:__+--      @[the "in table body" insertion mode]+--      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-intbody)@+-- +-- The parsing instructions corresponding to the 'InTableBody' section of the+-- state machine.+treeInTableBody :: TreeBuilder TreeOutput+treeInTableBody = next >>= switch+    [ If (isStartTag ["tr"]) $ \t' -> do+        switchMode InRow+        clear <- clearToContext tableBodyContext+        insert <- insertElement t'+        return $ clear ++| insert+    , If (isStartTag ["th", "td"]) $ \t' -> do+        push t'+        switchMode InRow+        clear <- clearToContext tableBodyContext+        insert <- fmap (consTreeError_ UnexpectedTableCellOutsideOfRow) . insertElement_ $ emptyTagParams+            { tagName = "tr"+            }+        packTree_ $ clear ++ insert+    , If (isEndTag ["tbody", "tfoot", "thead"]) $ \t' -> do+        hasTable <- hasInTableScope [tagName $ tokenTag t']+        if hasTable+            then do+                switchMode InTable+                close <- closeTableBody+                packTree t' close+            else packTreeErrors [UnmatchedEndTag $ tokenElement t'] t'+    , If (isStartTag+        [ "caption"+        , "col"+        , "colgroup"+        , "tbody"+        , "tfoot"+        , "thead"+        ]) $ \t' -> do+            hasMatch <- hasInTableScope ["tbody", "thead", "tfoot"]+            if hasMatch+                then do+                    push t'+                    switchMode InTable+                    close <- closeTableBody+                    packTree_ close+                else packTreeErrors [UnexpectedDescendantElement $ tokenElement t'] t'+    , If (isEndTag ["table"]) $ \t' -> do+        hasMatch <- hasInTableScope ["tbody", "thead", "tfoot"]+        if hasMatch+            then do+                push t'+                switchMode InTable+                close <- closeTableBody+                packTree_ close+            else packTreeErrors [UnexpectedDescendantElement $ tokenElement t'] t'+    , If (isEndTag+        [ "body"+        , "caption"+        , "col"+        , "colgroup"+        , "html"+        , "td"+        , "th"+        , "tr"+        ]) $ \t' ->+            packTreeErrors [UnexpectedEndTag $ tokenElement t'] t'+    , Else $ \t' -> do+        push t'+        treeInTable+    ]+  where closeTableBody = do+            clear <- clearToContext tableBodyContext+            close <- closeCurrentNode_+            return $ clear ++ close
+ src/Web/Mangrove/Parse/Tree/InTableText.hs view
@@ -0,0 +1,75 @@+{-|+Description:    Token processing rules for content misnested within a @\<table\>@.+Copyright:      (c) 2020-2021 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      stable+Portability:    portable+-}+module Web.Mangrove.Parse.Tree.InTableText+    ( treeInTableText+    ) where+++import qualified Data.Foldable as D+import qualified Data.List as L+import qualified Data.Tuple.HT as U.HT++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tokenize.Common+import Web.Mangrove.Parse.Tree.Common+import Web.Mangrove.Parse.Tree.InTable+import Web.Mangrove.Parse.Tree.Patch+import Web.Willow.Common.Encoding.Character+import Web.Willow.Common.Parser+import Web.Willow.Common.Parser.Switch++import {-# SOURCE #-} Web.Mangrove.Parse.Tree.Dispatcher+++-- | __HTML:__+--      @[the "in table text" insertion mode]+--      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-intabletext)@+-- +-- The parsing instructions corresponding to the 'InTableText' section of the+-- state machine.+treeInTableText :: TreeBuilder TreeOutput+treeInTableText = do+    resetMode+    pending <- treeInTableText'+    let processF = if not $ all (maybe True isAsciiWhitespace . U.HT.snd3) pending+            then anythingElse+            else insertCharacter+    ps <- D.foldrM (repack processF) [] pending+    start <- packTree_ []+    return $ L.foldr foldOut start ps+  where repack _ ([], Nothing, _) [] = return []+        repack _ (errs, Nothing, _) [] = pure . consTreeErrors errs <$> dispatchHtml+        repack _ (errs, Nothing, _) (t:ts) = return $ consTreeErrors errs t : ts+        repack f (errs, Just c, state) ts = (: ts) <$> f (dummyStateToken errs (Character c) state)+        foldOut ps ps' = ps' { treePatches = treePatches ps ++ treePatches ps' }+        consTreeErrors = flip $ foldr consTreeError++-- | __HTML:__+--      @[the "in table" insertion mode]+--      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-intable)@+-- +-- The parsing instructions corresponding to the 'InTableText' section of the+-- state machine.  Specifically, this consumes all following 'Character' tokens+-- to construct the list of [pending table character tokens]+-- (https://html.spec.whatwg.org/multipage/parsing.html#concept-pending-table-char-tokens),+-- while the surrounding 'treeInTableText' is what actually sends them to the+-- 'InTable' state for insertion.+treeInTableText' :: TreeBuilder [([ParseError], Maybe Char, TokenizerOutputState)]+treeInTableText' = next >>= switch+    [ If isNull $ \t' -> do+        cs <- treeInTableText'+        return $ (UnexpectedNullCharacter : tokenErrs t', Nothing, tokenState t') : cs+    , If isCharacter $ \t' -> do+        cs <- treeInTableText'+        return $ (tokenErrs t', tokenCharacter t', tokenState t') : cs+    , Else $ \t' -> do+        push t'+        return []+    ]
+ src/Web/Mangrove/Parse/Tree/InTemplate.hs view
@@ -0,0 +1,111 @@+{-# LANGUAGE OverloadedStrings #-}++{-|+Description:    Token processing rules within a @\<template\>@ section providing a fragment for script processing.++Copyright:      (c) 2020 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      stable+Portability:    portable+-}+module Web.Mangrove.Parse.Tree.InTemplate+    ( treeInTemplate+    ) where+++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tree.Common+import Web.Mangrove.Parse.Tree.InHead+import Web.Mangrove.Parse.Tree.Patch+import Web.Willow.Common.Parser+import Web.Willow.Common.Parser.Switch++import {-# SOURCE #-} Web.Mangrove.Parse.Tree.InBody+++-- | __HTML:__+--      @[the "in template" insertion mode]+--      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-intemplate)@+-- +-- The parsing instructions corresponding to the 'InTemplate' section of the+-- state machine.+treeInTemplate :: TreeBuilder TreeOutput+treeInTemplate = next >>= switch+    [ If isCharacter $ \t' -> do+        push t'+        treeInBody+    , If isComment $ \t' -> do+        push t'+        treeInBody+    , If isDoctype $ \t' -> do+        push t'+        treeInBody+    , If (isStartTag+        [ "base"+        , "basefont"+        , "bgsound"+        , "link"+        , "meta"+        , "noframes"+        , "script"+        , "style"+        , "template"+        , "title"+        ]) $ \t' -> do+            push t'+            treeInHead+    , If (isEndTag ["template"]) $ \t' -> do+        push t'+        treeInHead+    , If (isStartTag+        [ "caption"+        , "colgroup"+        , "tbody"+        , "tfoot"+        , "thead"+        ]) $ \t' -> do+            push t'+            _ <- popTemplateMode+            pushTemplateMode InTable+            switchMode InTable+            packTree_ []+    , If (isStartTag ["col"]) $ \t' -> do+        push t'+        popTemplateMode+        pushTemplateMode InColumnGroup+        switchMode InColumnGroup+        packTree_ []+    , If (isStartTag ["tr"]) $ \t' -> do+        push t'+        popTemplateMode+        pushTemplateMode InTableBody+        switchMode InTableBody+        packTree_ []+    , If (isStartTag ["td", "th"]) $ \t' -> do+        push t'+        popTemplateMode+        pushTemplateMode InRow+        switchMode InRow+        packTree_ []+    , If isAnyStartTag $ \t' -> do+        push t'+        popTemplateMode+        pushTemplateMode InBody+        switchMode InBody+        packTree_ []+    , If isAnyEndTag $ \t' ->+        packTreeErrors [UnmatchedEndTag $ tokenElement t'] t'+    , If isEOF $ \t' -> do+        hasTemplate <- hasOpenElement ["template"]+        if hasTemplate+            then do+                push t'+                close <- closeElement "template"+                clearFormattingElements+                popTemplateMode+                resetInsertionMode+                packTree_ close+            else stopParsing t'+    ]
+ src/Web/Mangrove/Parse/Tree/InText.hs view
@@ -0,0 +1,88 @@+{-|+Description:    Token processing rules for spans of raw character strings.++Copyright:      (c) 2020-2021 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      stable+Portability:    portable+-}+module Web.Mangrove.Parse.Tree.InText+    ( treeInText+    , genericRawTextElement+    , genericRCDataElement+    ) where+++import qualified Control.Monad.Trans.State as N.S++import qualified Data.Maybe as Y++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tokenize.Common+import Web.Mangrove.Parse.Tree.Common+import Web.Mangrove.Parse.Tree.Patch+import Web.Willow.Common.Parser+import Web.Willow.Common.Parser.Switch+++-- | __HTML:__+--      @[the "text" insertion mode]+--      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-incdata)@+-- +-- The parsing instructions corresponding to the 'InText' section of the state+-- machine.+treeInText :: TreeBuilder TreeOutput+treeInText = next >>= switch+    [ If isCharacter insertCharacter+    , If isEOF $ \t' -> do+        push t'+        N.S.modify $ \state -> state+            { insertionMode = Y.fromMaybe InText $ originalInsertionMode state+            , originalInsertionMode = Nothing+            }+        close <- consTreeError_ EOFInText <$> closeCurrentNode_+        packTree_ close+    , If (isEndTag ["script"]) $ \t' -> do+        resetMode+        close <- closeCurrentNode_+        packTree t' close+    , If isAnyEndTag $ \t' -> do+        resetMode+        closeCurrentNode t'+    ]+++-- | __HTML:__+--      @[generic raw text element parsing algorithm]+--      (https://html.spec.whatwg.org/multipage/parsing.html#generic-raw-text-element-parsing-algorithm)@+-- +-- Insert an element containing unescaped plain text.+genericRawTextElement :: TreeInput -> TreeBuilder TreeOutput+genericRawTextElement = genericParsingAlgorithm RawTextState++-- | __HTML:__+--      @[generic RCDATA element parsing algorithm]+--      (https://html.spec.whatwg.org/multipage/parsing.html#generic-rcdata-element-parsing-algorithm)@+-- +-- Insert an element containing plain text, potentially with character+-- references.+genericRCDataElement :: TreeInput -> TreeBuilder TreeOutput+genericRCDataElement = genericParsingAlgorithm RCDataState++-- | __HTML:__+--      @[parsing elements that contain only text]+--      (https://html.spec.whatwg.org/multipage/parsing.html#parsing-elements-that-contain-only-text)@+-- +-- The actual algorithm described for 'genericRawTextElement' and+-- 'genericRCDataElement', with all variables exported.+genericParsingAlgorithm :: CurrentTokenizerState -> TreeInput -> TreeBuilder TreeOutput+genericParsingAlgorithm mode t' = do+    N.S.modify $ \state -> state+        { insertionMode = InText+        , originalInsertionMode = Just $ insertionMode state+        }+    insertElement . mapTokenState t' $ \state -> state+        { currentState = mode+        }
+ src/Web/Mangrove/Parse/Tree/Initial.hs view
@@ -0,0 +1,136 @@+{-# LANGUAGE OverloadedStrings #-}++{-|+Description:    Token processing rules before the doctype declaration.++Copyright:      (c) 2020 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      stable+Portability:    portable+-}+module Web.Mangrove.Parse.Tree.Initial+    ( treeInitial+    ) where+++import qualified Data.Maybe as Y+import qualified Data.Text as T++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tokenize.Common+import Web.Mangrove.Parse.Tree.Common+import Web.Mangrove.Parse.Tree.Patch+import Web.Willow.Common.Encoding.Character+import Web.Willow.Common.Parser+import Web.Willow.Common.Parser.Switch+++-- | __HTML:__+--      @[the "initial" insertion mode]+--      (https://html.spec.whatwg.org/multipage/parsing.html#the-initial-insertion-mode)@+-- +-- The parsing instructions corresponding to the 'Initial' section of the+-- state machine.+treeInitial :: TreeBuilder TreeOutput+treeInitial = next >>= switch+    [ If isWhitespace $ packTreeErrors []+    , If isComment $ insertComment' InDocument+    , If isDoctype $ \t' -> do+        switchMode BeforeHtml+        doctype <- insertDoctype t'+        srcdoc <- inIFrameSrcDoc+        quirks <- setDocumentQuirks . requiredQuirks srcdoc $ tokenDoctype t'+        return $ doctype |++ quirks+    , Else $ \t' -> do+        push t'+        switchMode BeforeHtml+        srcdoc <- inIFrameSrcDoc+        ps <- if srcdoc+            then return []+            else do+                quirks <- setDocumentQuirks FullQuirks+                return $ consTreeError_ MissingDoctype quirks+        packTree_ ps+    ]++-- | Compare the addresses in the document type declaration to known values+-- which should trigger the backwards-compatible rendering mode.+requiredQuirks :: Bool -> DoctypeParams -> QuirksMode+requiredQuirks True _ = FullQuirks+requiredQuirks False d+    | doctypeQuirks d = FullQuirks+    | Y.isJust name && name /= Just "html" = FullQuirks+    | public == "-//W3O//DTD W3 HTML STRICT 3.0//EN//" = FullQuirks+    | public == "-/W3C/DTD HTML 4.0 TRANSITIONAL/EN" = FullQuirks+    | public == "HTML" = FullQuirks+    | system == "HTTP://WWW.IBM.COM/DATA/DTD/V11/IBMXHTML1-TRANSITIONAL.DTD" = FullQuirks+    | startsWith "+//SILMARIL//DTD HTML PRO V0R11 19970101//" public = FullQuirks+    | startsWith "-//AS//DTD HTML 3.0 ASWEDIT + EXTENSIONS//" public = FullQuirks+    | startsWith "//ADVASOFT LTD//DTD HTML 3.0 ASWEDIT + EXTENSIONS//" public = FullQuirks+    | startsWith "-//IETF//DTD HTML 2.0 LEVEL 1//" public = FullQuirks+    | startsWith "-//IETF//DTD HTML 2.0 LEVEL 2//" public = FullQuirks+    | startsWith "-//IETF//DTD HTML 2.0 STRICT LEVEL 1//" public = FullQuirks+    | startsWith "-//IETF//DTD HTML 2.0 STRICT LEVEL 2//" public = FullQuirks+    | startsWith "-//IETF//DTD HTML 2.0 STRICT//" public = FullQuirks+    | startsWith "-//IETF//DTD HTML 2.0//" public = FullQuirks+    | startsWith "-//IETF//DTD HTML 2.1E//" public = FullQuirks+    | startsWith "-//IETF//DTD HTML 3.0//" public = FullQuirks+    | startsWith "-//IETF//DTD HTML 3.2 FINAL//" public = FullQuirks+    | startsWith "-//IETF//DTD HTML 3.2//" public = FullQuirks+    | startsWith "-//IETF//DTD HTML 3//" public = FullQuirks+    | startsWith "-//IETF//DTD HTML LEVEL 0//" public = FullQuirks+    | startsWith "-//IETF//DTD HTML LEVEL 1//" public = FullQuirks+    | startsWith "-//IETF//DTD HTML LEVEL 2//" public = FullQuirks+    | startsWith "-//IETF//DTD HTML LEVEL 3//" public = FullQuirks+    | startsWith "-//IETF//DTD HTML STRICT LEVEL 0//" public = FullQuirks+    | startsWith "-//IETF//DTD HTML STRICT LEVEL 1//" public = FullQuirks+    | startsWith "-//IETF//DTD HTML STRICT LEVEL 2//" public = FullQuirks+    | startsWith "-//IETF//DTD HTML STRICT LEVEL 3//" public = FullQuirks+    | startsWith "-//IETF//DTD HTML STRICT//" public = FullQuirks+    | startsWith "-//IETF//DTD HTML//" public = FullQuirks+    | startsWith "-//METRIUS//DTD METRIUS PRESENTATIONAL//" public = FullQuirks+    | startsWith "-//MICROSOFT//DTD INTERNET EXPLORER 2.0 HTML STRICT//" public = FullQuirks+    | startsWith "-//MICROSOFT//DTD INTERNET EXPLORER 2.0 HTML//" public = FullQuirks+    | startsWith "-//MICROSOFT//DTD INTERNET EXPLORER 2.0 TABLES//" public = FullQuirks+    | startsWith "-//MICROSOFT//DTD INTERNET EXPLORER 3.0 HTML STRICT//" public = FullQuirks+    | startsWith "-//MICROSOFT//DTD INTERNET EXPLORER 3.0 HTML//" public = FullQuirks+    | startsWith "-//MICROSOFT//DTD INTERNET EXPLORER 3.0 TABLES//" public = FullQuirks+    | startsWith "-//NETSCAPE COMM. CORP.//DTD HTML//" public = FullQuirks+    | startsWith "-//NETSCAPE COMM. CORP.//DTD STRICT HTML//" public = FullQuirks+    | startsWith "-//O'REILLY AND ASSOCIATES//DTD HTML 2.0//" public = FullQuirks+    | startsWith "-//O'REILLY AND ASSOCIATES//DTD HTML EXTENDED 1.0//" public = FullQuirks+    | startsWith "-//O'REILLY AND ASSOCIATES//DTD HTML EXTENDED RELAXED 1.0//" public = FullQuirks+    | startsWith "-//SQ//DTD HTML 2.0 HOTMETAL + EXTENSIONS//" public = FullQuirks+    | startsWith "-//SOFTQUAD SOFTWARE//DTD HOTMETAL PRO 6.0::19990601::EXTENSIONS TO HTML 4.0//" public = FullQuirks+    | startsWith "-//SOFTQUAD//DTD HOTMETAL PRO 4.0::19971010::EXTENSIONS TO HTML 4.0//" public = FullQuirks+    | startsWith "-//SPYGLASS//DTD HTML 2.0 EXTENDED//" public = FullQuirks+    | startsWith "-//SUN MICROSYSTEMS CORP.//DTD HOTJAVA HTML//" public = FullQuirks+    | startsWith "-//SUN MICROSYSTEMS CORP.//DTD HOTJAVA STRICT HTML//" public = FullQuirks+    | startsWith "-//W3C//DTD HTML 3 1995-03-24//" public = FullQuirks+    | startsWith "-//W3C//DTD HTML 3.2 DRAFT//" public = FullQuirks+    | startsWith "-//W3C//DTD HTML 3.2 FINAL//" public = FullQuirks+    | startsWith "-//W3C//DTD HTML 3.2//" public = FullQuirks+    | startsWith "-//W3C//DTD HTML 3.2S DRAFT//" public = FullQuirks+    | startsWith "-//W3C//DTD HTML 4.0 FRAMESET//" public = FullQuirks+    | startsWith "-//W3C//DTD HTML 4.0 TRANSITIONAL//" public = FullQuirks+    | startsWith "-//W3C//DTD HTML EXPERIMENTAL 19960712//" public = FullQuirks+    | startsWith "-//W3C//DTD HTML EXPERIMENTAL 970421//" public = FullQuirks+    | startsWith "-//W3C//DTD W3 HTML//" public = FullQuirks+    | startsWith "-//W3O//DTD W3 HTML 3.0//" public = FullQuirks+    | startsWith "-//WEBTECHS//DTD MOZILLA HTML 2.0//" public = FullQuirks+    | startsWith "-//WEBTECHS//DTD MOZILLA HTML//" public = FullQuirks+    | startsWith "-//W3C//DTD HTML 4.01 FRAMESET//" public && Y.isNothing system' = FullQuirks+    | startsWith "-//W3C//DTD HTML 4.01 TRANSITIONAL//" public && Y.isNothing system' = FullQuirks+    | startsWith "-//W3C//DTD XHTML 1.0 FRAMESET//" public = LimitedQuirks+    | startsWith "-//W3C//DTD XHTML 1.0 TRANSITIONAL//" public = LimitedQuirks+    | startsWith "-//W3C//DTD HTML 4.01 FRAMESET//" public = LimitedQuirks+    | startsWith "-//W3C//DTD HTML 4.01 TRANSITIONAL//" public = LimitedQuirks+    | otherwise = NoQuirks+  where name = doctypeName d+        public = maybe T.empty (T.map toAsciiUpper) public'+        public' = doctypePublicId d+        system = maybe T.empty (T.map toAsciiUpper) system'+        system' = doctypeSystemId d+        startsWith = T.isPrefixOf
+ src/Web/Mangrove/Parse/Tree/Patch.hs view
@@ -0,0 +1,860 @@+{-# LANGUAGE OverloadedStrings #-}++{-|+Description:    A stateless reformulation of the components of the tree construction algorithm.++Copyright:      (c) 2020 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      provisional+Portability:    portable++This module provides the data structures used in the second half of this+implementation's split+__[HTML](https://html.spec.whatwg.org/multipage/parsing.html#tree-construction)__+tree construction algorithm, and the common functions generating them.  While+it would certainly be possible to implement the specification more directly,+the instructions it provides are not only (very strongly) build around mutable+data structures, they're also tailored to a pass-by-reference core library.+Haskell idiomatically being neither, it's actually easier (and likely more+performant, though that's not been tested) to implement a fourth stage from+scratch.++In effect, the main parser body handles all stateful computation ---tracking+the stack of open elements, managing transitions through the finite state+machine, etc.--- but doesn't know the shape of the document tree beyond its+very narrow window.  Instead, it emits a self-contained sequence of+instructions which can be very simply consumed with neither (much) further+modification nor external state; this then is the source from which the final+document tree is built.  Some duplication /is/ admittedly involved in doing so,+as the emitted instruction is typically accompanied by a corresponding change+to the parser state, but the relative logic simplicity makes doing so worth it.+-}+module Web.Mangrove.Parse.Tree.Patch+    ( -- * Types+      Patch ( .. )+    , TreeOutput ( .. )+    , treeRemainder+    , TokenizerOutputState+    , InsertAt ( .. )+    , TargetNode+    , ReparentDepth+      -- * Utility functions+      -- ** Tree types+      -- $utility-state+    , packTree+    , packTree_+    , packTreeErrors+    , packTreeErrors_+    , consTreeError+    , consTreeError_+    , (++|)+    , (|++)+    , (|++|)+      -- ** Token types+    , mapTokenErrs+    , mapTokenOut+      -- * Instructions+    , setDocumentQuirks+    , restartParsing+    , stopParsing+      -- ** Opening nodes+    , insertCharacter+    , insertComment+    , insertComment'+    , insertDoctype+    , addAttribute+      -- *** Elements+    , createElement+    , insertElement+    , insertElement_+    , insertNullElement+    , insertNullElement_+    , insertHeadElement+    , insertHeadElement_+      -- **** Formatting+    , insertForeignElement+    , insertForeignNullElement+    , insertFormattingElement+    , reconstructFormattingElements+      -- ** Closing nodes+      -- *** Single+    , closeCurrentNode+    , closeCurrentNode_+    , dropCurrentNode+    , softCloseCurrentNode_+    , closeAncestorNode_+    , closeAncestorNodes_+      -- *** Multiple+      -- **** Exclusive clear+    , clearToContext+    , tableContext+    , tableBodyContext+    , tableRowContext+      -- **** Inclusive clear+    , clearCount+    , closeElement+    , closeElements+    , closePElement+    , generateEndTags+    , impliedEndTags+    , thoroughlyImpliedEndTags+    ) where+++import qualified Control.Applicative as A+import qualified Control.Monad as N+import qualified Control.Monad.Trans.State as N.S++import qualified Data.ByteString as BS+import qualified Data.ByteString.Lazy as BS.L+import qualified Data.IntMap.Strict as M.I+import qualified Data.List as L+import qualified Data.Maybe as Y+import qualified Data.Text as T++import Data.Function ( (&) )+import Data.Functor ( ($>) )++import Web.Willow.DOM++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tokenize as Tokenize+import Web.Mangrove.Parse.Tokenize.Common+import Web.Mangrove.Parse.Tree.Common+import Web.Willow.Common.Encoding+++-- | The atomic, self-contained instruction set describing how to build a final+-- document tree.+data Patch+    = ErrorList [ParseError]+        -- ^ "Ignore the token", or add errors to, e.g., 'SetDocumentQuirks'.+    | SetDocumentQuirks QuirksMode+        -- ^ Specify which degree of backwards compatibility should be used in+        -- rendering the document.+    | CloseNodes (M.I.IntMap ReparentDepth)+        -- ^ "Pop the current node off the stack of open elements."+        -- +        -- The first 'TargetNode' of every tuple is treated identically to+        -- 'RelativeLocation', while the second 'ReparentDepth' indicates how+        -- many ancestors should be closed (i.e., @'CloseNodes' [(1, 1)]@+        -- closes the parent node).+    | SoftCloseCurrentNode+        -- ^ Float any following non-'Element' nodes (until the next+        -- 'CloseNodes' or 'DropCurrentNode') to the parent rather than+        -- inserting them under the current.+    | DropCurrentNode+        -- ^ "Remove the current node from its parent node, if it has one.  Pop+        -- it off the stack of open elements."+    | InsertCharacter [ParseError] Char+        -- ^ "Insert the character."+    | InsertComment [ParseError] InsertAt T.Text+        -- ^ "Insert a comment."+    | InsertElement [ParseError] ElementParams+        -- ^ "Insert a foreign element for the token, in the given namespace."+    | InsertAndSetDocumentType [ParseError] DocumentTypeParams+        -- ^ "Append a 'DocumentType' node to the @Document@ node.  Associate+        -- the former with the latter so that it is returned as the value of+        -- the @doctype@ attribute."+        -- +        -- While the specification requires the 'DocumentType' be explicitly+        -- associated as the @doctype@ attribute of the document, the parsing+        -- rules only allow for a single such node to be produced, so the extra+        -- processing isn't actually required.+    | AddAttribute InsertAt AttributeParams+        -- ^ "Add the attribute and its corresponding value to the top element+        -- of the stack of open elements."+    | RestartParsing+        -- ^ Discard the previous tree and begin a new, empty one.+  deriving ( Eq, Show, Read )+++-- | Some 'Patch'es represent instruction categories which may not always be+-- intended for the current location in the tree, and the resulting 'Tree's may+-- likewise migrate from where they're generated.  These are the "addresses"+-- which direct those datatypes through the hierarchy.+data InsertAt+    = RelativeLocation ReparentDepth+        -- ^ Insert a specific number of nodes /up/ the tree from where the+        -- 'Patch' or 'Tree' was generated.+    | InDocument+        -- ^ Insert as a direct child of the 'Web.Willow.IDL.Document.Document'+        -- itself.+    | InHtmlElement+        -- ^ Insert as a direct child of the top-level @html@ element in the+        -- document.+  deriving ( Eq, Show, Read )+++-- | Type-level clarification for a count of how many levels a 'Patch' should+-- rise in the hierarchy without affecting the intervening nodes; a value of+-- @0@ indicates the current node.+type TargetNode = Word++-- | Type-level clarification for how many levels up the hierarchy the+-- associated object should be moved.  Note that a value of @0@ won't result in+-- any noticable change.+type ReparentDepth = Word+++-- | The standard output of parsers used in the first tree construction stage.+-- Specifically, it contains the final state of the tokenization stage in+-- addition to the generated instructions, to enable the recursion loop to+-- detect the end of multi-token outputs and properly update the resume state.+data TreeOutput = TreeOutput+    { treePatches :: [Patch]+        -- ^ The instructions generated by the parser.+    , treeState :: TokenizerOutputState+        -- ^ The data required to resume tokenization immediately following the+        -- value, if possible.+    }+  deriving ( Eq, Show, Read )++-- | The unparsed portion of the binary stream, after building the+-- associated instruction set.+treeRemainder :: TreeOutput -> Maybe BS.ByteString+treeRemainder = fmap snd . treeState+++-- $utility-state+-- Many functions producing 'TreeOutput' objects have two forms: one taking a+-- 'TreeInput' (typically passed via the enclosing+-- 'Web.Willow.Common.Parser.Switch.switch') alongside some data to pack, and+-- one with only the latter.  That 'TreeInput' is the means by which errors+-- from earlier stages are propagated forward, and by which the support+-- framework identifies breakpoints in the tokenizer.  Therefore, most case+-- logic must be written to:+-- +-- * Emit the value returned by a function referencing the 'TreeInput',+--   unless that function is a 'Web.Willow.Common.Parser.push' to allow it to+--   be reconsumed.+-- * Only use the 'TreeInput' with a single function to avoid duplicating+--   errors.+-- +-- In many cases, it may be possible to pass the 'TreeInput' to one of+-- several functions.  So long as the above rules are followed, it often+-- doesn't matter which function performs the wrapping; attaching it in a way+-- that makes logical or proximal sense could provide clearer error display+-- positioning.+++-- | Emit a patchset from a tree construction parser, referencing the state+-- when the original token was produced.+-- +-- This produces a stateful output, and should only be used where the token+-- would otherwise be discarded; use 'packTree_' elsewhere.+packTree :: TreeInput -> [Patch] -> TreeBuilder TreeOutput+packTree t' ps = return $ TreeOutput+    { treePatches = ps+    , treeState = tokenState t'+    }++-- | Emit a patchset from a tree construction parser without any reference to+-- the original token.+-- +-- This produces a stateless output, and should only be used after the token+-- has been 'Web.Willow.Common.Parser.push'ed for reconsumption; use 'packTree'+-- elsewhere.+packTree_ :: [Patch] -> TreeBuilder TreeOutput+packTree_ ps = return $ TreeOutput+    { treePatches = ps+    , treeState = Nothing+    }+++-- | Modify the collection of errors associated with a wrapped token.+mapTokenErrs :: ([ParseError] -> [ParseError]) -> TreeInput -> TreeInput+mapTokenErrs f t' = t'+    { tokenErrs = f $ tokenErrs t'+    }++-- | Modify a wrapped token without affecting the associated data.+mapTokenOut :: (Token -> Token) -> TreeInput -> TreeInput+mapTokenOut f t' = t'+    { tokenOut = f $ tokenOut t'+    }+++-- | Emit a collection of errors from a tree construction parser, if and only+-- if that collection is non-empty.+-- +-- This produces a stateful output, and should only be used where the token+-- would otherwise be discarded; use 'packTreeErrors_' elsewhere.+packTreeErrors :: [ParseError] -> TreeInput -> TreeBuilder TreeOutput+packTreeErrors errs t' = do+    ps <- packTreeErrors_ $ errs ++ tokenErrs t'+    return $ ps+        { treeState = tokenState t'+        }++-- | Emit a collection of errors from a tree construction parser, if and only+-- if that collection is non-empty.+-- +-- This produces a stateless output, and should only be used after the token+-- has been 'Web.Willow.Common.Parser.push'ed for reconsumption; use+-- 'packTreeErrors' elsewhere, or 'consTreeError' and 'consTreeError_' if a+-- relevant set of patches already exists.+packTreeErrors_ :: [ParseError] -> TreeBuilder TreeOutput+packTreeErrors_ [] = packTree_ []+packTreeErrors_ errs = packTree_ [ErrorList errs]++-- | Prepend an error to the first patch in the list, or add an 'ErrorList'+-- entry if it doesn't support them.+-- +-- If this is the only patchset which may be generated, use 'packTreeErrors_'+-- rather than passing a null @['Patch']@.+consTreeError_ :: ParseError -> [Patch] -> [Patch]+consTreeError_ err (ErrorList errs:ps) =+    ErrorList (err : errs) : ps+consTreeError_ err (InsertCharacter errs c:ps) =+    InsertCharacter (err : errs) c : ps+consTreeError_ err (InsertComment errs loc d:ps) =+    InsertComment (err : errs) loc d : ps+consTreeError_ err (InsertElement errs d:ps) =+    InsertElement (err : errs) d : ps+consTreeError_ err (InsertAndSetDocumentType errs d:ps) =+    InsertAndSetDocumentType (err : errs) d : ps+consTreeError_ err ps = ErrorList [err] : ps++-- | Prepend an error to the wrapped patchset, adding an 'ErrorList' entry if+-- the first patch doesn't support them.+-- +-- If this is the only patchset which may be generated, use 'packTreeErrors'+-- rather than passing a null @['Patch']@.+consTreeError :: ParseError -> TreeOutput -> TreeOutput+consTreeError err out = out { treePatches = consTreeError_ err $ treePatches out }+++-- | Prepend a plain patchset to the contents of a wrapped one.+(++|) :: [Patch] -> TreeOutput -> TreeOutput+ps ++| out = out { treePatches = ps ++ treePatches out }+infixr 4 ++|++-- | Append a plain patchset to the contents of a wrapped one.+(|++) :: TreeOutput -> [Patch] -> TreeOutput+out |++ ps = out { treePatches = treePatches out ++ ps }+infixr 4 |++++-- | Concatenate the payloads of two wrapped patchsets, retaining the+-- associated state of the right one.+(|++|) :: TreeOutput -> TreeOutput -> TreeOutput+ps |++| ps' = ps' { treePatches = treePatches ps ++ treePatches ps' }+infixr 4 |++|+++-- | Set the degree of backwards compatibility the document seems to be written+-- for.+setDocumentQuirks :: QuirksMode -> TreeBuilder [Patch]+setDocumentQuirks quirks = do+    N.S.modify $ \state -> state+        { quirksMode = quirks+        }+    return [SetDocumentQuirks quirks]+++-- | Pass the stack of open elements into the given function in order to+-- determine how many elements should be closed, then close them.+clear :: ([ElementParams] -> Word) -> TreeBuilder [Patch]+clear f = do+    state <- N.S.get+    clearCount . f . map snd $ openElements state++-- | Pop a known number of nodes from the stack of open elements.+clearCount :: Word -> TreeBuilder [Patch]+clearCount 0 = return []+clearCount l = closeAncestorNodes_ 0 l+++-- | Close all markup tags until the current node is one of the given elements+-- in the HTML namespace, close it.  For the inverse (closing all tags which+-- /are/ listed), see 'generateEndTags'.+-- +-- 'tableContext', 'tableBodyContext', and 'tableRowContext' provide the+-- typical inputs.+clearToContext :: [ElementName] -> TreeBuilder [Patch]+clearToContext ns = clear countToContext+  where countToContext [] = 0+        countToContext (d:ds) = if elem (elementName d) ns && elementNamespace d == Just htmlNamespace+            then 0+            else succ $ countToContext ds++-- | __HTML:__+--      @[clear the stack back to a table context]+--      (https://html.spec.whatwg.org/multipage/parsing.html#clear-the-stack-back-to-a-table-context)@+-- +-- The list of elements to pass 'clearToContext' when closing all tags until+-- the start of the most recent table.+tableContext :: [ElementName]+tableContext =+    [ "table"+    , "template"+    , "html"+    ]++-- | __HTML:__+--      @[clear the stack back to a table body context]+--      (https://html.spec.whatwg.org/multipage/parsing.html#clear-the-stack-back-to-a-table-body-context)@+-- +-- The list of elements to pass 'clearToContext' when closing all tags until+-- the start of the most recent section of the current table.+tableBodyContext :: [ElementName]+tableBodyContext =+    [ "tbody"+    , "tfoot"+    , "thead"+    , "template"+    , "html"+    ]++-- | __HTML:__+--      @[clear the stack back to a table row context]+--      (https://html.spec.whatwg.org/multipage/parsing.html#clear-the-stack-back-to-a-table-row-context)@+-- +-- The list of elements to pass 'clearToContext' when closing all tags until+-- the start of the most recent row of the current table.+tableRowContext :: [ElementName]+tableRowContext =+    [ "tr"+    , "template"+    , "html"+    ]+++-- | "Pop elements from the stack of open elements until the given element in+-- the HTML namespace has been popped from the stack."+-- +-- If multiple elements may indicate an endpoint, use 'closeElements' instead.+closeElement :: ElementName -> TreeBuilder [Patch]+closeElement = closeElements . (: [])++-- | "Pop elements from the stack of open elements until one of the given+-- elements in the HTML namespace has been popped from the stack."+-- +-- If only a single element may indicate an endpoint, 'closeElement' could+-- provide a cleaner interface.+closeElements :: [ElementName] -> TreeBuilder [Patch]+closeElements names = clear countToElement+  where countToElement [] = 0+        countToElement (d:ds) = if elem (elementName d) names && elementNamespace d == Just htmlNamespace+            then 1+            else succ $ countToElement ds++-- | "Pop the current element from the stack of open elements."+-- +-- This produces a stateful output, and should only be used where the token+-- would otherwise be discarded; use 'closeCurrentNode_' elsewhere.+closeCurrentNode :: TreeInput -> TreeBuilder TreeOutput+closeCurrentNode t' = do+    close <- closeCurrentNode_+    state <- N.S.get+    return $ TreeOutput+        { treePatches = foldr consTreeError_ close $ tokenErrs t'+        , treeState = tokenState . mapTokenState t' $ resetNamespace state+        }+  where resetNamespace state tokState = tokState+            { currentNodeNamespace = Y.listToMaybe (openElements state) >>= elementNamespace . snd+            }++-- | "Pop the current element from the stack of open elements."+-- +-- This produces a stateless output, and should only be used where the token is+-- handled in another manner; use 'closeCurrentNode' elsewhere.  This has+-- identical behaviour to @'closeAncestorNode_' 0@.+closeCurrentNode_ :: TreeBuilder [Patch]+closeCurrentNode_ = closeAncestorNode_ 0++-- | As 'closeCurrentNode_', but the closed node and its descendants are not+-- retained in the document tree.+dropCurrentNode :: TreeBuilder [Patch]+dropCurrentNode = closeCurrentNode_ $> [DropCurrentNode]++-- | Partially close the current node, such that 'insertElement' (and related+-- element instructions) will still be inserted as children, but /every other/+-- type of node will instead be inserted as siblings.  This doesn't actually+-- change the internal state; a following call to 'closeCurrentNode' or similar+-- is still required.+softCloseCurrentNode_ :: [Patch]+softCloseCurrentNode_ = [SoftCloseCurrentNode]+++-- | Remove the node the specified number of ancestors up the tree from the+-- stack of open elements.  See 'closeCurrentNode_' for the special case where+-- the argument is @0@.+closeAncestorNode_ :: TargetNode -> TreeBuilder [Patch]+closeAncestorNode_ = flip closeAncestorNodes_ 1++-- | Remove several nodes the specified number of ancestors up the tree from+-- the stack of open elements.  See 'closeAncestorNode_' if only a single node+-- needs to be closed.+closeAncestorNodes_ :: TargetNode -> ReparentDepth -> TreeBuilder [Patch]+closeAncestorNodes_ l d = do+    state <- N.S.get+    let (es1, es2) = splitAt (fromIntegral l) $ openElements state+        d' = fromIntegral d+    N.S.put $ state+        { openElements = es1 ++ drop d' es2+        }+    return [CloseNodes . M.I.singleton (fromIntegral l) . fromIntegral . min d' $ length es2]+++-- | Close all markup tags up to and including the most recent @\<p\>@ element,+-- throwing an 'UnexpectedElementWithImpliedEndTag' if one of them does not+-- typically allow an implied end tag.  If the error is not required,+-- 'closeElement' provides similar behaviour.+closePElement :: TreeBuilder [Patch]+closePElement = do+    generate <- generateEndTags $ L.delete "p" impliedEndTags+    current <- currentNode+    let errF = if maybe True (nodeIsElement "p") current+            then id+            else consTreeError_ UnexpectedElementWithImpliedEndTag+    p <- closeElement "p"+    return $ generate ++ errF p++-- | Close all markup tags in a given list, until reaching an element which is+-- not in that list.  For the inverse (closing all tags /except/ what's+-- listed), see 'clearToContext'.+-- +-- 'impliedEndTags' and 'thoroughlyImpliedEndTags' provide the typical inputs.+generateEndTags :: [ElementName] -> TreeBuilder [Patch]+generateEndTags tags = clear countImpliable+  where countImpliable [] = 0+        countImpliable (d:ds) = if elem (elementName d) tags+            then succ $ countImpliable ds+            else 0++-- | __HTML:__+--      @[generate implied end tags]+--      (https://html.spec.whatwg.org/multipage/parsing.html#generate-implied-end-tags)@+-- +-- The list of elements to pass 'generateEndTags' when closing all markup tags+-- which typically allow their end tag to be omitted.+impliedEndTags :: [ElementName]+impliedEndTags =+    [ "dd"+    , "dt"+    , "li"+    , "optgroup"+    , "option"+    , "p"+    , "rb"+    , "rp"+    , "rt"+    , "rtc"+    ]++-- | __HTML:__+--      @[generate all implied end tags thoroughly]+--      (https://html.spec.whatwg.org/multipage/parsing.html#generate-all-implied-end-tags-thoroughly)@+-- +-- The list of elements to pass 'generateEndTags' when closing all markup tags,+-- including ones which may not usually allow their end tag to be omitted.+thoroughlyImpliedEndTags :: [ElementName]+thoroughlyImpliedEndTags =+    [ "caption"+    , "colgroup"+    , "tbody"+    , "td"+    , "tfoot"+    , "th"+    , "thead"+    , "tr"+    ] ++ impliedEndTags+++-- | __HTML:__+--      @[create an element for the token]+--      (https://html.spec.whatwg.org/multipage/parsing.html#create-an-element-for-the-token)@+-- +-- Reserve a place in the tree for a given node, but do not yet add it.  Note+-- that this does not otherwise affect the parser state, so any modification of+-- open elements or updating of the 'currentNodeNamespace', for example, needs+-- to be performed manually.  For that reason, 'insertElement_' is almost+-- always more desirable.+createElement :: ElementParams -> TreeBuilder (NodeIndex, ElementParams)+createElement d = do+    state <- N.S.get+    let index = elementIndex state+    N.S.put $ state+        { elementIndex = succ index+        }+    return (index, d)+++-- | __HTML:__+--      @[insert a character]+--      (https://html.spec.whatwg.org/multipage/parsing.html#insert-a-character)@+--      steps 1-3+-- +-- If the wrapped token is a 'Character', add it to the final tree; fails if+-- it's not.  Note that the concatenation of character into 'Text' nodes occurs+-- in the patch folding logic instead.+insertCharacter :: TreeInput -> TreeBuilder TreeOutput+insertCharacter t' = case tokenOut t' of+    Character c -> return $ TreeOutput+        { treePatches = [InsertCharacter (tokenErrs t') c]+        , treeState = tokenState t'+        }+    _ -> A.empty++-- | __HTML:__+--      @[insert a comment]+--      (https://html.spec.whatwg.org/multipage/parsing.html#insert-a-comment)@+--      with no explicit insertion postion+-- +-- If the wrapped token is a 'Tokenize.Comment', add it to the final tree at the current+-- position (@'RelativeLocation' 0@); fails if it's not.+insertComment :: TreeInput -> TreeBuilder TreeOutput+insertComment = insertComment' $ RelativeLocation 0++-- | __HTML:__+--      @[insert a comment]+--      (https://html.spec.whatwg.org/multipage/parsing.html#insert-a-comment)@+-- +-- If the wrapped token is a 'Tokenize.Comment', add it to the final tree at+-- the position specified; fails if it's not.+insertComment' :: InsertAt -> TreeInput -> TreeBuilder TreeOutput+insertComment' at t' = case tokenOut t' of+    Tokenize.Comment c -> return $ TreeOutput+        { treePatches = [InsertComment (tokenErrs t') at c]+        , treeState = tokenState t'+        }+    _ -> A.empty++-- | If the wrapped token is a 'Tokenize.Comment', add it to the final tree;+-- fails if it's not.+insertDoctype :: TreeInput -> TreeBuilder TreeOutput+insertDoctype t' = case tokenOut t' of+    Doctype d ->+        let system = doctypeSystemId d+            legacy+                | doctypeName d /= Just "html" = True+                | Y.isJust $ doctypePublicId d = True+                | Y.isJust system && system /= Just "about:legacy-compat" = True+                | otherwise = False+            errs'+                | legacy = LegacyDoctype : tokenErrs t'+                | otherwise = tokenErrs t'+        in  packTree t' [InsertAndSetDocumentType errs' $ tokenDocumentType t']+    _ -> A.empty++-- | __HTML:__+--      @[insert an HTML element]+--      (https://html.spec.whatwg.org/multipage/parsing.html#insert-an-html-element)@+-- +-- If the wrapped token is a 'StartTag', add it to the final tree as a markup+-- element in the HTML namespace; fails if it's not.+-- +-- For tag data generated /a priori/, use 'insertElement_' instead.+insertElement :: TreeInput -> TreeBuilder TreeOutput+insertElement = insertForeignElement htmlNamespace++-- | Add a markup element described by the input record to the tree in the HTML+-- namespace.+-- +-- If the tag data was obtained from the tokenizer, use 'insertElement' instead.+insertElement_ :: TagParams -> TreeBuilder [Patch]+insertElement_ d = treePatches <$> insertElement (dummyToken [] $ StartTag d)++-- | __HTML:__+--      @[insert a foreign element]+--      (https://html.spec.whatwg.org/multipage/parsing.html#insert-a-foreign-element)@+-- +-- If the wrapped token is a 'StartTag', add it to the final tree as a markup+-- element in the specified namespace; fails if it's not.+insertForeignElement :: Namespace -> TreeInput -> TreeBuilder TreeOutput+insertForeignElement ns t' = case tokenOut t' of+    StartTag d -> do+        let d' = packNodeData (Just ns) d+            errs'+                | tagIsSelfClosing d = NonVoidHtmlElementStartTagWithTrailingSolidus : tokenErrs t'+                | otherwise = tokenErrs t'+        e <- createElement d'+        N.S.modify $ \state -> state+            { openElements = e : openElements state+            }+        return $ TreeOutput+            { treePatches = [InsertElement errs' $ adjustAttributes d']+            , treeState = tokenState $ mapTokenState t' setNamespace+            }+    _ -> A.empty+  where adjustAttributes+            | ns == htmlNamespace = id+            | otherwise = adjustForeignAttributes+        setNamespace state = state+            { currentNodeNamespace = Just ns+            }++-- | If the wrapped token is a 'StartTag', add it to the final tree as a markup+-- element in the HTML namespace, and then immediately close it; fails if it's+-- not.+-- +-- For tag data generated /a priori/, use 'insertNullElement_' instead.+insertNullElement :: TreeInput -> TreeBuilder TreeOutput+insertNullElement = insertForeignNullElement htmlNamespace++-- | Add a markup element described by the input record to the tree in the HTML+-- namespace, and then immediately close it.+-- +-- If the tag data was obtained from the tokenizer, use 'insertNullElement'+-- instead.+insertNullElement_ :: TagParams -> TreeBuilder [Patch]+insertNullElement_ d = treePatches <$> insertNullElement (dummyToken [] $ StartTag d)++-- | If the wrapped token is a 'StartTag', add it to the final tree as a markup+-- element in the specified namespace, and then immediately close it; fails if+-- it's not.+insertForeignNullElement :: Namespace -> TreeInput -> TreeBuilder TreeOutput+insertForeignNullElement ns t' = case tokenOut t' of+    StartTag d -> return $ TreeOutput+        { treePatches =+            [ InsertElement (tokenErrs t') . adjustAttributes $ packNodeData (Just ns) d+            , CloseNodes $ M.I.singleton 0 1+            ]+        , treeState = tokenState t'+        }+    _ -> A.empty+  where adjustAttributes+            | ns == htmlNamespace = id+            | otherwise = adjustForeignAttributes+++-- | If the wrapped token is a 'StartTag', add it to the final tree as a markup+-- element in the HTML namespace and set it as the target of the head element+-- pointer; fails if it's not.+-- +-- For tag data generated /a priori/, use 'insertHeadElement_' instead.+insertHeadElement :: TreeInput -> TreeBuilder TreeOutput+insertHeadElement t' = do+    N.S.modify $ \state -> state+        { headElementPointer = Just $ elementIndex state+        }+    insertElement t'++-- | Add a markup element described by the input record to the tree in the HTML+-- namespace and set it as the target of the head element pointer.+-- +-- If the tag data was obtained from the tokenizer, use 'insertElement' instead.+insertHeadElement_ :: TagParams -> TreeBuilder [Patch]+insertHeadElement_ d = do+    N.S.modify $ \state -> state+        { headElementPointer = Just $ elementIndex state+        }+    insertElement_ d+++-- | Add an extra point of metadata to the indicated markup element, if that+-- element doesn't already have an attribute with that name.+addAttribute :: InsertAt -> NodeIndex -> BasicAttribute -> TreeBuilder [Patch]+addAttribute at i (name, value) = do+    state <- N.S.get+    let (es1, es2) = L.break ((==) i . fst) $ openElements state+    case es2 of+        ((_, e):es') | not (elem name . map attrName . toAttrList $ elementAttributes e) -> do+            N.S.put $ state+                { openElements = es1 ++ (i, addAttributeData e) : es'+                }+            return [AddAttribute at attr]+        _ -> return []+  where addAttributeData d = d+            { elementAttributes = insertAttribute attr $ elementAttributes d+            }+        attr = emptyAttributeParams+            { attrName = name+            , attrValue = value+            }+++-- | __HTML:__+--      @[push onto the list of active formatting elements]+--      (https://html.spec.whatwg.org/multipage/parsing.html#push-onto-the-list-of-active-formatting-elements)@+-- +-- Add a markup element described by the input record to the tree in the HTML+-- namespace.  The element is also added to the set of elements which are+-- recreated on overlapping markup spans via 'reconstructFormattingElements'.+insertFormattingElement :: TreeInput -> TreeBuilder TreeOutput+insertFormattingElement t' = do+    insert <- insertElement t'+    state <- N.S.get+    case Y.listToMaybe $ openElements state of+        Just e -> case tokenOut t' of+            StartTag d -> N.S.put $ state+                { activeFormattingElements =+                    pushFormattingElement (fst e, d) $ activeFormattingElements state+                }+            _ -> return ()+        Nothing -> return ()+    return insert+  where pushFormattingElement e [] = [[e]]+        pushFormattingElement e (fs:fss) = (e : fs') : fss+          where fs' = case L.findIndices (equalElement . snd) fs of+                    (_:_:i:is) -> case splitAts (i : is) fs of+                        [] -> []+                        (ds:dss) -> ds ++ concatMap (drop 1) dss+                    _ -> fs+                splitAts [] dss = [dss]+                splitAts (i:is) dss =+                    let (ds', dss') = splitAt i dss+                    in  ds' : splitAts (map (`subtract` i) is) dss'+                equalElement e' =+                    tagName (snd e) == tagName e'+                    -- This is only ever invoked on HTML elements, so the+                    -- namespaces shouldn't need to be checked for equality.+                    && tagAttributes (snd e) == tagAttributes e'++-- | __HTML:__+--      @[reconstruct the active formatting elements]+--      (https://html.spec.whatwg.org/multipage/parsing.html#reconstruct-the-active-formatting-elements)@+-- +-- Create new tokens for each of the yet-unclosed elements created by+-- 'insertFormattingElement', within the scope defined by+-- 'insertFormattingMarker'.+reconstructFormattingElements :: TreeBuilder [Patch]+reconstructFormattingElements = do+    state <- N.S.get+    let open = map fst $ openElements state+        (es, ess) = case activeFormattingElements state of+            [] -> ([], [])+            (es':ess') -> (es', ess')+        (toRebuild, alreadyOpened) = span (\(i, _) -> notElem i open) es+    N.S.put $ state+        { activeFormattingElements = alreadyOpened : ess+        }+    N.foldM reconstruct [] $ reverse toRebuild+  where reconstruct pss (_, d) = do+            ps <- treePatches <$> insertFormattingElement (dummyToken [] $ StartTag d)+            return $ pss ++ ps+++-- | __HTML:__+--      @[change the encoding]+--      (https://html.spec.whatwg.org/multipage/parsing.html#stop-parsing)@,+--      step 6+-- +-- Restore the tracked state to pristine condition, issue an instruction to+-- throw out any 'Tree' generated thus far, and set the given binary stream to+-- be used when the parser next resumes.+restartParsing :: BS.L.ByteString -> TreeBuilder TreeOutput+restartParsing initial = do+    N.S.put $ treeParserState defState+    return $ TreeOutput+        { treePatches = [RestartParsing]+        , treeState = Just (tokState, BS.L.toStrict initial)+        }+  where updateDecoder state = tokenizerState state &+            case decoderState_ $ tokenizerState state of+                Left initialize -> tokenizerEncoding $ fmap Just initialize+                Right decState -> tokenizerEncoding . Right $ fmap decoderEncoding decState+        defState = defaultTreeState+        tokState = updateDecoder defState++-- | __HTML:__+--      @[stop parsing]+--      (https://html.spec.whatwg.org/multipage/parsing.html#stop-parsing)@+-- +-- Close all remaining open elements, and perform other cleanup functions to+-- finalize the document tree.+stopParsing :: TreeInput -> TreeBuilder TreeOutput+stopParsing t' = clear (fromIntegral . length) >>= packTree t'
+ src/Web/Mangrove/Parse/Tree/Patch/Fold.hs view
@@ -0,0 +1,274 @@+{-|+Description:    Functions to collapse a state-free instruction list into a document tree.++Copyright:      (c) 2020 Sam May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      provisional+Portability:    portable++This module provides the logic powering the second half of this+implementation's split+__[HTML](https://html.spec.whatwg.org/multipage/parsing.html#tree-construction)__+tree construction algorithm.  Namely, the functions in this module operate over+the single-dimensional stream of static tree-building instructions generated by+the rest of the "Web.Mangrove.Parse.Tree" hierarchy, folding them into a+simplified DOM tree.  For a more detailed discussion of the design behind this,+see the documentation of "Web.Mangrove.Parse.Tree.Patch".+-}+module Web.Mangrove.Parse.Tree.Patch.Fold+    ( buildTree+    ) where+++import qualified Data.Bifunctor as F.B+import qualified Data.Either as E+import qualified Data.HashMap.Strict as M+import qualified Data.IntMap.Strict as M.I+import qualified Data.List as L+import qualified Data.Maybe as Y+import qualified Data.Text as T++import Web.Willow.DOM++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tree.Common hiding ( Token ( .. ) )+import Web.Mangrove.Parse.Tree.Patch+++-- | Fold a series of instructions describing how to build a document tree+-- (without reference to any persistent state) into the tree they describe.+buildTree :: [Patch] -> Tree+buildTree = buildTree' . foldPatchset+  where buildTree' (ts, []) = emptyTree+            { node = Document mode'+            , children = ts'+            }+          where (ms, ts') = L.partition (\t -> nodeType (node t) == Just DocumentNode) ts+                mode' = foldr max NoQuirks $ Y.mapMaybe (getQuirksMode . node) ms+                getQuirksMode (Document mode) = Just mode+                getQuirksMode _ = Nothing+        buildTree' (_, RestartParsing : ps) = buildTree' $ foldPatchset ps+        buildTree' (ts, ps) = buildTree' . F.B.first (ts ++) $ foldPatchset ps++-- | Consume the next minimal sequence of folding instructions which would+-- result in at least one complete tree.  The input patchset is read according+-- to a sensible behaviour for trees providing a root for the document+-- (typically the /only/ root).+foldPatchset :: [Patch] -> ([Tree], [Patch])+foldPatchset ps = F.B.first (joinTexts . Y.mapMaybe filterNull) $ foldPatchset' False ps+  where -- Preprocess the placeholder tuples, where the 'InsertAt' carries the+        -- semantic data rather than the 'Tree'.+        filterNull (Right (_, t)) = Just $ Right t+        filterNull (Left c) = Just $ Left c++-- | Consume the next sequence of folding instructions which would result in a+-- complete tree.  Any intervening instructions which are destined for other+-- parts of the final tree are extracted for re-direction upward.  Any textual+-- data to be inserted directly into the tree is returned as a 'Left' value for+-- later packing into a single 'Text' node, to avoid the exponential complexity+-- of, e.g., @'Data.Text'.'T.cons'@.+foldPatchset'+    :: Bool+        -- ^ Whether this function call originates from somewhere higher up the+        -- document tree; any explicit calls will almost always use 'False'.+    -> [Patch]+    -> ([Either ([ParseError], Char) (InsertAt, Tree)], [Patch])+foldPatchset' _ [] = ([], [])+foldPatchset' isInner (ErrorList _:ps) = foldPatchset' isInner ps+-- Break infinite loop on unmatched node-closing patches.+foldPatchset' False (CloseNodes _:ps) = foldPatchset' False ps+foldPatchset' False (SoftCloseCurrentNode:ps) = foldPatchset' False ps+foldPatchset' False (DropCurrentNode:ps) = foldPatchset' False ps+foldPatchset' False (RestartParsing:ps) = foldPatchset' False ps+-- Stop processing on node-closing patches to enable recursive consumption of+-- all patches until the end of the node.+foldPatchset' True ps@(DropCurrentNode:_) = ([], ps)+foldPatchset' True ps@(RestartParsing:_) = ([], ps)+-- Push a "close ancestors" along until it reaches one of the node-closing+-- patches; at that point, drop the payload destined for the current location.+foldPatchset' True (p@(CloseNodes ls):ps) = case decrementReparenting ls of+    _ | M.I.null ls -> foldPatchset' True ps+    (0, toFloat) -> case ps of+        [] -> ([], [p])+        (CloseNodes ls':ps') ->+            foldPatchset' True $ CloseNodes (M.I.unionWith (+) ls ls') : ps'+        (DropCurrentNode:ps') ->+            ([], DropCurrentNode : CloseNodes toFloat : ps')+        (RestartParsing:_) ->+            ([], ps)+        (p'@InsertElement{}:ps') ->+            foldPatchset' True $ p' : CloseNodes (incrementReparenting ls) : ps'+        (p':ps') ->+            foldPatchset' True $ p' : p : ps'+    _ -> ([], p : ps)+-- Simple patch -> node translation.+foldPatchset' isInner (SetDocumentQuirks mode:ps) =+    F.B.first (Right (InDocument, packQuirks mode) :) $ foldPatchset' isInner ps+foldPatchset' isInner (InsertAndSetDocumentType _ d:ps) =+    F.B.first (Right (InDocument, packDoctype d) :) $ foldPatchset' isInner ps+foldPatchset' isInner (InsertComment _ loc txt:ps) =+    F.B.first (Right (loc, packComment txt) :) $ foldPatchset' isInner ps+foldPatchset' isInner (AddAttribute loc attr:ps) =+    F.B.first (Right (loc, packAttribute attr) :) $ foldPatchset' isInner ps+-- Prepends to the following text rather than the spec's appending to the+-- previous.  That should still have an identical result.+foldPatchset' isInner (InsertCharacter errs h:ps) =+    F.B.first (Left (errs, h) :) $ foldPatchset' isInner ps+-- Float non-element nodes up one level.+foldPatchset' True (SoftCloseCurrentNode:ps) =+    F.B.first (map floatTrees) $ foldPatchset' True ps+  where floatTrees t'@(Right (RelativeLocation l, t))+            | nodeType (node t) == Just ElementNode = t'+            | otherwise = Right (RelativeLocation $ succ l, t)+        -- Text nodes here don't take advantage of ahead-of-time gathering+        -- (thus need more complex concatenation), but shouldn't be a large+        -- portion of the document.+        floatTrees (Left (_, c)) = Right (RelativeLocation 1, emptyTree+            { node = Text $ T.singleton c+            })+        floatTrees t' = t'+-- Consume patches until and including the (matching) node-closing patch.+foldPatchset' isInner (InsertElement _ tag:ps) =+    let (ts, ps') = foldPatchset' True ps+        (ts', bubble) = filterFloaters ts $ floatsElement isInner tag+        (attrs, ts'') = L.partition isAttribute ts'+        tag' = tag+            { elementAttributes = M.union (elementAttributes tag) .+                fromAttrList $ Y.mapMaybe toAttribute attrs+            }+        this = Right (RelativeLocation 0, packElement tag' $ joinTexts ts'')+    in  case ps' of+        [] -> (this : bubble, [])+        (DropCurrentNode:ps'') -> F.B.first (++ bubble) $ foldPatchset' True ps''+        (CloseNodes ls:ps'') ->+            let (here, toFloat) = decrementReparenting ls+                ls' = case here of+                    l | l >= 2 -> M.I.insertWith (+) 0 (pred l) toFloat+                    _ -> toFloat+                (out, trail) = foldPatchset' True $ CloseNodes ls' : ps''+            in  (this : bubble ++ out, trail)+        (RestartParsing:_) -> ([], ps')+        _ ->+            let (out, trail) = foldPatchset' True ps'+                cleaned = map (fmap $ \c -> (RelativeLocation 0, c)) ts''+            in  (cleaned ++ out ++ bubble, trail)+  where isAttribute (Right tok) = nodeType (node tok) == Just AttributeNode+        isAttribute (Left _) = False+        toAttribute (Right tok) = case node tok of+            Attribute attr -> Just attr+            _ -> Nothing+        toAttribute (Left _) = Nothing+++-- | Given a heterogeneous set of patches potentially destined for multiple+-- places within the document hierarchy, and a known set of addresses for the+-- current location, partition out the relevant patches from those continuing+-- onward.+filterFloaters+    :: [Either ([ParseError], Char) (InsertAt, Tree)]+    -> [InsertAt]+    -> ([Either ([ParseError], Char) Tree], [Either ([ParseError], Char) (InsertAt, Tree)])+filterFloaters ts here = F.B.bimap+    (map $ F.B.second snd)+    (map . fmap $ F.B.first decrementTarget) $+    L.partition isHere ts+  where decrementTarget loc = case loc of+            RelativeLocation 0 -> loc+            RelativeLocation i -> RelativeLocation $ pred i+            _ -> loc+        isHere (Right (l, _)) = elem l here+        isHere (Left _) = True++-- | Calculate the accepted addresses for the element at the current location;+-- usually just @'RelativeLocation' 0@, but a root @\<html\>@ node also accepts+-- 'InHtmlElement'.+floatsElement+    :: Bool+        -- ^ Whether this node is a descendant node of the root.+    -> ElementParams+    -> [InsertAt]+floatsElement isInner d+    | elementName d == T.pack "html" && not isInner = [RelativeLocation 0, InHtmlElement]+    | otherwise = [RelativeLocation 0]+++-- | Given the list of reparenting directives used by a 'CloseNodes'+-- instruction, increase the distance to the node-to-reparent to account for+-- the patch being pushed into a child node.+incrementReparenting :: M.I.IntMap ReparentDepth -> M.I.IntMap ReparentDepth+incrementReparenting = M.I.mapKeys succ++-- | Given the list of reparenting directives used by a 'CloseNodes'+-- instruction, partition out the accumulated directives intended for the+-- current node, and decrease the distance to the node-to-reparent to account+-- for the child node being closed.+decrementReparenting :: M.I.IntMap ReparentDepth -> (ReparentDepth, M.I.IntMap ReparentDepth)+decrementReparenting ls = (Y.fromMaybe 0 $ M.I.lookup 0 ls, M.I.mapKeys pred $ M.I.delete 0 ls)+++-- | Collapse all sequential 'Char' sequences in the list and pack them into a+-- single 'Text' node, preserving the interspersed complete subtrees.+joinTexts :: [Either ([ParseError], Char) Tree] -> [Tree]+joinTexts [] = []+joinTexts (Right t : xs) = t : joinTexts xs+joinTexts (Left c : xs) = txt' : joinTexts xs'+  where (cs', xs') = span E.isLeft xs+        cs = map (either id $ error "unexpected 'Right' after @span isLeft@") cs'+        (_, txt) = unzip $ c : cs+        txt' = emptyTree+            { node = Text $ T.pack txt+            }+++-- | Wrap the desired level of backwards compatibility into a 'Document' node+-- as a placeholder.+packQuirks :: QuirksMode -> Tree+packQuirks mode = emptyTree+    { node = Document mode+    }++-- | Wrap a string of characters in the data types expected for a 'Comment'+-- node in the output of 'buildTree'.+packComment :: T.Text -> Tree+packComment txt = emptyTree+    { node = Comment txt+    }++-- | Wrap the metadata contained in a document type declaration in the data+-- types expected for a 'DocumentType' node in the output of 'buildTree'.+packDoctype :: DocumentTypeParams -> Tree+packDoctype dtd = emptyTree+    { node = DocumentType dtd+    }++-- | Wrap the metadata contained in a markup tag in the data types expected for+-- an 'Element' node in the output of 'buildTree'.  If it represents an HTML+-- @\<template\>@ element, additionally wrap the children in a+-- 'DocumentFragment' node to indicate the @[template+-- contents](https://html.spec.whatwg.org/scripting.html#template-contents)@+-- parameter.+packElement :: ElementParams -> [Tree] -> Tree+packElement tag childTrees+    | nodeIsElement (T.pack "template") tag = emptyTree+        { node = e+        , children = [emptyTree+            { node = DocumentFragment+            , children = childTrees+            }]+        }+    | otherwise = emptyTree+        { node = e+        , children = childTrees+        }+  where e = Element tag++-- | Wrap the metadata contained in a metadata tag's attribute in the data+-- types expected for an 'Attribute' node in the output of 'buildTree'; this+-- will then be subsumed into the parent 'Element' node, and does not remain in+-- the final document tree.+packAttribute :: AttributeParams -> Tree+packAttribute attr = emptyTree+    { node = Attribute attr+    }
+ test/Test/Mangrove/Html5Lib.hs view
@@ -0,0 +1,31 @@+{-|+Description:    ++Copyright:      (c) 2020 Samuel May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      experimental+Portability:    portable+-}+module Main where+++import qualified Control.Monad as M+import qualified System.Exit as E+import qualified Test.HUnit as U++import qualified Test.Mangrove.Html5Lib.Tokenizer as Tokenizer+import qualified Test.Mangrove.Html5Lib.TreeConstruction as TreeConstruction+++main :: IO ()+main = do+    tokenizer <- Tokenizer.tests+    tree <- TreeConstruction.tests+    results <- U.runTestTT $ U.TestList+        [ tokenizer+        , tree+        ]+    +    M.unless (U.errors results == 0 && U.failures results == 0) E.exitFailure
+ test/Test/Mangrove/Html5Lib/Common.hs view
@@ -0,0 +1,13 @@+module Test.Mangrove.Html5Lib.Common+    ( dataFile+    ) where+++import Paths_mangrove++import System.FilePath ( (</>) )+++dataFile :: FilePath -> IO FilePath+dataFile f = getDataFileName $ dataDir </> f+  where dataDir = "test" </> "html5lib-tests"
+ test/Test/Mangrove/Html5Lib/Tokenizer.hs view
@@ -0,0 +1,96 @@+module Test.Mangrove.Html5Lib.Tokenizer+    ( tests+    ) where+++import qualified Data.ByteString.Short as BS.S+import qualified Data.List as L+import qualified Data.Text as T++import qualified Test.HUnit as U++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tokenize+import Web.Willow.DOM++import Test.Mangrove.Html5Lib.Tokenizer.JSON++import Test.HUnit ( (~:), (@?=) )+++tests :: IO U.Test+tests = U.TestLabel "tokenizer" . U.TestList <$> mapM runTestFile+    [ "test1"+    , "test2"+    , "test3"+    , "test4"+    , "entities"+    , "namedEntities"+    , "numericEntities"+    , "unicodeChars"+ -- , "unicodeCharsProblematic" -- Seems to contradict the standard in allowing surrogates to be decoded+    , "contentModelFlags"+    , "domjs"+    , "escapeFlag"+    , "pendingSpecChanges"+ -- , "xmlViolation"+    ]+++runTestFile :: FilePath -> IO U.Test+runTestFile p = U.TestLabel (p ++ ".test") . U.TestList . map run <$> parseTestFile p++run :: JsonTest -> U.Test+run test = T.unpack (description test) ~: U.TestList $ map (runState test)+    [ (show state, tokenizerMode state tokState)+    | state <- initialStates test+    ]+  where tokState = case lastStartTag test of+            Just n -> tokenizerStartTag Nothing n $ defaultTokenizerState+            Nothing -> defaultTokenizerState++runState :: JsonTest -> (String, TokenizerState) -> U.Test+runState test (mode, state) = mode ~: U.TestCase $ do+    let (out, state') = tokenize state $ input test+        final = finalizeTokenizer state'+        out' = out ++ final+    (L.sortOn show . concat $ map mapError out', trimEof $ map mapToken out') @?=+        (map errorCode $ errors test, concat $ output test)+  where normalizeToken (EndTag d) = EndTag $ emptyTagParams+            { tagName = tagName d+            }+        normalizeToken tok = tok+        mapToken = normalizeToken . snd+        normalizeError (CharacterReferenceOutsideUnicodeRange _) =+            CharacterReferenceOutsideUnicodeRange 0+        normalizeError (DuplicateAttribute _) =+            DuplicateAttribute (T.empty, T.empty)+        normalizeError (DuplicateSingletonElement _) =+            DuplicateSingletonElement emptyElementParams+        normalizeError (FramesetInBody _) =+            FramesetInBody emptyElementParams+        normalizeError (MalformedTableStructure _) =+            MalformedTableStructure emptyElementParams+        normalizeError (InvalidByteSequence _) =+            InvalidByteSequence $ BS.S.pack []+        normalizeError (NoncharacterCharacterReference _) =+            NoncharacterCharacterReference '\NUL'+        normalizeError (ObsoleteTagName _) =+            ObsoleteTagName T.empty+        normalizeError (SurrogateCharacterReference _) =+            SurrogateCharacterReference '\NUL'+        normalizeError (UnexpectedCharacterAfterDoctypeSystemIdentifier _) =+            UnexpectedCharacterAfterDoctypeSystemIdentifier '\NUL'+        normalizeError (UnexpectedDoctype _) =+            UnexpectedDoctype emptyDocumentTypeParams+        normalizeError (UnexpectedDescendantElement _) =+            UnexpectedDescendantElement emptyElementParams+        normalizeError (UnexpectedEndTag _) =+            UnexpectedEndTag emptyElementParams+        normalizeError (UnmatchedEndTag _) =+            UnmatchedEndTag emptyElementParams+        normalizeError err = err+        mapError = map normalizeError . fst+        trimEof ts = case reverse ts of+            (EndOfStream:ts') -> reverse ts'+            _ -> ts
+ test/Test/Mangrove/Html5Lib/Tokenizer/JSON.hs view
@@ -0,0 +1,179 @@+{-# LANGUAGE OverloadedStrings #-}++{-# OPTIONS_GHC -Wno-orphans #-}++module Test.Mangrove.Html5Lib.Tokenizer.JSON+    ( JsonTest ( .. )+    , TestError ( .. )+    , parseTestFile+    ) where+++import qualified Data.Aeson as J+import qualified Data.Aeson.Types as J+import qualified Data.ByteString as BS+import qualified Data.ByteString.Builder as BS.B+import qualified Data.ByteString.Lazy as BS.L+import qualified Data.ByteString.Short as BS.S+import qualified Data.HashMap.Strict as M+import qualified Data.List as L+import qualified Data.Maybe as Y+import qualified Data.Text as T++import qualified Numeric as N++import qualified Text.Read as R++import Web.Mangrove.Parse.Common.Error+import Web.Mangrove.Parse.Tokenize+import Web.Willow.DOM hiding ( Node ( .. ) )++import Test.Mangrove.Html5Lib.Common++import Data.Aeson ( (.:), (.:?), (.!=) )+import Data.Vector ( (!?) )+import System.FilePath ( (</>), (<.>) )+++testFile :: FilePath -> IO FilePath+testFile f = (</> f) <$> dataFile "tokenizer"++parseTestFile :: FilePath -> IO [JsonTest]+parseTestFile p = testFile (p <.> "test") >>= J.eitherDecodeFileStrict >>= either fail rewrap+  where rewrap (TestFile ts) = return ts+++data JsonTest = JsonTest+    { description :: T.Text+    , input :: BS.ByteString+    , output :: [[Token]]+    , initialStates :: [CurrentTokenizerState]+    , lastStartTag :: Maybe T.Text+    , errors :: [TestError]+    }+  deriving ( Eq, Show, Read )++instance J.FromJSON JsonTest where+    parseJSON = J.withObject "test" $ \v -> do+        doubleEscaped <- v .:? "doubleEscaped" .!= False+        JsonTest+            <$> v .: "description"+            <*> fmap (BS.L.toStrict . BS.B.toLazyByteString . BS.B.stringUtf8 . unescape doubleEscaped) (v .: "input")+            <*> fmap (map $ unpackTokens doubleEscaped) (v .: "output")+            <*> v .:? "initialStates" .!= [DataState]+            <*> v .:? "lastStartTag"+            <*> fmap (L.sortOn (show . errorCode)) (v .:? "errors" .!= [])+      where unpackTokens doubleEscaped (TestToken (Right (Comment str))) =+                [Comment . T.pack $ unescape doubleEscaped str]+            unpackTokens _ (TestToken (Right tok)) = [tok]+            unpackTokens doubleEscaped (TestToken (Left str)) =+                map Character . unescape doubleEscaped $ T.pack str+            unescape False = T.unpack+            unescape True = unescape' . T.splitOn "\\u"+            unescape' [] = ""+            unescape' (t:ts) = T.unpack t ++ concatMap unescapeCode ts+            unescapeCode t = maybe '\NUL' (toEnum . fst) (Y.listToMaybe $ N.readHex code) : ts+              where (code, ts) = splitAt 4 $ T.unpack t+++data TestError = TestError+    { errorCode :: ParseError+    , line :: Word+    , column :: Word+    }+  deriving ( Eq, Show, Read )++instance J.FromJSON TestError where+    parseJSON = J.withObject "error" $ \v -> TestError+        <$> v .: "code"+        <*> v .: "line"+        <*> v .: "col"+++instance J.FromJSON CurrentTokenizerState where+    parseJSON = J.withText "initialState" $ \v -> case v of+        "Data state" -> return DataState+        "PLAINTEXT state" -> return PlainTextState+        "RCDATA state" -> return RCDataState+        "RAWTEXT state" -> return RawTextState+        "Script data state" -> return ScriptDataState+        "CDATA section state" -> return CDataState+        _ -> fail $ "Unknown tokenizer state '" ++ T.unpack v ++ "'"+++newtype TestToken = TestToken (Either String Token)+  deriving ( Eq, Show, Read )++instance J.FromJSON TestToken where+    parseJSON = J.withArray "output" $ \v -> case v !? 0 of+        Just "DOCTYPE" -> do+            name <- maybe (return Nothing) parseTextOrNull $ v !? 1+            public <- maybe (return Nothing) parseTextOrNull $ v !? 2+            system <- maybe (return Nothing) parseTextOrNull $ v !? 3+            correctness <- maybe (return True) parseBool $ v !? 4+            return . TestToken . Right . Doctype $ emptyDoctypeParams+                { doctypeName = name+                , doctypePublicId = public+                , doctypeSystemId = system+                , doctypeQuirks = not correctness+                }+        Just "StartTag" -> do+            name <- maybe (return T.empty) parseText $ v !? 1+            attrs <- maybe (return []) parseAttributes $ v !? 2+            selfClosing <- maybe (return False) parseBool $ v !? 3+            return . TestToken . Right . StartTag $ emptyTagParams+                { tagName = name+                , tagIsSelfClosing = selfClosing+                , tagAttributes = M.fromList attrs+                }+        Just "EndTag" -> J.withText "EndTag" (return . TestToken . Right . packEndTag) $+            Y.fromMaybe (J.String T.empty) (v !? 1)+        Just "Comment" -> J.withText "Comment" (return . TestToken . Right . Comment) $+            Y.fromMaybe (J.String T.empty) (v !? 1)+        Just "Character" -> J.withText "Character" (return . TestToken . Left . T.unpack) $+            Y.fromMaybe (J.String T.empty) (v !? 1)+        Just _ -> fail $ "Unknown output token"+        Nothing -> fail $ "Empty output token"+      where packEndTag name = EndTag $ emptyTagParams+                { tagName = name+                }+            parseText = J.withText "text" return+            parseTextOrNull (J.String t) = return $ Just t+            parseTextOrNull J.Null = return Nothing+            parseTextOrNull v = J.typeMismatch "String or Null" v+            parseBool = J.withBool "bool" return+            parseAttributes = J.withObject "attributes" $ fmap M.toList . mapM parseText+++instance J.FromJSON ParseError where+    parseJSON = J.withText "code" $ \v -> case v of+        "character-reference-outside-unicode-range" ->+            return $ CharacterReferenceOutsideUnicodeRange 0+        "duplicate-attribute" -> return $ DuplicateAttribute (T.empty, T.empty)+        "duplicate-singleton-element" -> return $ DuplicateSingletonElement emptyElementParams+        "frameset-in-body" -> return $ FramesetInBody emptyElementParams+        "malformed-table-structure" -> return $ MalformedTableStructure emptyElementParams+        "invalid-byte-sequence" -> return . InvalidByteSequence $ BS.S.pack []+        "noncharacter-character-reference" -> return $ NoncharacterCharacterReference '\NUL'+        "obsolete-tag-name" -> return $ ObsoleteTagName T.empty+        "surrogate-character-reference" -> return $ SurrogateCharacterReference '\NUL'+        "unexpected-character-after-doctype-system-identifier" ->+            return $ UnexpectedCharacterAfterDoctypeSystemIdentifier '\NUL'+        "unexpected-descendant-element" -> return $ UnexpectedDescendantElement emptyElementParams+        "unexpected-doctype" -> return $ UnexpectedDoctype emptyDocumentTypeParams+        "unexpected-end-tag" -> return $ UnexpectedEndTag emptyElementParams+        "unmatched-end-tag" -> return $ UnmatchedEndTag emptyElementParams+        _ -> maybe (fail $ err v) return . R.readMaybe . concat .+            map (T.unpack . conCase) $ T.split (== '-') v+      where conCase "cdata" = "CData"+            conCase "eof" = "EOF"+            conCase t = T.toTitle t+            err v = "Could not parse error code '" ++ T.unpack v ++ "'"+++newtype TestFile = TestFile [JsonTest]+  deriving ( Eq, Show, Read )++instance J.FromJSON TestFile where+    parseJSON = J.withObject "file" $ \v -> TestFile+        <$> v .: "tests"
+ test/Test/Mangrove/Html5Lib/TreeConstruction.hs view
@@ -0,0 +1,68 @@+module Test.Mangrove.Html5Lib.TreeConstruction+    ( tests+    ) where+++import qualified Control.Exception as E++import qualified Test.HUnit as U++import Web.Mangrove.Parse.Tree++import Test.Mangrove.Html5Lib.TreeConstruction.Parser++import Test.HUnit ( (@=?) )+++tests :: IO U.Test+tests = U.TestLabel "tree-construction" . U.TestList <$> mapM runTestFile files+  where files =+            [ "tests" ++ show (i :: Word)+            | i <- [1..12]+                -- "tests13" doesn't exist+                ++ [14..26]+            ] +++            [ "adoption01"+            , "adoption02"+            , "blocks"+            , "comments01"+            , "doctype01"+            , "domjs-unsafe"+            , "entities01"+            , "entities02"+            , "foreign-fragment"+            , "html5test-com"+            , "inbody01"+            , "isindex"+            , "main-element"+            , "math"+            , "menuitem-element"+            , "namespace-sensitivity"+            , "plain-text-unsafe"+            , "ruby"+            , "scriptdata01"+            , "tables01"+            , "template"+            , "tests_innerHTML_1"+            , "tricky01"+            , "webkit01"+            , "webkit02"+            ]++runTestFile :: FilePath -> IO U.Test+runTestFile p = U.TestLabel (p ++ ".dat") . U.TestList . map run <$> parseTestFile p+++run :: TreeTest -> U.Test+run t = U.TestCase $ do+    result <- E.tryJust filterErrors . E.evaluate .+        uncurry finalizeTree . tree (state t) $ input t+    either return (output t @=?) $ normalizeOutput <$> result+  where filterErrors (E.ErrorCall "Adoption agency not yet implemented") = Just ()+        filterErrors (E.ErrorCall "Foster parenting not yet implemented") = Just ()+        filterErrors _ = Nothing+        normalizeOutput n = n+            { node = normalizeQuirks $ node n+            }+        normalizeQuirks (Document _) = Document NoQuirks+        normalizeQuirks n = n
+ test/Test/Mangrove/Html5Lib/TreeConstruction/Parser.hs view
@@ -0,0 +1,208 @@+{-# LANGUAGE CPP #-}+{-# LANGUAGE OverloadedStrings #-}+{-# OPTIONS_GHC -Wno-incomplete-uni-patterns #-}++module Test.Mangrove.Html5Lib.TreeConstruction.Parser+    ( parseTestFile+    , TreeTest ( .. )+    ) where+++import qualified Data.ByteString.Char8 as BS.C+import qualified Data.HashMap.Strict as M+import qualified Data.List as L+import qualified Data.Maybe as Y+import qualified Data.Text as T+import qualified Data.Text.Encoding as T++import Web.Mangrove.Parse.Tree+import Web.Willow.DOM++import Test.Mangrove.Html5Lib.Common++#if MIN_VERSION_base(4,11,0)+#else+import Data.Semigroup ( (<>) )+#endif++import System.FilePath ( (</>), (<.>) )+++data TreeTest = TreeTest+    { input :: BS.C.ByteString+    , errors :: [(Word, Word, T.Text)]+    , output :: Tree+    , state :: TreeState+    }+  deriving ( Eq, Show, Read )++data RawTest = RawTest+    { input' :: [BS.C.ByteString]+    , errors' :: [BS.C.ByteString]+    , output' :: [BS.C.ByteString]+    , state' :: Maybe BS.C.ByteString+    , scripting' :: Maybe Bool+    }+  deriving ( Eq, Show, Read )+++testFile :: FilePath -> IO FilePath+testFile f = (</> f) <$> dataFile "tree-construction"++parseTestFile :: FilePath -> IO [TreeTest]+parseTestFile p = fmap parse $ testFile (p <.> "dat") >>= BS.C.readFile++parse :: BS.C.ByteString -> [TreeTest]+parse = Y.mapMaybe (parseTest . rawTest) . breakTests+  where breakTests = breakTests' . BS.C.split '\n'+        breakTests' [] = []+        breakTests' bss = case break BS.C.null bss of+            (bs, []) -> [bs]+            (bs, bss') -> bs : breakTests' (drop 1 bss')++rawTest :: [BS.C.ByteString] -> RawTest+rawTest ("#data":bss) = (rawTest bss')+    { input' = d+    }+  where (d, bss') = break isInstruction bss+rawTest ("#errors":bss) = (rawTest bss')+    { errors' = e+    }+  where (e, bss') = break isInstruction bss+rawTest ("#document-fragment":bss) = (rawTest bss')+    { state' = Y.listToMaybe x+    }+  where (x, bss') = break isInstruction bss+rawTest ("#document":bss) = (rawTest bss')+    { output' = t+    }+  where (t, bss') = break isInstruction bss+rawTest ("#script-off":bss) = (rawTest bss')+    { scripting' = Just False+    }+  where (_, bss') = break isInstruction bss+rawTest ("#script-on":bss) = (rawTest bss')+    { scripting' = Just True+    }+  where (_, bss') = break isInstruction bss+rawTest (_:bss) = rawTest $ dropWhile (not . isInstruction) bss+rawTest [] = RawTest+    { input' = []+    , errors' = []+    , output' = []+    , state' = Nothing+    , scripting' = Nothing+    }++isInstruction :: BS.C.ByteString -> Bool+isInstruction bs = fmap fst (BS.C.uncons bs) == Just '#'+++parseTest :: RawTest -> Maybe TreeTest+parseTest t+    | null $ input' t = Nothing+    | null $ output' t = Nothing+    | scripting' t == Just True = Nothing+    | otherwise = Just $ TreeTest+        { input = BS.C.intercalate "\n" $ input' t+        , errors = Y.mapMaybe parseError $ errors' t+        , output = parseOutput . map (BS.C.drop 2) . foldr joinMultiline [] $ output' t+        , state = case state' t of+            Just bs -> treeFragment (packElement bs) [] Nothing Nothing defaultTreeState+            Nothing -> defaultTreeState+        }+  where joinMultiline bs [] = [bs]+        joinMultiline bs (bs':bss) = if BS.C.take 2 bs' == "| "+            then bs : bs' : bss+            else bs <> "\n" <> bs' : bss+        htmlElement = emptyElementParams+            { elementName = "html"+            , elementNamespace = Just htmlNamespace+            }+        packElement = packElement' . parseNode . BS.C.cons '<' . flip BS.C.snoc '>'+        packElement' (Element d) = d+        packElement' _ = htmlElement++parseError :: BS.C.ByteString -> Maybe (Word, Word, T.Text)+parseError _ = Nothing --TODO++parseOutput :: [BS.C.ByteString] -> Tree+parseOutput bss = Tree+    { node = Document NoQuirks+    , children = map snd . foldr encapsulate [] $ map dropParents bss+    }+  where dropParents bs =+            let (ws, n) = BS.C.span (== ' ') bs+            in  (BS.C.length ws, parseNode n)+        encapsulate (d, e) es =+            let (cs, ss) = span ((d <) . fst) es+                (attrs, nonAttrs) = L.partition (\c -> nodeType (node $ snd c) == Just AttributeNode) cs+                (e', cs') = if nodeType e == Just ElementNode+                    then let Element t = e+                             t' = t+                                { elementAttributes =+                                    M.union (elementAttributes t) . fromAttrList $ Y.mapMaybe (unpack . node . snd) attrs+                                }+                             unpack (Attribute a) = Just a+                             unpack _ = Nothing+                         in  (Element t', nonAttrs)+                    else (e, cs)+            in  (d, Tree+                    { node = e'+                    , children = map snd cs'+                    }) : ss++parseNode :: BS.C.ByteString -> Node+parseNode bs+    | bs == "content" = DocumentFragment+    | BS.C.take 10 bs == "<!DOCTYPE " =+        let (n', bs') = BS.C.break (\c -> c == ' ' || c == '>') $ BS.C.drop 10 bs+            (n, p, s) = if BS.C.take 1 bs' == ">"+                then (repack 0 0 n', T.empty, T.empty)+                else let (p', s') = BS.C.break (== '"') $ BS.C.drop 2 bs'+                     in  (repack 0 0 n', repack 0 0 p', repack 3 2 s')+        in  DocumentType $ emptyDocumentTypeParams+                { documentTypeName = n+                , documentTypePublicId = p+                , documentTypeSystemId = s+                }+    | BS.C.take 5 bs == "<!-- " = Comment $ repack 5 4 bs+    | BS.C.take 1 bs == "<" && fmap snd (BS.C.unsnoc bs) == Just '>' =+        let (ns, n) = trimTag $ repack 1 1 bs+        in  Element $ emptyElementParams+                { elementNamespace = Just ns+                , elementName = n+                }+    | BS.C.take 1 bs == "\"" = Text $ repack 1 1 bs+    | BS.C.isInfixOf "=\"" bs && BS.C.last bs == '"' =+        let (n', v) = BS.C.break (== '=') bs+        in  Attribute $ (trimAttribute $ repack 0 0 n')+                { attrValue = repack 2 1 v+                }+    | otherwise = Document NoQuirks+  where repack h f bs' = T.decodeUtf8 . BS.C.drop h $ BS.C.take (BS.C.length bs' - f) bs'+        trimTag t+            | T.take 4 t == "svg " = (svgNamespace, T.drop 4 t)+            | T.take 5 t == "math " = (mathMLNamespace, T.drop 5 t)+            | otherwise = (htmlNamespace, t)+        trimAttribute t+            | T.take 6 t == "xlink " = emptyAttributeParams+                { attrNamespace = Just xlinkNamespace+                , attrPrefix = Just "xlink"+                , attrName = T.drop 6 t+                }+            | T.take 4 t == "xml " = emptyAttributeParams+                { attrNamespace = Just xmlNamespace+                , attrPrefix = Just "xml"+                , attrName = T.drop 4 t+                }+            | T.take 6 t == "xmlns " = emptyAttributeParams+                { attrNamespace = Just xmlnsNamespace+                , attrPrefix = Just "xmlns"+                , attrName = T.drop 6 t+                }+            | otherwise = emptyAttributeParams+                { attrNamespace = Nothing+                , attrPrefix = Nothing+                , attrName = t+                }
+ test/Test/Mangrove/Unit.hs view
@@ -0,0 +1,28 @@+{-|+Description:    ++Copyright:      (c) 2020 Samuel May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      experimental+Portability:    portable+-}+module Main where+++import qualified Control.Monad as N+import qualified System.Exit as E+import qualified Test.HUnit as U++import qualified Test.Mangrove.Unit.Tokenize as Tokenize+++main :: IO ()+main = do+    tokenize <- Tokenize.tests+    results <- U.runTestTT $ U.TestList+        [ tokenize+        ]+    +    N.unless (U.errors results == 0 && U.failures results == 0) E.exitFailure
+ test/Test/Mangrove/Unit/Common.hs view
@@ -0,0 +1,23 @@+{-|+Description:    ++Copyright:      (c) 2020 Samuel May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      experimental+Portability:    portable+-}+module Test.Mangrove.Unit.Common+    ( dataFile+    ) where+++import Paths_mangrove++import System.FilePath ( (</>) )+++dataFile :: FilePath -> IO FilePath+dataFile f = getDataFileName $ dataDir </> f+  where dataDir = "test" </> "unit"
+ test/Test/Mangrove/Unit/Tokenize.hs view
@@ -0,0 +1,26 @@+{-|+Description:    ++Copyright:      (c) 2020 Samuel May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      experimental+Portability:    portable+-}+module Test.Mangrove.Unit.Tokenize+    ( tests+    ) where+++import qualified Test.HUnit as U++import Test.HUnit ( (~:) )++import qualified Test.Mangrove.Unit.Tokenize.Tag as Tag+++tests :: IO U.Test+tests = return $ "tokenization" ~: U.TestList+    [ Tag.tests+    ]
+ test/Test/Mangrove/Unit/Tokenize/Tag.hs view
@@ -0,0 +1,49 @@+{-|+Description:    ++Copyright:      (c) 2020 Samuel May+License:        MPL-2.0+Maintainer:     ag.eitilt@gmail.com++Stability:      experimental+Portability:    portable+-}+module Test.Mangrove.Unit.Tokenize.Tag+    ( tests+    ) where+++import qualified Data.ByteString.Char8 as BS.C+import qualified Data.Text as T+import qualified Data.Tuple.HT as U.HT++import qualified Test.HUnit as U++import Test.HUnit ( (~:), (~?=) )++import Web.Mangrove.Parse.Tokenize+++tests :: U.Test+tests = "UTF-8 encoding" ~: U.TestList+        [ "appropriate end tags allow re-entry" ~: U.TestList $ map stepBreakpoint+            [ RawTextState+            , RCDataState+            , ScriptDataState+            , ScriptDataEscapedState+            ]+        ]+++stepBreakpoint :: CurrentTokenizerState -> U.Test+stepBreakpoint state = show state ~:+    step' ~?= ([([], EndTag tag)], tokenState', BS.C.pack "</xmp>")+  where step = tokenizeStep tokenState $ BS.C.pack "</script></xmp>"+        step' = U.HT.mapSnd3 (tokenizerEncoding (Right $ Just Utf8)) $ step+        script = T.pack "script"+        tag = emptyTagParams+            { tagName = script+            }+        tokenState = tokenizerStartTag Nothing script $+            tokenizerMode state defaultTokenizerState+        tokenState' = tokenizerMode DataState tokenState