diff --git a/LICENSE b/LICENSE new file mode 100644 --- /dev/null +++ b/LICENSE @@ -0,0 +1,675 @@ + GNU GENERAL PUBLIC LICENSE + Version 3, 29 June 2007 + + Copyright (C) 2007 Free Software Foundation, Inc. + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The GNU General Public License is a free, copyleft license for +software and other kinds of works. + + The licenses for most software and other practical works are designed +to take away your freedom to share and change the works. By contrast, +the GNU General Public License is intended to guarantee your freedom to +share and change all versions of a program--to make sure it remains free +software for all its users. We, the Free Software Foundation, use the +GNU General Public License for most of our software; it applies also to +any other work released this way by its authors. You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +them if you wish), that you receive source code or can get it if you +want it, that you can change the software or use pieces of it in new +free programs, and that you know you can do these things. + + To protect your rights, we need to prevent others from denying you +these rights or asking you to surrender the rights. Therefore, you have +certain responsibilities if you distribute copies of the software, or if +you modify it: responsibilities to respect the freedom of others. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must pass on to the recipients the same +freedoms that you received. You must make sure that they, too, receive +or can get the source code. And you must show them these terms so they +know their rights. + + Developers that use the GNU GPL protect your rights with two steps: +(1) assert copyright on the software, and (2) offer you this License +giving you legal permission to copy, distribute and/or modify it. + + For the developers' and authors' protection, the GPL clearly explains +that there is no warranty for this free software. For both users' and +authors' sake, the GPL requires that modified versions be marked as +changed, so that their problems will not be attributed erroneously to +authors of previous versions. + + Some devices are designed to deny users access to install or run +modified versions of the software inside them, although the manufacturer +can do so. This is fundamentally incompatible with the aim of +protecting users' freedom to change the software. The systematic +pattern of such abuse occurs in the area of products for individuals to +use, which is precisely where it is most unacceptable. Therefore, we +have designed this version of the GPL to prohibit the practice for those +products. If such problems arise substantially in other domains, we +stand ready to extend this provision to those domains in future versions +of the GPL, as needed to protect the freedom of users. + + Finally, every program is threatened constantly by software patents. +States should not allow patents to restrict development and use of +software on general-purpose computers, but in those that do, we wish to +avoid the special danger that patents applied to a free program could +make it effectively proprietary. To prevent this, the GPL assures that +patents cannot be used to render the program non-free. + + The precise terms and conditions for copying, distribution and +modification follow. + + TERMS AND CONDITIONS + + 0. Definitions. + + "This License" refers to version 3 of the GNU General Public License. + + "Copyright" also means copyright-like laws that apply to other kinds of +works, such as semiconductor masks. + + "The Program" refers to any copyrightable work licensed under this +License. Each licensee is addressed as "you". "Licensees" and +"recipients" may be individuals or organizations. + + To "modify" a work means to copy from or adapt all or part of the work +in a fashion requiring copyright permission, other than the making of an +exact copy. The resulting work is called a "modified version" of the +earlier work or a work "based on" the earlier work. + + A "covered work" means either the unmodified Program or a work based +on the Program. + + To "propagate" a work means to do anything with it that, without +permission, would make you directly or secondarily liable for +infringement under applicable copyright law, except executing it on a +computer or modifying a private copy. Propagation includes copying, +distribution (with or without modification), making available to the +public, and in some countries other activities as well. + + To "convey" a work means any kind of propagation that enables other +parties to make or receive copies. Mere interaction with a user through +a computer network, with no transfer of a copy, is not conveying. + + An interactive user interface displays "Appropriate Legal Notices" +to the extent that it includes a convenient and prominently visible +feature that (1) displays an appropriate copyright notice, and (2) +tells the user that there is no warranty for the work (except to the +extent that warranties are provided), that licensees may convey the +work under this License, and how to view a copy of this License. If +the interface presents a list of user commands or options, such as a +menu, a prominent item in the list meets this criterion. + + 1. Source Code. + + The "source code" for a work means the preferred form of the work +for making modifications to it. "Object code" means any non-source +form of a work. + + A "Standard Interface" means an interface that either is an official +standard defined by a recognized standards body, or, in the case of +interfaces specified for a particular programming language, one that +is widely used among developers working in that language. + + The "System Libraries" of an executable work include anything, other +than the work as a whole, that (a) is included in the normal form of +packaging a Major Component, but which is not part of that Major +Component, and (b) serves only to enable use of the work with that +Major Component, or to implement a Standard Interface for which an +implementation is available to the public in source code form. A +"Major Component", in this context, means a major essential component +(kernel, window system, and so on) of the specific operating system +(if any) on which the executable work runs, or a compiler used to +produce the work, or an object code interpreter used to run it. + + The "Corresponding Source" for a work in object code form means all +the source code needed to generate, install, and (for an executable +work) run the object code and to modify the work, including scripts to +control those activities. However, it does not include the work's +System Libraries, or general-purpose tools or generally available free +programs which are used unmodified in performing those activities but +which are not part of the work. For example, Corresponding Source +includes interface definition files associated with source files for +the work, and the source code for shared libraries and dynamically +linked subprograms that the work is specifically designed to require, +such as by intimate data communication or control flow between those +subprograms and other parts of the work. + + The Corresponding Source need not include anything that users +can regenerate automatically from other parts of the Corresponding +Source. + + The Corresponding Source for a work in source code form is that +same work. + + 2. Basic Permissions. + + All rights granted under this License are granted for the term of +copyright on the Program, and are irrevocable provided the stated +conditions are met. This License explicitly affirms your unlimited +permission to run the unmodified Program. The output from running a +covered work is covered by this License only if the output, given its +content, constitutes a covered work. This License acknowledges your +rights of fair use or other equivalent, as provided by copyright law. + + You may make, run and propagate covered works that you do not +convey, without conditions so long as your license otherwise remains +in force. You may convey covered works to others for the sole purpose +of having them make modifications exclusively for you, or provide you +with facilities for running those works, provided that you comply with +the terms of this License in conveying all material for which you do +not control copyright. Those thus making or running the covered works +for you must do so exclusively on your behalf, under your direction +and control, on terms that prohibit them from making any copies of +your copyrighted material outside their relationship with you. + + Conveying under any other circumstances is permitted solely under +the conditions stated below. Sublicensing is not allowed; section 10 +makes it unnecessary. + + 3. Protecting Users' Legal Rights From Anti-Circumvention Law. + + No covered work shall be deemed part of an effective technological +measure under any applicable law fulfilling obligations under article +11 of the WIPO copyright treaty adopted on 20 December 1996, or +similar laws prohibiting or restricting circumvention of such +measures. + + When you convey a covered work, you waive any legal power to forbid +circumvention of technological measures to the extent such circumvention +is effected by exercising rights under this License with respect to +the covered work, and you disclaim any intention to limit operation or +modification of the work as a means of enforcing, against the work's +users, your or third parties' legal rights to forbid circumvention of +technological measures. + + 4. Conveying Verbatim Copies. + + You may convey verbatim copies of the Program's source code as you +receive it, in any medium, provided that you conspicuously and +appropriately publish on each copy an appropriate copyright notice; +keep intact all notices stating that this License and any +non-permissive terms added in accord with section 7 apply to the code; +keep intact all notices of the absence of any warranty; and give all +recipients a copy of this License along with the Program. + + You may charge any price or no price for each copy that you convey, +and you may offer support or warranty protection for a fee. + + 5. Conveying Modified Source Versions. + + You may convey a work based on the Program, or the modifications to +produce it from the Program, in the form of source code under the +terms of section 4, provided that you also meet all of these conditions: + + a) The work must carry prominent notices stating that you modified + it, and giving a relevant date. + + b) The work must carry prominent notices stating that it is + released under this License and any conditions added under section + 7. This requirement modifies the requirement in section 4 to + "keep intact all notices". + + c) You must license the entire work, as a whole, under this + License to anyone who comes into possession of a copy. This + License will therefore apply, along with any applicable section 7 + additional terms, to the whole of the work, and all its parts, + regardless of how they are packaged. This License gives no + permission to license the work in any other way, but it does not + invalidate such permission if you have separately received it. + + d) If the work has interactive user interfaces, each must display + Appropriate Legal Notices; however, if the Program has interactive + interfaces that do not display Appropriate Legal Notices, your + work need not make them do so. + + A compilation of a covered work with other separate and independent +works, which are not by their nature extensions of the covered work, +and which are not combined with it such as to form a larger program, +in or on a volume of a storage or distribution medium, is called an +"aggregate" if the compilation and its resulting copyright are not +used to limit the access or legal rights of the compilation's users +beyond what the individual works permit. Inclusion of a covered work +in an aggregate does not cause this License to apply to the other +parts of the aggregate. + + 6. Conveying Non-Source Forms. + + You may convey a covered work in object code form under the terms +of sections 4 and 5, provided that you also convey the +machine-readable Corresponding Source under the terms of this License, +in one of these ways: + + a) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by the + Corresponding Source fixed on a durable physical medium + customarily used for software interchange. + + b) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by a + written offer, valid for at least three years and valid for as + long as you offer spare parts or customer support for that product + model, to give anyone who possesses the object code either (1) a + copy of the Corresponding Source for all the software in the + product that is covered by this License, on a durable physical + medium customarily used for software interchange, for a price no + more than your reasonable cost of physically performing this + conveying of source, or (2) access to copy the + Corresponding Source from a network server at no charge. + + c) Convey individual copies of the object code with a copy of the + written offer to provide the Corresponding Source. This + alternative is allowed only occasionally and noncommercially, and + only if you received the object code with such an offer, in accord + with subsection 6b. + + d) Convey the object code by offering access from a designated + place (gratis or for a charge), and offer equivalent access to the + Corresponding Source in the same way through the same place at no + further charge. You need not require recipients to copy the + Corresponding Source along with the object code. If the place to + copy the object code is a network server, the Corresponding Source + may be on a different server (operated by you or a third party) + that supports equivalent copying facilities, provided you maintain + clear directions next to the object code saying where to find the + Corresponding Source. Regardless of what server hosts the + Corresponding Source, you remain obligated to ensure that it is + available for as long as needed to satisfy these requirements. + + e) Convey the object code using peer-to-peer transmission, provided + you inform other peers where the object code and Corresponding + Source of the work are being offered to the general public at no + charge under subsection 6d. + + A separable portion of the object code, whose source code is excluded +from the Corresponding Source as a System Library, need not be +included in conveying the object code work. + + A "User Product" is either (1) a "consumer product", which means any +tangible personal property which is normally used for personal, family, +or household purposes, or (2) anything designed or sold for incorporation +into a dwelling. In determining whether a product is a consumer product, +doubtful cases shall be resolved in favor of coverage. For a particular +product received by a particular user, "normally used" refers to a +typical or common use of that class of product, regardless of the status +of the particular user or of the way in which the particular user +actually uses, or expects or is expected to use, the product. A product +is a consumer product regardless of whether the product has substantial +commercial, industrial or non-consumer uses, unless such uses represent +the only significant mode of use of the product. + + "Installation Information" for a User Product means any methods, +procedures, authorization keys, or other information required to install +and execute modified versions of a covered work in that User Product from +a modified version of its Corresponding Source. The information must +suffice to ensure that the continued functioning of the modified object +code is in no case prevented or interfered with solely because +modification has been made. + + If you convey an object code work under this section in, or with, or +specifically for use in, a User Product, and the conveying occurs as +part of a transaction in which the right of possession and use of the +User Product is transferred to the recipient in perpetuity or for a +fixed term (regardless of how the transaction is characterized), the +Corresponding Source conveyed under this section must be accompanied +by the Installation Information. But this requirement does not apply +if neither you nor any third party retains the ability to install +modified object code on the User Product (for example, the work has +been installed in ROM). + + The requirement to provide Installation Information does not include a +requirement to continue to provide support service, warranty, or updates +for a work that has been modified or installed by the recipient, or for +the User Product in which it has been modified or installed. Access to a +network may be denied when the modification itself materially and +adversely affects the operation of the network or violates the rules and +protocols for communication across the network. + + Corresponding Source conveyed, and Installation Information provided, +in accord with this section must be in a format that is publicly +documented (and with an implementation available to the public in +source code form), and must require no special password or key for +unpacking, reading or copying. + + 7. Additional Terms. + + "Additional permissions" are terms that supplement the terms of this +License by making exceptions from one or more of its conditions. +Additional permissions that are applicable to the entire Program shall +be treated as though they were included in this License, to the extent +that they are valid under applicable law. If additional permissions +apply only to part of the Program, that part may be used separately +under those permissions, but the entire Program remains governed by +this License without regard to the additional permissions. + + When you convey a copy of a covered work, you may at your option +remove any additional permissions from that copy, or from any part of +it. (Additional permissions may be written to require their own +removal in certain cases when you modify the work.) You may place +additional permissions on material, added by you to a covered work, +for which you have or can give appropriate copyright permission. + + Notwithstanding any other provision of this License, for material you +add to a covered work, you may (if authorized by the copyright holders of +that material) supplement the terms of this License with terms: + + a) Disclaiming warranty or limiting liability differently from the + terms of sections 15 and 16 of this License; or + + b) Requiring preservation of specified reasonable legal notices or + author attributions in that material or in the Appropriate Legal + Notices displayed by works containing it; or + + c) Prohibiting misrepresentation of the origin of that material, or + requiring that modified versions of such material be marked in + reasonable ways as different from the original version; or + + d) Limiting the use for publicity purposes of names of licensors or + authors of the material; or + + e) Declining to grant rights under trademark law for use of some + trade names, trademarks, or service marks; or + + f) Requiring indemnification of licensors and authors of that + material by anyone who conveys the material (or modified versions of + it) with contractual assumptions of liability to the recipient, for + any liability that these contractual assumptions directly impose on + those licensors and authors. + + All other non-permissive additional terms are considered "further +restrictions" within the meaning of section 10. If the Program as you +received it, or any part of it, contains a notice stating that it is +governed by this License along with a term that is a further +restriction, you may remove that term. If a license document contains +a further restriction but permits relicensing or conveying under this +License, you may add to a covered work material governed by the terms +of that license document, provided that the further restriction does +not survive such relicensing or conveying. + + If you add terms to a covered work in accord with this section, you +must place, in the relevant source files, a statement of the +additional terms that apply to those files, or a notice indicating +where to find the applicable terms. + + Additional terms, permissive or non-permissive, may be stated in the +form of a separately written license, or stated as exceptions; +the above requirements apply either way. + + 8. Termination. + + You may not propagate or modify a covered work except as expressly +provided under this License. Any attempt otherwise to propagate or +modify it is void, and will automatically terminate your rights under +this License (including any patent licenses granted under the third +paragraph of section 11). + + However, if you cease all violation of this License, then your +license from a particular copyright holder is reinstated (a) +provisionally, unless and until the copyright holder explicitly and +finally terminates your license, and (b) permanently, if the copyright +holder fails to notify you of the violation by some reasonable means +prior to 60 days after the cessation. + + Moreover, your license from a particular copyright holder is +reinstated permanently if the copyright holder notifies you of the +violation by some reasonable means, this is the first time you have +received notice of violation of this License (for any work) from that +copyright holder, and you cure the violation prior to 30 days after +your receipt of the notice. + + Termination of your rights under this section does not terminate the +licenses of parties who have received copies or rights from you under +this License. If your rights have been terminated and not permanently +reinstated, you do not qualify to receive new licenses for the same +material under section 10. + + 9. Acceptance Not Required for Having Copies. + + You are not required to accept this License in order to receive or +run a copy of the Program. Ancillary propagation of a covered work +occurring solely as a consequence of using peer-to-peer transmission +to receive a copy likewise does not require acceptance. However, +nothing other than this License grants you permission to propagate or +modify any covered work. These actions infringe copyright if you do +not accept this License. Therefore, by modifying or propagating a +covered work, you indicate your acceptance of this License to do so. + + 10. Automatic Licensing of Downstream Recipients. + + Each time you convey a covered work, the recipient automatically +receives a license from the original licensors, to run, modify and +propagate that work, subject to this License. You are not responsible +for enforcing compliance by third parties with this License. + + An "entity transaction" is a transaction transferring control of an +organization, or substantially all assets of one, or subdividing an +organization, or merging organizations. If propagation of a covered +work results from an entity transaction, each party to that +transaction who receives a copy of the work also receives whatever +licenses to the work the party's predecessor in interest had or could +give under the previous paragraph, plus a right to possession of the +Corresponding Source of the work from the predecessor in interest, if +the predecessor has it or can get it with reasonable efforts. + + You may not impose any further restrictions on the exercise of the +rights granted or affirmed under this License. For example, you may +not impose a license fee, royalty, or other charge for exercise of +rights granted under this License, and you may not initiate litigation +(including a cross-claim or counterclaim in a lawsuit) alleging that +any patent claim is infringed by making, using, selling, offering for +sale, or importing the Program or any portion of it. + + 11. Patents. + + A "contributor" is a copyright holder who authorizes use under this +License of the Program or a work on which the Program is based. The +work thus licensed is called the contributor's "contributor version". + + A contributor's "essential patent claims" are all patent claims +owned or controlled by the contributor, whether already acquired or +hereafter acquired, that would be infringed by some manner, permitted +by this License, of making, using, or selling its contributor version, +but do not include claims that would be infringed only as a +consequence of further modification of the contributor version. For +purposes of this definition, "control" includes the right to grant +patent sublicenses in a manner consistent with the requirements of +this License. + + Each contributor grants you a non-exclusive, worldwide, royalty-free +patent license under the contributor's essential patent claims, to +make, use, sell, offer for sale, import and otherwise run, modify and +propagate the contents of its contributor version. + + In the following three paragraphs, a "patent license" is any express +agreement or commitment, however denominated, not to enforce a patent +(such as an express permission to practice a patent or covenant not to +sue for patent infringement). To "grant" such a patent license to a +party means to make such an agreement or commitment not to enforce a +patent against the party. + + If you convey a covered work, knowingly relying on a patent license, +and the Corresponding Source of the work is not available for anyone +to copy, free of charge and under the terms of this License, through a +publicly available network server or other readily accessible means, +then you must either (1) cause the Corresponding Source to be so +available, or (2) arrange to deprive yourself of the benefit of the +patent license for this particular work, or (3) arrange, in a manner +consistent with the requirements of this License, to extend the patent +license to downstream recipients. "Knowingly relying" means you have +actual knowledge that, but for the patent license, your conveying the +covered work in a country, or your recipient's use of the covered work +in a country, would infringe one or more identifiable patents in that +country that you have reason to believe are valid. + + If, pursuant to or in connection with a single transaction or +arrangement, you convey, or propagate by procuring conveyance of, a +covered work, and grant a patent license to some of the parties +receiving the covered work authorizing them to use, propagate, modify +or convey a specific copy of the covered work, then the patent license +you grant is automatically extended to all recipients of the covered +work and works based on it. + + A patent license is "discriminatory" if it does not include within +the scope of its coverage, prohibits the exercise of, or is +conditioned on the non-exercise of one or more of the rights that are +specifically granted under this License. You may not convey a covered +work if you are a party to an arrangement with a third party that is +in the business of distributing software, under which you make payment +to the third party based on the extent of your activity of conveying +the work, and under which the third party grants, to any of the +parties who would receive the covered work from you, a discriminatory +patent license (a) in connection with copies of the covered work +conveyed by you (or copies made from those copies), or (b) primarily +for and in connection with specific products or compilations that +contain the covered work, unless you entered into that arrangement, +or that patent license was granted, prior to 28 March 2007. + + Nothing in this License shall be construed as excluding or limiting +any implied license or other defenses to infringement that may +otherwise be available to you under applicable patent law. + + 12. No Surrender of Others' Freedom. + + If conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot convey a +covered work so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you may +not convey it at all. For example, if you agree to terms that obligate you +to collect a royalty for further conveying from those to whom you convey +the Program, the only way you could satisfy both those terms and this +License would be to refrain entirely from conveying the Program. + + 13. Use with the GNU Affero General Public License. + + Notwithstanding any other provision of this License, you have +permission to link or combine any covered work with a work licensed +under version 3 of the GNU Affero General Public License into a single +combined work, and to convey the resulting work. The terms of this +License will continue to apply to the part which is the covered work, +but the special requirements of the GNU Affero General Public License, +section 13, concerning interaction through a network will apply to the +combination as such. + + 14. Revised Versions of this License. + + The Free Software Foundation may publish revised and/or new versions of +the GNU General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + + Each version is given a distinguishing version number. If the +Program specifies that a certain numbered version of the GNU General +Public License "or any later version" applies to it, you have the +option of following the terms and conditions either of that numbered +version or of any later version published by the Free Software +Foundation. If the Program does not specify a version number of the +GNU General Public License, you may choose any version ever published +by the Free Software Foundation. + + If the Program specifies that a proxy can decide which future +versions of the GNU General Public License can be used, that proxy's +public statement of acceptance of a version permanently authorizes you +to choose that version for the Program. + + Later license versions may give you additional or different +permissions. However, no additional obligations are imposed on any +author or copyright holder as a result of your choosing to follow a +later version. + + 15. Disclaimer of Warranty. + + THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY +APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT +HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY +OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, +THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM +IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF +ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + + 16. Limitation of Liability. + + IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS +THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY +GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE +USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF +DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD +PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), +EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF +SUCH DAMAGES. + + 17. Interpretation of Sections 15 and 16. + + If the disclaimer of warranty and limitation of liability provided +above cannot be given local legal effect according to their terms, +reviewing courts shall apply local law that most closely approximates +an absolute waiver of all civil liability in connection with the +Program, unless a warranty or assumption of liability accompanies a +copy of the Program in return for a fee. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +state the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + + Copyright (C) + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see . + +Also add information on how to contact you by electronic and paper mail. + + If the program does terminal interaction, make it output a short +notice like this when it starts in an interactive mode: + + Copyright (C) + This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, your program's commands +might be different; for a GUI interface, you would use an "about box". + + You should also get your employer (if you work as a programmer) or school, +if any, to sign a "copyright disclaimer" for the program, if necessary. +For more information on this, and how to apply and follow the GNU GPL, see +. + + The GNU General Public License does not permit incorporating your program +into proprietary programs. If your program is a subroutine library, you +may consider it more useful to permit linking proprietary applications with +the library. If this is what you want to do, use the GNU Lesser General +Public License instead of this License. But first, please read +. + diff --git a/Linguistics/Bigram.hs b/Linguistics/Bigram.hs new file mode 100644 --- /dev/null +++ b/Linguistics/Bigram.hs @@ -0,0 +1,131 @@ + +-- | Map between 'String's that represent characters and their 'Int'-based +-- representation. +-- +-- NOTE filtering the scores list and creating a single bigram map takes about +-- 70 seconds. +-- +-- NOTE A single bigram map costs around 160 MByte ram. This includes the +-- overhead for actually storing the bigrams once (creating pointers instead of +-- multiple copied 'Bigram' data structures. + +module Linguistics.Bigram where + +import Control.Applicative +import Control.Arrow +import Control.DeepSeq +import Control.Lens +import Data.Attoparsec.ByteString.Lazy (()) +import Data.ByteString (ByteString) +import Data.Function +import Data.Hashable +import Data.Interned +import Data.List +import Data.Strict.Tuple +import GHC.Generics (Generic) +import qualified Data.Attoparsec.ByteString as AB +import qualified Data.Attoparsec.ByteString.Char8 as AB hiding (takeWhile1) +import qualified Data.Attoparsec.ByteString.Lazy as ABL +import qualified Data.ByteString as B +import qualified Data.ByteString.Lazy as BL hiding (unpack) +import qualified Data.ByteString.Lazy.Char8 as BL hiding (readFile) +import qualified Data.ByteString.Short as BS +import qualified Data.HashMap.Strict as HM +import qualified Data.Map.Strict as M +import qualified Data.Set as S +import qualified Data.Stringable as SA + +import NLP.Text.BTI + + + +data Bigram = Bigram + { peekChar :: {-# UNPACK #-} !BTI + , hitChar :: {-# UNPACK #-} !BTI + } + deriving (Show,Eq,Ord,Generic) + +instance Hashable Bigram where + hashWithSalt s (Bigram p h) = hashWithSalt s (p,h) -- (uninternMultiChar p, uninternMultiChar h) + hash (Bigram p h) = hash (hash p , hash h) + {-# Inline hashWithSalt #-} + {-# Inline hash #-} + +instance NFData Bigram where + rnf !(Bigram a b) = () + +instance Hashable (Pair Int Int) where + hashWithSalt s (a:!:b) = hashWithSalt s (a,b) + {-# Inline hashWithSalt #-} + +-- | Try to read the first line to figure out if there is a default score there + +withDefault :: Double -> [BL.ByteString] -> (Double,[BL.ByteString]) +withDefault d [] = (d,[]) +withDefault d (x:xs) + | [(rd,"")] <- readsPrec 0 (BL.unpack x) = (rd,xs) + | otherwise = (d,(x:xs)) + +parseLine l = case ABL.eitherResult (ABL.parse go l) of + Left err -> error err + Right p -> force p + where + go = (,,,,) <$> lang <*> lang <*> bigram <*> bigram <*> score "go" + lang = wrd "lang" + bigram = Bigram <$> wrd <*> wrd "bigram" + score = AB.double "score" + wrd = SA.fromByteString <$> AB.takeWhile1 (not . AB.isHorizontalSpace) <* AB.space + +type Lang = BTI +type Line = (Lang, Lang, Bigram, Bigram, Double) +type Scores = HM.HashMap (Bigram:!:Bigram) Double + +data Mapping = Mapping + { bigrams :: !(M.Map Bigram Bigram) + , lliid :: !(M.Map (Lang:!:Lang) Scores) + } + deriving (Show) + +instance Hashable (Pair Bigram Bigram) where + hashWithSalt s (a:!:b) = hashWithSalt s (a,b) + +lines2mapping :: [Line] -> Mapping +lines2mapping = foldl' mkMapping emptyMapping . concatMap dupGroup . groupBy ((==) `on` ((^._1) &&& (^._2))) where + dupGroup ls@(l:_) + | l^._1 == l^._2 = [ls] + | otherwise = [ls,ls'] + where ls' = map (\(l1,l2,b1,b2,d) -> (l2,l1,b2,b1,d)) . filter (\l -> l^._1 /= l^._2) $ ls + +emptyMapping = let b = Bigram "" "" + in Mapping (M.singleton b b) M.empty + +mkMapping :: Mapping -> [Line] -> Mapping +mkMapping !m [] = m +mkMapping !(Mapping bs ll) xs@(x:_) + | otherwise = Mapping bs' ll' + where + nom = filter (`M.notMember` bs) $ map (^._3) xs ++ map (^._4) xs + bs' = bs `M.union` (M.fromList $ map (\a -> (a,a)) nom) + ll' = M.insertWith HM.union (x^._1 :!: x^._2) ys ll + ys :: Scores + ys = HM.fromList + [ ((k1:!:k2),d) + | y <- xs + , let k1 = bs' M.! (y^._3) + , let k2 = bs' M.! (y^._4) + , let d = y ^._5 + ] + +-- | Given a set of acceptable languages, a default score, and the lazy +-- bytestring of scores, create the 'Mapping' of languages and scores. + +generateLookups :: S.Set BTI -> Double -> BL.ByteString -> Mapping +generateLookups langs wd b = lines2mapping xs where + (d,ls) = withDefault wd $ BL.lines b + xs = filter inLangSet $ map parseLine ls + inLangSet l + | S.null langs = True + | (l^._1) `S.member` langs + && (l^._2) `S.member` langs = True + | otherwise = False + diff --git a/Linguistics/Common.hs b/Linguistics/Common.hs new file mode 100644 --- /dev/null +++ b/Linguistics/Common.hs @@ -0,0 +1,90 @@ + +-- | Some common functions and things that are not of immediate importance to +-- understand the algorithms. + +module Linguistics.Common where + +import Control.Monad (forM_) +import Data.ByteString (ByteString) +import Data.Char +import Data.List (transpose,reverse) +import qualified Data.ByteString.Short as S +import qualified Data.Text as T +import qualified Data.Text.Lazy as TL +import qualified Data.Text.Lazy.IO as TL +import qualified Data.Text.Lazy.Builder as TL +import Data.Text (Text) +import qualified Data.Text.Format as TF +import qualified Data.Text.Encoding as T +import Text.Printf +import Data.Stringable (toString) +import GHC.IO.Handle + +import NLP.Text.BTI + + + +type IMCp = (BTI, BTI) + +-- | Actually align something prettily + +alignPretty :: [[IMCp]] -> [String] +alignPretty xss = map concat . transpose . map (\xs -> map (f xs) xs) . transpose . map reverse $ xss where + f zs x = printAligned x zs + +-- | Prettyprint ``characters'', which are actually small bytestrings. + +printAligned = printAlignedPad ' ' + +-- | Print with special padding character + +printAlignedPad :: Char -> IMCp -> [IMCp] -> String +printAlignedPad p (_,c) zs = printf " %s%s" (replicate pad p) (toUtf8String c) where + pad :: Int + pad = (1+) . maximum $ 0 : map (\(_,x) -> printLength x - printLength c) zs + +-- | Length in /printed characters/ of an UTF8 string wrapped as a 'ByteString' +-- +-- NOTE 'isMark' selects unicode symbols that modify a character, thereby not +-- increasing the length of the /printed/ string. + +printLength :: BTI -> Int +printLength = length . filter isAN . toUtf8String where + isAN c = not (isMark c) -- isAlphaNum c || c `elem` [ '\\', '\'', '^', '$', '-', '\'' ] + + +{- + } where prnt x z = let pad = max 0 (length (filter isAN $ pp z) - length (filter isAN $ pp x)) + in printf " %s%s" (replicate pad ' ') (pp x) + ds x = ' ' : replicate (length $ filter isAN $ pp x) '-' + isAN c = isAlphaNum c || c `elem` [ '\\', '\'' ] +-} + + +toUtf8String :: BTI -> String +toUtf8String = toString -- T.unpack . T.decodeUtf8 . conv +{-# INLINE toUtf8String #-} + +buildLines :: [[Text]] -> TL.Builder +buildLines xss = s where + n = (1+) . maximum $ 1 : (map (T.length . T.filter (not . isMark)) . concat $ xss) + yss = transpose xss + fmt = "%" ++ show n ++ "s" + s = mconcat [ (mconcat $ map (TF.left n ' ') ys) `mappend` "\n" + | ys <- yss ] + +printLines :: Handle -> [[Text]] -> IO () +printLines hndl xss = do + let n = (1+) . maximum $ 1 : (map (T.length . T.filter (not . isMark)) . concat $ xss) + let yss = transpose xss + let fmt = "%" ++ show n ++ "s" + let s = mconcat [ (mconcat $ map (TF.left n ' ') ys) `mappend` "\n" + | ys <- yss ] + TL.hPutStrLn hndl $ TL.toLazyText s + --forM_ yss $ \ys -> do + -- forM_ ys $ \y -> hPrintf hndl fmt (T.unpack y) + -- hPrintf hndl "\n" + +--conv = S.fromShort . getMultiChar . uninternMultiChar +--{-# INLINE conv #-} + diff --git a/Linguistics/TwoWay/Bigram.hs b/Linguistics/TwoWay/Bigram.hs new file mode 100644 --- /dev/null +++ b/Linguistics/TwoWay/Bigram.hs @@ -0,0 +1,167 @@ +{-# LANGUAGE MultiWayIf #-} +{-# LANGUAGE OverloadedStrings #-} +{-# LANGUAGE TypeOperators #-} +{-# LANGUAGE BangPatterns #-} +{-# LANGUAGE NoMonomorphismRestriction #-} + +{-# OPTIONS_GHC -fno-liberate-case #-} + +module Linguistics.TwoWay.Bigram where + +import Data.ByteString.Char8 (ByteString) +import Data.FMList (FMList) +import Data.Sequence (Seq) +import Data.Strict.Tuple (Pair (..)) +import Data.Stringable (toString,toText) +import Data.Text (Text,pack) +import Data.Vector.Fusion.Util (Id(..)) +import Data.Vector.Unboxed (Vector) +import qualified Data.ByteString.Char8 as B +import qualified Data.HashMap.Strict as HM +import qualified Data.List as L +import qualified Data.Vector.Fusion.Stream.Monadic as SM +import qualified Data.Vector.Unboxed as VU +import Text.Printf +import qualified Data.Text.Format as TF + +import ADP.Fusion +import Data.PrimitiveArray +import DP.Alignment.Global.Tapes2 +import NLP.Scoring.SimpleUnigram +import NLP.Text.BTI + +import Linguistics.Common +import Linguistics.Bigram + + + +type IMC = BTI +type SigT m x r = SigGlobal m x r IMCp IMCp + + + +sScore :: Monad m => Double -> Double -> Scores -> SigT m Double Double +sScore dS gapopen s = SigGlobal + { delin = \ww (Z:.c :._ ) -> ww + gapopen + , indel = \ww (Z:._ :.c ) -> ww + gapopen + , align = \ww (Z:.(lp,l):.(up,u)) -> ww + lkup up u lp l + , done = const 0 + , h = SM.foldl' max (-888888) + } where + lkup mc' c nd' d = {-# SCC "lkup" #-} HM.lookupDefault dS (Bigram mc' c :!: Bigram nd' d) s + {-# INLINE lkup #-} +{-# INLINE sScore #-} +{- +sScore dS gapOpen s = SigGlobal + { indel = \ww (Z:.():.(mc,c)) -> if | c=="$" -> -500000 + | otherwise -> ww + gapOpen + , delin = \ww (Z:.(mc,c):.()) -> if | c=="$" -> -500000 + | otherwise -> ww + gapOpen + , align = \ww (Z:.(mc,c):.(nd,d)) -> if | c=="^" && d=="$" -> -500000 + | c=="$" && d=="^" -> -500000 + | otherwise -> case (mc,nd) of + (Nothing , Nothing ) -> 0 + (Just mc' , Just nd') -> ww + lkup mc' c nd' d + _ -> -500000 + , done = const 0 + , h = S.foldl' max (-500000) + } where + lkup mc' c nd' d = maybe dS id . unsafePerformIO $ H.lookup s (Bigram mc' c :!: Bigram nd' d) + {-# INLINE lkup #-} +{-# INLINE sScore #-} +-} + +sBacktrack :: Monad m => SigT m (FMList (IMCp,IMCp)) [FMList (IMCp,IMCp)] +sBacktrack = backtrack ("-","-") ("-","-") +{-# Inline sBacktrack #-} + +sBacktrackFun :: Monad m => Double -> Double -> Scores -> SigT m (FMList [Text]) [FMList [Text]] +sBacktrackFun defS go sco = backtrackFun f g ("-","-") ("-","-") where + f cc@(mc',c) dd@(nd',d) = let z = HM.lookupDefault defS (Bigram mc' c :!: Bigram nd' d) sco + in [toText c,toText d, pack $ printf "%3.1f" z] + g (_ ,c) ( _ ,"-") = [toText c,"-", pack $ printf "%3.1f" go] + g (_, "-") (_,d) = ["-", toText d, pack $ printf "%3.1f" go] +{-# Inline sBacktrackFun #-} + +alignGlobal :: Double -> Double -> Scores -> Int -> Vector IMC -> Vector IMC -> (Double,[[[Text]]]) +alignGlobal ds gapopen scoring k i1' i2' = (d, take k bs) where -- . L.map runPrettyF . S.toList . unId $ axiom b) where + i1 = VU.zip i1' (VU.tail i1') ; i2 = VU.zip i2' (VU.tail i2') + n1 = VU.length i1 ; n2 = VU.length i2 + !(Z:.t) = alignGlobalForward ds gapopen scoring i1 i2 + d = unId $ axiom t + bs = alignGlobalBacktrack ds gapopen scoring i1 i2 t +{-# NoInline alignGlobal #-} + +alignGlobalForward :: Double -> Double -> Scores -> Vector IMCp -> Vector IMCp -> Z:.ITbl Id Unboxed (Z:.PointL I:.PointL I) Double +alignGlobalForward ds gapopen scoring i1 i2 = {-# SCC "ali_forw" #-} mutateTablesDefault $ + gGlobal (sScore ds gapopen scoring) + (ITbl 0 0 (Z:.EmptyOk:.EmptyOk) (fromAssocs (Z:.PointL 0:.PointL 0) (Z:.PointL n1:.PointL n2) (-999999) [])) + (chr i1) (chr i2) + where n1 = VU.length i1 + n2 = VU.length i2 +{-# NoInline alignGlobalForward #-} + +alignGlobalBacktrack :: Double -> Double -> Scores -> Vector IMCp -> Vector IMCp -> ITbl Id Unboxed (Z:.PointL I:.PointL I) Double -> [[[Text]]] +alignGlobalBacktrack ds gapopen scoring i1 i2 t = {-# SCC "ali_back" #-} L.map runBacktrack . unId $ axiom b + where (Z:.b) = gGlobal (sScore ds gapopen scoring <|| sBacktrackFun ds gapopen scoring) (toBacktrack t (undefined :: Id a -> Id a)) (chr i1) (chr i2) +{-# NoInline alignGlobalBacktrack #-} + +{- +-- | Backtrack the alignment + +sAlign :: Monad m => STwoWay m Aligned (S.Stream m Aligned) (Maybe InternedMultiChar,InternedMultiChar) () +sAlign = STwoWay + { loop_step = \(w1,w2) (Z:.():.(_,c)) -> ( w1 ++ ["-"], w2 ++ [c] ) -- (w1++padd "" c, w2++prnt c "") + , step_loop = \(w1,w2) (Z:.(_,c):.()) -> ( w1 ++ [c] , w2 ++ ["-"] ) -- (w1++prnt c "", w2++padd "" c) + , step_step = \(w1,w2) (Z:.(_,a):.(_,b)) -> ( w1 ++ [a], w2 ++ [b] ) -- (w1++prnt a b,w2++prnt b a) + , nil_nil = const ([],[]) + , h = return . id + } where prnt x z = printAligned x [z] + padd x z = printAlignedPad '-' x [z] +{-# INLINE sAlign #-} + +-- | Wrap calculations + +twoWay dS gapOpen scores i1 i2 = (ws ! (Z:.pointL 0 n1:.pointL 0 n2), bt) where + ws = unsafePerformIO (twoWayFill dS gapOpen scores i1 i2) + n1 = V.length i1 + n2 = V.length i2 + bt = backtrack dS gapOpen scores i1 i2 ws +{-# NOINLINE twoWay #-} + +-- | Forward phase + +twoWayFill + :: Double + -> Double + -> Scores + -> V.Vector InternedMultiChar + -> V.Vector InternedMultiChar + -> IO (PA.Unboxed (Z:.PointL:.PointL) Double) +twoWayFill dS gapOpen scores i1 i2 = do + let n1 = V.length i1 + let n2 = V.length i2 + !t' <- newWithM (Z:.pointL 0 0:.pointL 0 0) (Z:.pointL 0 n1:.pointL 0 n2) 0 + let w = mTbl (Z:.EmptyT:.EmptyT) t' + fillTable2 $ gTwoWay (sScore dS gapOpen scores) w (chrLeft i1) (chrLeft i2) Empty Empty + freeze t' +{-# NOINLINE twoWayFill #-} + +backtrack + :: Double + -> Double + -> Scores + -> V.Vector InternedMultiChar + -> V.Vector InternedMultiChar + -> PA.Unboxed (Z:.PointL:.PointL) Double + -> [Aligned] +backtrack dS gapOpen scores i1 i2 tbl = unId . S.toList . unId $ g $ Z:.pointL 0 n1 :.pointL 0 n2 where + n1 = V.length i1 + n2 = V.length i2 + w :: DefBtTbl Id (Z:.PointL:.PointL) Double Aligned + w = btTbl (Z:.EmptyT:.EmptyT) tbl (g :: (Z:.PointL:.PointL) -> Id (S.Stream Id Aligned)) + (Z:.(_,g)) = gTwoWay (sScore dS gapOpen scores <** sAlign) w (chrLeft i1) (chrLeft i2) Empty Empty +{-# NOINLINE backtrack #-} + +-} + diff --git a/Linguistics/TwoWay/Simple.hs b/Linguistics/TwoWay/Simple.hs new file mode 100644 --- /dev/null +++ b/Linguistics/TwoWay/Simple.hs @@ -0,0 +1,150 @@ +{-# LANGUAGE RecordWildCards #-} +{-# LANGUAGE OverloadedStrings #-} +{-# LANGUAGE TypeOperators #-} +{-# LANGUAGE BangPatterns #-} +{-# LANGUAGE MultiWayIf #-} +{-# LANGUAGE NoMonomorphismRestriction #-} + +{-# OPTIONS_GHC -fno-liberate-case #-} + +module Linguistics.TwoWay.Simple where + +import Data.ByteString.Char8 (ByteString) +import Data.Vector.Fusion.Util (Id(..)) +import GHC.Exts +import qualified Data.ByteString.Char8 as B +import qualified Data.List as L +import qualified Data.Text as T +import qualified Data.Text.Encoding as T +import qualified Data.Vector.Fusion.Stream.Monadic as SM +--import qualified Data.Vector.Fusion.Stream as S +import qualified Data.Vector.Unboxed as VU +import Data.Vector.Unboxed (Vector) +import System.IO.Unsafe (unsafePerformIO) +import Data.Sequence (Seq) +import Data.FMList (FMList) +import Data.Text (Text) +import Data.Stringable (toText) + +import ADP.Fusion +import Data.PrimitiveArray +import DP.Alignment.Global.Tapes2 +import NLP.Scoring.SimpleUnigram +import NLP.Text.BTI + +import Linguistics.Common +--import Linguistics.TwoWay.Common + + +--type IMC = InternedMultiChar +type SigT m x r = SigGlobal m x r BTI BTI + +sScore :: Monad m => SimpleScoring -> SigT m Double Double +sScore ss@SimpleScoring{..} = SigGlobal + { delin = \ww (Z:.c:._) -> ww + gapScore + , indel = \ww (Z:._:.c) -> ww + gapScore + , align = \ww (Z:.l:.u) -> ww + scoreUnigram ss l u + , done = const 0 + , h = SM.foldl' max (-888888) + } +{-# Inline sScore #-} + +sBacktrack :: Monad m => SigT m (FMList (BTI,BTI)) [FMList (BTI,BTI)] +sBacktrack = backtrack "-" "-" +{-# Inline sBacktrack #-} + +-- | Create a backtracking function +-- +-- TODO includes scores as well? + +sBacktrackFun :: Monad m => SigT m (FMList [Text]) [FMList [Text]] +sBacktrackFun = backtrackFun f g "-" "-" where + f c d = [toText c, toText d] + g c d = [toText c, toText d] + +alignGlobal :: SimpleScoring -> Int -> Vector BTI -> Vector BTI -> (Double,[[[Text]]]) +alignGlobal scoring k i1 i2 = (d, take k bs) where + n1 = VU.length i1 ; n2 = VU.length i2 + !(Z:.t) = alignGlobalForward scoring i1 i2 + d = unId $ axiom t + bs = alignGlobalBacktrack scoring i1 i2 t +{-# NoInline alignGlobal #-} + +alignGlobalForward :: SimpleScoring -> Vector BTI -> Vector BTI -> Z:.ITbl Id Unboxed (Z:.PointL I:.PointL I) Double +alignGlobalForward scoring i1 i2 = {-# SCC "alignGlobalForward" #-} mutateTablesDefault $ + gGlobal (sScore scoring) + (ITbl 0 0 (Z:.EmptyOk:.EmptyOk) (fromAssocs (Z:.PointL 0:.PointL 0) (Z:.PointL n1:.PointL n2) (-999999) [])) + (chr i1) (chr i2) + where n1 = VU.length i1 + n2 = VU.length i2 +{-# NoInline alignGlobalForward #-} + +alignGlobalBacktrack :: SimpleScoring -> Vector BTI -> Vector BTI -> ITbl Id Unboxed (Z:.PointL I:.PointL I) Double -> [[[Text]]] +alignGlobalBacktrack scoring i1 i2 t = {-# SCC "alignGlobalBacktrack" #-} L.map runBacktrack . unId $ axiom b + where (Z:.b) = gGlobal (sScore scoring <|| sBacktrackFun) (toBacktrack t (undefined :: Id a -> Id a)) (chr i1) (chr i2) +{-# NoInline alignGlobalBacktrack #-} + +-- | Decoupling the forward phase for CORE observation. + +{- +alignGlobalForward :: Vector IMC -> Vector IMC -> Z:.(ITbl Id Unboxed (Z:.PointL:.PointL) Double) +alignGlobalForward i1 i2 = let n1 = VU.length i1; n2 = VU.length i2 in mutateTablesDefault $ + gGlobal score + (ITbl 0 0 (Z:.EmptyOk:.EmptyOk) (PA.fromAssocs (Z:.PointL 0:.PointL 0) (Z:.PointL n1:.PointL n2) (-999999) [])) + (chr i1) (chr i2) +{-# NoInline runNeedlemanWunschForward #-} +-} + + + +{- +sAlign :: Monad m => STwoWay m Aligned (S.Stream m Aligned) InternedMultiChar () +sAlign = STwoWay + { loop_step = \[w1,w2] (Z:.():.c) -> [ "-" : w1 , c : w2 ] + , step_loop = \[w1,w2] (Z:.c:.()) -> [ c : w1 , "-" : w2 ] + , step_step = \[w1,w2] (Z:.a:.b) -> [ a : w1 , b : w2 ] + , nil_nil = const [[],[]] + , h = return . id + } +{-# INLINE sAlign #-} + +--twoWay :: VU.Vector Char -> VU.Vector Char -> [Double] -> Double -> V.Vector ByteString -> V.Vector ByteString -> (Double +twoWay simpleScoring i1 i2 = (ws ! (Z:.pointL 0 n1:.pointL 0 n2), bt) where + ws = unsafePerformIO $ twoWayFill simpleScoring i1 i2 + n1 = V.length i1 + n2 = V.length i2 + bt = backtrack simpleScoring i1 i2 ws +{-# NOINLINE twoWay #-} + +twoWayFill + :: SimpleScoring + -> V.Vector InternedMultiChar + -> V.Vector InternedMultiChar + -> IO (PA.Unboxed (Z:.PointL:.PointL) Double) +twoWayFill simpleScoring i1 i2 = do + let n1 = V.length i1 + let n2 = V.length i2 + !t' <- newWithM (Z:.pointL 0 0:.pointL 0 0) (Z:.pointL 0 n1:.pointL 0 n2) 0 + let w = mTblD (Z:.EmptyOk:.EmptyOk) t' + fillTable2 $ gTwoWay (sScore simpleScoring) w (chr i1) (chr i2) Empty Empty + freeze t' +{-# NOINLINE twoWayFill #-} + +backtrack + :: SimpleScoring + -> V.Vector InternedMultiChar + -> V.Vector InternedMultiChar + -> PA.Unboxed (Z:.PointL:.PointL) Double + -> [Aligned] +backtrack simpleScoring i1 i2 tbl = unId . S.toList . unId $ g $ Z:.pointL 0 n1 :.pointL 0 n2 where + n1 = V.length i1 + n2 = V.length i2 + w = btTblD (Z:.EmptyOk:.EmptyOk) tbl (g :: (Z:.PointL:.PointL) -> Id (S.Stream Id Aligned)) + (Z:.(_,g)) = gTwoWay (sScore simpleScoring <** sAlign) w (chr i1) (chr i2) Empty Empty +{-# NOINLINE backtrack #-} + +--test s = twoWay (VU.fromList "aeiou") (VU.fromList $ ['a' .. 'z'] L.\\ "aeiou") [3,1,1,0,0,-1] (-1) s' s' where +-- s' = V.fromList $ L.map (B.pack . (:[])) $ s + +-} + diff --git a/Linguistics/Word.hs b/Linguistics/Word.hs new file mode 100644 --- /dev/null +++ b/Linguistics/Word.hs @@ -0,0 +1,77 @@ +{-# LANGUAGE OverloadedStrings #-} +{-# LANGUAGE BangPatterns #-} + +-- | A single word in a language. Uses a 'MultiChar' encoding for the actual +-- characters. MultiChar encodings need to be decoded for printing on screen. + +module Linguistics.Word where + +import Control.Applicative +import Control.DeepSeq +import Data.ByteString (ByteString) +import Data.Interned +import Data.Interned.ByteString +import qualified Data.Attoparsec.ByteString as AB +import qualified Data.Attoparsec.ByteString.Char8 as AB hiding (takeWhile1) +import qualified Data.Attoparsec.ByteString.Lazy as ABL +import qualified Data.ByteString as B +import qualified Data.ByteString.Lazy as BL hiding (unpack) +import qualified Data.ByteString.Lazy.Char8 as BL hiding (readFile) +import qualified Data.ByteString.Short as S +import qualified Data.Vector.Unboxed as VU +import Prelude hiding (Word) +import Data.Stringable + +import NLP.Text.BTI + + + +-- | A single word we want to align to another word. It comes with an id (here +-- 9), the language name (which we intern), a word class (interned as well), +-- the length of the word (so that we don't have to check wordWord length and +-- check for word delims), and finally the word itself. Indivitual 'MultiChar' +-- characters are interned to reduce memory cost (and we might want to do stuff +-- with the Id's). +-- +-- 9 Albanian_Tosk 1.214 6 \' b a lʸ t ə +-- + +data Word = Word + { wordID :: {-# UNPACK #-} !Int + , wordClass :: {-# UNPACK #-} !BTI -- InternedByteString + , wordLang :: {-# UNPACK #-} !BTI -- InternedByteString + , wordLength :: {-# UNPACK #-} !Int + , wordWord :: {-# UNPACK #-} !(VU.Vector BTI) + } + deriving (Show,Eq,Ord) + +instance NFData Word where + rnf !(Word {}) = () + +parseWord :: BL.ByteString -> Word +parseWord w = case ABL.eitherResult (ABL.parse go w) of + Left err -> error err + Right p -> force p + where + go = Word <$> AB.decimal + <* AB.many1 AB.space + <*> (wW <$> wrd) + <*> (wW <$> wrd) + <*> AB.decimal + <* AB.many1 AB.space + <*> ((VU.fromList . map wW) <$> (AB.takeWhile1 (not . AB.isHorizontalSpace) `AB.sepBy` AB.space)) + wrd = AB.takeWhile1 (not . AB.isHorizontalSpace) <* AB.space + wW = fromByteString + +addWordDelims :: Word -> Word +addWordDelims w + | VU.length ww >= 2 && VU.head ww == "^" && VU.last ww == "$" = w + | otherwise = w { wordWord = "^" `VU.cons` wordWord w `VU.snoc` "$" } + where ww = wordWord w + +removeWordDelims :: Word -> Word +removeWordDelims w + | VU.length ww >= 2 && VU.head ww == "^" && VU.last ww == "$" = w { wordWord = VU.init . VU.tail $ wordWord w } + | otherwise = w + where ww = wordWord w + diff --git a/README.md b/README.md new file mode 100644 --- /dev/null +++ b/README.md @@ -0,0 +1,177 @@ +[![Build Status](https://travis-ci.org/choener/WordAlignment.svg?branch=master)](https://travis-ci.org/choener/WordAlignment) + +# Word alignments in natural languages + +This library and program are designed for the alignment of *words* in human +languages. Each word is encoded as a list of *characters*. A single +*character*, however, is encoded as a *list of unicode symbols*, not just a +single symbol. + +This particular encoding is necessary because there is no general one-to-one +mapping of atomic alphabet symbols to unicode characters. Just consider +character decorations, such as *Umlaute* (even though Umlaute are available in +unicode). + +As user, you send a list to stdin that is formatted as follows (with all fields +mandatory; the tab symbol being the separator between each column): + +* word id +* language name +* meaning identifier (this is a string, not a number) +* length of the word +* space-separated characters (each character is actually a string) + +(In case you read the plain-text version of this document, the 4 whitespace +characters should be left out.) + + 0 Albanian_Tosk 1.100 5 \' b o t ə + 2 Albanian_Tosk 1.100 10 r̃ o k u lʸ i a\' lʸ e m + + + +The WordAlignment program comes with two modes, *twowaysimple* and *twoway*. + +## TwoWay - Simple + +```WordAlign twowaysimple``` aligns words based on a simple scoring model. Gaps +are scored with linear costs. Matches are scored based on a simple scoring file +handling unigrams of characters. Even the simple model allows for the concept +of equivalence classes. This makes it possible to not only score exact matches, +but to give somewhat high scores to, say, two vowels that should be matched. + +The following command-line options are provided: + + --scorefile=ITEM + --lpblock=ITEM,ITEM + --showmanual + --prettystupid + --outfile=ITEM + +```--scorefile``` is the simple score file to be used by the mode. the +*defaultSimpleScoring* file can be used as a template. + +```--lpblock``` expects a pair of language names (Breton,Breton) or a pair of +integers (3,3 or 4,6) and will then align only the given language pairs with +each other. This option should be very helpful in case you want to parallelize +the program. + +```--showmanual``` will show this manual in plain text. + +```--prettystupid``` will show a progress bar of the current language pair. +It's pretty and helpful with smaller tasks but should not be used when you +parallelize on a grid engine. + +```--outfile``` writes to the given output file, not stdout. Actually required +when ```--prettystupid``` is active. + +```--nobacktrack``` disables backtracking. Sometimes just the alignment score +is enough. + + + +## TwoWay (Complex Scoring) + +The complex scoring model uses linear gap costs. In contrast to the simple +model above, however, character matching is now performed in a bigram context. + +The required score file is currently using an in-house format with the +following columns all required (with whitespace, not tab between the entries): + +* language name +* character (bigram 1.1) +* character (bigram 1.2) +* language name +* character (bigram 2.1) +* character (bigram 2.2) +* score + +Three example lines + + Albanian_Tosk \' a Albanian_Tosk \' a 4.25238 + Albanian_Tosk \' a Albanian_Tosk \' b 0.402228 + Albanian_Tosk \' a Albanian_Tosk \' g 1.07432 + +In addition to the scoring file, set via ```--scorefile```, the default score +constant and the gap cost constant need to be set. + +```--bigramdef=NUM``` is the score given to unknown matches. + +```--gapopen=NUM``` is the score given to each gap character (and not just the +opening score) + +```--lpblock``` as above + +```--showmanual``` as above + +```--prettystupid``` as above + +```--outfile``` as above + +```--nobacktrack``` as above + +An example output is given below. + +The output are four lines for each alignment. An info line with the word ids +(IDS), the alignment score (SCORE), the normalized scores (NSCORE) and the +actual words, started by (WORDS) and interleaved by (WORD). The next two lines +are the alignment, with deletions showning up as minus symbols (---) in the +deletion field. Note that a deletion does not delete a character from the +input, it merely aligns an existing character in one alignment with the symbol +for deletion (--) in the other. The final line provides the per-column score +for the alignment. After the alignment follows one empty line. + +The normalized score is defined as SCORE / maximum of input word lengths + +```cat albanian.input | head -n 3 | ./dist/build/WordAlign/WordAlign twoway -s albanian.scores``` + +yields: + + IDS: 1 2 SCORE: 32.85 NSCORE: 5.48 WORDS: ^ d o u a r $ WORD ^ d o u a r $ + ^ d o u a r $ + ^ d o u a r $ + 0.0 4.8 5.9 4.1 6.6 4.6 6.8 + + IDS: 1 3 SCORE: -12.36 NSCORE: -1.77 WORDS: ^ d o u a r $ WORD ^ p o u l t r $ + ^ d - o u a r - - $ + ^ p o - u - l t r $ + 0.0 0.3 -5.0 -5.0 4.1 -5.0 -20.0 -5.0 -5.0 6.8 + + + +## Performance Notes + +Measured on a core i5-3570K @ 3.40 GHz; single-threaded. + +The program is not compiled for multi-threading, if you need this consider the +```--lpblock``` option first. Otherwise, send a mail. + +The running time for calculating 100 000 alignments is: + + Mode Seconds Alignments per Second + ==== ======= ===================== + TwoWay Simple: 6.7 s ~ 15 000 + TwoWay Complex: 15.8 s ~ 6 300 + +Disabling backtracking (to get just the alignment scores) improves performance +by roughly a factor of ```x 2```. Words are very short, backtracking overhead +is quite large! + +As in: the number of actual /unicode/ symbols is approximately the same as the +number of forward DP cells to be filled, since each /character/ can be a +combination of multiple unicdoe symbols. + + + +## Three- and Four-way Alignments + +These are currently disabled. If you need them, consider contacting me. + + + +#### Contact + +Christian Hoener zu Siederdissen +Leipzig University, Leipzig, Germany +choener@bioinf.uni-leipzig.de +http://www.bioinf.uni-leipzig.de/~choener/ + diff --git a/Setup.hs b/Setup.hs new file mode 100644 --- /dev/null +++ b/Setup.hs @@ -0,0 +1,2 @@ +import Distribution.Simple +main = defaultMain diff --git a/WordAlignment.cabal b/WordAlignment.cabal new file mode 100644 --- /dev/null +++ b/WordAlignment.cabal @@ -0,0 +1,159 @@ +name: WordAlignment +version: 0.1.0.0 +author: Christian Hoener zu Siederdissen, 2013-2015 +copyright: Christian Hoener zu Siederdissen, 2013-2015 +homepage: https://github.com/choener/WordAlignment +bug-reports: https://github.com/choener/WordAlignment/issues +maintainer: choener@bioinf.uni-leipzig.de +category: Linguistics +license: GPL-3 +license-file: LICENSE +build-type: Simple +stability: experimental +cabal-version: >= 1.10.0 +tested-with: GHC == 7.8.4, GHC == 7.10.2 +synopsis: Bigram word pair alignments. +description: + The library provides fast dynamic programming algorithms to + align word pairs using either a simple or a bigram scoring + scheme. + + + +Extra-Source-Files: + README.md + changelog.md + + + +library + build-depends: base >= 4.7 && < 4.9 + , ADPfusion >= 0.4.0.2 && < 0.5.1 + , AlignmentAlgorithms >= 0.0.2 && < 0.0.3 + , attoparsec >= 0.10 && < 0.14 + , bytestring + , containers + , control-monad-omega >= 0.3 && < 0.4 + , deepseq >= 1.3 && < 1.5 + , file-embed >= 0.0.6 && < 0.1 + , fmlist >= 0.9 && < 0.10 + , FormalGrammars >= 0.2.1 && < 0.2.2 + , ghc-prim + , GrammarProducts >= 0.1.1 && < 0.1.2 + , hashable >= 1.1 && < 1.3 + , intern >= 0.9 && < 0.10 + , lens >= 4.0 && < 4.14 + , LinguisticsTypes >= 0.0.0 && < 0.0.1 + , NaturalLanguageAlphabets >= 0.1.0 && < 0.2.0 + , primitive >= 0.5 && < 0.7 + , PrimitiveArray >= 0.6.0 && < 0.7.1 + , strict >= 0.3.2 && < 0.3.3 + , stringable >= 0.1.2 && < 0.1.4 + , template-haskell + , text >= 1.0 && < 1.3 + , text-format >= 0.3 && < 0.4 + , transformers + , tuple-th >= 0.2.4 && < 0.2.6 + , unordered-containers >= 0.2.3 && < 0.2.6 + , vector >= 0.10 && < 0.12 + exposed-modules: + Linguistics.Bigram + Linguistics.Common +-- Linguistics.FourWay +-- Linguistics.FourWay.Backward +-- Linguistics.FourWay.Common +-- Linguistics.FourWay.Forward +-- Linguistics.ThreeWay +-- Linguistics.ThreeWay.Bigram +-- Linguistics.ThreeWay.Common +-- Linguistics.ThreeWay.Simple +-- Linguistics.TwoWay +-- Linguistics.TwoWay.AdvancedBigram + Linguistics.TwoWay.Bigram +-- Linguistics.TwoWay.Common + Linguistics.TwoWay.Simple + Linguistics.Word + default-language: + Haskell2010 + default-extensions: BangPatterns + , DeriveGeneric + , FlexibleContexts + , FlexibleInstances + , OverloadedStrings + , PatternGuards + , QuasiQuotes + , StandaloneDeriving + , TemplateHaskell + , TypeFamilies + , TypeOperators + ghc-options: + -fsimpl-tick-factor=900 + -O2 + + + +executable WordAlign + build-depends: base + , ascii-progress >= 0.3.2 && < 0.4 + , bytestring + , cmdargs == 0.10.* + , containers + , file-embed + , intern + , LinguisticsTypes + , NaturalLanguageAlphabets + , parallel >= 3.2 && < 3.3 + , strict + , text + , unordered-containers + , vector + , WordAlignment + main-is: + WordAlign.hs + hs-source-dirs: + src + default-language: + Haskell2010 + default-extensions: BangPatterns + , DeriveDataTypeable + , OverloadedStrings + , PatternGuards + , RecordWildCards + , ScopedTypeVariables + , TemplateHaskell + ghc-options: + -fsimpl-tick-factor=900 + -rtsopts + -O2 + -funfolding-use-threshold100 + -funfolding-keeness-factor100 + + + +test-suite properties + type: + exitcode-stdio-1.0 + main-is: + properties.hs + ghc-options: + -threaded -rtsopts -with-rtsopts=-N + hs-source-dirs: + tests + default-language: + Haskell2010 + default-extensions: BangPatterns + , ScopedTypeVariables + , TemplateHaskell + build-depends: base + , QuickCheck + , test-framework >= 0.8 && < 0.9 + , test-framework-quickcheck2 >= 0.3 && < 0.4 + , test-framework-th >= 0.2 && < 0.3 + , WordAlignment + + + +source-repository head + type: git + location: git://github.com/choener/WordAlignment + diff --git a/changelog.md b/changelog.md new file mode 100644 --- /dev/null +++ b/changelog.md @@ -0,0 +1,27 @@ +0.1.0.0 +------- + +- using BTI types instead of IMMC +- backtracking pretty printing with Text-Builders +- updated README.md (TODO!) + +0.0.4.0 +------- + +- pretty, because we can! + +0.0.3.1 +------- + +- bugfixes + +0.0.2.0 +------- + +- partially rewritten + +0.0.1.0 +------- + +- internal, non-release version + diff --git a/src/WordAlign.hs b/src/WordAlign.hs new file mode 100644 --- /dev/null +++ b/src/WordAlign.hs @@ -0,0 +1,584 @@ + +{-# Options_GHC -fno-cse #-} + +module Main where + +import Control.Arrow ((***)) +import Control.Concurrent (threadDelay) +import Control.Monad (forM_,when) +import Control.Parallel.Strategies (rdeepseq,parMap,parBuffer,using,evalTuple2,r0,rseq,evalBuffer,parList,evalList,evalTuple3,evalTuple5) +import Data.FileEmbed +import Data.Function (on) +import Data.List (sortBy,groupBy,intersperse,genericLength) +import Data.Maybe (isJust) +import Data.Sequence (Seq) +import Data.Strict.Tuple +import Data.Text (Text) +import Data.Version (showVersion) +import Debug.Trace +import Debug.Trace (trace) +import GHC.Conc (numCapabilities) +import GHC.IO.Handle +import Prelude hiding (Word) +import qualified Data.ByteString.Char8 as BS +import qualified Data.ByteString.Lazy.Char8 as BL +import qualified Data.HashMap.Strict as HM +import qualified Data.Map.Strict as M +import qualified Data.Set as S +import qualified Data.Text as T +import qualified Data.Text.Lazy.Builder as TL +import qualified Data.Text.Lazy.IO as TL +import qualified Data.Vector as V +import qualified Data.Vector.Unboxed as VU +import qualified System.Console.AsciiProgress as CAP +import System.Console.CmdArgs +import System.Exit +import System.IO +import Text.Printf +import Text.Read (readMaybe) + +import NLP.Scoring.SimpleUnigram +import NLP.Scoring.SimpleUnigram.Import +import NLP.Text.BTI + +import Linguistics.Bigram +import Linguistics.Common +import Linguistics.TwoWay.Simple +import Linguistics.Word (parseWord,Word(..),addWordDelims) +import qualified Linguistics.TwoWay.Bigram as BI + +import Paths_WordAlignment (version) + + + +data Config + = TwoWaySimple + { scoreFile :: String + , lpblock :: Maybe (String,String) + , showManual :: Bool + , prettystupid :: Bool + , outfile :: String + , nobacktrack :: Bool + } + | TwoWay + { scoreFile :: String + , bigramDef :: Double +-- , unibiDef :: Double + , gapOpen :: Double +-- , gapExtend :: Double + , lpblock :: Maybe (String,String) + , showManual :: Bool + , prettystupid :: Bool + , outfile :: String + , nobacktrack :: Bool + } + deriving (Show,Data,Typeable) + +twowaySimple = TwoWaySimple + { scoreFile = def &= help "the file to read the simple scores from" + , lpblock = Nothing + , showManual = False &= help "show the manual and quit" + , prettystupid = False + , outfile = "" + , nobacktrack = False + } &= help "Align words based on a simple, linear scoring model for gaps, and an unigram model for matches." + +twoway = TwoWay + { scoreFile = "" &= help "the file to read the scores from" + , bigramDef = (-20) &= help "score to use for unknown bigram matches" +-- , unibiDef = (-5) &= help "score to close a gap if the closing characters are unknown" + , gapOpen = (-5) &= help "cost to open a gap" +-- , gapExtend = (-1) &= help "cost to extend a gap" -- we currently do not use the affine cost model. Should come later once a generic affine system is written in AlignmentAlgorithms + , lpblock = Nothing &= help "compare ONLY the given pair of languages: i.e 'Breton','Breton' or 2,3 (with the latter notation '2' being the 2nd language in the input file)" + , prettystupid = False &= help "a pretty stupid developer option" + , outfile = "" &= help "write output to this file" + , showManual = False + , nobacktrack = False + } &= help "Align words based on a linear scoring model for gaps, but with bigram-based scoring for matches." + +config = [twowaySimple, twoway] + &= program "WordAlign" + &= summary ("WordAlign " ++ showVersion version ++ " (c) Christian Höner zu Siederdissen 2014--2015, choener@bioinf.uni-leipzig.de") + +embeddedManual = $(embedFile "README.md") + + + +main = do + o <- cmdArgs $ modes config + when (showManual o) $ do + BS.putStrLn embeddedManual + exitSuccess + when (prettystupid o && null (outfile o)) $ do + putStrLn "The --prettystupid mode requires giving and --outfile" + exitFailure + (if prettystupid o then CAP.displayConsoleRegions else id) $ do + ws <- BL.getContents >>= return . map parseWord . BL.lines + case o of + TwoWaySimple{..} -> run2Simple o (blockSelection2 lpblock ws) + TwoWay{..} -> run2 o (blockSelection2 lpblock $ map addWordDelims ws) + + + +-- | Given a @Config@ and a @List of Lists of Word-Pairs@ align everything. + +run2Simple :: Config -> [[(Word,Word)]] -> IO () +run2Simple TwoWaySimple{..} wss = do + hndl <- if null outfile then return stdout else openFile outfile AppendMode + scoring <- simpleScoreFromFile scoreFile + let wsslen = length wss + -- for each language pairing + forM_ (zip [1 :: Int ..] wss) $ \(k,ws) -> do + pg <- if prettystupid + then do + let len = genericLength ws + let (x,y) = head ws + printf "[%4d / %4d] Language pair: %s / %s with %d alignments:\n" k wsslen (show $ wordLang x) (show $ wordLang y) len + Just <$> CAP.newProgressBar CAP.def { CAP.pgWidth = 100, CAP.pgTotal = len } + else return Nothing + let alis = [ (x,y,d,bts,sss) + | (x,y) <- ws + , let (d,bts') = alignGlobal scoring 1 (wordWord x) (wordWord y) -- calculate score and all co-optimal backtraces + , let bts = if nobacktrack then [] else bts' + , let sss = TL.toLazyText $ buildAlignmentSimple 0 ([x,y],(d,bts)) -- make nice strings + ] + -- print the actual alignments + forM_ alis $ \(x,y,d,bts,sss) -> do + TL.hPutStr hndl sss + when (isJust pg) $ let Just pg' = pg in CAP.tick pg' + +-- | Given a @Config@ and a @List of List of Word-Pairs@ align everything. + +run2 :: Config -> [[(Word,Word)]] -> IO () +run2 TwoWay{..} wss = do + hndl <- if null outfile then return stdout else openFile outfile AppendMode + let wsslen = length wss + -- build up scoring system + let chkLs = S.fromList . map wordLang . concat . map (\(x,y) -> [x,y]) . map head $ wss + scoring <- BL.readFile scoreFile >>= return . generateLookups chkLs (-999999) + -- for each language pairing + forM_ (zip [1 :: Int ..] wss) $ \(k,ws) -> do + pg <- if prettystupid + then do + let len = genericLength ws + let (x,y) = head ws + printf "[%4d / %4d] Language pair: %s / %s with %d alignments:\n" k wsslen (show $ wordLang x) (show $ wordLang y) len + Just <$> CAP.newProgressBar CAP.def { CAP.pgWidth = 100, CAP.pgTotal = len } + else return Nothing + -- get score pairing + let (hx,hy) = head ws + let sco = getScores2 scoring (wordLang hx) (wordLang hy) + -- align the words the in @ws@ pairing + let as = map (\(x,y) -> ( let (d,bts) = BI.alignGlobal bigramDef gapOpen sco 1 (wordWord x) (wordWord y) + in TL.toLazyText $ buildAlignment (-1) ([x,y],(d,if nobacktrack then [] else bts)) + ) + ) ws + forM_ (as) {- `using` parBuffer 100 rseq)-} $ \ali -> do + when (isJust pg) $ let Just pg' = pg in CAP.tick pg' + TL.hPutStr hndl ali + +-- | Given a set of words from different languages, we want to do two +-- things: +-- +-- (i) We want to block alignments into groups. This will make alignments +-- faster as we do not have to reload the scoring system every time. +-- +-- (ii) We want to decide on alignment maybe only a subset of words. We +-- make this selection rather coarse-grained by giving just the name or +-- running id of the language. I.e you can type @Breton@ or @1@ (if Breton +-- happens to be the first language). We allow numeric identification as +-- that is easier for scripts to handle. + +blockSelection2 :: Maybe (String,String) -> [Word] -> [[(Word,Word)]] +blockSelection2 s ws = go (mkCmp s) + -- grouping words by their languages + where gs = zip [1..] $ groupBy ((==) `on` wordLang) ws + ls = M.fromList $ map (\(k,(v:_)) -> (show $ wordLang v,k)) gs + go f = [ [ (x,y) + | (kk,x) <- zip [1..] xs, (ll,y) <- zip [1..] ys + , k/=l || kk < ll + ] + | (k,xs) <- gs, (l,ys) <- gs -- @k@ and @l@ are word groups, i.e. language identifiers + , f k l + ] + -- Create comparator function for group selection + mkCmp :: Maybe (String,String) -> (Int -> Int -> Bool) + -- nothing selected, produce all upper triangular groups + mkCmp Nothing = \k l -> k <= l + mkCmp (Just (a,b)) + -- we have two strings that actually are numbers + | Just a' <- readMaybe a + , Just b' <- readMaybe b = \k l -> a'==k && b'==l + -- we have two language names + | Just a' <- M.lookup a ls + , Just b' <- M.lookup b ls = \k l -> a'==k && b'==l + -- the user did provide crappy input + mkCmp (Just (a,b)) = \k l -> traceShow ("Unknown languages or ID's: " ++ a ++ " , " ++ b) $ False + +-- | (write me) + +getScores2 :: Mapping -> Lang -> Lang -> Scores +getScores2 ss a b + | Just z <- M.lookup (a:!:b) (lliid ss) = z + | otherwise = trace (printf "Language pair %s %s not found in mapping! Returning empty hashmap\n" (toUtf8String a) (toUtf8String b)) + HM.empty + +-- | (write me) + +prettyAli2 :: Double -> [(BTI,BTI)] -> IO () +prettyAli2 d s = do + print d + forM_ s $ \(x,xs) -> do + putStr $ show x + putStr $ show xs + putStrLn "" + forM_ s $ \(_,y) -> do + putStr $ show y + putStrLn "" + +buildAlignmentSimple :: Double -> ([Linguistics.Word.Word],(Double,[[[Text]]])) -> TL.Builder +buildAlignmentSimple k (ws,(s,(xs))) = TL.fromText hdr `mappend` ls `mappend` "\n" where + ids = concat . intersperse " " . map (show . wordID) $ ws + wds = concat . intersperse " WORD " . map (concat . intersperse " " . map toUtf8String . VU.toList . wordWord) $ ws + ns = s / (maximum $ 1 : map ((+k) . fromIntegral . VU.length . wordWord) ws) + hdr = T.pack $ printf "IDS: %s SCORE: %.2f NSCORE: %.2f WORDS: %s\n" ids s ns wds + ls = case xs of [] -> "" ; [xs'] -> buildLines xs' + +buildAlignment :: Double -> ([Linguistics.Word.Word],(Double,[[[Text]]])) -> TL.Builder +buildAlignment k (ws,(s,(xss))) = TL.fromText hdr `mappend` ls `mappend` "\n" where + ids = concat . intersperse " " . map (show . wordID) $ ws + wds = concat . intersperse " WORD " . map (concat . intersperse " " . map toUtf8String . VU.toList . wordWord) $ ws + ns = s / (maximum $ 1 : map ((+k) . fromIntegral . VU.length . wordWord) ws) + hdr = T.pack $ printf "IDS: %s SCORE: %.2f NSCORE: %.2f WORDS: %s\n" ids s ns wds + ls = case xss of [] -> "" ; [xs'] -> buildLines $ ["^","^","0.0"] : xs' + +--printAlignment :: Double -> ([Linguistics.Word.Word],(Double,[[(IMCp,IMCp)]])) -> IO () +--printAlignment k (ws,(s,(xs))) = do +-- let ids = concat . intersperse " " . map (show . wordID) $ ws +-- let wds = concat . intersperse " WORD " . map (concat . intersperse " " . map toUtf8String . VU.toList . wordWord) $ ws +-- --let ns = s / (maximum $ 1 : map ((+k) . fromIntegral . VU.length . wordWord) ws) +-- let ns = s / (maximum $ 1 : map ((+k) . fromIntegral . VU.length . wordWord) ws) +-- printf "IDS: %s SCORE: %.2f NSCORE: %.2f WORDS: %s\n" ids s ns wds +-- let xs1 = map ({- tail . -} reverse . map Prelude.fst) $ xs +-- let xs2 = map ({- tail . -} reverse . map Prelude.snd) $ xs +-- mapM_ (putStrLn) (alignPretty $ xs1 ++ xs2) +-- putStrLn "" + + + + + + + + + + + + + + + + + + + + + + + +{- +import Control.Arrow +import Control.Monad (unless) +import Control.Monad (when) +import Control.Parallel.Strategies +import Data.ByteString (ByteString) +import Data.ByteString.Char8 (unpack) +import Data.Function +import Data.List (intersperse) +import Data.List (tails,genericLength,genericTake,genericDrop,group) +import Data.Ord +import Data.Strict.Tuple +import Data.Vector (fromList) +import Debug.Trace (trace) +import qualified Data.ByteString as BS +import qualified Data.ByteString.Char8 as BC +import qualified Data.ByteString.Lazy.Char8 as BL +import qualified Data.HashTable.IO as H +import qualified Data.List as L +import qualified Data.Map.Strict as M +import qualified Data.Set as S +import qualified Data.Vector as V +import qualified Data.Vector.Unboxed as VU +import System.Console.CmdArgs +import System.IO.Unsafe (unsafePerformIO) +import Text.Printf + +import NLP.Alphabet.MultiChar +import NLP.Scoring.SimpleUnigram +import NLP.Scoring.SimpleUnigram.Default +import NLP.Scoring.SimpleUnigram.Import + +import Linguistics.TwoWay.Simple +{- we test interning with TwoWay alignments only +import Linguistics.ThreeWay +import Linguistics.FourWay +-} +import Linguistics.Bigram +import Linguistics.Common +import Linguistics.Word +import Linguistics.TwoWay.AdvancedBigram + +data Config + = TwoWay + { scoreFile :: String + , bigramDef :: Double + , unibiDef :: Double + , gapOpen :: Double + , gapExtend :: Double + , block :: Maybe (Integer,Integer) + , selfAlign :: Bool + } + | TwoWaySimple + { scoreFile :: String + , block :: Maybe (Integer,Integer) + , selfAlign :: Bool + } + {- NLA + | ThreeWay + { scoreFile :: String + , defaultScore :: Double + , gapOpen :: Double + , block :: Maybe (Integer,Integer) + } + | ThreeWaySimple + { scoreFile :: String + , gapOpen :: Double + , block :: Maybe (Integer,Integer) + } + | FourWay + { scoreFile :: String + , defaultScore :: Double + , gapOpen :: Double + , block :: Maybe (Integer,Integer) + } + | FourWaySimple + { scoreFile :: String + , gapOpen :: Double + , block :: Maybe (Integer,Integer) + } + -} + | Info + { + } + deriving (Show,Data,Typeable) + +twoway = TwoWay + { scoreFile = "" &= help "the file to read the scores from" + , bigramDef = (-20) &= help "score to use for unknown bigram matches" + , unibiDef = (-5) &= help "score to close a gap if the closing characters are unknown" + , gapOpen = (-5) &= help "cost to open a gap" + , gapExtend = (-1) &= help "cost to extend a gap" + , block = Nothing &= help "when using --block N,k calculate only the k'th block (starting at 1) with length N. For parallelized computations." + , selfAlign = False &= help "align each word with itself as well" + } &= help "Align two words at a time for all ordered word combinations" + +twowaySimple = TwoWaySimple + { scoreFile = def &= help "" + } + +{- NLA +threeway = ThreeWay + { + } + +threewaySimple = ThreeWaySimple + { + } + +fourway = FourWay + { + } + +fourwaySimple = FourWaySimple + { + } +-} + +info = Info + { + } + +config = [twoway,twowaySimple, {- threeway,threewaySimple,fourway,fourwaySimple, -} info] + &= program "WordAlign" + &= summary "WordAlign v.0.0.1" + +main = do + o <- cmdArgs $ modes config + ws' <- BL.getContents >>= return . map parseWord . BL.lines + case o of + Info{} -> do + let l :: Integer = genericLength ws' + let c2 = l * (l-1) `div` 2 -- number of alignments + let t2 :: Double = fromIntegral c2 / 5000 / 60 / 60 -- approximate time in hours + let c3 = l * (l-1) * (l-2) `div` 3 + let t3 :: Double = fromIntegral c3 / 5000 / 60 / 60 -- TODO fix time constant + let c4 = l * (l-1) * (l-2) * (l-3) `div` 4 + let t4 :: Double = fromIntegral c4 / 5000 / 60 / 60 -- TODO fix time constant + printf "%d %.1f %d %.1f %d %.1f\n" c2 t2 c3 t3 c4 t4 + TwoWay{..} -> do + let ws = map addWordDelims ws' + let bs = blockWith block $ [ (a,b) | (a:as) <- tails ws, b <- if selfAlign then (a:as) else as ] + let chkLs = S.fromList . map wordLang $ ws + {- + let chkLs = if block==Nothing + then S.fromList . map wordLang $ ws + else S.fromLIst . map wordLang $ ws -- S.fromList . map head . group . map wordLang . concatMap (\(a,b) -> [a,b]) $ bs + -} + ss <- BL.readFile scoreFile >>= return . generateLookups chkLs (-999999) + let ts = map (\(a,b) -> ( [a,b], alignTwo (BigramScores {gapOpen = gapOpen, gapExtend = gapExtend, bigramDef = bigramDef, unigramDef = -999999, biUniDef = unibiDef, scores = getScores2 ss (wordLang a) (wordLang b)}) + (wordWord a) (wordWord b) + ) + ) bs + mapM_ (printAlignment (-2)) ts + TwoWaySimple{..} -> do + simpleScoring <- if null scoreFile then return $ error "scorefile missing" + else simpleScoreFromFile scoreFile + let ws = ws' + let bs = blockWith block $ [ (a,b) | (a:as) <- tails ws, b <- if selfAlign then (a:as) else as ] + let ts = map (\(a,b) -> ( [a,b], alignTwoSimple simpleScoring (wordWord a) (wordWord b) + ) + ) bs + mapM_ (printAlignment 0) ts + {- + ThreeWay{..} -> do + let ws = map addWordDelims ws' + let bs = blockWith block $ [ (a,b,c) | (a:as) <- tails ws, (b:bs) <- tails as, c <- bs ] + let chkLs = S.fromList . map wordLang $ ws + {- + let chkLs = if block==Nothing + then S.fromList . map wordLang $ ws + --else S.fromList . map head . group . map wordLang . concatMap (\(a,b,c) -> [a,b,c]) $ bs + else S.fromLIst . map wordLang $ ws -- S.fromList . map head . group . map wordLang . concatMap (\(a,b) -> [a,b]) $ bs + -} + ss <- BL.readFile scoreFile >>= return . generateLookups chkLs defaultScore + let ts = ss `seq` map (\(a,b,c) -> ( [a,b,c], alignThree defaultScore gapOpen (getScores3 ss (wordLang a) (wordLang b) (wordLang c)) + (wordWord a) (wordWord b) (wordWord c) + ) + ) bs + mapM_ (printAlignment (-2)) ts + ThreeWaySimple{..} -> do + [vwl,cns] <- readFile vowelConsonantFile >>= return . map VU.fromList . lines + scs <- if null scoreFile then return [3,1,1,0,0,-1] else (readFile scoreFile >>= return . map read . words) + let ws = ws' + let bs = blockWith block $ [ (a,b,c) | (a:as) <- tails ws, (b:bs) <- tails as, c <- bs ] + let ts = map (\(a,b,c) -> ( [a,b,c], alignThreeSimple vwl cns scs gapOpen (wordWord a) (wordWord b) (wordWord c) + ) + ) bs + mapM_ (printAlignment 0) ts + FourWay{..} -> do + let ws = map addWordDelims ws' + let bs = blockWith block $ [ (a,b,c,d) | (a:as) <- tails ws, (b:bs) <- tails as, (c:cs) <- tails bs, d <- cs ] + let chkLs = S.fromList . map wordLang $ ws + {- + let chkLs = if block==Nothing + then S.fromList . map wordLang $ ws + --else S.fromList . map head . group . map wordLang . concatMap (\(a,b,c,d) -> [a,b,c,d]) $ bs + else S.fromLIst . map wordLang $ ws -- S.fromList . map head . group . map wordLang . concatMap (\(a,b) -> [a,b]) $ bs + -} + ss <- BL.readFile scoreFile >>= return . generateLookups chkLs defaultScore + let ts = map (\(a,b,c,d) -> ( [a,b,c,d], alignFour defaultScore gapOpen (getScores4 ss (wordLang a) (wordLang b) (wordLang c) (wordLang d)) + (wordWord a) (wordWord b) (wordWord c) (wordWord d) + ) + ) bs + mapM_ (printAlignment (-2)) ts + FourWaySimple{..} -> do + [vwl,cns] <- readFile vowelConsonantFile >>= return . map VU.fromList . lines + scs <- if null scoreFile then return [3,1,1,0,0,-1] else (readFile scoreFile >>= return . map read . words) + let ws = ws' + let bs = blockWith block $ [ (a,b,c,d) | (a:as) <- tails ws, (b:bs) <- tails as, (c:cs) <- tails bs, d <- cs ] + let ts = map (\(a,b,c,d) -> ( [a,b,c,d], alignFourSimple vwl cns scs gapOpen (wordWord a) (wordWord b) (wordWord c) (wordWord d) + ) + ) bs + mapM_ (printAlignment 0) ts + -} + +alignTwo :: BigramScores -> V.Vector InternedMultiChar -> V.Vector InternedMultiChar -> (Double, [[String]]) +alignTwo scores x y + = second (map alignPretty) -- . filter (any (\c -> c/= "$" && c/= "^"))) + $ runBigram scores 1 x y + +alignTwoSimple + :: SimpleScoring + -> V.Vector InternedMultiChar + -> V.Vector InternedMultiChar + -> (Double, [[String]]) +alignTwoSimple simpleScoring x y = second (map alignPretty) $ twoWaySimple simpleScoring x y + +{- +alignThree :: Double -> Double -> (Scores,Scores,Scores) -> V.Vector ByteString -> V.Vector ByteString -> V.Vector ByteString -> (Double, [[String]]) +alignThree sDef sGapOpen scores x y z = second (map (alignPretty . map (filter (\c -> c/= "$" && c/="^")) . tup3List)) $ threeWayBigram sDef sGapOpen scores x y z + +alignThreeSimple + :: VU.Vector Char + -> VU.Vector Char + -> [Double] + -> Double + -> V.Vector ByteString + -> V.Vector ByteString + -> V.Vector ByteString + -> (Double, [[String]]) +alignThreeSimple v c scores sGapOpen x y z = second (map (alignPretty . tup3List)) $ threeWaySimple v c scores sGapOpen x y z + +alignFour :: Double -> Double -> (Scores,Scores,Scores,Scores,Scores,Scores) -> V.Vector ByteString -> V.Vector ByteString -> V.Vector ByteString -> V.Vector ByteString -> (Double, [[String]]) +alignFour sDef sGapOpen scores w x y z = second (map (alignPretty . map (filter (\c -> c/= "$" && c/="^")) . tup4List)) $ fourWayBigram sDef sGapOpen scores w x y z + +alignFourSimple + :: VU.Vector Char + -> VU.Vector Char + -> [Double] + -> Double + -> V.Vector ByteString + -> V.Vector ByteString + -> V.Vector ByteString + -> V.Vector ByteString + -> (Double, [[String]]) +alignFourSimple v c scores sGapOpen w x y z = second (map (alignPretty . tup4List)) $ fourWaySimple v c scores sGapOpen w x y z +-} + +blockWith Nothing xs = xs +blockWith (Just (l,k)) xs = genericTake l . genericDrop (l * (k-1)) $ xs + +getScores2 :: Mapping -> Lang -> Lang -> Scores +getScores2 ss a b + | Just z <- M.lookup (a:!:b) (lliid ss) = z + | otherwise = trace (printf "Language pair %s %s not found in mapping! Returning empty hashmap\n" (toUtf8String a) (toUtf8String b)) + (unsafePerformIO H.new) + +{- +getScores3 :: Mapping -> Lang -> Lang -> Lang -> (Scores,Scores,Scores) +getScores3 ss a b c = (getScores2 ss a b, getScores2 ss a c, getScores2 ss b c) -- (lliid ss M.! (a:!:b), lliid ss M.! (a:!:c), lliid ss M.! (b:!:c)) + +getScores4 :: Mapping -> Lang -> Lang -> Lang -> Lang -> (Scores,Scores,Scores,Scores,Scores,Scores) +getScores4 ss a b c d = (getScores2 ss a b, getScores2 ss a c, getScores2 ss a d, getScores2 ss b c, getScores2 ss b d, getScores2 ss c d) +-- (lliid ss M.! (a:!:b), lliid ss M.! (a:!:c), lliid ss M.! (a:!:d), lliid ss M.! (b:!:c), lliid ss M.! (b:!:d), lliid ss M.! (c:!:d)) +-} + +printAlignment :: Double -> ([Word], (Double, [[String]])) -> IO () +printAlignment k (ws,(s,[])) = do + printf "DEBUG!\nScore: %f\nDEBUG!\n\n" s + +printAlignment k (ws,(s,(x:xs))) = do + let ids = concat . intersperse " " . map (show . wordID) $ ws + let wds = concat . intersperse " WORD " . map (concat . intersperse " " . map toUtf8String . V.toList . wordWord) $ ws + let ns = s / (maximum $ 1 : map ((+k) . fromIntegral . V.length . wordWord) ws) + printf "IDS: %s SCORE: %.2f NSCORE: %.2f WORDS: %s\n" ids s ns wds + mapM_ putStrLn x + putStrLn "" + + +tup2List (a,b) = [a,b] +tup3List (a,b,c) = [a,b,c] +tup4List (a,b,c,d) = [a,b,c,d] + +-} + diff --git a/tests/properties.hs b/tests/properties.hs new file mode 100644 --- /dev/null +++ b/tests/properties.hs @@ -0,0 +1,11 @@ + +module Main where + +import Test.Framework.Providers.QuickCheck2 +import Test.Framework.TH + + + +main :: IO () +main = $(defaultMainGenerator) +