sequor (empty) → 0.1
raw patch · 22 files changed
+4294/−0 lines, 22 filesdep +arraydep +basedep +binarysetup-changed
Dependencies added: array, base, binary, bytestring, containers, mtl, utf8-string, vector
Files
- LICENSE +26/−0
- Makefile +15/−0
- README +115/−0
- Setup.lhs +8/−0
- data/all.features +11/−0
- data/devel.conll +1000/−0
- data/example.features +13/−0
- data/test.conll +1000/−0
- data/train.conll +1000/−0
- lib/haskell-utils/Atom.hs +85/−0
- lib/haskell-utils/ListZipper.hs +69/−0
- lib/haskell-utils/Text.hs +85/−0
- lib/haskell-utils/Utils.hs +43/−0
- sequor.cabal +28/−0
- src/Config.hs +23/−0
- src/CorpusReader.hs +45/−0
- src/FeatureTemplate.hs +33/−0
- src/Features.hs +117/−0
- src/Labeler.hs +187/−0
- src/Main.hs +49/−0
- src/Perceptron/Sequence.hs +259/−0
- src/Perceptron/Vector.hs +83/−0
+ LICENSE view
@@ -0,0 +1,26 @@+Copyright (c) 2010, Grzegorz Chrupala+All rights reserved.++Redistribution and use in source and binary forms, with or without+modification, are permitted provided that the following conditions are+met:++ * Redistributions of source code must retain the above copyright+ notice, this list of conditions and the following disclaimer.++ * Redistributions in binary form must reproduce the above+ copyright notice, this list of conditions and the following+ disclaimer in the documentation and/or other materials provided+ with the distribution.++THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT+HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ Makefile view
@@ -0,0 +1,15 @@+INCLUDES=lib/haskell-utils++all:run+ ++lib/haskell-utils:+ svn co http://ws9lx.lsv.uni-saarland.de/repos/gchrupala/haskell-utils/ lib/haskell-utils++run: lib/haskell-utils+ ghc --make -O2 -isrc:$(INCLUDES) -o bin/sequor src/Main.hs++clean:+ -find . -name '*.o' | xargs rm+ -find . -name '*.hi' | xargs rm+
+ README view
@@ -0,0 +1,115 @@+sequor 0.1++AUTHOR: Grzegorz Chrupała <pitekus@gmail.com>++Sequor is a sequence labeler based on Collins's sequence+perceptron. Sequor has a flexible feature template language and is+meant mainly for NLP applications such as Part of Speech tagging,+syntactic chunking or Named Entity labeling.+ +++USAGE++With Sequor you can learn a model from sequences manually annotated+with labels, and then apply this model to new data in order to add+labels. Sequor is meant to be used mainly with linguistic data, for+example to learn Part of Speech tagging, syntactic chunking or Named+Entity labeling.+ +In order to learn a model from labeled data call sequor like this:++sequor train FEATURE-TEMPLATE LEARNING-RATE BEAM-SIZE MAX-ITERATIONS \+ MIN-DICT-COUNT TRAIN-FILE HELDOUT-FILE MODEL-FILE+FEATURE-TEMPLATE - specification of which features to use. For an + example see data/AllFeatures.txt+LEARNING-RATE - positive number (=<1) which controls how fast learning is+ 0.1 is a reasonable default+BEAM-SIZE - positive integer controlling the size of the beam. +ITERATIONS - positive integer controlling for how many iterations to train+MIN-DICT-COUNT - positive integer specifying how many times an indexed + feature has to use the label dictionary for this feature. + Using a large number will effectively disable use of label+ dictionary+TRAIN-FILE - annotated data in CoNLL format. Sequences separated by + blank lines, features separated by space+HELDOUT-FILE - annotated heldout data. To disable use an empty file + (/dev/null) +MODEL-FILE - name of the file where the learned model will be stored++++In order to apply the learned model to new data, call:++sequor predict MODEL-FILE < NEW-DATA > NEW-LABELS++Data files should be in the UTF-8 encoding.++As an example we can use data annotated with syntactic chunk labels in+the data directory. For example:++./bin/sequor train data/all.features 0.1 10 5 50 \+ data/train.conll data/devel.conll model++./bin/sequor predict model < data/test.conll > data/test.labels++FEATURE TEMPLATE SYNTAX++Sequor uses a small language for specifying feature templates to use+when learing. This section gives an informal overview of this+language. Sequor uses the simple CoNLL format for the input files. In+this format sentences are separated by blank lines. Each line+represents a single token (word). Each token should have the same+fixed number of space-separated fields, where the last field is the+label, e.g.++der d ART I-NC O+Europäischen europäisch ADJA I-NC ORG+Union Union NN I-NC ORG++The template language treats the input sentence as a matrix of+features (i.e. field values) and allows you to select and apply some+transformations to those features.++The language consists of a number predefined functions. By calling the+functions with certain argument you can specify the feature set to+use. As an example consider the following template: +Cat [ Cell 0 0, Suffix 2 (Cell 0 0), Row -1, Row 1 ]. +It specifies the following features: the first field in the current+token, the two-characted suffix of the first field of the current+token, all the fields of the previous token and all the fields of the+following token. ++Functions:++Cell r c Selects field in row r and column c. +Rect r c r' c Selects all features in the rectangle+ whose upper-left corner is in row r column c+ and lower-right corner is in row r' column c'.+Row r Selects all features in row r.+MarkNull f If feature does not exist, replace it with a NULL mark+ Typically used when absence of feature is significant, + e.g. to mark the beginning of the sentence.+Index f Marks the feature f to use in indexing for label + dictionary.+Cat [f1,f2,...,fn] Selects features in the list.+Cart f f' Creates Cartesian product of feature sets f and f'. + If f and f' are singletons, simply conjoins the + two features.+Lower f Maps f to lower case characters.+Suffix i f Takes suffix of i character length of feature f.+Prefix i f Takes prefix of i character langth of feature f.+WordShape f Creates a specification of which charater classes such + as lower case and upper case letters, digits or + punctuation occur in feature f.++Remarks: Rows are indexed relative to the current token (0). Columns+are indexed starting with 0. Functions which take features as+arguments can be passed either singleton features or sequences of+features. If passed a sequence they are applied to each of its+elements. For example (Suffix 3 (Row 0)) will return the sequence of+features formed by taking the suffix of length 3 of each field of row+0.++For more examples see files all.features and example.features in the+directory data.
+ Setup.lhs view
@@ -0,0 +1,8 @@+#! /usr/bin/env runhaskell++> module Main (main) where++> import Distribution.Simple++> main :: IO ()+> main = defaultMain
+ data/all.features view
@@ -0,0 +1,11 @@+Cat [ Index (Cell 0 0)+ , Rect 0 1 0 10 + , MarkNull (Cell -1 0)+ , Rect -1 1 -1 10+ , Rect -2 0 -2 10+ , Row 1 + , Row 2+ ]+++
+ data/devel.conll view
@@ -0,0 +1,1000 @@+the DT B-NP+Friendly NNP I-NP+Skies NNPS I-NP+-- : O+free JJ B-NP+first-class JJ I-NP+travel NN I-NP+, , O+and CC O+$ $ B-NP+20,000 CD I-NP+a DT B-NP+year NN I-NP+for IN B-PP+life NN B-NP+as IN B-ADVP+well RB I-ADVP+. . O++Conceivably RB B-ADVP+, , O+in IN B-PP+a DT B-NP+scaled-back JJ I-NP+buy-out NN I-NP+, , O+they PRP B-NP+could MD B-VP+be VB I-VP+bumped VBN I-VP+back RB B-ADVP+to TO B-PP+coach NN B-NP+seats NNS I-NP+for IN B-PP+life NN B-NP+. . O++Thomas NNP B-NP+H. NNP I-NP+Johnson NNP I-NP+, , O+president NN B-NP+of IN B-PP+the DT B-NP+Coatedboard NNP I-NP+division NN I-NP+of IN B-PP+Mead NNP B-NP+Corp. NNP I-NP+, , O+was VBD B-VP+named VBN I-VP+president NN B-NP+of IN B-PP+Manville NNP B-NP+Forest NNP I-NP+Products NNP I-NP+Corp. NNP I-NP+, , O+a DT B-NP+Manville NNP I-NP+unit NN I-NP+, , O+and CC O+senior JJ B-NP+vice NN I-NP+president NN I-NP+of IN B-PP+Manville NNP B-NP+Corp NNP I-NP+. . O++Mr. NNP B-NP+Johnson NNP I-NP+succeeds VBZ B-VP+Harry NNP B-NP+W. NNP I-NP+Sherman NNP I-NP+, , O+who WP B-NP+resigned VBD B-VP+to TO B-VP+pursue VB I-VP+other JJ B-NP+interests NNS I-NP+, , O+in IN B-PP+both DT B-NP+positions NNS I-NP+. . O++Manville NNP B-NP+is VBZ B-VP+a DT B-NP+building NN I-NP+and CC I-NP+forest NN I-NP+products NNS I-NP+concern VBP I-NP+. . O++US PRP B-NP+Facilities NNP I-NP+Corp. NNP I-NP+said VBD B-VP+Robert NNP B-NP+J. NNP I-NP+Percival NNP I-NP+agreed VBD B-VP+to TO I-VP+step VB I-VP+down RB B-ADVP+as IN B-PP+vice NN B-NP+chairman NN I-NP+of IN B-PP+the DT B-NP+insurance NN I-NP+holding VBG I-NP+company NN I-NP+. . O++`` `` O+There EX B-NP+was VBD B-VP+a DT B-NP+difference NN I-NP+of IN B-PP+opinion NN B-NP+as IN B-PP+to TO B-PP+the DT B-NP+future NN I-NP+direction NN I-NP+of IN B-PP+the DT B-NP+company NN I-NP+, , O+'' '' O+a DT B-NP+spokeswoman NN I-NP+said VBD B-VP+. . O++Mr. NNP B-NP+Percival NNP I-NP+declined VBD B-VP+to TO I-VP+comment VB I-VP+. . O++In IN B-PP+a DT B-NP+statement NN I-NP+, , O+US PRP B-NP+Facilities NNPS I-NP+said VBD B-VP+Mr. NNP B-NP+Percival NNP I-NP+'s POS B-NP+employment NN I-NP+contract NN I-NP+calls VBZ B-VP+for IN B-SBAR+him PRP B-NP+to TO B-VP+act VB I-VP+as IN B-PP+a DT B-NP+consultant NN I-NP+to TO B-PP+the DT B-NP+company NN I-NP+for IN B-PP+two CD B-NP+years NNS I-NP+. . O++He PRP B-NP+will MD B-VP+also RB I-VP+remain VB I-VP+a DT B-NP+director NN I-NP+, , O+US PRP B-NP+Facilities NNPS I-NP+said VBD B-VP+, , O+but CC O+wo MD B-VP+n't RB I-VP+serve VB I-VP+on IN B-PP+any DT B-NP+board NN I-NP+committees NNS I-NP+. . O++Mr. NNP B-NP+Percival NNP I-NP+will MD B-VP+be VB I-VP+succeeded VBN I-VP+on IN B-PP+an DT B-NP+interim JJ I-NP+basis NN I-NP+by IN B-PP+George NNP B-NP+Kadonada NNP I-NP+, , O+US PRP B-NP+Facilities NNPS I-NP+chairman NN I-NP+and CC I-NP+president NN I-NP+. . O++In IN B-PP+the DT B-NP+same JJ I-NP+statement NN I-NP+, , O+US PRP B-NP+Facilities NNPS I-NP+also RB B-ADVP+said VBD B-VP+it PRP B-NP+had VBD B-VP+bought VBN I-VP+back RB B-ADVP+112,000 CD B-NP+of IN B-PP+its PRP$ B-NP+common JJ I-NP+shares NNS I-NP+in IN B-PP+a DT B-NP+private JJ I-NP+transaction NN I-NP+. . O++Terms NNS B-NP+were VBD B-VP+n't RB I-VP+disclosed VBN I-VP+. . O++The DT B-NP+buy-back NN I-NP+represents VBZ B-VP+about IN B-NP+3 CD I-NP+% NN I-NP+of IN B-PP+the DT B-NP+company NN I-NP+'s POS B-NP+shares NNS I-NP+, , O+based VBN B-VP+on IN B-PP+the DT B-NP+3.7 CD I-NP+million CD I-NP+shares NNS I-NP+outstanding JJ B-ADJP+as IN B-PP+of IN B-PP+Sept. NNP B-NP+30 CD I-NP+. . O++In IN B-PP+national JJ B-NP+over-the-counter JJ I-NP+trading NN I-NP+yesterday NN B-NP+, , O+US PRP B-NP+Facilities NNPS I-NP+closed VBD B-VP+at IN B-PP+$ $ B-NP+3.625 CD I-NP+, , O+unchanged JJ B-ADJP+. . O++Three CD B-NP+leading VBG I-NP+drug NN I-NP+companies NNS I-NP+reported VBD B-VP+robust JJ B-NP+third-quarter JJ I-NP+earnings NNS I-NP+, , O+bolstered VBN B-VP+by IN B-PP+strong JJ B-NP+sales NNS I-NP+of IN B-PP+newer JJR B-NP+, , I-NP+big-selling JJ I-NP+prescriptions NNS I-NP+drugs NNS I-NP+that WDT B-NP+provide VBP B-VP+hefty JJ B-NP+profit NN I-NP+margins NNS I-NP+. . O++Merck NNP B-NP+& CC I-NP+Co. NNP I-NP+reported VBD B-VP+a DT B-NP+25 CD I-NP+% NN I-NP+increase NN I-NP+in IN B-PP+earnings NNS B-NP+; : O+Warner-Lambert NNP B-NP+Co. NNP I-NP+'s POS B-NP+profit NN I-NP+rose VBD B-VP+22 CD B-NP+% NN I-NP+and CC O+Eli NNP B-NP+Lilly NNP I-NP+& CC I-NP+Co. NNP I-NP+'s POS B-NP+net JJ I-NP+income NN I-NP+rose VBD B-VP+24 CD B-NP+% NN I-NP+. . O++The DT B-NP+results NNS I-NP+were VBD B-VP+in IN B-PP+line NN B-NP+with IN B-PP+analysts NNS B-NP+' POS B-NP+expectations NNS I-NP+. . O++Merck NNP B-NP+& CC I-NP+Co NNP I-NP+. . O++Merck NNP B-NP+, , O+Rahway NNP B-NP+, , O+N.J. NNP B-NP+, , O+continued VBD B-VP+to TO I-VP+lead VB I-VP+the DT B-NP+industry NN I-NP+with IN B-PP+a DT B-NP+strong JJ I-NP+sales NNS I-NP+performance NN I-NP+in IN B-PP+the DT B-NP+human NN I-NP+and CC I-NP+animal NN I-NP+health-products NNS I-NP+segment VBP I-NP+. . O++A DT B-NP+stronger JJR I-NP+U.S. NNP I-NP+dollar NN I-NP+reduced VBD B-VP+third-quarter JJ B-NP+and CC I-NP+first-nine-month JJ I-NP+sales NNS I-NP+growth NN I-NP+2 CD B-NP+% NN I-NP+and CC I-NP+3 CD I-NP+% NN I-NP+, , O+respectively RB B-ADVP+. . O++International JJ B-NP+sales NNS I-NP+accounted VBD B-VP+for IN B-PP+47 CD B-NP+% NN I-NP+of IN B-PP+total JJ B-NP+company NN I-NP+sales NNS I-NP+for IN B-PP+the DT B-NP+nine CD I-NP+months NNS I-NP+, , O+compared VBN B-PP+with IN B-PP+50 CD B-NP+% NN I-NP+a DT B-NP+year NN I-NP+earlier RBR B-ADVP+. . O++Sales NNS B-NP+for IN B-PP+the DT B-NP+quarter NN I-NP+rose VBD B-VP+to TO B-PP+$ $ B-NP+1.63 CD I-NP+billion CD I-NP+from IN B-PP+$ $ B-NP+1.47 CD I-NP+billion CD I-NP+. . O++Mevacor NNP B-NP+, , O+Merck NNP B-NP+'s POS B-NP+new JJ I-NP+cholesterol-lowering JJ I-NP+drug NN I-NP+, , O+had VBD B-VP+higher JJR B-NP+sales NNS I-NP+than IN B-SBAR+any DT B-NP+other JJ I-NP+prescription NN I-NP+medicine NN I-NP+has VBZ B-VP+ever RB I-VP+achieved VBN I-VP+in IN B-PP+the DT B-NP+U.S. NNP I-NP+in IN B-PP+the DT B-NP+year NN I-NP+following VBG B-PP+introduction NN B-NP+, , O+the DT B-NP+company NN I-NP+said VBD B-VP+. . O++The DT B-NP+drug NN I-NP+was VBD B-VP+introduced VBN I-VP+in IN B-PP+West NNP B-NP+Germany NNP I-NP+this DT B-NP+year NN I-NP+. . O++Intense JJ B-NP+competition NN I-NP+, , O+however RB B-ADVP+, , O+led VBN B-VP+to TO B-PP+unit NN B-NP+sales NNS I-NP+declines NNS I-NP+for IN B-PP+a DT B-NP+group NN I-NP+of IN B-PP+Merck NNP B-NP+'s POS B-NP+established VBN I-NP+human NN I-NP+and CC I-NP+animal-health NN I-NP+products NNS I-NP+, , O+including VBG B-PP+Aldomet NNP B-NP+and CC I-NP+Indocin NNP I-NP+. . O++In IN B-PP+New NNP B-NP+York NNP I-NP+Stock NNP I-NP+Exchange NNP I-NP+composite JJ I-NP+trading NN I-NP+yesterday NN B-NP+, , O+Merck NNP B-NP+shares NNS I-NP+closed VBD B-VP+at IN B-PP+$ $ B-NP+75.25 CD I-NP+, , O+up IN B-ADVP+50 CD B-NP+cents NNS I-NP+. . O++Warner-Lambert NNP B-NP+Co NNP I-NP+. . O++Warner-Lambert NNP B-NP+, , O+Morris NNP B-NP+Plains NNP I-NP+, , O+N.J. NNP B-NP+, , O+reported VBD B-VP+sales NNS B-NP+that WDT B-NP+were VBD B-VP+a DT B-NP+record NN I-NP+for IN B-PP+any DT B-NP+quarter NN I-NP+and CC O+the DT B-NP+eighth JJ I-NP+quarter NN I-NP+in IN B-PP+a DT B-NP+row NN I-NP+of IN B-PP+20 CD B-NP+% NN I-NP+or CC I-NP+more RBR I-NP+per-share JJ I-NP+earnings NNS I-NP+growth NN I-NP+. . O++Spurred VBN B-VP+by IN B-PP+growth NN B-NP+in IN B-PP+world-wide JJ B-NP+sales NNS I-NP+of IN B-PP+the DT B-NP+company NN I-NP+'s POS B-NP+prescription NN I-NP+drugs NNS I-NP+, , O+Warner-Lambert NNP B-NP+said VBD B-VP+1989 CD B-NP+will MD B-VP+be VB I-VP+the DT B-NP+best JJS I-NP+year NN I-NP+in IN B-PP+its PRP$ B-NP+history NN I-NP+, , O+with IN B-SBAR+per-share JJ B-NP+earnings NNS I-NP+expected VBN B-VP+to TO I-VP+increase VB I-VP+more JJR B-NP+than IN I-NP+20 CD I-NP+% NN I-NP+to TO B-PP+about RB B-NP+$ $ I-NP+6.10 CD I-NP+. . O++Sales NNS B-NP+for IN B-PP+the DT B-NP+quarter NN I-NP+rose VBD B-VP+to TO B-PP+$ $ B-NP+1.11 CD I-NP+billion CD I-NP+from IN B-PP+$ $ B-NP+1.03 CD I-NP+billion CD I-NP+. . O++Prescription-drug NN B-NP+world-wide JJ I-NP+sales NNS I-NP+rose VBD B-VP+9 CD B-NP+% NN I-NP+in IN B-PP+the DT B-NP+quarter NN I-NP+to TO B-PP+$ $ B-NP+340 CD I-NP+million CD I-NP+; : O+U.S. NNP B-NP+sales NNS I-NP+rose VBD B-VP+15 CD B-NP+% NN I-NP+. . O++The DT B-NP+segment NN I-NP+'s POS B-NP+growth NN I-NP+was VBD B-VP+led VBN I-VP+by IN B-PP+sales NNS B-NP+of IN B-PP+the DT B-NP+cardiovascular JJ I-NP+drugs NNS I-NP+Lopid NNP B-NP+, , O+a DT B-NP+lipid NN I-NP+regulator NN I-NP+, , O+and CC O+Dilzem NNP B-NP+, , O+a DT B-NP+calcium NN I-NP+channel NN I-NP+blocker NN I-NP+. . O++World-wide JJ B-NP+sales NNS I-NP+of IN B-PP+Warner-Lambert NNP B-NP+'s POS B-NP+non-prescription JJ I-NP+health-care NN I-NP+products NNS I-NP+, , O+such JJ B-PP+as IN I-PP+Halls NNP B-NP+cough NN I-NP+tablets NNS I-NP+, , O+Rolaids NNP B-NP+antacid NN I-NP+, , O+and CC O+Lubriderm NNP B-NP+skin NN I-NP+lotion NN I-NP+, , O+increased VBN B-VP+3 CD B-NP+% NN I-NP+to TO B-PP+$ $ B-NP+362 CD I-NP+million CD I-NP+in IN B-PP+the DT B-NP+third JJ I-NP+quarter NN I-NP+; : O+U.S. NNP B-NP+sales NNS I-NP+rose VBD B-VP+5 NN B-NP+% NN I-NP+. . O++Confectionery JJ B-NP+products NNS I-NP+sales NNS I-NP+also RB B-ADVP+had VBD B-VP+strong JJ B-NP+growth NN I-NP+in IN B-PP+the DT B-NP+quarter NN I-NP+. . O++World-wide JJ B-NP+sales NNS I-NP+of IN B-PP+Trident NNP B-NP+gum NN I-NP+, , O+Certs NNP B-NP+breath NN I-NP+mints NNS I-NP+, , O+and CC O+Clorets NNP B-NP+gum NN I-NP+and CC I-NP+breath NN I-NP+mints NNS I-NP+, , O+increased VBN B-VP+12 CD B-NP+% NN I-NP+to TO B-PP+$ $ B-NP+277 CD I-NP+million CD I-NP+. . O++Warner-Lambert NNP B-NP+shares NNS I-NP+closed VBD B-VP+at IN B-PP+$ $ B-NP+109.50 CD I-NP+a DT B-NP+share NN I-NP+, , O+up IN B-ADVP+$ $ B-NP+1.50 CD I-NP+, , O+in IN B-PP+Big NNP B-NP+Board NNP I-NP+composite JJ I-NP+trading NN I-NP+yesterday NN B-NP+. . O++Eli NNP B-NP+Lilly NNP I-NP+& CC I-NP+Co NNP I-NP+. . O++Lilly NNP B-NP+attributed VBD B-VP+record NN B-NP+third-quarter JJ I-NP+and CC I-NP+nine-month JJ I-NP+results NNS I-NP+to TO B-PP+world-wide JJ B-NP+gains NNS I-NP+for IN B-PP+pharmaceuticals NNS B-NP+, , O+medical JJ B-NP+instruments NNS I-NP+and CC O+plant-science NN B-NP+products NNS I-NP+despite IN B-PP+poor JJ B-NP+exchange NN I-NP+rates NNS I-NP+for IN B-PP+the DT B-NP+dollar NN I-NP+that WDT B-NP+slowed VBD B-VP+sales NNS B-NP+abroad RB B-ADVP+. . O++Earnings NNS B-NP+continued VBD B-VP+to TO I-VP+pace VB I-VP+sales NNS B-NP+because IN B-PP+of IN I-PP+a DT B-NP+lower JJR I-NP+tax NN I-NP+rate NN I-NP+, , O+profit NN B-NP+from IN B-PP+the DT B-NP+renegotiation NN I-NP+of IN B-PP+the DT B-NP+debt NN I-NP+instrument NN I-NP+received VBD B-VP+from IN B-PP+Faberge NNP B-NP+Inc. NNP I-NP+in IN B-PP+connection NN B-NP+with IN B-PP+Lilly NNP B-NP+'s POS B-NP+sale NN I-NP+of IN B-PP+Elizabeth NNP B-NP+Arden NNP I-NP+Inc. NNP I-NP+in IN B-PP+1987 CD B-NP+, , O+and CC O+net JJ B-NP+proceeds NNS I-NP+from IN B-PP+the DT B-NP+settlement NN I-NP+of IN B-PP+patent NN B-NP+litigation NN I-NP+at IN B-PP+Lilly NNP B-NP+'s POS B-NP+Hybritech NNP I-NP+Inc. NNP I-NP+unit NN I-NP+. . O++Third-quarter JJ B-NP+sales NNS I-NP+of IN B-PP+the DT B-NP+Indianapolis NNP I-NP+, , I-NP+Ind. NNP I-NP+, , I-NP+company NN I-NP+rose VBD B-VP+11 CD B-NP+% NN I-NP+to TO B-PP+$ $ B-NP+1.045 CD I-NP+billion CD I-NP+from IN B-PP+$ $ B-NP+940.6 CD I-NP+million CD I-NP+. . O++Nine-month JJ B-NP+sales NNS I-NP+grew VBD B-VP+12 CD B-NP+% NN I-NP+to TO B-PP+$ $ B-NP+3.39 CD I-NP+billion CD I-NP+from IN B-PP+$ $ B-NP+3.03 CD I-NP+billion CD I-NP+a DT B-NP+year NN I-NP+earlier RBR B-ADVP+. . O++Sales NNS B-NP+of IN B-PP+Prozac NNP B-NP+, , O+an DT B-NP+anti-depressant NN I-NP+, , O+led VBN B-VP+drug-sales NNS B-NP+increases NNS I-NP+. . O++Higher JJR B-NP+sales NNS I-NP+of IN B-PP+pesticides NNS B-NP+and CC O+other JJ B-NP+plant-science NN I-NP+products NNS I-NP+more JJR B-VP+than IN I-VP+offset VB I-VP+a DT B-NP+slight JJ I-NP+decline NN I-NP+in IN B-PP+the DT B-NP+sales NNS I-NP+of IN B-PP+animal-health NN B-NP+products NNS I-NP+to TO B-VP+fuel VB I-VP+the DT B-NP+increase NN I-NP+in IN B-PP+world-wide JJ B-NP+agricultural JJ I-NP+product NN I-NP+sales NNS I-NP+, , O+Lilly NNP B-NP+said VBD B-VP+. . O++Advanced NNP B-NP+Cardiovascular NNP I-NP+Systems NNP I-NP+Inc. NNP I-NP+and CC O+Cardiac NNP B-NP+Pacemakers NNPS I-NP+Inc. NNP I-NP+units NNS I-NP+led VBD B-VP+growth NN B-NP+in IN B-PP+the DT B-NP+medical-instrument JJ I-NP+systems NNS I-NP+division NN I-NP+. . O++Lilly NNP B-NP+shares NNS I-NP+closed VBD B-VP+yesterday NN B-NP+in IN B-PP+composite JJ B-NP+trading NN I-NP+on IN B-PP+the DT B-NP+Big NNP I-NP+Board NNP I-NP+at IN B-PP+$ $ B-NP+62.25 CD I-NP+, , O+down RB B-ADVP+12.5 CD B-NP+cents NNS I-NP+. . O++Reuben NNP B-NP+Mark NNP I-NP
+ data/example.features view
@@ -0,0 +1,13 @@+Cat [ Index (Cell 0 0)+ , Lower (Cell 0 0)+ , WordShape (Cell 0 0)+ , Cell 0 1+ , Cell 0 2+ , Cell 0 3+ , Cell -2 0+ , Cell -1 0+ , Cell 1 0+ , Cell 2 0 + , Cart (Cell -2 0) (Cell -1 0)+ ]+
+ data/test.conll view
@@ -0,0 +1,1000 @@+, , O+chairman NN B-NP+of IN B-PP+Colgate-Palmolive NNP B-NP+Co. NNP I-NP+, , O+said VBD B-VP+he PRP B-NP+is VBZ B-VP+`` `` B-ADJP+comfortable JJ I-ADJP+'' '' O+with IN B-PP+analysts NNS B-NP+' POS B-NP+estimates NNS I-NP+that IN B-SBAR+third-quarter JJ B-NP+earnings NNS I-NP+rose VBD B-VP+to TO B-PP+between IN B-NP+95 CD I-NP+cents NNS I-NP+and CC I-NP+$ $ I-NP+1.05 CD I-NP+a DT B-NP+share NN I-NP+. . O++That DT B-NP+compares VBZ B-VP+with IN B-PP+per-share JJ B-NP+earnings NNS I-NP+from IN B-PP+continuing VBG B-NP+operations NNS I-NP+of IN B-PP+69 CD B-NP+cents NNS I-NP+the DT B-NP+year NN I-NP+earlier RBR B-ADVP+; : O+including VBG B-PP+discontinued VBN B-NP+operations NNS I-NP+, , O+per-share NN B-NP+was VBD B-VP+88 CD B-NP+cents NNS I-NP+a DT B-NP+year NN I-NP+ago RB B-ADVP+. . O++The DT B-NP+per-share JJ I-NP+estimates NNS I-NP+mean VBP B-VP+the DT B-NP+consumer-products NNS I-NP+company NN I-NP+'s POS B-NP+net JJ I-NP+income NN I-NP+, , O+increased VBN B-VP+to TO B-PP+between IN B-NP+$ $ I-NP+69.5 CD I-NP+million CD I-NP+and CC I-NP+$ $ I-NP+76 CD I-NP+million CD I-NP+, , O+from IN B-PP+$ $ B-NP+47.1 CD I-NP+million CD I-NP+the DT B-NP+year-before JJ I-NP+period NN I-NP+. . O++Analysts NNS B-NP+estimate VBP B-VP+Colgate NNP B-NP+'s POS B-NP+world-wide JJ I-NP+third-quarter JJ I-NP+sales NNS I-NP+rose VBD B-VP+about IN B-NP+8 CD I-NP+% NN I-NP+to TO B-PP+$ $ B-NP+1.29 CD I-NP+billion CD I-NP+. . O++Mr. NNP B-NP+Mark NNP I-NP+attributed VBD B-VP+the DT B-NP+earnings NNS I-NP+growth NN I-NP+to TO B-PP+strong JJ B-NP+sales NNS I-NP+in IN B-PP+Latin NNP B-NP+America NNP I-NP+, , O+Asia NNP B-NP+and CC O+Europe NNP B-NP+. . O++Results NNS B-NP+were VBD B-VP+also RB I-VP+bolstered VBN I-VP+by IN B-PP+`` `` B-NP+a DT I-NP+very RB I-NP+meaningful JJ I-NP+'' '' I-NP+increase NN I-NP+in IN B-PP+operating VBG B-NP+profit NN I-NP+by IN B-PP+Colgate NNP B-NP+'s POS B-NP+U.S. NNP I-NP+business NN I-NP+, , O+Mr. NNP B-NP+Mark NNP I-NP+said VBD B-VP+. . O++Operating NN B-NP+profit NN I-NP+at IN B-PP+Colgate NNP B-NP+'s POS B-NP+U.S. NNP I-NP+household NN I-NP+products NNS I-NP+and CC I-NP+personal-care JJ I-NP+businesses NNS I-NP+jumped VBD B-VP+25 CD B-NP+% NN I-NP+in IN B-PP+the DT B-NP+quarter NN I-NP+, , O+Mr. NNP B-NP+Mark NNP I-NP+added VBD B-VP+. . O++He PRP B-NP+said VBD B-VP+the DT B-NP+improvement NN I-NP+was VBD B-VP+a DT B-NP+result NN I-NP+of IN B-PP+cost NN B-NP+savings NNS I-NP+achieved VBN B-VP+by IN B-PP+consolidating VBG B-VP+manufacturing NN B-NP+operations NNS B-NP+, , O+blending VBG B-VP+two CD B-NP+sales NNS I-NP+organizations NNS I-NP+and CC O+focusing VBG B-VP+more RBR B-ADVP+carefully RB I-ADVP+the DT B-NP+company NN I-NP+'s POS B-NP+promotional JJ I-NP+activities NNS I-NP+. . O++The DT B-NP+estimated VBN I-NP+improvement NN I-NP+in IN B-PP+Colgate NNP B-NP+'s POS B-NP+U.S. NNP I-NP+operations NNS I-NP+took VBD B-VP+some DT B-NP+analysts NNS I-NP+by IN B-PP+surprise NN B-NP+. . O++Colgate NNP B-NP+'s POS B-NP+household NN I-NP+products NNS I-NP+business NN I-NP+, , O+which WDT B-NP+includes VBZ B-VP+such JJ B-NP+brands NNS I-NP+as IN B-PP+Fab NNP B-NP+laundry NN I-NP+detergent NN I-NP+and CC O+Ajax NNP B-NP+cleanser NN I-NP+, , O+has VBZ B-VP+been VBN I-VP+a DT B-NP+weak JJ I-NP+performer NN I-NP+. . O++Analysts NNS B-NP+estimate VBP B-VP+Colgate NNP B-NP+'s POS B-NP+sales NNS I-NP+of IN B-PP+household NN B-NP+products NNS I-NP+in IN B-PP+the DT B-NP+U.S. NNP I-NP+were VBD B-VP+flat JJ B-ADJP+for IN B-PP+the DT B-NP+quarter NN I-NP+, , O+and CC O+they PRP B-NP+estimated VBD B-VP+operating VBG B-NP+margins NNS I-NP+at IN B-PP+only RB B-NP+1 CD I-NP+% NN I-NP+to TO I-NP+3 CD I-NP+% NN I-NP+. . O++`` `` O+If IN B-SBAR+you PRP B-NP+could MD B-VP+say VB I-VP+their PRP$ B-NP+business NN I-NP+in IN B-PP+the DT B-NP+U.S. NNP I-NP+was VBD B-VP+mediocre JJ B-ADJP+, , O+but CC O+great JJ B-ADJP+everywhere RB B-ADVP+else RB I-ADVP+, , O+that WDT B-NP+would MD B-VP+be VB I-VP+fine JJ B-ADJP+, , O+'' '' O+says VBZ B-VP+Bonita NNP B-NP+Austin NNP I-NP+, , O+an DT B-NP+analyst NN I-NP+with IN B-PP+Wertheim NNP B-NP+Schroder NNP I-NP+& CC I-NP+Co NNP I-NP+. . O++`` `` O+But CC O+it PRP B-NP+'s VBZ B-VP+not RB O+mediocre JJ B-ADJP+, , O+it PRP B-NP+'s VBZ B-VP+a DT B-NP+real JJ I-NP+problem NN I-NP+. . O+'' '' O++Mr. NNP B-NP+Mark NNP I-NP+conceded VBD B-VP+that IN O+Colgate NNP B-NP+'s POS B-NP+domestic JJ I-NP+business NN I-NP+, , O+apart RB B-ADVP+from IN B-PP+its PRP$ O+highly RB O+profitable JJ O+Hill NNP B-NP+'s POS B-NP+Pet NNP I-NP+Products NNPS I-NP+unit NN I-NP+, , O+has VBZ B-VP+lagged VBN I-VP+. . O++`` `` O+We PRP B-NP+'ve VBP B-VP+done VBN I-VP+a DT B-NP+lot NN I-NP+to TO B-VP+improve VB I-VP+-LCB- ( B-NP+U.S. NNP I-NP+. . I-NP+-RCB- ) I-NP+results NNS I-NP+, , O+and CC O+a DT B-NP+lot NN I-NP+more JJR I-NP+will MD B-VP+be VB I-VP+done VBN I-VP+, , O+'' '' O+Mr. NNP B-NP+Mark NNP I-NP+said VBD B-VP+. . O++`` `` O+Improving VBG B-VP+profitability NN B-NP+of IN B-PP+U.S. NNP B-NP+operations NNS I-NP+is VBZ B-VP+an DT B-NP+extremely RB I-NP+high JJ I-NP+priority NN I-NP+in IN B-PP+the DT B-NP+company NN I-NP+. . O+'' '' O++To TO B-VP+focus VB I-VP+on IN B-PP+its PRP$ B-NP+global JJ I-NP+consumer-products NNS I-NP+business NN I-NP+, , O+Colgate NNP B-NP+sold VBD B-VP+its PRP$ B-NP+Kendall NNP I-NP+health-care NN I-NP+business NN I-NP+in IN B-PP+1988 CD B-NP+. . O++H. NNP B-NP+Anthony NNP I-NP+Ittleson NNP I-NP+was VBD B-VP+elected VBN I-VP+a DT B-NP+director NN I-NP+of IN B-PP+this DT B-NP+company NN I-NP+, , O+which WDT B-NP+primarily RB B-ADVP+has VBZ B-VP+interests NNS B-NP+in IN B-PP+radio NN B-NP+and CC I-NP+television NN I-NP+stations NNS I-NP+, , O+increasing VBG B-VP+the DT B-NP+number NN I-NP+of IN B-PP+seats NNS B-NP+to TO B-PP+five CD B-NP+. . O++Osborn NNP B-NP+also RB B-ADVP+operates VBZ B-VP+Muzak NNP B-NP+franchises NNS I-NP+, , O+entertainment NN B-NP+properties NNS I-NP+and CC O+small JJ B-NP+cable-television NN I-NP+systems NNS I-NP+. . O++Mr. NNP B-NP+Ittleson NNP I-NP+is VBZ B-VP+executive NN B-NP+, , O+special JJ B-NP+projects NNS I-NP+, , O+at IN B-PP+CIT NNP B-NP+Group NNP I-NP+Holdings NNP I-NP+Inc. NNP I-NP+, , O+which WDT B-NP+is VBZ B-VP+controlled VBN I-VP+by IN B-PP+Manufacturers NNP B-NP+Hanover NNP I-NP+Corp NNP I-NP+. . O++The DT B-NP+Boston NNP I-NP+Globe NNP I-NP+says VBZ B-VP+its PRP$ B-NP+newly RB I-NP+redesigned VBN I-NP+pages NNS I-NP+have VBP B-VP+a DT B-NP+`` `` I-NP+crisper NN I-NP+'' '' I-NP+look VB I-NP+with IN B-PP+revamped VBN B-NP+fixtures NNS I-NP+aimed VBN B-VP+at IN B-PP+making VBG B-VP+the DT B-NP+paper NN I-NP+`` `` O+more RBR B-ADJP+consistent JJ I-ADJP+'' '' O+and CC O+`` `` O+easier JJR B-ADJP+to TO B-VP+read VB I-VP+. . O+'' '' O++Maybe RB B-ADVP+so RB I-ADVP+-- : O+if IN B-SBAR+you PRP B-NP+can MD B-VP+find VB I-VP+where WRB B-ADVP+your PRP$ B-NP+favorite JJ I-NP+writer NN I-NP+went VBD B-VP+. . O++Beantown NNP B-NP+scribes NNS I-NP+, , O+who WP B-NP+spare VB B-VP+no DT B-NP+invective NN I-NP+when WRB B-ADVP+taking VBG B-VP+on IN B-PRT+local JJ B-NP+luminaries NNS I-NP+such JJ B-PP+as IN I-PP+Michael NNP B-NP+`` `` I-NP+Pee NNP I-NP+Wee NNP I-NP+'' '' I-NP+Dukakis NNP I-NP+, , O+or CC O+New NNP B-NP+England NNP I-NP+Patriots NNPS I-NP+Coach NNP I-NP+Raymond NNP I-NP+`` `` I-NP+Rev. NNP I-NP+Ray NNP I-NP+'' '' I-NP+Berry NNP I-NP+, , O+yesterday NN B-NP+poured VBD B-VP+ridicule NN B-NP+on IN B-PP+new JJ B-NP+drawings NNS I-NP+of IN B-PP+Globe NNP B-NP+columnists NNS I-NP+that WDT B-NP+replaced VBD B-VP+old JJ B-NP+photos NNS I-NP+in IN B-PP+the DT B-NP+revamped VBN I-NP+pages NNS I-NP+this DT B-NP+week NN I-NP+. . O++By IN B-PP+late JJ B-NP+last JJ I-NP+night NN I-NP+, , O+Globe NNP B-NP+Managing NNP I-NP+Editor NNP I-NP+Thomas NNP B-NP+Mulvoy NNP I-NP+, , O+bending VBG B-VP+to TO B-PP+the DT B-NP+will MD I-NP+of IN B-PP+his PRP$ B-NP+troops NNS I-NP+, , O+scrapped VBD B-VP+the DT B-NP+new JJ I-NP+drawings NNS I-NP+. . O++For IN B-PP+a DT B-NP+few JJ I-NP+days NNS I-NP+at IN B-PP+least JJS B-ADJP+, , O+he PRP B-NP+says VBZ B-VP+, , O+no DT B-NP+pictures NNS I-NP+or CC I-NP+drawings NNS I-NP+of IN B-PP+any DT B-NP+kind NN I-NP+will MD B-VP+adorn VB I-VP+the DT B-NP+columns NNS I-NP+. . O++Trouble NN B-NP+was VBD B-VP+, , O+nobody NN B-NP+thought VBD B-VP+they PRP B-NP+looked VBD B-VP+right NN B-ADJP+. . O++Globe NNP B-NP+columnist NN I-NP+Mike NNP I-NP+Barnicle NNP I-NP+-- : O+in IN B-PP+the DT B-NP+second JJ I-NP+attack NN I-NP+on IN B-PP+his PRP$ B-NP+employer NN I-NP+in IN B-PP+as IN B-NP+many JJ I-NP+weeks NNS I-NP+-- : O+averred VBD B-VP+that IN B-SBAR+his PRP$ B-NP+shadowy JJ I-NP+countenance NN I-NP+was VBD B-VP+so RB B-ADJP+bad JJ I-ADJP+, , O+it PRP B-NP+looked VBD B-VP+`` `` O+like IN B-PP+a DT B-NP+face NN I-NP+you PRP B-NP+'d MD B-VP+find VB I-VP+on IN B-PP+a DT B-NP+bottle NN I-NP+of IN B-PP+miracle NN B-NP+elixir NN I-NP+that WDT B-NP+promises VBZ B-VP+to TO I-VP+do VB I-VP+away RB B-ADVP+with IN B-PP+diarrhea NN B-NP+in IN B-PP+our PRP$ B-NP+lifetime NN I-NP+. . O+'' '' O++Mr. NNP B-NP+Barnicle NNP I-NP+reminded VBD B-VP+readers NNS B-NP+that IN B-SBAR+he PRP B-NP+still RB B-ADVP+has VBZ B-VP+n't RB I-VP+forgiven VBN I-VP+Globe NNP B-NP+management NN I-NP+for IN B-PP+questioning VBG B-VP+a DT B-NP+$ $ I-NP+20 CD I-NP+expense NN I-NP+chit NN I-NP+he PRP B-NP+submitted VBD B-VP+for IN B-PP+parking VBG B-VP+his PRP$ B-NP+car NN I-NP+while IN B-SBAR+chasing VBG B-VP+a DT B-NP+story NN I-NP+. . O++`` `` O+I PRP B-NP+thought VBD B-VP+-LCB- ( O+the DT B-NP+drawing VBG I-NP+-RCB- ) O+a DT B-NP+cross NN I-NP+between IN B-PP+someone NN B-NP+you PRP B-NP+'d MD B-VP+spot VB I-VP+whipping VBG I-VP+open JJ I-VP+his PRP$ B-NP+trench NN I-NP+coat NN I-NP+... : O+or CC O+a DT B-NP+guy NN I-NP+who WP B-NP+boasted VBD B-VP+he PRP B-NP+'d MD B-VP+been VBN I-VP+Charles NNP B-NP+Manson NNP I-NP+'s POS B-NP+roommate NN I-NP+for IN B-PP+the DT B-NP+last JJ I-NP+19 CD I-NP+years NNS I-NP+, , O+'' '' O+he PRP B-NP+said VBD B-VP+. . O++Mr. NNP B-NP+Barnicle NNP I-NP+was VBD B-VP+hardly RB B-ADJP+kinder JJR I-ADJP+to TO B-PP+the DT B-NP+renderings NNS I-NP+of IN B-PP+colleagues NNS B-NP+Michael NNP B-NP+Madden NNP I-NP+-LRB- ( O+`` `` O+appears VBZ B-VP+to TO I-VP+be VB I-VP+a DT B-NP+pervert NN I-NP+'' '' O+-RRB- ) O+, , O+Will MD B-NP+McDonough NNP I-NP+-LRB- ( O+`` `` O+looks VBZ B-VP+as IN B-SBAR+if IN I-SBAR+he PRP B-NP+drove VBD B-VP+for IN B-PP+Abe NNP B-NP+Lincoln NNP I-NP+'' '' O+-RRB- ) O+or CC O+Bella NNP B-NP+English NNP I-NP+, , O+whose WP$ B-NP+`` `` I-NP+little JJ I-NP+girl NN I-NP+now RB B-ADVP+screams VBZ B-VP+hysterically RB B-ADVP+every DT B-NP+time NN I-NP+she PRP B-NP+sees VBZ B-VP+a DT B-NP+newspaper NN I-NP+. . O+'' '' O++Lynn NNP B-NP+Staley NNP I-NP+, , O+the DT B-NP+Globe NNP I-NP+'s POS B-NP+assistant NN I-NP+managing VBG I-NP+editor NN I-NP+for IN B-PP+design NN B-NP+, , O+acknowledges VBZ B-VP+that IN B-SBAR+the DT B-NP+visages NNS I-NP+were VBD B-VP+`` `` O+on IN B-PP+the DT B-NP+low JJ I-NP+end NN I-NP+of IN B-PP+the DT B-NP+likeness NN I-NP+spectrum NN I-NP+. . O+'' '' O++Rival NNP B-NP+Boston NNP I-NP+Herald NNP I-NP+columnist NN I-NP+Howie NNP I-NP+Carr NNP I-NP+, , O+who WP B-NP+usually RB B-ADVP+rails VBZ B-VP+at IN B-PP+Statehouse NN B-NP+`` `` I-NP+hacks NNS I-NP+'' '' I-NP+and CC I-NP+nepotism NN I-NP+, , O+argued VBD B-VP+that IN B-SBAR+the DT B-NP+new JJ I-NP+drawings NNS I-NP+were VBD B-VP+designed VBN I-VP+to TO B-VP+hide VB I-VP+Mr. NNP B-NP+Madden NNP I-NP+'s POS B-NP+`` `` O+rapidly RB B-ADJP+growing VBG I-ADJP+forehead NN B-NP+'' '' O+and CC O+the DT B-NP+facial JJ I-NP+defects NNS I-NP+of IN B-PP+`` `` B-NP+chinless JJ I-NP+'' '' I-NP+Dan NNP I-NP+Shaughnessy NNP I-NP+, , O+a DT B-NP+Globe NNP I-NP+sports NNS I-NP+columnist NN I-NP+. . O++`` `` O+But CC O+think VBP B-VP+of IN B-PP+the DT B-NP+money NN I-NP+you PRP B-NP+, , O+the DT B-NP+reader NN I-NP+, , O+will MD B-VP+save VB I-VP+on IN B-PP+Halloween NNP B-NP+, , O+'' '' O+said VBD B-VP+Mr. NNP B-NP+Barnicle NNP I-NP+. . O++`` `` O+Instead RB B-PP+of IN I-PP+buying VBG B-VP+masks NNS B-NP+for IN B-PP+your PRP$ B-NP+kids NNS I-NP+, , O+just RB B-ADVP+cut VB B-VP+out RP B-PRT+the IN B-NP+columnists NNS I-NP+' POS B-NP+pictures NNS I-NP+... : O+. . O++Deeply RB B-ADJP+ingrained JJ I-ADJP+in IN B-PP+both DT O+the DT B-NP+book NN I-NP+review NN I-NP+`` `` O+Kissing VBG B-VP+Nature NNP B-NP+Good-bye UH B-NP+'' '' O+by IN B-PP+Stephen NNP B-NP+MacDonald NNP I-NP+-LRB- ( O+Leisure NNP B-NP+& CC I-NP+Arts NNP I-NP+, , O+Sept. NNP B-NP+27 CD I-NP+-RRB- ) O+and CC O+the DT B-NP+books NNS I-NP+reviewed VBN B-VP+is VBZ B-VP+the DT B-NP+assumption NN I-NP+that IN B-SBAR+global JJ B-NP+warming NN I-NP+is VBZ B-VP+entirely RB B-ADVP+a DT B-NP+result NN I-NP+of IN B-PP+human JJ B-NP+activity NN I-NP+. . O++Is VBZ O+such JJ B-NP+a DT I-NP+view NN I-NP+justified VBN B-VP+? . O++In IN B-PP+the DT B-NP+absence NN I-NP+of IN B-PP
+ data/train.conll view
@@ -0,0 +1,1000 @@+Rockwell NNP B-NP+International NNP I-NP+Corp. NNP I-NP+'s POS B-NP+Tulsa NNP I-NP+unit NN I-NP+said VBD B-VP+it PRP B-NP+signed VBD B-VP+a DT B-NP+tentative JJ I-NP+agreement NN I-NP+extending VBG B-VP+its PRP$ B-NP+contract NN I-NP+with IN B-PP+Boeing NNP B-NP+Co. NNP I-NP+to TO B-VP+provide VB I-VP+structural JJ B-NP+parts NNS I-NP+for IN B-PP+Boeing NNP B-NP+'s POS B-NP+747 CD I-NP+jetliners NNS I-NP+. . O++Rockwell NNP B-NP+said VBD B-VP+the DT B-NP+agreement NN I-NP+calls VBZ B-VP+for IN B-SBAR+it PRP B-NP+to TO B-VP+supply VB I-VP+200 CD B-NP+additional JJ I-NP+so-called JJ I-NP+shipsets NNS I-NP+for IN B-PP+the DT B-NP+planes NNS I-NP+. . O++These DT B-NP+include VBP B-VP+, , O+among IN B-PP+other JJ B-NP+parts NNS I-NP+, , O+each DT B-NP+jetliner NN I-NP+'s POS B-NP+two CD I-NP+major JJ I-NP+bulkheads NNS I-NP+, , O+a DT B-NP+pressure NN I-NP+floor NN I-NP+, , O+torque NN B-NP+box NN I-NP+, , O+fixed VBN B-NP+leading VBG I-NP+edges NNS I-NP+for IN B-PP+the DT B-NP+wings NNS I-NP+and CC O+an DT B-NP+aft JJ I-NP+keel NN I-NP+beam NN I-NP+. . O++Under IN B-PP+the DT B-NP+existing VBG I-NP+contract NN I-NP+, , O+Rockwell NNP B-NP+said VBD B-VP+, , O+it PRP B-NP+has VBZ B-VP+already RB I-VP+delivered VBN I-VP+793 CD B-NP+of IN B-PP+the DT B-NP+shipsets NNS I-NP+to TO B-PP+Boeing NNP B-NP+. . O++Rockwell NNP B-NP+, , O+based VBN B-VP+in IN B-PP+El NNP B-NP+Segundo NNP I-NP+, , O+Calif. NNP B-NP+, , O+is VBZ B-VP+an DT B-NP+aerospace NN I-NP+, , I-NP+electronics NNS I-NP+, , I-NP+automotive JJ I-NP+and CC I-NP+graphics NNS I-NP+concern VBP I-NP+. . O++Frank NNP B-NP+Carlucci NNP I-NP+III NNP I-NP+was VBD B-VP+named VBN I-VP+to TO B-PP+this DT B-NP+telecommunications NNS I-NP+company NN I-NP+'s POS B-NP+board NN I-NP+, , O+filling VBG B-VP+the DT B-NP+vacancy NN I-NP+created VBN B-VP+by IN B-PP+the DT B-NP+death NN I-NP+of IN B-PP+William NNP B-NP+Sobey NNP I-NP+last JJ B-NP+May NNP I-NP+. . O++Mr. NNP B-NP+Carlucci NNP I-NP+, , O+59 CD B-NP+years NNS I-NP+old JJ B-ADJP+, , O+served VBN B-VP+as IN B-PP+defense NN B-NP+secretary NN I-NP+in IN B-PP+the DT B-NP+Reagan NNP I-NP+administration NN I-NP+. . O++In IN B-PP+January NNP B-NP+, , O+he PRP B-NP+accepted VBD B-VP+the DT B-NP+position NN I-NP+of IN B-PP+vice NN B-NP+chairman NN I-NP+of IN B-PP+Carlyle NNP B-NP+Group NNP I-NP+, , O+a DT B-NP+merchant NN I-NP+banking NN I-NP+concern NN I-NP+. . O++SHEARSON NNP B-NP+LEHMAN NNP I-NP+HUTTON NNP I-NP+Inc NNP I-NP+. . O++Thomas NNP B-NP+E. NNP I-NP+Meador NNP I-NP+, , O+42 CD B-NP+years NNS I-NP+old JJ B-ADJP+, , O+was VBD B-VP+named VBN I-VP+president NN B-NP+and CC O+chief JJ B-NP+operating VBG I-NP+officer NN I-NP+of IN B-PP+Balcor NNP B-NP+Co. NNP I-NP+, , O+a DT B-NP+Skokie NNP I-NP+, , I-NP+Ill. NNP I-NP+, , I-NP+subsidiary NN I-NP+of IN B-PP+this DT B-NP+New NNP I-NP+York NNP I-NP+investment NN I-NP+banking NN I-NP+firm NN I-NP+. . O++Balcor NNP B-NP+, , O+which WDT B-NP+has VBZ B-VP+interests NNS B-NP+in IN B-PP+real JJ B-NP+estate NN I-NP+, , O+said VBD B-VP+the DT B-NP+position NN I-NP+is VBZ B-VP+newly RB I-VP+created VBN I-VP+. . O++Mr. NNP B-NP+Meador NNP I-NP+had VBD B-VP+been VBN I-VP+executive JJ B-NP+vice NN I-NP+president NN I-NP+of IN B-PP+Balcor NNP B-NP+. . O++In IN B-PP+addition NN B-NP+to TO B-PP+his PRP$ B-NP+previous JJ I-NP+real-estate NN I-NP+investment NN I-NP+and CC I-NP+asset-management NN I-NP+duties NNS I-NP+, , O+Mr. NNP B-NP+Meador NNP I-NP+takes VBZ B-VP+responsibility NN B-NP+for IN B-PP+development NN B-NP+and CC O+property NN B-NP+management NN I-NP+. . O++Those DT B-NP+duties NNS I-NP+had VBD B-VP+been VBN I-VP+held VBN I-VP+by IN B-PP+Van NNP B-NP+Pell NNP I-NP+, , O+44 CD B-NP+, , O+who WP B-NP+resigned VBD B-VP+as IN B-PP+an DT B-NP+executive JJ I-NP+vice NN I-NP+president NN I-NP+. . O++Shearson NNP B-NP+is VBZ B-VP+about IN B-ADJP+60%-held JJ I-ADJP+by IN B-PP+American NNP B-NP+Express NNP I-NP+Co NNP I-NP+. . O++Great NNP B-NP+American NNP I-NP+Bank NNP I-NP+, , O+citing VBG B-VP+depressed JJ B-NP+Arizona NNP I-NP+real JJ I-NP+estate NN I-NP+prices NNS I-NP+, , O+posted VBD B-VP+a DT B-NP+third-quarter JJ I-NP+loss NN I-NP+of IN B-PP+$ $ B-NP+59.4 CD I-NP+million CD I-NP+, , O+or CC O+$ $ B-NP+2.48 CD I-NP+a DT B-NP+share NN I-NP+. . O++A DT B-NP+year NN I-NP+earlier RBR B-ADVP+, , O+the DT B-NP+savings NNS I-NP+bank VBP I-NP+had VBD B-VP+earnings NNS B-NP+of IN B-PP+$ $ B-NP+8.1 CD I-NP+million CD I-NP+, , O+or CC O+33 CD B-NP+cents NNS I-NP+a DT B-NP+share NN I-NP+. . O++For IN B-PP+the DT B-NP+nine CD I-NP+months NNS I-NP+, , O+it PRP B-NP+had VBD B-VP+a DT B-NP+loss NN I-NP+of IN B-PP+$ $ B-NP+58.3 CD I-NP+million CD I-NP+, , O+or CC O+$ $ B-NP+2.44 CD I-NP+a DT B-NP+share NN I-NP+, , O+after IN B-PP+earnings NNS B-NP+of IN B-PP+$ $ B-NP+29.5 CD I-NP+million CD I-NP+, , O+or CC O+$ $ B-NP+1.20 CD I-NP+a DT B-NP+share NN I-NP+, , O+in IN B-PP+the DT B-NP+1988 CD I-NP+period NN I-NP+. . O++Great NNP B-NP+American NNP I-NP+said VBD B-VP+it PRP B-NP+increased VBD B-VP+its PRP$ B-NP+loan-loss NN I-NP+reserves NNS I-NP+by IN B-PP+$ $ B-NP+93 CD I-NP+million CD I-NP+after IN B-PP+reviewing VBG B-VP+its PRP$ B-NP+loan NN I-NP+portfolio NN I-NP+, , O+raising VBG B-VP+its PRP$ B-NP+total JJ I-NP+loan NN I-NP+and CC I-NP+real JJ I-NP+estate NN I-NP+reserves NNS I-NP+to TO B-PP+$ $ B-NP+217 CD I-NP+million CD I-NP+. . O++Before IN B-PP+the DT B-NP+loan-loss NN I-NP+addition NN I-NP+, , O+it PRP B-NP+said VBD B-VP+, , O+it PRP B-NP+had VBD B-VP+operating VBG B-NP+profit NN I-NP+of IN B-PP+$ $ B-NP+10 CD I-NP+million CD I-NP+for IN B-PP+the DT B-NP+quarter NN I-NP+. . O++The DT B-NP+move NN I-NP+followed VBD B-VP+a DT B-NP+round NN I-NP+of IN B-PP+similar JJ B-NP+increases NNS I-NP+by IN B-PP+other JJ B-NP+lenders NNS I-NP+against IN B-PP+Arizona NNP B-NP+real JJ I-NP+estate NN I-NP+loans NNS I-NP+, , O+reflecting VBG B-VP+a DT B-NP+continuing VBG I-NP+decline NN I-NP+in IN B-PP+that DT B-NP+market NN I-NP+. . O++In IN B-PP+addition NN B-NP+to TO B-PP+the DT B-NP+increased VBN I-NP+reserve NN I-NP+, , O+the DT B-NP+savings NNS I-NP+bank VBP I-NP+took VBD B-VP+a DT B-NP+special JJ I-NP+charge NN I-NP+of IN B-PP+$ $ B-NP+5 NN I-NP+million CD I-NP+representing VBG B-VP+general JJ B-NP+and CC I-NP+administrative JJ I-NP+expenses NNS I-NP+from IN B-PP+staff NN B-NP+reductions NNS I-NP+and CC O+other JJ B-NP+matters NNS I-NP+, , O+and CC O+it PRP B-NP+posted VBD B-VP+a DT B-NP+$ $ I-NP+7.6 CD I-NP+million CD I-NP+reduction NN I-NP+in IN B-PP+expected VBN B-NP+mortgage NN I-NP+servicing NN I-NP+fees NNS I-NP+, , O+reflecting VBG B-VP+the DT B-NP+fact NN I-NP+that IN B-SBAR+more JJR B-NP+borrowers NNS I-NP+are VBP B-VP+prepaying VBG I-VP+their PRP$ B-NP+mortgages NNS I-NP+. . O++Arbitragers NNS B-NP+were VBD B-VP+n't RB O+the DT B-NP+only RB I-NP+big JJ I-NP+losers NNS I-NP+in IN B-PP+the DT B-NP+collapse NN I-NP+of IN B-PP+UAL NNP B-NP+Corp. NNP I-NP+stock NN I-NP+. . O++Look VB B-VP+at IN B-PP+what WP B-NP+happened VBD B-VP+to TO B-PP+UAL NNP B-NP+'s POS B-NP+chairman NN I-NP+, , O+Stephen NNP B-NP+M. NNP I-NP+Wolf NNP I-NP+, , O+and CC O+its PRP$ B-NP+chief JJ I-NP+financial JJ I-NP+officer NN I-NP+, , O+John NNP B-NP+C. NNP I-NP+Pope NNP I-NP+. . O++On IN B-PP+a DT B-NP+day NN I-NP+some DT B-NP+United NNP I-NP+Airlines NNPS I-NP+employees NNS I-NP+wanted VBD B-VP+Mr. NNP B-NP+Wolf NNP I-NP+fired VBD B-VP+and CC O+takeover NN B-NP+stock NN I-NP+speculators NNS I-NP+wanted VBD B-VP+his PRP$ B-NP+scalp NN I-NP+, , O+Messrs. NNP B-NP+Wolf NNP I-NP+and CC I-NP+Pope NNP I-NP+saw VBD B-VP+their PRP$ B-NP+prospective JJ I-NP+personal JJ I-NP+fortunes NNS I-NP+continue VBP B-VP+to TO I-VP+plummet VB I-VP+as IN B-SBAR+shares NNS B-NP+of IN B-PP+UAL NNP B-NP+, , O+United NNP B-NP+'s POS B-NP+parent NN I-NP+company NN I-NP+, , O+dived VBD B-VP+$ $ B-NP+24.875 CD I-NP+on IN B-PP+the DT B-NP+Big NNP I-NP+Board NNP I-NP+to TO B-VP+close VB I-VP+at IN B-PP+$ $ B-NP+198 CD I-NP+. . O++Including VBG O+Monday NNP B-NP+'s POS B-NP+plunge NN I-NP+, , O+that WDT B-NP+has VBZ B-VP+given VBN I-VP+the DT B-NP+two CD I-NP+executives NNS I-NP+paper NN B-NP+losses NNS I-NP+of IN B-PP+$ $ B-NP+49.5 CD I-NP+million CD I-NP+, , O+based VBN B-VP+on IN B-PP+what WP B-NP+they PRP B-NP+would MD B-VP+have VB I-VP+realized VBN I-VP+had VBN O+the DT B-NP+pilots NNS I-NP+and CC I-NP+management-led JJ I-NP+buy-out NN I-NP+of IN B-PP+UAL NNP B-NP+gone VBN B-VP+through IN B-ADVP+at IN B-PP+$ $ B-NP+300 CD I-NP+a DT B-NP+share NN I-NP+. . O++When WRB B-ADVP+bank NN B-NP+financing NN I-NP+for IN B-PP+the DT B-NP+buy-out NN I-NP+collapsed VBD B-VP+last JJ B-NP+week NN I-NP+, , O+so RB B-ADVP+did VBD O+UAL NNP B-NP+'s POS B-NP+stock NN I-NP+. . O++Even RB B-ADVP+if IN B-SBAR+the DT B-NP+banks NNS I-NP+resurrect VBP B-VP+a DT B-NP+financing NN I-NP+package NN I-NP+at IN B-PP+$ $ B-NP+250 CD I-NP+a DT B-NP+share NN I-NP+, , O+the DT B-NP+two CD I-NP+executives NNS I-NP+would MD B-VP+still RB I-VP+get VB I-VP+about RB B-NP+$ $ I-NP+25 CD I-NP+million CD I-NP+less JJR B-NP+than IN B-SBAR+they PRP B-NP+stood VBD B-VP+to TO I-VP+gain VB I-VP+in IN B-PP+the DT B-NP+initial JJ I-NP+transaction NN I-NP+. . O++Mr. NNP B-NP+Wolf NNP I-NP+owns VBZ B-VP+75,000 CD B-NP+UAL NNP I-NP+shares NNS I-NP+and CC O+has VBZ B-VP+options NNS B-NP+to TO B-VP+buy VB I-VP+another DT B-NP+250,000 CD I-NP+at IN B-PP+$ $ B-NP+83.3125 CD I-NP+each DT B-NP+. . O++In IN B-PP+the DT B-NP+$ $ I-NP+300-a-share JJ I-NP+buyout NN I-NP+, , O+that WDT B-NP+totaled VBD B-VP+about RB B-NP+$ $ I-NP+76.7 CD I-NP+million CD I-NP+. . O++By IN B-PP+yesterday NN B-NP+'s POS B-NP+close NN I-NP+of IN B-PP+trading NN B-NP+, , O+it PRP B-NP+was VBD B-VP+good JJ B-ADJP+for IN B-PP+a DT B-NP+paltry JJ I-NP+$ $ I-NP+43.5 CD I-NP+million CD I-NP+. . O++Of IN B-PP+course NN B-NP+, , O+Mr. NNP B-NP+Wolf NNP I-NP+, , O+48 CD B-NP+years NNS I-NP+old JJ B-ADJP+, , O+has VBZ B-VP+some DT B-NP+savings NNS I-NP+. . O++He PRP B-NP+left VBD B-VP+his PRP$ B-NP+last JJ I-NP+two CD I-NP+jobs NNS I-NP+at IN B-PP+Republic NNP B-NP+Airlines NNPS I-NP+and CC O+Flying NNP B-NP+Tiger NNP I-NP+with IN B-PP+combined VBN B-NP+stock-option NN I-NP+gains NNS I-NP+of IN B-PP+about RB B-NP+$ $ I-NP+22 CD I-NP+million CD I-NP+, , O+and CC O+UAL NNP B-NP+gave VBD B-VP+him PRP B-NP+a DT B-NP+$ $ I-NP+15 CD I-NP+million CD I-NP+bonus NN I-NP+when WRB B-ADVP+it PRP B-NP+hired VBD B-VP+him PRP B-NP+. . O++His PRP$ B-NP+1988 CD I-NP+salary NN I-NP+was VBD B-VP+$ $ B-NP+575,000 CD I-NP+, , O+with IN B-PP+a DT B-NP+$ $ I-NP+575,000 CD I-NP+bonus NN I-NP+. . O++The DT B-NP+40-year JJ I-NP+old JJ I-NP+Mr. NNP I-NP+Pope NNP I-NP+has VBZ B-VP+n't RB I-VP+changed VBN I-VP+jobs NNS B-NP+enough RB B-ADVP+-- : O+at IN B-PP+least JJS B-ADJP+the DT B-NP+right NN I-NP+ones NNS I-NP+-- : O+to TO B-VP+stash VB I-VP+away RB B-ADVP+that DT B-NP+kind NN I-NP+of IN B-PP+money NN B-NP+. . O++United NNP B-NP+paid VBD B-VP+him PRP B-NP+a DT B-NP+$ $ I-NP+375,000 CD I-NP+bonus NN I-NP+to TO B-VP+lure VB I-VP+him PRP B-NP+away RB B-PP+from IN B-PP+American NNP B-NP+Airlines NNPS I-NP+, , O+and CC O+he PRP B-NP+was VBD B-VP+paid VBN I-VP+a DT B-NP+salary NN I-NP+of IN B-PP+$ $ B-NP+342,122 CD I-NP+last JJ B-NP+year NN I-NP+with IN B-PP+a DT B-NP+$ $ I-NP+280,000 CD I-NP+bonus NN I-NP+. . O++Mr. NNP B-NP+Pope NNP I-NP+owns VBZ B-VP+10,000 CD B-NP+UAL NNP I-NP+shares NNS I-NP+and CC O+has VBZ B-VP+options NNS B-NP+to TO B-VP+buy VB I-VP+another DT B-NP+150,000 CD I-NP+at IN B-PP+$ $ B-NP+69 CD I-NP+each DT B-NP+. . O++That DT B-NP+came VBD B-VP+to TO B-PP+a DT B-NP+combined VBN I-NP+$ $ I-NP+37.7 CD I-NP+million CD I-NP+under IN B-PP+the DT B-NP+$ $ I-NP+300-a-share JJ I-NP+buy-out NN I-NP+, , O+but CC O+just RB B-NP+$ $ I-NP+21.3 CD I-NP+million CD I-NP+at IN B-PP+yesterday NN B-NP+'s POS B-NP+close NN I-NP+. . O++Of IN B-PP+the DT B-NP+combined VBN I-NP+$ $ I-NP+114.4 CD I-NP+million CD I-NP+the DT B-NP+two CD I-NP+men NNS I-NP+were VBD B-VP+scheduled VBN I-VP+to TO I-VP+reap VB I-VP+under IN B-PP+the DT B-NP+buy-out NN I-NP+, , O+they PRP B-NP+agreed VBD B-VP+to TO I-VP+invest VB I-VP+in IN B-PP+the DT B-NP+buy-out NN I-NP+just RB B-NP+$ $ I-NP+15 CD I-NP+million CD I-NP+, , O+angering VBG B-VP+many NN B-NP+of IN B-PP+the DT B-NP+thousands NNS I-NP+of IN B-PP+workers NNS B-NP+asked VBD B-VP+to TO B-VP+make VB I-VP+pay NN B-NP+concessions NNS I-NP+so RB B-SBAR+the DT B-NP+buy-out NN I-NP+would MD B-VP+be VB I-VP+a DT B-NP+success NN I-NP+. . O++United NNP B-NP+'s POS B-NP+directors NNS I-NP+voted VBD B-VP+themselves PRP B-NP+, , O+and CC O+their PRP$ B-NP+spouses NNS I-NP+, , O+lifetime NN B-NP+access NN I-NP+to TO B-PP
+ lib/haskell-utils/Atom.hs view
@@ -0,0 +1,85 @@+{-# LANGUAGE GeneralizedNewtypeDeriving , NoMonomorphismRestriction #-}+module Atom ( MonadAtoms (..)+ , AtomTable (..)+ , Atoms+ , AtomsT+ , empty+ , evalAtoms+ , evalAtomsT+ , runAtoms+ , runAtomsT+ )+where+import Control.Monad.State+import Control.Monad.Identity+import qualified Data.Map as Map+import qualified Data.IntMap as IntMap+import qualified Data.Binary as B+import qualified Text+type Txt = Text.Txt+++data AtomTable = T { lastID :: !Int + , to :: Map.Map Txt Int + , from :: IntMap.IntMap Txt } + deriving (Eq,Show)+++instance B.Binary AtomTable where+ put t = do B.put (lastID t) + B.put (to t)+ B.put (from t)+ get = do liftM3 T B.get B.get B.get+++class Monad m => MonadAtoms m where+ toAtom :: Txt -> m Int+ maybeToAtom :: Txt -> m (Maybe Int)+ fromAtom :: Int -> m Txt+ table :: m AtomTable++instance Monad m => MonadAtoms (AtomsT m) where+ toAtom x = AtomsT $ do+ t <- get+ case Map.lookup x (to t) of+ Just j -> return $! j+ Nothing -> do + let i = lastID t+ i' = i + 1 + !t' = t { lastID = i'+ , to = Map.insert x i (to t) + , from = IntMap.insert i x (from t) }+ put t'+ return $! lastID t++ maybeToAtom x = + AtomsT $ do+ t <- get+ return . Map.lookup x . to $ t+ + fromAtom i = AtomsT $ do+ t <- get+ return $ (from t) IntMap.! i+ table = AtomsT get++empty :: AtomTable+empty = T 0 Map.empty IntMap.empty++runAtomsT :: AtomsT t t1 -> AtomTable -> t (t1, AtomTable)+runAtomsT (AtomsT x) s = runStateT x s++runAtoms :: Atoms t -> AtomTable -> (t, AtomTable)+runAtoms (Atoms x) s = runIdentity (runAtomsT x s)+++evalAtoms :: Atoms t -> t+evalAtoms = fst . flip runAtoms empty++evalAtomsT :: (Monad m) => AtomsT m a -> m a+evalAtomsT = liftM fst . flip runAtomsT empty++newtype AtomsT m r = AtomsT (StateT AtomTable m r)+ deriving (Functor,Monad,MonadTrans,MonadIO)++newtype Atoms r = Atoms (AtomsT Identity r)+ deriving (Functor,Monad,MonadAtoms)
+ lib/haskell-utils/ListZipper.hs view
@@ -0,0 +1,69 @@+module ListZipper ( ListZipper(..)+ , focus+ , left+ , right+ , reset+ , fromList+ , toWindows+ , toList+ , next+ , atEnd + , at+ )+where+import Data.Maybe+import Data.Monoid+import Data.Foldable (Foldable,foldMap)+++data ListZipper a = LZ [a] (Maybe a) [a] deriving (Show,Eq,Ord)+++left (LZ xs _ _) = xs+focus (LZ _ x _) = x+right (LZ _ _ ys) = ys++fromList [] = LZ [] Nothing [] +fromList (x:xs) = LZ [] (Just x) xs++toWindows xs = let zs = iterate next . fromList $ xs+ in map snd . zip xs $ zs++toList (LZ xs (Just x) ys) = reverse xs ++ [x] ++ ys+toList (LZ xs Nothing []) = reverse xs++next :: ListZipper a -> ListZipper a+next = nextWith id id+nextWith f g (LZ lxs (Just x) rxs) =+ case rxs of+ y:ys -> LZ (f x:lxs) (Just (g y)) ys+ [] -> LZ (f x:lxs) Nothing []++atEnd = isNothing . focus++reset (LZ [] (Just y) ys) = LZ [] (Just y) ys+reset (LZ [] Nothing []) = LZ [] Nothing []+reset (LZ xs (Just y) ys) = LZ [] (Just z) (zs++[y]++ys)+ where z:zs = reverse xs +reset (LZ xs Nothing []) = LZ [] (Just z) zs+ where z:zs = reverse xs++from :: (Monoid m) => Maybe m -> m+from Nothing = mempty+from (Just x) = x++index 1 (x:xs) = Just x+index i [] = Nothing+index i (x:xs) = index (i-1) xs++at :: (Monoid m) => ListZipper m -> Int -> m+at z 0 = from . focus $ z+at z i | i < 0 = from . index (negate i) . left $ z+at z i = from . index i . right $ z++instance Functor ListZipper where+ fmap f (LZ ls x rs) = LZ (fmap f ls) (fmap f x) (fmap f rs) +instance Foldable ListZipper where+ foldMap f (LZ ls x rs) = mconcat $ fmap f (reverse ls) + ++ [from $ fmap f x] ++ fmap f rs+
+ lib/haskell-utils/Text.hs view
@@ -0,0 +1,85 @@+module Text ( Txt+ , StrictTxt+ , module Data.ByteString.Lazy.UTF8+ , Char8.unlines+ , Char8.words+ , Char8.unwords+ , Char8.append+ , Char8.concat+ , Char8.null+ , Char8.readInt++ , BS.getContents+ , BS.readFile+ , BS.writeFile+ , BS.putStr+ , BS.putStrLn+ , BS.interact+ , BS.hPutStr++ , map+ , all+ , show+ , read+ , reverse+ , normalize+ , toStrict+ , fromStrict+ , takeWhile+ , dropWhile+ , isPrefixOf+ , isSuffixOf+ , splitOn+ )+where+import Data.ByteString.Lazy.UTF8+import qualified Data.ByteString.Lazy.Char8 as Char8+import qualified Data.ByteString.Lazy as BS+import qualified Data.ByteString.Char8 as Strict +import Prelude hiding (show,reverse,map,read,takeWhile,dropWhile,break,all)+import qualified Prelude+import qualified Utils ++type Txt = ByteString+type StrictTxt = Strict.ByteString++show :: (Show a) => a -> Txt+show = fromString . Prelude.show++read :: (Read a) => Txt -> a+read = Prelude.read . toString++reverse :: Txt -> Txt+reverse = fromString . Prelude.reverse . toString++map :: (Char -> Char) -> Txt -> Txt+map f = fromString . Prelude.map f . toString -- FIXME++all :: (Char -> Bool) -> Txt -> Bool+all f = Prelude.all f . toString++normalize :: Txt -> Txt+normalize = Char8.fromChunks . return . Strict.concat . Char8.toChunks++toStrict :: Txt -> StrictTxt+toStrict = Strict.concat . Char8.toChunks++fromStrict :: StrictTxt -> Txt+fromStrict = Char8.fromChunks . return++takeWhile :: (Char -> Bool) -> Txt -> Txt+takeWhile f = fst . break (not . f)++dropWhile :: (Char -> Bool) -> Txt -> Txt+dropWhile f = snd . break (not . f)++isPrefixOf :: Txt -> Txt -> Bool+xs `isPrefixOf` ys = Prelude.all id . Char8.zipWith (==) xs $ ys++isSuffixOf :: Txt -> Txt -> Bool+xs `isSuffixOf` ys = reverse xs `isPrefixOf` Char8.reverse ys+++splitOn :: Char -> Txt -> [Txt]+splitOn c = Prelude.map fromString . Utils.splitOn c . toString+
+ lib/haskell-utils/Utils.hs view
@@ -0,0 +1,43 @@+{-# LANGUAGE BangPatterns #-}++module Utils ( splitOn+ , splitWith+ , padRight+ , uniq+ , accuracy+ , mean+ , (#)+ )+where+import Data.List (foldl')+import qualified Data.Set as Set+import qualified Data.Map as Map++splitOn :: (Eq a) => a -> [a] -> [[a]]+splitOn = splitWith . (==)++splitWith :: (a -> Bool) -> [a] -> [[a]]+splitWith f s = case dropWhile f s of+ [] -> []+ s' -> w : splitWith f s''+ where (w, s'') = break f s'++padRight :: a -> Int -> [a] -> [a]+padRight x i xs = take i (xs ++ repeat x)++uniq :: (Ord a) => [a] -> [a]+uniq = Set.toList . Set.fromList++accuracy predicted testset = + fromIntegral (foldl' (+) 0 (zipWith ((fromEnum .) . (==)) predicted + testset))+ / + fromIntegral (length testset)++{-# SPECIALIZE mean :: [Double] -> Double #-}+mean :: (Fractional a) => [a] -> a+mean = uncurry (/) + . foldl' (\ (!s,!n) x -> (s+x,n+1)) (0,0)++(#) :: (Show k,Ord k) => Map.Map k a -> k -> a+m # i = Map.findWithDefault (error $ "#: not found: " ++ show i) i m
+ sequor.cabal view
@@ -0,0 +1,28 @@+Name: sequor+Version: 0.1+Description: A sequence labeler based on Collins's sequence perceptron.+Synopsis: A sequence labeler based on Collins's sequence perceptron.+Homepage: http://code.google.com/p/sequor/+License: BSD3+License-file: LICENSE+Author: Grzegorz Chrupała+Maintainer: gchrupala@lsv.uni-saarland.de+Build-Type: Simple+Cabal-Version: >=1.2+Category: Natural Language Processing+Extra-source-files: README, Makefile+Data-dir: data+Data-files: all.features, example.features, train.conll+ devel.conll, test.conll++Executable sequor+ Main-is: Main.hs+ Other-modules: ListZipper, Utils, Text, CorpusReader, Atom, + FeatureTemplate, Config, Perceptron.Vector,+ Perceptron.Sequence, Features, Labeler+ Build-Depends: base >= 3 && < 5, containers >= 0.2, + bytestring >= 0.9, utf8-string >= 0.3,+ binary >= 0.5, mtl >= 1.1,+ vector >= 0.5, array >= 0.2+ hs-source-dirs: src,lib/haskell-utils+ ghc-options: -O2
+ src/Config.hs view
@@ -0,0 +1,23 @@+{-# LANGUAGE NoMonomorphismRestriction #-}+module Config + ( Config (..) )+where+import Data.Char+import Atom (AtomTable)+import qualified Data.Binary as B+import Control.Monad (ap)+import FeatureTemplate (Feature)++data Config = Config { wordMinCount :: Int+ , atomTable :: AtomTable + , minLabelFreq :: Int+ , featureTemplate :: Feature+ }++instance B.Binary Config where+ get = let g = B.get+ in return Config `ap` g `ap` g `ap` g `ap` g ++ put (Config a b c d) =+ let p = B.put + in p a >> p b >> p c >> p d
+ src/CorpusReader.hs view
@@ -0,0 +1,45 @@+{-# LANGUAGE OverloadedStrings #-}+module CorpusReader + ( Token+ , corpus+ , corpusLabeled + , fromWords+ )+where+import ListZipper +import Utils (splitWith)+import qualified Text+import Text (Txt)+import Data.Maybe (isJust)+++type Token = [Txt]++corpus :: Txt -> [[ListZipper Token]]+corpus =+ map toZippers+ . map (map $ parseFields)+ . splitWith null+ . map Text.words+ . Text.lines ++corpusLabeled ::Txt -> [([ListZipper Token], [Txt])]+corpusLabeled = + map (\xys -> let (xs,ys) = unzip xys in (toZippers xs,ys))+ . map (map $ parseFieldsLabeled)+ . splitWith null+ . map Text.words+ . Text.lines+ +fromWords :: [Txt] -> [ListZipper Token] +fromWords = toZippers . map (\ w -> [w])++parseFieldsLabeled :: [Txt] -> (Token, Txt)+parseFieldsLabeled ws = (init ws,last ws)++parseFields :: [Txt] -> Token+parseFields ws = ws++toZippers :: [a] -> [ListZipper a]+toZippers = takeWhile (isJust . focus) . iterate next . fromList +
+ src/FeatureTemplate.hs view
@@ -0,0 +1,33 @@+module FeatureTemplate + ( Feature(..) + , parse+ )+where+import Data.Binary +import Text++type Row = Int+type Col = Int++data Feature =+ Cell Row Col+ | Rect Row Col Row Col+ | Row Row+ | Index Feature+ | MarkNull Feature+ | Cat [Feature]+ | Cart Feature Feature+ | Lower Feature+ | Suffix Int Feature+ | Prefix Int Feature+ | WordShape Feature+ deriving (Show,Read)++parse :: Txt -> Feature+parse = Text.read . Text.unwords . Text.lines++instance Binary Feature where+ put f = put $ Text.show f+ get = do+ f <- get+ return $ Text.read f
+ src/Features.hs view
@@ -0,0 +1,117 @@+{-# LANGUAGE OverloadedStrings #-}++module Features + ( features+ , inputFeatures+ , outputFeatures+ , indexFeatures+ , maybeFeatures , eval + )+where++import qualified Text+import Text (Txt)+import qualified ListZipper as LZ+import ListZipper (ListZipper,at)+import CorpusReader (Token,fromWords)+import qualified Data.Char as Char+import Data.List (group,sort)+import qualified Data.IntSet as IntSet+import qualified Data.IntMap as IntMap+import Atom (MonadAtoms,AtomTable,from,toAtom,maybeToAtom)+import Data.Maybe (catMaybes,isNothing)+import Control.Monad (liftM2)+import Data.Monoid (mappend)+import Config +import qualified Data.Vector.Unboxed as V+import FeatureTemplate (Feature(..))+ +iNDEX_SUFFIX :: Txt+iNDEX_SUFFIX="::index"+iNPUT_PREFIX :: Txt+iNPUT_PREFIX="in:"+oUTPUT_PREFIX :: Txt+oUTPUT_PREFIX="out:"+nULL_MARK :: Txt+nULL_MARK = "<NULL>"+++eval :: ListZipper Token -> Feature -> [Maybe Txt]+eval z (Cell r c) = case z `at` r of + [] -> [Nothing]+ fs -> [fs `index` c] +eval z (Rect r c r' c') = concat [ eval z (Cell i j) | i <- [r..r'] + , j <- [c..c'] ]+eval z (Row r) = concat [ eval z (Cell r j) + | j <- [0..length (z `at` 0)-1] ]+eval z (MarkNull f) = [ maybe (Just nULL_MARK) Just fi + | fi <- eval z f ]+eval z (Index f) = [ fi+++Just iNDEX_SUFFIX | fi <- eval z f ]+eval z (Cat fs) = concatMap (eval z) fs+eval z (Cart f f') = [ fmap Text.normalize $ fi +++ Just "," +++ fi' + | fi <- eval z f , fi' <- eval z f' ]+eval z (Lower f) = [ fmap (Text.map Char.toLower) fi | fi <- eval z f ]+eval z (Suffix i f) = [ fmap (Text.reverse+ . Text.take (fromIntegral i)+ . Text.reverse )+ $ fi | fi <- eval z f ]+eval z (Prefix i f) = [ fmap (Text.take (fromIntegral i)) $ fi + | fi <- eval z f ]+eval z (WordShape f) = [ fmap (spellingSpec) fi | fi <- eval z f ] ++spellingSpec = Text.fromString + . map (\(x:xs) -> x) + . group + . map collapse + . Text.toString++collapse c | Char.isAlpha c && Char.isUpper c = 'X'+ | Char.isAlpha c && Char.isLower c = 'x'+ | Char.isDigit c = '0'+ | c == '-' = '-'+ | c == '_' = '_'+ | otherwise = '*'++indexFeatures :: AtomTable -> IntSet.IntSet +indexFeatures =+ IntMap.keysSet + . IntMap.filter (iNDEX_SUFFIX `Text.isSuffixOf`) + . from ++inputFeatures :: Config -> ListZipper Token -> [Txt]+inputFeatures config x =+ catMaybes . prefixIndex oUTPUT_PREFIX . eval x . featureTemplate $ config++outputFeatures :: [Txt] -> [Txt]+outputFeatures ys = catMaybes . prefixIndex oUTPUT_PREFIX . map Just $+ case ys of+ (y:y':_) -> [y,y`Text.append`y']+ [y] -> [y]+ [] -> []++features :: (MonadAtoms m) => Config -> ListZipper Token -> m (V.Vector Int)+features config x = do+ ifs <- mapM toAtom (inputFeatures config x)+ return $ V.fromList ifs++maybeFeatures :: (MonadAtoms m) => + Config -> ListZipper Token -> m (V.Vector Int)+maybeFeatures config x = do+ ifs <- mapM maybeToAtom (inputFeatures config x)+ return (V.fromList $ catMaybes $ ifs)+++prefixIndex :: Txt -> [Maybe Txt] -> [Maybe Txt]+prefixIndex str = zipWith (\i x -> Just str +++ Just (Text.show i) + +++ Just "=" + +++ x ) + [1..]++(+++) = liftM2 Text.append++index [] _ = Nothing+index (x:_) 0 = Just x+index (_:xs) i = index xs (i-1)+++sent = LZ.fromList [["I","pro"],["like","v"],["Ike","pn"]] :: ListZipper Token
+ src/Labeler.hs view
@@ -0,0 +1,187 @@+{-# LANGUAGE OverloadedStrings #-}+module Labeler + ( ModelData(..)+ , Config(..)+ , train+ , predict+ )+ +where++import qualified Data.Map as Map+import qualified Data.Set as Set+import qualified Data.IntMap as IntMap+import qualified Data.IntSet as IntSet+import Data.List (foldl',tails)+import Data.Maybe (fromMaybe)+import ListZipper+import qualified Perceptron.Sequence as P+import Perceptron.Sequence (Options(..))+import CorpusReader (Token)+import Utils (splitWith,uniq)+import Text.Printf+import Atom+import Control.Monad.RWS+import Features (maybeFeatures,features,outputFeatures,indexFeatures)+import qualified Data.Array as A+import qualified Data.Vector.Unboxed as V+import qualified Data.Binary as Binary+import qualified Text+import Text(Txt)+import Data.Char+import Data.Maybe (catMaybes)+import Config ++data ModelData = ModelData { model :: P.Model+ , config :: Config+ } +instance Binary.Binary ModelData where+ get = return ModelData `ap` Binary.get `ap` Binary.get+ put (ModelData a b) = Binary.put a >> Binary.put b +++++-- Main exported functions +predict :: ModelData -> [[ListZipper Token]] -> [[Txt]]+predict m testdat = + fst . flip runAtoms (atomTable . config $ m) $+ do flip mapM testdat $ \x -> + do x' <- mapM (maybeFeatures (config m)) $ x+ predict' (P.decode (model m)) $ x'++train :: Config + -> Float+ -> Int + -> Int + -> [([ListZipper Token],[Txt])]+ -> [([ListZipper Token],[Txt])]+ -> ModelData+train conf rate limit beam traindat heldout = + let ((m,_predicted),_atoms) = + runAtoms (run conf + (rate,limit,beam) + traindat + heldout) + $ empty+ in m++-- Implementation+type F = Int+type Tag = Int+tagDictionary :: IntSet.IntSet + -> Int + -> [([V.Vector Int], [F])] + -> IntMap.IntMap [Tag]+tagDictionary indexFeatureSet wmin trainset = + let tags = concat . map snd $ trainset+ ws = catMaybes + . map (V.find (`IntSet.member` indexFeatureSet))+ . concat + . map fst + $ trainset+ count_ws = IntMap.fromListWith (+) [ (w,1) | w <- ws ]+ dict = IntMap.map Set.toList+ . IntMap.fromListWith Set.union + $ [ (w,Set.singleton t) | (w,t) <- zip ws tags + , count_ws IntMap.! w >= wmin]+ in dict == dict `seq` dict++pruneLabels :: Int -> [(x,[Txt])] -> [(x,[Txt])]+pruneLabels lim xys =+ let freq = Map.fromListWith (+)+ . map (\y -> (y,1))+ . concat+ . map snd+ $ xys+ undet = "UNDETERMINED"+ in [ (x,[ if freq Map.! yi < lim then undet else yi | yi <- y ]) + | (x,y) <- xys ]++run :: (Functor m, MonadAtoms m) =>+ Config+ -> (Float, Int,Int)+ -> [([ListZipper Token], [Txt])]+ -> [([ListZipper Token], [Txt])]+ -> m (ModelData, [[Txt]])+run conf (rate, limit,beamp) trainset_in_full testset_in = do+ let trainset_in = pruneLabels (minLabelFreq conf) trainset_in_full+ ys = uniq . concat . map snd $ trainset_in :: [Txt]+ ys' <- mapM toAtom ys+ trainset <- mapM (mkfs $ features conf) trainset_in+ outm <- mkOutputFeatureAtoms . map snd $ trainset_in + testset <- mapM (mkfs $ maybeFeatures conf) testset_in + tab <- table+ let indexFeatureSet = indexFeatures tab+ conf' = conf {atomTable = tab }+ opts = Options { oYMap = outm+ , oIndexSet = indexFeatureSet+ , oYDict = tagDictionary indexFeatureSet + (wordMinCount conf') trainset+ , oYs = ys'+ , oBeam = beamp+ , oRate = rate+ , oEpochs = limit+ }+ m = P.train opts testset formatEval trainset+ ps <- mapM (predict' (P.decode m . fst)) testset+ return $ (ModelData { model = m , config = conf' }+ ,ps)++predict' :: (MonadAtoms m) =>+ (t -> [Int]) -> t -> m [Txt]+predict' dec x = do+ let xr = dec x+ xr'<- mapM fromAtom xr+ return xr'++mkOutputFeatureAtoms :: (MonadAtoms m) => [[Txt]] -> m P.YMap+mkOutputFeatureAtoms yss = do+ let unigrams = map return . uniq . concat $ yss+ bigrams = uniq $ concat [ filter ((==2) . length) + . map (take 2) + . tails + $ ys | ys <- yss ]+ unigramis <- mapM (mapM toAtom) unigrams+ bigramis <- mapM (mapM toAtom) bigrams+ let ys = map head unigramis+ (lo,hi) = (minimum ys,maximum ys)+ unigramfs <- mapM (mapM toAtom) . map outputFeatures $ unigrams+ bigramfs <- mapM (mapM toAtom) . map outputFeatures $ bigrams+ zerofs <- mapM toAtom . outputFeatures $ []+ let ymap1 = A.accumArray (V.++) V.empty (lo,hi) + . zip (map head unigramis) + . map V.fromList+ $ unigramfs+ ymap2 = A.accumArray (V.++) V.empty ((lo,lo),(hi,hi)) + . zip (map (\ [y1,y2] -> (y1,y2)) bigramis)+ . map V.fromList+ $ bigramfs + return $ (V.fromList zerofs, ymap1, ymap2)+ +mkfs :: (MonadAtoms m) => + (ListZipper Token -> m (V.Vector F)) + -> ([ListZipper Token], [Txt]) + -> m ([V.Vector F], [Tag])+mkfs f (x,y) = do+ fs <- mapM f x+ fs == fs `seq` return ()+ y' <- mapM toAtom y+ y' == y' `seq` return ()+ return $ (fs,y')+++formatEval :: P.Eval +formatEval 0 _ _ = printf "%10s %10s %10s" ("Iter"::String) + ("Train"::String)+ ("Heldout"::String)+formatEval i ss heldout = printf "%10d %10.4f %10.4f" i (eval ss) (eval heldout)+ ++eval :: Eq a => [([a],[a])] -> Double+eval ys = + let corr = foldl' (+) 0 + . concat+ $ [ [ 1 | (y,y') <- ys , (yi,yi') <- zip y y' + , yi == yi' ] ]+ in corr / fromIntegral (length . concatMap fst $ ys)
+ src/Main.hs view
@@ -0,0 +1,49 @@+module Main (main)+where+import Labeler (Config(..),ModelData(..)+ ,train,predict)+import CorpusReader (corpus,corpusLabeled)+import qualified Text+import qualified Data.Binary as Binary+import System.Environment (getArgs)+import System.IO (hPutStrLn,stderr)+import FeatureTemplate (parse)++main :: IO ()+main = do+ (command:args) <- getArgs+ case command of+ "train" -> do + let [ templatef+ ,rate+ ,beamp+ ,limit+ ,mincount+ ,trainf+ ,testf+ ,outf+ ] = args+ template <- parse `fmap` Text.readFile templatef+ traindat <- fmap corpusLabeled $ Text.readFile trainf+ testdat <- fmap corpusLabeled $ Text.readFile testf+ let conf = Config { featureTemplate = template + , wordMinCount = read mincount+ , atomTable = error + "main:Config.atomTable undefined" + , minLabelFreq = 1+ }+ Text.writeFile outf + . Binary.encode + . train conf (read rate) (read limit) (read beamp) traindat + $ testdat+ "predict" -> do+ let [modelf] = args+ m <- fmap Binary.decode (Text.readFile modelf)+ testdat <- fmap corpus $ Text.getContents+ Text.putStr + . Text.unlines + . map Text.unlines + . predict m+ $ testdat+ _ -> hPutStrLn stderr "Invalid command"+
+ src/Perceptron/Sequence.hs view
@@ -0,0 +1,259 @@+{-# LANGUAGE NoMonomorphismRestriction + , BangPatterns+ , FlexibleInstances+ #-}+module Perceptron.Sequence+ (+ Model+ , Options(..)+ , Eval+ , YMap+ , train+ , decode+ )+where++import Data.Array.ST+import Data.Array.Unboxed+import qualified Data.Array as A+import qualified Data.Vector.Unboxed as V+import Control.Monad.ST+import Data.STRef+import Control.Monad+import qualified Data.Map as Map+import qualified Data.IntMap as IntMap+import qualified Data.IntSet as IntSet+import Perceptron.Vector+import System.IO+import Debug.Trace+import Config +import Data.List (inits,foldl',sortBy)+import Data.Ord (comparing)+import ListZipper +import qualified Data.Binary as Binary+import Utils (uniq)++data Model = Model { options :: Options + , weights :: UArray I Float }+type X = [Xi]+type Y = [Yi]+type I = (Yi,Xii) +type Xi = V.Vector Xii+type Xii = Int+type Yi = Int+type Dot = LocalSparseVector Yi Xii -> Float++data Options = Options { oYMap :: YMap+ , oIndexSet :: IntSet.IntSet+ , oYDict :: IntMap.IntMap [Yi]+ , oYs :: [Yi]+ , oBeam :: Int + , oRate :: Float+ , oEpochs :: Int + } deriving Eq++type YMap = (Xi,A.Array Yi Xi,A.Array (Yi,Yi) Xi)++instance Binary.Binary (V.Vector Int) where+ put v = Binary.put $ V.toList v+ get = V.fromList `fmap` Binary.get+ +instance Binary.Binary Model where+ put m = do + Binary.put (options m)+ -- Binary.put (weights m)+ let (lo,hi) = bounds . weights $ m+ xs = filter (\(_,e) -> e /= 0.0) . assocs . weights $ m+ Binary.put (lo,hi)+ Binary.put xs++ get = {-# SCC "get1" #-} do + os <- Binary.get + os == os `seq` return ()+ ws <- do+ (lo,hi) <- Binary.get+ xs <- Binary.get+ xs == xs `seq` return ()+ return $ accumArray (+) 0 (lo,hi) $ xs+ ws == ws `seq` return ()+ return $ Model os ws++instance Binary.Binary Options where+ put (Options a b c d e f g) = Binary.put a >> Binary.put b >> Binary.put c + >> Binary.put d >> Binary.put e >> Binary.put f+ >> Binary.put g + get = {-# SCC "get2" #-} do+ a <- Binary.get+ a == a `seq` return ()+ b <- Binary.get+ b == b `seq` return ()+ c <- Binary.get + c == c `seq` return ()+ d <- Binary.get + d == d `seq` return ()+ e <- Binary.get+ e == e `seq` return ()+ f <- Binary.get+ f == f `seq` return ()+ g <- Binary.get+ g == g `seq` return ()+ return $ Options a b c d e f g++yDictFind :: Options -> Xi -> [Yi]+yDictFind opts fs = + let mk = V.find (`IntSet.member` oIndexSet opts) $ fs+ def = oYs opts+ in case mk of+ Just k -> IntMap.findWithDefault def k . oYDict $ opts+ Nothing -> def++-- | DECODING +decode :: Model -> X -> Y+decode m = fst . decode' (options m) (weights m `dot`) ++data Cell = Cell { cScore :: !Float+ , cPhi :: SparseVector I+ , cPath :: Y+ , cStep :: ListZipper Xi } deriving (Show,Eq)++decode' :: Options -> Dot -> X -> (Y,SparseVector I)+decode' opts w x = + bestPath opts w [Cell { cScore = 0 + , cPhi = Map.empty+ , cPath = []+ , cStep = fromList x } ]+++phi :: Options -> X -> Y -> SparseVector I+phi opts x y = foldl' f Map.empty . zip x . map reverse . tail . inits $ y+ where f z (xi,ys) = z `plus` toSV (features (oYMap opts) xi ys)++{-# INLINE features #-} +features :: YMap -> Xi -> [Yi] -> LocalSparseVector Yi Xii+features (!zero,uni,bi) xi (y:ys) = + case ys of+ [] -> (y, zero V.++ xi)+ [y1] -> (y, uni A.! y1 V.++ xi)+ (y1 : y2 : _) -> let r = bi A.! (y1,y2) + in if V.null r + then (y, uni A.! y1 V.++ xi)+ else (y, r V.++ xi)++beamSearch :: Options+ -> Dot+ -> [Cell] + -> [Cell]+beamSearch opts w cs = + let f cs = if any (atEnd . cStep) cs then cs + else + let cs' = [ let fs = features (oYMap opts) xi (y':ys)+ in Cell { cScore = + s + w fs + , cPhi = ph `plus` (toSV fs)+ , cPath = (y':ys)+ , cStep = next x } + | Cell { cScore = s + , cPhi = ph + , cPath = ys + , cStep = x } <- cs + , let Just xi = focus x::Maybe Xi+ , y' <- yDictFind opts xi+ ]+ in f . take (oBeam opts) + . sortBy (flip $ comparing cScore) + $ cs'+ in f cs ++bestPath :: Options+ -> Dot+ -> [Cell]+ -> (Y, SparseVector I)+bestPath opts w xs = + let xs' = beamSearch opts w xs+ first = (\(x:_) -> x) xs'+ in ( reverse . cPath $ first+ , cPhi first )++-- | TRAINING++iter :: Options + -> Int+ -> [(X,Y)]+ -> (STRef s Int, DenseVectorST s I, DenseVectorST s I)+ -> ST s ()+iter opts _ ss (c,params,params_a) = do+ for_ ss $ \ (x,y) -> do+ params' <- unsafeFreeze params+ let (y',phi_xy') = decode' opts (params'`dot`) x+ when (y' /= y) $ do + let phi_xy = phi opts x y + update = (phi_xy `minus` phi_xy') `scale` oRate opts+ params `plus_` update+ c' <- readSTRef c+ params_a `plus_` (update `scale` fromIntegral c')+ modifySTRef c (+1)+++type Eval = Int -> [(Y,Y)] -> [(Y,Y)] -> String++train :: Options -> [(X, Y)] -> Eval -> [(X,Y)] -> Model+train opts heldout eval ss = Model opts $ runSTUArray $ do+ let bs = computeBounds opts ss+ trace (show bs) () `seq` return ()+ params <- newArray bs 0+ params_a <- newArray bs 0+ c <- newSTRef 1+ let undef = error "Perceptron.Sequence.train: undefined"+ runLogger . hPutStrLn stderr $ eval 0 undef undef+ for_ [1..oEpochs opts] $ + \i -> do iter opts i ss (c,params,params_a)+ c' <- readSTRef c+ params' <- unsafeFreeze params+ params_a' <- unsafeFreeze params_a+ let w = (fromIntegral c',params',params_a')+ ys xys = [ fst . decode' opts (w`dot'`) $ x + | (x,_) <- xys ]+ runLogger + . hPutStrLn stderr+ $ eval i (zip (map snd ss) (ys ss))+ (zip (map snd heldout) (ys heldout)) + + finalParams (c, params, params_a)+ return params+++{-# NOINLINE runLogger #-}+runLogger f = unsafeIOToST f++finalParams :: (STRef s Int, DenseVectorST s I, DenseVectorST s I) + -> ST s ()+finalParams (c,params,params_a) = do+ (l,u) <- getBounds params+ c' <- fmap fromIntegral (readSTRef c)+ for_ (range (l,u)) $ \i -> do+ e <- readArray params i+ e_a <- readArray params_a i+ writeArray params i (e - (e_a * (1/c')))++computeBounds :: Options -> [(X,Y)] -> (I,I)+computeBounds opts = foldl' f ((maxBound,minimum xis)+ ,(minBound,maximum xis)) + . (\(xs,ys) -> zip (concat xs) (concat ys))+ . unzip+ where f ((!miny,!minx),(!maxy,!maxx)) (xs,!y) =+ ((min miny y,V.minimum $ minx`V.cons`xs)+ ,(max maxy y,V.maximum $ maxx`V.cons`xs))+ xis = let (zero,uni,bi) = oYMap opts+ in uniq+ . concatMap V.toList+ $+ [zero]+ +++ (filter (not . V.null)+ . A.elems+ $ bi)+ +++ (filter (not . V.null) + . A.elems + $ uni)+
+ src/Perceptron/Vector.hs view
@@ -0,0 +1,83 @@+{-# LANGUAGE FlexibleContexts , BangPatterns #-}+module Perceptron.Vector + ( SparseVector+ , LocalSparseVector+ , DenseVector+ , DenseVectorST+ , toSV+ , for_+ , plus_+ , minus_+ , plus+ , minus+ , scale+ , dot + , dot'+ )+where++import Data.Array.ST+import Data.Array.Unboxed+import Control.Monad.ST+import Data.STRef+import Control.Monad+import qualified Data.Map as Map+import Data.List (foldl',sort)+import Config+import qualified Data.Vector.Unboxed as V++type SparseVector i = Map.Map i Float+type LocalSparseVector y i = (y,V.Vector i)+type DenseVectorST s i = STUArray s i Float+type DenseVector i = UArray i Float++++for_ xs f = mapM_ f xs++plus_ :: (Show i,Ix i) => DenseVectorST s i -> SparseVector i -> ST s ()+plus_ w v = do+ for_ (Map.toList v) $ \(i,vi) -> do+ wi <- readArray w i + writeArray w i (wi + vi)+minus_ w v = plus_ w (v `scale` (-1))++scale :: (Ix i) => SparseVector i -> Float -> SparseVector i+scale v n = Map.map (*n) v++plus :: (Ix i) => SparseVector i -> SparseVector i -> SparseVector i+plus u v = Map.unionWith (+) u v+minus :: (Ix i) => SparseVector i -> SparseVector i -> SparseVector i+minus u v = u `plus` (v `scale` (-1))+++dot :: DenseVector (Int,Int) -> LocalSparseVector Int Int -> Float+{-# INLINE dot #-}+dot w (!y,x) = V.foldl' (\ !z !i -> z + w ! (y,i)) 0 x+{-+dot w (!y,v) = go 0 v+ where go !s [] = s+ go !s (!i:v) = go (s + w ! (y,i)) v+-}++dot' :: (Float,DenseVector (Int,Int),DenseVector (Int,Int)) + -> LocalSparseVector Int Int + -> Float+{-# INLINE dot' #-}+dot' (!c,params,params_a) (!y,x) = V.foldl' (\ !z !i -> + let e = params ! (y,i)+ e_a = params_a ! (y,i)+ in z + (e - (e_a * (1/c))))+ 0+ x+{-+dot' (!c,params,params_a) (!y,x) = go 0 x+ where go !s [] = s+ go !s (!i:x) = + let e = params ! (y,i)+ e_a = params_a ! (y,i)+ in go (s + (e - (e_a * (1/c)))) x+-}++toSV :: (V.Unbox i, Ord y,Ord i) => LocalSparseVector y i -> SparseVector (y,i)+toSV (y,v) = Map.fromList [ ((y,i),1) | i <- V.toList v ]