Haggressive (empty) → 0.1.0.0
raw patch · 10 files changed
+969/−0 lines, 10 filesdep +Cabaldep +HUnitdep +Haggressivesetup-changed
Dependencies added: Cabal, HUnit, Haggressive, PSQueue, base, bytestring, cassava, containers, directory, text, tokenize, tuple, vector
Files
- Haggressive.cabal +51/−0
- LICENSE +339/−0
- Setup.hs +2/−0
- src-exe/Main.hs +8/−0
- src-lib/Evaluation.hs +79/−0
- src-lib/Hag.hs +232/−0
- src-lib/Helpers.hs +41/−0
- src-lib/Preprocess.hs +19/−0
- src-lib/Tweets.hs +36/−0
- testsuite/tests/My.hs +162/−0
+ Haggressive.cabal view
@@ -0,0 +1,51 @@+name: Haggressive+version: 0.1.0.0+synopsis: Aggression analysis for Tweets on Twitter+description: Aggression analysis for Tweets on Twitter+homepage: http://github.io/pold87/Haggressive+license: GPL-2+license-file: LICENSE+author: Volker Strobel+maintainer: volker.strobel87@gmail.com+category: Web+build-type: Simple++cabal-version: >=1.10++library+ hs-source-dirs: src-lib+ build-depends: base >=4.7 && <5,+ vector,+ containers,+ Cabal,+ HUnit,+ text,+ cassava,+ tuple,+ bytestring, + directory,+ tokenize,+ PSQueue+ exposed-modules: Hag, Preprocess, Tweets, Helpers, Evaluation+ default-language: Haskell2010+ ghc-options: -fllvm++executable Haggressive+ main-is: Main.hs+ build-depends: base, Haggressive, Cabal, HUnit+ hs-source-dirs: src-exe+ default-language: Haskell2010+ ghc-options: -fllvm++test-suite tests+ type: detailed-0.9+ test-module: My+ hs-source-dirs: testsuite/tests+ build-depends: base+ , Haggressive+ , HUnit+ , containers+ , vector+ , Cabal+ , tuple+ default-language: Haskell2010
+ LICENSE view
@@ -0,0 +1,339 @@+ GNU GENERAL PUBLIC LICENSE+ Version 2, June 1991++ Copyright (C) 1989, 1991 Free Software Foundation, Inc.,+ 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA+ Everyone is permitted to copy and distribute verbatim copies+ of this license document, but changing it is not allowed.++ Preamble++ The licenses for most software are designed to take away your+freedom to share and change it. By contrast, the GNU General Public+License is intended to guarantee your freedom to share and change free+software--to make sure the software is free for all its users. This+General Public License applies to most of the Free Software+Foundation's software and to any other program whose authors commit to+using it. (Some other Free Software Foundation software is covered by+the GNU Lesser General Public License instead.) You can apply it to+your programs, too.++ When we speak of free software, we are referring to freedom, not+price. Our General Public Licenses are designed to make sure that you+have the freedom to distribute copies of free software (and charge for+this service if you wish), that you receive source code or can get it+if you want it, that you can change the software or use pieces of it+in new free programs; and that you know you can do these things.++ To protect your rights, we need to make restrictions that forbid+anyone to deny you these rights or to ask you to surrender the rights.+These restrictions translate to certain responsibilities for you if you+distribute copies of the software, or if you modify it.++ For example, if you distribute copies of such a program, whether+gratis or for a fee, you must give the recipients all the rights that+you have. You must make sure that they, too, receive or can get the+source code. And you must show them these terms so they know their+rights.++ We protect your rights with two steps: (1) copyright the software, and+(2) offer you this license which gives you legal permission to copy,+distribute and/or modify the software.++ Also, for each author's protection and ours, we want to make certain+that everyone understands that there is no warranty for this free+software. If the software is modified by someone else and passed on, we+want its recipients to know that what they have is not the original, so+that any problems introduced by others will not reflect on the original+authors' reputations.++ Finally, any free program is threatened constantly by software+patents. We wish to avoid the danger that redistributors of a free+program will individually obtain patent licenses, in effect making the+program proprietary. To prevent this, we have made it clear that any+patent must be licensed for everyone's free use or not licensed at all.++ The precise terms and conditions for copying, distribution and+modification follow.++ GNU GENERAL PUBLIC LICENSE+ TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION++ 0. This License applies to any program or other work which contains+a notice placed by the copyright holder saying it may be distributed+under the terms of this General Public License. The "Program", below,+refers to any such program or work, and a "work based on the Program"+means either the Program or any derivative work under copyright law:+that is to say, a work containing the Program or a portion of it,+either verbatim or with modifications and/or translated into another+language. (Hereinafter, translation is included without limitation in+the term "modification".) Each licensee is addressed as "you".++Activities other than copying, distribution and modification are not+covered by this License; they are outside its scope. The act of+running the Program is not restricted, and the output from the Program+is covered only if its contents constitute a work based on the+Program (independent of having been made by running the Program).+Whether that is true depends on what the Program does.++ 1. You may copy and distribute verbatim copies of the Program's+source code as you receive it, in any medium, provided that you+conspicuously and appropriately publish on each copy an appropriate+copyright notice and disclaimer of warranty; keep intact all the+notices that refer to this License and to the absence of any warranty;+and give any other recipients of the Program a copy of this License+along with the Program.++You may charge a fee for the physical act of transferring a copy, and+you may at your option offer warranty protection in exchange for a fee.++ 2. You may modify your copy or copies of the Program or any portion+of it, thus forming a work based on the Program, and copy and+distribute such modifications or work under the terms of Section 1+above, provided that you also meet all of these conditions:++ a) You must cause the modified files to carry prominent notices+ stating that you changed the files and the date of any change.++ b) You must cause any work that you distribute or publish, that in+ whole or in part contains or is derived from the Program or any+ part thereof, to be licensed as a whole at no charge to all third+ parties under the terms of this License.++ c) If the modified program normally reads commands interactively+ when run, you must cause it, when started running for such+ interactive use in the most ordinary way, to print or display an+ announcement including an appropriate copyright notice and a+ notice that there is no warranty (or else, saying that you provide+ a warranty) and that users may redistribute the program under+ these conditions, and telling the user how to view a copy of this+ License. (Exception: if the Program itself is interactive but+ does not normally print such an announcement, your work based on+ the Program is not required to print an announcement.)++These requirements apply to the modified work as a whole. If+identifiable sections of that work are not derived from the Program,+and can be reasonably considered independent and separate works in+themselves, then this License, and its terms, do not apply to those+sections when you distribute them as separate works. But when you+distribute the same sections as part of a whole which is a work based+on the Program, the distribution of the whole must be on the terms of+this License, whose permissions for other licensees extend to the+entire whole, and thus to each and every part regardless of who wrote it.++Thus, it is not the intent of this section to claim rights or contest+your rights to work written entirely by you; rather, the intent is to+exercise the right to control the distribution of derivative or+collective works based on the Program.++In addition, mere aggregation of another work not based on the Program+with the Program (or with a work based on the Program) on a volume of+a storage or distribution medium does not bring the other work under+the scope of this License.++ 3. You may copy and distribute the Program (or a work based on it,+under Section 2) in object code or executable form under the terms of+Sections 1 and 2 above provided that you also do one of the following:++ a) Accompany it with the complete corresponding machine-readable+ source code, which must be distributed under the terms of Sections+ 1 and 2 above on a medium customarily used for software interchange; or,++ b) Accompany it with a written offer, valid for at least three+ years, to give any third party, for a charge no more than your+ cost of physically performing source distribution, a complete+ machine-readable copy of the corresponding source code, to be+ distributed under the terms of Sections 1 and 2 above on a medium+ customarily used for software interchange; or,++ c) Accompany it with the information you received as to the offer+ to distribute corresponding source code. (This alternative is+ allowed only for noncommercial distribution and only if you+ received the program in object code or executable form with such+ an offer, in accord with Subsection b above.)++The source code for a work means the preferred form of the work for+making modifications to it. For an executable work, complete source+code means all the source code for all modules it contains, plus any+associated interface definition files, plus the scripts used to+control compilation and installation of the executable. However, as a+special exception, the source code distributed need not include+anything that is normally distributed (in either source or binary+form) with the major components (compiler, kernel, and so on) of the+operating system on which the executable runs, unless that component+itself accompanies the executable.++If distribution of executable or object code is made by offering+access to copy from a designated place, then offering equivalent+access to copy the source code from the same place counts as+distribution of the source code, even though third parties are not+compelled to copy the source along with the object code.++ 4. You may not copy, modify, sublicense, or distribute the Program+except as expressly provided under this License. Any attempt+otherwise to copy, modify, sublicense or distribute the Program is+void, and will automatically terminate your rights under this License.+However, parties who have received copies, or rights, from you under+this License will not have their licenses terminated so long as such+parties remain in full compliance.++ 5. You are not required to accept this License, since you have not+signed it. However, nothing else grants you permission to modify or+distribute the Program or its derivative works. These actions are+prohibited by law if you do not accept this License. Therefore, by+modifying or distributing the Program (or any work based on the+Program), you indicate your acceptance of this License to do so, and+all its terms and conditions for copying, distributing or modifying+the Program or works based on it.++ 6. Each time you redistribute the Program (or any work based on the+Program), the recipient automatically receives a license from the+original licensor to copy, distribute or modify the Program subject to+these terms and conditions. You may not impose any further+restrictions on the recipients' exercise of the rights granted herein.+You are not responsible for enforcing compliance by third parties to+this License.++ 7. If, as a consequence of a court judgment or allegation of patent+infringement or for any other reason (not limited to patent issues),+conditions are imposed on you (whether by court order, agreement or+otherwise) that contradict the conditions of this License, they do not+excuse you from the conditions of this License. If you cannot+distribute so as to satisfy simultaneously your obligations under this+License and any other pertinent obligations, then as a consequence you+may not distribute the Program at all. For example, if a patent+license would not permit royalty-free redistribution of the Program by+all those who receive copies directly or indirectly through you, then+the only way you could satisfy both it and this License would be to+refrain entirely from distribution of the Program.++If any portion of this section is held invalid or unenforceable under+any particular circumstance, the balance of the section is intended to+apply and the section as a whole is intended to apply in other+circumstances.++It is not the purpose of this section to induce you to infringe any+patents or other property right claims or to contest validity of any+such claims; this section has the sole purpose of protecting the+integrity of the free software distribution system, which is+implemented by public license practices. Many people have made+generous contributions to the wide range of software distributed+through that system in reliance on consistent application of that+system; it is up to the author/donor to decide if he or she is willing+to distribute software through any other system and a licensee cannot+impose that choice.++This section is intended to make thoroughly clear what is believed to+be a consequence of the rest of this License.++ 8. If the distribution and/or use of the Program is restricted in+certain countries either by patents or by copyrighted interfaces, the+original copyright holder who places the Program under this License+may add an explicit geographical distribution limitation excluding+those countries, so that distribution is permitted only in or among+countries not thus excluded. In such case, this License incorporates+the limitation as if written in the body of this License.++ 9. The Free Software Foundation may publish revised and/or new versions+of the General Public License from time to time. Such new versions will+be similar in spirit to the present version, but may differ in detail to+address new problems or concerns.++Each version is given a distinguishing version number. If the Program+specifies a version number of this License which applies to it and "any+later version", you have the option of following the terms and conditions+either of that version or of any later version published by the Free+Software Foundation. If the Program does not specify a version number of+this License, you may choose any version ever published by the Free Software+Foundation.++ 10. If you wish to incorporate parts of the Program into other free+programs whose distribution conditions are different, write to the author+to ask for permission. For software which is copyrighted by the Free+Software Foundation, write to the Free Software Foundation; we sometimes+make exceptions for this. Our decision will be guided by the two goals+of preserving the free status of all derivatives of our free software and+of promoting the sharing and reuse of software generally.++ NO WARRANTY++ 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY+FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN+OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES+PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED+OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF+MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS+TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE+PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,+REPAIR OR CORRECTION.++ 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR+REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,+INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING+OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED+TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY+YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER+PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE+POSSIBILITY OF SUCH DAMAGES.++ END OF TERMS AND CONDITIONS++ How to Apply These Terms to Your New Programs++ If you develop a new program, and you want it to be of the greatest+possible use to the public, the best way to achieve this is to make it+free software which everyone can redistribute and change under these terms.++ To do so, attach the following notices to the program. It is safest+to attach them to the start of each source file to most effectively+convey the exclusion of warranty; and each file should have at least+the "copyright" line and a pointer to where the full notice is found.++ <one line to give the program's name and a brief idea of what it does.>+ Copyright (C) <year> <name of author>++ This program is free software; you can redistribute it and/or modify+ it under the terms of the GNU General Public License as published by+ the Free Software Foundation; either version 2 of the License, or+ (at your option) any later version.++ This program is distributed in the hope that it will be useful,+ but WITHOUT ANY WARRANTY; without even the implied warranty of+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the+ GNU General Public License for more details.++ You should have received a copy of the GNU General Public License along+ with this program; if not, write to the Free Software Foundation, Inc.,+ 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.++Also add information on how to contact you by electronic and paper mail.++If the program is interactive, make it output a short notice like this+when it starts in an interactive mode:++ Gnomovision version 69, Copyright (C) year name of author+ Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.+ This is free software, and you are welcome to redistribute it+ under certain conditions; type `show c' for details.++The hypothetical commands `show w' and `show c' should show the appropriate+parts of the General Public License. Of course, the commands you use may+be called something other than `show w' and `show c'; they could even be+mouse-clicks or menu items--whatever suits your program.++You should also get your employer (if you work as a programmer) or your+school, if any, to sign a "copyright disclaimer" for the program, if+necessary. Here is a sample; alter the names:++ Yoyodyne, Inc., hereby disclaims all copyright interest in the program+ `Gnomovision' (which makes passes at compilers) written by James Hacker.++ <signature of Ty Coon>, 1 April 1989+ Ty Coon, President of Vice++This General Public License does not permit incorporating your program into+proprietary programs. If your program is a subroutine library, you may+consider it more useful to permit linking proprietary applications with the+library. If this is what you want to do, use the GNU Lesser General+Public License instead of this License.
+ Setup.hs view
@@ -0,0 +1,2 @@+import Distribution.Simple+main = defaultMain
+ src-exe/Main.hs view
@@ -0,0 +1,8 @@+module Main where++import qualified Evaluation+import qualified Hag++main :: IO ()+main =+ Hag.main
+ src-lib/Evaluation.hs view
@@ -0,0 +1,79 @@+module Evaluation+ (Evaluation.main)+ where++import Control.Monad+import qualified Data.ByteString.Char8 as BS+import qualified Data.ByteString.Lazy as L+import qualified Data.ByteString.Lazy.Char8 as BSL+import Data.Char+import Data.Csv+import Data.Either+import Data.List+import qualified Data.Map as M+import Data.Ord+import qualified Data.PSQueue as PS+import qualified Data.Text as T+import Data.Text.Encoding+import qualified Data.Text.IO as TI+import qualified Data.Vector as V+import Hag+import Helpers+import NLP.Tokenize+import Preprocess+import Preprocess+import qualified System.Directory as S+import System.Environment+import Tweets+++-- |Create a dictionary ('M.Map String Float' /TODO/: probably define+-- dictionary type) from a 'V.Vector' of 'Tweet's+createDictionary :: V.Vector Tweet -> FeatureMap+createDictionary tweets = V.foldl (M.unionWith (+)) M.empty allTweets+ where allTweets = V.map extractFeatures tweets++-- |Create a 'grand dictionary' ('M.Map String Float' /TODO/: probably+-- define dictionary type) from a 'mini dictionary' ('M.Map Tweet+-- (M.Map String Float)'+createDictionaryFromMap :: M.Map Tweet FeatureMap -> FeatureMap+createDictionaryFromMap tweetMap = foldl (M.unionWith (+)) M.empty $ M.elems tweetMap++main :: IO ()+main = do+ (dir:_) <- getArgs++ files <- getFiles dir+ csvs <- mapM TI.readFile $ sort files+ let processedCsvs = map preprocess csvs+ r = map parseCsv processedCsvs+ listOfVectorOfTweets = rights r :: [V.Vector Tweet]+ -- Tweet vectors+ vTweets = V.concat listOfVectorOfTweets :: V.Vector Tweet+ aggTweets = filterByLabel vTweets "aggressive" :: V.Vector Tweet+ nonAggTweets = filterByLabel vTweets "non_aggressive" :: V.Vector Tweet++ -- Tweet dictionaries+ dict = createDictionary vTweets+ aggDict = createDictionary aggTweets+ nonAggDict = createDictionary nonAggTweets++ words = (encode $ sortBy (comparing $ snd) $+ M.toList dict) :: L.ByteString++ aggWords = (encode $ sortBy (comparing $ snd) $+ M.toList aggDict) :: L.ByteString++ nonAggWords = (encode $ sortBy (comparing $ snd) $+ M.toList nonAggDict) :: L.ByteString++ -- Create a header+ header = encode ["word","frequency"]++ -- Write output to csv+ L.writeFile "agg_words.csv" $ header `L.append` aggWords+ L.writeFile "non_agg_words.csv" $ header `L.append` nonAggWords+ L.writeFile "words.csv" $ header `L.append` words+ print $ V.length aggTweets++
+ src-lib/Hag.hs view
@@ -0,0 +1,232 @@+{-# OPTIONS_GHC -Wall #-}+{-# LANGUAGE OverloadedStrings #-}+++{-|+Module : Hag+Description : Classify Tweets (aggressive vs. non_aggressive) and evaluate+ classification performance.+License : None+Maintainer : Volker Strobel (volker.strobel87@gmail.com)+Stability : experimental+Portability : None++This module is the main interface for Tweet classification.++-}+module Hag+( module Hag+ ) where++import qualified Data.ByteString.Lazy as L+import Data.Char+import Data.Csv+import Data.Either+import Data.List+import qualified Data.Map as M+import qualified Data.PSQueue as PS+import qualified Data.Text as T+import Data.Text.Encoding+import qualified Data.Text.IO as TI+import qualified Data.Vector as V+import Helpers+import NLP.Tokenize+import Preprocess+import qualified System.Directory as S+import System.Environment+import Tweets+++-- | Features are represented by a 'M.Map', where the keys are+-- 'String's (e.g., the words in the message of a Tweet) and the+-- values are 'Float's (e.g., the number of occurrence of a word)+type FeatureMap = M.Map String Float++-- |+-- =IO and Parsing++-- |'parseCsv' parses a 'T.Text' input for fields in CSV format and+-- returns a 'Vector' of 'Tweet's+parseCsv :: T.Text -> Either String (V.Vector Tweet)+parseCsv text = decodeWith+ defaultDecodeOptions {decDelimiter = fromIntegral $ ord '\t' }+ NoHeader+ (L.fromStrict $+ encodeUtf8 text)++-- |Get directory contents of 'FilePath'. A better variant is at:+getFiles :: FilePath -> IO [FilePath]+getFiles dir = S.getDirectoryContents dir+ >>= return . map (dir ++) . filter (`notElem` [".", ".."])++-- |Extract features (the bag of unigrams) for one Tweet.+-- Thereby, the Tweet will be (in order of application):+-- * tokenized+-- * converted to a 'V.Vector'+-- * 'String's will be converted to lowercase+-- * 'String's that are element of 'stopWords' are removed+-- * Empty 'String's will be removed+extractFeatures :: Tweet -> FeatureMap+extractFeatures tweet = bagOfUnigrams+ where+ pre = V.filter (`notElem` ["USER", "RT"])+ . V.fromList+ . tokenize+ . tMessage+ bagOfUnigrams = frequency+ . V.filter (/= "")+ . V.filter (`notElem` stopWords)+ . V.map (map toLower)+ . pre+ $ tweet++-- |Calculate the 'frequency' of items in a 'V.Vector' and return them+-- in a 'M.Map'.+frequency :: V.Vector String -> FeatureMap+frequency = V.foldl' countItem M.empty++-- |Insert an item into a 'M.Map'. Default value is 1 if the item is+-- not existing. If the item is already existing, its frequency will+-- be increased by 1.+countItem :: M.Map String Float -> String -> FeatureMap+countItem myMap item = M.insertWith (+) item 1 myMap++-- |Take a 'M.Map', consisting of+-- key: 'Tweet'+-- value: 'FeatureMap'+-- and one Tweet and create a new 'M.Map' with the added features+-- from the 'Tweet'+insertInMap :: M.Map Tweet FeatureMap+ -> Tweet+ -> M.Map Tweet FeatureMap+insertInMap oldMap tweet = M.insert tweet val oldMap+ where val = extractFeatures tweet++-- | Compare two vectors of 'Tweet's, the first is the test vector,+-- the second the train vector and return the all neighbors for each+-- 'Tweet'. 'grandDict' is a 'M.Map', where each entry consits of a+-- 'Tweet' and its features+getNeighbors :: (V.Vector Tweet, V.Vector Tweet)+ -> V.Vector (Tweet, [PS.Binding Tweet Float])+getNeighbors (v1,v2) = V.map (featureIntersection dictionary) v1+ where dictionary = V.foldl insertInMap M.empty v2 :: M.Map Tweet FeatureMap++-- | Take a dictionary and a 'Tweet' and return a pair of this 'Tweet'+-- and all its nearest neighbors+featureIntersection :: M.Map Tweet FeatureMap+ -> Tweet+ -> (Tweet, [PS.Binding Tweet Float])+featureIntersection tweetMap tweet = (tweet, mini)+ where+ mini = PS.fromList+ $ M.elems+ $ M.mapWithKey (mergeTweetFeatures cosineDistance tweet) tweetMap++-- |Take a distance function, 'Tweet' 1, 'Tweet' 2 and a dictionary as+-- 'FeatureMap' and create a 'PS.Binding' between 'Tweet' 2 and the+-- distance from this 'Tweet' to the other 'Tweet'.+mergeTweetFeatures :: (FeatureMap -> FeatureMap -> Float)+ -> Tweet+ -> Tweet+ -> FeatureMap+ -> PS.Binding Tweet Float++mergeTweetFeatures distF t1 t2 _ = queue+ where featuresT1 = extractFeatures t1+ featuresT2 = extractFeatures t2+ distance = distF featuresT1 featuresT2+ queue = t2 PS.:-> distance++-- |Take the features of two 'Tweet's and return the distance as+-- 'Num'.+cosineDistance :: FeatureMap -> FeatureMap -> Float+cosineDistance t1 t2 = negate (mySum / (wordsInT1 * wordsInT2))+ where+ wordsInT1 = M.foldl (+) 0 t1+ wordsInT2 = M.foldl (+) 0 t2+ intersection = M.elems $ M.intersectionWith (*) t1 t2+ mySum = foldl (+) 0 intersection++-- |Takes a dictionary and a mini dictionary (frequency of words in+-- one Tweet) and calculates the idftf values for all words in the+-- mini dictionary.+idftf :: FeatureMap -> FeatureMap -> FeatureMap+idftf grandDict miniDict = M.mapWithKey (iFrequency grandDict) miniDict++iFrequency :: FeatureMap -> String -> Float -> Float+iFrequency dict word freq = freq * (log (totalNumberOfWords / freqWord))+ where freqWord = M.findWithDefault 1 word dict+ totalNumberOfWords = M.foldl (+) 0 dict++-- | Calculate the amount of tweets where the predicted label matches+-- the actual label.+compareLabels :: Int -> V.Vector (Tweet,[PS.Binding Tweet Float]) -> V.Vector Float+compareLabels k vec = V.map+ (\(a,b) -> if (tLabel a) == getLabel k b then 1 else 0)+ vec++compareLabelsForScheme :: [V.Vector (Tweet,[PS.Binding Tweet Float])] -> Int -> [Float]+compareLabelsForScheme vecs k = map (getAccuracy . compareLabels k) vecs+++-- | Get the label for a 'Tweet' by looking at the k nearest+-- neighbors. If there are more aggressive than non_aggressive+-- 'Tweet's, the label will be aggressive, otherwise, it will be+-- non-aggressive.+getLabel :: Int -> [PS.Binding Tweet Float] -> String+getLabel k queue = if agg >= nonAgg then "aggressive" else "non_aggressive"+ where -- tweets = queueTake k queue+ tweets = take k queue+ labels = map (tLabel . PS.key) tweets+ agg = length $ filter (== "aggressive") labels+ nonAgg = length $ filter (== "non_aggressive") labels++-- | Get sum total of a vector of floats (i.e., the number of+-- correctly classified tweets) and return the accuracy+getAccuracy :: V.Vector Float -> Float+getAccuracy vec = (V.foldl (+) 0 vec) / fromIntegral (V.length vec)++main :: IO ()+main = do+ -- Retrieve command line args+ (dir:_) <- getArgs++ -- Get list of files in directory dir+ files <- getFiles dir++ -- Read content of all files into list, one Text per item+ csvs <- mapM TI.readFile $ sort files++ let+ -- Add quotation marks for CSV parsing+ processedCsvs = map preprocess csvs++ -- Create Tweets from Text+ r = map parseCsv processedCsvs++ -- If parsing was successful, extract Right elements from the+ -- Either list+ tweets = rights r++ -- Create a leave-one-out cross-validation scheme+ scheme = mkCrossValScheme tweets++ -- Get all neighbors for all tweets for all schemes+ allNeighbors = map getNeighbors scheme++ -- k should be in {1..100}+ ks = [1..100]++ -- Compare all training tweets to test tweets for all ks+ comparedTweets = map (compareLabelsForScheme allNeighbors) ks++ -- Prepare for CSV+ results = encode comparedTweets++ -- Create header+ -- header = encode+ -- $ map (("fold_" ++) . show) ([1..10] :: [Integer])++ -- Write output to a file+ -- L.writeFile "resultsK.csv" $ header `L.append` results+ L.writeFile "resultsK.csv" $ results
+ src-lib/Helpers.hs view
@@ -0,0 +1,41 @@+module Helpers+ (mkCrossValScheme+ , queueTake+ , stopWords)+ where++import qualified Data.PSQueue as PS+import qualified Data.Vector as V+import Debug.Trace+import System.IO.Unsafe++-- | 'FilePath' of the 'stopWords' that are removed before the+-- dictionary is created+stopWordsFile :: FilePath+stopWordsFile = "dutch-stop-words.txt"++-- | Read the 'stopWords' from the 'stopWordsFile'+stopWords :: [String]+stopWords = lines file+ where file = unsafePerformIO $ readFile stopWordsFile++-- | Make a cross-validation scheme from a list of vectors+mkCrossValScheme :: (Eq a) => [V.Vector a] -> [(V.Vector a,V.Vector a)]+mkCrossValScheme xs = map (leaveOneOut xs) xs++-- | Create pair of a list of vectors and a vector that specifies+-- which vector should be left out+leaveOneOut :: (Eq a) => [V.Vector a] -> V.Vector a -> (V.Vector a,V.Vector a)+leaveOneOut all test = (test, V.concat $ filter (/= test) all)++-- | Take the first k elements of a queue+queueTake :: (Ord k, Ord p, Show k) => Int -> PS.PSQ k p -> [k]+queueTake k queue = queueTake' k queue []++-- | Helper function for 'queueTake'+queueTake' :: (Ord k, Ord p) => Int -> PS.PSQ k p -> [k] -> [k]+queueTake' 0 _ acc = acc+queueTake' k queue acc = case mini of+ Nothing -> []+ Just m -> queueTake' (k - 1) (PS.deleteMin queue) (PS.key m:acc)+ where mini = PS.findMin queue
+ src-lib/Preprocess.hs view
@@ -0,0 +1,19 @@+{-# OPTIONS_GHC -Wall #-}+{-# LANGUAGE OverloadedStrings #-}++module Preprocess+ (preprocess)+ where++import qualified Data.Text as T++preprocess :: T.Text -> T.Text+preprocess txt = T.cons '\"' $ T.snoc escaped '\"'+ where escaped = T.concatMap escaper txt++escaper :: Char -> T.Text+escaper c+ | c == '\t' = "\"\t\""+ | c == '\n' = "\"\n\""+ | c == '\"' = "\"\""+ | otherwise = T.singleton c
+ src-lib/Tweets.hs view
@@ -0,0 +1,36 @@+module Tweets+ (filterByLabel+, Tweet(..)) where++import Control.Applicative ((<$>), (<*>), (<|>))+import Control.Monad (mzero)+import Data.Csv+import qualified Data.PSQueue as PS+import qualified Data.Vector as V++-- | Parsing Record to Tweet+instance FromRecord Tweet where+ parseRecord v+ | V.length v == 5 = Tweet <$>+ v.! 0 <*>+ v.! 1 <*>+ v.! 2 <*>+ v.! 3 <*>+ v.! 4+ | otherwise = mzero++-- | A Tweet consists of a category, a user, a date, a time, and a+-- message+data Tweet = Tweet { tLabel :: String+ , tUser :: String+ , tDate :: String+ , tTime :: String+ , tMessage :: String+ } deriving (Show, Eq, Ord)+++-- | Filter 'Tweet's by label+filterByLabel :: V.Vector Tweet -> String -> V.Vector Tweet+filterByLabel tweets label = V.filter (\t -> tLabel t == label) tweets++
+ testsuite/tests/My.hs view
@@ -0,0 +1,162 @@+module My where++import Data.Map as M+import Data.Tuple.Curry+import Data.Vector as V+import qualified Distribution.TestSuite as TS+import qualified Hag as H+import qualified Test.HUnit as HU++++-- Little helpers++-- parseTweet(s) were in Hag but it turned out that they are not necessary+-- there, since you can parse the CSV ad hoc, i.e., during reading the file++parseTweets :: V.Vector (String, String, String, String, String) -> V.Vector H.Tweet+parseTweets = V.map $ uncurryN H.Tweet++parseTweet :: (String, String, String, String, String) -> H.Tweet+parseTweet (a,b,c,d,e) = H.Tweet a b c d e+++-- Lists+list1, list2, list3 :: [String]+list1 = [ "ich", "bin", "aber", "ich", "bin", "nicht" ]+list2 = ["hey", "du", "hey", "du"]+list3 = []++-- Vectors+vec1, vec2, vec3 :: V.Vector String+vec1 = V.fromList list1+vec2 = V.fromList list2+vec3 = V.fromList list3+++-- Maps+map1, map2 :: M.Map String Int+map1 = M.fromList [ ("ich", 2)+ , ("bin", 2)+ , ("aber", 1)+ , ("nicht", 1)+ ]+map2 = M.fromList [ ("hab", 2)+ , ("bin", 2)+ , ("aber", 1)+ , ("nicht", 2)+ ]++-- Tuples+tup1 = ("aggressive", "Monday", "Time", "User", "Whatup?? I like this sh******t!!")+tup2 = ("a", "b", "c", "d", "e")+tup3 = ("foo", "baa", "haa", "mu", "sa")++tup4 = ("1", "2", "3", "4", "5")+tup5 = ("z", "y", "x", "a", "Oh my goodness \"!§$%&/()=?")++tup6 = ("b", "b", "b", "b", "Text Mining is cool")+tup7 = ("a", "a", "a", "a", "Text Mining sucks")++-- Tweets+[tweet1, tweet2, tweet3, tweet4, tweet5, tweet6, tweet7] = Prelude.map parseTweet [tup1, tup2, tup3, tup4, tup5, tup6, tup7]++-- Define tests++-- frequency --+testFrequency1, testFrequency3 :: HU.Test+testFrequency1 = H.frequency vec1 HU.~?= map1+testFrequency2 = H.frequency vec2 HU.~?= map2+testFrequency3 = H.frequency vec3 HU.~?= M.empty++++vecTweets1, vecTweets2, vecTweets3 :: V.Vector H.Tweet+vecTweets1 = parseTweets $ V.fromList [tup1, tup2, tup3]+vecTweets2 = parseTweets $ V.fromList [tup4, tup5]+vecTweets3 = parseTweets $ V.fromList [tup2, tup5]++++tweet1_m :: H.Tweet+tweet1_m = H.Tweet "aggressive"+ "Monday"+ "Time"+ "User"+ "Whatup?? I like this sh******t!!"+++++-- parseTweet+parseTweetTest1 :: HU.Test+parseTweetTest1 = parseTweet tup1 HU.~?= tweet1_m+++mkCrossValSchemeTest0, mkCrossValSchemeTest1, mkCrossValSchemeTest2 :: HU.Test+mkCrossValSchemeTest0= H.mkCrossValScheme ([] :: [V.Vector H.Tweet]) HU.~?= []+mkCrossValSchemeTest1 =+ H.mkCrossValScheme [vecTweets1, vecTweets2] HU.~?= [(vecTweets1, vecTweets2)+ , (vecTweets2, vecTweets1)+ ]+mkCrossValSchemeTest2 =+ H.mkCrossValScheme [vecTweets1, vecTweets2, vecTweets3] HU.~?= [(vecTweets1, vecTweets2 V.++ vecTweets3)+ , (vecTweets2, vecTweets1 V.++ vecTweets3)+ , (vecTweets3, vecTweets1 V.++ vecTweets2)]++++featureIntersectionTest1 :: HU.Test+featureIntersectionTest1 = H.featureIntersection tweet6 (V.fromList [tweet6]) HU.~?=+ V.fromList [M.fromList [("text", 2)+ , ("mining", 2)+ , ("is", 2)+ , ("cool", 2)]]++featureDistanceTest1 :: HU.Test+featureDistanceTest1 = H.featureDistance tweet6 (V.fromList [tweet6, tweet7]) HU.~?=+ V.fromList [4, 2]++++createDictionaryTest1 :: HU.Test+createDictionaryTest1 = H.createDictionary (V.fromList [tweet6, tweet7]) HU.~?=+ M.fromList [("text", 2)+ , ("mining", 2)+ , ("is", 1)+ , ("sucks", 1)+ , ("cool", 1)]++hunitTests = HU.TestList [testFrequency1+ , testFrequency3+ -- , parseTweetTest1+ , mkCrossValSchemeTest0+ , mkCrossValSchemeTest1+ , mkCrossValSchemeTest2+ , createDictionaryTest1+ , featureIntersectionTest1+ , featureDistanceTest1+ ]++runHUnitTests :: HU.Test -> IO TS.Progress+runHUnitTests tests = do+ (HU.Counts cases tried errors failures) <- HU.runTestTT tests+ return $ if errors > 0+ then TS.Finished $ TS.Error "There were errors in the HUnit tests"+ else if failures > 0+ then TS.Finished $ TS.Fail "There were failures in the HUnit tests"+ else TS.Finished TS.Pass++tests :: IO [TS.Test]+tests = return [ TS.Test hunit ]+ where+ hunit = TS.TestInstance+ { TS.run = runHUnitTests hunitTests+ , TS.name = "HUnit Test Cases"+ , TS.tags = ["hunit"]+ , TS.options = []+ , TS.setOption = \_ _ -> Right hunit+ }++main :: IO [TS.Test]+main = tests