diff --git a/ChangeLog.md b/ChangeLog.md --- a/ChangeLog.md +++ b/ChangeLog.md @@ -1,5 +1,11 @@ # Changelog for Learning +## 0.0.2 *February 15th 2018* + * Implement confusion matrices + * Write PCA tutorials + * Add `pcaVariance` function parametrizing variance to retain + * Add `accuracy` function for classification evaluation + ## 0.0.1 *February 9th 2018* * Define core data structures * Provide linear classifiers and regressors for the supervised learning diff --git a/Learning.cabal b/Learning.cabal --- a/Learning.cabal +++ b/Learning.cabal @@ -2,17 +2,17 @@ -- -- see: https://github.com/sol/hpack -- --- hash: 6c4606a45d47a42344ba884c903f3f574904cf74f1bf8ba2b085dc33882900dd +-- hash: 17f24c6acefbeb01d1dbaf720f248d2ff7b3d82ff28f368f58a60e25cdc2a4d3 name: Learning -version: 0.0.2 +version: 0.0.3 synopsis: The most frequently used machine learning tools description: Please see the README on Github at category: Machine Learning homepage: https://github.com/masterdezign/Learning#readme bug-reports: https://github.com/masterdezign/Learning/issues author: Bogdan Penkovsky -maintainer: dev at penkovsky [dot] com +maintainer: dev () penkovsky dot com copyright: Bogdan Penkovsky license: BSD3 license-file: LICENSE @@ -42,9 +42,9 @@ default-language: Haskell2010 executable learning-pca - main-is: MainPCA.lhs + main-is: Main.lhs hs-source-dirs: - app + examples/PCA ghc-options: -threaded -rtsopts -with-rtsopts=-N build-depends: Learning @@ -53,14 +53,13 @@ , hmatrix >=0.18.0.0 , vector other-modules: - MainPCA2 Paths_Learning default-language: Haskell2010 executable learning-pca-advanced - main-is: MainPCA2.lhs + main-is: Main.lhs hs-source-dirs: - app + examples/PCA2 ghc-options: -threaded -rtsopts -with-rtsopts=-N build-depends: Learning @@ -69,7 +68,6 @@ , hmatrix >=0.18.0.0 , vector other-modules: - MainPCA Paths_Learning default-language: Haskell2010 diff --git a/README.md b/README.md --- a/README.md +++ b/README.md @@ -5,7 +5,7 @@ The name of the package can be interpreted in two ways: 1. Either as "Learning" in "Machine Learning". -2. Or "Learning" meaning that [examples](https://github.com/masterdezign/Learning/tree/master/app) +2. Or "Learning" meaning that [examples](https://github.com/masterdezign/Learning/tree/master/examples) are written in [literate style](https://en.wikipedia.org/wiki/Literate_programming) and can be used to discover machine learning techniques. @@ -21,20 +21,20 @@ ## Getting Started -Use [Stack](http://haskellstack.org). +Use [Stack](http://haskellstack.org) $ git clone https://github.com/masterdezign/Learning.git && cd Learning $ stack build --install-ghc ### Demo 1: principal components analysis (PCA) -Launch the [PCA demo](https://github.com/masterdezign/Learning/blob/master/app/MainPCA.lhs) +Launch the [PCA demo](https://github.com/masterdezign/Learning/blob/master/examples/PCA/Main.lhs) $ stack exec learning-pca -### Demo 2: advanced principal components analysis (PCA) +### Demo 2: advanced PCA -Launch the advanced [PCA demo](https://github.com/masterdezign/Learning/blob/master/app/MainPCA2.lhs) +Launch the advanced [PCA demo](https://github.com/masterdezign/Learning/blob/master/examples/PCA2/Main.lhs) $ stack exec learning-pca-advanced diff --git a/app/MainPCA.lhs b/app/MainPCA.lhs deleted file mode 100644 --- a/app/MainPCA.lhs +++ /dev/null @@ -1,62 +0,0 @@ -Principal Components Analysis (PCA) demo ----------------------------------------- - - -The tutorial is based on http://setosa.io/ev/principal-component-analysis/ - -Suppose, we study nutrition habits of the citizens of four countries. -Here, we provide the food consumption data among those countries. - -> import Learning -> import qualified Numeric.LinearAlgebra as LA - -> england = [375, 57, 245, 1472, 105, 54, 193, 147, 1102, -> 720, 253, 685, 488, 198, 360, 1374, 156] - -> northernIreland = [135, 47, 267, 1494, 66, 41, 209, 93, 674, -> 1033, 143, 586, 355, 187, 334, 1506, 139] - -> scotland = [458, 53, 242, 1462, 103, 62, 184, 122, 957, -> 566, 171, 750, 418, 220, 337, 1572, 147] - -> wales = [475, 73, 227, 1582, 103, 64, 235, 160, 1137, -> 874, 265, 803, 570, 203, 365, 1256, 175] - -We want to know how differ the countries based on those data. -For that purpose, we would like to reduce the redundant information -or, in other words, perform PCA. - -We create a single list of feature vectors (each country) used later -for the analysis. - -> countries = map LA.fromList [england, northernIreland, scotland, wales] - -We perform PCA, i.e. calculate the compression (dimensionality reduction) -function `compress`. The `pca` function is given -`principalComponents` parameter. Here it's 1, that means that -`countries` vectors of 17 features will be reduced into scalars (1D vectors). - -> compress = let principalComponents = 1 -> pca1 = pca principalComponents countries -> in _compress pca1 - -Output the resulting scalar values for each country - -> main = mapM_ (print. compress) countries - -Here is a summary: - - England -702.9850482521952 - Northern Ireland -80.6002572540017 - Scotland -649.8612350689822 - Wales -798.521043705295 - -Wales - \/ England Northern Ireland - \/ \/ -..o....o..o.................................o - /\ - Scotland - -Now, we can clearly see that there exists a difference between -Northern Ireland and the rest of the countries. diff --git a/app/MainPCA2.lhs b/app/MainPCA2.lhs deleted file mode 100644 --- a/app/MainPCA2.lhs +++ /dev/null @@ -1,77 +0,0 @@ -Advanced Principal Components Analysis (PCA) demo -------------------------------------------------- - - -The tutorial is a continuation of ./MainPCA.lhs - -Previously, we were able to quickly determine that the food ration -in Northern Ireland is somewhat different comparing to the other three -countries. We have projected data from 17 dimensions into -one dimension. However, we could also make a projection into -two or more dimensions. So how do we determine which dimensionality -reduction does preserve the most of the information? How do we make -sure that only redundant information was removed? - -For that purpose, we will calculate retained variance [1] -depending on the number of the principal components. - -[1] http://www.dsc.ufcg.edu.br/~hmg/disciplinas/posgraduacao/rn-copin-2014.3/material/SignalProcPCA.pdf - -Let's start with the imports and data definition. - -> import Learning ( pca' ) -> import qualified Numeric.LinearAlgebra as LA -> import Data.List ( scanl' ) -> import Text.Printf ( printf ) - -> england = [375, 57, 245, 1472, 105, 54, 193, 147, 1102, -> 720, 253, 685, 488, 198, 360, 1374, 156] - -> northernIreland = [135, 47, 267, 1494, 66, 41, 209, 93, 674, -> 1033, 143, 586, 355, 187, 334, 1506, 139] - -> scotland = [458, 53, 242, 1462, 103, 62, 184, 122, 957, -> 566, 171, 750, 418, 220, 337, 1572, 147] - -> wales = [475, 73, 227, 1582, 103, 64, 235, 160, 1137, -> 874, 265, 803, 570, 203, 365, 1256, 175] - -> countries = map LA.fromList [england, northernIreland, scotland, wales] - -In order to compute the eigenvectors u and eigenvalues eig of -a covariance matrix, we use function pca'. In fact, pca' was -already called under the hood in the previous tutorial. - -> (u, eig) = pca' countries - -Sums of the first N eigenvalues: - -> cumul = drop 1 $ scanl' (+) 0 $ LA.toList eig - -Sum of all eigenvalues: - -> total = last cumul - -> main = mapM_ (\(i, s) -> -> let retained = s / total * 100 :: Double -> msg = "%d principal component(s): Retained variance %.1f%%" -> in putStrLn $ printf msg i retained) -> $ zip [1::Int ..] cumul - - 1 principal component(s): Retained variance 67.4% - 2 principal component(s): Retained variance 96.5% - 3 principal component(s): Retained variance 100.0% - 4 principal component(s): Retained variance 100.0% - - ... - - 17 principal component(s): Retained variance 100.0% - -From this data we can conclude that the first two principal components -contain 96.5% of information. Therefore, we will loose 3.5% of information -after projecting into two orthogonal axes in the transformed coordinate system -obtained after PCA. - -Hint: to compute compression (dimensionality reduction) and -decompression functions for specified variance to retain, use -(_compress. pcaVariance) and (_decompress. pcaVariance) functions. diff --git a/examples/PCA/Main.lhs b/examples/PCA/Main.lhs new file mode 100644 --- /dev/null +++ b/examples/PCA/Main.lhs @@ -0,0 +1,62 @@ +Principal Components Analysis (PCA) demo +---------------------------------------- + + +The tutorial is based on http://setosa.io/ev/principal-component-analysis/ + +Suppose, we study nutrition habits of the citizens of four countries. +Here, we provide the food consumption data among those countries. + +> import Learning +> import qualified Numeric.LinearAlgebra as LA + +> england = [375, 57, 245, 1472, 105, 54, 193, 147, 1102, +> 720, 253, 685, 488, 198, 360, 1374, 156] + +> northernIreland = [135, 47, 267, 1494, 66, 41, 209, 93, 674, +> 1033, 143, 586, 355, 187, 334, 1506, 139] + +> scotland = [458, 53, 242, 1462, 103, 62, 184, 122, 957, +> 566, 171, 750, 418, 220, 337, 1572, 147] + +> wales = [475, 73, 227, 1582, 103, 64, 235, 160, 1137, +> 874, 265, 803, 570, 203, 365, 1256, 175] + +We want to know how differ the countries based on those data. +For that purpose, we would like to reduce the redundant information +or, in other words, perform PCA. + +We create a single list of feature vectors (each country) used later +for the analysis. + +> countries = map LA.fromList [england, northernIreland, scotland, wales] + +We perform PCA, i.e. calculate the compression (dimensionality reduction) +function `compress`. The `pca` function is given +`principalComponents` parameter. Here it's 1, that means that +`countries` vectors of 17 features will be reduced into scalars (1D vectors). + +> compress = let principalComponents = 1 +> pca1 = pca principalComponents countries +> in _compress pca1 + +Output the resulting scalar values for each country + +> main = mapM_ (print. compress) countries + +Here is a summary: + + England -702.9850482521952 + Northern Ireland -80.6002572540017 + Scotland -649.8612350689822 + Wales -798.521043705295 + +Wales + \/ England Northern Ireland + \/ \/ +..o....o..o.................................o + /\ + Scotland + +Now, we can clearly see that there exists a difference between +Northern Ireland and the rest of the countries. diff --git a/examples/PCA2/Main.lhs b/examples/PCA2/Main.lhs new file mode 100644 --- /dev/null +++ b/examples/PCA2/Main.lhs @@ -0,0 +1,77 @@ +Advanced Principal Components Analysis (PCA) demo +------------------------------------------------- + + +The tutorial is a continuation of examples/PCA/Main.lhs + +Previously, we were able to quickly determine that the food ration +in Northern Ireland is somewhat different comparing to the other three +countries. We have projected data from 17 dimensions into +one dimension. However, we could also make a projection into +two or more dimensions. So how do we determine which dimensionality +reduction does preserve the most of the information? How do we make +sure that only redundant information was removed? + +For that purpose, we will calculate retained variance [1] +depending on the number of the principal components. + +[1] http://www.dsc.ufcg.edu.br/~hmg/disciplinas/posgraduacao/rn-copin-2014.3/material/SignalProcPCA.pdf + +Let's start with the imports and data definition. + +> import Learning ( pca' ) +> import qualified Numeric.LinearAlgebra as LA +> import Data.List ( scanl' ) +> import Text.Printf ( printf ) + +> england = [375, 57, 245, 1472, 105, 54, 193, 147, 1102, +> 720, 253, 685, 488, 198, 360, 1374, 156] + +> northernIreland = [135, 47, 267, 1494, 66, 41, 209, 93, 674, +> 1033, 143, 586, 355, 187, 334, 1506, 139] + +> scotland = [458, 53, 242, 1462, 103, 62, 184, 122, 957, +> 566, 171, 750, 418, 220, 337, 1572, 147] + +> wales = [475, 73, 227, 1582, 103, 64, 235, 160, 1137, +> 874, 265, 803, 570, 203, 365, 1256, 175] + +> countries = map LA.fromList [england, northernIreland, scotland, wales] + +In order to compute the eigenvectors u and eigenvalues eig of +a covariance matrix, we use function pca'. In fact, pca' was +already called under the hood in the previous tutorial. + +> (u, eig) = pca' countries + +Sums of the first N eigenvalues: + +> cumul = drop 1 $ scanl' (+) 0 $ LA.toList eig + +Sum of all eigenvalues: + +> total = last cumul + +> main = mapM_ (\(i, s) -> +> let retained = s / total * 100 :: Double +> msg = "%d principal component(s): Retained variance %.1f%%" +> in putStrLn $ printf msg i retained) +> $ zip [1::Int ..] cumul + + 1 principal component(s): Retained variance 67.4% + 2 principal component(s): Retained variance 96.5% + 3 principal component(s): Retained variance 100.0% + 4 principal component(s): Retained variance 100.0% + + ... + + 17 principal component(s): Retained variance 100.0% + +From this data we can conclude that the first two principal components +contain 96.5% of information. Therefore, we will loose 3.5% of information +after projecting into two orthogonal axes in the transformed coordinate system +obtained after PCA. + +Hint: to compute compression (dimensionality reduction) and +decompression functions for specified variance to retain, use +(_compress. pcaVariance) and (_decompress. pcaVariance) functions.