elynx-tools 0.1.0 → 0.2.1
raw patch · 21 files changed
+1310/−673 lines, 21 filesdep +base16-bytestringdep +directorydep −lifted-basedep ~aesondep ~basedep ~bytestring
Dependencies added: base16-bytestring, directory
Dependencies removed: lifted-base
Dependency ranges changed: aeson, base, bytestring, containers, fast-logger, megaparsec, monad-logger, optparse-applicative, primitive, template-haskell, text, time, vector
Files
- ChangeLog.md +8/−5
- README.md +375/−88
- elynx-tools.cabal +22/−20
- src/ELynx/Tools.hs +52/−0
- src/ELynx/Tools/ByteString.hs +16/−10
- src/ELynx/Tools/Concurrent.hs +15/−30
- src/ELynx/Tools/Definitions.hs +27/−5
- src/ELynx/Tools/Equality.hs +6/−5
- src/ELynx/Tools/InputOutput.hs +108/−57
- src/ELynx/Tools/LinearAlgebra.hs +11/−8
- src/ELynx/Tools/List.hs +11/−9
- src/ELynx/Tools/Logger.hs +130/−66
- src/ELynx/Tools/Matrix.hs +51/−21
- src/ELynx/Tools/Misc.hs +5/−4
- src/ELynx/Tools/Numeric.hs +8/−6
- src/ELynx/Tools/Options.hs +0/−211
- src/ELynx/Tools/Parallel.hs +24/−0
- src/ELynx/Tools/Random.hs +17/−14
- src/ELynx/Tools/Reproduction.hs +396/−91
- src/ELynx/Tools/Text.hs +11/−9
- src/ELynx/Tools/Vector.hs +17/−14
ChangeLog.md view
@@ -1,10 +1,13 @@ -# Table of Contents--1. [Changelog for ELynx](#org263dcac)+# Changelog for ELynx -<a id="org263dcac"></a>+## Version 0.2.1 -# Changelog for ELynx+- Validation and repetition of previous analyses is finally possible with the+ new `elynx` binary.+- A library `elynx-markov` for running Markov processes along phylogenetic trees+ has been split off `elynx-seq`. This library performs the computations when+ executing `slynx simulate ...`.+- Many other small improvements.
README.md view
@@ -2,30 +2,31 @@ # The ELynx Suite -Version: 0.1.0.+Version: 0.2.1. Reproducible evolution made easy. -The ELynx Suite is a Haskell library and a tool set for computational biology.-The goal of the ELynx Suite is reproducible research. Evolutionary sequences and-phylogenetic trees can be read, viewed, modified and simulated. Exact-specification of all options is necessary, and nothing is assumed about the data-(e.g., the type of the genetic code). The command line with all arguments is-consistently, and automatically logged. The work overhead in the beginning-usually pays off in the end.+A Haskell library and tool set for computational biology. The goal of ELynx is+reproducible research. Evolutionary sequences and phylogenetic trees can be+read, viewed, modified and simulated. The command line with all arguments is+logged consistently, and automatically. Data integrity is verified using SHA256+sums so that validation of past analyses is possible without the need to+recompute the result. -The Elynx Suite consists of three library packages and two executables providing+The Elynx Suite consists of four library packages and three executables providing a range of sub commands. The library packages are: +- **elynx-markov:** Simulate multi sequence alignments along phylogenetic trees. - **elynx-seq:** Handle evolutionary sequences and multi sequence alignments.-- **elynx-tree:** Handle phylogenetic trees. - **elynx-tools:** Tools for the provided executables.+- **elynx-tree:** Handle phylogenetic trees. The executables are: -- **slynx:** Analyze, modify, and simulate evolutionary sequences (FASTA format).-- **tlynx:** Analyze, modify, and simulate phylogenetic trees (Newick format).+- **slynx:** Analyze, modify, and simulate evolutionary sequences.+- **tlynx:** Analyze, modify, and simulate phylogenetic trees.+- **elynx:** Validate and redo past analyses. **ELynx is actively developed. We happily receive comments, ideas, feature requests, and pull requests!**@@ -67,10 +68,12 @@ slynx --help - ELynx Suite version 0.1.0. Developed by Dominik Schrempf. Compiled on January- 30, 2020, at 09:18 am, UTC.+ ELynx Suite version 0.2.1.+ Developed by Dominik Schrempf.+ Compiled on April 27, 2020, at 06:31 am, UTC. - Usage: slynx [-v|--verbosity VALUE] [-o|--output-file-basename NAME] COMMAND+ Usage: slynx [-v|--verbosity VALUE] [-o|--output-file-basename NAME] + [-f|--force] COMMAND Analyze, and simulate multi sequence alignments. Available options:@@ -80,21 +83,22 @@ Debug (default: Info) -o,--output-file-basename NAME Specify base name of output file+ -f,--force Ignore previous analysis and overwrite existing+ output files. Available commands: concatenate Concatenate sequences found in input files.- examine Examine sequences. If data is a multi sequence- alignment, additionally analyze columns.+ examine Examine sequences. If data is a multi sequence alignment, additionally analyze columns.+ filter-columns Filter columns of multi sequence alignments. filter-rows Filter rows (or sequences) found in input files.- filter-columns Filter columns of multi-sequence alignments. simulate Simulate multi sequence alignments. sub-sample Sub-sample columns from multi sequence alignments. translate Translate from DNA to Protein or DNAX to ProteinX. - File formats:+ Available sequence file formats: - FASTA - Alphabet types:+ Available alphabets: - DNA (nucleotides) - DNAX (nucleotides; including gaps) - DNAI (nucleotides; including gaps, and IUPAC codes)@@ -103,19 +107,20 @@ - ProteinS (amino acids; including gaps, and translation stops) - ProteinI (amino acids; including gaps, translation stops, and IUPAC codes) - The ELynx Suite- ---------------- A Haskell library and a tool set for computational biology. The goal of the- ELynx Suite is reproducible research. Evolutionary sequences and phylogenetic- trees can be read, viewed, modified and simulated. Exact specification of all- options is necessary, and nothing is assumed about the data (e.g., the type of- code). The command line with all arguments is consistently, and automatically- logged.+ ELynx+ -----+ A Haskell library and tool set for computational biology. The goal of ELynx is+ reproducible research. Evolutionary sequences and phylogenetic trees can be+ read, viewed, modified and simulated. The command line with all arguments is+ logged consistently, and automatically. Data integrity is verified using SHA256+ sums so that validation of past analyses is possible without the need to+ recompute the result. slynx Analyze, modify, and simulate evolutionary sequences. tlynx Analyze, modify, and simulate phylogenetic trees.+ elynx Validate and redo past analyses. - Get help for specific commands:+ Get help for sub commands: slynx examine --help @@ -125,13 +130,36 @@ slynx concatenate --help + ELynx Suite version 0.2.1.+ Developed by Dominik Schrempf.+ Compiled on April 27, 2020, at 06:31 am, UTC.+ Usage: slynx concatenate (-a|--alphabet NAME) INPUT-FILE Concatenate sequences found in input files. Available options:+ -h,--help Show this help text+ -V,--version Show version -a,--alphabet NAME Specify alphabet type NAME INPUT-FILE Read sequences from INPUT-FILE -h,--help Show this help text+ + + ELynx+ -----+ A Haskell library and tool set for computational biology. The goal of ELynx is+ reproducible research. Evolutionary sequences and phylogenetic trees can be+ read, viewed, modified and simulated. The command line with all arguments is+ logged consistently, and automatically. Data integrity is verified using SHA256+ sums so that validation of past analyses is possible without the need to+ recompute the result.+ + slynx Analyze, modify, and simulate evolutionary sequences.+ tlynx Analyze, modify, and simulate phylogenetic trees.+ elynx Validate and redo past analyses.+ + Get help for sub commands:+ slynx examine --help ## Examine@@ -140,15 +168,37 @@ slynx examine --help - Usage: slynx examine (-a|--alphabet NAME) [INPUT-FILE] [--per-site]- Examine sequences. If data is a multi sequence alignment, additionally analyze- columns.+ ELynx Suite version 0.2.1.+ Developed by Dominik Schrempf.+ Compiled on April 27, 2020, at 06:31 am, UTC. + Usage: slynx examine (-a|--alphabet NAME) INPUT-FILE [--per-site]+ Examine sequences. If data is a multi sequence alignment, additionally analyze columns.+ Available options:+ -h,--help Show this help text+ -V,--version Show version -a,--alphabet NAME Specify alphabet type NAME INPUT-FILE Read sequences from INPUT-FILE --per-site Report per site summary statistics -h,--help Show this help text+ + + ELynx+ -----+ A Haskell library and tool set for computational biology. The goal of ELynx is+ reproducible research. Evolutionary sequences and phylogenetic trees can be+ read, viewed, modified and simulated. The command line with all arguments is+ logged consistently, and automatically. Data integrity is verified using SHA256+ sums so that validation of past analyses is possible without the need to+ recompute the result.+ + slynx Analyze, modify, and simulate evolutionary sequences.+ tlynx Analyze, modify, and simulate phylogenetic trees.+ elynx Validate and redo past analyses.+ + Get help for sub commands:+ slynx examine --help ## Filter@@ -157,12 +207,17 @@ slynx filter-rows --help - Usage: slynx filter-rows (-a|--alphabet NAME) [INPUT-FILE]- [--longer-than LENGTH] [--shorter-than LENGTH]- [--standard-characters]+ ELynx Suite version 0.2.1.+ Developed by Dominik Schrempf.+ Compiled on April 27, 2020, at 06:31 am, UTC.+ + Usage: slynx filter-rows (-a|--alphabet NAME) INPUT-FILE [--longer-than LENGTH] + [--shorter-than LENGTH] [--standard-characters] Filter rows (or sequences) found in input files. Available options:+ -h,--help Show this help text+ -V,--version Show version -a,--alphabet NAME Specify alphabet type NAME INPUT-FILE Read sequences from INPUT-FILE --longer-than LENGTH Only keep sequences longer than LENGTH@@ -170,21 +225,61 @@ --standard-characters Only keep sequences containing at least one standard (i.e., non-IUPAC) character -h,--help Show this help text+ + + ELynx+ -----+ A Haskell library and tool set for computational biology. The goal of ELynx is+ reproducible research. Evolutionary sequences and phylogenetic trees can be+ read, viewed, modified and simulated. The command line with all arguments is+ logged consistently, and automatically. Data integrity is verified using SHA256+ sums so that validation of past analyses is possible without the need to+ recompute the result.+ + slynx Analyze, modify, and simulate evolutionary sequences.+ tlynx Analyze, modify, and simulate phylogenetic trees.+ elynx Validate and redo past analyses.+ + Get help for sub commands:+ slynx examine --help Filter columns of multi sequence alignments with `filter-columns`. slynx filter-columns --help - Usage: slynx filter-columns (-a|--alphabet NAME) [INPUT-FILE]+ ELynx Suite version 0.2.1.+ Developed by Dominik Schrempf.+ Compiled on April 27, 2020, at 06:31 am, UTC.+ + Usage: slynx filter-columns (-a|--alphabet NAME) INPUT-FILE [--standard-chars DOUBLE]- Filter columns of multi-sequence alignments.+ Filter columns of multi sequence alignments. Available options:+ -h,--help Show this help text+ -V,--version Show version -a,--alphabet NAME Specify alphabet type NAME INPUT-FILE Read sequences from INPUT-FILE --standard-chars DOUBLE Keep columns with a proportion standard (non-IUPAC) characters larger than DOUBLE in [0,1] -h,--help Show this help text+ + + ELynx+ -----+ A Haskell library and tool set for computational biology. The goal of ELynx is+ reproducible research. Evolutionary sequences and phylogenetic trees can be+ read, viewed, modified and simulated. The command line with all arguments is+ logged consistently, and automatically. Data integrity is verified using SHA256+ sums so that validation of past analyses is possible without the need to+ recompute the result.+ + slynx Analyze, modify, and simulate evolutionary sequences.+ tlynx Analyze, modify, and simulate phylogenetic trees.+ elynx Validate and redo past analyses.+ + Get help for sub commands:+ slynx examine --help ## Simulate@@ -193,16 +288,22 @@ slynx simulate --help - Usage: slynx simulate (-t|--tree-file Name) [-s|--substitution-model MODEL]- [-m|--mixture-model MODEL] [-e|--edm-file NAME]- [-p|--siteprofile-files NAMES]- [-w|--mixture-model-weights "[DOUBLE,DOUBLE,...]"]+ ELynx Suite version 0.2.1.+ Developed by Dominik Schrempf.+ Compiled on April 27, 2020, at 06:31 am, UTC.+ + Usage: slynx simulate (-t|--tree-file Name) [-s|--substitution-model MODEL] + [-m|--mixture-model MODEL] [-e|--edm-file NAME] + [-p|--siteprofile-files NAMES] + [-w|--mixture-model-weights "[DOUBLE,DOUBLE,...]"] [-g|--gamma-rate-heterogeneity "(NCAT,SHAPE)"] (-l|--length NUMBER) [-S|--seed [INT]] Simulate multi sequence alignments. Available options:- -t,--tree-file Name Read trees from file NAME+ -h,--help Show this help text+ -V,--version Show version+ -t,--tree-file Name Read tree from Newick file NAME -s,--substitution-model MODEL Set the phylogenetic substitution model; available models are shown below (mutually exclusive with -m@@ -224,17 +325,24 @@ Substitution models: -s "MODEL[PARAMETER,PARAMETER,...]{STATIONARY_DISTRIBUTION}"- Supported DNA models: JC, HKY.+ Supported DNA models: JC, F81, HKY, GTR4. For example, -s HKY[KAPPA]{DOUBLE,DOUBLE,DOUBLE,DOUBLE}- Supported Protein models: Poisson, Poisson-Custom, LG, LG-Custom, WAG, WAG-Custom.+ -s GTR4[e_AC,e_AG,e_AT,e_CG,e_CT,e_GT]{DOUBLE,DOUBLE,DOUBLE,DOUBLE}+ where the 'e_XY' are the exchangeabilities from nucleotide X to Y.+ Supported Protein models: Poisson, Poisson-Custom, LG, LG-Custom, WAG, WAG-Custom, GTR20. MODEL-Custom means that only the exchangeabilities of MODEL are used, and a custom stationary distribution is provided. For example,+ -s LG -s LG-Custom{...}+ -s GTR20[e_AR,e_AN,...]{...}+ the 'e_XY' are the exchangeabilities from amino acid X to Y (alphabetical order).+ Notes: The F81 model for DNA is equivalent to the Poisson-Custom for proteins.+ The GTR4 model for DNA is equivalent to the GTR20 for proteins. Mixture models:- -m "MIXTURE(SUBSTITUTION_MODEL_1,SUBSTITUTION_MODEL_2)"+ -m "MIXTURE(SUBSTITUTION_MODEL_1,SUBSTITUTION_MODEL_2[PARAMETERS]{STATIONARY_DISTRIBUTION},...)" For example, -m "MIXTURE(JC,HKY[6.0]{0.3,0.2,0.2,0.3})" Mixture weights have to be provided with the -w option.@@ -249,6 +357,22 @@ LG exchangeabilities with stationary distributions given in FILE. -m "EDM(LG-Custom)" -e FILE For special mixture models, mixture weights are optional.+ + ELynx+ -----+ A Haskell library and tool set for computational biology. The goal of ELynx is+ reproducible research. Evolutionary sequences and phylogenetic trees can be+ read, viewed, modified and simulated. The command line with all arguments is+ logged consistently, and automatically. Data integrity is verified using SHA256+ sums so that validation of past analyses is possible without the need to+ recompute the result.+ + slynx Analyze, modify, and simulate evolutionary sequences.+ tlynx Analyze, modify, and simulate phylogenetic trees.+ elynx Validate and redo past analyses.+ + Get help for sub commands:+ slynx examine --help ## Sub-sample@@ -257,12 +381,18 @@ slynx sub-sample --help - Usage: slynx sub-sample (-a|--alphabet NAME) [INPUT-FILE]+ ELynx Suite version 0.2.1.+ Developed by Dominik Schrempf.+ Compiled on April 27, 2020, at 06:31 am, UTC.+ + Usage: slynx sub-sample (-a|--alphabet NAME) INPUT-FILE (-n|--number-of-sites INT) (-m|--number-of-alignments INT) [-S|--seed [INT]] Sub-sample columns from multi sequence alignments. Available options:+ -h,--help Show this help text+ -V,--version Show version -a,--alphabet NAME Specify alphabet type NAME INPUT-FILE Read sequences from INPUT-FILE -n,--number-of-sites INT Number of sites randomly drawn with replacement@@ -272,8 +402,23 @@ integers with up to 256 elements (default: random) -h,--help Show this help text - Create a given number of multi sequence alignments, each of which contains a- given number of random sites drawn from the original multi sequence alignment.+ Create a given number of multi sequence alignments, each of which contains a given number of random sites drawn from the original multi sequence alignment.+ + ELynx+ -----+ A Haskell library and tool set for computational biology. The goal of ELynx is+ reproducible research. Evolutionary sequences and phylogenetic trees can be+ read, viewed, modified and simulated. The command line with all arguments is+ logged consistently, and automatically. Data integrity is verified using SHA256+ sums so that validation of past analyses is possible without the need to+ recompute the result.+ + slynx Analyze, modify, and simulate evolutionary sequences.+ tlynx Analyze, modify, and simulate phylogenetic trees.+ elynx Validate and redo past analyses.+ + Get help for sub commands:+ slynx examine --help ## Translate@@ -282,17 +427,40 @@ slynx translate --help - Usage: slynx translate (-a|--alphabet NAME) [INPUT-FILE]- (-r|--reading-frame INT) (-u|--universal-code CODE)+ ELynx Suite version 0.2.1.+ Developed by Dominik Schrempf.+ Compiled on April 27, 2020, at 06:31 am, UTC.+ + Usage: slynx translate (-a|--alphabet NAME) INPUT-FILE (-r|--reading-frame INT)+ (-u|--universal-code CODE) Translate from DNA to Protein or DNAX to ProteinX. Available options:+ -h,--help Show this help text+ -V,--version Show version -a,--alphabet NAME Specify alphabet type NAME INPUT-FILE Read sequences from INPUT-FILE -r,--reading-frame INT Reading frame [0|1|2]. -u,--universal-code CODE universal code; one of: Standard, VertebrateMitochondrial. -h,--help Show this help text+ + + ELynx+ -----+ A Haskell library and tool set for computational biology. The goal of ELynx is+ reproducible research. Evolutionary sequences and phylogenetic trees can be+ read, viewed, modified and simulated. The command line with all arguments is+ logged consistently, and automatically. Data integrity is verified using SHA256+ sums so that validation of past analyses is possible without the need to+ recompute the result.+ + slynx Analyze, modify, and simulate evolutionary sequences.+ tlynx Analyze, modify, and simulate phylogenetic trees.+ elynx Validate and redo past analyses.+ + Get help for sub commands:+ slynx examine --help # TLynx@@ -301,10 +469,12 @@ tlynx --help - ELynx Suite version 0.1.0. Developed by Dominik Schrempf. Compiled on January- 30, 2020, at 09:18 am, UTC.+ ELynx Suite version 0.2.1.+ Developed by Dominik Schrempf.+ Compiled on April 27, 2020, at 06:31 am, UTC. - Usage: tlynx [-v|--verbosity VALUE] [-o|--output-file-basename NAME] COMMAND+ Usage: tlynx [-v|--verbosity VALUE] [-o|--output-file-basename NAME] + [-f|--force] COMMAND Compare, examine, and simulate phylogenetic trees. Available options:@@ -314,35 +484,37 @@ Debug (default: Info) -o,--output-file-basename NAME Specify base name of output file+ -f,--force Ignore previous analysis and overwrite existing+ output files. Available commands:+ coalesce Simulate phylogenetic trees using the coalescent processes (see also the 'simulate' command for simulations using the birth and death process).+ compare Compare two phylogenetic trees (compute distances and branch-wise differences).+ connect Connect two phylogenetic trees in all ways (possibly honoring constraints). distance Compute distances between many phylogenetic trees. examine Compute summary statistics of phylogenetic trees.- simulate Simulate phylogenetic trees using birth and death- processes.- compare Compare two phylogenetic trees (compute distances and- branch-wise differences).- connect Connect two phylogenetic trees in all ways (possibly- honoring constraints).- shuffle Shuffle a phylogenetic tree (keep coalescent times,- but shuffle topology and leaves).+ shuffle Shuffle a phylogenetic tree (keep coalescent times, but shuffle topology and leaves).+ simulate Simulate phylogenetic trees using birth and death processes (see also the 'coalesce' command for simulations using the coalescent process). - File formats:- - Newick+ Available tree file formats:+ - Newick Standard: Branch support values are stored in square brackets after branch lengths.+ - Newick IqTree: Branch support values are stored as node names after the closing bracket of forests.+ - Newick RevBayes Key-value pairs is provided in square brackets after node names as well as branch lengths. XXX: Key value pairs are IGNORED at the moment. - The ELynx Suite- ---------------- A Haskell library and a tool set for computational biology. The goal of the- ELynx Suite is reproducible research. Evolutionary sequences and phylogenetic- trees can be read, viewed, modified and simulated. Exact specification of all- options is necessary, and nothing is assumed about the data (e.g., the type of- code). The command line with all arguments is consistently, and automatically- logged.+ ELynx+ -----+ A Haskell library and tool set for computational biology. The goal of ELynx is+ reproducible research. Evolutionary sequences and phylogenetic trees can be+ read, viewed, modified and simulated. The command line with all arguments is+ logged consistently, and automatically. Data integrity is verified using SHA256+ sums so that validation of past analyses is possible without the need to+ recompute the result. slynx Analyze, modify, and simulate evolutionary sequences. tlynx Analyze, modify, and simulate phylogenetic trees.+ elynx Validate and redo past analyses. - Get help for specific commands:+ Get help for sub commands: slynx examine --help @@ -352,18 +524,43 @@ tlynx compare --help - Usage: tlynx compare [-n|--normalize] [-b|--bipartitions] [-i|--newick-iqtree]- NAME- Compare two phylogenetic trees (compute distances and branch-wise- differences).+ ELynx Suite version 0.2.1.+ Developed by Dominik Schrempf.+ Compiled on April 27, 2020, at 06:31 am, UTC. + Usage: tlynx compare [-n|--normalize] [-b|--bipartitions] [-t|--intersect] + [-f|--newick-format FORMAT] NAME+ Compare two phylogenetic trees (compute distances and branch-wise differences).+ Available options:+ -h,--help Show this help text+ -V,--version Show version -n,--normalize Normalize trees before comparison- -b,--bipartitions Print common and missing bipartitions- -i,--newick-iqtree Use IQ-TREE Newick format (internal node labels are- branch support values)+ -b,--bipartitions Print and plot common and missing bipartitions+ -t,--intersect Compare intersections; i.e., before comparison, drop+ leaves that are not present in the other tree+ -f,--newick-format FORMAT+ Newick tree format; see 'tlynx+ --help' (default: Standard) NAME Tree file -h,--help Show this help text+ + + ELynx+ -----+ A Haskell library and tool set for computational biology. The goal of ELynx is+ reproducible research. Evolutionary sequences and phylogenetic trees can be+ read, viewed, modified and simulated. The command line with all arguments is+ logged consistently, and automatically. Data integrity is verified using SHA256+ sums so that validation of past analyses is possible without the need to+ recompute the result.+ + slynx Analyze, modify, and simulate evolutionary sequences.+ tlynx Analyze, modify, and simulate phylogenetic trees.+ elynx Validate and redo past analyses.+ + Get help for sub commands:+ slynx examine --help ## Examine@@ -372,14 +569,38 @@ tlynx examine --help - Usage: tlynx examine [INPUT-FILE] [-i|--newick-iqtree]+ ELynx Suite version 0.2.1.+ Developed by Dominik Schrempf.+ Compiled on April 27, 2020, at 06:31 am, UTC.+ + Usage: tlynx examine INPUT-FILE [-f|--newick-format FORMAT] Compute summary statistics of phylogenetic trees. Available options:+ -h,--help Show this help text+ -V,--version Show version INPUT-FILE Read trees from INPUT-FILE- -i,--newick-iqtree Use IQ-TREE Newick format (internal node labels are- branch support values)+ -f,--newick-format FORMAT+ Newick tree format; see 'tlynx+ --help' (default: Standard) -h,--help Show this help text+ + + ELynx+ -----+ A Haskell library and tool set for computational biology. The goal of ELynx is+ reproducible research. Evolutionary sequences and phylogenetic trees can be+ read, viewed, modified and simulated. The command line with all arguments is+ logged consistently, and automatically. Data integrity is verified using SHA256+ sums so that validation of past analyses is possible without the need to+ recompute the result.+ + slynx Analyze, modify, and simulate evolutionary sequences.+ tlynx Analyze, modify, and simulate phylogenetic trees.+ elynx Validate and redo past analyses.+ + Get help for sub commands:+ slynx examine --help ## Simulate@@ -388,13 +609,19 @@ tlynx simulate --help - Usage: tlynx simulate [-t|--nTrees INT] [-n|--nLeaves INT] [-H|--height DOUBLE]- [-M|--condition-on-mrca] [-l|--lambda DOUBLE]- [-m|--mu DOUBLE] [-r|--rho DOUBLE] [-u|--sub-sample]+ ELynx Suite version 0.2.1.+ Developed by Dominik Schrempf.+ Compiled on April 27, 2020, at 06:31 am, UTC.+ + Usage: tlynx simulate [-t|--nTrees INT] [-n|--nLeaves INT] [-H|--height DOUBLE] + [-M|--condition-on-mrca] [-l|--lambda DOUBLE] + [-m|--mu DOUBLE] [-r|--rho DOUBLE] [-u|--sub-sample] [-s|--summary-statistics] [-S|--seed [INT]]- Simulate phylogenetic trees using birth and death processes.+ Simulate phylogenetic trees using birth and death processes (see also the 'coalesce' command for simulations using the coalescent process). Available options:+ -h,--help Show this help text+ -V,--version Show version -t,--nTrees INT Number of trees (default: 10) -n,--nLeaves INT Number of leaves per tree (default: 5) -H,--height DOUBLE Fix tree height (no default)@@ -413,15 +640,75 @@ Summary statistics only: only print (NumberOfExtantChildren BranchLength) pairs for each branch of each tree. The trees are separated by a newline character. Sub-sampling: simulate one big tree with n'=round(n/rho), n'>=n, leaves, and randomly sample sub-trees with n leaves. Hence, with rho=1.0, the same tree is reported over and over again. Gernhard, T. (2008). The conditioned reconstructed process. Journal of Theoretical Biology, 253(4), 769–778. http://doi.org/10.1016/j.jtbi.2008.04.005+ + ELynx+ -----+ A Haskell library and tool set for computational biology. The goal of ELynx is+ reproducible research. Evolutionary sequences and phylogenetic trees can be+ read, viewed, modified and simulated. The command line with all arguments is+ logged consistently, and automatically. Data integrity is verified using SHA256+ sums so that validation of past analyses is possible without the need to+ recompute the result.+ + slynx Analyze, modify, and simulate evolutionary sequences.+ tlynx Analyze, modify, and simulate phylogenetic trees.+ elynx Validate and redo past analyses.+ + Get help for sub commands:+ slynx examine --help # ELynx -Documentation of the library can be found on [Hackage](https://hackage.haskell.org/):+Validate and (optionally) redo past ELynx analyses. + elynx --help++ ELynx Suite version 0.2.1.+ Developed by Dominik Schrempf.+ Compiled on April 27, 2020, at 06:31 am, UTC.+ + Usage: elynx COMMAND+ Validate and redo past ELynx analyses+ + Available options:+ -h,--help Show this help text+ -V,--version Show version+ + Available commands:+ validate Validate an ELynx analysis+ redo Redo an ELynx analysis+ + + ELynx+ -----+ A Haskell library and tool set for computational biology. The goal of ELynx is+ reproducible research. Evolutionary sequences and phylogenetic trees can be+ read, viewed, modified and simulated. The command line with all arguments is+ logged consistently, and automatically. Data integrity is verified using SHA256+ sums so that validation of past analyses is possible without the need to+ recompute the result.+ + slynx Analyze, modify, and simulate evolutionary sequences.+ tlynx Analyze, modify, and simulate phylogenetic trees.+ elynx Validate and redo past analyses.+ + Get help for sub commands:+ slynx examine --help+++# Library documentation++Documentation of the libraries can be found on [Hackage](https://hackage.haskell.org/):++- [elynx-markov](https://hackage.haskell.org/package/elynx-markov) - [elynx-seq](https://hackage.haskell.org/package/elynx-seq)-- [elynx-tree](https://hackage.haskell.org/package/elynx-tree) - [elynx-tools](https://hackage.haskell.org/package/elynx-tools)+- [elynx-tree](https://hackage.haskell.org/package/elynx-tree)++Documentation of the executables is also available:++- [elynx](https://hackage.haskell.org/package/elynx) - [slynx](https://hackage.haskell.org/package/slynx) - [tlynx](https://hackage.haskell.org/package/tlynx)
elynx-tools.cabal view
@@ -1,9 +1,9 @@ cabal-version: 1.12 name: elynx-tools-version: 0.1.0+version: 0.2.1 license: GPL-3 license-file: LICENSE-copyright: Dominik Schrempf (2019)+copyright: Dominik Schrempf (2020) maintainer: dominik.schrempf@gmail.com author: Dominik Schrempf homepage: https://github.com/dschrempf/elynx#readme@@ -23,6 +23,9 @@ library exposed-modules:+ ELynx.Tools+ hs-source-dirs: src+ other-modules: ELynx.Tools.ByteString ELynx.Tools.Concurrent ELynx.Tools.Definitions@@ -34,39 +37,38 @@ ELynx.Tools.Matrix ELynx.Tools.Misc ELynx.Tools.Numeric- ELynx.Tools.Options+ ELynx.Tools.Parallel ELynx.Tools.Random ELynx.Tools.Reproduction ELynx.Tools.Text ELynx.Tools.Vector- hs-source-dirs: src- other-modules: Paths_elynx_tools default-language: Haskell2010- ghc-options: -Wall -O2+ ghc-options: -Wall -fllvm build-depends:- aeson >=1.4.5.0 && <1.5,+ aeson >=1.4.7.1 && <1.5, async >=2.2.2 && <2.3,- base >=4.12.0.0 && <4.13,- bytestring >=0.10.8.2 && <0.11,- containers >=0.6.0.1 && <0.7,+ base >=4.13.0.0 && <4.14,+ base16-bytestring >=0.1.1.6 && <0.2,+ bytestring >=0.10.10.0 && <0.11,+ containers >=0.6.2.1 && <0.7, cryptohash-sha256 >=0.11.101.0 && <0.12, deepseq >=1.4.4.0 && <1.5,- fast-logger >=2.4.17 && <2.5,+ directory >=1.3.6.0 && <1.4,+ fast-logger >=3.0.1 && <3.1, hmatrix >=0.20.0.0 && <0.21,- lifted-base >=0.2.3.12 && <0.3, matrices >=0.5.0 && <0.6,- megaparsec >=7.0.5 && <7.1,+ megaparsec >=8.0.0 && <8.1, monad-control >=1.0.2.3 && <1.1,- monad-logger >=0.3.30 && <0.4,+ monad-logger >=0.3.32 && <0.4, mwc-random >=0.14.0.0 && <0.15,- optparse-applicative >=0.14.3.0 && <0.15,+ optparse-applicative >=0.15.1.0 && <0.16, parallel >=3.2.2.0 && <3.3,- primitive >=0.6.4.0 && <0.7,- template-haskell >=2.14.0.0 && <2.15,- text >=1.2.3.1 && <1.3,- time >=1.8.0.2 && <1.9,+ primitive >=0.7.0.1 && <0.8,+ template-haskell >=2.15.0.0 && <2.16,+ text >=1.2.4.0 && <1.3,+ time >=1.9.3 && <1.10, transformers >=0.5.6.2 && <0.6, transformers-base >=0.4.5.2 && <0.5,- vector >=0.12.0.3 && <0.13,+ vector >=0.12.1.2 && <0.13, zlib >=0.6.2.1 && <0.7
+ src/ELynx/Tools.hs view
@@ -0,0 +1,52 @@+{- |+Module : ELynx.Tools+Description : Tools and other niceties+Copyright : (c) Dominik Schrempf 2020+License : GPL-3.0-or-later++Maintainer : dominik.schrempf@gmail.com+Stability : unstable+Portability : portable++Creation date: Sat Mar 21 13:56:03 2020.++A large collection of tools.++-}++module ELynx.Tools+ ( module ELynx.Tools.ByteString+ , module ELynx.Tools.Concurrent+ , module ELynx.Tools.Definitions+ , module ELynx.Tools.Equality+ , module ELynx.Tools.InputOutput+ , module ELynx.Tools.LinearAlgebra+ , module ELynx.Tools.List+ , module ELynx.Tools.Logger+ , module ELynx.Tools.Matrix+ , module ELynx.Tools.Misc+ , module ELynx.Tools.Numeric+ , module ELynx.Tools.Parallel+ , module ELynx.Tools.Random+ , module ELynx.Tools.Reproduction+ , module ELynx.Tools.Text+ , module ELynx.Tools.Vector+ )+where++import ELynx.Tools.ByteString+import ELynx.Tools.Concurrent+import ELynx.Tools.Definitions+import ELynx.Tools.Equality+import ELynx.Tools.InputOutput+import ELynx.Tools.LinearAlgebra+import ELynx.Tools.List+import ELynx.Tools.Logger+import ELynx.Tools.Matrix+import ELynx.Tools.Misc+import ELynx.Tools.Numeric+import ELynx.Tools.Parallel+import ELynx.Tools.Random+import ELynx.Tools.Reproduction+import ELynx.Tools.Text+import ELynx.Tools.Vector
src/ELynx/Tools/ByteString.hs view
@@ -1,7 +1,7 @@ {- | Module : ELynx.Tools.ByteString-Copyright : (c) Dominik Schrempf 2019-License : GPL-3+Copyright : (c) Dominik Schrempf 2020+License : GPL-3.0-or-later Maintainer : dominik.schrempf@gmail.com Stability : unstable@@ -15,7 +15,7 @@ module ELynx.Tools.ByteString (- -- * ByteString handling.+ -- * ByteString handling alignRightWith , alignRight , alignLeftWith@@ -24,15 +24,19 @@ , c2w , w2c , bShow- ) where+ )+where -import Data.ByteString.Internal (c2w, w2c)-import qualified Data.ByteString.Lazy.Char8 as L+import Data.ByteString.Internal ( c2w+ , w2c+ )+import qualified Data.ByteString.Lazy.Char8 as L -- | For a given width, align string to the right; use given fill character; -- trim on the left if string is longer. alignRightWith :: Char -> Int -> L.ByteString -> L.ByteString-alignRightWith c n s = L.replicate (fromIntegral n - l) c <> L.take (fromIntegral n) s+alignRightWith c n s =+ L.replicate (fromIntegral n - l) c <> L.take (fromIntegral n) s where l = L.length s -- | For a given width, align string to the right; trim on the left if string is@@ -43,7 +47,8 @@ -- | For a given width, align string to the left; use given fill character; trim -- on the right if string is longer. alignLeftWith :: Char -> Int -> L.ByteString -> L.ByteString-alignLeftWith c n s = L.take (fromIntegral n) s <> L.replicate (fromIntegral n - l) c+alignLeftWith c n s =+ L.take (fromIntegral n) s <> L.replicate (fromIntegral n - l) c where l = L.length s -- | For a given width, align string to the left; trim on the right if string is@@ -54,8 +59,9 @@ -- | If a string is longer than a given value, trim it and add some dots. summarizeByteString :: Int -> L.ByteString -> L.ByteString-summarizeByteString l s | L.length s >= fromIntegral l = L.take (fromIntegral l) s <> L.pack "..."- | otherwise = s+summarizeByteString l s+ | L.length s >= fromIntegral l = L.take (fromIntegral l) s <> L.pack "..."+ | otherwise = s -- | Conversion to 'L.ByteString'. bShow :: Show a => a -> L.ByteString
src/ELynx/Tools/Concurrent.hs view
@@ -2,8 +2,8 @@ {- | Module : ELynx.Tools.Concurrent Description : Tools for concurrent random calculations-Copyright : (c) Dominik Schrempf 2019-License : GPL-3+Copyright : (c) Dominik Schrempf 2020+License : GPL-3.0-or-later Maintainer : dominik.schrempf@gmail.com Stability : unstable@@ -16,39 +16,28 @@ module ELynx.Tools.Concurrent ( -- * MWC- getNGen- , splitGen- -- * Parallel stuff+ splitGen+ -- * Concurrent calculations , parComp , getChunks- , parMapChunk- ) where+ )+where import Control.Concurrent import Control.Concurrent.Async import Control.Monad--- import qualified Control.Monad.Parallel as P import Control.Monad.Primitive-import Control.Parallel.Strategies-import qualified Data.Vector as V+import qualified Data.Vector as V import Data.Word import System.Random.MWC --- import ELynx.Tools.Definitions---- | Get a number of generators, possibly with a fixed seed.-getNGen :: Int -> Maybe [Word32] -> IO [GenIO]-getNGen n ms = do- g <- maybe createSystemRandom (initialize . V.fromList) ms- splitGen n g- -- | Split a generator. splitGen :: PrimMonad m => Int -> Gen (PrimState m) -> m [Gen (PrimState m)] splitGen n gen- | n <= 0 = return []+ | n <= 0 = return [] | otherwise = do- seeds :: [V.Vector Word32] <- replicateM n $ uniformVector gen 256- mapM initialize seeds+ seeds :: [V.Vector Word32] <- replicateM n $ uniformVector gen 256+ mapM initialize seeds -- -- XXX: This just doesn't work... The only thing I found: -- -- https://stackoverflow.com/a/16250010.@@ -63,7 +52,7 @@ -- | Perform random calculation in parallel. Does only work with 'IO' and the moment. parComp :: Int -> (Int -> GenIO -> IO b) -> GenIO -> IO [b] parComp num fun gen = do- ncap <- getNumCapabilities+ ncap <- getNumCapabilities let chunks = getChunks ncap num gs <- splitGen ncap gen mapConcurrently (uncurry fun) (zip chunks gs)@@ -73,11 +62,7 @@ -- to the number of calculations. getChunks :: Int -> Int -> [Int] getChunks c n = ns- where n' = n `div` c- r = n `mod` c- ns = replicate r (n'+1) ++ replicate (c - r) n'---- XXX: This should actually be in a module called Parallel, not Concurrent.--- | Parallel map with given chunk size.-parMapChunk :: Int -> (a -> b) -> [a] -> [b]-parMapChunk n f as = map f as `using` parListChunk n rseq+ where+ n' = n `div` c+ r = n `mod` c+ ns = replicate r (n' + 1) ++ replicate (c - r) n'
src/ELynx/Tools/Definitions.hs view
@@ -1,8 +1,8 @@ {- | Module : ELynx.Tools.Definitions Description : Some global definitions-Copyright : (c) Dominik Schrempf 2019-License : GPL-3+Copyright : (c) Dominik Schrempf 2020+License : GPL-3.0-or-later Maintainer : dominik.schrempf@gmail.com Stability : unstable@@ -13,14 +13,36 @@ -} module ELynx.Tools.Definitions- ( eps+ ( -- * Definitions+ chunksize+ , eps+ , eps' , precision- , chunksize- ) where+ , version+ )+where +import Data.Version++-- XXX: Manually handled, because automatic handling with @Paths_...@ modules+-- results in errors during source tarball creation and upload.++-- Careful, this version number has to be synchronized with the one specified in+-- the *.cabal files. This is done with the @bump-version@ script.++-- | ELynx version.+version :: Version+version = read "0.2.1"+ -- | Required precision when comparing 'Double' values. eps :: Double eps = 1e-12++-- | Relaxed required precision when comparing 'Double' values. Often, input+-- files do not provide the required double precision. In this case, it is only+-- possible to check input with relaxed bounds.+eps' :: Double+eps' = 1e-5 -- | Default output precision. precision :: Int
src/ELynx/Tools/Equality.hs view
@@ -1,7 +1,7 @@ {- | Module : ELynx.Tools.Equality-Copyright : (c) Dominik Schrempf 2019-License : GPL-3+Copyright : (c) Dominik Schrempf 2020+License : GPL-3.0-or-later Maintainer : dominik.schrempf@gmail.com Stability : unstable@@ -14,7 +14,7 @@ -} module ELynx.Tools.Equality- (+ ( -- * Equality allEqual , allNearlyEqualWith , allNearlyEqual@@ -28,7 +28,8 @@ , nearlyEqVec , nearlyEqMatWith , nearlyEqMat- ) where+ )+where import Numeric.LinearAlgebra @@ -52,7 +53,7 @@ -- | Test for equality with given tolerance (needed because of machine precision). nearlyEqWith :: Double -> Double -> Double -> Bool-nearlyEqWith tol a b = tol > abs (a-b)+nearlyEqWith tol a b = tol > abs (a - b) -- | Test for equality with predefined tolerance 'eps' (needed because of -- machine precision).
src/ELynx/Tools/InputOutput.hs view
@@ -1,10 +1,13 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE TemplateHaskell #-}+{-# LANGUAGE LambdaCase #-}+{-# LANGUAGE ScopedTypeVariables #-}+{-# LANGUAGE TypeApplications #-} {- | Module : ELynx.Tools.InputOutput-Copyright : (c) Dominik Schrempf 2019-License : GPL-3+Copyright : (c) Dominik Schrempf 2020+License : GPL-3.0-or-later Maintainer : dominik.schrempf@gmail.com Stability : unstable@@ -17,42 +20,79 @@ -} module ELynx.Tools.InputOutput- (- -- * Input, output.- readGZFile+ ( -- * Input, output+ getOutFilePath+ , openFile'+ , readGZFile , writeGZFile , out , outHandle- -- * Parsing.+ -- * Parsing , runParserOnFile , parseFileWith , parseIOWith , parseFileOrIOWith , parseStringWith , parseByteStringWith- ) where+ )+where -import Codec.Compression.GZip (compress, decompress)-import Control.DeepSeq (force)-import Control.Exception (evaluate)-import Control.Monad ((<=<))+import Codec.Compression.GZip ( compress+ , decompress+ )+import Control.Monad.Trans.Reader ( ask )+import Control.DeepSeq ( force )+import Control.Exception ( evaluate )+import Control.Monad ( (<=<) ) import Control.Monad.IO.Class import Control.Monad.Logger-import qualified Data.ByteString.Lazy.Char8 as L-import Data.List (isSuffixOf)+import qualified Data.ByteString.Lazy.Char8 as L+import Data.List ( isSuffixOf ) import Data.Maybe-import qualified Data.Text as T+import qualified Data.Text as T import System.IO+import System.Directory ( doesFileExist ) import Text.Megaparsec +import ELynx.Tools.Reproduction ( ELynx+ , Force(..)+ , Arguments(..)+ , outFileBaseName+ , forceReanalysis+ , Reproducible(..)+ )++-- | Get out file path with extension.+getOutFilePath+ :: forall a . Reproducible a => String -> ELynx a (Maybe FilePath)+getOutFilePath ext = do+ a <- ask+ let bn = outFileBaseName . global $ a+ sfxs = outSuffixes a+ if ext `elem` sfxs+ then return $ (++ ext) <$> bn+ else+ error+ "getOutFilePath: out file suffix not registered. Please contact maintainer."++checkFile :: Force -> FilePath -> IO ()+checkFile (Force True ) _ = return ()+checkFile (Force False) fp = doesFileExist fp >>= \case+ True ->+ error+ $ "File exists: "+ <> fp+ <> ". Please use the --force option to repeat an analysis."+ False -> return ()++-- | Open existing files only if 'Force' is true.+openFile' :: Force -> FilePath -> IOMode -> IO Handle+openFile' frc fp md = checkFile frc fp >> openFile fp md+ -- XXX: For now, all files are read strictly (see help of -- Control.DeepSeq.force). readFile' :: FilePath -> IO L.ByteString--- -- Doesn't work.--- readFile' fn = withFile fn ReadMode $ (evaluate . force) <=< L.hGetContents--- This works!-readFile' fn = withFile fn ReadMode $- (evaluate . force) <=< L.hGetContents+readFile' fn = withFile fn ReadMode $ (evaluate . force) <=< L.hGetContents -- | Read file. If file path ends with ".gz", assume gzipped file and decompress -- before read.@@ -62,76 +102,87 @@ -- | Write file. If file path ends with ".gz", assume gzipped file and compress -- before write.-writeGZFile :: FilePath -> L.ByteString -> IO ()-writeGZFile f | ".gz" `isSuffixOf` f = L.writeFile f . compress- | otherwise = L.writeFile f+writeGZFile :: Force -> FilePath -> L.ByteString -> IO ()+writeGZFile frc f r+ | ".gz" `isSuffixOf` f = checkFile frc f >> L.writeFile f (compress r)+ | otherwise = checkFile frc f >> L.writeFile f r -- | Parse a possibly gzipped file.-runParserOnFile :: Parsec e L.ByteString a- -> FilePath -> IO (Either (ParseErrorBundle L.ByteString e) a)+runParserOnFile+ :: Parsec e L.ByteString a+ -> FilePath+ -> IO (Either (ParseErrorBundle L.ByteString e) a) runParserOnFile p f = parse p f <$> readGZFile f -- | Parse a possibly gzipped file and extract the result.-parseFileWith :: (ShowErrorComponent e)- => Parsec e L.ByteString a -- ^ The parser.- -> FilePath- -> IO a+parseFileWith+ :: (ShowErrorComponent e)+ => Parsec e L.ByteString a -- ^ The parser.+ -> FilePath+ -> IO a parseFileWith p f = parseFileOrIOWith p (Just f) -- | Parse standard input.-parseIOWith :: (ShowErrorComponent e)- => Parsec e L.ByteString a -- ^ The parser.- -> IO a+parseIOWith+ :: (ShowErrorComponent e)+ => Parsec e L.ByteString a -- ^ The parser.+ -> IO a parseIOWith p = parseByteStringWith "Standard input" p <$> L.getContents -- | Parse a possibly gzipped file, or standard input, and extract the result.-parseFileOrIOWith :: (ShowErrorComponent e)- => Parsec e L.ByteString a -- ^ The parser.- -> Maybe FilePath -- ^ If no file path is given, standard input is used.- -> IO a+parseFileOrIOWith+ :: (ShowErrorComponent e)+ => Parsec e L.ByteString a -- ^ The parser.+ -> Maybe FilePath -- ^ If no file path is given, standard input is used.+ -> IO a parseFileOrIOWith p mf = do- contents <- case mf of- Nothing -> L.getContents- Just f -> readGZFile f+ contents <- maybe L.getContents readGZFile mf return $ parseByteStringWith (fromMaybe "Standard input" mf) p contents -- | Parse a 'String' and extract the result.-parseStringWith :: (ShowErrorComponent e)- => String -- ^ Name of string.- -> Parsec e L.ByteString a -- ^ Parser.- -> String -- ^ Input.- -> a+parseStringWith+ :: (ShowErrorComponent e)+ => String -- ^ Name of string.+ -> Parsec e L.ByteString a -- ^ Parser.+ -> String -- ^ Input.+ -> a parseStringWith s p l = parseByteStringWith s p (L.pack l) -- | Parse a 'L.ByteString' and extract the result.-parseByteStringWith :: (ShowErrorComponent e)- => String -- ^ Name of byte string.- -> Parsec e L.ByteString a -- ^ Parser.- -> L.ByteString -- ^ Input.- -> a+parseByteStringWith+ :: (ShowErrorComponent e)+ => String -- ^ Name of byte string.+ -> Parsec e L.ByteString a -- ^ Parser.+ -> L.ByteString -- ^ Input.+ -> a parseByteStringWith s p l = case parse p s l of- Left err -> error $ errorBundlePretty err- Right val -> val+ Left err -> error $ errorBundlePretty err+ Right val -> val --- | Write a result with a given name to file or standard output.-out :: (MonadLogger m, MonadIO m) => String -> L.ByteString -> Maybe FilePath -> m ()-out name res mfp =+-- | Write a result with a given name to file with given extension or standard+-- output. Supports compression.+out :: Reproducible a => String -> L.ByteString -> String -> ELynx a ()+out name res ext = do+ mfp <- getOutFilePath ext case mfp of Nothing -> do $(logInfo) $ T.pack $ "Write " <> name <> " to standard output." liftIO $ L.putStr res Just fp -> do $(logInfo) $ T.pack $ "Write " <> name <> " to file '" <> fp <> "'."- liftIO $ writeGZFile fp res+ frc <- forceReanalysis . global <$> ask+ liftIO $ writeGZFile frc fp res -- | Get an output handle, does not support compression. The handle has to be -- closed after use!-outHandle :: (MonadLogger m, MonadIO m) => String -> Maybe FilePath -> m Handle-outHandle name mfp =+outHandle :: Reproducible a => String -> String -> ELynx a Handle+outHandle name ext = do+ mfp <- getOutFilePath ext case mfp of Nothing -> do $(logInfo) $ T.pack $ "Write " <> name <> " to standard output." return stdout Just fp -> do $(logInfo) $ T.pack $ "Write " <> name <> " to file '" <> fp <> "'."- liftIO $ openFile fp WriteMode+ frc <- forceReanalysis . global <$> ask+ liftIO $ openFile' frc fp WriteMode
src/ELynx/Tools/LinearAlgebra.hs view
@@ -1,7 +1,7 @@ {- | Module : ELynx.Tools.LinearAlgebra-Copyright : (c) Dominik Schrempf 2019-License : GPL-3+Copyright : (c) Dominik Schrempf 2020+License : GPL-3.0-or-later Maintainer : dominik.schrempf@gmail.com Stability : unstable@@ -15,13 +15,14 @@ module ELynx.Tools.LinearAlgebra (- -- * Matrices.+ -- * Linear Algebra matrixSeparateSymSkew , matrixSetDiagToZero , dispv , dispm , dispmi- ) where+ )+where import Data.List @@ -30,9 +31,10 @@ -- | Separate a square matrix into a symmetric and a skew-symmetric matrix. matrixSeparateSymSkew :: Matrix R -> (Matrix R, Matrix R) matrixSeparateSymSkew m = (mSym, mSkew)- where trM = tr m- mSym = scale 0.5 $ m + trM- mSkew = scale 0.5 $ m - trM+ where+ trM = tr m+ mSym = scale 0.5 $ m + trM+ mSkew = scale 0.5 $ m - trM -- | Set the diagonal entries of a matrix to zero. matrixSetDiagToZero :: Matrix R -> Matrix R@@ -48,4 +50,5 @@ -- | Display a matrix with given precision and indent. dispmi :: Int -> Int -> Matrix R -> String-dispmi p i m = intercalate "\n" $ map (replicate i ' ' ++) $ tail $ lines $ dispf p m+dispmi p i m =+ intercalate "\n" $ map (replicate i ' ' ++) $ tail $ lines $ dispf p m
src/ELynx/Tools/List.hs view
@@ -1,8 +1,8 @@ {- | Module : ELynx.Tools.List Description : Additional tools for lists-Copyright : (c) Dominik Schrempf 2019-License : GPL-3+Copyright : (c) Dominik Schrempf 2020+License : GPL-3.0-or-later Maintainer : dominik.schrempf@gmail.com Stability : unstable@@ -13,9 +13,11 @@ -} module ELynx.Tools.List- ( sortWithIndices- , randomInsert- ) where+ ( -- * Lists+ sortListWithIndices+ , randomInsertList+ )+where import Control.Monad.Primitive import Data.Function@@ -23,12 +25,12 @@ import System.Random.MWC -- | Sort a list and also return original indices.-sortWithIndices :: Ord a => [a] -> [(a, Int)]-sortWithIndices xs = sortBy (compare `on` fst) $ zip xs ([0..] :: [Int])+sortListWithIndices :: Ord a => [a] -> [(a, Int)]+sortListWithIndices xs = sortBy (compare `on` fst) $ zip xs ([0 ..] :: [Int]) -- | Insert element into random position of list.-randomInsert :: PrimMonad m => a -> [a] -> Gen (PrimState m) -> m [a]-randomInsert e v g = do+randomInsertList :: PrimMonad m => a -> [a] -> Gen (PrimState m) -> m [a]+randomInsertList e v g = do let l = length v i <- uniformR (0, l) g return $ take i v ++ [e] ++ drop i v
src/ELynx/Tools/Logger.hs view
@@ -1,12 +1,14 @@ {-# LANGUAGE FlexibleContexts #-} {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE TemplateHaskell #-}+{-# LANGUAGE TypeApplications #-}+{-# LANGUAGE ScopedTypeVariables #-} {- | Module : ELynx.Tools.Logger Description : Monad logger utility functions-Copyright : (c) Dominik Schrempf 2019-License : GPL-3+Copyright : (c) Dominik Schrempf 2020+License : GPL-3.0-or-later Maintainer : dominik.schrempf@gmail.com Stability : unstable@@ -17,81 +19,143 @@ -} module ELynx.Tools.Logger- ( logNewSection+ ( -- * Logger+ logNewSection , eLynxWrapper- ) where+ )+where -import Control.Exception.Lifted-import Control.Monad.Base-import Control.Monad.IO.Class-import Control.Monad.Logger-import Control.Monad.Trans.Control-import qualified Data.ByteString.Char8 as B-import Data.Text-import System.IO-import System.Log.FastLogger+import Control.Monad.Base ( liftBase )+import Control.Monad.IO.Class ( MonadIO+ , liftIO+ )+import Control.Monad.Logger ( Loc+ , LogLevel+ , LogSource+ , LoggingT+ , MonadLogger+ , filterLogger+ , logInfo+ , runLoggingT+ )+import Control.Monad.Trans.Control ( MonadBaseControl )+import Control.Monad.Trans.Reader ( ReaderT(runReaderT) )+import qualified Data.ByteString.Char8 as B+import Data.Text ( Text+ , pack+ )+import System.IO ( BufferMode(LineBuffering)+ , Handle+ , IOMode(WriteMode)+ , hClose+ , hSetBuffering+ , stderr+ )+import System.Log.FastLogger ( LogStr+ , fromLogStr+ )+import System.Random.MWC ( createSystemRandom+ , save+ , fromSeed+ ) -import ELynx.Tools.Options+import ELynx.Tools.Reproduction ( Reproducible(..)+ , writeReproduction+ , ToJSON+ , toLogLevel+ , ELynx+ , Arguments(..)+ , Force+ , GlobalArguments(..)+ , Seed(..)+ , logHeader+ , logFooter+ )+import ELynx.Tools.InputOutput ( openFile' ) -- | Unified way of creating a new section in the log. logNewSection :: MonadLogger m => Text -> m ()-logNewSection s = $(logInfo) $ "-- " <> s+logNewSection s = $(logInfo) $ "== " <> s --- | The 'LoggingT' wrapper for ELynx. Prints a header and a footer, logs to--- 'stderr' if no file is provided. If a log file is provided, log to the file--- and to 'stderr'.-eLynxWrapper :: (MonadBaseControl IO m, MonadIO m)- => LogLevel -> Maybe FilePath -> String -> LoggingT m () -> m ()-eLynxWrapper lvl logFile headerMsg worker = runELynxLoggingT lvl logFile $- do- h <- liftIO $ logHeader headerMsg- $(logInfo) $ pack h- worker- f <- liftIO logFooter- $(logInfo) $ pack f+-- | The 'ReaderT' and 'LoggingT' wrapper for ELynx. Prints a header and a+-- footer, logs to 'stderr' if no file is provided. Initializes the seed if none+-- is provided. If a log file is provided, log to the file and to 'stderr'.+eLynxWrapper+ :: forall a b+ . (Eq a, Show a, Reproducible a, ToJSON a)+ => Arguments a+ -> (Arguments a -> Arguments b)+ -> ELynx b ()+ -> IO ()+eLynxWrapper args f worker = do+ -- Arguments.+ let gArgs = global args+ lArgs = local args+ let lvl = toLogLevel $ verbosity gArgs+ rd = forceReanalysis gArgs+ outBn = outFileBaseName gArgs+ logFile = (++ ".log") <$> outBn+ runELynxLoggingT lvl rd logFile $ do+ -- Header.+ h <- liftIO $ logHeader (cmdName @a) (cmdDsc @a)+ $(logInfo) $ pack $ h ++ "\n"+ -- Fix seed.+ lArgs' <- case getSeed lArgs of+ Nothing -> return lArgs+ Just Random -> do+ -- XXX: Have to go via a generator here, since creation of seed is not+ -- supported.+ g <- liftIO createSystemRandom+ s <- liftIO $ fromSeed <$> save g+ $(logInfo) $ pack $ "Seed: random; set to " <> show s <> "."+ return $ setSeed lArgs s+ Just (Fixed s) -> do+ $(logInfo) $ pack $ "Seed: " <> show s <> "."+ return lArgs+ let args' = Arguments gArgs lArgs'+ -- Run the worker with the fixed seed.+ runReaderT worker $ f args'+ -- Reproduction file.+ case outBn of+ Nothing ->+ $(logInfo)+ "No output file given --- skip writing ELynx file for reproducible runs."+ Just bn -> do+ $(logInfo) "Write ELynx reproduction file."+ liftIO $ writeReproduction bn args'+ -- Footer.+ ftr <- liftIO logFooter+ $(logInfo) $ pack ftr -runELynxLoggingT :: (MonadBaseControl IO m, MonadIO m)- => LogLevel -> Maybe FilePath -> LoggingT m a -> m a-runELynxLoggingT lvl f = case f of- Nothing -> runELynxStderrLoggingT . filterLogger (\_ l -> l >= lvl)- Just fn -> runELynxFileLoggingT fn . filterLogger (\_ l -> l >= lvl)+runELynxLoggingT+ :: (MonadBaseControl IO m, MonadIO m)+ => LogLevel+ -> Force+ -> Maybe FilePath+ -> LoggingT m a+ -> m a+runELynxLoggingT lvl _ Nothing =+ runELynxStderrLoggingT . filterLogger (\_ l -> l >= lvl)+runELynxLoggingT lvl frc (Just fn) =+ runELynxFileLoggingT frc fn . filterLogger (\_ l -> l >= lvl) -runELynxFileLoggingT :: MonadBaseControl IO m => FilePath -> LoggingT m a -> m a-runELynxFileLoggingT fp logger = bracket- (liftBase $ openFile fp AppendMode)- (liftBase . hClose)- $ \h -> liftBase (hSetBuffering h LineBuffering) >> runLoggingT logger (output2H stderr h)+runELynxFileLoggingT+ :: MonadBaseControl IO m => Force -> FilePath -> LoggingT m a -> m a+runELynxFileLoggingT frc fp logger = do+ h <- liftBase $ openFile' frc fp WriteMode+ liftBase (hSetBuffering h LineBuffering)+ r <- runLoggingT logger (output2H stderr h)+ liftBase (hClose h)+ return r runELynxStderrLoggingT :: MonadIO m => LoggingT m a -> m a runELynxStderrLoggingT = (`runLoggingT` output stderr) -output :: Handle- -> Loc- -> LogSource- -> LogLevel- -> LogStr- -> IO ()-output h loc src lvl msg =- B.hPutStr h ls- where- ls = logStrToBS loc src lvl msg--output2H :: Handle- -> Handle- -> Loc- -> LogSource- -> LogLevel- -> LogStr- -> IO ()-output2H h1 h2 loc src lvl msg = do- B.hPutStr h1 ls- B.hPutStr h2 ls- where- ls = logStrToBS loc src lvl msg+output :: Handle -> Loc -> LogSource -> LogLevel -> LogStr -> IO ()+output h _ _ _ msg = B.hPutStrLn h ls where ls = fromLogStr msg -logStrToBS :: Loc- -> LogSource- -> LogLevel- -> LogStr- -> B.ByteString-logStrToBS _ _ _ msg = fromLogStr msg <> "\n"+output2H :: Handle -> Handle -> Loc -> LogSource -> LogLevel -> LogStr -> IO ()+output2H h1 h2 _ _ _ msg = do+ B.hPutStrLn h1 ls+ B.hPutStrLn h2 ls+ where ls = fromLogStr msg
src/ELynx/Tools/Matrix.hs view
@@ -1,7 +1,7 @@ {- | Module : ELynx.Tools.Matrix-Copyright : (c) Dominik Schrempf 2019-License : GPL-3+Copyright : (c) Dominik Schrempf 2020+License : GPL-3.0-or-later Maintainer : dominik.schrempf@gmail.com Stability : unstable@@ -14,7 +14,7 @@ -} module ELynx.Tools.Matrix- (+ ( -- * Matrices fMapRowSeq , fMapRowPar , fMapRowParChunk@@ -24,51 +24,81 @@ , (|||) , (===) , subSampleMatrix- ) where+ )+where import Control.Parallel.Strategies import Data.List-import qualified Data.Matrix.Generic as M-import qualified Data.Vector.Generic as V+import qualified Data.Matrix.Generic as M+import qualified Data.Vector.Generic as V -- | Map a function on each row of a DIM2 array; sequential version.-fMapRowSeq :: (V.Vector v a, V.Vector v b) => (v a -> v b) -> M.Matrix v a -> M.Matrix v b+fMapRowSeq+ :: (V.Vector v a, V.Vector v b)+ => (v a -> v b)+ -> M.Matrix v a+ -> M.Matrix v b fMapRowSeq f m = M.fromRows $ map f (M.toRows m) -- | Map a function on each row of a DIM2 array; parallel version.-fMapRowPar :: (V.Vector v a, V.Vector v b) => (v a -> v b) -> M.Matrix v a -> M.Matrix v b+fMapRowPar+ :: (V.Vector v a, V.Vector v b)+ => (v a -> v b)+ -> M.Matrix v a+ -> M.Matrix v b fMapRowPar f m = M.fromRows $ parMap rseq f (M.toRows m) -- | Map a function on each row of a DIM2 array; parallel version with given chunk size.-fMapRowParChunk :: (V.Vector v a, V.Vector v b) => Int -> (v a -> v b) -> M.Matrix v a -> M.Matrix v b-fMapRowParChunk n f m = M.fromRows (map f (M.toRows m) `using` parListChunk n rseq)+fMapRowParChunk+ :: (V.Vector v a, V.Vector v b)+ => Int+ -> (v a -> v b)+ -> M.Matrix v a+ -> M.Matrix v b+fMapRowParChunk n f m =+ M.fromRows (map f (M.toRows m) `using` parListChunk n rseq) -- | Map a function on each row of a DIM2 array; sequential version.-fMapColSeq :: (V.Vector v a, V.Vector v b) => (v a -> v b) -> M.Matrix v a -> M.Matrix v b+fMapColSeq+ :: (V.Vector v a, V.Vector v b)+ => (v a -> v b)+ -> M.Matrix v a+ -> M.Matrix v b fMapColSeq f m = M.fromColumns $ map f (M.toColumns m) -- | Map a function on each row of a DIM2 array; parallel version.-fMapColPar :: (V.Vector v a, V.Vector v b) => (v a -> v b) -> M.Matrix v a -> M.Matrix v b+fMapColPar+ :: (V.Vector v a, V.Vector v b)+ => (v a -> v b)+ -> M.Matrix v a+ -> M.Matrix v b fMapColPar f m = M.fromColumns $ parMap rseq f (M.toColumns m) -- | Map a function on each row of a DIM2 array; parallel version with given chunk size.-fMapColParChunk :: (V.Vector v a, V.Vector v b) => Int -> (v a -> v b) -> M.Matrix v a -> M.Matrix v b-fMapColParChunk n f m = M.fromColumns (map f (M.toColumns m) `using` parListChunk n rseq)+fMapColParChunk+ :: (V.Vector v a, V.Vector v b)+ => Int+ -> (v a -> v b)+ -> M.Matrix v a+ -> M.Matrix v b+fMapColParChunk n f m =+ M.fromColumns (map f (M.toColumns m) `using` parListChunk n rseq) -- | Horizontal concatenation. (|||) :: (V.Vector v a) => M.Matrix v a -> M.Matrix v a -> M.Matrix v a (|||) l r = M.fromColumns $ lCs ++ rCs- where- lCs = M.toColumns l- rCs = M.toColumns r+ where+ lCs = M.toColumns l+ rCs = M.toColumns r -- | Vertical concatenation. (===) :: (V.Vector v a) => M.Matrix v a -> M.Matrix v a -> M.Matrix v a (===) l r = M.fromRows $ lRs ++ rRs- where- lRs = M.toRows l- rRs = M.toRows r+ where+ lRs = M.toRows l+ rRs = M.toRows r -- | Sample the given sites from a matrix. subSampleMatrix :: (V.Vector v a) => [Int] -> M.Matrix v a -> M.Matrix v a-subSampleMatrix is m = M.fromColumns $ foldl' (\a i -> M.takeColumn m i : a) [] is+subSampleMatrix is m =+ M.fromColumns $ foldl' (\a i -> M.takeColumn m i : a) [] is
src/ELynx/Tools/Misc.hs view
@@ -1,7 +1,7 @@ {- | Module : ELynx.Tools.Misc-Copyright : (c) Dominik Schrempf 2019-License : GPL-3+Copyright : (c) Dominik Schrempf 2020+License : GPL-3.0-or-later Maintainer : dominik.schrempf@gmail.com Stability : unstable@@ -21,7 +21,8 @@ , compose , allValues , horizontalConcat- ) where+ )+where import Data.List @@ -31,7 +32,7 @@ -- | Get all values of a bounded enumerated type. allValues :: (Bounded a, Enum a) => [a]-allValues = [minBound..]+allValues = [minBound ..] -- | A brain f***. As an example, let @xss@ be a list of alignments (i.e., a -- list of a list of a list of alleles). This function horizontally concatenates
src/ELynx/Tools/Numeric.hs view
@@ -1,7 +1,7 @@ {- | Module : ELynx.Tools.Numeric-Copyright : (c) Dominik Schrempf 2019-License : GPL-3+Copyright : (c) Dominik Schrempf 2020+License : GPL-3.0-or-later Maintainer : dominik.schrempf@gmail.com Stability : unstable@@ -14,17 +14,19 @@ -} module ELynx.Tools.Numeric- ( harmonic+ ( -- * Numeric+ harmonic , xLogX , roundN- ) where+ )+where import ELynx.Tools.Equality -- | Calculate the nth harmonic number. harmonic :: Int -> Double harmonic 1 = 1.0-harmonic n = 1.0 / fromIntegral n + harmonic (n-1)+harmonic n = 1.0 / fromIntegral n + harmonic (n - 1) -- | Calculate x*log(x) but set to 0.0 when x is smaller than 'eps'. xLogX :: Double -> Double@@ -34,4 +36,4 @@ -- | Round double to a given precision. roundN :: Int -> Double -> Double-roundN n v = fromInteger (round $ v * (10^n)) / (10.0^^n)+roundN n v = fromInteger (round $ v * (10 ^ n)) / (10.0 ^^ n)
− src/ELynx/Tools/Options.hs
@@ -1,211 +0,0 @@-{-# LANGUAGE TemplateHaskell #-}--{- |-Module : ELynx.Tools.Options-Description : Global command line options and arguments-Copyright : (c) Dominik Schrempf 2019-License : GPL-3--Maintainer : dominik.schrempf@gmail.com-Stability : unstable-Portability : portable--Creation date: Fri May 3 18:20:11 2019.---}--module ELynx.Tools.Options- (- -- * Log file- logHeader- , logFooter- -- * Options- , parseArgumentsWith- , Verbosity (..)- , Arguments (..)- , GlobalArguments (..)- , globalArguments- , seedOpt- -- * Options meta- , megaReadM- -- * Formatting- , fillParagraph- ) where--import Control.Monad.Logger (LogLevel (..))-import Data.List hiding (group)-import Data.Time-import Data.Version (showVersion)-import Data.Void-import Data.Word-import Language.Haskell.TH-import Options.Applicative hiding (empty)-import Options.Applicative.Help.Pretty-import System.Environment-import Text.Megaparsec (Parsec, errorBundlePretty,- runParser)--import ELynx.Tools.Misc-import Paths_elynx_tools (version)---- Be careful; it is necessary to synchronize the evomod-xxx libraries, so that--- the version number of elynx-tools matches the others.-versionString :: String-versionString = "ELynx Suite version " ++ showVersion version ++ "."--copyrightString :: String-copyrightString = "Developed by Dominik Schrempf."--compilationString :: String-compilationString = "Compiled on "- ++ $(stringE =<< runIO- ( formatTime defaultTimeLocale "%B %-e, %Y, at %H:%M %P, %Z."- `fmap` Data.Time.getCurrentTime ))---- A short header to be used in executables. 'unlines' doesn't work here because--- it adds an additional newline at the end :(.-hdr :: String-hdr = intercalate "\n" [ versionString- , copyrightString- , compilationString- ]--time :: IO String-time = formatTime defaultTimeLocale "%B %-e, %Y, at %H:%M %P, %Z." `fmap` Data.Time.getCurrentTime---- | Short, globally usable string preceding all logs with obligatory description.-logHeader :: String -> IO String-logHeader desc = do- t <- time- p <- getProgName- as <- getArgs- -- let l = length desc- return $ intercalate "\n"- -- [ replicate (l+4) '-'- [ "=== " <> desc- , hdr- , "Start time: " ++ t- , "Command line: " ++ p ++ " " ++ unwords as ]---- | See 'logHeader' but footer.-logFooter :: IO String-logFooter = do- t <- time- let timeStr = "=== End time: " ++ t- -- l = length timeStr- return $ intercalate "\n"- [ timeStr ]- -- , replicate l '-' ]--versionOpt :: Parser (a -> a)-versionOpt = infoOption hdr- ( long "version"- -- Lower case 'v' clashes with verbosity.- <> short 'V'- <> help "Show version"- <> hidden )--evoModSuiteFooter :: [Doc]-evoModSuiteFooter =- [ empty- -- , bold $ text "The ELynx Suite."- , text "The ELynx Suite"- , text "---------------"- , fillParagraph "A Haskell library and a tool set for computational biology. The goal of the ELynx Suite is reproducible research. Evolutionary sequences and phylogenetic trees can be read, viewed, modified and simulated. Exact specification of all options is necessary, and nothing is assumed about the data (e.g., the type of code). The command line with all arguments is consistently, and automatically logged."- -- , fill 9 (bold $ text "slynx") <+> text "Analyze, modify, and simulate evolutionary sequences."- -- , fill 9 (bold $ text "tlynx") <+> text "Analyze, modify, and simulate phylogenetic trees." ]- , empty- , fill 9 (text "slynx") <+> text "Analyze, modify, and simulate evolutionary sequences."- , fill 9 (text "tlynx") <+> text "Analyze, modify, and simulate phylogenetic trees."- , empty- , text "Get help for specific commands:"- , text " slynx examine --help"]---- | Parse arguments. Provide a global description, header, footer, and so on.--- Custom additional description (first argument) and footer (second argument)--- can be provided. print help if needed.-parseArgumentsWith :: [String] -> [String] -> Parser a -> IO (Arguments a)-parseArgumentsWith desc ftr p = execParser $- info (helper <*> versionOpt <*> p')- (fullDesc- <> header hdr- <> progDesc dsc'- <> footerDoc (Just ftr'))- where- p' = Arguments <$> globalArguments <*> p- dsc' = unlines desc- ftr' = vsep $ map pretty ftr ++ evoModSuiteFooter---- | Verbosity levels.-data Verbosity = Quiet | Warning | Info | Debug- deriving (Show, Read, Eq, Enum, Bounded, Ord)--toLogLevel :: Verbosity -> LogLevel-toLogLevel Quiet = LevelError-toLogLevel Warning = LevelWarn-toLogLevel Info = LevelInfo-toLogLevel Debug = LevelDebug---- | Argument skeleton to be used with all commands.-data Arguments a = Arguments { global :: GlobalArguments- , local :: a- }---- | A set of global arguments used by all programs. The idea is to provide a--- common framework for shared arguments.----data GlobalArguments = GlobalArguments- { verbosity :: LogLevel- , outFileBaseName :: Maybe FilePath }---- | See 'GlobalArguments', parser function.------ TODO: Provide global --redo, -r option. Only overwrite files if this option--- is specified.-globalArguments :: Parser GlobalArguments-globalArguments = GlobalArguments- <$> (toLogLevel <$> verbosityOpt)- <*> optional outFileBaseNameOpt---- | Boolean option; be verbose; default NO.-verbosityOpt :: Parser Verbosity-verbosityOpt = option auto- ( long "verbosity"- <> short 'v'- <> metavar "VALUE"- <> value Info- <> showDefault- <> help ("Be verbose; one of: " ++ unwords (map show vs) ))- where- vs = allValues :: [Verbosity]---- | Output filename.-outFileBaseNameOpt :: Parser FilePath-outFileBaseNameOpt = strOption- ( long "output-file-basename"- <> short 'o'- <> metavar "NAME"- <> help "Specify base name of output file")---- | Seed option for MWC. Defaults to RANDOM.-seedOpt :: Parser (Maybe [Word32])-seedOpt = optional $ option auto- ( long "seed"- <> short 'S'- <> metavar "[INT]"- <> help ("Seed for random number generator; "- ++ "list of 32 bit integers with up to 256 elements (default: random)" ) )---- | See 'eitherReader', but for Megaparsec.-megaReadM :: Parsec Void String a -> ReadM a-megaReadM p = eitherReader $ \input ->- let eea = runParser p "" input- in- case eea of- Left eb -> Left $ errorBundlePretty eb- Right a -> Right a---- | Fill a string so that it becomes a paragraph with line breaks. Useful for--- descriptions, headers and footers.-fillParagraph :: String -> Doc-fillParagraph = fillSep . map text . words
+ src/ELynx/Tools/Parallel.hs view
@@ -0,0 +1,24 @@+{- |+Module : ELynx.Tools.Parallel+Description : Utility functions for parallel calculations+Copyright : (c) Dominik Schrempf 2020+License : GPL-3.0-or-later++Maintainer : dominik.schrempf@gmail.com+Stability : unstable+Portability : portable++Creation date: Tue Apr 21 15:19:44 2020.++-}++module ELynx.Tools.Parallel+ ( parMapChunk+ )+where++import Control.Parallel.Strategies++-- | Parallel map with given chunk size.+parMapChunk :: Int -> (a -> b) -> [a] -> [b]+parMapChunk n f as = map f as `using` parListChunk n rseq
src/ELynx/Tools/Random.hs view
@@ -3,8 +3,8 @@ {- | Module : ELynx.Tools.Random Description : Tools for random sampling-Copyright : (c) Dominik Schrempf 2019-License : GPL-3+Copyright : (c) Dominik Schrempf 2020+License : GPL-3.0-or-later Maintainer : dominik.schrempf@gmail.com Stability : unstable@@ -15,23 +15,26 @@ -} module ELynx.Tools.Random- ( sample- ) where+ ( -- * Random+ sample+ )+where import Control.Monad.Primitive-import Data.Foldable (toList)-import qualified Data.Sequence as Seq+import Data.Foldable ( toList )+import qualified Data.Sequence as Seq import System.Random.MWC -- | From a given sequence, randomly sample without replacement a number of elements. sample :: PrimMonad m => Seq.Seq a -> Int -> Gen (PrimState m) -> m [a] sample ys size = go 0 (l - 1) ys where- l = Seq.length ys- go !n !i xs g | n >= size = return $! (toList . Seq.drop (l - size)) xs- | otherwise = do- j <- uniformR (0, i) g- let toI = xs `Seq.index` j- toJ = xs `Seq.index` i- next = (Seq.update i toI . Seq.update j toJ) xs- go (n + 1) (i - 1) next g+ l = Seq.length ys+ go !n !i xs g+ | n >= size = return $! (toList . Seq.drop (l - size)) xs+ | otherwise = do+ j <- uniformR (0, i) g+ let toI = xs `Seq.index` j+ toJ = xs `Seq.index` i+ next = (Seq.update i toI . Seq.update j toJ) xs+ go (n + 1) (i - 1) next g {-# INLINE sample #-}
src/ELynx/Tools/Reproduction.hs view
@@ -1,11 +1,15 @@+{-# LANGUAGE TemplateHaskell #-} {-# LANGUAGE DeriveGeneric #-} {-# LANGUAGE ScopedTypeVariables #-}+{-# LANGUAGE FlexibleContexts #-}+{-# LANGUAGE AllowAmbiguousTypes #-}+{-# LANGUAGE TypeApplications #-} {- | Module : ELynx.Tools.Reproduction Description : Functions to ease reproduction of analyses-Copyright : (c) Dominik Schrempf 2019-License : GPL-3+Copyright : (c) Dominik Schrempf 2020+License : GPL-3.0-or-later Maintainer : dominik.schrempf@gmail.com Stability : unstable@@ -13,113 +17,414 @@ Creation date: Tue Nov 19 15:07:09 2019. -XXX: This module is actively developed. It is not yet used by slynx nor tlynx.+Use of standard input is not supported. -} module ELynx.Tools.Reproduction- ( Reproducible (..)- , Reproduction (..)- , readR- , writeR- ) where+ ( -- * Log file+ logHeader+ , logFooter+ -- * Options+ , Verbosity(..)+ , toLogLevel+ , Force(..)+ , forceOpt+ , GlobalArguments(..)+ , globalArguments+ , Seed(..)+ , seedOpt+ , Arguments(..)+ , parseArguments+ -- * Reproduction+ , ELynx+ , Reproducible(..)+ , getReproductionHash+ , Reproduction(..)+ , writeReproduction+ , hashFile+ -- * Misc+ , createCommandReproducible+ , createCommand+ , elynxParserInfo+ , megaReadM+ -- * Re-exports+ , Generic+ , FromJSON+ , ToJSON+ )+where -import Control.Monad (zipWithM)-import Crypto.Hash.SHA256 (hash)-import Data.Aeson (FromJSON, ToJSON,- eitherDecodeFileStrict', encodeFile)-import Data.Bifunctor (first)-import qualified Data.ByteString.Char8 as B-import Data.Either (either)-import GHC.Generics (Generic)-import Options.Applicative (Parser, briefDesc, defaultPrefs,- execParserPure, getParseResult, info)-import System.Environment (getArgs, getProgName)+import Data.Aeson ( FromJSON+ , ToJSON+ , encodeFile+ )+import Control.Monad ( void )+import Crypto.Hash.SHA256 ( hash )+import Data.ByteString.Base16 ( encode )+import qualified Data.ByteString.Char8 as B+import Data.Word+import Data.Vector.Unboxed ( Vector )+import GHC.Generics ( Generic )+import System.Environment ( getArgs+ , getProgName+ )+import Control.Monad.Logger ( LoggingT+ , LogLevel(..)+ )+import Control.Monad.Trans.Reader ( ReaderT )+import Data.List hiding ( group )+import Data.Time+import Data.Version ( Version+ , showVersion+ )+import Data.Void+import Language.Haskell.TH+import Options.Applicative hiding ( empty )+import Options.Applicative.Help.Pretty+import Text.Megaparsec ( Parsec+ , errorBundlePretty+ , runParser+ ) --- | Reproducible commands have a set of input files that have to be checked for--- consistency.+import ELynx.Tools.Definitions ( version )+import ELynx.Tools.Misc ( allValues )++-- Be careful; it is necessary to synchronize the version numbers across packages.+versionString :: String+versionString = "ELynx Suite version " ++ showVersion version ++ "."++copyrightString :: String+copyrightString = "Developed by Dominik Schrempf."++compilationString :: String+compilationString = "Compiled on "+ ++ $(stringE =<< runIO+ ( formatTime defaultTimeLocale "%B %-e, %Y, at %H:%M %P, %Z."+ `fmap` Data.Time.getCurrentTime ))++-- A short header to be used in executables. 'unlines' doesn't work here because+-- it adds an additional newline at the end.+hdr :: [String]+hdr = [versionString, copyrightString, compilationString]++time :: IO String+time =+ formatTime defaultTimeLocale "%B %-e, %Y, at %H:%M %P, %Z."+ `fmap` Data.Time.getCurrentTime++-- | Short, globally usable string preceding all logs with obligatory description.+logHeader :: String -> [String] -> IO String+logHeader h dsc = do+ t <- time+ p <- getProgName+ as <- getArgs+ return+ $ intercalate "\n"+ $ ("=== " <> h)+ : dsc+ ++ hdr+ ++ ["Start time: " ++ t, "Command line: " ++ p ++ " " ++ unwords as]++-- | See 'logHeader' but footer.+logFooter :: IO String+logFooter = do+ t <- time+ let timeStr = "=== End time: " ++ t+ return $ intercalate "\n" [timeStr]++versionOpt :: Parser (a -> a)+versionOpt = infoOption+ (intercalate "\n" hdr)+ ( long "version"+ -- Lower case 'v' clashes with verbosity.+ <> short 'V'+ <> help "Show version"+ <> hidden+ )++evoModSuiteFooter :: [Doc]+evoModSuiteFooter =+ [ empty+ , text "ELynx"+ , text "-----"+ , fillParagraph+ "A Haskell library and tool set for computational biology. The goal of ELynx is reproducible research. Evolutionary sequences and phylogenetic trees can be read, viewed, modified and simulated. The command line with all arguments is logged consistently, and automatically. Data integrity is verified using SHA256 sums so that validation of past analyses is possible without the need to recompute the result."+ , empty+ , fill 9 (text "slynx")+ <+> text "Analyze, modify, and simulate evolutionary sequences."+ , fill 9 (text "tlynx")+ <+> text "Analyze, modify, and simulate phylogenetic trees."+ , fill 9 (text "elynx") <+> text "Validate and redo past analyses."+ , empty+ , text "Get help for sub commands:"+ , text " slynx examine --help"+ ]++-- | Verbosity levels.+data Verbosity = Quiet | Warning | Info | Debug+ deriving (Show, Read, Eq, Enum, Bounded, Ord, Generic)++instance FromJSON Verbosity++instance ToJSON Verbosity++-- | Conert verbosity option to log level.+toLogLevel :: Verbosity -> LogLevel+toLogLevel Quiet = LevelError+toLogLevel Warning = LevelWarn+toLogLevel Info = LevelInfo+toLogLevel Debug = LevelDebug++-- | Exit when output exists, or overwrite.+newtype Force = Force Bool+ deriving (Eq, Show, Generic)++instance FromJSON Force++instance ToJSON Force++-- | A set of global arguments used by all programs. The idea is to provide a+-- common framework for shared arguments.+--+data GlobalArguments = GlobalArguments+ { verbosity :: Verbosity+ , outFileBaseName :: Maybe FilePath+ , forceReanalysis :: Force }+ deriving (Eq, Show, Generic)++instance FromJSON GlobalArguments++instance ToJSON GlobalArguments++-- | See 'GlobalArguments', parser function.+globalArguments :: Parser GlobalArguments+globalArguments =+ GlobalArguments <$> verbosityOpt <*> optional outFileBaseNameOpt <*> forceOpt++-- | Boolean option; be verbose; default NO.+verbosityOpt :: Parser Verbosity+verbosityOpt = option+ auto+ ( long "verbosity"+ <> short 'v'+ <> metavar "VALUE"+ <> value Info+ <> showDefault+ <> help ("Be verbose; one of: " ++ unwords (map show vs))+ )+ where vs = allValues :: [Verbosity]++-- | Output filename.+outFileBaseNameOpt :: Parser FilePath+outFileBaseNameOpt = strOption+ (long "output-file-basename" <> short 'o' <> metavar "NAME" <> help+ "Specify base name of output file"+ )++-- | Force option parser.+forceOpt :: Parser Force+forceOpt = flag+ (Force False)+ (Force True)+ -- DO NOT CHANGE --force nor -f; they are used by 'elynx redo'.+ (long "force" <> short 'f' <> help+ "Ignore previous analysis and overwrite existing output files."+ )++-- | Random or fixed seed.+data Seed = Random | Fixed (Vector Word32)+ deriving (Show, Generic)++-- | Upon equality check, a random seed is not different from a fixed one.+instance Eq Seed where+ Random == _ = True+ _ == Random = True+ Fixed s == Fixed t = s == t++instance FromJSON Seed++instance ToJSON Seed++-- | Seed option for MWC. Defaults to Random.+seedOpt :: Parser Seed+seedOpt = toSeed <$> seedPar++toSeed :: Maybe (Vector Word32) -> Seed+toSeed Nothing = Random+toSeed (Just w) = Fixed w++seedPar :: Parser (Maybe (Vector Word32))+seedPar = optional $ option+ auto+ (long "seed" <> short 'S' <> metavar "[INT]" <> help+ ( "Seed for random number generator; "+ ++ "list of 32 bit integers with up to 256 elements (default: random)"+ )+ )++-- | Argument skeleton to be used with all commands.+data Arguments a = Arguments { global :: GlobalArguments+ , local :: a+ }+ deriving (Eq, Show, Generic)++instance FromJSON a => FromJSON (Arguments a)++instance ToJSON a => ToJSON (Arguments a)++instance Reproducible a => Reproducible (Arguments a) where+ inFiles = inFiles . local+ outSuffixes = outSuffixes . local+ getSeed = getSeed . local+ setSeed (Arguments g l) s = Arguments g $ setSeed l s+ parser = argumentsParser (parser @a)+ cmdName = cmdName @a+ cmdDsc = cmdDsc @a+ cmdFtr = cmdFtr @a++argumentsParser :: Parser a -> Parser (Arguments a)+argumentsParser p = Arguments <$> globalArguments <*> p++elynxParser :: Parser a -> Parser a+elynxParser p = helper <*> versionOpt <*> p++-- | Parse arguments. Provide a global description, header, footer, and so on.+-- Custom additional description (first argument) and footer (second argument)+-- can be provided. print help if needed.+parseArguments :: forall a . Reproducible a => IO (Arguments a)+parseArguments = execParser+ $ elynxParserInfo (cmdDsc @a) (cmdFtr @a) (argumentsParser $ parser @a)++-- | Logging transformer to be used with all executables.+type ELynx a = ReaderT (Arguments a) (LoggingT IO)++-- | Reproducible commands have+-- - a set of input files to be checked for consistency,+-- - a set of output suffixes which define output files to be checked for consistency,+-- - a function to get the seed, if available,+-- - a function to set the seed, if applicable,+-- - a parser to read the command line,+-- - a nice program name, description, and footer. class Reproducible a where- inFiles :: a -> [FilePath]- parser :: a -> Parser a+ inFiles :: a -> [FilePath]+ outSuffixes :: a -> [String]+ getSeed :: a -> Maybe Seed+ setSeed :: a -> Vector Word32 -> a+ parser :: Parser a+ cmdName :: String+ cmdDsc :: [String]+ cmdFtr :: [String]+ cmdFtr = [] +-- | A unique hash of the reproduction data type.+getReproductionHash :: forall a . Reproducible a => Reproduction a -> String+getReproductionHash r =+ B.unpack+ $ encode+ $ hash+ $ B.pack+ $ unlines+ $+ -- Reproduction.+ progName r+ : argsStr r+ <> [showVersion (rVersion r)]+ <> files r+ <> checkSums r+ -- Reproducible.+ <> inFiles ri+ <> outSuffixes ri+ <> [cmdName @a]+ <> cmdDsc @a+ <> cmdFtr @a+ where ri = reproducible r++setHash :: Reproducible a => Reproduction a -> Reproduction a+setHash r = r { rHash = Just h } where h = getReproductionHash r++ -- | Necessary information for a reproducible run. Notably, the input files are -- checked for consistency! data Reproduction a = Reproduction- { progName :: String -- ^ Program name.- , args :: [String] -- ^ Command line arguments without program name.- , filePaths :: [FilePath] -- ^ File paths of in files.- , checkSums :: [String] -- ^ SHA256 sums of in files.- , cmd :: a -- ^ Command argument.+ { progName :: String -- ^ Program name.+ , argsStr :: [String] -- ^ Command line arguments without program name.+ , rVersion :: Version+ , rHash :: Maybe String -- ^ Unique hash; see 'getReproductionHash'.+ , files :: [FilePath] -- ^ File paths of used files.+ , checkSums :: [String] -- ^ SHA256 sums of used files.+ , reproducible :: a -- ^ Command argument. } deriving (Generic) -instance ToJSON a => ToJSON (Reproduction a) where- instance FromJSON a => FromJSON (Reproduction a) --- Does the command line fit the provided command?-checkArgs :: (Eq a, Show a, Reproducible a)- => [String] -> a -> IO (Either String ())-checkArgs as c = do- let p = parser c- pres = execParserPure defaultPrefs (info p briefDesc) as- return $ case getParseResult pres of- Nothing ->- Left $ unlines [ "Could not parse command line string:"- , concat as ]- Just c' ->- if c' /= c- then Left $ unlines [ "Command line string and command arguments do not fit:"- , concat as- , show c ]- else Right ()---- Does the file match the checksum?-checkFile :: FilePath -> B.ByteString -> IO (Either String ())-checkFile fp h = do- h' <- hashFile fp- return $ if h' == h- then Right ()- else Left $ unlines [ "SHA256 sum does not match for a file."- , fp ++ " has check sum " ++ B.unpack h'- , "Stored sum is " ++ B.unpack h ]---- | Check if command line arguments and files check sums are matching.-checkReproduction :: (Eq a, Show a, Reproducible a)- => Reproduction a -> IO (Either String ())-checkReproduction (Reproduction _ as fs ss c) = do- chA <- checkArgs as c- chFs <- zipWithM checkFile fs (map B.pack ss)- let ch = sequence_ (chA : chFs)- return $ first ("Failed validating the reproduction file.\n" ++) ch---- | Read an ELynx reproduction file. Check consistency of arguments and input files.-readR :: forall a . (Eq a, Show a, Reproducible a, FromJSON a)- => FilePath -> IO (Reproduction a)-readR fp = do- res <- eitherDecodeFileStrict' fp :: IO (Either String (Reproduction a))- case res of- Left err -> do- putStrLn "Failed reading the ELynx reproduction file."- putStrLn "The following error was encountered."- error err- Right r -> do- ch <- checkReproduction r- return $ either error (const r) ch+instance ToJSON a => ToJSON (Reproduction a) -- | Helper function. hashFile :: FilePath -> IO B.ByteString-hashFile f = hash <$> B.readFile f+hashFile f = encode . hash <$> B.readFile f --- | Write an ELynx reproduction file. Check arguments.-writeR :: (Eq a, Show a, Reproducible a, ToJSON a) => FilePath -> a -> IO ()-writeR fp c = do- p <- getProgName+-- | Write an ELynx reproduction file.+writeReproduction+ :: forall a+ . (Eq a, Show a, Reproducible a, ToJSON a)+ => String+ -> a+ -> IO ()+writeReproduction bn r = do+ pn <- getProgName as <- getArgs- let fs = inFiles c+ let outFs = map (bn ++) (outSuffixes r)+ let fs = inFiles r ++ outFs cs <- mapM hashFile fs- let- cs' = map B.unpack cs- r = Reproduction p as fs cs' c- -- XXX: Actually, it is only necessary to to checkArgs here. But let's just be safe.- ch <- checkReproduction r- either error (const $ encodeFile fp r) ch+ let cs' = map B.unpack cs+ s = Reproduction pn as version Nothing fs cs' r+ void $ encodeFile (bn <> ".elynx") (setHash s)++-- | Create a command; convenience function.+createCommandReproducible+ :: forall a b . Reproducible a => (a -> b) -> Mod CommandFields b+createCommandReproducible f = command (cmdName @a) $ f <$> elynxParserInfo+ (cmdDsc @a)+ (cmdFtr @a)+ (parser @a)++-- | Create a command; convenience function.+createCommand+ :: String+ -> [String]+ -> [String]+ -> Parser a+ -> (a -> b)+ -> Mod CommandFields b+createCommand nm dsc ftr p f = command nm $ f <$> parserInfo dsc' ftr' p+ where+ dsc' = if null dsc then Nothing else Just $ vsep $ map pretty dsc+ ftr' = if null ftr then Nothing else Just $ vsep $ map pretty ftr++-- | ELynx parser info; convenience function.+elynxParserInfo :: [String] -> [String] -> Parser a -> ParserInfo a+elynxParserInfo dsc ftr = parserInfo dsc' ftr'+ where+ dsc' = if null dsc then Nothing else Just $ vsep $ map pretty dsc+ ftr' = Just $ vsep $ map pretty ftr ++ evoModSuiteFooter++-- Short version of ELynx parser info for sub commands.+parserInfo :: Maybe Doc -> Maybe Doc -> Parser a -> ParserInfo a+parserInfo dsc ftr p = info+ (elynxParser p)+ (fullDesc <> headerDoc (Just hdr') <> progDescDoc dsc <> footerDoc ftr)+ where hdr' = vsep $ map pretty hdr++-- | See 'eitherReader', but for Megaparsec.+megaReadM :: Parsec Void String a -> ReadM a+megaReadM p = eitherReader $ \input ->+ let eea = runParser p "" input+ in case eea of+ Left eb -> Left $ errorBundlePretty eb+ Right a -> Right a++-- | Fill a string so that it becomes a paragraph with line breaks. Useful for+-- descriptions, headers and footers.+fillParagraph :: String -> Doc+fillParagraph = fillSep . map text . words
src/ELynx/Tools/Text.hs view
@@ -1,7 +1,7 @@ {- | Module : ELynx.Tools.Text-Copyright : (c) Dominik Schrempf 2019-License : GPL-3+Copyright : (c) Dominik Schrempf 2020+License : GPL-3.0-or-later Maintainer : dominik.schrempf@gmail.com Stability : unstable@@ -14,16 +14,18 @@ -} module ELynx.Tools.Text- (- -- * Text handling.+ ( -- * Text handling tShow , fromBs- ) where+ )+where -import Data.ByteString.Lazy (ByteString)-import Data.Text (Text, pack)-import Data.Text.Lazy (toStrict)-import Data.Text.Lazy.Encoding (decodeUtf8)+import Data.ByteString.Lazy ( ByteString )+import Data.Text ( Text+ , pack+ )+import Data.Text.Lazy ( toStrict )+import Data.Text.Lazy.Encoding ( decodeUtf8 ) -- | Conversion to 'Text' from showable values. tShow :: Show a => a -> Text
src/ELynx/Tools/Vector.hs view
@@ -1,7 +1,7 @@ {- | Module : ELynx.Tools.Vector-Copyright : (c) Dominik Schrempf 2019-License : GPL-3+Copyright : (c) Dominik Schrempf 2020+License : GPL-3.0-or-later Maintainer : dominik.schrempf@gmail.com Stability : unstable@@ -14,17 +14,18 @@ -} module ELynx.Tools.Vector- (+ ( -- * Vectors sumVec , normalizeSumVec , uniformVec , meanVec- , chop- , randomInsert- ) where+ , chopVec+ , randomInsertVec+ )+where import Control.Monad.Primitive-import qualified Data.Vector.Generic as V+import qualified Data.Vector.Generic as V import System.Random.MWC -- | Sum of elements.@@ -34,8 +35,9 @@ -- | Normalize a vector such that elements sum to a given value. normalizeSumVec :: (Fractional a, V.Vector v a) => a -> v a -> v a normalizeSumVec c v = V.map (* c') v- where s = sumVec v- c' = c/s+ where+ s = sumVec v+ c' = c / s -- | A uniform vector of given length. uniformVec :: (Fractional a, V.Vector v a) => Int -> v a@@ -47,13 +49,14 @@ -- | Chop list into chunks of given length. If the last chop is shorter than -- length, it is dropped.-chop :: V.Vector v a => Int -> v a -> [v a]-chop n xs | V.length xs < n = []- | otherwise = V.take n xs : chop n (V.drop n xs)+chopVec :: V.Vector v a => Int -> v a -> [v a]+chopVec n xs | V.length xs < n = []+ | otherwise = V.take n xs : chopVec n (V.drop n xs) -- | Insert element into random position of vector.-randomInsert :: (PrimMonad m, V.Vector v a) => a -> v a -> Gen (PrimState m) -> m (v a)-randomInsert e v g = do+randomInsertVec+ :: (PrimMonad m, V.Vector v a) => a -> v a -> Gen (PrimState m) -> m (v a)+randomInsertVec e v g = do let l = V.length v i <- uniformR (0, l) g return $ V.take i v V.++ V.singleton e V.++ V.drop i v