diff --git a/ChangeLog.md b/ChangeLog.md --- a/ChangeLog.md +++ b/ChangeLog.md @@ -5,6 +5,16 @@ ## Unreleased changes +## Version 0.6.0.0 + +- **elynx-tree:** remove parallel folds with layers (`parBranchFoldMapWithLayer` + too special and slow). +- **elynx-tree:** fix various tree instances; add zip trees with appropriate + instances. +- Remove `monad-logger` dependency and implement lighter alternative. +- Significant changes to the tool chain. + + ## Version 0.5.1.0 - **elynx-tree:** new functions `isValidPath`, `isLeaf`, `depth`; add conversion diff --git a/README.md b/README.md --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ # The ELynx Suite -Version: 0.5.1.0. +Version: 0.6.0.0. Reproducible evolution made easy.

@@ -39,7 +39,7 @@ # Installation -ELynx is written in [Haskell](https://www.haskell.org/) and can be installed with [Stack](https://docs.haskellstack.org/en/stable/README/). +ELynx is written in [Haskell](https://www.haskell.org/) and can be installed with [cabal-install](https://cabal.readthedocs.io/en/3.4/cabal-commands.html) or [Stack](https://docs.haskellstack.org/en/stable/README/). 1. Install Stack with your package manager, or directly from the web page. @@ -67,15 +67,15 @@ [PATH](https://en.wikipedia.org/wiki/PATH_(variable)) environment variable. Then, they can be used directly. -# SLynx +# Get help -Handle evolutionary sequences. +For example: - stack exec slynx -- --help | head -n -16 + slynx --help - ELynx Suite version 0.5.1.0. + ELynx Suite version 0.6.0.0. Developed by Dominik Schrempf. - Compiled on June 12, 2021, at 14:54 pm, UTC. + Compiled on September 3, 2021, at 20:56 pm, UTC. Usage: slynx [-v|--verbosity VALUE] [-o|--output-file-basename NAME] [-f|--force] [--no-elynx-file] COMMAND @@ -84,7 +84,7 @@ Available options: -h,--help Show this help text -V,--version Show version - -v,--verbosity VALUE Be verbose; one of: Quiet Warning Info Debug + -v,--verbosity VALUE Be verbose; one of: Quiet Warn Info Debug (default: Info) -o,--output-file-basename NAME Specify base name of output file @@ -112,107 +112,33 @@ - Protein (amino acids) - ProteinX (amino acids; including gaps) - ProteinS (amino acids; including gaps, and translation stops) - - -## Concatenate - -Concatenate multi sequence alignments. - - stack exec slynx -- concatenate --help - - ELynx Suite version 0.5.1.0. - Developed by Dominik Schrempf. - Compiled on June 12, 2021, at 14:54 pm, UTC. - - Usage: slynx concatenate (-a|--alphabet NAME) INPUT-FILE - Concatenate sequences found in input files. - - Available options: - -h,--help Show this help text - -V,--version Show version - -a,--alphabet NAME Specify alphabet type NAME - INPUT-FILE Read sequences from INPUT-FILE - -h,--help Show this help text - - -## Examine - -Examine sequence with `slynx examine`. - - stack exec slynx -- examine --help - - ELynx Suite version 0.5.1.0. - Developed by Dominik Schrempf. - Compiled on June 12, 2021, at 14:54 pm, UTC. - - Usage: slynx examine (-a|--alphabet NAME) INPUT-FILE [--per-site] - Examine sequences. If data is a multi sequence alignment, additionally analyze columns. - - Available options: - -h,--help Show this help text - -V,--version Show version - -a,--alphabet NAME Specify alphabet type NAME - INPUT-FILE Read sequences from INPUT-FILE - --per-site Report per site summary statistics - -h,--help Show this help text - - -## Filter - -Filter sequences with `filer-rows`. - - stack exec slynx -- filter-rows --help - - ELynx Suite version 0.5.1.0. - Developed by Dominik Schrempf. - Compiled on June 12, 2021, at 14:54 pm, UTC. - - Usage: slynx filter-rows (-a|--alphabet NAME) INPUT-FILE [--longer-than LENGTH] - [--shorter-than LENGTH] [--standard-characters] - Filter rows (or sequences) found in input files. - - Available options: - -h,--help Show this help text - -V,--version Show version - -a,--alphabet NAME Specify alphabet type NAME - INPUT-FILE Read sequences from INPUT-FILE - --longer-than LENGTH Only keep sequences longer than LENGTH - --shorter-than LENGTH Only keep sequences shorter than LENGTH - --standard-characters Only keep sequences containing at least one standard - (i.e., non-IUPAC) character - -h,--help Show this help text - -Filter columns of multi sequence alignments with `filter-columns`. - - stack exec slynx -- filter-columns --help - - ELynx Suite version 0.5.1.0. - Developed by Dominik Schrempf. - Compiled on June 12, 2021, at 14:54 pm, UTC. + - ProteinI (amino acids; including gaps, translation stops, and IUPAC codes) + ELynx + ----- + A Haskell library and tool set for computational biology. The goal of ELynx is + reproducible research. Evolutionary sequences and phylogenetic trees can be + read, viewed, modified and simulated. The command line with all arguments is + logged consistently, and automatically. Data integrity is verified using SHA256 + sums so that validation of past analyses is possible without the need to + recompute the result. - Usage: slynx filter-columns (-a|--alphabet NAME) INPUT-FILE - [--standard-chars DOUBLE] - Filter columns of multi sequence alignments. + slynx Analyze, modify, and simulate evolutionary sequences. + tlynx Analyze, modify, and simulate phylogenetic trees. + elynx Validate and redo past analyses. - Available options: - -h,--help Show this help text - -V,--version Show version - -a,--alphabet NAME Specify alphabet type NAME - INPUT-FILE Read sequences from INPUT-FILE - --standard-chars DOUBLE Keep columns with a proportion standard (non-IUPAC) - characters larger than DOUBLE in [0,1] - -h,--help Show this help text + Get help for sub commands: + slynx examine --help -## Simulate +## Sub command -Simulate sequences with `slynx simulate`. +The documentation of sub commands can be accessed separately: - stack exec slynx -- simulate --help + slynx simulate --help - ELynx Suite version 0.5.1.0. + ELynx Suite version 0.6.0.0. Developed by Dominik Schrempf. - Compiled on June 12, 2021, at 14:54 pm, UTC. + Compiled on September 3, 2021, at 20:56 pm, UTC. Usage: slynx simulate (-t|--tree-file Name) [-s|--substitution-model MODEL] [-m|--mixture-model MODEL] [-e|--edm-file NAME] @@ -281,315 +207,4 @@ LG exchangeabilities with site profiles (Phylobayes) given in FILES. -m "EDM(LG-Custom)" -p FILES For special mixture models, mixture weights are optional. - - -## Sub-sample - -Sub-sample columns from multi sequence alignments. - - stack exec slynx -- sub-sample --help - - ELynx Suite version 0.5.1.0. - Developed by Dominik Schrempf. - Compiled on June 12, 2021, at 14:54 pm, UTC. - - Usage: slynx sub-sample (-a|--alphabet NAME) INPUT-FILE - (-n|--number-of-sites INT) - (-m|--number-of-alignments INT) [-S|--seed [INT]] - Sub-sample columns from multi sequence alignments. - - Available options: - -h,--help Show this help text - -V,--version Show version - -a,--alphabet NAME Specify alphabet type NAME - INPUT-FILE Read sequences from INPUT-FILE - -n,--number-of-sites INT Number of sites randomly drawn with replacement - -m,--number-of-alignments INT - Number of multi sequence alignments to be created - -S,--seed [INT] Seed for random number generator; list of 32 bit - integers with up to 256 elements (default: random) - -h,--help Show this help text - - Create a given number of multi sequence alignments, each of which contains a given number of random sites drawn from the original multi sequence alignment. - - -## Translate - -Translate sequences. - - stack exec slynx -- translate --help - - ELynx Suite version 0.5.1.0. - Developed by Dominik Schrempf. - Compiled on June 12, 2021, at 14:54 pm, UTC. - - Usage: slynx translate (-a|--alphabet NAME) INPUT-FILE (-r|--reading-frame INT) - (-u|--universal-code CODE) - Translate from DNA to Protein or DNAX to ProteinX. - - Available options: - -h,--help Show this help text - -V,--version Show version - -a,--alphabet NAME Specify alphabet type NAME - INPUT-FILE Read sequences from INPUT-FILE - -r,--reading-frame INT Reading frame [0|1|2]. - -u,--universal-code CODE universal code; one of: Standard, - VertebrateMitochondrial. - -h,--help Show this help text - - -# TLynx - -Handle phylogenetic trees in Newick format. - - stack exec tlynx -- --help | head -n -16 - - ELynx Suite version 0.5.1.0. - Developed by Dominik Schrempf. - Compiled on June 12, 2021, at 14:54 pm, UTC. - - Usage: tlynx [-v|--verbosity VALUE] [-o|--output-file-basename NAME] - [-f|--force] [--no-elynx-file] COMMAND - Compare, examine, and simulate phylogenetic trees. - - Available options: - -h,--help Show this help text - -V,--version Show version - -v,--verbosity VALUE Be verbose; one of: Quiet Warning Info Debug - (default: Info) - -o,--output-file-basename NAME - Specify base name of output file - -f,--force Ignore previous analysis and overwrite existing - output files. - --no-elynx-file Do not write data required to reproduce an analysis. - - Available commands: - compare Compare two phylogenetic trees (compute distances and branch-wise differences). - connect Connect two phylogenetic trees in all ways (possibly honoring constraints). - distance Compute distances between many phylogenetic trees. - examine Compute summary statistics of phylogenetic trees. - shuffle Shuffle a phylogenetic tree (keep coalescent times, but shuffle topology and leaves). - simulate Simulate phylogenetic trees using a birth and death or coalescent process. - - - Available tree file formats: - - Newick Standard: Branch support values are stored in square brackets after branch lengths. - - Newick IqTree: Branch support values are stored as node names after the closing bracket of forests. - - Newick RevBayes: Key-value pairs is provided in square brackets after node names as well as branch lengths. XXX: Key value pairs are ignored at the moment. - - -## Compare - -Compute distances between phylogenetic trees. - - stack exec tlynx -- compare --help - - ELynx Suite version 0.5.1.0. - Developed by Dominik Schrempf. - Compiled on June 12, 2021, at 14:54 pm, UTC. - - Usage: tlynx compare [-n|--normalize] [-b|--bipartitions] [-t|--intersect] - [-f|--newick-format FORMAT] NAMES - Compare two phylogenetic trees (compute distances and branch-wise differences). - - Available options: - -h,--help Show this help text - -V,--version Show version - -n,--normalize Normalize trees before comparison - -b,--bipartitions Print and plot common and missing bipartitions - -t,--intersect Compare intersections; i.e., before comparison, drop - leaves that are not present in the other tree - -f,--newick-format FORMAT - Newick tree format: Standard, IqTree, or RevBayes; - default: Standard; for detailed help, see 'tlynx - --help' - NAMES Tree files - -h,--help Show this help text - - -## Connect - -Connect two phylogenetic tree in all ways (possibly honoring constraints). - - stack exec tlynx -- connect --help - - ELynx Suite version 0.5.1.0. - Developed by Dominik Schrempf. - Compiled on June 12, 2021, at 14:54 pm, UTC. - - Usage: tlynx connect [-f|--newick-format FORMAT] [-c|--contraints CONSTRAINTS] - TREE-FILE-A TREE-FILE-B - Connect two phylogenetic trees in all ways (possibly honoring constraints). - - Available options: - -h,--help Show this help text - -V,--version Show version - -f,--newick-format FORMAT - Newick tree format: Standard, IqTree, or RevBayes; - default: Standard; for detailed help, see 'tlynx - --help' - -c,--contraints CONSTRAINTS - File containing one or more Newick trees to be used - as constraints - TREE-FILE-A File containing the first Newick tree - TREE-FILE-B File containing the second Newick tree - -h,--help Show this help text - - -## Distancce - -Compute distances between many phylogenetic trees. - - stack exec tlynx -- distance --help - - ELynx Suite version 0.5.1.0. - Developed by Dominik Schrempf. - Compiled on June 12, 2021, at 14:54 pm, UTC. - - Usage: tlynx distance (-d|--distance MEASURE) [-n|--normalize] [-t|--intersect] - [-s|--summary-statistics] - [-m|--master-tree-file MASTER-TREE-File] - [-f|--newick-format FORMAT] [INPUT-FILES] - Compute distances between many phylogenetic trees. - - Available options: - -h,--help Show this help text - -V,--version Show version - -d,--distance MEASURE Type of distance to calculate (available distance - measures are listed below) - -n,--normalize Normalize trees before distance calculation; only - affect distances depending on branch lengths - -t,--intersect Compare intersections; i.e., before comparison, drop - leaves that are not present in the other tree - -s,--summary-statistics Report summary statistics only - -m,--master-tree-file MASTER-TREE-File - Compare all trees to the tree in the master tree - file. - -f,--newick-format FORMAT - Newick tree format: Standard, IqTree, or RevBayes; - default: Standard; for detailed help, see 'tlynx - --help' - INPUT-FILES Read tree(s) from INPUT-FILES; if more files are - given, one tree is expected per file - -h,--help Show this help text - - Distance measures: - symmetric Symmetric distance (Robinson-Foulds distance). - incompatible-split[VAL] Incompatible split distance. Collapse branches with (normalized) - support less than 0.0<=VAL<=1.0 before distance calculation; - if, let's say, VAL>0.7, only well supported differences contribute - to the total distance. - branch-score Branch score distance. - - -## Examine - -Compute summary statistics of phylogenetic trees. - - stack exec tlynx -- examine --help - - ELynx Suite version 0.5.1.0. - Developed by Dominik Schrempf. - Compiled on June 12, 2021, at 14:54 pm, UTC. - - Usage: tlynx examine INPUT-FILE [-f|--newick-format FORMAT] - Compute summary statistics of phylogenetic trees. - - Available options: - -h,--help Show this help text - -V,--version Show version - INPUT-FILE Read trees from INPUT-FILE - -f,--newick-format FORMAT - Newick tree format: Standard, IqTree, or RevBayes; - default: Standard; for detailed help, see 'tlynx - --help' - -h,--help Show this help text - - -## Shuffle - -Shuffle a phylogenetic tree (keep coalescent times, but shuffle topology and -leaves). - - stack exec tlynx -- shuffle --help - - ELynx Suite version 0.5.1.0. - Developed by Dominik Schrempf. - Compiled on June 12, 2021, at 14:54 pm, UTC. - - Usage: tlynx shuffle [-f|--newick-format FORMAT] [-n|--replicates N] TREE-FILE - [-S|--seed [INT]] - Shuffle a phylogenetic tree (keep coalescent times, but shuffle topology and leaves). - - Available options: - -h,--help Show this help text - -V,--version Show version - -f,--newick-format FORMAT - Newick tree format: Standard, IqTree, or RevBayes; - default: Standard; for detailed help, see 'tlynx - --help' - -n,--replicates N Number of trees to generate - TREE-FILE File containing a Newick tree - -S,--seed [INT] Seed for random number generator; list of 32 bit - integers with up to 256 elements (default: random) - -h,--help Show this help text - - -## Simulate - -Simulate phylogenetic trees using birth and death processes. - - stack exec tlynx -- simulate --help - - ELynx Suite version 0.5.1.0. - Developed by Dominik Schrempf. - Compiled on June 12, 2021, at 14:54 pm, UTC. - - Usage: tlynx simulate (-t|--nTrees INT) (-n|--nLeaves INT) PROCESS - [-u|--sub-sample DOUBLE] [-s|--summary-statistics] - [-S|--seed [INT]] - Simulate phylogenetic trees using a birth and death or coalescent process. - - Available options: - -h,--help Show this help text - -V,--version Show version - -t,--nTrees INT Number of trees - -n,--nLeaves INT Number of leaves per tree - -u,--sub-sample DOUBLE Perform sub-sampling; see below. - -s,--summary-statistics For each branch, print length and number of children - -S,--seed [INT] Seed for random number generator; list of 32 bit - integers with up to 256 elements (default: random) - -h,--help Show this help text - - Available processes: - birthdeath Birth and death process - coalescent Coalescent process - - See, for example, 'tlynx simulate birthdeath --help'. - Sub-sample with probability p: - 1. Simulate one big tree with n'=round(n/p), n'>=n, leaves; - 2. Randomly sample sub-trees with n leaves. - (With p=1.0, the same tree is reported over and over again.) - - -# ELynx - -Validate and (optionally) redo past ELynx analyses. - - stack exec elynx -- --help | head -n -16 - - ELynx Suite version 0.5.1.0. - Developed by Dominik Schrempf. - Compiled on June 12, 2021, at 14:54 pm, UTC. - - Usage: elynx COMMAND - Validate and redo past ELynx analyses - - Available options: - -h,--help Show this help text - -V,--version Show version - - Available commands: - validate Validate an ELynx analysis - redo Redo an ELynx analysis diff --git a/elynx-seq.cabal b/elynx-seq.cabal --- a/elynx-seq.cabal +++ b/elynx-seq.cabal @@ -1,19 +1,18 @@ -cabal-version: 2.2 -name: elynx-seq -version: 0.5.1.1 -license: GPL-3.0-or-later -license-file: LICENSE -copyright: Dominik Schrempf (2021) -maintainer: dominik.schrempf@gmail.com -author: Dominik Schrempf -homepage: https://github.com/dschrempf/elynx#readme -bug-reports: https://github.com/dschrempf/elynx/issues -synopsis: Handle molecular sequences -description: - Examine, modify, and simulate molecular sequences in a reproducible way. Please see the README on GitHub at . +cabal-version: 2.2 +name: elynx-seq +version: 0.6.0.0 +synopsis: Handle molecular sequences +description: Examine, modify, and simulate molecular sequences in a reproducible way. Please see the README on GitHub at . +category: Bioinformatics +homepage: https://github.com/dschrempf/elynx#readme +bug-reports: https://github.com/dschrempf/elynx/issues +author: Dominik Schrempf +maintainer: dominik.schrempf@gmail.com +copyright: Dominik Schrempf (2021) +license: GPL-3.0-or-later +license-file: LICENSE +build-type: Simple -category: Bioinformatics -build-type: Simple extra-source-files: README.md ChangeLog.md @@ -27,70 +26,72 @@ data/TranslateMitochondrialVertebrateProtein.fasta source-repository head - type: git - location: https://github.com/dschrempf/elynx + type: git + location: https://github.com/dschrempf/elynx library - exposed-modules: - ELynx.Data.Alphabet.Alphabet - ELynx.Data.Alphabet.Character - ELynx.Data.Alphabet.DistributionDiversity - ELynx.Data.Character.AminoAcid - ELynx.Data.Character.AminoAcidI - ELynx.Data.Character.AminoAcidS - ELynx.Data.Character.AminoAcidX - ELynx.Data.Character.Character - ELynx.Data.Character.Codon - ELynx.Data.Character.Nucleotide - ELynx.Data.Character.NucleotideI - ELynx.Data.Character.NucleotideX - ELynx.Data.Sequence.Alignment - ELynx.Data.Sequence.Defaults - ELynx.Data.Sequence.Distance - ELynx.Data.Sequence.Sequence - ELynx.Data.Sequence.Translate - ELynx.Export.Sequence.Fasta - ELynx.Import.Sequence.Fasta - - hs-source-dirs: src - other-modules: Paths_elynx_seq - autogen-modules: Paths_elynx_seq - default-language: Haskell2010 - ghc-options: -Wall -Wunused-packages - build-depends: - aeson >=1.5.6.0, - attoparsec >=0.13.2.5, - base >=4.7 && <5, - bytestring >=0.10.12.0, - containers >=0.6.2.1, - matrices >=0.5.0, - mwc-random >=0.15.0.1, - parallel >=3.2.2.0, - primitive >=0.7.1.0, - vector >=0.12.3.0, - vector-th-unbox >=0.2.1.9, - word8 >=0.1.3 + exposed-modules: + ELynx.Data.Alphabet.Alphabet + ELynx.Data.Alphabet.Character + ELynx.Data.Alphabet.DistributionDiversity + ELynx.Data.Character.AminoAcid + ELynx.Data.Character.AminoAcidI + ELynx.Data.Character.AminoAcidS + ELynx.Data.Character.AminoAcidX + ELynx.Data.Character.Character + ELynx.Data.Character.Codon + ELynx.Data.Character.Nucleotide + ELynx.Data.Character.NucleotideI + ELynx.Data.Character.NucleotideX + ELynx.Data.Sequence.Alignment + ELynx.Data.Sequence.Defaults + ELynx.Data.Sequence.Distance + ELynx.Data.Sequence.Sequence + ELynx.Data.Sequence.Translate + ELynx.Export.Sequence.Fasta + ELynx.Import.Sequence.Fasta + other-modules: + Paths_elynx_seq + autogen-modules: + Paths_elynx_seq + hs-source-dirs: src + ghc-options: -Wall -Wunused-packages + build-depends: + aeson + , attoparsec + , base >=4.7 && <5 + , bytestring + , containers + , matrices + , mwc-random + , parallel + , primitive + , vector + , vector-th-unbox + , word8 + default-language: Haskell2010 test-suite seq-test - type: exitcode-stdio-1.0 - main-is: Spec.hs - hs-source-dirs: test - other-modules: - ELynx.Data.Alphabet.DistributionDiversitySpec - ELynx.Data.Sequence.AlignmentSpec - ELynx.Data.Sequence.SequenceSpec - ELynx.Data.Sequence.TranslateSpec - ELynx.Export.Sequence.FastaSpec - ELynx.Import.Sequence.FastaSpec - Paths_elynx_seq - - default-language: Haskell2010 - ghc-options: -Wall -Wunused-packages - build-depends: - base >=4.7 && <5, - bytestring >=0.10.12.0, - elynx-seq -any, - elynx-tools >=0.5.1.1, - hspec >=2.7.10, - matrices >=0.5.0, - vector >=0.12.3.0 + type: exitcode-stdio-1.0 + main-is: Spec.hs + other-modules: + ELynx.Data.Alphabet.DistributionDiversitySpec + ELynx.Data.Sequence.AlignmentSpec + ELynx.Data.Sequence.SequenceSpec + ELynx.Data.Sequence.TranslateSpec + ELynx.Export.Sequence.FastaSpec + ELynx.Import.Sequence.FastaSpec + Paths_elynx_seq + autogen-modules: + Paths_elynx_seq + hs-source-dirs: test + ghc-options: -Wall -Wunused-packages + build-depends: + base >=4.7 && <5 + , bytestring + , elynx-seq + , elynx-tools + , hspec + , matrices + , vector + default-language: Haskell2010 diff --git a/src/ELynx/Data/Alphabet/Character.hs b/src/ELynx/Data/Alphabet/Character.hs --- a/src/ELynx/Data/Alphabet/Character.hs +++ b/src/ELynx/Data/Alphabet/Character.hs @@ -37,6 +37,12 @@ newtype Character = Character Word8 deriving (Read, Show, Eq, Ord, Bounded) +derivingUnbox + "Character" + [t|Character -> Word8|] + [|\(Character w) -> w|] + [|Character|] + -- | Conversion of 'Character's. toWord :: Character -> Word8 toWord (Character w) = w @@ -68,9 +74,3 @@ -- | Conversion of 'Character's. fromCVec :: C.Character a => V.Vector a -> V.Vector Character fromCVec = V.map (fromWord . C.toWord) - -derivingUnbox - "Character" - [t|Character -> Word8|] - [|\(Character w) -> w|] - [|Character|] diff --git a/src/ELynx/Data/Sequence/Alignment.hs b/src/ELynx/Data/Sequence/Alignment.hs --- a/src/ELynx/Data/Sequence/Alignment.hs +++ b/src/ELynx/Data/Sequence/Alignment.hs @@ -218,7 +218,7 @@ al = alphabet a f v = case V.find (A.isStd al) v of Nothing -> False - Just c -> V.all (\x -> x == c || (A.isGap al) x || (A.isUnknown al) x) v + Just c -> V.all (\x -> x == c || A.isGap al x || A.isUnknown al x) v -- | Only keep columns with standard characters. Alignment columns with IUPAC -- characters are removed.