diff --git a/MOO.TXT b/MOO.TXT --- a/MOO.TXT +++ b/MOO.TXT @@ -2,7 +2,182 @@ moo: the dbmigrations management tool ------------------------------------- -Before using moo, you must set two environment variables: +The dbmigrations package exposes a library and a tool. The tool, called +"moo", is responsible for creating, installing, and reverting migrations +in your database backend. + +At present, PostgreSQL and Sqlite3 are the only supported database backends. + +The moo tool works by creating migration files in a specific location, +called a migration store, on your filesystem. This directory is where +all possible migrations for your project will be kept. Moo allows you to +create migrations that depend on each other. When you use moo to upgrade +your database schema, it determines which migrations are missing, what +their dependencies are, and installs the required migrations in the +correct order (based on dependencies). + +Moo works by prompting you for new migration information. It then +creates a migration YAML file (whose format is described below), which +you then edit by hand. + +When migrations are installed into your database, the set of installed +migrations is tracked by way of a migration table that is installed into +your database. + +Getting started +--------------- + + 1. Create a directory in which to store migration files. + + 2. Set an environment variable DBM_MIGRATION_STORE to the path to the + directory you created in step 1. + + 3. Set an environment variable DBM_DATABASE_TYPE to a supported + backend. Valid values are "postgresql" and "sqlite3". + + 4. Set an environment variable DBM_DATABASE to a database connection + string that is appropriate for the value of DBM_DATABASE_TYPE you + chose. + + 5. Run "moo upgrade". This command will not actually install any + migrations, since you have not created any, but it will attempt to + connect to your database and install a migration-tracking table. + + If this step succeeds, you should see this output: + + Database is up to date. + + 6. Create a migration with "moo new". Here is an example output: + + $ moo new hello-world + Selecting dependencies for new migration: hello-world + + Confirm: create migration 'hello-world' + (No dependencies) + Are you sure? (yn): y + Migration created successfully: ".../hello-world.txt" + + 7. Edit the migration you created. In this case, moo created a file + $DBM_MIGRATION_STORE/hello_world.txt that looks like this: + + Description: (Description here.) + Created: 2015-02-18 00:50:12.041176 UTC + Depends: + Apply: | + (Apply SQL here.) + + Revert: | + (Revert SQL here.) + + This migration has no valid apply or revert SQL yet; that's for you + to provide. You might edit the apply and revert fields as follows: + + Apply: | + CREATE TABLE foo (a int); + + Revert: | + DROP TABLE foo; + + 8. Test the new migration with "moo test". This will install the + migration in a transaction and roll it back. Here is example output: + + $ moo test hello-world + Applying: hello-world... done. + Reverting: hello-world... done. + Successfully tested migrations. + + 9. Install the migration. This can be done in one of two ways: with + "moo upgrade" or with "moo apply". Here are examples: + + $ moo apply hello-world + Applying: hello-world... done. + Successfully applied migrations. + + $ moo upgrade + Applying: hello-world... done. + Database successfully upgraded. + + 10. List installed migrations with "moo list". + + $ moo list + hello-world + + 10. Revert the migration. + + $ moo revert hello-world + Reverting: hello-world... done. + Successfully reverted migrations. + + 11. List migrations that have not been installed. + + $ moo upgrade-list + Migrations to install: + hello-world + +Configuration file format +------------------------- + +All moo commands accept a --config-file option which you can use to +specify the path to a configuration file containing your settings. This +approach is an alternative to setting environment variables. The +configuration file format uses the same environment variable names for +its fields. An example configuration is as follows: + + DBM_DATABASE_TYPE = "sqlite3" + DBM_DATABASE = "/path/to/database.db" + DBM_MIGRATION_STORE = "/path/to/migration/store" + +Migration file format +--------------------- + +A migration used by this package is a structured document in YAML +format containing these fields: + + Description: (optional) a textual description of the migration + + Dependencies: (required, but may be empty) a whitespace-separated + list of migration names on which the migration + depends; these names are the migration filenames + without the filename extension + + Created: The UTC date and time at which this migration was + created + + Apply: The SQL necessary to apply this migration to the + database + + Revert: (optional) The SQL necessary to revert this migration + from the database + +The format of this file is somewhat flexible; please see the YAML 1.2 +format specification for a full description of syntax features. I +recommend appending "|" to the Apply and Revert fields if they contain +multi-line SQL that you want to keep that way, e.g., + + Apply: | + CREATE OR REPLACE FUNCTION ... + ... + ... + + Revert: | + DROP TABLE foo; + DROP TABLE bar; + +Note that this is only *necessary* when concatenating the lines would +have a different meaning, e.g., + + Apply: + -- Comment here + CREATE TABLE; + +Without "|" on the "Apply:" line, the above text would be collapsed to +"-- Comment here CREATE TABLE;" which is probably not what you want. +For a full treatment of this behavior, see the YAML spec. + +Environment +----------- + +Moo depends on these environment variables: DBM_DATABASE_TYPE diff --git a/README b/README deleted file mode 100644 --- a/README +++ /dev/null @@ -1,127 +0,0 @@ - -dbmigrations ------------- - -This package contains a library and program for the creation, -management, and installation of schema updates (called "migrations") -for a relational database. In particular, this package lets the -migration author express explicit dependencies between migrations and -the management tool automatically installs or reverts migrations -accordingly, using transactions for safety. - -This package operates on two logical entities: - - - The "backend", which is the relational database whose schema you - want to manage. - - - The "migration store" (or simply "store"), which is the collection - of schema changes you want to apply to the database. These - migrations are expressed using plain text files collected together - in a single directory, although the library is general enough to - permit easy implementation of other storage backends for - migrations. - -Migration file format ---------------------- - -While the "moo" management tool included in this package will create -new migration files for you, a description of the file format follows. - -A migration used by this package is a structured document in YAML -format containing these fields: - - Description: (optional) a textual description of the migration - - Dependencies: (required, but may be empty) a whitespace-separated - list of migration names on which the migration - depends; these names are the migration filenames - without the filename extension - - Created: The UTC date and time at which this migration was - created - - Apply: The SQL necessary to apply this migration to the - database - - Revert: (optional) The SQL necessary to revert this migration - from the database - -The format of this file is somewhat flexible; please see the YAML 1.2 -format specification for a full description of syntax features. I -recommend appending "|" to the Apply and Revert fields if they contain -multi-line SQL that you want to keep that way, e.g., - - Apply: | - CREATE OR REPLACE FUNCTION ... - ... - ... - - Revert: | - DROP TABLE foo; - DROP TABLE bar; - -Note that this is only *necessary* when concatenating the lines would -have a different meaning, e.g., - - Apply: - -- Comment here - CREATE TABLE; - -Without "|" on the "Apply:" line, the above text would be collapsed to -"-- Comment here CREATE TABLE;" which is probably not what you want. -For a full treatment of this behavior, see the YAML spec. - -moo: the management tool ------------------------- - -This package includes one program, "moo". For information about moo's -usage and features, please see the file MOO.TXT. - -Installation ------------- - -If you've obtained this package in source form and would like to -install it, you'll need the "cabal" program. To install this package, -run: - - cabal install - -Getting the source ------------------- - -If you've obtained this package by some other means and would like to -get a copy of the source code, you can do so with darcs: - - darcs get http://repos.codevine.org/dbmigrations - -For more information about the "darcs" revision control system, please -see: - - http://darcs.net/ - -Submitting patches ------------------- - -I'll gladly consider accepting patches to this package; please do not -hesitate to send them to me with "darcs send", or mail them to me -manually (see below). If you'd like to send me a patch, I'll be more -likely to accept it if you can follow these guidelines where -appropriate: - - - Keep patches small; a single patch should make a single logical - change with minimal scope. - - - If possible, include tests with your patch. - - - If possible, include haddock with your patch. - -Submitting bug reports and feedback ------------------------------------ - -If you've found a bug or would like to provide any feedback on the -design, implementation, or behavior of this library or its programs, -please do not hesitate to contact me directly at - - drcygnus AT gmail DOT com - -Enjoy! diff --git a/README.md b/README.md new file mode 100644 --- /dev/null +++ b/README.md @@ -0,0 +1,49 @@ + +dbmigrations +------------ + +This package contains a library and program for the creation, +management, and installation of schema updates (called "migrations") +for a relational database. In particular, this package lets the +migration author express explicit dependencies between migrations and +the management tool automatically installs or reverts migrations +accordingly, using transactions for safety. + +This package operates on two logical entities: + + - The "backend", which is the relational database whose schema you + want to manage. + + - The "migration store" (or simply "store"), which is the collection + of schema changes you want to apply to the database. These + migrations are expressed using plain text files collected together + in a single directory, although the library is general enough to + permit easy implementation of other storage backends for + migrations. + +Getting started with dbmigrations +--------------------------------- + +This package includes one program, "moo". See MOO.TXT for details on how +to use this tool to manage your database migrations. + +Installation +------------ + +If you've obtained this package in source form and would like to +install it, you'll need the "cabal" program. To install this package +from the source directory, run `cabal install`. + +Submitting patches +------------------ + +I'll gladly consider accepting patches to this package; please do not +hesitate to submit GitHub pull requests. I'll be more likely to accept +a patch if you can follow these guidelines where appropriate: + + - Keep patches small; a single patch should make a single logical + change with minimal scope. + + - If possible, include tests with your patch. + + - If possible, include haddock with your patch. diff --git a/dbmigrations.cabal b/dbmigrations.cabal --- a/dbmigrations.cabal +++ b/dbmigrations.cabal @@ -1,5 +1,5 @@ Name: dbmigrations -Version: 0.8 +Version: 0.8.1 Synopsis: An implementation of relational database "migrations" Description: A library and program for the creation, management, and installation of schema updates @@ -14,7 +14,7 @@ HDBC-supported database, although at present only PostgreSQL is fully supported. - To get started, see the included 'README' and + To get started, see the included 'README.md' and 'MOO.TXT' files and the usage output for the 'moo' command. @@ -24,18 +24,15 @@ Build-Type: Simple License: BSD3 License-File: LICENSE -Cabal-Version: >= 1.6 -Data-Files: README MOO.TXT +Cabal-Version: >= 1.10 +Data-Files: README.md MOO.TXT Source-Repository head type: git location: git://github.com/jtdaugherty/dbmigrations.git -Flag testing - Description: Build for testing - Default: False - Library + default-language: Haskell2010 if impl(ghc >= 6.12.0) ghc-options: -Wall -fwarn-tabs -funbox-strict-fields -fno-warn-unused-do-bind @@ -45,8 +42,8 @@ Build-Depends: base >= 4 && < 5, HDBC >= 2.2.1, - time == 1.4.*, - random >= 1.0 && < 1.1, + time >= 1.4 && < 1.6, + random >= 1.0 && < 1.2, containers >= 0.2 && < 0.6, mtl >= 2.1 && < 3, filepath >= 1.1 && < 1.4, @@ -55,8 +52,10 @@ template-haskell, yaml-light >= 0.1 && < 0.2, bytestring >= 0.9 && < 1.0, - text >= 0.11 && < 1.2, - configurator == 0.2.* + text >= 0.11 && < 1.3, + configurator >= 0.2 && < 0.4, + HDBC-postgresql, + HDBC-sqlite3 Hs-Source-Dirs: src Exposed-Modules: @@ -76,8 +75,21 @@ Moo.CommandUtils Moo.Core -Executable dbmigrations-tests +test-suite dbmigrations-tests + default-language: Haskell2010 + type: exitcode-stdio-1.0 Build-Depends: + base >= 4 && < 5, + time >= 1.4 && < 1.6, + containers >= 0.2 && < 0.6, + mtl >= 2.1 && < 3, + filepath >= 1.1 && < 1.4, + directory >= 1.0 && < 1.3, + fgl >= 5.4 && < 5.6, + template-haskell, + yaml-light >= 0.1 && < 0.2, + bytestring >= 0.9 && < 1.0, + HDBC >= 2.2.1, HDBC-postgresql, HDBC-sqlite3, HUnit >= 1.2 && < 1.3, @@ -89,14 +101,26 @@ else ghc-options: -threaded -Wall -fwarn-tabs -funbox-strict-fields - if !flag(testing) - Buildable: False - Hs-Source-Dirs: src,test Main-is: TestDriver.hs Executable moo + default-language: Haskell2010 Build-Depends: + base >= 4 && < 5, + HDBC >= 2.2.1, + time >= 1.4 && < 1.6, + random >= 1.0 && < 1.2, + containers >= 0.2 && < 0.6, + mtl >= 2.1 && < 3, + filepath >= 1.1 && < 1.4, + directory >= 1.0 && < 1.3, + fgl >= 5.4 && < 5.6, + template-haskell, + yaml-light >= 0.1 && < 0.2, + bytestring >= 0.9 && < 1.0, + text >= 0.11 && < 1.3, + configurator >= 0.2 && < 0.4, HDBC-postgresql, HDBC-sqlite3 diff --git a/src/Database/Schema/Migrations/Filesystem.hs b/src/Database/Schema/Migrations/Filesystem.hs --- a/src/Database/Schema/Migrations/Filesystem.hs +++ b/src/Database/Schema/Migrations/Filesystem.hs @@ -53,11 +53,7 @@ fullMigrationName s name = return $ storePath s name ++ filenameExtension - loadMigration s theId = do - result <- liftIO $ migrationFromFile s theId - return $ case result of - Left _ -> Nothing - Right m -> Just m + loadMigration s theId = liftIO $ migrationFromFile s theId getMigrations s = do contents <- liftIO $ getDirectoryContents $ storePath s @@ -86,7 +82,7 @@ migrationFromPath :: FilePath -> IO (Either String Migration) migrationFromPath path = do let name = takeBaseName $ takeFileName path - (Right <$> process name) `catch` (\(FilesystemStoreError s) -> return $ Left s) + (Right <$> process name) `catch` (\(FilesystemStoreError s) -> return $ Left $ "Could not parse migration " ++ path ++ ":" ++ s) where process name = do diff --git a/src/Database/Schema/Migrations/Filesystem/Serialize.hs b/src/Database/Schema/Migrations/Filesystem/Serialize.hs --- a/src/Database/Schema/Migrations/Filesystem/Serialize.hs +++ b/src/Database/Schema/Migrations/Filesystem/Serialize.hs @@ -37,11 +37,11 @@ serializeRevert m = case mRevert m of Nothing -> Nothing - Just revert -> Just $ "Revert:\n" ++ + Just revert -> Just $ "Revert: |\n" ++ (serializeMultiline revert) serializeApply :: FieldSerializer -serializeApply m = Just $ "Apply:\n" ++ (serializeMultiline $ mApply m) +serializeApply m = Just $ "Apply: |\n" ++ (serializeMultiline $ mApply m) commonPrefix :: String -> String -> String commonPrefix a b = map fst $ takeWhile (uncurry (==)) (zip a b) diff --git a/src/Database/Schema/Migrations/Store.hs b/src/Database/Schema/Migrations/Store.hs --- a/src/Database/Schema/Migrations/Store.hs +++ b/src/Database/Schema/Migrations/Store.hs @@ -22,7 +22,7 @@ ) where -import Data.Maybe ( catMaybes, isJust ) +import Data.Maybe ( isJust ) import Control.Monad ( mzero ) import Control.Applicative ( (<$>) ) import qualified Data.Map as Map @@ -51,7 +51,7 @@ -- which existing migrations can be loaded. class (Monad m) => MigrationStore s m where -- |Load a migration from the store. - loadMigration :: s -> String -> m (Maybe Migration) + loadMigration :: s -> String -> m (Either String Migration) -- |Save a migration to the store. saveMigration :: s -> Migration -> m () @@ -73,6 +73,8 @@ -- ^ An error was encountered when -- constructing the dependency graph for -- this store. + | InvalidMigration String + -- ^ The specified migration is invalid. deriving (Eq) instance Show MapValidationError where @@ -80,6 +82,8 @@ "Migration " ++ (show from) ++ " references nonexistent dependency " ++ show to show (DependencyGraphError msg) = "There was an error constructing the dependency graph: " ++ msg + show (InvalidMigration msg) = + "There was an error loading a migration: " ++ msg -- |A convenience function for extracting the list of 'Migration's -- extant in the specified 'StoreData'. @@ -100,12 +104,19 @@ loadMigrations :: (MigrationStore s m) => s -> m (Either [MapValidationError] StoreData) loadMigrations store = do migrations <- getMigrations store - loaded <- mapM (\name -> loadMigration store name) migrations - let mMap = Map.fromList $ [ (mId e, e) | e <- catMaybes $ loaded ] + loadedWithErrors <- mapM (\name -> loadMigration store name) migrations + + let mMap = Map.fromList $ [ (mId e, e) | e <- loaded ] validationErrors = validateMigrationMap mMap + (loaded, loadErrors) = sortResults loadedWithErrors ([], []) + allErrors = validationErrors ++ (InvalidMigration <$> loadErrors) - case null validationErrors of - False -> return $ Left validationErrors + sortResults [] v = v + sortResults (Left e:rest) (ms, es) = sortResults rest (ms, e:es) + sortResults (Right m:rest) (ms, es) = sortResults rest (m:ms, es) + + case null allErrors of + False -> return $ Left allErrors True -> do -- Construct a dependency graph and, if that succeeds, return -- StoreData.