migrant-core 0.1.0.3 → 0.1.1.1

raw patch · 4 files changed

+220/−14 lines, 4 filesdep ~tastydep ~tasty-quickcheckPVP ok

version bump matches the API change (PVP)

Dependency ranges changed: tasty, tasty-quickcheck

API changes (from Hackage documentation)

Files

README.markdown +204/−0
migrant-core.cabal +4/−3
src/Database/Migrant/Run.hs +5/−4
test/Database/Migrant/Run_Tests.hs +7/−7

+ README.markdown view

@@ -0,0 +1,204 @@+# Migrant++Opinionated SQL schema migration management++## Introduction++Migrant instruments SQL schema migrations in a semi-automated way. Writing the+actual up- and downgrade scripts is still a manual effort; but Migrant takes+care of tracking which scripts have been run already and which still need to be+run, runs them for you.++## The SQL Schema Migration Problem++Databases are notoriously difficult to version-control.++Versioning source code is a solved problem: we write code, put it in a+repository, and the source control software gives us a unique identifier for+that exact version. We can now deploy the code in whichever state we want, and+as long as we keep code and data separated, we can do this in a fairly+brute-force manner: we just delete the old code, copy the new code where it+needs to be, and restart what needs to be restarted. Easy. And because we can+deploy any version of the code we want on any host we want, we can test our+code on one machine (a test server), and then deploy it on another (a+production server), and be reasonably sure that if it works on the test+environment, it will also work in production. We can also, just as easily,+revert the code to an older version, if one of our changes turns out to have+introduced a fault.++But with SQL databases, this doesn't work. The schema and the data stored in+the database are intertwined; we want to manage the schema, but we want to do+it such that no data is lost. We cannot simply overwrite the schema: if we+delete a schema, we also delete all the data in it, because in an SQL database,+data cannot exist without the associated schema. Even small changes, such as+changing the type of a column, can be destructive, and so there is a real risk+of permanently losing data as a result of schema changes. And this means we+must be more surgical about our database mutations.++There are two fundamental approaches to this, which I call "snapshot-based" and+"delta-based".++The "snapshot-based" approach stores a snapshot of the schema at a given+version in the source control system; to migrate the database to that version,+it looks at the current schema, and infers the schema changes that are required+to get the schema into the desired state (or a compatible one). For example, if+there is a table `products` in the database that has three columns (`id`,+`name`, `price`), and the version-controlled schema description says it should+have columns `id`, `name`, `price`, `image`, then the migration code infers+that the `image` column must be added.++The "delta-based" approach, which is what Migrant uses, stores descriptions of+the steps required to arrive at the current schema. A migration run, then,+figures out which migration steps have already been executed, and which ones+are needed to get the schema where we want it, and executes the required steps.+In our example above, there may be two upgrade steps: `create-products-table`,+and `add-product-image`. The migration code detects that only the+`create-products-table` step has been run, and decides to run the+`add-product-image` step.++## How Migrant Works++Migrant migrations are implemented using three key parts:++- A `_migrations` table in the database. Migrant creates this table+  automatically; it is used to track which migrations have been run. Migrant+  will never inspect the schema itself or make any guesses: if the migrations+  table says a migration has been run, then Migrant trusts it.+- A list of migrations that represent the desired state of the database.+  Migrant takes this as a list of strings (`[MigrationName]`; `MigrationName`+  is a newtype over `Text`); you can hard-code that list (which means it will+  be committed to source control along with the rest of your code), you can+  load it dynamically at runtime (which means you can manage migrations+  independently from the application code), or you can use the `embed-file`+  package to compile a separate text file into your application code, and+  commit that text file along with the rest of your code. What matters is that+  the list is version controlled, and that is lists migrations in the order you+  want them to be run.+- A set of `up` and `down` migration actions. Migrant takes two functions that+  will be used to look up these scripts, both of type `MigrationName ->+  connection -> IO ()`, where the `connection -> IO ()` part is the function to+  be run for the "up" or "down" migration. Note that there is no way to signal+  a lookup failure other than throwing an exception: this is by design, and+  works fine, because the migration runner code will respond to a failed query+  exactly the same way as it responds to a missing migration script: it aborts,+  and rolls back the entire migration.++## Using Migrant++1. Add `migrant-core` to your project, and one of the backends+   (`migrant-sqlite-simple`, `migrant-postgresql-simple`, or `migrant-hdbc`).+2. Write some glue code to make your application call `migrate`+3. Write schema migrations as pairs of "up" and "down" scripts, and write a+   list that says in which order to run these scripts.+4. To deploy your migrations, compile the project, and make it run the+   `migrate` function. Migrant will now look for a `_migrations` table in the+   database, creating it if necessary, and run "up" and "down" scripts as+   needed to get the database into the state that your application expects.++Example:++```haskell+{-#LANGUAGE OverloadedStrings #-}++import qualified Database.HDBC as HDBC+import Database.Migrant+import Data.Text (Text)++runMigrations :: HDBC.ConnWrapper -> IO ()+runMigrations conn =+  migrate myMigrations migrateUp migrateDown conn++myMigrations :: [MigrationName]+myMigrations =+  [ "create-users-table"+  , "user-email"+  ]++migrateUp :: MigrationName -> HDBC.ConnWrapper -> IO ()+migrateUp name conn = case name of+  "create-users-table" ->+    HDBC.quickQuery+      conn+      "CREATE TABLE users (id INTEGER NOT NULL SERIAL, username TEXT NOT NULL, password BLOB NULL)"+      []+  "user-email" ->+    HDBC.quickQuery+      conn+      "ALTER TABLE users ADD COLUMN email TEXT NULL"+      []++migrateDown :: MigrationName -> HDBC.ConnWrapper -> IO ()+migrateDown name conn = case name of+  "create-users-table" ->+    HDBC.quickQuery+      conn+      "DROP TABLE users"+      []+  "user-email" ->+    HDBC.quickQuery+      conn+      "ALTER TABLE users DROP COLUMN email"+      []+  +```++## Suggested Practices++- Never manipulate the database schema manually. Migrant cannot know about+  manual changes to the database, so anything you do manually may violate the+  assumptions that the up and down scripts are based on. Any changes you make+  for development purposes should go in pairs of "up" and "down" scripts.+- Test your migrations. A good approach is to clone a production database into+  a test environment and run the updated migrations code against that, then+  check if everything works as intended.+- Also test your "down" scripts: after running the "up" scripts, revert the+  code to the previous version, and run another migration. It should undo the+  upgrade, and the database schema should be in the same (or compatible) state+  as it was before the upgrade.+- The migrations list should be considered "append-only". To undo changes that+  have already been deployed, add another script that undoes them, rather than+  deleting an entry from the migrations list.+- Likewise, do not change migration scripts once committed (unless they were+  committed to a branch that isn't integrated with upstream, such as a local+  WIP branch).+- Different branches can introduce conflicting changes. This is why it is+  important to have both the migration scripts and the migrations list in+  source control: conflicts will appear as merge conflicts, and you can+  manually resolve them the usual way, deciding on an appropriate ordering of+  the conflicting scripts.+- If at all possible, write your "up" and "down" scripts such that they are+  reversible and do not lead to data loss. Dropping tables or columns is a+  prime candidate for data loss, and usually not necessary, at least not+  immediately.+- Consider a staged upgrade strategy using three deployment cycles for each+  change:+    1. Upgrade the database to add the new feature+    2. Upgrade client code to use the new feature+    3. Once all clients are upgraded, drop support for the old feature from the+       database.+- To avoid naming conflicts between independent lines of development, it may be+  a good idea to tag migration names with a sufficiently entropic identifier,+  such as a randomized nonce or a fine-grained timestamp.++## Suggested Development Flow++1. In a development environment, write your intended database change, and an+   "undo", as Migrant migration pair, and reference it in your migrations list.+2. Run migrations on the dev environment.+3. Verify that the migration does what you want. If it does, proceed; if not,+   remove the reference from the list, run migrations (this will run the "undo"+   script), and try again. If your "up" script failed, there is no need to roll+   it back. If the "down" script doesn't work, reset your database by cloning a+   production database or starting with a blank database (this will recreate+   the entire thing from scratch, running all the migrations one by one).+4. Once you're happy with the migration script, commit it.+5. When merging, resolve conflicts in the migrations list and test the merged+   version against a development database. Changing the order of entries in the+   list and re-running migrations will roll back changes up to the last common+   situation, and then re-apply the migrations in the order you specified.+6. As a final test before deploying, clone a production database and run all+   migrations on it.+7. To deploy, first install the application code on the server, and then make+   it run migrations.+8. To undo a deployment, the procedure is exactly the same: install application+   code, then run migrations.

migrant-core.cabal view

@@ -3,7 +3,7 @@ -- For further documentation, see http://haskell.org/cabal/users-guide/  name: migrant-core-version: 0.1.0.3+version: 0.1.1.1 synopsis: Semi-automatic database schema migrations -- description: homepage: https://github.com/tdammers/migrant@@ -15,6 +15,7 @@ -- copyright: category: Database extra-source-files: +extra-doc-files: README.markdown  library   exposed-modules: Database.Migrant@@ -39,7 +40,7 @@                , migrant-core                , HUnit >=1.6.1.0 && <1.7                , QuickCheck >=2.14.2 && <2.15-               , tasty >=1.4 && <1.5+               , tasty >=1.4 && <1.6                , tasty-hunit >=0.10.0.2 && <0.11-               , tasty-quickcheck >=0.10.1.1 && <0.11+               , tasty-quickcheck >=0.10.1.1 && <0.12                , text >=1.2 && <3

src/Database/Migrant/Run.hs view

@@ -41,7 +41,7 @@ makePlan [] xs   -- Situation 1: no more "up" targets left, but more migrations exist, so   -- we need to roll those back.-  = [(MigrateDown, n) | n <- xs]+  = [(MigrateDown, n) | n <- reverse xs] makePlan xs []   -- Situation 2: only "up" targets left, run them.   = [(MigrateUp, n) | n <- xs]@@ -52,10 +52,11 @@   | t == c   = makePlan ts cs   -- Situation 4: both "up" targets and existing migrations are present but the-  -- do not match, so we need to roll back existing migrations until a-  -- consistent situation is restored.+  -- do not match, so we need to roll back all existing migrations, and then+  -- apply all remaining target migrations.   | otherwise-  = (MigrateDown, c):makePlan (t:ts) cs+  = [(MigrateDown, n) | n <- reverse (c:cs)] +++    [(MigrateUp, n) | n <- t:ts]  -- | Apply a migration plan to a database. -- This should generally be done within the same transaction as loading the

test/Database/Migrant/Run_Tests.hs view

@@ -40,21 +40,21 @@   ]  propSameInputEmptyDiff migs =-  makePlan migs migs == []+  makePlan migs migs === []  propAppendIsDiff base ext =-  makePlan (base ++ ext) base == [(MigrateUp, m) | m <- ext]+  makePlan (base ++ ext) base === [(MigrateUp, m) | m <- ext]  propRemoveIsDiff base ext =-  makePlan base (base ++ ext) == [(MigrateDown, m) | m <- ext]+  makePlan base (base ++ ext) === [(MigrateDown, m) | m <- reverse ext]  propDiffNotLargerThanInputs target current =   length (makePlan target current) <= length target + length current  propRemoveBeforeAdd ext1 ext2 =   all not ((==) <$> ext1 <*> ext2) ==>-  makePlan ext1 ext2 ==-  [(MigrateDown, e) | e <- ext2] +++  makePlan ext1 ext2 ===+  [(MigrateDown, e) | e <- reverse ext2] ++   [(MigrateUp, e) | e <- ext1]  executePlanTests :: [TestTree]@@ -114,10 +114,10 @@   let expected =         [ "BEGIN TRANSACTION"         , "INIT"-        , "MigrateDown bar"-        , "MARK MigrateDown bar"         , "MigrateDown quux"         , "MARK MigrateDown quux"+        , "MigrateDown bar"+        , "MARK MigrateDown bar"         , "MigrateUp foo"         , "MARK MigrateUp foo"         , "MigrateUp bar"