diff --git a/LICENSE b/LICENSE new file mode 100644 --- /dev/null +++ b/LICENSE @@ -0,0 +1,30 @@ +Copyright (c) 2017 Gabriel Gonzalez + +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above + copyright notice, this list of conditions and the following + disclaimer in the documentation and/or other materials provided + with the distribution. + + * Neither the name of Gabriel Gonzalez nor the names of other + contributors may be used to endorse or promote products derived + from this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. diff --git a/README.md b/README.md new file mode 100644 --- /dev/null +++ b/README.md @@ -0,0 +1,113 @@ +# `nix-diff 1.0.0` + +This package provides a `nix-diff` executable which explains why two Nix +derivations differ. The most common use cases for this are: + +* Understanding why you have a cache miss +* Verifying a Nix change had the intended effect at the derivation level +* Improving your understanding of how Nix works + +For example, consider the following Nix derivation to build a bare-bones NixOS +system: + +```nix +let + nixos = import { + system = "x86_64-linux"; + + configuration = { + boot.loader.grub.devices = [ "/dev/sda" ]; + + fileSystems."/" = { + device = "/dev/sda"; + }; + }; + }; + +in + nixos.system +``` + +We can use `nix-instantiate` to compute the derivation for the above file: + +```bash +$ nix-instantiate example.nix # Your exact hash might differ +warning: you did not specify ‘--add-root’; the result might be removed by the garbage collector +/nix/store/6z9nr5pzs4j1v9mld517dmlcz61zy78z-nixos-system-nixos-18.03pre119245.5cfd049a03.drv +``` + +Now, let's add a service to our system definition: + +```nix +let + nixos = import { + system = "x86_64-linux"; + + configuration = { + boot.loader.grub.devices = [ "/dev/sda" ]; + + fileSystems."/" = { + device = "/dev/sda"; + }; + + services.apache-kafka.enable = true; + }; + }; + +in + nixos.system +``` + +... and compute the derivation for the updated system: + +```bash +$ nix-instantiate example.nix +warning: you did not specify ‘--add-root’; the result might be removed by the garbage collector +/nix/store/k05ibijg0kknvwrgfyb7dxwjrs8qrlbj-nixos-system-nixos-18.03pre119245.5cfd049a03.drv +``` + +We can use `nix-diff` to compare the two computed derivations to determine what +changed about our system: + +```bash +$ nix-diff /nix/store/6z9nr5pzs4j1v9mld517dmlcz61zy78z-nixos-system-nixos-18.03pre119245.5cfd049a03.drv /nix/store/k05ibijg0kknvwrgfyb7dxwjrs8qrlbj-nixos-system-nixos-18.03pre119245.5cfd049a03.drv +``` + +... which produces the following output: + +![](https://i.imgur.com/KUB4rXx.png) + +## Development status + +[![Build Status](https://travis-ci.org/Gabriel439/nix-diff.png)](https://travis-ci.org/Gabriel439/nix-diff) + +I don't currently plan to add any new features, but I do welcome feature +requests. Just open an issue on the issue tracker if you would like to request +an improvement. + +## License (BSD 3-clause) + + Copyright (c) 2017 Gabriel Gonzalez + All rights reserved. + + Redistribution and use in source and binary forms, with or without modification, + are permitted provided that the following conditions are met: + * Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + * Neither the name of Gabriel Gonzalez nor the names of other contributors + may be used to endorse or promote products derived from this software + without specific prior written permission. + + THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND + ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED + WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE + DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR + ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES + (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; + LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON + ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS + SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. diff --git a/Setup.hs b/Setup.hs new file mode 100644 --- /dev/null +++ b/Setup.hs @@ -0,0 +1,2 @@ +import Distribution.Simple +main = defaultMain diff --git a/nix-diff.cabal b/nix-diff.cabal new file mode 100644 --- /dev/null +++ b/nix-diff.cabal @@ -0,0 +1,33 @@ +name: nix-diff +version: 1.0.0 +synopsis: Explain why two Nix derivations differ +description: This package provides a @nix-diff@ executable which + explains why two Nix derivations (i.e. @*.drv@ files) + differ +homepage: https://github.com/Gabriel439/nix-diff +license: BSD3 +license-file: LICENSE +author: Gabriel Gonzalez +maintainer: Gabriel439@gmail.com +copyright: 2017 Gabriel Gonzalez +category: System +build-type: Simple +tested-with: GHC == 8.0.2, GHC == 8.2.2 +cabal-version: >= 1.10 +extra-source-files: README.md + +executable nix-diff + main-is: Main.hs + build-depends: base >= 4.9 && < 4.10 + , attoparsec >= 0.13 && < 0.14 + , containers >= 0.5 && < 0.6 + , Diff >= 0.3 && < 0.4 + , text >= 1.2 && < 1.3 + , system-filepath >= 0.4 && < 0.5 + , optparse-generic >= 1.1 && < 1.3 + , nix-derivation >= 1.0 && < 1.1 + , mtl >= 2.2 && < 2.3 + , unix < 2.8 + , vector >= 0.12 && < 0.13 + hs-source-dirs: src + default-language: Haskell2010 diff --git a/src/Main.hs b/src/Main.hs new file mode 100644 --- /dev/null +++ b/src/Main.hs @@ -0,0 +1,469 @@ +{-# LANGUAGE DeriveGeneric #-} +{-# LANGUAGE GeneralizedNewtypeDeriving #-} +{-# LANGUAGE NamedFieldPuns #-} +{-# LANGUAGE OverloadedStrings #-} + +module Main where + +import Control.Monad (forM, forM_) +import Control.Monad.IO.Class (MonadIO, liftIO) +import Control.Monad.Reader (MonadReader, ReaderT, ask, local) +import Control.Monad.State (MonadState, StateT, get, put) +import Data.Attoparsec.Text (IResult(..)) +import Data.Map (Map) +import Data.Monoid ((<>)) +import Data.Set (Set) +import Data.Text (Text) +import Data.Vector (Vector) +import Filesystem.Path (FilePath) +import Nix.Derivation (Derivation, DerivationOutput) +import Numeric.Natural (Natural) +import Options.Generic (Generic, ParseRecord) +import Prelude hiding (FilePath) + +import qualified Control.Monad.Reader +import qualified Control.Monad.State +import qualified Data.Algorithm.Diff +import qualified Data.Attoparsec.Text +import qualified Data.Map +import qualified Data.Set +import qualified Data.Text +import qualified Data.Text.IO +import qualified Data.Vector +import qualified Filesystem.Path.CurrentOS +import qualified Nix.Derivation +import qualified Options.Generic +import qualified System.Posix.IO +import qualified System.Posix.Terminal + +data Options = Options FilePath FilePath + deriving (Generic) + +instance ParseRecord Options + +data Context = Context + { tty :: TTY + , indent :: Natural + } + +newtype Status = Status { visited :: Set Diffed } + +data Diffed = Diffed + { leftDerivation :: FilePath + , leftOutput :: Set Text + , rightDerivation :: FilePath + , rightOutput :: Set Text + } deriving (Eq, Ord) + +newtype Diff a = Diff { unDiff :: ReaderT Context (StateT Status IO) a } + deriving + ( Functor + , Applicative + , Monad + , MonadReader Context + , MonadState Status + , MonadIO + ) + +echo :: Text -> Diff () +echo text = do + Context { indent } <- ask + let n = fromIntegral indent + liftIO (Data.Text.IO.putStrLn (Data.Text.replicate n " " <> text)) + +indented :: Natural -> Diff a -> Diff a +indented n = local adapt + where + adapt context = context { indent = indent context + n } + +pathToText :: FilePath -> Text +pathToText path = + case Filesystem.Path.CurrentOS.toText path of + Left text -> text + Right text -> text + +{-| Extract the name of a derivation (i.e. the part after the hash) + + This is used to guess which derivations are related to one another, even + though their hash might differ + + Note that this assumes that the path name is: + + > /nix/store/${32_CHARACTER_HASH}-${NAME}.drv + + Nix technically does not require that the Nix store is actually stored + underneath `/nix/store`, but this is the overwhelmingly common use case +-} +derivationName :: FilePath -> Text +derivationName = Data.Text.dropEnd 4 . Data.Text.drop 44 . pathToText + +-- | Group input derivations by their name +groupByName :: Map FilePath (Set Text) -> Map Text (Map FilePath (Set Text)) +groupByName m = Data.Map.fromList assocs + where + toAssoc key = (derivationName key, Data.Map.filterWithKey predicate m) + where + predicate key' _ = derivationName key == derivationName key' + + assocs = fmap toAssoc (Data.Map.keys m) + +-- | Read and parse a derivation from a file +readDerivation :: FilePath -> Diff Derivation +readDerivation path = do + let string = Filesystem.Path.CurrentOS.encodeString path + text <- liftIO (Data.Text.IO.readFile string) + case Data.Attoparsec.Text.parse Nix.Derivation.parseDerivation text of + Done _ derivation -> do + return derivation + _ -> do + fail ("Could not parse a derivation from this file: " ++ string) + +{-| Join two `Map`s on shared keys, discarding keys which are not present in + both `Map`s +-} +innerJoin :: Ord k => Map k a -> Map k b -> Map k (a, b) +innerJoin = Data.Map.mergeWithKey both left right + where + both _ a b = Just (a, b) + + left _ = Data.Map.empty + + right _ = Data.Map.empty + +data TTY = IsTTY | NotTTY + +-- | Color text red +red :: TTY -> Text -> Text +red IsTTY text = "\ESC[1;31m" <> text <> "\ESC[0m" +red NotTTY text = text + +-- | Color text background red +redBackground :: TTY -> Text -> Text +redBackground IsTTY text = "\ESC[41m" <> text <> "\ESC[0m" +redBackground NotTTY text = "←" <> text <> "←" + +-- | Color text green +green :: TTY -> Text -> Text +green IsTTY text = "\ESC[1;32m" <> text <> "\ESC[0m" +green NotTTY text = text + +-- | Color text background green +greenBackground :: TTY -> Text -> Text +greenBackground IsTTY text = "\ESC[42m" <> text <> "\ESC[0m" +greenBackground NotTTY text = "→" <> text <> "→" + +-- | Color text grey +grey :: TTY -> Text -> Text +grey IsTTY text = "\ESC[1;2m" <> text <> "\ESC[0m" +grey NotTTY text = text + +-- | Format the left half of a diff +minus :: TTY -> Text -> Text +minus tty text = red tty ("- " <> text) + +-- | Format the right half of a diff +plus :: TTY -> Text -> Text +plus tty text = green tty ("+ " <> text) + +-- | Format text explaining a diff +explain :: Text -> Text +explain text = "• " <> text + +{-| Utility to automate a common pattern of printing the two halves of a diff. + This passes the correct formatting function to each half +-} +diffWith :: a -> a -> ((Text -> Text, a) -> Diff ()) -> Diff () +diffWith l r k = do + Context { tty } <- ask + k (minus tty, l) + k (plus tty, r) + +-- | Format the derivation outputs +renderOutputs :: Set Text -> Text +renderOutputs outputs = + ":{" <> Data.Text.intercalate "," (Data.Set.toList outputs) <> "}" + +-- | Diff two outputs +diffOutput + :: Text + -- ^ Output name + -> DerivationOutput + -- ^ Left derivation outputs + -> DerivationOutput + -- ^ Right derivation outputs + -> Diff () +diffOutput outputName leftOutput rightOutput = do + -- We deliberately do not include output paths or hashes in the diff since + -- we already expect them to differ if the inputs differ. Instead, we focus + -- only displaying differing inputs. + let leftHashAlgo = Nix.Derivation.hashAlgo leftOutput + let rightHashAlgo = Nix.Derivation.hashAlgo rightOutput + if leftHashAlgo == rightHashAlgo + then return () + else do + echo (explain ("{" <> outputName <> "}:")) + echo (explain " Hash algorithm:") + diffWith leftHashAlgo rightHashAlgo $ \(sign, hashAlgo) -> do + echo (" " <> sign hashAlgo) + +-- | Diff two sets of outputs +diffOutputs + :: Map Text DerivationOutput + -- ^ Left derivation outputs + -> Map Text DerivationOutput + -- ^ Right derivation outputs + -> Diff () +diffOutputs leftOutputs rightOutputs = do + let leftExtraOutputs = Data.Map.difference leftOutputs rightOutputs + let rightExtraOutputs = Data.Map.difference rightOutputs leftOutputs + + let bothOutputs = innerJoin leftOutputs rightOutputs + + if Data.Map.null leftExtraOutputs && Data.Map.null rightExtraOutputs + then return () + else do + echo (explain "The set of outputs do not match:") + diffWith leftExtraOutputs rightExtraOutputs $ \(sign, extraOutputs) -> do + forM_ (Data.Map.toList extraOutputs) $ \(key, _value) -> do + echo (" " <> sign ("{" <> key <> "}")) + forM_ (Data.Map.toList bothOutputs) $ \(key, (leftOutput, rightOutput)) -> do + if leftOutput == rightOutput + then return () + else diffOutput key leftOutput rightOutput + +-- | Diff two `Text` values +diffText + :: Text + -- ^ Left value to compare + -> Text + -- ^ Right value to compare + -> Diff Text +diffText left right = do + Context { indent, tty } <- ask + let n = fromIntegral indent + + let leftString = Data.Text.unpack left + rightString = Data.Text.unpack right + + let chunks = Data.Algorithm.Diff.getGroupedDiff leftString rightString + + let prefix = Data.Text.replicate n " " + + let format text = + if 80 <= n + Data.Text.length text + then "''\n" <> indentedText <> prefix <> "''" + else text + where + indentedText = + (Data.Text.unlines . fmap indentLine . Data.Text.lines) text + where + indentLine line = prefix <> " " <> line + + let renderChunk (Data.Algorithm.Diff.First l) = + redBackground tty (Data.Text.pack l) + renderChunk (Data.Algorithm.Diff.Second r) = + greenBackground tty (Data.Text.pack r) + renderChunk (Data.Algorithm.Diff.Both l _) = + grey tty (Data.Text.pack l) + + return (format (Data.Text.concat (fmap renderChunk chunks))) + +-- | Diff two environments +diffEnv + :: Set Text + -- ^ Left derivation outputs + -> Set Text + -- ^ Right derivation outputs + -> Map Text Text + -- ^ Left environment to compare + -> Map Text Text + -- ^ Right environment to compare + -> Diff () +diffEnv leftOutputs rightOutputs leftEnv rightEnv = do + let leftExtraEnv = Data.Map.difference leftEnv rightEnv + let rightExtraEnv = Data.Map.difference rightEnv leftEnv + + let bothEnv = innerJoin leftEnv rightEnv + + let predicate key (left, right) = + left == right + || ( Data.Set.member key leftOutputs + && Data.Set.member key rightOutputs + ) + || key == "builder" + || key == "system" + + if Data.Map.null leftExtraEnv + && Data.Map.null rightExtraEnv + && Data.Map.null + (Data.Map.filterWithKey (\k v -> not (predicate k v)) bothEnv) + then return () + else do + echo (explain "The environments do not match:") + diffWith leftExtraEnv rightExtraEnv $ \(sign, extraEnv) -> do + forM_ (Data.Map.toList extraEnv) $ \(key, value) -> do + echo (" " <> sign (key <> "=" <> value)) + forM_ (Data.Map.toList bothEnv) $ \(key, (leftValue, rightValue)) -> do + if predicate key (leftValue, rightValue) + then return () + else do + text <- indented 4 (diffText leftValue rightValue) + echo (" " <> key <> "=" <> text) + +-- | Diff two environments +diffSrcs + :: Set FilePath + -- ^ Left derivation outputs + -> Set FilePath + -- ^ Right derivation outputs + -> Diff Bool +diffSrcs leftSrcs rightSrcs = do + let leftExtraSrcs = Data.Set.difference leftSrcs rightSrcs + let rightExtraSrcs = Data.Set.difference rightSrcs leftSrcs + + if Data.Set.null leftExtraSrcs && Data.Set.null rightExtraSrcs + then return True + else do + echo (explain "The set of input sources do not match:") + diffWith leftExtraSrcs rightExtraSrcs $ \(sign, extraSrcs) -> do + forM_ extraSrcs $ \extraSrc -> do + echo (" " <> sign (pathToText extraSrc)) + return False + +diffPlatform :: Text -> Text -> Diff () +diffPlatform leftPlatform rightPlatform = do + if leftPlatform == rightPlatform + then return () + else do + echo (explain "The platforms do not match") + diffWith leftPlatform rightPlatform $ \(sign, platform) -> do + echo (" " <> sign platform) + +diffBuilder :: Text -> Text -> Diff () +diffBuilder leftBuilder rightBuilder = do + if leftBuilder == rightBuilder + then return () + else do + echo (explain "The builders do not match") + diffWith leftBuilder rightBuilder $ \(sign, builder) -> do + echo (" " <> sign builder) + +diffArgs :: Vector Text -> Vector Text -> Diff () +diffArgs leftArgs rightArgs = do + Context { tty } <- ask + if leftArgs == rightArgs + then return () + else do + echo (explain "The arguments do not match") + let leftList = Data.Vector.toList leftArgs + let rightList = Data.Vector.toList rightArgs + let diffs = Data.Algorithm.Diff.getDiff leftList rightList + let renderDiff (Data.Algorithm.Diff.First arg) = + echo (" " <> minus tty arg) + renderDiff (Data.Algorithm.Diff.Second arg) = + echo (" " <> plus tty arg) + renderDiff (Data.Algorithm.Diff.Both arg _) = + echo (" " <> explain arg) + mapM_ renderDiff diffs + +diff :: FilePath -> Set Text -> FilePath -> Set Text -> Diff () +diff leftPath leftOutputs rightPath rightOutputs = do + Status { visited } <- get + let diffed = Diffed leftPath leftOutputs rightPath rightOutputs + if leftPath == rightPath + then return () + else if Data.Set.member diffed visited + then do + echo (explain "These two derivations have already been compared") + else do + put (Status (Data.Set.insert diffed visited)) + diffWith (leftPath, leftOutputs) (rightPath, rightOutputs) $ \(sign, (path, outputs)) -> do + echo (sign (pathToText path <> renderOutputs outputs)) + + if derivationName leftPath /= derivationName rightPath + then do + echo (explain "The derivation names do not match") + else if leftOutputs /= rightOutputs + then do + echo (explain "The requested outputs do not match") + else do + leftDerivation <- readDerivation leftPath + rightDerivation <- readDerivation rightPath + + let leftOuts = Nix.Derivation.outputs leftDerivation + let rightOuts = Nix.Derivation.outputs rightDerivation + diffOutputs leftOuts rightOuts + + let leftPlatform = Nix.Derivation.platform leftDerivation + let rightPlatform = Nix.Derivation.platform rightDerivation + diffPlatform leftPlatform rightPlatform + + let leftBuilder = Nix.Derivation.builder leftDerivation + let rightBuilder = Nix.Derivation.builder rightDerivation + diffBuilder leftBuilder rightBuilder + + let leftArgs = Nix.Derivation.args leftDerivation + let rightArgs = Nix.Derivation.args rightDerivation + diffArgs leftArgs rightArgs + + let leftInputs = groupByName (Nix.Derivation.inputDrvs leftDerivation) + let rightInputs = groupByName (Nix.Derivation.inputDrvs rightDerivation) + + let leftNames = Data.Map.keysSet leftInputs + let rightNames = Data.Map.keysSet rightInputs + let leftExtraNames = Data.Set.difference leftNames rightNames + let rightExtraNames = Data.Set.difference rightNames leftNames + + if leftNames /= rightNames + then do + echo (explain "The set of input names do not match:") + diffWith leftExtraNames rightExtraNames $ \(sign, names) -> do + forM_ names $ \name -> do + echo (" " <> sign name) + else do + let assocs = Data.Map.toList (innerJoin leftInputs rightInputs) + descended <- forM assocs $ \(inputName, (leftPaths, rightPaths)) -> do + let leftExtraPaths = + Data.Map.difference leftPaths rightPaths + let rightExtraPaths = + Data.Map.difference rightPaths leftPaths + case (Data.Map.toList leftExtraPaths, Data.Map.toList rightExtraPaths) of + _ | leftPaths == rightPaths -> do + return False + ([(leftPath', leftOutputs')], [(rightPath', rightOutputs')]) + | leftOutputs' == rightOutputs' -> do + echo (explain ("The input named `" <> inputName <> "` differs")) + indented 2 (diff leftPath' leftOutputs' rightPath' rightOutputs') + return True + _ -> do + echo (explain ("The set of inputs named `" <> inputName <> "` do not match")) + diffWith leftExtraPaths rightExtraPaths $ \(sign, extraPaths) -> do + forM_ (Data.Map.toList extraPaths) $ \(extraPath, outputs) -> do + echo (" " <> sign (pathToText extraPath <> renderOutputs outputs)) + return False + if or descended + then return () + else do + let leftSrcs = Nix.Derivation.inputSrcs leftDerivation + let rightSrcs = Nix.Derivation.inputSrcs rightDerivation + differed <- diffSrcs leftSrcs rightSrcs + + if not differed + then return () + else do + let leftEnv = Nix.Derivation.env leftDerivation + let rightEnv = Nix.Derivation.env rightDerivation + let leftOutNames = Data.Map.keysSet leftOuts + let rightOutNames = Data.Map.keysSet rightOuts + diffEnv leftOutNames rightOutNames leftEnv rightEnv + +main :: IO () +main = do + Options left right <- Options.Generic.getRecord "Explain why two derivations differ" + b <- System.Posix.Terminal.queryTerminal System.Posix.IO.stdOutput + let tty = if b then IsTTY else NotTTY + let indent = 0 + let context = Context { tty, indent } + let status = Status Data.Set.empty + let action = diff left (Data.Set.singleton "out") right (Data.Set.singleton "out") + Control.Monad.State.evalStateT (Control.Monad.Reader.runReaderT (unDiff action) context) status