diff --git a/README.md b/README.md --- a/README.md +++ b/README.md @@ -229,19 +229,26 @@ # Build modes -The Buck2 build system has two build modes: +The Buck2 build system has the following build mode options, which can +be selected using `-m` on the Buck2 command line (e.g. `-m opt`) - * `dev`: the default, builds everything with `-O0` and dynamic linking. This is intended to give you the quickest edit-compile-test turnaround. - * `opt`: enable `-O` and link statically. This takes longer but the code runs faster. + * Optimisation: + * `dev`: (default) no optimisation, `-g` for C/C++. + * `opt`: adds `-O` for Haskell, `-O3 -DNDEBUG` for C/C++, and also enables + `static` unless overridden by `shared`. + * Linking: + * `shared`: (default) shared libraries, dynamic linking + * `static`: static linking, but also builds shared libraries using + `-dynamic-too` because those are required for Template Haskell. + * Profiling: + * `prof`: enables Haskell profiling; also forces `static`. -To build with `opt`, use `-m opt`, e.g. +To use any of these modes, add them using `-m` on the command line: ``` buck2 build my-package:my-program -m opt ``` -There are other build options that can be selected in a similar way, such as `-m prof` to enable profiling. See `constraints/BUCK` for details. - # Sharing build results (a build cache) By default buck2 keeps nothing between runs of its daemon, nor between @@ -428,32 +435,6 @@ including the package's `setup-depends`. It has to be built with the same Cabal version as `cabal-buck2`, so `cabal-buck2` injects a solver constraint to ensure that. - -## **Template Haskell and `prof`** - -A module that defines a splice must live in a *different* -`haskell_library()` from any module that uses it, when profiling (`-m -prof`). If not, the build will likely complain about a link error or a -missing object file at compile-time. - -The situation with Template Haskell and profiling is complex, as is -the reason for this limitation. - -* Without `-fexternal-interpreter`: GHC loads object code at - compile-time into its own process. Since GHC is itself a - dynamically-linked non-profiled executable, the objects it loads - must be shared, non-profiled, objects. So we have to build all the - dependencies of the current packages as shared libraries. This is - fine, except for the current package: GHC expects to find the - `.dyn_o` objects for the current package in the current `-odir`. But - Buck2 doesn't work this way: it builds the two instances of the - package separately. It's not clear if this is easily fixable. - -* With `-fexternal-interpreter`, we could load the profiled non-shared - objects. However, this method uses the RTS runtime linker, which is - known to have some limitations and can't load some objects, - particularly on certain architectures. This is the main reason that - GHC switched to dynamic linking. So we don't go this route. ## No support for Cabal's `foreign-library` diff --git a/cabal-buck2.cabal b/cabal-buck2.cabal --- a/cabal-buck2.cabal +++ b/cabal-buck2.cabal @@ -1,6 +1,6 @@ cabal-version: 3.8 name: cabal-buck2 -version: 0.4.0.0 +version: 0.5.0.0 -- To ship a new version: -- - bump the version above diff --git a/changelog.md b/changelog.md --- a/changelog.md +++ b/changelog.md @@ -1,3 +1,30 @@ +# 0.5.0.0 + + * For `cabal-install` version 3.18 + * Now works on MacOS X + * A component's C, C++, assembler and Cmm sources are part of its + library, as in Cabal: their objects are in the library's archive + and shared library, rather than in libraries of their own linked + alongside it. A library with no Haskell modules (only C sources, + say) no longer gets a placeholder module. + * A library that isn't exposed (`exposed: False`) is registered as + such, and one whose `ghc-options` give it a unit id + (`-this-unit-id`) has that unit id. + * `-m static` and `-m shared` say how executables are linked, + independently of the build mode (whose default is shared for dev, + static for opt and prof). + * `haskell_library` has a `soname` attribute, for a shared library + whose soname isn't its file's name. + * Configurations can be named by a project's own constraints too: + `named_constraint_settings` in the `[cfg]` section of its + `.buckconfig`. + * Fix for using a combination of a Cabal `main-is` declaration + together with `ghc-options: -main-is ...`. + * Modules in `other-modules` are now correctly hidden outside the + component. + * A Template Haskell splice can use a module of the same library + when profiling (`-m prof`). + # 0.4.0.0 * For `cabal-install` version 3.18 diff --git a/src/Distribution/Client/Buck2/Generate.hs b/src/Distribution/Client/Buck2/Generate.hs --- a/src/Distribution/Client/Buck2/Generate.hs +++ b/src/Distribution/Client/Buck2/Generate.hs @@ -71,6 +71,7 @@ , hcOptions , hsSourceDirs , includeDirs + , libExposed , otherModules , buildType , lookupComponent @@ -343,16 +344,18 @@ Left problems -> skipBecause problems ("library " ++ targetName ++ " (couldn't resolve all its modules)") Right (srcs, srcAutogen) -> let reexports = [SpecReexport (prettyShow (moduleReexportName r)) (prettyShow original) (specDep <$> from) | (r, Just (from, original)) <- reexportOrigins localIndex pkgDesc lbi clbi lib] - -- A library with no modules (a compatibility package, say) has - -- nothing to build, which a rule can't have. + -- A library with no code at all (a compatibility package, say) + -- has nothing to build, which a rule can't have. extra :: [(ModuleName.ModuleName, AutogenFile)] extra - | null srcs = [(ModuleName.fromString "CabalBuck2Empty", emptyModule targetName)] + | null srcs && null (cSources bi) && null (cxxSources bi) && null (asmSources bi) && null (cmmSources bi) = [(ModuleName.fromString "CabalBuck2Empty", emptyModule targetName)] | otherwise = [] in componentTargets (specComponent localIndex externalBuildTools Library targetName bi) { scSrcs = srcs ++ [(prettyShow m, SrcAutogen (autogenName f)) | (m, f) <- extra] , scReexports = reexports + , scHiddenModules = map prettyShow (otherModules bi) + , scExposed = libExposed lib , scGeneratedIncludeDirs = generatedIncludeDirs projectRoot pkgDesc lbi bi , scHscOptions = platformDefines lbi } @@ -491,9 +494,11 @@ { specPackageName = unPackageName (packageName pkgDesc) , specPackageVersion = prettyShow (packageVersion pkgDesc) , specGhcOptions = projectGhcOptions - , specDataDir = getSymbolicPath (dataDir pkgDesc) - , specDataFiles = map getSymbolicPath (dataFiles pkgDesc) - , specExtraSourceFiles = map getSymbolicPath (extraSrcFiles pkgDesc) + , -- These are patterns buck2 globs for, as the .cabal file wrote them: + -- `./foo.jpg` is a fine Cabal pattern but not a valid buck2 path. + specDataDir = normalise (getSymbolicPath (dataDir pkgDesc)) + , specDataFiles = map (normalise . getSymbolicPath) (dataFiles pkgDesc) + , specExtraSourceFiles = map (normalise . getSymbolicPath) (extraSrcFiles pkgDesc) , specComponents = components } @@ -519,6 +524,7 @@ , scExtraLibDirs = map getSymbolicPath (extraLibDirs bi) , scDeps = map specDep (libraryDeps bi) , scReexports = [] + , scHiddenModules = [] , scBuildTools = mapMaybe buildToolSpec (ordNub [(pn, exe) | ExeDependency pn exe _ <- buildToolDepends bi]) , scCSources = map getSymbolicPath (cSources bi) , scCxxSources = map getSymbolicPath (cxxSources bi) @@ -528,6 +534,7 @@ , scCcOptions = ccOptions bi , scCxxOptions = cxxOptions bi , scIncludeDirs = map getSymbolicPath (includeDirs bi) + , scExposed = True , scHscOptions = [] , scGeneratedIncludeDirs = [] , scPkgconfig = ordNub [unPkgconfigName n | PkgconfigDependency n _ <- pkgconfigDepends bi] diff --git a/src/Distribution/Client/Buck2/LocalPackages.hs b/src/Distribution/Client/Buck2/LocalPackages.hs --- a/src/Distribution/Client/Buck2/LocalPackages.hs +++ b/src/Distribution/Client/Buck2/LocalPackages.hs @@ -28,7 +28,7 @@ import qualified Data.Map as Map import qualified Data.Set as Set import System.Directory (doesDirectoryExist, doesFileExist, listDirectory, removeDirectoryRecursive) -import System.FilePath (makeRelative, ()) +import System.FilePath (makeRelative, normalise, ()) import Distribution.Client.DistDirLayout (DistDirLayout (distUnpackedSrcDirectory, distUnpackedSrcRootDirectory)) import qualified Distribution.Client.InstallPlan as InstallPlan @@ -300,9 +300,11 @@ where name = unUnqualComponentName component --- | A directory relative to the project root, @.@ for the root itself. +-- | A directory relative to the project root, @.@ for the root itself. The +-- directory is as @cabal.project@ names it, so it may be relative already, +-- with a @.\/@ in front (@packages: .\/foo@) that a buck2 label can't have. rootRelativeDir :: FilePath -> FilePath -> FilePath -rootRelativeDir projectRoot dir = case makeRelative projectRoot dir of +rootRelativeDir projectRoot dir = case makeRelative projectRoot (normalise dir) of "" -> "." rel -> rel diff --git a/src/Distribution/Client/Buck2/Prebuilt.hs b/src/Distribution/Client/Buck2/Prebuilt.hs --- a/src/Distribution/Client/Buck2/Prebuilt.hs +++ b/src/Distribution/Client/Buck2/Prebuilt.hs @@ -32,7 +32,6 @@ import qualified Data.ByteString as BS import qualified Data.ByteString.Char8 as BS8 -import Data.Char (isHexDigit) import Data.List (stripPrefix) import qualified Data.Map as Map import qualified Data.Set as Set @@ -76,7 +75,7 @@ import Distribution.InstalledPackageInfo (parseInstalledPackageInfo) import Distribution.Package (HasUnitId (installedUnitId), packageName, packageVersion) -import Distribution.Simple.BuildPaths (exeExtension) +import Distribution.Simple.BuildPaths (dllExtension, exeExtension) import Distribution.Simple.Compiler ( Compiler (compilerProperties) , compilerVersion @@ -87,6 +86,7 @@ import Distribution.Simple.Program.Db (lookupProgram) import Distribution.Simple.Program.Types (programPath) import Distribution.Simple.Utils (die', notice, ordNub, warn) +import Distribution.System (OS (..), Platform (..)) import Distribution.Utils.MD5 (md5, showMD5) import Distribution.Types.InstalledPackageInfo ( InstalledPackageInfo @@ -181,6 +181,10 @@ , rpGlobalRootAbs = globalRootAbs , rpStoreRootAbs = storeRootAbs , rpStoreDbRel = makeRelative storeRootAbs storeDB + , rpDllExtension = dllExtension (pkgConfigPlatform shared) + , rpPicByDefault = case pkgConfigPlatform shared of + Platform _ OSX -> True + _ -> False , rpExternal = [] } @@ -261,32 +265,25 @@ , rpStoreRootAbs :: FilePath , rpStoreDbRel :: FilePath -- ^ The store's package db, relative to 'rpStoreRootAbs'. + , rpDllExtension :: String + -- ^ The extension of a shared library, without the dot: @so@, or @dylib@ on macos. + , rpPicByDefault :: Bool + -- ^ Whether GHC's code is always position-independent (as on macOS). , rpExternal :: [(FilePath, FilePath)] -- ^ Directories that are neither under GHC's libdir nor the cabal store, -- and the symlinks (relative to @third-party\/haskell@) to them, which -- they are reached through. See 'externalDirs'. } --- | Which of the two package dbs a unit id's @.conf@ lives in - decided --- entirely by the unit id's own shape, no searching required (unlike the --- Python predecessor, which had to glob for the store root, not knowing --- its ABI-tag suffix in advance): GHC's global\/boot packages have a plain --- @name-version@ id; a package installed to the store gets a --- hash-suffixed one. (An @-inplace@ id - a package buck2 builds itself - --- never gets here: see 'generatePrebuilt's plan parameter.) +-- | Which of the two package dbs a unit id's @.conf@ lives in - found +-- by looking in the store's db first, then GHC's global one (see +-- 'readPackage'). The unit id's shape can't be relied on to tell them +-- apart: on macOS and Windows cabal shortens store unit ids (e.g. +-- LstLk-4.7.8.4-0e779db1). (An @-inplace@ id - a package buck2 builds +-- itself - never gets here: see 'generatePrebuilt's plan parameter.) data PkgDbKind = GlobalDb | StoreDb deriving (Eq) -classifyUnitId :: UnitId -> PkgDbKind -classifyUnitId uid - | hasStoreHashSuffix s = StoreDb - | otherwise = GlobalDb - where - s = unUnitId uid - hasStoreHashSuffix rs = case break (== '-') (reverse rs) of - (revSuffix, '-' : _) -> length revSuffix >= 20 && all isHexDigit revSuffix - _ -> False - -- | A resolved package: its metadata plus which package db it came from -- (needed to pick the right @db =@ value in the generated rule, and to -- know where to find its @.conf@ file). @@ -307,26 +304,32 @@ ++ [("profiled", f) | f <- rpProfiledLibs p] ++ [("shared", f) | (_, f) <- rpSharedLibs p] -confPath :: FilePath -> FilePath -> UnitId -> FilePath -confPath globalRootAbs storeDB uid = case classifyUnitId uid of +confPath :: FilePath -> FilePath -> PkgDbKind -> UnitId -> FilePath +confPath globalRootAbs storeDB kind uid = case kind of GlobalDb -> globalRootAbs "package.conf.d" unUnitId uid <.> "conf" StoreDb -> storeDB unUnitId uid <.> "conf" readPackage :: Verbosity -> RepoPaths -> FilePath -> UnitId -> IO (Maybe ResolvedPackage) readPackage verbosity paths storeDB uid = do - let dbKind = classifyUnitId uid - path = confPath (rpGlobalRootAbs paths) storeDB uid - exists <- doesFileExist path - if not exists - then do + let storePath = confPath (rpGlobalRootAbs paths) storeDB StoreDb uid + globalPath = confPath (rpGlobalRootAbs paths) storeDB GlobalDb uid + storeHas <- doesFileExist storePath + globalHas <- doesFileExist globalPath + case (storeHas, globalHas) of + (True, _) -> parseFrom StoreDb storePath + (_, True) -> parseFrom GlobalDb globalPath + _ -> do warn verbosity $ "cabal buck2: no .conf file for " ++ unUnitId uid - ++ " (expected at " - ++ path + ++ " (looked in " + ++ storePath + ++ " and " + ++ globalPath ++ "); packages are only found in GHC's global package db and the cabal store, not in other package dbs" return Nothing - else do + where + parseFrom dbKind path = do contents <- BS.readFile path case parseInstalledPackageInfo contents of Left errs -> do @@ -347,7 +350,17 @@ if isRts then return [] else findLibs paths (libraryDirs ipi) ["lib" ++ stem ++ "_p" <.> "a" | stem <- hsLibraries ipi] - sharedLibs <- findSharedLibs paths ipi + -- A Haskell shared library mustn't be linked against the + -- RTS: which RTS (threaded or not) is only decided when the + -- final executable is, and on macOS, where each symbol is + -- bound to the library it was linked against, one loaded + -- into a process already running another RTS (GHC itself, + -- running a Template Haskell splice) uses its own, + -- uninitialized, RTS. Cabal doesn't link against it either. + sharedLibs <- + if isRts + then return [] + else findSharedLibs paths ipi return $ Just (ResolvedPackage ipi uid dbKind staticLibs profiledLibs sharedLibs) -- | Resolve each candidate filename against @dirs@ in turn, keeping only @@ -377,7 +390,7 @@ where dirs = libraryDynDirs ipi ++ libraryDirs ipi oneStem stem = do - let soname = "lib" ++ stem ++ "-ghc" ++ rpGhcVersion paths ++ ".so" + let soname = "lib" ++ stem ++ "-ghc" ++ rpGhcVersion paths <.> rpDllExtension paths mpath <- findOne paths dirs soname return ((,) soname <$> mpath) @@ -493,6 +506,7 @@ , ("db", str (if rpDbKind p == GlobalDb then globalDbRel else "cabal-store" rpStoreDbRel paths)) , ("static_libs", strList (rel (rpStaticLibs p))) ] + ++ [("pic_static_libs", strList (rel (rpStaticLibs p))) | rpPicByDefault paths] ++ [("profiled_static_libs", strList profiled) | not (null profiled)] ++ [("pic_profiled_static_libs", strList profiled) | not (null profiled)] ++ [("shared_libs", VDict [(soname, str sopath) | (soname, abs') <- rpSharedLibs p, Just sopath <- [toRepoRelative paths abs']])] diff --git a/src/Distribution/Client/Buck2/Spec.hs b/src/Distribution/Client/Buck2/Spec.hs --- a/src/Distribution/Client/Buck2/Spec.hs +++ b/src/Distribution/Client/Buck2/Spec.hs @@ -78,6 +78,9 @@ , scReexports :: [SpecReexport] -- ^ Modules a library re-exports from the library of another package (or -- another library of its own). + , scHiddenModules :: [String] + -- ^ Of a library: the modules that its package doesn't expose (Cabal's + -- @other-modules@). , scBuildTools :: [SpecBuildTool] , scCSources :: [FilePath] , scCcOptions :: [String] @@ -87,6 +90,8 @@ , scAsmSources :: [FilePath] , scAsmOptions :: [String] , scIncludeDirs :: [FilePath] + , scExposed :: Bool + -- ^ Of a library: whether its package is exposed (Cabal's @exposed@). , scHscOptions :: [String] -- ^ Defines for the C compiler of @hsc2hs@, which say what the code is -- compiled for (GHC does this itself for Haskell code). diff --git a/src/Distribution/Client/Buck2/Write.hs b/src/Distribution/Client/Buck2/Write.hs --- a/src/Distribution/Client/Buck2/Write.hs +++ b/src/Distribution/Client/Buck2/Write.hs @@ -235,6 +235,7 @@ ++ listField "ld_options" (scLdOptions c) ++ valuesField "deps" (map depValue (scDeps c)) ++ valuesField "reexports" (map reexportValue (scReexports c)) + ++ listField "hidden_modules" (scHiddenModules c) ++ valuesField "build_tools" (map buildToolValue (scBuildTools c)) ++ listField "c_sources" (scCSources c) ++ listField "cc_options" (scCcOptions c) @@ -244,6 +245,7 @@ ++ listField "asm_sources" (scAsmSources c) ++ listField "asm_options" (scAsmOptions c) ++ listField "include_dirs" (scIncludeDirs c) + ++ [("exposed", VBool False) | not (scExposed c)] ++ listField "hsc_options" (scHscOptions c) ++ listField "generated_include_dirs" (scGeneratedIncludeDirs c) ++ listField "pkgconfig" (scPkgconfig c) diff --git a/test/Main.hs b/test/Main.hs --- a/test/Main.hs +++ b/test/Main.hs @@ -78,6 +78,7 @@ , ("pkgconfig", pkgconfig) , ("project-overrides", projectOverrides) , ("same-name", sameName) + , ("normalise-paths", normalisePaths) ] -- | A library and an executable with the same name in one package, and @@ -99,6 +100,23 @@ assertContains "user/BUCK.cabal.bzl" "'exe': 'tool-exe'" userBzl assertContains "user/BUCK.cabal.bzl" "'package': 'tool'" userBzl +-- | Paths written with a @.\/@ prefix, in cabal.project and in the file +-- patterns of a package: buck2 accepts neither in a label nor in a glob. +normalisePaths :: Project -> IO () +normalisePaths project = do + _ <- buck2 project [] + + -- `./pkg/pkg.cabal` in cabal.project is the package in `pkg`. + packages <- readIn project ("cabal-buck2" "packages.bzl") + assertContains "cabal-buck2/packages.bzl" "'pkg': 'pkg'" packages + assertNotContains "cabal-buck2/packages.bzl" "./" packages + + -- The patterns buck2 globs for come out without the prefix too. + pkgBzl <- readIn project ("pkg" "BUCK.cabal.bzl") + assertContains "pkg/BUCK.cabal.bzl" "'data/x.txt'" pkgBzl + assertContains "pkg/BUCK.cabal.bzl" "'README'" pkgBzl + assertNotContains "pkg/BUCK.cabal.bzl" "'./" pkgBzl + -- | A copy of a fixture project, and how to run @cabal-buck2@ in it. data Project = Project { projectDir :: FilePath @@ -652,7 +670,7 @@ -- @ghcWithPackages@) are reached through a symlink for each directory. externalDirsTest :: IO () externalDirsTest = do - let paths = RepoPaths{rpGhcVersion = "9.6.7", rpGlobalRootAbs = "/ghc/lib", rpStoreRootAbs = "/home/u/.cabal/store/ghc-9.6.7", rpStoreDbRel = "package.db", rpExternal = []} + let paths = RepoPaths{rpGhcVersion = "9.6.7", rpGlobalRootAbs = "/ghc/lib", rpStoreRootAbs = "/home/u/.cabal/store/ghc-9.6.7", rpStoreDbRel = "package.db", rpExternal = [], rpDllExtension = "so", rpPicByDefault = False} aeson = "/nix/store/aaa-aeson-lib-aeson-2.2.5.1/lib/x86_64-linux-ghc-9.6.7" text = "/nix/store/bbb-text-lib-text-2.1.4/lib/x86_64-linux-ghc-9.6.7" dirs = diff --git a/test/fixtures/normalise-paths/.buckconfig b/test/fixtures/normalise-paths/.buckconfig new file mode 100644 --- /dev/null +++ b/test/fixtures/normalise-paths/.buckconfig @@ -0,0 +1,4 @@ +# Placeholder: this test only checks `cabal buck2`'s generated files, it +# never actually invokes a real buck2 binary, so this doesn't need to be +# a functional buck2 config - it only needs to exist so that +# `ensureBuckconfigAndPackage` sees it's already there and leaves it alone. diff --git a/test/fixtures/normalise-paths/PACKAGE b/test/fixtures/normalise-paths/PACKAGE new file mode 100644 --- /dev/null +++ b/test/fixtures/normalise-paths/PACKAGE @@ -0,0 +1,1 @@ +# Placeholder - see .buckconfig. diff --git a/test/fixtures/normalise-paths/buck2/.gitkeep b/test/fixtures/normalise-paths/buck2/.gitkeep new file mode 100644 --- /dev/null +++ b/test/fixtures/normalise-paths/buck2/.gitkeep diff --git a/test/fixtures/normalise-paths/cabal.project b/test/fixtures/normalise-paths/cabal.project new file mode 100644 --- /dev/null +++ b/test/fixtures/normalise-paths/cabal.project @@ -0,0 +1,5 @@ +-- A package path with a `./` prefix, naming the .cabal file: how `cabal init` +-- and many hand-written projects write it. +packages: ./pkg/pkg.cabal + +active-repositories: :none diff --git a/test/fixtures/normalise-paths/pkg/data/x.txt b/test/fixtures/normalise-paths/pkg/data/x.txt new file mode 100644 --- /dev/null +++ b/test/fixtures/normalise-paths/pkg/data/x.txt @@ -0,0 +1,1 @@ +x diff --git a/test/fixtures/normalise-paths/pkg/pkg.cabal b/test/fixtures/normalise-paths/pkg/pkg.cabal new file mode 100644 --- /dev/null +++ b/test/fixtures/normalise-paths/pkg/pkg.cabal @@ -0,0 +1,13 @@ +cabal-version: 2.4 +name: pkg +version: 0.1.0.0 +build-type: Simple +-- File patterns with a `./` prefix, as Cabal allows. +data-files: ./data/x.txt +extra-source-files: ./README + +library + exposed-modules: Pkg + hs-source-dirs: src + build-depends: base + default-language: Haskell2010 diff --git a/test/fixtures/normalise-paths/pkg/src/Pkg.hs b/test/fixtures/normalise-paths/pkg/src/Pkg.hs new file mode 100644 --- /dev/null +++ b/test/fixtures/normalise-paths/pkg/src/Pkg.hs @@ -0,0 +1,4 @@ +module Pkg (x) where + +x :: Int +x = 1