diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,6 @@ +# Revision history for tahoe-great-black-swamp-types + +## 0.5.0.0 -- 2023-12-21 + +* First version. Released on an unsuspecting world. +* Types related to Tahoe-LAFS Great Black Swamp split out of tahoe-great-black-swamp package. diff --git a/LICENSE b/LICENSE new file mode 100644 --- /dev/null +++ b/LICENSE @@ -0,0 +1,202 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/src/Tahoe/Storage/Backend.hs b/src/Tahoe/Storage/Backend.hs new file mode 100644 --- /dev/null +++ b/src/Tahoe/Storage/Backend.hs @@ -0,0 +1,361 @@ +{- | Types involved in the implementation of the backend for a Tahoe-LAFS + storage server. +-} +module Tahoe.Storage.Backend where + +import Control.Exception ( + Exception, + ) +import Data.Aeson ( + FromJSON (..), + FromJSONKey (..), + ToJSON (..), + Value (String), + object, + withObject, + withText, + (.:), + (.=), + ) +import Data.ByteArray (constEq) +import qualified Data.ByteString as B +import qualified "base64-bytestring" Data.ByteString.Base64 as Base64 +import Data.Hashable (Hashable (hashWithSalt)) +import Data.Map.Merge.Strict (merge, preserveMissing, zipWithMatched) +import qualified Data.Map.Strict as Map +import qualified Data.Set as Set +import qualified Data.Text.Encoding as T +import GHC.Generics ( + Generic, + ) +import Network.HTTP.Types ( + ByteRanges, + ) + +-- | A human-readable description of the backend software in use. +type ApplicationVersion = B.ByteString + +-- | Give certain operational details about this storage server. +data Version1Parameters = Version1Parameters + { maximumImmutableShareSize :: Size + , maximumMutableShareSize :: Size + , availableSpace :: Size + } + deriving (Show, Eq, Generic) + +-- | Carry a version string and operational parameters. +data Version = Version + { parameters :: Version1Parameters + , applicationVersion :: ApplicationVersion + } + deriving (Show, Eq, Generic) + +-- | The number of bytes in some data. +type Size = Integer + +{- | A position in some data identified by a number of bytes into that data + from the beginning. +-} +type Offset = Integer + +{- | Some ranges of data, possibly. Nothing conventionally refers to the + entire data. +-} +type QueryRange = Maybe ByteRanges + +{- | An opaque identifier for a single storage area. Multiple storage objects + may exist at one storage index if they have different share numbers. TODO: + This should probably be ByteString instead. +-} +type StorageIndex = String + +-- | Some data. +type ShareData = B.ByteString + +{- | The identifier of a distinct storage object, unique within the context of + a particular storage index. In practice, values are in the range [0..255] + (endpoints included). +-} +newtype ShareNumber = ShareNumber Integer + deriving + ( Show + , Eq + , Ord + , Generic + ) + deriving newtype + ( ToJSON + , FromJSON + , FromJSONKey + ) + +instance Hashable ShareNumber where + hashWithSalt i (ShareNumber num) = hashWithSalt i num + +{- | A new type for which we can define our own CBOR serialisation rules. The + cborg library provides a Serialise instance for Set which is not compatible + with the representation required by Tahoe-LAFS. +-} +newtype CBORSet a = CBORSet + { getCBORSet :: Set.Set a + } + deriving newtype (ToJSON, FromJSON, Show, Eq) + +-- | Describe a client-detected incidence of data corruption. +newtype CorruptionDetails = CorruptionDetails + { reason :: String + } + deriving (Show, Eq, Generic) + +{- | A secret shared between a client and this storage server for the purpose + of authorizing certain operations related to immutable uploads. +-} +newtype UploadSecret = UploadSecret B.ByteString + +instance Eq UploadSecret where + (UploadSecret left) == (UploadSecret right) = constEq left right + +{- | A secret shared between an SSK write capability holder and this storage + server for the purpose of authorizing write operations on a mutable object. +-} +newtype WriteEnablerSecret = WriteEnablerSecret B.ByteString + +instance Eq WriteEnablerSecret where + (WriteEnablerSecret left) == (WriteEnablerSecret right) = constEq left right + +-- | Describe tests, reads, and writes to perform on a mutable object. +data ReadTestWriteVectors = ReadTestWriteVectors + { testWriteVectors :: Map.Map ShareNumber TestWriteVectors + , readVector :: [ReadVector] + } + deriving (Show, Eq, Generic) + +-- | Describe tests and writes to perform on a mutable object. +data TestWriteVectors = TestWriteVectors + { test :: [TestVector] + , write :: [WriteVector] + , -- | If given, truncate or extend the object to the given size. If + -- necessary, fill new space with NUL. + newLength :: Maybe Integer + } + deriving (Show, Eq, Generic, ToJSON, FromJSON) + +-- | Describe a single test to perform on a mutable object. +data TestVector = TestVector + { testOffset :: Offset + , testSize :: Size + , operator :: TestOperator + , specimen :: ShareData + } + deriving (Show, Eq, Generic) + +-- | Describe a single write to perform on a mutable object. +data WriteVector = WriteVector + { writeOffset :: Offset + , shareData :: ShareData + } + deriving (Show, Eq, Generic) + +-- | Describe one read to perform on an immutable object. +data ReadVector = ReadVector + { offset :: Offset + , readSize :: Size + } + deriving (Show, Eq, Generic) + +{- | The result of a request to read and/or write some data from/to some + shares. +-} +data ReadTestWriteResult = ReadTestWriteResult + { success :: Bool + , readData :: ReadResult + } + deriving (Show, Eq, Generic) + +-- | The result of a request to read some data from some shares. +type ReadResult = Map.Map ShareNumber [ShareData] + +instance Semigroup TestWriteVectors where + (TestWriteVectors testL writeL _) <> (TestWriteVectors testR writeR newLengthR) = + TestWriteVectors (testL <> testR) (writeL <> writeR) newLengthR + +instance Monoid TestWriteVectors where + mempty = TestWriteVectors mempty mempty Nothing + +instance Monoid ReadTestWriteVectors where + mempty = ReadTestWriteVectors mempty [] + +instance Semigroup ReadTestWriteVectors where + (ReadTestWriteVectors wv0 rv0) <> (ReadTestWriteVectors wv1 rv1) = + ReadTestWriteVectors (merge preserveMissing preserveMissing (zipWithMatched $ \_ l r -> l <> r) wv0 wv1) (rv0 <> rv1) + +{- | Create a ReadTestWriteVectors which performs one read at the given offset + of the given size. +-} +readv :: Offset -> Size -> ReadTestWriteVectors +readv offset size = mempty{readVector = [ReadVector offset size]} + +{- | Create a ReadTestWriteVectors which performs one write on the given share + number at the given offset. +-} +writev :: ShareNumber -> Offset -> ShareData -> ReadTestWriteVectors +writev shareNum offset bytes = mempty{testWriteVectors = Map.singleton shareNum (mempty{write = [WriteVector offset bytes]})} + +{- | Create a ReadTestWriteVectors which performs one test on the given share + number using the given specimen. +-} +testv :: ShareNumber -> Offset -> ShareData -> ReadTestWriteVectors +testv shareNum offset specimen = + mempty + { testWriteVectors = Map.singleton shareNum (mempty{test = [TestVector offset (fromIntegral $ B.length specimen) Eq specimen]}) + } + +-- | Describe some allocations for immutable objects requested by a client. +data AllocateBuckets = AllocateBuckets + { shareNumbers :: [ShareNumber] + , allocatedSize :: Size + } + deriving (Show, Eq, Generic) + +{- | Describe the server's willingness to allocate space for some immutable + objects. +-} +data AllocationResult = AllocationResult + { alreadyHave :: [ShareNumber] + , allocated :: [ShareNumber] + } + deriving (Show, Eq, Generic) + +instance Semigroup AllocationResult where + AllocationResult xa ya <> AllocationResult xb yb = AllocationResult (xa <> xb) (ya <> yb) + +instance Monoid AllocationResult where + mempty = AllocationResult mempty mempty + +{- | One of several kinds of shared secrets for authorizing a client to + perform various operations. +-} +data LeaseSecret = Renew B.ByteString | Cancel B.ByteString | Upload UploadSecret | Write WriteEnablerSecret + +-- XXX Most of these operators have been removed from the spec. +data TestOperator + = Lt + | Le + | Eq + | Ne + | Ge + | Gt + deriving (Show, Eq, Generic, ToJSON, FromJSON) + +-- | A ByteString wrapper which can have its own serialization semantics. +newtype Base64 = Base64 {unBase64 :: B.ByteString} + +-- | Serialize a ByteString by encoding with Base64. +instance ToJSON Base64 where + toJSON = String . T.decodeLatin1 . Base64.encode . unBase64 + +-- | Deserialize a ByteString which has been encoded with Base64. +instance FromJSON Base64 where + parseJSON = + withText + "Base64-encoded ByteString" + ( \bs -> + case Base64.decode $ T.encodeUtf8 bs of + Left err -> fail ("Base64 decoding failed: " <> err) + Right bytes -> pure $ Base64 bytes + ) + +-- | Serialize to the JSON as specified by the specification. +instance ToJSON TestVector where + toJSON (TestVector{..}) = + object ["offset" .= testOffset, "size" .= testSize, "specimen" .= Base64 specimen] + +-- | Deserialize from JSON as specified by the specification. +instance FromJSON TestVector where + parseJSON = withObject "TestVector" $ \v -> + TestVector <$> v .: "offset" <*> v .: "size" <*> pure Eq <*> (unBase64 <$> v .: "specimen") + +-- | Serialize to the JSON as specified by the specification. +instance ToJSON WriteVector where + toJSON (WriteVector{..}) = + object ["offset" .= writeOffset, "data" .= Base64 shareData] + +-- | Deserialize from JSON as specified by the specification. +instance FromJSON WriteVector where + parseJSON = withObject "WriteVector" $ \v -> + WriteVector <$> v .: "offset" <*> (unBase64 <$> v .: "data") + +{- | Exceptional cases which might be encounted by various backend for + operations related to immutable shares. +-} +data WriteImmutableError + = -- | Used to reject an immutable allocate or upload with no upload secret. + MissingUploadSecret + | -- | Used to reject an immutable upload with an incorrect upload secret. + IncorrectUploadSecret + | -- | Used to reject an immutable allocate of a size that does not match + -- existing shares with the same storage index. + ShareSizeMismatch + | -- | Used to reject an immutable allocate of a size greater than the + -- maximum allowed by the server. + MaximumShareSizeExceeded + { maximumShareSizeExceededLimit :: Integer + , maximumShareSizeExceededGiven :: Integer + } + | -- | Used to reject an immutable write to a share that is already + -- completely written. + ImmutableShareAlreadyWritten + | -- | Used to reject an immutable write to a share which has not been + -- allocated. + ShareNotAllocated + | -- | Used to reject an immutable write that overlaps with data that has + -- already been written. + ConflictingWrite + deriving (Ord, Eq, Show) + +instance Exception WriteImmutableError + +{- | Exceptional cases which might be encounted by various backend for + operations related to mutable shares. +-} +data WriteMutableError + = -- | Used to reject a mutable write with no write enabler secret. + MissingWriteEnablerSecret + | -- | Used to reject a mutable write with an incorrect write enabler secret. + IncorrectWriteEnablerSecret + deriving (Ord, Eq, Show) + +instance Exception WriteMutableError + +class Backend b where + version :: b -> IO Version + + -- | Update the lease expiration time on the shares associated with the + -- given storage index. + renewLease :: b -> StorageIndex -> [LeaseSecret] -> IO () + + createImmutableStorageIndex :: b -> StorageIndex -> Maybe [LeaseSecret] -> AllocateBuckets -> IO AllocationResult + + -- May throw ImmutableShareAlreadyWritten + -- XXX Return should indicate what remains to be written + writeImmutableShare :: b -> StorageIndex -> ShareNumber -> Maybe [LeaseSecret] -> ShareData -> Maybe ByteRanges -> IO () + abortImmutableUpload :: b -> StorageIndex -> ShareNumber -> Maybe [LeaseSecret] -> IO () + adviseCorruptImmutableShare :: b -> StorageIndex -> ShareNumber -> CorruptionDetails -> IO () + getImmutableShareNumbers :: b -> StorageIndex -> IO (CBORSet ShareNumber) + readImmutableShare :: b -> StorageIndex -> ShareNumber -> QueryRange -> IO ShareData + + -- | Read some ranges of all shares held and/or, if test conditions are + -- met, overwrite some ranges of some shares. + readvAndTestvAndWritev :: + b -> + -- | The storage index at which to operate. + StorageIndex -> + -- | A shared secret which the backend can use to authorize the writes. + WriteEnablerSecret -> + -- | The reads, tests, and writes to perform. + ReadTestWriteVectors -> + IO ReadTestWriteResult + + readMutableShare :: b -> StorageIndex -> ShareNumber -> QueryRange -> IO ShareData + getMutableShareNumbers :: b -> StorageIndex -> IO (CBORSet ShareNumber) + adviseCorruptMutableShare :: b -> StorageIndex -> ShareNumber -> CorruptionDetails -> IO () diff --git a/tahoe-great-black-swamp-types.cabal b/tahoe-great-black-swamp-types.cabal new file mode 100644 --- /dev/null +++ b/tahoe-great-black-swamp-types.cabal @@ -0,0 +1,104 @@ +cabal-version: 2.4 + +-- The cabal-version field refers to the version of the .cabal specification, +-- and can be different from the cabal-install (the tool) version and the +-- Cabal (the library) version you are using. As such, the Cabal (the library) +-- version used must be equal or greater than the version stated in this field. +-- Starting from the specification version 2.2, the cabal-version field must be +-- the first thing in the cabal file. + +-- Initial package description 'tahoe-great-black-swamp-types' generated by +-- 'cabal init'. For further documentation, see: +-- http://haskell.org/cabal/users-guide/ +-- +-- The name of the package. +name: tahoe-great-black-swamp-types + +-- The package version. +-- See the Haskell package versioning policy (PVP) for standards +-- guiding when and how versions should be incremented. +-- https://pvp.haskell.org +-- PVP summary: +-+------- breaking API changes +-- | | +----- non-breaking API additions +-- | | | +--- code changes with no API change +version: 0.5.0.0 + +-- A short (one-line) description of the package. +synopsis: + Types related to implementation of a Tahoe-LAFS Great Black Swamp server + +-- A longer description of the package. +description: + Types used by the server provided by `tahoe-great-black-swamp` as + well as any implementation of a storage backend for use by that + server. + +-- The license under which the package is released. +license: Apache-2.0 + +-- The file containing the license text. +license-file: LICENSE + +-- The package author(s). +author: Jean-Paul Calderone + +-- An email address to which users can send suggestions, bug reports, and patches. +maintainer: exarkun@twistedmatrix.com + +-- A copyright notice. +-- copyright: +category: Data +build-type: Simple + +-- Extra doc files to be distributed with the package, such as a CHANGELOG or a README. +extra-doc-files: CHANGELOG.md + +-- Extra source files to be distributed with the package, such as examples, or a tutorial module. +-- extra-source-files: + +source-repository head + type: git + location: + git@gitlab.com:tahoe-lafs/tahoe-great-black-swamp-testing.git + +common warnings + ghc-options: -Wall + +library + -- Import common warning flags. + import: warnings + + -- Modules exported by the library. + exposed-modules: Tahoe.Storage.Backend + + -- Modules included in this library but not exported. + -- other-modules: + + -- LANGUAGE extensions used by modules in this package. + -- other-extensions: + + -- Other library packages from which modules are imported. + build-depends: + , aeson >=1.4.7 && <2.2 + , base ^>=4.14.3.0 + , base64-bytestring >=1.0.0.3 && <1.3 + , bytestring >=0.10.8.2 && <0.11 + , containers >=0.6.0.1 && <0.7 + , hashable >=1.4 && <1.5 + , http-types >=0.12.3 && <0.13 + , memory >=0.15 && <0.19 + , text >=1.2.3.1 && <1.3 + + -- Directories containing source files. + hs-source-dirs: src + + -- Base language which the package is written in. + default-language: Haskell2010 + default-extensions: + DeriveAnyClass + DeriveGeneric + DerivingStrategies + GeneralizedNewtypeDeriving + OverloadedStrings + PackageImports + RecordWildCards