diff --git a/LICENSE b/LICENSE new file mode 100644 --- /dev/null +++ b/LICENSE @@ -0,0 +1,202 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright © 2019-2020 47 Degrees. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/mu-avro.cabal b/mu-avro.cabal new file mode 100644 --- /dev/null +++ b/mu-avro.cabal @@ -0,0 +1,51 @@ +cabal-version: >=1.10 +name: mu-avro +version: 0.1.0.0 +synopsis: Avro serialization support for Mu microservices +description: You can use @mu-avro@ to read AVRO Schema Declarations for mu-haskell +license: Apache-2.0 +license-file: LICENSE +author: Alejandro Serrano, Flavio Corpa +maintainer: alejandro.serrano@47deg.com +copyright: Copyright © 2019-2020 +category: Network +build-type: Simple +data-files: test/avro/*.avsc +homepage: https://higherkindness.io/mu-haskell/ +bug-reports: https://github.com/higherkindness/mu-haskell/issues + +source-repository head + type: git + location: https://github.com/higherkindness/mu-haskell + +library + exposed-modules: Mu.Adapter.Avro + , Mu.Quasi.Avro + , Mu.Quasi.Avro.Example + build-depends: base >=4.12 && <5 + , mu-schema + , avro + , tagged + , aeson + , text + , vector + , containers + , unordered-containers + , sop-core + , bytestring + , template-haskell >= 2.12 + hs-source-dirs: src + default-language: Haskell2010 + ghc-options: -Wall + -fprint-potential-instances + +executable test-avro + main-is: Avro.hs + build-depends: base >=4.12 && <5 + , mu-schema + , mu-avro + , avro + , bytestring + hs-source-dirs: test + default-language: Haskell2010 + ghc-options: -Wall diff --git a/src/Mu/Adapter/Avro.hs b/src/Mu/Adapter/Avro.hs new file mode 100644 --- /dev/null +++ b/src/Mu/Adapter/Avro.hs @@ -0,0 +1,323 @@ +{-# language DataKinds #-} +{-# language FlexibleContexts #-} +{-# language FlexibleInstances #-} +{-# language GADTs #-} +{-# language MultiParamTypeClasses #-} +{-# language PolyKinds #-} +{-# language RankNTypes #-} +{-# language ScopedTypeVariables #-} +{-# language TypeApplications #-} +{-# language TypeOperators #-} +{-# language UndecidableInstances #-} +{-# OPTIONS_GHC -fno-warn-orphans #-} +{-| +Description : Adapter for Avro serialization + +Just import the module and you can turn any +value with a 'ToSchema' and 'FromSchema' from +and to Avro values. +-} +module Mu.Adapter.Avro () where + +import Control.Arrow ((***)) +import qualified Data.Avro as A +import qualified Data.Avro.Schema as ASch +import qualified Data.Avro.Types.Value as AVal +-- 'Tagged . unTagged' can be replaced by 'coerce' +-- eliminating some run-time overhead +import Data.Coerce (coerce) +import Data.Functor.Identity +import qualified Data.HashMap.Strict as HM +import Data.List.NonEmpty (NonEmpty (..)) +import qualified Data.List.NonEmpty as NonEmptyList +import qualified Data.Map as M +import Data.Tagged +import qualified Data.Text as T +import qualified Data.Vector as V +import GHC.TypeLits + +import Mu.Schema +import qualified Mu.Schema.Interpretation.Schemaless as SLess + +instance SLess.ToSchemalessTerm (AVal.Value t) Identity where + toSchemalessTerm (AVal.Record _ r) + = SLess.TRecord $ map (\(k,v) -> SLess.Field k (Identity $ SLess.toSchemalessValue v)) + $ HM.toList r + toSchemalessTerm (AVal.Enum _ i _) + = SLess.TEnum i + toSchemalessTerm (AVal.Union _ _ v) + = SLess.toSchemalessTerm v + toSchemalessTerm v = SLess.TSimple (SLess.toSchemalessValue v) + +instance SLess.ToSchemalessValue (AVal.Value t) Identity where + toSchemalessValue AVal.Null = SLess.FNull + toSchemalessValue (AVal.Boolean b) = SLess.FPrimitive b + toSchemalessValue (AVal.Int b) = SLess.FPrimitive b + toSchemalessValue (AVal.Long b) = SLess.FPrimitive b + toSchemalessValue (AVal.Float b) = SLess.FPrimitive b + toSchemalessValue (AVal.Double b) = SLess.FPrimitive b + toSchemalessValue (AVal.String b) = SLess.FPrimitive b + toSchemalessValue (AVal.Fixed _ b) = SLess.FPrimitive b + toSchemalessValue (AVal.Bytes b) = SLess.FPrimitive b + toSchemalessValue (AVal.Array v) + = SLess.FList $ map SLess.toSchemalessValue $ V.toList v + toSchemalessValue (AVal.Map hm) + = SLess.FMap $ M.fromList + $ map (SLess.FPrimitive *** SLess.toSchemalessValue) + $ HM.toList hm + toSchemalessValue (AVal.Union _ _ v) + = SLess.toSchemalessValue v + toSchemalessValue r@(AVal.Record _ _) + = SLess.FSchematic (SLess.toSchemalessTerm r) + toSchemalessValue e@AVal.Enum {} + = SLess.FSchematic (SLess.toSchemalessTerm e) + +instance HasAvroSchemas sch sch + => A.HasAvroSchema (WithSchema f sch sty t) where + -- the previous iteration added only the schema of the type + -- schema = coerce $ A.schema @(Term sch (sch :/: sty)) + -- but now we prefer to have all of them + schema = Tagged $ ASch.Union (schemas (Proxy @sch) (Proxy @sch)) +instance ( FromSchema f sch sty t, HasAvroSchemas sch sch + , A.FromAvro (Term f sch (sch :/: sty)) ) + => A.FromAvro (WithSchema f sch sty t) where + fromAvro (AVal.Union _ _ v) = WithSchema . fromSchema' @_ @_ @sch @f <$> A.fromAvro v + fromAvro v = ASch.badValue v "top-level" +instance ( ToSchema Identity sch sty t, HasAvroSchemas sch sch + , A.ToAvro (Term Identity sch (sch :/: sty)) ) + => A.ToAvro (WithSchema Identity sch sty t) where + toAvro (WithSchema v) = AVal.Union (schemas (Proxy @sch) (Proxy @sch)) + (unTagged $ A.schema @(Term Identity sch (sch :/: sty))) + (A.toAvro (toSchema' @_ @_ @sch @Identity v)) + +class HasAvroSchemas (r :: Schema tn fn) (sch :: Schema tn fn) where + schemas :: Proxy r -> Proxy sch -> V.Vector ASch.Type +instance HasAvroSchemas r '[] where + schemas _ _ = V.empty +instance forall r d ds. + (A.HasAvroSchema (Term Identity r d), HasAvroSchemas r ds) + => HasAvroSchemas r (d ': ds) where + schemas pr _ = V.cons thisSchema (schemas pr (Proxy @ds)) + where thisSchema = unTagged $ A.schema @(Term Identity r d) + +-- HasAvroSchema instances + +instance (KnownName name, HasAvroSchemaFields sch args) + => A.HasAvroSchema (Term f sch ('DRecord name args)) where + schema = Tagged $ ASch.Record recordName [] Nothing Nothing fields + where recordName = nameTypeName (Proxy @name) + fields = schemaF (Proxy @sch) (Proxy @args) +instance (KnownName name, HasAvroSchemaEnum choices) + => A.HasAvroSchema (Term f sch ('DEnum name choices)) where + schema = Tagged $ ASch.mkEnum enumName [] Nothing choicesNames + where enumName = nameTypeName (Proxy @name) + choicesNames = schemaE (Proxy @choices) +instance A.HasAvroSchema (FieldValue f sch t) + => A.HasAvroSchema (Term f sch ('DSimple t)) where + schema = coerce $ A.schema @(FieldValue f sch t) + +instance A.HasAvroSchema (FieldValue f sch 'TNull) where + schema = Tagged ASch.Null +instance A.HasAvroSchema t + => A.HasAvroSchema (FieldValue f sch ('TPrimitive t)) where + schema = coerce $ A.schema @t +instance KnownName t + => A.HasAvroSchema (FieldValue f sch ('TSchematic t)) where + -- schema = coerce $ A.schema @(Term sch (sch :/: t)) + schema = Tagged $ ASch.NamedType (nameTypeName (Proxy @t)) +instance forall sch f choices. + HasAvroSchemaUnion (FieldValue f sch) choices + => A.HasAvroSchema (FieldValue f sch ('TUnion choices)) where + schema = Tagged $ ASch.mkUnion $ schemaU (Proxy @(FieldValue f sch)) (Proxy @choices) +instance A.HasAvroSchema (FieldValue f sch t) + => A.HasAvroSchema (FieldValue f sch ('TOption t)) where + schema = coerce $ A.schema @(Maybe (FieldValue f sch t)) +instance A.HasAvroSchema (FieldValue f sch t) + => A.HasAvroSchema (FieldValue f sch ('TList t)) where + schema = coerce $ A.schema @[FieldValue f sch t] +-- These are the only two versions of Map supported by the library +instance A.HasAvroSchema (FieldValue f sch v) + => A.HasAvroSchema (FieldValue f sch ('TMap ('TPrimitive T.Text) v)) where + schema = coerce $ A.schema @(M.Map T.Text (FieldValue f sch v)) +instance A.HasAvroSchema (FieldValue f sch v) + => A.HasAvroSchema (FieldValue f sch ('TMap ('TPrimitive String) v)) where + schema = coerce $ A.schema @(M.Map String (FieldValue f sch v)) + +class HasAvroSchemaUnion (f :: k -> *) (xs :: [k]) where + schemaU :: Proxy f -> Proxy xs -> NonEmpty ASch.Type +instance A.HasAvroSchema (f v) => HasAvroSchemaUnion f '[v] where + schemaU _ _ = vSchema :| [] + where vSchema = unTagged (A.schema @(f v)) +instance (A.HasAvroSchema (f x), HasAvroSchemaUnion f (y ': zs)) + => HasAvroSchemaUnion f (x ': y ': zs) where + schemaU p _ = xSchema :| NonEmptyList.toList yzsSchema + where xSchema = unTagged (A.schema @(f x)) + yzsSchema = schemaU p (Proxy @(y ': zs)) + +class HasAvroSchemaFields sch (fs :: [FieldDef tn fn]) where + schemaF :: Proxy sch -> Proxy fs -> [ASch.Field] +instance HasAvroSchemaFields sch '[] where + schemaF _ _ = [] +instance (KnownName name, A.HasAvroSchema (FieldValue Identity sch t), HasAvroSchemaFields sch fs) + => HasAvroSchemaFields sch ('FieldDef name t ': fs) where + schemaF psch _ = schemaThis : schemaF psch (Proxy @fs) + where fieldName = nameText (Proxy @name) + schemaT = unTagged $ A.schema @(FieldValue Identity sch t) + schemaThis = ASch.Field fieldName [] Nothing Nothing schemaT Nothing + +class HasAvroSchemaEnum (fs :: [ChoiceDef fn]) where + schemaE :: Proxy fs -> [T.Text] +instance HasAvroSchemaEnum '[] where + schemaE _ = [] +instance (KnownName name, HasAvroSchemaEnum fs) + => HasAvroSchemaEnum ('ChoiceDef name ': fs) where + schemaE _ = nameText (Proxy @name) : schemaE (Proxy @fs) + +-- FromAvro instances + +instance (KnownName name, HasAvroSchemaFields sch args, FromAvroFields f sch args) + => A.FromAvro (Term f sch ('DRecord name args)) where + fromAvro (AVal.Record _ fields) = TRecord <$> fromAvroF fields + fromAvro v = A.badValue v "record" +instance (KnownName name, HasAvroSchemaEnum choices, FromAvroEnum choices) + => A.FromAvro (Term f sch ('DEnum name choices)) where + fromAvro v@(AVal.Enum _ n _) = TEnum <$> fromAvroEnum v n + fromAvro v = A.badValue v "enum" +instance A.FromAvro (FieldValue f sch t) + => A.FromAvro (Term f sch ('DSimple t)) where + fromAvro v = TSimple <$> A.fromAvro v + +instance A.FromAvro (FieldValue f sch 'TNull) where + fromAvro AVal.Null = return FNull + fromAvro v = A.badValue v "null" +instance A.FromAvro t => A.FromAvro (FieldValue f sch ('TPrimitive t)) where + fromAvro v = FPrimitive <$> A.fromAvro v +instance (KnownName t, A.FromAvro (Term f sch (sch :/: t))) + => A.FromAvro (FieldValue f sch ('TSchematic t)) where + fromAvro v = FSchematic <$> A.fromAvro v +instance (HasAvroSchemaUnion (FieldValue f sch) choices, FromAvroUnion f sch choices) + => A.FromAvro (FieldValue f sch ('TUnion choices)) where + fromAvro (AVal.Union _ branch v) = FUnion <$> fromAvroU branch v + fromAvro v = A.badValue v "union" +instance A.FromAvro (FieldValue f sch t) + => A.FromAvro (FieldValue f sch ('TOption t)) where + fromAvro v = FOption <$> A.fromAvro v +instance A.FromAvro (FieldValue f sch t) + => A.FromAvro (FieldValue f sch ('TList t)) where + fromAvro v = FList <$> A.fromAvro v +-- These are the only two versions of Map supported by the library +instance A.FromAvro (FieldValue f sch v) + => A.FromAvro (FieldValue f sch ('TMap ('TPrimitive T.Text) v)) where + fromAvro v = FMap . M.mapKeys FPrimitive <$> A.fromAvro v +instance A.FromAvro (FieldValue f sch v) + => A.FromAvro (FieldValue f sch ('TMap ('TPrimitive String) v)) where + fromAvro v = FMap . M.mapKeys (FPrimitive . T.unpack) <$> A.fromAvro v + +class FromAvroEnum (vs :: [ChoiceDef fn]) where + fromAvroEnum :: AVal.Value ASch.Type -> Int -> A.Result (NS Proxy vs) +instance FromAvroEnum '[] where + fromAvroEnum v _ = A.badValue v "element not found" +instance FromAvroEnum vs => FromAvroEnum (v ': vs) where + fromAvroEnum _ 0 = return (Z Proxy) + fromAvroEnum v n = S <$> fromAvroEnum v (n-1) + +class FromAvroUnion f sch choices where + fromAvroU :: ASch.Type -> AVal.Value ASch.Type -> ASch.Result (NS (FieldValue f sch) choices) +instance FromAvroUnion f sch '[] where + fromAvroU _ v = A.badValue v "union choice not found" +instance (A.FromAvro (FieldValue f sch u), FromAvroUnion f sch us) + => FromAvroUnion f sch (u ': us) where + fromAvroU branch v + | ASch.matches branch (unTagged (A.schema @(FieldValue f sch u))) + = Z <$> A.fromAvro v + | otherwise + = S <$> fromAvroU branch v + +class FromAvroFields f sch (fs :: [FieldDef Symbol Symbol]) where + fromAvroF :: HM.HashMap T.Text (AVal.Value ASch.Type) -> A.Result (NP (Field f sch) fs) +instance FromAvroFields f sch '[] where + fromAvroF _ = return Nil +instance (Applicative f, KnownName name, A.FromAvro (FieldValue f sch t), FromAvroFields f sch fs) + => FromAvroFields f sch ('FieldDef name t ': fs) where + fromAvroF v = case HM.lookup fieldName v of + Nothing -> A.badValue v "field not found" + Just f -> (:*) <$> (Field . pure <$> A.fromAvro f) <*> fromAvroF v + where fieldName = nameText (Proxy @name) + +-- ToAvro instances + +instance (KnownName name, HasAvroSchemaFields sch args, ToAvroFields sch args) + => A.ToAvro (Term Identity sch ('DRecord name args)) where + toAvro (TRecord fields) = AVal.Record wholeSchema (toAvroF fields) + where wholeSchema = unTagged (A.schema @(Term Identity sch ('DRecord name args))) +instance (KnownName name, HasAvroSchemaEnum choices, ToAvroEnum choices) + => A.ToAvro (Term Identity sch ('DEnum name choices)) where + toAvro (TEnum n) = AVal.Enum wholeSchema choice text + where wholeSchema = unTagged (A.schema @(Term Identity sch ('DEnum name choices))) + (choice, text) = toAvroE n +instance A.ToAvro (FieldValue Identity sch t) + => A.ToAvro (Term Identity sch ('DSimple t)) where + toAvro (TSimple v) = A.toAvro v + +instance A.ToAvro (FieldValue Identity sch 'TNull) where + toAvro FNull = AVal.Null +instance A.ToAvro t => A.ToAvro (FieldValue Identity sch ('TPrimitive t)) where + toAvro (FPrimitive v) = A.toAvro v +instance (KnownName t, A.ToAvro (Term Identity sch (sch :/: t))) + => A.ToAvro (FieldValue Identity sch ('TSchematic t)) where + toAvro (FSchematic v) = A.toAvro v +instance forall sch choices. + (HasAvroSchemaUnion (FieldValue Identity sch) choices, ToAvroUnion sch choices) + => A.ToAvro (FieldValue Identity sch ('TUnion choices)) where + toAvro (FUnion v) = AVal.Union wholeSchema' chosenTy chosenVal + where wholeSchema = schemaU (Proxy @(FieldValue Identity sch)) (Proxy @choices) + wholeSchema' = V.fromList (NonEmptyList.toList wholeSchema) + (chosenTy, chosenVal) = toAvroU v +instance A.ToAvro (FieldValue Identity sch t) + => A.ToAvro (FieldValue Identity sch ('TOption t)) where + toAvro (FOption v) = A.toAvro v +instance A.ToAvro (FieldValue Identity sch t) + => A.ToAvro (FieldValue Identity sch ('TList t)) where + toAvro (FList v) = AVal.Array $ V.fromList $ A.toAvro <$> v +-- These are the only two versions of Map supported by the library +instance A.ToAvro (FieldValue Identity sch v) + => A.ToAvro (FieldValue Identity sch ('TMap ('TPrimitive T.Text) v)) where + toAvro (FMap v) = A.toAvro $ M.mapKeys (\(FPrimitive k) -> k) v +instance A.ToAvro (FieldValue Identity sch v) + => A.ToAvro (FieldValue Identity sch ('TMap ('TPrimitive String) v)) where + toAvro (FMap v) = A.toAvro $ M.mapKeys (\(FPrimitive k) -> k) v + +class ToAvroUnion sch choices where + toAvroU :: NS (FieldValue Identity sch) choices -> (ASch.Type, AVal.Value ASch.Type) +instance ToAvroUnion sch '[] where + toAvroU _ = error "ToAvro in an empty union" +instance forall sch u us. + (A.ToAvro (FieldValue Identity sch u), ToAvroUnion sch us) + => ToAvroUnion sch (u ': us) where + toAvroU (Z v) = (unTagged (A.schema @(FieldValue Identity sch u)), A.toAvro v) + toAvroU (S n) = toAvroU n + +class ToAvroEnum choices where + toAvroE :: NS Proxy choices -> (Int, T.Text) +instance ToAvroEnum '[] where + toAvroE = error "ToAvro in an empty enum" +instance (KnownName u, ToAvroEnum us) + => ToAvroEnum ('ChoiceDef u ': us) where + toAvroE (Z _) = (0, nameText (Proxy @u)) + toAvroE (S v) = let (n, t) = toAvroE v in (n + 1, t) + +class ToAvroFields sch (fs :: [FieldDef Symbol Symbol]) where + toAvroF :: NP (Field Identity sch) fs -> HM.HashMap T.Text (AVal.Value ASch.Type) +instance ToAvroFields sch '[] where + toAvroF _ = HM.empty +instance (KnownName name, A.ToAvro (FieldValue Identity sch t), ToAvroFields sch fs) + => ToAvroFields sch ('FieldDef name t ': fs) where + toAvroF (Field (Identity v) :* rest) = HM.insert fieldName fieldValue (toAvroF rest) + where fieldName = nameText (Proxy @name) + fieldValue = A.toAvro v + +-- Conversion of symbols to other things +nameText :: KnownName s => proxy s -> T.Text +nameText = T.pack . nameVal +nameTypeName :: KnownName s => proxy s -> ASch.TypeName +nameTypeName = ASch.parseFullname . nameText diff --git a/src/Mu/Quasi/Avro.hs b/src/Mu/Quasi/Avro.hs new file mode 100644 --- /dev/null +++ b/src/Mu/Quasi/Avro.hs @@ -0,0 +1,132 @@ +{-# language DataKinds #-} +{-# language LambdaCase #-} +{-# language NamedFieldPuns #-} +{-# language TemplateHaskell #-} +{-# language ViewPatterns #-} +{-| +Description : Quasi-quoters for Avro IDL format + +This module turns schema definitions written in + +into Mu 'Schema's. We provide versions for writing +the IDL inline ('avro') and import it from a file +('avroFile'). + +/Note/: as of now, only the JSON-based IDL format +is supported, not the Java-like one. +-} +module Mu.Quasi.Avro ( + -- * Quasi-quoters for @.avsc@ files + avro + , avroFile + -- * Only for internal use + , schemaFromAvroType + ) where + +import Data.Aeson (decode) +import qualified Data.Avro.Schema as A +import qualified Data.ByteString as B +import Data.ByteString.Lazy.Char8 (pack) +import Data.Int +import qualified Data.Text as T +import Data.Vector (fromList, toList) +import Language.Haskell.TH +import Language.Haskell.TH.Quote + +import Mu.Schema.Definition + +-- | Imports an avro definition written in-line as a 'Schema'. +avro :: QuasiQuoter +avro = + QuasiQuoter + (const $ fail "cannot use as expression") + (const $ fail "cannot use as pattern") + schemaFromAvroString + (const $ fail "cannot use as declaration") + +-- | Imports an avro definition from a file as a 'Schema'. +avroFile :: QuasiQuoter +avroFile = quoteFile avro + +schemaFromAvroString :: String -> Q Type +schemaFromAvroString s = + case decode (pack s) of + Nothing -> fail "could not parse avro spec!" + Just (A.Union us) -> schemaFromAvro (toList us) + Just t -> schemaFromAvro [t] + where + schemaFromAvro = + (typesToList <$>) . mapM schemaDecFromAvroType . flattenAvroDecls + +schemaDecFromAvroType :: A.Type -> Q Type +schemaDecFromAvroType (A.Record name _ _ _ fields) = + [t|'DRecord $(textToStrLit $ A.baseName name) + $(typesToList <$> mapM avroFieldToType fields)|] + where + avroFieldToType :: A.Field -> Q Type + avroFieldToType field = + [t|'FieldDef $(textToStrLit $ A.fldName field) + $(schemaFromAvroType $ A.fldType field)|] +schemaDecFromAvroType (A.Enum name _ _ symbols) = + [t|'DEnum $(textToStrLit $ A.baseName name) + $(typesToList <$> mapM avChoiceToType (toList symbols))|] + where + avChoiceToType :: T.Text -> Q Type + avChoiceToType c = [t|'ChoiceDef $(textToStrLit c)|] +schemaDecFromAvroType t = [t|'DSimple $(schemaFromAvroType t)|] + +-- | Turns a schema from Avro into a Template Haskell 'Type'. +schemaFromAvroType :: A.Type -> Q Type +schemaFromAvroType = + \case + A.Null -> [t|'TPrimitive 'TNull|] + A.Boolean -> [t|'TPrimitive Bool|] + A.Int -> [t|'TPrimitive Int32|] + A.Long -> [t|'TPrimitive Int64|] + A.Float -> [t|'TPrimitive Float|] + A.Double -> [t|'TPrimitive Double|] + A.Bytes -> [t|'TPrimitive B.ByteString|] + A.String -> [t|'TPrimitive T.Text|] + A.Array item -> [t|'TList $(schemaFromAvroType item)|] + A.Map values -> [t|'TMap T.Text $(schemaFromAvroType values)|] + A.NamedType typeName -> + [t|'TSchematic $(textToStrLit (A.baseName typeName))|] + A.Enum {} -> fail "should never happen, please, file an issue" + A.Record {} -> fail "should never happen, please, file an issue" + A.Union options -> + case toList options of + [A.Null, x] -> toOption x + [x, A.Null] -> toOption x + _ -> + [t|'TUnion $(typesToList <$> mapM schemaFromAvroType (toList options))|] + where toOption x = [t|'TOption $(schemaFromAvroType x)|] + A.Fixed {} -> fail "fixed integers are not currently supported" + +flattenAvroDecls :: [A.Type] -> [A.Type] +flattenAvroDecls = concatMap (uncurry (:) . flattenDecl) + where + flattenDecl :: A.Type -> (A.Type, [A.Type]) + flattenDecl (A.Record name a d o fields) = + let (flds, tts) = unzip (flattenAvroField <$> fields) + in (A.Record name a d o flds, concat tts) + flattenDecl (A.Union _) = error "should never happen, please, file an issue" + flattenDecl t = (t, []) + flattenAvroType :: A.Type -> (A.Type, [A.Type]) + flattenAvroType (A.Record name a d o fields) = + let (flds, tts) = unzip (flattenAvroField <$> fields) + in (A.NamedType name, A.Record name a d o flds : concat tts) + flattenAvroType (A.Union (toList -> ts)) = + let (us, tts) = unzip (map flattenAvroType ts) + in (A.Union $ fromList us, concat tts) + flattenAvroType e@A.Enum {A.name} = (A.NamedType name, [e]) + flattenAvroType t = (t, []) + flattenAvroField :: A.Field -> (A.Field, [A.Type]) + flattenAvroField f = + let (t, decs) = flattenAvroType (A.fldType f) + in (f {A.fldType = t}, decs) + +typesToList :: [Type] -> Type +typesToList = foldr (\y ys -> AppT (AppT PromotedConsT y) ys) PromotedNilT + +textToStrLit :: T.Text -> Q Type +textToStrLit s = return $ LitT $ StrTyLit $ T.unpack s diff --git a/src/Mu/Quasi/Avro/Example.hs b/src/Mu/Quasi/Avro/Example.hs new file mode 100644 --- /dev/null +++ b/src/Mu/Quasi/Avro/Example.hs @@ -0,0 +1,47 @@ +{-# language CPP #-} +{-# language DataKinds #-} +{-# language QuasiQuotes #-} +{-| +Description : Examples for Avro quasi-quoters + +Look at the source code of this module. +-} +module Mu.Quasi.Avro.Example where + +import Mu.Quasi.Avro (avro, avroFile) + +type Example = [avro| +{ + "type": "record", + "name": "person", + "fields": [ + { "name": "firstName", "type": "string" }, + { "name": "lastName", "type": "string" }, + { "name": "age", "type": ["long", "null"] }, + { "name": "gender", "type": [ + { + "type": "enum", + "name": "gender", + "symbols": [ "male", "female", "nb"] + }, + "null" + ] + }, + { "name": "address", "type": { + "type": "record", + "name": "address", + "fields": [ + { "name": "postcode", "type": "string" }, + { "name": "country", "type": "string" } + ] + } + } + ] +} +|] + +#if __GHCIDE__ +type ExampleFromFile = [avroFile|adapter/avro/test/avro/example.avsc|] +#else +type ExampleFromFile = [avroFile|test/avro/example.avsc|] +#endif diff --git a/test/Avro.hs b/test/Avro.hs new file mode 100644 --- /dev/null +++ b/test/Avro.hs @@ -0,0 +1,41 @@ +{-# language DataKinds #-} +{-# language DerivingVia #-} +{-# language OverloadedStrings #-} +{-# language StandaloneDeriving #-} +{-# language TypeApplications #-} +{-# options_ghc -fno-warn-orphans #-} +module Main where + +import Data.Avro +import qualified Data.ByteString.Lazy as BS +import Data.Functor.Identity +import System.Environment + +import Mu.Adapter.Avro () +import Mu.Schema (WithSchema (..)) +import Mu.Schema.Examples + +exampleAddress :: Address +exampleAddress = Address "1111BB" "Spain" + +examplePerson1, examplePerson2 :: Person +examplePerson1 = Person "Haskellio" "Gómez" (Just 30) (Just Male) exampleAddress +examplePerson2 = Person "Cuarenta" "Siete" Nothing Nothing exampleAddress + +deriving via (WithSchema Identity ExampleSchema "person" Person) instance HasAvroSchema Person +deriving via (WithSchema Identity ExampleSchema "person" Person) instance FromAvro Person +deriving via (WithSchema Identity ExampleSchema "person" Person) instance ToAvro Person + +main :: IO () +main = do -- Obtain the filenames + [genFile, conFile] <- getArgs + -- Read the file produced by Python + putStrLn "haskell/consume" + cbs <- BS.readFile conFile + let [people] = decodeContainer @Person cbs + print people + -- Encode a couple of values + putStrLn "haskell/generate" + print [examplePerson1, examplePerson2] + gbs <- encodeContainer [[examplePerson1, examplePerson2]] + BS.writeFile genFile gbs diff --git a/test/avro/example.avsc b/test/avro/example.avsc new file mode 100644 --- /dev/null +++ b/test/avro/example.avsc @@ -0,0 +1,22 @@ +[ { "type": "enum", + "name": "gender", + "symbols" : ["male", "female", "nb"] + } +, { "type": "record" + , "name": "address" + , "fields": [ + {"name": "postcode", "type": "string"}, + {"name": "country", "type": "string"} + ] + } +,{ "type": "record", + "name": "person", + "fields": [ + {"name": "firstName", "type": "string"}, + {"name": "lastName", "type": "string"}, + {"name": "age", "type": ["long", "null"]}, + {"name": "gender", "type": ["gender", "null"]}, + {"name": "address", "type": "address"} + ] + } +]