diff --git a/CHANGELOG.md b/CHANGELOG.md --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,160 @@ # CHANGELOG for crypton +## 2.2.5 + +**RSA private-key operations check their CRT result.** A fault in one +half of the computation could hand out a signature that reveals a prime of +the key; the result is now raised back to the public exponent first, as +OpenSSL does, at a cost of two or three per cent on signing. + +* perf(rsa): verification in C at every RSA size on Apple AArch64 + [#340](https://github.com/kazu-yamamoto/crypton/pull/340) +* perf(rsa): a generated squaring at 48 limbs on AArch64 + [#341](https://github.com/kazu-yamamoto/crypton/pull/341) +* perf(rsa): verification without lists or Integer round trips + [#342](https://github.com/kazu-yamamoto/crypton/pull/342) +* ci: time out instead of waiting on a stalled package mirror + [#343](https://github.com/kazu-yamamoto/crypton/pull/343) +* fix(rsa): check the CRT result before letting it out + [#344](https://github.com/kazu-yamamoto/crypton/pull/344) +* perf(hash): one context for a one-shot hash and for MGF1 + [#345](https://github.com/kazu-yamamoto/crypton/pull/345) +* perf(rsa): RSA-3072's 24-limb square from 8-limb pieces on x86-64 + [#346](https://github.com/kazu-yamamoto/crypton/pull/346) + +## 2.2.4 + +**New module `Crypto.PubKey.RSA.Blinded`.** A `BlindedKey` carries RSA +blinding from one signature or decryption to the next. The existing +`*Safer` functions use it through a small process-wide table, and +verification keeps R^2 per modulus in another. + +* perf(rsa): keep the blinding between operations + [#335](https://github.com/kazu-yamamoto/crypton/pull/335) +* perf(rsa): a generated Montgomery multiplication for AArch64 + [#336](https://github.com/kazu-yamamoto/crypton/pull/336) +* fix: the C warnings CI reports + [#337](https://github.com/kazu-yamamoto/crypton/pull/337) +* perf(rsa): both CRT halves at once with AVX-512 IFMA + [#338](https://github.com/kazu-yamamoto/crypton/pull/338) +* perf(rsa): keep R^2 per modulus for verification + [#339](https://github.com/kazu-yamamoto/crypton/pull/339) + +## 2.2.3 + +**crypton builds on Windows again.** 2.2.2's tarball lacked a header that +only Windows includes, so it built from git but not from Hackage. With +2.2.2, `-f-support_s2n_bignum` works around it. + +* bench: RSA-3072 in the tables + [#331](https://github.com/kazu-yamamoto/crypton/pull/331) +* perf(rsa): a generated Montgomery squaring for AArch64 + [#333](https://github.com/kazu-yamamoto/crypton/pull/333) +* fix: ship the Windows wrapper for s2n-bignum + [#334](https://github.com/kazu-yamamoto/crypton/pull/334) + +## 2.2.2 + +**crypton builds on an Intel Mac again.** From 2.1.2 onwards, Apple's +assembler rejected one of the vendored RSA assembly files, so the default +configuration did not build on x86-64 macOS. With an older release, +`-f-support_s2n_bignum` works around it. + +* doc: the Ed25519 verify step was the AVX2 flag, and the tables move to 2.2.1 + [#318](https://github.com/kazu-yamamoto/crypton/pull/318) +* test: the window the exponentiation walks in, and the macro that hid it + [#319](https://github.com/kazu-yamamoto/crypton/pull/319) +* perf: the Montgomery constant in assembly, on both architectures + [#320](https://github.com/kazu-yamamoto/crypton/pull/320) +* perf: verification need not hide an exponent that is public + [#321](https://github.com/kazu-yamamoto/crypton/pull/321) +* doc: the generic AArch64 routines, measured at last + [#322](https://github.com/kazu-yamamoto/crypton/pull/322) +* doc: a VM's CPUID is not the hardware: two descriptions that said otherwise + [#323](https://github.com/kazu-yamamoto/crypton/pull/323) +* test: switches to assume what CPUID hides, for testing only + [#324](https://github.com/kazu-yamamoto/crypton/pull/324) +* perf: RSA-3072's halves through s2n-bignum on x86-64 + [#325](https://github.com/kazu-yamamoto/crypton/pull/325) +* perf: 3072- and 4096-bit secure exponentiation through s2n-bignum + [#326](https://github.com/kazu-yamamoto/crypton/pull/326) +* test: the two AES differential tests, in CI + [#327](https://github.com/kazu-yamamoto/crypton/pull/327) +* perf: s2n-bignum on x86-64 Windows + [#328](https://github.com/kazu-yamamoto/crypton/pull/328) +* fix: crypton builds on an Intel Mac again + [#329](https://github.com/kazu-yamamoto/crypton/pull/329) +* ci: link with the Command Line Tools on macOS + [#330](https://github.com/kazu-yamamoto/crypton/pull/330) + +## 2.2.1 + +**A binary built on one x86-64 machine runs on another again.** Since +2.1.8 the `-mavx2 -mbmi2` written for the ML-KEM and ML-DSA backends have +been compiling every C file in the library -- `cc-options` in a conditional +block belong to the component, not to the block -- so the compiler put AVX2 +and BMI2 into ordinary code that has no dispatch and asks nothing at +runtime: Ed25519, Argon2, Curve25519, P-256, Ed448, and the older hashes +among them. A binary built where AVX2 exists then died with `SIGILL` where +it does not, in Ed25519 rather than in anything post-quantum. + +The two backends also chose their implementation when the library was +compiled and called it unchecked. They ask the processor now, as everything +else in crypton that uses an instruction beyond the baseline already did, +and fall back to the portable C where the answer is no. AArch64 asks about +the ARMv8.4 Keccak instructions for the same reason. + +Nothing in the API changes. + +* fix: stop raising the x86-64 baseline for the whole library + [#317](https://github.com/kazu-yamamoto/crypton/pull/317) +* fix: ask the processor before using the ML-KEM and ML-DSA backends + [#317](https://github.com/kazu-yamamoto/crypton/pull/317) +* test(ct): link the processor check into the ML-KEM backend build + [#317](https://github.com/kazu-yamamoto/crypton/pull/317) + +## 2.2.0 + +`ProcessorOption` is no longer a sum of constructors. It is a number with +pattern synonyms for the names it knows, so it has `Ord` now and has lost +`Enum` and `Data`, and a `case` over the names needs a catch-all. `RDRAND` +is gone from it, and with it the `support_rdrand` flag. An AArch64 machine +reports `ARMAES` and `ARMPMULL` where it used to report `AESNI` and `PCLMUL`. + +Two more things change under a caller without the type checker saying so: +`MonadRandom IO` draws from a per-thread generator rather than reading the +system on every call, and a missing or failing entropy source raises +`EntropyError` rather than an `ErrorCall` or an `IOException`. + +* feat: take entropy from the kernel without a file descriptor + [#300](https://github.com/kazu-yamamoto/crypton/pull/300) +* feat: a ChaCha20 generator per operating system thread + [#300](https://github.com/kazu-yamamoto/crypton/pull/300) +* feat: MonadRandom IO goes through the per-thread generator + [#300](https://github.com/kazu-yamamoto/crypton/pull/300) +* fix: say that the system gave no entropy, rather than error + [#300](https://github.com/kazu-yamamoto/crypton/pull/300) +* fix: stop using RDRAND at all, since it cannot add anything + [#300](https://github.com/kazu-yamamoto/crypton/pull/300) +* feat: name every processor feature crypton dispatches on, not three of them + [#302](https://github.com/kazu-yamamoto/crypton/pull/302) +* fix: an AArch64 machine no longer reports AESNI and PCLMUL + [#302](https://github.com/kazu-yamamoto/crypton/pull/302) +* chore(rsa): deprecate `sign`, `signSafer` and `verify` in favour of them + [#305](https://github.com/kazu-yamamoto/crypton/pull/305) +* fix: compute the portable AES and GHASH without a table + [#308](https://github.com/kazu-yamamoto/crypton/pull/308) +* feat: use the AES instructions on 32-bit ARM + [#309](https://github.com/kazu-yamamoto/crypton/pull/309) +* test: skip the accelerated constant-time driver where the processor has no instructions + [#310](https://github.com/kazu-yamamoto/crypton/pull/310) +* feat: use the POWER8 AES and GHASH instructions on ppc64le + [#311](https://github.com/kazu-yamamoto/crypton/pull/311) +* test: the big-endian harness now asks about AES + [#312](https://github.com/kazu-yamamoto/crypton/pull/312) +* doc(tutorial): compile the examples, and teach the things people come for + [#313](https://github.com/kazu-yamamoto/crypton/pull/313) + ## 2.1.10 * build: ask the processor for AES-NI on every x86 system, not four of them diff --git a/Crypto/Hash.hs b/Crypto/Hash.hs --- a/Crypto/Hash.hs +++ b/Crypto/Hash.hs @@ -53,12 +53,20 @@ import qualified Crypto.Internal.ByteArray as B import qualified Data.ByteString.Lazy as L import Data.Int (Int32) +import Data.Word (Word8) +import Foreign.Marshal.Alloc (allocaBytes) import qualified Foreign.Marshal.Utils as FMU import Foreign.Ptr (Ptr, castPtr, plusPtr) -- | Hash a strict bytestring into a digest. -hash :: (ByteArrayAccess ba, HashAlgorithm a) => ba -> Digest a -hash bs = hashFinalize $ hashUpdate hashInit bs +hash :: forall ba a. (ByteArrayAccess ba, HashAlgorithm a) => ba -> Digest a +hash bs = Digest $ + allocAndFreezePrim (hashDigestSize (undefined :: a)) $ \(dig :: Ptr (Digest a)) -> + -- one context, used in place: the pure steps would copy it three times + allocaBytes (hashInternalContextSize (undefined :: a)) $ \(ctx :: Ptr (Context a)) -> do + hashInternalInit ctx + B.withByteArray bs (updateBlocks ctx (B.length bs)) + hashInternalFinalize ctx dig -- | Hash the first N bytes of a bytestring, with code path independent from N. hashPrefix @@ -92,20 +100,19 @@ hashUpdates c l | null ls = c | otherwise = Context $ B.copyAndFreeze c $ \(ctx :: Ptr (Context a)) -> - mapM_ (\b -> B.withByteArray b (processBlocks ctx (B.length b))) ls + mapM_ (\b -> B.withByteArray b (updateBlocks ctx (B.length b))) ls where ls = filter (not . B.null) l - -- process the data in 2GB chunks to fit in uint32_t and Int on 32 bit systems - processBlocks ctx bytesLeft dataPtr - | bytesLeft == 0 = return () - | otherwise = do - hashInternalUpdate ctx dataPtr (fromIntegral actuallyProcessed) - processBlocks - ctx - (bytesLeft - actuallyProcessed) - (dataPtr `plusPtr` actuallyProcessed) - where - actuallyProcessed = min bytesLeft (fromIntegral (maxBound :: Int32)) + +-- | Process the data in 2GB chunks to fit in uint32_t and Int on 32 bit systems +updateBlocks :: HashAlgorithm a => Ptr (Context a) -> Int -> Ptr Word8 -> IO () +updateBlocks ctx bytesLeft dataPtr + | bytesLeft == 0 = return () + | otherwise = do + hashInternalUpdate ctx dataPtr (fromIntegral actuallyProcessed) + updateBlocks ctx (bytesLeft - actuallyProcessed) (dataPtr `plusPtr` actuallyProcessed) + where + actuallyProcessed = min bytesLeft (fromIntegral (maxBound :: Int32)) -- | Finalize a context and return a digest. hashFinalize diff --git a/Crypto/Number/ModArithmetic.hs b/Crypto/Number/ModArithmetic.hs --- a/Crypto/Number/ModArithmetic.hs +++ b/Crypto/Number/ModArithmetic.hs @@ -43,6 +43,7 @@ import Crypto.Internal.Compat (unsafeDoIO) import Crypto.Number.Basic import Crypto.Number.Compat +import Crypto.Number.ModArithmetic.Pub (expPub) import qualified Crypto.Number.Serialize.Internal as Internal import Data.Memory.PtrMethods (memSet) import Data.Word (Word32, Word8) @@ -141,6 +142,7 @@ -- what GMP's own secure exponentiation lets slip !eLen = 8 * ((numBytes e + 7) `div` 8) + foreign import ccall safe "crypton_powm_sec" c_powm_sec :: Ptr Word8 @@ -158,6 +160,14 @@ -- -- Use this function when all the parameters are public, -- otherwise 'expSafe' should be preferred. +-- +-- A short exponent on an odd modulus -- RSA verification, where the exponent +-- is 65537 and comes out of the public key -- is walked a bit at a time in +-- C, squaring at each step and multiplying where a bit is set, which reads +-- the exponent and branches on it and is why this is not 'expSafe'. That +-- path is taken only where it is the faster of the two, which the C decides; +-- everywhere else, and for every longer exponent, this is GMP's windowed +-- exponentiation as it has always been. expFast :: Integer -- ^ base @@ -167,7 +177,13 @@ -- ^ modulo -> Integer -- ^ result -expFast b e m = gmpPowModInteger b e m `onGmpUnsupported` exponentiation b e m +expFast b e m + | odd m && m > 1 && e > 0 = case expPub (b `mod` m) e m of + Just r -> r + Nothing -> slow + | otherwise = slow + where + slow = gmpPowModInteger b e m `onGmpUnsupported` exponentiation b e m -- | @exponentiation@ computes modular exponentiation as /b^e mod m/ -- using repetitive squaring. diff --git a/Crypto/Number/ModArithmetic/Pub.hs b/Crypto/Number/ModArithmetic/Pub.hs new file mode 100644 --- /dev/null +++ b/Crypto/Number/ModArithmetic/Pub.hs @@ -0,0 +1,158 @@ +{-# LANGUAGE BangPatterns #-} + +-- | +-- Module : Crypto.Number.ModArithmetic.Pub +-- License : BSD-style +-- Stability : experimental +-- Portability : Good +-- +-- The C exponentiation for an exponent that is not a secret, and the R^2 +-- it keeps per modulus. Internal: 'expPub' is behind 'expFast', and +-- 'expPubBytes' behind RSA's public-key operation. +module Crypto.Number.ModArithmetic.Pub ( + expPub, + expPubBytes, +) where + +import Crypto.Internal.ByteArray ( + ByteArray, + ByteArrayAccess, + Bytes, + allocRet, + withByteArray, + ) +import qualified Crypto.Internal.ByteArray as B +import Crypto.Internal.Compat (unsafeDoIO) +import Crypto.Number.Basic +import qualified Crypto.Number.Serialize.Internal as Internal +import Data.IORef (IORef, atomicModifyIORef', newIORef) +import Data.Word (Word32, Word8) +import Foreign.C.Types (CInt (..)) +import Foreign.Marshal.Alloc (allocaBytes) +import Foreign.Ptr (Ptr, plusPtr) +import System.IO.Unsafe (unsafePerformIO) + +-- | The windowless exponentiation in C, for an exponent that is not a +-- secret. 'Nothing' when the C declines, which it does on every machine +-- where what the caller already has is faster -- so this is an +-- optimisation and never the only path. The base has to be reduced and +-- the exponent positive, which the caller sees to. +expPub :: Integer -> Integer -> Integer -> Maybe Integer +expPub b e m = unsafeDoIO $ do + known <- r2For m + case known of + Nothing -> return Nothing + Just (r2, modu) -> + allocaBytes (sum widths) $ \start -> case scanl plusPtr start widths of + (out : base : expo : _) -> do + _ <- Internal.i2ospOf b base mLen + _ <- Internal.i2ospOf e expo eLen + r <- withByteArray modu $ \pm -> withByteArray r2 $ \pr2 -> + c_powm_pub_r2 + out + base + (fromIntegral mLen) + expo + (fromIntegral eLen) + pm + (fromIntegral mLen) + pr2 + if r == 0 + then do + !v <- Internal.os2ip out mLen + return (Just v) + else return Nothing + _ -> return Nothing + where + !mLen = numBytes m + !eLen = numBytes e + widths = [mLen, mLen, eLen] + +-- | The same with the base and the answer as big-endian bytes, as long as +-- the modulus: what RSA verification has in hand, so neither goes through +-- an 'Integer'. 'Nothing' also when the base is not below the modulus. +expPubBytes + :: (ByteArrayAccess bi, ByteArray bo) => Integer -> Integer -> bi -> Maybe bo +expPubBytes e m b + | e <= 0 || m <= 1 || even m || B.length b /= mLen = Nothing + | otherwise = unsafeDoIO $ do + known <- r2For m + case known of + Nothing -> return Nothing + Just (r2, modu) + | not (below b modu) -> return Nothing + | otherwise -> allocaBytes eLen $ \expo -> do + _ <- Internal.i2ospOf e expo eLen + (r, out) <- allocRet mLen $ \pout -> + withByteArray b $ \pb -> withByteArray modu $ \pm -> + withByteArray r2 $ \pr2 -> + c_powm_pub_r2 + pout + pb + (fromIntegral mLen) + expo + (fromIntegral eLen) + pm + (fromIntegral mLen) + pr2 + return (if r == 0 then Just out else Nothing) + where + !mLen = numBytes m + !eLen = numBytes e + +-- big-endian numbers of one length: the first byte that differs decides +below :: (ByteArrayAccess a, ByteArrayAccess b) => a -> b -> Bool +below x y = go 0 + where + n = B.length x + go i + | i >= n = False + | otherwise = case compare (B.index x i) (B.index y i) of + EQ -> go (i + 1) + o -> o == LT + +-- | R^2 mod m for the public path, and m in bytes, kept for the last few +-- moduli, since a program verifies under the same keys again and again. +-- 'Nothing' is kept too: it means the C declines this modulus, which is +-- then not asked again. +r2For :: Integer -> IO (Maybe (Bytes, Bytes)) +r2For m = do + found <- atomicModifyIORef' r2Table $ \ks -> case break ((== m) . fst) ks of + (before, hit : after) -> (hit : before ++ after, Just (snd hit)) + _ -> (ks, Nothing) + case found of + Just r2 -> return r2 + Nothing -> do + let mLen = numBytes m + (r2, modu) <- allocRet mLen $ \modu -> do + _ <- Internal.i2ospOf m modu mLen + (r, out) <- allocRet mLen $ \pout -> c_powm_r2 pout modu (fromIntegral mLen) + return (if r == 0 then Just out else Nothing) + let kept = fmap (\x -> (x, modu)) r2 + atomicModifyIORef' r2Table $ \ks -> (take r2TableSize ((m, kept) : ks), ()) + return kept + +r2TableSize :: Int +r2TableSize = 8 + +r2Table :: IORef [(Integer, Maybe (Bytes, Bytes))] +r2Table = unsafePerformIO (newIORef []) +{-# NOINLINE r2Table #-} + +-- safe, as the secure one is: the call is tens of microseconds at RSA +-- sizes, and the hundred nanoseconds this costs over an unsafe call is +-- nothing beside holding a capability for that long in a server +foreign import ccall safe "crypton_powm_pub_r2" + c_powm_pub_r2 + :: Ptr Word8 + -> Ptr Word8 + -> Word32 + -> Ptr Word8 + -> Word32 + -> Ptr Word8 + -> Word32 + -> Ptr Word8 + -> IO CInt + +foreign import ccall unsafe "crypton_powm_r2" + c_powm_r2 :: Ptr Word8 -> Ptr Word8 -> Word32 -> IO CInt diff --git a/Crypto/PubKey/MaskGenFunction.hs b/Crypto/PubKey/MaskGenFunction.hs --- a/Crypto/PubKey/MaskGenFunction.hs +++ b/Crypto/PubKey/MaskGenFunction.hs @@ -1,4 +1,4 @@ -{-# LANGUAGE BangPatterns #-} +{-# LANGUAGE ScopedTypeVariables #-} -- | -- Module : Crypto.PubKey.MaskGenFunction @@ -11,10 +11,17 @@ mgf1, ) where -import Crypto.Hash -import Crypto.Internal.ByteArray (ByteArray, ByteArrayAccess, Bytes) +import Control.Monad (forM_) +import Crypto.Hash (Context, Digest) +import Crypto.Hash.IO (HashAlgorithm (..)) +import Crypto.Internal.ByteArray (ByteArray, ByteArrayAccess) import qualified Crypto.Internal.ByteArray as B -import Crypto.Number.Serialize (i2ospOf_) +import Data.Bits (shiftR) +import Data.Word (Word8) +import Foreign.Marshal.Alloc (allocaBytes) +import Foreign.Marshal.Utils (copyBytes) +import Foreign.Ptr (Ptr, castPtr, plusPtr) +import Foreign.Storable (pokeByteOff) -- | Represent a mask generation algorithm type MaskGenAlgorithm seed output = @@ -26,20 +33,33 @@ -- | Mask generation algorithm MGF1 mgf1 - :: (ByteArrayAccess seed, ByteArray output, HashAlgorithm hashAlg) + :: forall seed output hashAlg + . (ByteArrayAccess seed, ByteArray output, HashAlgorithm hashAlg) => hashAlg -> seed -> Int -> output -mgf1 hashAlg seed len = - let !seededCtx = hashUpdate (hashInitWith hashAlg) seed - in B.take len $ - B.concat $ - map (hashCounter seededCtx) [0 .. fromIntegral (maxCounter - 1)] +mgf1 hashAlg seed len + | len <= 0 = B.empty + | otherwise = B.allocAndFreeze len $ \out -> + -- the seeded context is kept, and copied for each block rather than + -- rebuilt; each block's digest is written straight into the output + allocaBytes ctxSize $ \(seeded :: Ptr (Context hashAlg)) -> + allocaBytes ctxSize $ \(ctx :: Ptr (Context hashAlg)) -> + allocaBytes digestLen $ \(dig :: Ptr (Digest hashAlg)) -> + allocaBytes 4 $ \(cnt :: Ptr Word8) -> do + hashInternalInit seeded + B.withByteArray seed $ \p -> + hashInternalUpdate seeded p (fromIntegral (B.length seed)) + forM_ [0 .. maxCounter - 1] $ \i -> do + copyBytes ctx seeded ctxSize + forM_ [0 .. 3] $ \k -> + pokeByteOff cnt k (fromIntegral (i `shiftR` (8 * (3 - k))) :: Word8) + hashInternalUpdate ctx cnt 4 + hashInternalFinalize ctx dig + let off = i * digestLen + copyBytes (out `plusPtr` off) (castPtr dig) (min digestLen (len - off)) where + ctxSize = hashInternalContextSize hashAlg digestLen = hashDigestSize hashAlg - (chunks, left) = len `divMod` digestLen - maxCounter = if left > 0 then chunks + 1 else chunks - - hashCounter :: HashAlgorithm a => Context a -> Integer -> Digest a - hashCounter ctx counter = hashFinalize $ hashUpdate ctx (i2ospOf_ 4 counter :: Bytes) + maxCounter = (len + digestLen - 1) `div` digestLen diff --git a/Crypto/PubKey/RSA/Blinded.hs b/Crypto/PubKey/RSA/Blinded.hs new file mode 100644 --- /dev/null +++ b/Crypto/PubKey/RSA/Blinded.hs @@ -0,0 +1,29 @@ +-- | +-- Module : Crypto.PubKey.RSA.Blinded +-- License : BSD-style +-- Stability : experimental +-- Portability : Good +-- +-- An RSA private key that keeps its blinding between operations. +-- +-- Blinding hides the private-key operation's input from timing: it is +-- multiplied by @r^e@ for a random @r@ first, and the result by @r^-1@ +-- after. Drawing a new @r@ costs a modular inverse and an exponentiation, +-- about a tenth of an RSA-2048 signature. A t'BlindedKey' draws one and +-- squares the pair for each use after it, which is what OpenSSL does, and +-- draws a new one every 32 uses, from the 'Crypto.Random.Types.MonadRandom' +-- the operation runs in. +-- +-- The key is threaded through the operations as a hash context is: each +-- function in "Crypto.PubKey.RSA.PKCS15", "Crypto.PubKey.RSA.PSS" and +-- "Crypto.PubKey.RSA.OAEP" whose name ends in @Blinded@ returns the key for +-- the next one. The @Safer@ functions there keep a t'BlindedKey' per key +-- themselves, for the last few keys used, so they get the same saving +-- without one being passed. +module Crypto.PubKey.RSA.Blinded ( + BlindedKey, + blindedKey, + blindedPrivateKey, +) where + +import Crypto.PubKey.RSA.Blinded.Internal diff --git a/Crypto/PubKey/RSA/Blinded/Internal.hs b/Crypto/PubKey/RSA/Blinded/Internal.hs new file mode 100644 --- /dev/null +++ b/Crypto/PubKey/RSA/Blinded/Internal.hs @@ -0,0 +1,136 @@ +{-# LANGUAGE BangPatterns #-} +{-# LANGUAGE ScopedTypeVariables #-} +{-# OPTIONS_GHC -fno-cse -fno-full-laziness #-} + +-- | +-- Module : Crypto.PubKey.RSA.Blinded.Internal +-- License : BSD-style +-- Stability : experimental +-- Portability : Good +-- +-- A private key with a blinding pair that is reused by squaring, as OpenSSL +-- does, rather than generated afresh for every operation. The key is a value +-- threaded through the operations, as a hash 'Crypto.Hash.Context' is. +module Crypto.PubKey.RSA.Blinded.Internal ( + BlindedKey (..), + blindedKey, + dpBlinded, + Draw, + drawFor, + withBlindingFor, + dpWithPair, +) where + +import Data.IORef +import System.IO.Unsafe (unsafePerformIO) + +import Crypto.Internal.ByteArray (ByteArray, ScrubbedBytes) +import Crypto.Number.Generate (generateMax) +import Crypto.Number.ModArithmetic (expFast, inverseCoprimes) +import Crypto.Number.Serialize (i2ospOf_, os2ip) +import Crypto.PubKey.RSA.Prim (dp) +import Crypto.PubKey.RSA.Types +import Crypto.Random.Types + +-- | A private key together with the blinding its next operation uses. +-- +-- Each operation multiplies the input by @r^e@ and the output by @r^-1@, and +-- returns the key with both squared for the next one, so a pair costs two +-- multiplications after the first. A new @r@ is drawn every 32 operations. +-- Use the key an operation returns: handing the same one to two operations +-- gives both the same pair. +data BlindedKey = BlindedKey + { blindedPrivateKey :: PrivateKey + -- ^ the key it was made from + , blindedPair :: Pair + } + +-- | @r^e@ and @r^-1@ modulo n, and how many more uses they have after this. +data Pair = Pair !Int !Integer !Integer | Spent + +-- | Uses of one @r@ before another is drawn, OpenSSL's number. +refresh :: Int +refresh = 32 + +-- | A t'BlindedKey' for a private key. The first pair is drawn when it is +-- first used. +blindedKey :: PrivateKey -> BlindedKey +blindedKey pk = BlindedKey pk Spent + +-- | The random numbers a new pair is made from: drawing them is cheap, and +-- making the pair, an inverse and an exponentiation, is done only if needed. +data Draw = Draw !Integer !Integer + +-- | Draw for a key, as 'Crypto.PubKey.RSA.generateBlinder' does. +drawFor :: forall m. MonadRandom m => PrivateKey -> m Draw +drawFor pk = do + r <- generateMax (private_n pk) + u <- os2ip <$> (getRandomBytes 16 :: m ScrubbedBytes) + return (Draw r u) + +-- | @r^e@ and @r^-1@. The inverse is taken of @r*u@ and multiplied by @u@, +-- so the extended Euclidean algorithm never sees @r@, as in +-- 'Crypto.PubKey.RSA.generateBlinder'. +makePair :: PrivateKey -> Draw -> (Integer, Integer) +makePair pk (Draw r u) = + (expFast r (private_e pk) n, (inverseCoprimes ((r * u) `mod` n) n * u) `mod` n) + where + n = private_n pk + +-- | The pair for this operation, and the key holding the next one. The +-- draw is used only when a new pair is due. +takePair :: Draw -> BlindedKey -> ((Integer, Integer), BlindedKey) +takePair d (BlindedKey pk pair) = case pair of + Pair k a ai -> ((a, ai), BlindedKey pk (next k a ai)) + Spent -> case makePair pk d of + (a, ai) -> ((a, ai), BlindedKey pk (next refresh a ai)) + where + n = private_n pk + next k a ai + | k > 1 = Pair (k - 1) ((a * a) `mod` n) ((ai * ai) `mod` n) + | otherwise = Spent + +-- | The decryption primitive, blinded with a given pair. +dpWithPair :: ByteArray ba => (Integer, Integer) -> PrivateKey -> ba -> ba +dpWithPair (a, ai) pk c = i2ospOf_ k ((ai * os2ip m) `mod` n) + where + n = private_n pk + k = private_size pk + m = dp Nothing pk (i2ospOf_ k ((a * os2ip c) `mod` n) `asTypeOf` c) + +-- | The decryption primitive, blinded with the key's pair, and the key for +-- the next operation. Randomness is drawn only when a new pair is due. +dpBlinded + :: (MonadRandom m, ByteArray ba) => BlindedKey -> ba -> m (ba, BlindedKey) +dpBlinded bk c = do + d <- case blindedPair bk of + Pair {} -> return (Draw 0 0) + Spent -> drawFor (blindedPrivateKey bk) + let (pair, bk') = takePair d bk + return (dpWithPair pair (blindedPrivateKey bk) c, bk') + +-- | Run an operation with the pair kept for a key, for the functions that +-- take a bare t'PrivateKey'. One t'BlindedKey' is kept per public key, for +-- the last few keys used; taking its pair and leaving the next is atomic, so +-- no two operations get the same pair. The caller draws every time, since +-- whether a new pair is due is known only in here. +withBlindingFor :: PrivateKey -> Draw -> ((Integer, Integer) -> a) -> a +withBlindingFor pk d f = unsafePerformIO $ do + pair <- atomicModifyIORef' table step + return (f pair) + where + pub = private_pub pk + step ks = case break ((== pub) . private_pub . blindedPrivateKey) ks of + (before, bk : after) -> found bk (before ++ after) + _ -> found (blindedKey pk) ks + found bk rest = case takePair d bk of + (pair@(!_, !_), bk') -> (take tableSize (bk' : rest), pair) +{-# NOINLINE withBlindingFor #-} + +-- | How many keys 'withBlindingFor' remembers. +tableSize :: Int +tableSize = 4 + +table :: IORef [BlindedKey] +table = unsafePerformIO (newIORef []) +{-# NOINLINE table #-} diff --git a/Crypto/PubKey/RSA/OAEP.hs b/Crypto/PubKey/RSA/OAEP.hs --- a/Crypto/PubKey/RSA/OAEP.hs +++ b/Crypto/PubKey/RSA/OAEP.hs @@ -18,13 +18,14 @@ -- * OAEP decryption decrypt, decryptSafer, + decryptBlinded, ) where import Crypto.Hash import Crypto.Number.Serialize (os2ip) import Crypto.PubKey.Internal (and') import Crypto.PubKey.MaskGenFunction -import Crypto.PubKey.RSA (generateBlinder) +import Crypto.PubKey.RSA.Blinded.Internal import Crypto.PubKey.RSA.Prim import Crypto.PubKey.RSA.Types import Crypto.Random.Types @@ -212,7 +213,8 @@ k = private_size pk hashLen = hashDigestSize (oaepHash oaep) --- | Decrypt a ciphertext using OAEP and by automatically generating a blinder. +-- | Decrypt a ciphertext using OAEP, blinded with the pair kept for this key; +-- see "Crypto.PubKey.RSA.Blinded". decryptSafer :: (HashAlgorithm hash, MonadRandom m) => OAEPParams hash ByteString ByteString @@ -222,6 +224,34 @@ -> ByteString -- ^ Cipher text -> m (Either Error ByteString) -decryptSafer oaep pk cipher = do - blinder <- generateBlinder (private_n pk) - return (decrypt (Just blinder) oaep pk cipher) +decryptSafer oaep pk cipher + | B.length cipher /= k = return (Left MessageSizeIncorrect) + | os2ip cipher >= private_n pk = return (Left MessageSizeIncorrect) + | k < 2 * hashLen + 2 = return (Left InvalidParameters) + | otherwise = do + d <- drawFor pk + return $ withBlindingFor pk d $ \pair -> unpad oaep k (dpWithPair pair pk cipher) + where + k = private_size pk + hashLen = hashDigestSize (oaepHash oaep) + +-- | 'decrypt' with the blinding a t'BlindedKey' keeps, and the key for the +-- next operation. +decryptBlinded + :: (HashAlgorithm hash, MonadRandom m) + => OAEPParams hash ByteString ByteString + -> BlindedKey + -> ByteString + -- ^ Cipher text + -> m (Either Error ByteString, BlindedKey) +decryptBlinded oaep bk cipher + | B.length cipher /= k = return (Left MessageSizeIncorrect, bk) + | os2ip cipher >= private_n pk = return (Left MessageSizeIncorrect, bk) + | k < 2 * hashLen + 2 = return (Left InvalidParameters, bk) + | otherwise = do + (m, bk') <- dpBlinded bk cipher + return (unpad oaep k m, bk') + where + pk = blindedPrivateKey bk + k = private_size pk + hashLen = hashDigestSize (oaepHash oaep) diff --git a/Crypto/PubKey/RSA/PKCS15.hs b/Crypto/PubKey/RSA/PKCS15.hs --- a/Crypto/PubKey/RSA/PKCS15.hs +++ b/Crypto/PubKey/RSA/PKCS15.hs @@ -20,6 +20,11 @@ signDigestInfo, signSaferDigestInfo, + -- * Private key operations with a kept blinding + decryptBlinded, + signDigestBlinded, + signDigestInfoBlinded, + -- * Public key operations encrypt, verify, @@ -33,7 +38,7 @@ import Crypto.Hash import Crypto.Number.Serialize (os2ip) import Crypto.PubKey.Internal (and') -import Crypto.PubKey.RSA (generateBlinder) +import Crypto.PubKey.RSA.Blinded.Internal import Crypto.PubKey.RSA.Prim import Crypto.PubKey.RSA.Types import Crypto.Random.Types @@ -410,10 +415,10 @@ :: ByteArray signature => Int -> signature -> Either Error signature padSignature klen signature | klen < siglen + 11 = Left SignatureTooLong - | otherwise = Right (B.pack padding `B.append` signature) + | otherwise = + Right (B.concat [B.pack [0, 1], B.replicate (klen - siglen - 3) 0xff, B.pack [0], signature]) where siglen = B.length signature - padding = 0 : 1 : (replicate (klen - siglen - 3) 0xff ++ [0]) -- | Try to remove a standard PKCS1.5 encryption padding. -- @@ -486,7 +491,8 @@ -- "convert" must be apply to "c". | otherwise = unpad $ dp blinder pk $ B.convert c --- | decrypt message using the private key and by automatically generating a blinder. +-- | decrypt message using the private key, blinded with the pair kept for +-- this key; see "Crypto.PubKey.RSA.Blinded". decryptSafer :: (MonadRandom m, ByteArray ba) => PrivateKey @@ -494,10 +500,30 @@ -> ByteString -- ^ cipher text -> m (Either Error ba) -decryptSafer pk b = do - blinder <- generateBlinder (private_n pk) - return (decrypt (Just blinder) pk b) +decryptSafer pk b + | B.length b /= private_size pk = return (Left MessageSizeIncorrect) + | os2ip b >= private_n pk = return (Left MessageSizeIncorrect) + | otherwise = do + d <- drawFor pk + return $ withBlindingFor pk d $ \pair -> unpad (dpWithPair pair pk (B.convert b)) +-- | 'decrypt' with the blinding a t'BlindedKey' keeps, and the key for the +-- next operation. +decryptBlinded + :: (MonadRandom m, ByteArray ba) + => BlindedKey + -> ByteString + -- ^ cipher text + -> m (Either Error ba, BlindedKey) +decryptBlinded bk c + | B.length c /= private_size pk = return (Left MessageSizeIncorrect, bk) + | os2ip c >= private_n pk = return (Left MessageSizeIncorrect, bk) + | otherwise = do + (m, bk') <- dpBlinded bk (B.convert c) + return (unpad m, bk') + where + pk = blindedPrivateKey bk + -- | encrypt a bytestring using the public key. -- -- The message needs to be smaller than the key size - 11. @@ -512,6 +538,12 @@ -- | sign message using private key, a hash and its ASN1 description -- +-- __Deprecated.__ Use 'signDigest', or 'signDigestInfo' where this would +-- have been given 'Nothing'. The @Maybe hashAlg@ argument says both \"hash +-- it with this\" and \"do not hash it\", and down the first path the value +-- inside the 'Just' is never read: it is there to fix the type, which +-- leaves a caller that has only a type with nothing to pass. +-- -- The blinder is optional and 'Nothing' is accepted, but see t'Blinder' for -- what it covers and when leaving it out is a decision rather than a default. -- 'signSafer' generates one for you. @@ -527,8 +559,15 @@ -- ^ message to sign -> Either Error ByteString sign blinder hashDescr pk m = dp blinder pk `fmap` makeSignature hashDescr (private_size pk) m +{-# DEPRECATED + sign + "Use signDigest, or signDigestInfo when the message is already a DigestInfo" + #-} --- | sign message using the private key and by automatically generating a blinder. +-- | sign message using the private key, blinded with the pair kept for this +-- key; see "Crypto.PubKey.RSA.Blinded". +-- +-- __Deprecated.__ Use 'signSaferDigest', or 'signSaferDigestInfo'. signSafer :: (HashAlgorithmASN1 hashAlg, MonadRandom m) => Maybe hashAlg @@ -539,11 +578,18 @@ -- ^ message to sign -> m (Either Error ByteString) signSafer hashAlg pk m = do - blinder <- generateBlinder (private_n pk) - return (sign (Just blinder) hashAlg pk m) + d <- drawFor pk + return $ signWithBlindingFor pk d (makeSignature hashAlg (private_size pk) m) +{-# DEPRECATED + signSafer + "Use signSaferDigest, or signSaferDigestInfo when the message is already a DigestInfo" + #-} -- | verify message with the signed message -- +-- __Deprecated.__ Use 'verifyDigest', or 'verifyDigestInfo' where this +-- would have been given 'Nothing'. +-- -- Following RFC 8017, the signature is rejected unless it is exactly as long -- as the modulus (section 8.2.2, step 1) and its integer representative is -- below the modulus (section 5.2.2, step 1). Verification works by @@ -562,6 +608,10 @@ -> Bool verify hashAlg pk m sm = verifyEncoded pk (makeSignature hashAlg (public_size pk) m) sm +{-# DEPRECATED + verify + "Use verifyDigest, or verifyDigestInfo when the message is already a DigestInfo" + #-} -- | The two checks of RFC 8017 and the comparison, shared by the three -- verification entry points. The expected encoding is a thunk and is @@ -609,8 +659,8 @@ signDigest blinder pk digest = dp blinder pk `fmap` padSignature (private_size pk) (hashDigestASN1 digest) --- | 'signDigest' with a blinder generated for the occasion, as 'signSafer' --- is to 'sign'. +-- | 'signDigest' blinded with the pair kept for this key, as 'signSafer' is +-- to 'sign'; see "Crypto.PubKey.RSA.Blinded". signSaferDigest :: (HashAlgorithmASN1 hashAlg, MonadRandom m) => PrivateKey @@ -619,9 +669,22 @@ -- ^ digest of the message to sign -> m (Either Error ByteString) signSaferDigest pk digest = do - blinder <- generateBlinder (private_n pk) - return (signDigest (Just blinder) pk digest) + d <- drawFor pk + return $ + signWithBlindingFor pk d (padSignature (private_size pk) (hashDigestASN1 digest)) +-- | 'signDigest' with the blinding a t'BlindedKey' keeps, and the key for the +-- next operation. +signDigestBlinded + :: (HashAlgorithmASN1 hashAlg, MonadRandom m) + => BlindedKey + -> Digest hashAlg + -- ^ digest of the message to sign + -> m (Either Error ByteString, BlindedKey) +signDigestBlinded bk digest = + signBlindedEncoded bk $ + padSignature (private_size (blindedPrivateKey bk)) (hashDigestASN1 digest) + -- | Sign something that is already a DigestInfo, the ASN.1 structure -- naming a hash algorithm and carrying a digest under it. -- @@ -643,7 +706,7 @@ signDigestInfo blinder pk di = dp blinder pk `fmap` padSignature (private_size pk) di --- | 'signDigestInfo' with a blinder generated for the occasion. +-- | 'signDigestInfo' blinded with the pair kept for this key. signSaferDigestInfo :: MonadRandom m => PrivateKey @@ -652,8 +715,37 @@ -- ^ a DigestInfo, encoded -> m (Either Error ByteString) signSaferDigestInfo pk di = do - blinder <- generateBlinder (private_n pk) - return (signDigestInfo (Just blinder) pk di) + d <- drawFor pk + return $ signWithBlindingFor pk d (padSignature (private_size pk) di) + +-- | 'signDigestInfo' with the blinding a t'BlindedKey' keeps, and the key for +-- the next operation. +signDigestInfoBlinded + :: MonadRandom m + => BlindedKey + -> ByteString + -- ^ a DigestInfo, encoded + -> m (Either Error ByteString, BlindedKey) +signDigestInfoBlinded bk di = + signBlindedEncoded bk $ padSignature (private_size (blindedPrivateKey bk)) di + +-- | The private-key operation on an encoded signature, if there is one; the +-- key is used up only if there is. +signBlindedEncoded + :: MonadRandom m + => BlindedKey + -> Either Error ByteString + -> m (Either Error ByteString, BlindedKey) +signBlindedEncoded bk (Left err) = return (Left err, bk) +signBlindedEncoded bk (Right em) = do + (s, bk') <- dpBlinded bk em + return (Right s, bk') + +-- | The same, with the pair the @Safer@ functions keep for this key. +signWithBlindingFor + :: PrivateKey -> Draw -> Either Error ByteString -> Either Error ByteString +signWithBlindingFor _ _ (Left err) = Left err +signWithBlindingFor pk d (Right em) = Right (withBlindingFor pk d (\pair -> dpWithPair pair pk em)) -- | Verify a signature over a digest. The checks are 'verify's. verifyDigest diff --git a/Crypto/PubKey/RSA/PSS.hs b/Crypto/PubKey/RSA/PSS.hs --- a/Crypto/PubKey/RSA/PSS.hs +++ b/Crypto/PubKey/RSA/PSS.hs @@ -16,6 +16,8 @@ signDigest, signSafer, signDigestSafer, + signBlinded, + signDigestBlinded, verify, verifyDigest, ) where @@ -24,15 +26,15 @@ import Crypto.Number.Basic (numBits) import Crypto.Number.Serialize (os2ip) import Crypto.PubKey.MaskGenFunction -import Crypto.PubKey.RSA (generateBlinder) +import Crypto.PubKey.RSA.Blinded.Internal import Crypto.PubKey.RSA.Prim import Crypto.PubKey.RSA.Types import Crypto.Random.Types -import Data.Bits (complement, shiftR, xor, (.&.)) +import Data.Bits (complement, shiftR, (.&.)) import Data.Word import Crypto.Internal.ByteArray (ByteArray, ByteArrayAccess) -import qualified Crypto.Internal.ByteArray as B (convert, eq) +import qualified Crypto.Internal.ByteArray as B (bxor, convert, eq) import Data.ByteString (ByteString) import qualified Data.ByteString as B @@ -85,9 +87,20 @@ -> Digest hash -- ^ Message digest -> Either Error ByteString -signDigestWithSalt salt blinder params pk digest +signDigestWithSalt salt blinder params pk digest = + dp blinder pk <$> encodeDigest salt params pk digest + +-- | The encoded message the private-key operation is applied to. +encodeDigest + :: HashAlgorithm hash + => ByteString + -> PSSParams hash ByteString ByteString + -> PrivateKey + -> Digest hash + -> Either Error ByteString +encodeDigest salt params pk digest | emLen < hashLen + saltLen + 2 = Left InvalidParameters - | otherwise = Right $ dp blinder pk em + | otherwise = Right em where k = private_size pk emLen = if emTruncate pubBits then k - 1 else k @@ -100,7 +113,7 @@ h = B.convert $ hashWith (pssHash params) m' db = B.concat [B.replicate (dbLen - saltLen - 1) 0, B.singleton 1, salt] dbmask = pssMaskGenAlg params h dbLen - maskedDB = B.pack $ normalizeToKeySize pubBits $ B.zipWith xor db dbmask + maskedDB = normalizeToKeySize pubBits (B.bxor db dbmask :: ByteString) em = B.concat [maskedDB, h, B.singleton (pssTrailerField params)] -- | Sign using the PSS parameters and the salt explicitely passed as parameters. @@ -164,7 +177,8 @@ salt <- getRandomBytes (pssSaltLength params) return (signDigestWithSalt salt blinder params pk digest) --- | Sign using the PSS Parameters and an automatically generated blinder. +-- | Sign using the PSS Parameters, blinded with the pair kept for this key; +-- see "Crypto.PubKey.RSA.Blinded". signSafer :: (HashAlgorithm hash, MonadRandom m) => PSSParams hash ByteString ByteString @@ -174,11 +188,10 @@ -> ByteString -- ^ message to sign -> m (Either Error ByteString) -signSafer params pk m = do - blinder <- generateBlinder (private_n pk) - sign (Just blinder) params pk m +signSafer params pk m = signDigestSafer params pk (hashWith (pssHash params) m) --- | Sign using the PSS Parameters and an automatically generated blinder. +-- | Sign using the PSS Parameters, blinded with the pair kept for this key; +-- see "Crypto.PubKey.RSA.Blinded". signDigestSafer :: (HashAlgorithm hash, MonadRandom m) => PSSParams hash ByteString ByteString @@ -189,9 +202,40 @@ -- ^ message digst -> m (Either Error ByteString) signDigestSafer params pk digest = do - blinder <- generateBlinder (private_n pk) - signDigest (Just blinder) params pk digest + salt <- getRandomBytes (pssSaltLength params) + d <- drawFor pk + return $ case encodeDigest salt params pk digest of + Left err -> Left err + Right em -> Right (withBlindingFor pk d (\pair -> dpWithPair pair pk em)) +-- | 'signSafer' with the blinding a t'BlindedKey' keeps, and the key for the +-- next operation. +signBlinded + :: (HashAlgorithm hash, MonadRandom m) + => PSSParams hash ByteString ByteString + -> BlindedKey + -> ByteString + -- ^ message to sign + -> m (Either Error ByteString, BlindedKey) +signBlinded params bk m = signDigestBlinded params bk (hashWith (pssHash params) m) + +-- | 'signDigestSafer' with the blinding a t'BlindedKey' keeps, and the key +-- for the next operation. +signDigestBlinded + :: (HashAlgorithm hash, MonadRandom m) + => PSSParams hash ByteString ByteString + -> BlindedKey + -> Digest hash + -- ^ message digest + -> m (Either Error ByteString, BlindedKey) +signDigestBlinded params bk digest = do + salt <- getRandomBytes (pssSaltLength params) + case encodeDigest salt params (blindedPrivateKey bk) digest of + Left err -> return (Left err, bk) + Right em -> do + (s, bk') <- dpBlinded bk em + return (Right s, bk') + -- | Verify a signature using the PSS Parameters verify :: HashAlgorithm hash @@ -259,7 +303,7 @@ maskedDB = B.take dbLen em h = B.take hashLen $ B.drop (B.length maskedDB) em dbmask = pssMaskGenAlg params h dbLen - db = B.pack $ normalizeToKeySize pubBits $ B.zipWith xor maskedDB dbmask + db = normalizeToKeySize pubBits (B.bxor maskedDB dbmask :: ByteString) (ps0, z) = B.break (== 1) db (b1, salt) = B.splitAt 1 z m' = B.concat [B.replicate 8 0, mHash, salt] @@ -269,9 +313,10 @@ emTruncate :: Int -> Bool emTruncate bits = ((bits - 1) .&. 0x7) == 0 -normalizeToKeySize :: Int -> [Word8] -> [Word8] -normalizeToKeySize _ [] = [] -- very unlikely -normalizeToKeySize bits (x : xs) = x .&. normalizeMask bits : xs +normalizeToKeySize :: Int -> ByteString -> ByteString +normalizeToKeySize bits x = case B.uncons x of + Nothing -> x -- very unlikely + Just (h, t) -> B.cons (h .&. normalizeMask bits) t -- | The bits of the leftmost octet that belong to the encoding: the low -- @emBits `mod` 8@ of them, or all eight when that is zero. Its complement diff --git a/Crypto/PubKey/RSA/Prim.hs b/Crypto/PubKey/RSA/Prim.hs --- a/Crypto/PubKey/RSA/Prim.hs +++ b/Crypto/PubKey/RSA/Prim.hs @@ -1,3 +1,5 @@ +{-# LANGUAGE BangPatterns #-} + -- | -- Module : Crypto.PubKey.RSA.Prim -- License : BSD-style @@ -12,9 +14,19 @@ ep, ) where +import Data.Memory.PtrMethods (memSet) +import Data.Word (Word32, Word8) +import Foreign.C.Types (CInt (..)) +import Foreign.Marshal.Alloc (allocaBytes) +import Foreign.Ptr (Ptr, plusPtr) + import Crypto.Internal.ByteArray (ByteArray) +import Crypto.Internal.Compat (unsafeDoIO) +import Crypto.Number.Basic (numBytes) import Crypto.Number.ModArithmetic (expFast, expSafe) +import Crypto.Number.ModArithmetic.Pub (expPubBytes) import Crypto.Number.Serialize (i2ospOf_, os2ip) +import qualified Crypto.Number.Serialize.Internal as Internal import Crypto.PubKey.RSA.Types {- dpSlow computes the decrypted message not using any precomputed cache value. @@ -29,22 +41,78 @@ dpFast (Blinder r rm1) pk c = i2ospOf_ (private_size pk) - (multiplication rm1 (m2 + h * (private_q pk)) (private_n pk)) + (multiplication rm1 (crtChecked pk iC (m2 + h * (private_q pk))) (private_n pk)) where re = expFast r (public_e $ private_pub pk) (private_n pk) iC = multiplication re (os2ip c) (private_n pk) - m1 = expSafe iC (private_dP pk) (private_p pk) - m2 = expSafe iC (private_dQ pk) (private_q pk) + (m1, m2) = crtHalves iC pk h = ((private_qinv pk) * (m1 - m2)) `mod` (private_p pk) dpFastNoBlinder :: ByteArray ba => PrivateKey -> ba -> ba -dpFastNoBlinder pk c = i2ospOf_ (private_size pk) (m2 + h * (private_q pk)) +dpFastNoBlinder pk c = i2ospOf_ (private_size pk) (crtChecked pk iC (m2 + h * (private_q pk))) where iC = os2ip c - m1 = expSafe iC (private_dP pk) (private_p pk) - m2 = expSafe iC (private_dQ pk) (private_q pk) + (m1, m2) = crtHalves iC pk h = ((private_qinv pk) * (m1 - m2)) `mod` (private_p pk) +-- | The CRT answer m for c, raised back to e before it is let out: a fault +-- in one half gives an m whose gcd with n is the other prime (Boneh, DeMillo +-- and Lipton). On a mismatch it is computed again without CRT, as OpenSSL does. +crtChecked :: PrivateKey -> Integer -> Integer -> Integer +crtChecked pk c m + | expFast m (public_e (private_pub pk)) n == c `mod` n = m + | private_d pk > 0 = expSafe c (private_d pk) n + | otherwise = m + where + n = private_n pk + +-- | c^dP mod p and c^dQ mod q, the halves of the private-key operation: both +-- at once where the C can (AVX-512 IFMA, and the sizes of RSA-2048, -3072 and +-- -4096), one after the other where it declines. +crtHalves :: Integer -> PrivateKey -> (Integer, Integer) +crtHalves c pk = case expSafeX2 (c `mod` p) dP p (c `mod` q) dQ q of + Just r -> r + Nothing -> (expSafe c dP p, expSafe c dQ q) + where + p = private_p pk + q = private_q pk + dP = private_dP pk + dQ = private_dQ pk + +-- | Two secure exponentiations in one call. The bases are reduced already +-- and the moduli are the same length; both exponents go in at that length, +-- so the length the C sees is the modulus', as 'expSafe' pads to whole words. +expSafeX2 :: Integer -> Integer -> Integer -> Integer -> Integer -> Integer -> Maybe (Integer, Integer) +expSafeX2 b1 e1 m1 b2 e2 m2 + | numBytes m2 /= len || not (odd m1 && odd m2) = Nothing + | e1 < 0 || e2 < 0 || numBytes e1 > len || numBytes e2 > len = Nothing + | otherwise = unsafeDoIO $ + allocaBytes (8 * len) $ \start -> case take 8 (iterate (`plusPtr` len) start) of + [o1, pb1, pe1, pm1, o2, pb2, pe2, pm2] -> do + mapM_ + (\(v, ptr) -> Internal.i2ospOf v ptr len) + [(b1, pb1), (e1, pe1), (m1, pm1), (b2, pb2), (e2, pe2), (m2, pm2)] + r <- c_powm_sec_x2 o1 pb1 l pe1 l pm1 l o2 pb2 l pe2 l pm2 l + -- the exponents are the caller's secrets + memSet pe1 0 len + memSet pe2 0 len + if r /= 0 + then return Nothing + else do + !v1 <- Internal.os2ip o1 len + !v2 <- Internal.os2ip o2 len + return (Just (v1, v2)) + _ -> return Nothing + where + !len = numBytes m1 + l = fromIntegral len :: Word32 + +foreign import ccall safe "crypton_powm_sec_x2" + c_powm_sec_x2 + :: Ptr Word8 -> Ptr Word8 -> Word32 -> Ptr Word8 -> Word32 -> Ptr Word8 -> Word32 + -> Ptr Word8 -> Ptr Word8 -> Word32 -> Ptr Word8 -> Word32 -> Ptr Word8 -> Word32 + -> IO CInt + -- | Compute the RSA decrypt primitive. -- if the p and q numbers are available, then dpFast is used -- otherwise, we use dpSlow which only need d and n. @@ -56,7 +124,9 @@ -- | Compute the RSA encrypt primitive ep :: ByteArray ba => PublicKey -> ba -> ba -ep pk m = i2ospOf_ (public_size pk) $ expFast (os2ip m) (public_e pk) (public_n pk) +ep pk m = case expPubBytes (public_e pk) (public_n pk) m of + Just r -> r + Nothing -> i2ospOf_ (public_size pk) $ expFast (os2ip m) (public_e pk) (public_n pk) -- | multiply 2 integers in Zm only performing the modulo operation if necessary multiplication :: Integer -> Integer -> Integer -> Integer diff --git a/Crypto/Random.hs b/Crypto/Random.hs --- a/Crypto/Random.hs +++ b/Crypto/Random.hs @@ -7,35 +7,159 @@ -- Maintainer : Vincent Hanquez -- Stability : stable -- Portability : good +-- +-- Random bytes, drawn either from the system or from a generator whose +-- seed you hold. +-- +-- == Drawing from the system +-- +-- 'getRandomBytes' in 'IO' is the ordinary way to get bytes nobody can +-- predict. The length is in bytes and the type is any 'ByteArray', so the +-- result is usually pinned down by where it goes: +-- +-- > import Crypto.Random (getRandomBytes) +-- > import Data.ByteString (ByteString) +-- > +-- > nonce <- getRandomBytes 12 :: IO ByteString +-- +-- Everything in this library that needs randomness takes it the same way, +-- through a @MonadRandom m =>@ constraint, so running it in 'IO' is the +-- whole of choosing this source: +-- +-- > import qualified Crypto.PubKey.RSA as RSA +-- > +-- > (pub, priv) <- RSA.generate 256 0x10001 +-- +-- Where the operating system offers @getrandom(2)@ or @getentropy(3)@ the +-- bytes come from a ChaCha20 generator belonging to the operating system +-- thread the call runs on, seeded from the system and reseeded as it goes; +-- see "Crypto.Random.SysDRG" for what that is and what it is not. Where +-- it does not -- Windows, for now -- they come from the system on every +-- call, as they always did. Either way the caller writes the same line. +-- +-- Any Haskell thread may draw, and none has to say which generator it +-- wants: +-- +-- > import Control.Concurrent (forkIO) +-- > +-- > mapM_ (\_ -> forkIO (getRandomBytes 32 >>= use)) [1 .. 64 :: Int] +-- +-- The generator belongs to the operating system thread that happens to be +-- carrying the Haskell thread when the call is made, and is found through +-- that thread's own storage rather than by anything the caller passes. A +-- @forkIO@ thread moves between capabilities, so two draws from one Haskell +-- thread may be answered by two generators; they are seeded independently +-- of each other, so it makes no difference which one answers. +-- +-- == Drawing from a generator you hold +-- +-- A generator of your own is the way to draw many times from one seed. +-- The system is asked once, when the generator is made, and not again. +-- That is how @tls@ does a connection: 'seedNew' when it opens, and +-- everything the handshake needs afterwards from the generator, whose +-- state the connection carries from one draw to the next. +-- +-- > import Crypto.Random (ChaChaDRG, drgNewSeed, seedNew, withDRG) +-- > +-- > data Connection = Connection { connRNG :: ChaChaDRG } +-- > +-- > newConnection :: IO Connection +-- > newConnection = do +-- > seed <- seedNew -- the only draw from the system +-- > return (Connection (drgNewSeed seed)) +-- > +-- > -- every later draw advances the connection's own generator +-- > connRandom :: Int -> Connection -> (ByteString, Connection) +-- > connRandom n conn = +-- > let (bytes, rng') = withDRG (connRNG conn) (getRandomBytes n) +-- > in (bytes, conn{connRNG = rng'}) +-- +-- Inside 'withDRG' the same 'getRandomBytes' resolves to the instance for +-- 'MonadPseudoRandom' rather than the one for 'IO', so it touches neither +-- the system nor the per-thread generator: it advances the generator it was +-- given and hands it back. Keep that generator and the bytes are +-- reproducible from the seed, which is what makes a test repeatable -- and +-- what makes a generator unfit for keys unless its seed came from the +-- system. +-- +-- 'drgNew' is 'seedNew' and 'drgNewSeed' in one step; 'drgNewTest' takes +-- four numbers instead of a seed, for a test that must give the same answer +-- twice. +-- +-- == Drawing from a monad stacked on IO +-- +-- There are two instances, one for 'IO' and one for 'MonadPseudoRandom', +-- and none for the transformers, so a @ReaderT env IO@ or a @StateT s IO@ +-- reaches the system through 'liftIO': +-- +-- > import Control.Monad.IO.Class (liftIO) +-- > import Control.Monad.Trans.Reader (ReaderT) +-- > +-- > newKey :: ReaderT env IO ByteString +-- > newKey = liftIO (getRandomBytes 32) +-- +-- What that draws is what the first section describes, no more and no less: +-- the 'IO' instance, the generator of the operating system thread the call +-- lands on, seeded from the system and reseeded as it goes. 'liftIO' says +-- which monad the draw happens in and nothing about where the bytes come +-- from. +-- +-- Writing an instance for the stack instead would make the choice invisible +-- at the call site, and the class cannot tell a strong source from a weak +-- one -- see 'MonadRandom'. +-- +-- == When the system will not give any +-- +-- Drawing randomness is the one thing here with nothing to fall back on, +-- so the failure is an exception rather than a value: there is no sensible +-- 'Maybe' to return and no partial answer worth having. Since 2.2 it is an +-- 'EntropyError' and not a @Control.Exception.ErrorCall@ or an +-- @Control.Exception.IOException@, which is worth knowing if you were +-- catching one of those: +-- +-- > import Control.Exception (catch) +-- > import Crypto.Random (EntropyError (..), getRandomBytes) +-- > +-- > key <- getRandomBytes 32 `catch` \e -> case e of +-- > NoEntropySource -> fail "this system offers no randomness at all" +-- > EntropyShort w g -> fail (show w ++ " bytes wanted, " ++ show g ++ " arrived") +-- > EntropySourceLost s -> fail ("the source " ++ s ++ " went away") +-- +-- Catching it at all is a decision rather than a default. A program that +-- cannot get randomness cannot make a key, and stopping is usually the +-- honest thing; the reason to catch is to say so in the program's own +-- terms rather than in crypton's. module Crypto.Random ( - -- * Deterministic instances - ChaChaDRG, - SystemDRG, - Seed, + -- * Drawing from the system + MonadRandom (..), - -- * Seed + -- * Drawing from a generator you hold + Seed, seedNew, seedFromInteger, seedToInteger, seedFromBinary, - - -- * Deterministic Random class - getSystemDRG, - drgNew, drgNewSeed, + drgNew, drgNewTest, withDRG, withRandomBytes, + MonadPseudoRandom, DRG (..), - -- * Random abstraction - MonadRandom (..), - MonadPseudoRandom, + -- * The generators + ChaChaDRG, + SystemDRG, + getSystemDRG, + + -- * When the system will not give any + EntropyError (..), ) where import Crypto.Error import Crypto.Internal.Imports import Crypto.Random.ChaChaDRG +import Crypto.Random.Entropy (EntropyError (..)) import Crypto.Random.SystemDRG import Crypto.Random.Types import Data.ByteArray (ByteArray, ByteArrayAccess, ScrubbedBytes) @@ -50,6 +174,9 @@ import Foreign.Ptr (Ptr, castPtr) #endif +-- | The material a deterministic generator is built from. Two generators +-- made from one seed produce the same bytes, which is what 'drgNewSeed' is +-- for and why a seed kept anywhere is as good as the keys drawn from it. newtype Seed = Seed ScrubbedBytes deriving (ByteArrayAccess) diff --git a/Crypto/Random/Entropy.hs b/Crypto/Random/Entropy.hs --- a/Crypto/Random/Entropy.hs +++ b/Crypto/Random/Entropy.hs @@ -6,16 +6,37 @@ -- Portability : Good module Crypto.Random.Entropy ( getEntropy, + EntropyError (..), ) where import Crypto.Internal.ByteArray (ByteArray) import qualified Crypto.Internal.ByteArray as B -import Data.Maybe (catMaybes) +import System.IO.Unsafe (unsafeInterleaveIO, unsafePerformIO) import Crypto.Random.Entropy.Unsafe +-- | The backends this system has, worked out once and no further than +-- needed. +-- +-- Opening one is not free: for a device file it means opening and closing +-- @\/dev\/random@ or @\/dev\/urandom@ just to learn that it is there. This +-- used to be done on every call, and for the whole list before any backend +-- was asked for a byte, so a program paid for both devices even when the +-- first backend answered everything. +-- +-- Once, now, and lazily: 'replenish' stops as soon as the buffer is full, +-- which leaves the tail of this list unforced, so a system where the first +-- backend answers never opens a device at all. +{-# NOINLINE openedBackends #-} +openedBackends :: [EntropyBackend] +openedBackends = unsafePerformIO (openAsNeeded supportedBackends) + where + openAsNeeded [] = return [] + openAsNeeded (o : os) = do + m <- o + rest <- unsafeInterleaveIO (openAsNeeded os) + return $ maybe rest (: rest) m + -- | Get some entropy from the system source of entropy getEntropy :: ByteArray byteArray => Int -> IO byteArray -getEntropy n = do - backends <- catMaybes `fmap` sequence supportedBackends - B.alloc n (replenish n backends) +getEntropy n = B.alloc n (replenish n openedBackends) diff --git a/Crypto/Random/Entropy/Backend.hs b/Crypto/Random/Entropy/Backend.hs --- a/Crypto/Random/Entropy/Backend.hs +++ b/Crypto/Random/Entropy/Backend.hs @@ -11,27 +11,39 @@ ( EntropyBackend , supportedBackends , gatherBackend + , EntropyError(..) ) where import Foreign.Ptr import Data.Proxy import Data.Word (Word8) import Crypto.Random.Entropy.Source -#ifdef SUPPORT_RDRAND -import Crypto.Random.Entropy.RDRand -#endif #ifdef WINDOWS import Crypto.Random.Entropy.Windows #else +import Crypto.Random.Entropy.SysRandom import Crypto.Random.Entropy.Unix #endif --- | All supported backends +-- | All supported backends, best first. +-- +-- The system call comes before everything else: it is the kernel's own +-- generator, it needs no descriptor, and it is what the rest of the world +-- reaches for now. The device files are what is left when the system has +-- no such call. +-- +-- RDRAND is deliberately not here, though it used to be first on x86. A +-- list like this one is a list of alternatives, and whichever answers +-- first decides the bytes on its own -- which is the one thing #298 says +-- RDRAND should not do. Nor does it contribute anywhere else: every +-- system this builds for already feeds the instruction into the pool the +-- call above draws from, so asking it again adds nothing. See the note in +-- @cbits\/crypton_sysdrg.c@. supportedBackends :: [IO (Maybe EntropyBackend)] supportedBackends = [ -#ifdef SUPPORT_RDRAND - openBackend (Proxy :: Proxy RDRand), +#ifndef WINDOWS + openBackend (Proxy :: Proxy SysRandom), #endif #ifdef WINDOWS openBackend (Proxy :: Proxy WinCryptoAPI) diff --git a/Crypto/Random/Entropy/RDRand.hs b/Crypto/Random/Entropy/RDRand.hs deleted file mode 100644 --- a/Crypto/Random/Entropy/RDRand.hs +++ /dev/null @@ -1,39 +0,0 @@ -{-# LANGUAGE ForeignFunctionInterface #-} - --- | --- Module : Crypto.Random.Entropy.RDRand --- License : BSD-style --- Maintainer : Vincent Hanquez --- Stability : experimental --- Portability : Good -module Crypto.Random.Entropy.RDRand ( - RDRand, -) where - -import Crypto.Random.Entropy.Source -import Data.Word (Word8) -import Foreign.C.Types -import Foreign.Ptr - -foreign import ccall unsafe "crypton_cpu_has_rdrand" - c_cpu_has_rdrand :: IO CInt - -foreign import ccall unsafe "crypton_get_rand_bytes" - c_get_rand_bytes :: Ptr Word8 -> CInt -> IO CInt - --- | Fake handle to Intel RDRand entropy CPU instruction -data RDRand = RDRand - -instance EntropySource RDRand where - entropyOpen = rdrandGrab - entropyGather _ = rdrandGetBytes - entropyClose _ = return () - -rdrandGrab :: IO (Maybe RDRand) -rdrandGrab = supported `fmap` c_cpu_has_rdrand - where - supported 0 = Nothing - supported _ = Just RDRand - -rdrandGetBytes :: Ptr Word8 -> Int -> IO Int -rdrandGetBytes ptr sz = fromIntegral `fmap` c_get_rand_bytes ptr (fromIntegral sz) diff --git a/Crypto/Random/Entropy/Source.hs b/Crypto/Random/Entropy/Source.hs --- a/Crypto/Random/Entropy/Source.hs +++ b/Crypto/Random/Entropy/Source.hs @@ -6,8 +6,32 @@ -- Portability : Good module Crypto.Random.Entropy.Source where +import Control.Exception (Exception) import Data.Word (Word8) import Foreign.Ptr + +-- | The system would not give the entropy it was asked for. +-- +-- This is the one failure in the library with nothing to fall back on and +-- nothing sensible to return: a key drawn from bytes that are not random +-- is worse than no key. It used to be reported with 'error' and 'fail', +-- which left a caller no way to tell it from a bug in the library, and no +-- way to say anything useful about it. +data EntropyError + = -- | The system offers no source of entropy at all. On Unix that + -- means no @getrandom(2)@, no @getentropy(3)@ and no @\/dev@ to read + -- from; a container built from nothing would look like this. + NoEntropySource + | -- | The sources between them gave fewer bytes than were asked for, + -- three times over. The two numbers are how many were wanted and + -- how many arrived. + EntropyShort Int Int + | -- | A source that could be opened once could not be opened again. + -- The string names it. + EntropySourceLost String + deriving (Show, Eq) + +instance Exception EntropyError -- | A handle to an entropy maker, either a system capability -- or a hardware generator. diff --git a/Crypto/Random/Entropy/SysRandom.hs b/Crypto/Random/Entropy/SysRandom.hs new file mode 100644 --- /dev/null +++ b/Crypto/Random/Entropy/SysRandom.hs @@ -0,0 +1,50 @@ +{-# LANGUAGE ForeignFunctionInterface #-} + +-- | +-- Module : Crypto.Random.Entropy.SysRandom +-- License : BSD-style +-- Maintainer : Kazu Yamamoto +-- Stability : experimental +-- Portability : Unix +-- +-- The kernel's own generator, reached without a file descriptor: +-- @getrandom(2)@ on Linux and FreeBSD, @getentropy(3)@ where that is what +-- the system has. +-- +-- This is the source to prefer over reading @\/dev\/urandom@. It needs no +-- path and no descriptor, so it still answers where @\/dev@ is not mounted +-- or not populated, and it cannot be defeated by a full descriptor table. +module Crypto.Random.Entropy.SysRandom ( + SysRandom, +) where + +import Crypto.Random.Entropy.Source +import Data.Word (Word8) +import Foreign.C.Types +import Foreign.Ptr + +-- Both are @safe@ rather than @unsafe@: at early boot, before the kernel +-- pool is initialised, these calls block, and an unsafe call that blocks +-- holds the capability it runs on. +foreign import ccall safe "crypton_sysrandom_available" + c_sysrandom_available :: IO CInt + +foreign import ccall safe "crypton_sysrandom_bytes" + c_sysrandom_bytes :: Ptr Word8 -> CInt -> IO CInt + +-- | The system call, where there is one. +data SysRandom = SysRandom + +instance EntropySource SysRandom where + -- Asked at run time and not only at compile time: a binary built where + -- the header declares the call can still run on a kernel that answers + -- ENOSYS. + entropyOpen = available `fmap` c_sysrandom_available + where + available 0 = Nothing + available _ = Just SysRandom + + entropyGather _ ptr n = + fromIntegral `fmap` c_sysrandom_bytes ptr (fromIntegral n) + + entropyClose _ = return () diff --git a/Crypto/Random/Entropy/Unix.hs b/Crypto/Random/Entropy/Unix.hs --- a/Crypto/Random/Entropy/Unix.hs +++ b/Crypto/Random/Entropy/Unix.hs @@ -61,7 +61,7 @@ withDev filepath f = openDev filepath >>= \h -> case h of - Nothing -> error ("device " ++ filepath ++ " cannot be grabbed") + Nothing -> E.throwIO (EntropySourceLost filepath) Just fd -> f fd `E.finally` closeDev fd closeDev :: H -> IO () diff --git a/Crypto/Random/Entropy/Unsafe.hs b/Crypto/Random/Entropy/Unsafe.hs --- a/Crypto/Random/Entropy/Unsafe.hs +++ b/Crypto/Random/Entropy/Unsafe.hs @@ -9,25 +9,27 @@ module Crypto.Random.Entropy.Backend, ) where +import Control.Exception (throwIO) + import Crypto.Random.Entropy.Backend import Data.Word (Word8) import Foreign.Ptr (Ptr, plusPtr) -- | Refill the entropy in a buffer -- --- Call each entropy backend in turn until the buffer has --- been replenished. +-- Call each entropy backend in turn until the buffer has been replenished. -- --- If the buffer cannot be refill after 3 loopings, this will raise --- an User Error exception +-- Throws 'EntropyError': 'NoEntropySource' when there is no backend at all, +-- and 'EntropyShort' when three passes over the backends still leave the +-- buffer unfilled. replenish :: Int -> [EntropyBackend] -> Ptr Word8 -> IO () -replenish _ [] _ = fail "crypton: random: cannot get any source of entropy on this system" +replenish _ [] _ = throwIO NoEntropySource replenish poolSize backends ptr = loop 0 backends ptr poolSize where loop :: Int -> [EntropyBackend] -> Ptr Word8 -> Int -> IO () loop _ _ _ 0 = return () loop retry [] p n - | retry == 3 = error "crypton: random: cannot fully replenish" + | retry == 3 = throwIO $ EntropyShort poolSize (poolSize - n) | otherwise = loop (retry + 1) backends p n loop retry (b : bs) p n = do r <- gatherBackend b p n diff --git a/Crypto/Random/Entropy/Windows.hs b/Crypto/Random/Entropy/Windows.hs --- a/Crypto/Random/Entropy/Windows.hs +++ b/Crypto/Random/Entropy/Windows.hs @@ -23,6 +23,7 @@ import Foreign.Storable (peek) import System.Win32.Types (getLastError) +import Control.Exception (throwIO) import Crypto.Random.Entropy.Source @@ -38,7 +39,8 @@ case mctx of Nothing -> do lastError <- getLastError - fail $ "cannot re-grab win crypto api: error " ++ show lastError + throwIO $ EntropySourceLost + ("the Windows crypto API: error " ++ show lastError) Just ctx -> do r <- cryptGenRandom ctx ptr n cryptReleaseCtx ctx @@ -100,4 +102,5 @@ then return () else do lastError <- getLastError - fail $ "cryptReleaseCtx: error " ++ show lastError + throwIO $ EntropySourceLost + ("cryptReleaseCtx: error " ++ show lastError) diff --git a/Crypto/Random/SysDRG.hs b/Crypto/Random/SysDRG.hs new file mode 100644 --- /dev/null +++ b/Crypto/Random/SysDRG.hs @@ -0,0 +1,82 @@ +{-# LANGUAGE ForeignFunctionInterface #-} + +-- | +-- Module : Crypto.Random.SysDRG +-- License : BSD-style +-- Maintainer : Kazu Yamamoto +-- Stability : experimental +-- Portability : Unix, Windows +-- +-- The generator behind the 'Crypto.Random.MonadRandom' instance for 'IO'. +-- +-- A ChaCha20 generator per operating system thread, seeded from a +-- process-wide generator, seeded in turn from the system entropy pool. +-- The state and the reseeding live in +-- @cbits\/crypton_sysdrg.c@, because a @forkIO@ thread is not an operating +-- system thread -- it moves between capabilities -- so state held against +-- one would be shared by threads running at the same time. +-- +-- == What it is, and what it is not +-- +-- It is not a DRBG of NIST SP 800-90A. That standard names three -- +-- @Hash_DRBG@, @HMAC_DRBG@ and @CTR_DRBG@ -- and none of them is built on a +-- stream cipher. The name here is the older and looser sense of the word. +-- +-- What it is is the shape @arc4random@ on OpenBSD and @get_random_bytes@ in +-- the Linux kernel both have, and the one asked for in +-- . That shape is +-- common; it is not specified anywhere, so the parts a standard would have +-- fixed were chosen here instead: +-- +-- * ChaCha20, rather than AES in counter mode. +-- * SHA-512 over the system's bytes, rather than a derivation function a +-- standard would have named. +-- * A mebibyte per thread, and a mebibyte of issued seed for the process +-- generator, as the points at which to reseed. Those numbers are a +-- choice, not a result. +-- +-- The pieces underneath are specified: ChaCha20 is RFC 8439, SHA-512 is +-- FIPS 180-4. The way they are put together is not. +-- +-- == What a compromised state gives away +-- +-- Each draw ends by taking the next forty bytes of keystream as the key and +-- nonce and dropping the ones that made the output, so a state read after a +-- draw is not the state that produced it. ChaCha20 does not run backwards +-- and the key that would be needed is gone, which is backtracking +-- resistance in the terms of SP 800-90A. Without it, a key stands until +-- the next reseed and anyone holding the state can wind the counter back +-- over everything issued since -- a mebibyte of output that was meant to be +-- secret. It is what @arc4random@ does, and for the same reason. +-- +-- Prediction resistance is what the reseeding gives. A reseed draws from +-- the system again, so a state that has been read does not determine what +-- comes after one. +module Crypto.Random.SysDRG ( + sysDRGBytes, +) where + +import Data.Word (Word8) +import Foreign.C.Types (CInt (..)) +import Foreign.Ptr (Ptr) + +import Crypto.Internal.ByteArray (ByteArray) +import qualified Crypto.Internal.ByteArray as B + +-- Safe, not unsafe: seeding can reach a getrandom(2) that blocks until the +-- kernel pool is ready, and an unsafe call that blocks holds the capability +-- it runs on. +foreign import ccall safe "crypton_sysdrg_bytes" + c_sysdrg_bytes :: Ptr Word8 -> CInt -> IO CInt + +-- | Draw bytes, or 'Nothing' if the generator cannot be seeded. +-- +-- It cannot be seeded where the system has no @getrandom(2)@ or +-- @getentropy(3)@; the caller falls back to +-- 'Crypto.Random.Entropy.getEntropy' there, which is the path this system +-- had before the generator existed. +sysDRGBytes :: ByteArray byteArray => Int -> IO (Maybe byteArray) +sysDRGBytes n = do + (got, out) <- B.allocRet n $ \ptr -> + fromIntegral `fmap` c_sysdrg_bytes ptr (fromIntegral n) + return $ if got == n then Just out else Nothing diff --git a/Crypto/Random/Types.hs b/Crypto/Random/Types.hs --- a/Crypto/Random/Types.hs +++ b/Crypto/Random/Types.hs @@ -13,6 +13,7 @@ import Crypto.Internal.ByteArray import Crypto.Random.Entropy +import Crypto.Random.SysDRG (sysDRGBytes) -- | A monad constraint that allows to generate random bytes -- @@ -35,8 +36,13 @@ -- | Generate N bytes of randomness from a DRG randomBytesGenerate :: ByteArray byteArray => Int -> gen -> (byteArray, gen) +-- | Through the generator of 'Crypto.Random.SysDRG': a ChaCha20 generator +-- per operating system thread, reseeded from the system. Where that +-- cannot be seeded -- a system with no @getrandom(2)@ or @getentropy(3)@, +-- which includes Windows for now -- this is the system entropy source +-- directly, as it was before the generator existed. instance MonadRandom IO where - getRandomBytes = getEntropy + getRandomBytes n = sysDRGBytes n >>= maybe (getEntropy n) return -- | A simple Monad class very similar to a State Monad -- with the state being a DRG. diff --git a/Crypto/System/CPU.hs b/Crypto/System/CPU.hs --- a/Crypto/System/CPU.hs +++ b/Crypto/System/CPU.hs @@ -1,6 +1,6 @@ {-# LANGUAGE CPP #-} -{-# LANGUAGE DeriveDataTypeable #-} {-# LANGUAGE ForeignFunctionInterface #-} +{-# LANGUAGE PatternSynonyms #-} -- | -- Module : Crypto.System.CPU @@ -11,57 +11,218 @@ -- -- Gives information about crypton runtime environment. module Crypto.System.CPU ( - ProcessorOption (..), + -- The names are bundled with the type rather than listed as + -- `pattern' exports, so an importer writes ProcessorOption (..), or + -- names the ones it wants, as it would for a type with constructors. + ProcessorOption ( + -- x86 + AESNI, + PCLMUL, + SSSE3, + AVX, + AVX2, + SHANI, + MOVBE, + ADX, + VAES, + VAES512, + AVX512IFMA, + -- AArch64 + NEON, + ARMAES, + ARMPMULL, + ARMSHA1, + ARMSHA2, + ARMSHA512, + -- PowerISA + PPCAES, + PPCVPMSUM + ), processorOptions, -) where -import Data.Data -import Data.List (findIndices) -#ifdef SUPPORT_RDRAND -import Data.Maybe (isJust) -#endif -import Data.Word (Word8) -import Foreign.Ptr -import Foreign.Storable + -- * Questions that do not name an architecture + hasAESAcceleration, + hasGHASHAcceleration, +) where +import Control.Monad (filterM) +import Data.List (sort) +import Data.Word (Word16) +import Foreign.C.Types (CInt (..), CUInt (..)) import Crypto.Internal.Compat -#ifdef SUPPORT_RDRAND -import Crypto.Random.Entropy.RDRand -import Crypto.Random.Entropy.Source -#endif +-- | A processor feature crypton looked for, and dispatches on where it +-- finds it. +-- +-- This is a number with names rather than a sum of constructors, and the +-- names are pattern synonyms with no @COMPLETE@ pragma, so a @case@ over +-- them needs a catch-all and a feature named in a later release breaks +-- nothing that compiled against this one. The same reason 'Show' is +-- written out below: a program built against an older crypton still says +-- something useful about a value from a newer one. +-- +-- The names are the processor's, not the operation's. 'AESNI' is x86's +-- and 'ARMAES' is AArch64's, and a machine reports only the ones it has; +-- ask 'hasAESAcceleration' if the question is whether AES is fast here. +-- +-- They are bundled with the type in the export list, so @ProcessorOption +-- (..)@ brings in all of them and naming one brings in that one, as for a +-- type with constructors. The constructor underneath is not exported: +-- these values say what the processor was found to have, and a caller has +-- nothing to build. +newtype ProcessorOption = ProcessorOption Word16 + deriving (Eq, Ord) --- | CPU options impacting cryptography implementation and library performance. -data ProcessorOption - = -- | Support for AES instructions, with flag @support_aesni@ - AESNI - | -- | Support for CLMUL instructions, with flag @support_pclmuldq@ - PCLMUL - | -- | Support for RDRAND instruction, with flag @support_rdrand@ - RDRAND - deriving (Show, Eq, Enum, Data) +-- | Support for AES instructions, with flag @support_aesni@. +pattern AESNI :: ProcessorOption +pattern AESNI = ProcessorOption 0 --- | Options which have been enabled at compile time and are supported by the --- current CPU. +-- | Support for CLMUL instructions, with flag @support_pclmuldq@. +pattern PCLMUL :: ProcessorOption +pattern PCLMUL = ProcessorOption 1 + +-- | Supplemental SSE3. +pattern SSSE3 :: ProcessorOption +pattern SSSE3 = ProcessorOption 3 + +-- | AVX, and an operating system that saves its registers. +pattern AVX :: ProcessorOption +pattern AVX = ProcessorOption 4 + +-- | AVX2, and an operating system that saves its registers. +pattern AVX2 :: ProcessorOption +pattern AVX2 = ProcessorOption 5 + +-- | The SHA extensions, @sha1rnds4@ and @sha256rnds2@ and their neighbours. +pattern SHANI :: ProcessorOption +pattern SHANI = ProcessorOption 6 + +-- | The byte-swapping load. +pattern MOVBE :: ProcessorOption +pattern MOVBE = ProcessorOption 7 + +-- | @MULX@, @ADCX@ and @ADOX@: the two independent carry chains. +pattern ADX :: ProcessorOption +pattern ADX = ProcessorOption 8 + +-- | The AES and carry-less multiply instructions in their 256-bit form. +pattern VAES :: ProcessorOption +pattern VAES = ProcessorOption 9 + +-- | The same pair in their 512-bit form. +pattern VAES512 :: ProcessorOption +pattern VAES512 = ProcessorOption 10 + +-- | The AVX-512 52-bit integer multiply-adds, with AVX-512 F and VL, and the +-- operating system saving their state: RSA's private-key operation uses them. +pattern AVX512IFMA :: ProcessorOption +pattern AVX512IFMA = ProcessorOption 19 + +-- | Advanced SIMD, which is not optional on AArch64. +pattern NEON :: ProcessorOption +pattern NEON = ProcessorOption 11 + +-- | The ARMv8 AES instructions. +pattern ARMAES :: ProcessorOption +pattern ARMAES = ProcessorOption 12 + +-- | @PMULL@, the ARMv8 carry-less multiply. +pattern ARMPMULL :: ProcessorOption +pattern ARMPMULL = ProcessorOption 13 + +-- | The ARMv8 SHA-1 instructions. +pattern ARMSHA1 :: ProcessorOption +pattern ARMSHA1 = ProcessorOption 14 + +-- | The ARMv8 SHA-256 instructions. +pattern ARMSHA2 :: ProcessorOption +pattern ARMSHA2 = ProcessorOption 15 + +-- | The ARMv8.2 SHA-512 instructions, which are optional where SHA-256's +-- are not. +pattern ARMSHA512 :: ProcessorOption +pattern ARMSHA512 = ProcessorOption 16 + +-- | Support for the PowerISA 2.07 vector AES instructions, which POWER8 was +-- the first to implement. +pattern PPCAES :: ProcessorOption +pattern PPCAES = ProcessorOption 17 + +-- | Support for @vpmsumd@, the vector carry-less multiply that came with +-- them, which is what makes GHASH fast. +pattern PPCVPMSUM :: ProcessorOption +pattern PPCVPMSUM = ProcessorOption 18 + +-- | Named where the name is known, numbered where it is not, so that a +-- binary built against an older crypton can still print a value a newer one +-- produced. +instance Show ProcessorOption where + show AESNI = "AESNI" + show PCLMUL = "PCLMUL" + show SSSE3 = "SSSE3" + show AVX = "AVX" + show AVX2 = "AVX2" + show SHANI = "SHANI" + show MOVBE = "MOVBE" + show ADX = "ADX" + show VAES = "VAES" + show VAES512 = "VAES512" + show AVX512IFMA = "AVX512IFMA" + show NEON = "NEON" + show ARMAES = "ARMAES" + show ARMPMULL = "ARMPMULL" + show ARMSHA1 = "ARMSHA1" + show ARMSHA2 = "ARMSHA2" + show ARMSHA512 = "ARMSHA512" + show PPCAES = "PPCAES" + show PPCVPMSUM = "PPCVPMSUM" + show (ProcessorOption n) = "ProcessorOption " ++ show n + +-- | Options which have been enabled at compile time and which this machine +-- says may be used: on x86-64 what the processor declares through CPUID, and +-- for the vector extensions only where the operating system saves their +-- state as well; elsewhere what the operating system reports of the +-- processor. +-- +-- That is what crypton itself decides by, and it is not always everything +-- the core underneath executes. A guest under a hypervisor may be shown an +-- older processor model than the host's, so that it can be moved to another +-- host -- and an instruction used because it happens to run would fail +-- there. So an option missing here means crypton will not use it, not that +-- the hardware lacks it. +-- +-- Sorted, and without repeats. A machine reports the names of its own +-- architecture only: an AArch64 processor with AES says 'ARMAES', not +-- 'AESNI', which it does not have. processorOptions :: [ProcessorOption] -processorOptions = unsafeDoIO $ do - p <- crypton_aes_cpu_init - options <- traverse (getOption p) aesOptions - rdrand <- hasRDRand - return (decodeOptions options ++ [RDRAND | rdrand]) +processorOptions = unsafeDoIO (sort <$> filterM askC allOptions) where - aesOptions = [AESNI .. PCLMUL] - getOption p = peekElemOff p . fromEnum - decodeOptions = map toEnum . findIndices (> 0) + -- 2 is not asked for and has no name: it was RDRAND, which crypton no + -- longer dispatches on. The number is left out rather than reused, so + -- that the others keep the values they had. + allOptions = [ProcessorOption n | n <- [0 .. 19], n /= 2] + askC (ProcessorOption n) = + (/= 0) <$> crypton_cpu_option (fromIntegral n) {-# NOINLINE processorOptions #-} -hasRDRand :: IO Bool -#ifdef SUPPORT_RDRAND -hasRDRand = fmap isJust getRDRand - where getRDRand = entropyOpen :: IO (Maybe RDRand) -#else -hasRDRand = return False -#endif +-- | Is there hardware AES on this machine? +-- +-- The instructions have different names on different architectures, and a +-- caller that wants to know whether AES-GCM will be fast wants this rather +-- than either name. +hasAESAcceleration :: Bool +hasAESAcceleration = + any + (`elem` processorOptions) + [AESNI, ARMAES, PPCAES] -foreign import ccall unsafe "crypton_aes_cpu_init" - crypton_aes_cpu_init :: IO (Ptr Word8) +-- | Is there a hardware carry-less multiply, which is what GHASH, and so +-- AES-GCM, spends its time in once AES itself is fast? +hasGHASHAcceleration :: Bool +hasGHASHAcceleration = + any + (`elem` processorOptions) + [PCLMUL, ARMPMULL, PPCVPMSUM] + +foreign import ccall unsafe "crypton_cpu_option" + crypton_cpu_option :: CUInt -> IO CInt diff --git a/Crypto/Tutorial.hs b/Crypto/Tutorial.hs --- a/Crypto/Tutorial.hs +++ b/Crypto/Tutorial.hs @@ -1,4 +1,7 @@ -- | Examples of how to use @crypton@. +-- +-- Every code block here is extracted and compiled against this version of +-- the library by @tests\/tutorial\/run.sh@, so what is written below builds. module Crypto.Tutorial ( -- * API design -- $api_design @@ -6,9 +9,21 @@ -- * Hash algorithms -- $hash_algorithms - -- * Symmetric block ciphers - -- $symmetric_block_ciphers + -- * Authenticated encryption + -- $authenticated_encryption + -- * Comparing secrets + -- $comparing_secrets + + -- * Password storage + -- $password_storage + + -- * Key derivation + -- $key_derivation + + -- * Digital signatures + -- $digital_signatures + -- * Combining primitives -- $combining_primitives ) where @@ -31,6 +46,14 @@ -- Error conditions are returned with data type 'Crypto.Error.CryptoFailable'. -- Functions in module "Crypto.Error" can convert those values to runtime -- exceptions, 'Maybe' or 'Either' values. +-- +-- Types that hold a secret do not print it. A private key's 'Show' renders +-- whatever is public and @\@ or @\@ for the rest, +-- because 'Show' is what @print@, @error@, an exception and a failing test +-- all reach for, and a key arriving in a log that way is an accident nobody +-- asked for. "Crypto.Debug" is how one is printed when printing it is what +-- was meant. Several of those types keep their bytes in +-- 'Data.ByteArray.ScrubbedBytes', which is wiped when it is collected. -- $hash_algorithms -- @@ -90,72 +113,258 @@ -- > hashMutableUpdate ctx ("dog" :: ByteString) -- > hashMutableFinalize ctx >>= print --- $symmetric_block_ciphers +-- $authenticated_encryption -- +-- Encrypting hides a message; it does not stop anyone changing it. Under a +-- counter or stream mode, flipping a bit of the ciphertext flips the same +-- bit of the plaintext, and the receiver has no way to tell. An AEAD mode +-- binds the message, and anything else named as associated data, to a short +-- authentication tag, and decrypting something that does not match that tag +-- returns nothing at all. +-- +-- That is the mode to use. The unauthenticated ones are in this library +-- for protocols that authenticate separately, not as a starting point. +-- -- > {-# LANGUAGE OverloadedStrings #-} --- > {-# LANGUAGE ScopedTypeVariables #-} --- > {-# LANGUAGE GADTs #-} -- > -- > import Crypto.Cipher.AES (AES256) --- > import Crypto.Cipher.Types (BlockCipher(..), Cipher(..), nullIV, KeySizeSpecifier(..), IV, makeIV) --- > import Crypto.Error (CryptoFailable(..), CryptoError(..)) +-- > import Crypto.Cipher.Types +-- > ( AEADMode (AEAD_GCM) +-- > , AuthTag +-- > , BlockCipher (aeadInit) +-- > , Cipher (cipherInit, cipherKeySize) +-- > , KeySizeSpecifier (..) +-- > , aeadSimpleDecrypt +-- > , aeadSimpleEncrypt +-- > ) +-- > import Crypto.Error (CryptoError, eitherCryptoError) +-- > import qualified Crypto.Random.Types as CRT -- > +-- > import Data.ByteArray (ByteArray, ByteArrayAccess, ScrubbedBytes) +-- > import Data.ByteString (ByteString) +-- > +-- > -- | A key of the length the cipher asks for, rather than a length +-- > -- written out here. ScrubbedBytes rather than ByteString, so that it +-- > -- is wiped when it is collected and does not print. +-- > genSecretKey :: (Cipher c, CRT.MonadRandom m) => c -> m ScrubbedBytes +-- > genSecretKey c = CRT.getRandomBytes (longest (cipherKeySize c)) +-- > where +-- > longest (KeySizeFixed n) = n +-- > longest (KeySizeRange _ n) = n +-- > longest (KeySizeEnum ns) = maximum ns +-- > +-- > -- | A fresh nonce for every message. GCM must never see one twice +-- > -- under the same key: a repeat does not just expose those two messages, +-- > -- it hands over the key that authenticates all of them. Twelve random +-- > -- bytes, sent along with the ciphertext. +-- > genNonce :: CRT.MonadRandom m => m ByteString +-- > genNonce = CRT.getRandomBytes 12 +-- > +-- > -- | Encrypt and authenticate. The associated data is authenticated but +-- > -- not encrypted: it is for what the receiver can already see and must +-- > -- not have had altered, such as a header or an address. +-- > encrypt +-- > :: (ByteArray key, ByteArrayAccess nonce, ByteArrayAccess aad, ByteArray ba) +-- > => key -> nonce -> aad -> ba -> Either CryptoError (AuthTag, ba) +-- > encrypt key nonce aad plaintext = do +-- > cipher <- eitherCryptoError (cipherInit key) :: Either CryptoError AES256 +-- > aead <- eitherCryptoError (aeadInit AEAD_GCM cipher nonce) +-- > return (aeadSimpleEncrypt aead aad plaintext 16) +-- > +-- > -- | And back, with two different failures. Left is this code used +-- > -- wrongly -- a key of the wrong length, a mode the cipher has not got. +-- > -- Nothing is a message that is not the one that was sent; it carries no +-- > -- plaintext and says nothing about which byte was wrong, both of which +-- > -- are the point. +-- > decrypt +-- > :: (ByteArray key, ByteArrayAccess nonce, ByteArrayAccess aad, ByteArray ba) +-- > => key -> nonce -> aad -> AuthTag -> ba -> Either CryptoError (Maybe ba) +-- > decrypt key nonce aad tag ciphertext = do +-- > cipher <- eitherCryptoError (cipherInit key) :: Either CryptoError AES256 +-- > aead <- eitherCryptoError (aeadInit AEAD_GCM cipher nonce) +-- > return (aeadSimpleDecrypt aead aad ciphertext tag) +-- > +-- > exampleAES256GCM :: ByteString -> IO () +-- > exampleAES256GCM msg = do +-- > key <- genSecretKey (undefined :: AES256) +-- > nonce <- genNonce +-- > let aad = "to: alice" :: ByteString +-- > case encrypt key nonce aad msg of +-- > Left err -> error (show err) +-- > Right (tag, ciphertext) -> do +-- > putStrLn $ "ciphertext: " ++ show ciphertext +-- > putStrLn $ " tag: " ++ show tag +-- > putStrLn $ " recovered: " +-- > ++ show (decrypt key nonce aad tag ciphertext) +-- > -- The same bytes and the same tag, with one thing changed +-- > -- that was never encrypted: Right Nothing. +-- > putStrLn $ "redirected: " +-- > ++ show (decrypt key nonce ("to: eve" :: ByteString) tag ciphertext) +-- +-- The two functions above work for any cipher that has an AEAD mode; what +-- changes is the mode given to 'Crypto.Cipher.Types.aeadInit'. +-- "Crypto.Cipher.ChaChaPoly1305" is the one to prefer on a machine with no +-- AES instructions, and "Crypto.Cipher.AESGCMSIV" is the one that survives +-- a repeated nonce, at the price of needing the whole message before it can +-- begin. + +-- $comparing_secrets +-- +-- Comparing two byte strings with '==' stops at the first byte that +-- differs, so how long it takes says where that byte was. Against an +-- authentication tag that is the whole secret: someone who can send a guess +-- and time the answer finds the first byte in a few hundred tries, then the +-- second, and has a tag that was supposed to cost 2^128 in a few thousand. +-- +-- crypton's own authentication types already compare in constant time, so +-- for those there is nothing to do: 'Crypto.MAC.HMAC.HMAC', +-- 'Crypto.MAC.CMAC.CMAC', 'Crypto.MAC.Poly1305.Auth' and +-- 'Crypto.Cipher.Types.AuthTag' have an 'Eq' that looks at every byte +-- whatever it finds. What needs care is a tag that arrives as bytes, and +-- 'Data.ByteArray.constEq' is the comparison for it. +-- +-- > {-# LANGUAGE OverloadedStrings #-} +-- > +-- > import Crypto.Hash.Algorithms (SHA256) +-- > import Crypto.MAC.HMAC (HMAC, hmac) +-- > +-- > import qualified Data.ByteArray as BA +-- > import Data.ByteString (ByteString) +-- > +-- > -- | The tag came off the wire as bytes, so it is compared as bytes. +-- > authentic :: ByteString -> ByteString -> ByteString -> Bool +-- > authentic key message tag = +-- > BA.constEq tag (hmac key message :: HMAC SHA256) +-- > +-- > -- | Once it has been parsed into the library's own type, (==) is +-- > -- already the constant-time comparison. +-- > authentic' :: ByteString -> ByteString -> HMAC SHA256 -> Bool +-- > authentic' key message tag = tag == hmac key message + +-- $password_storage +-- +-- A password is not a key. It is short and it is guessable, and whoever +-- takes the database can try every likely one without being watched. What +-- answers that is a function that is deliberately expensive to compute, and +-- crypton has four: "Crypto.KDF.BCrypt", "Crypto.KDF.Scrypt", +-- "Crypto.KDF.Argon2" and "Crypto.KDF.PBKDF2". A plain hash is not one of +-- them, however many times it is applied by hand. +-- +-- bcrypt leaves the least to get wrong, because the record it returns +-- carries the salt and the cost inside it: +-- +-- > import Crypto.KDF.BCrypt (hashPassword, validatePassword) +-- > +-- > import Data.ByteString (ByteString) +-- > +-- > -- | What goes in the database. The salt is drawn inside and ends up in +-- > -- the result, so two accounts with the same password do not look alike. +-- > register :: ByteString -> IO ByteString +-- > register password = hashPassword 12 password +-- > +-- > -- | And what is checked against it. The cost comes out of the stored +-- > -- record, so raising it for new accounts leaves the old ones working. +-- > login :: ByteString -> ByteString -> Bool +-- > login password stored = validatePassword password stored +-- +-- Argon2 is the stronger choice and the one to pick for something new: it +-- asks for memory as well as time, which is what takes the advantage away +-- from the hardware that bcrypt's small working set leaves room for. +-- Nothing is encoded for the caller, though -- the salt and the options are +-- theirs to store, and without all three the hash cannot be recomputed when +-- the user comes back. +-- +-- > import Crypto.Error (CryptoFailable) +-- > import qualified Crypto.KDF.Argon2 as Argon2 -- > import qualified Crypto.Random.Types as CRT -- > --- > import Data.ByteArray (ByteArray) -- > import Data.ByteString (ByteString) -- > --- > -- | Not required, but most general implementation --- > data Key c a where --- > Key :: (BlockCipher c, ByteArray a) => a -> Key c a +-- > -- | Argon2id, which is the variant to prefer: it resists both a machine +-- > -- built to guess and a process watching the cache. +-- > options :: Argon2.Options +-- > options = Argon2.defaultOptions{Argon2.variant = Argon2.Argon2id} -- > --- > -- | Generates a string of bytes (key) of a specific length for a given block cipher --- > genSecretKey :: forall m c a. (CRT.MonadRandom m, BlockCipher c, ByteArray a) => c -> Int -> m (Key c a) --- > genSecretKey _ = fmap Key . CRT.getRandomBytes +-- > newSalt :: CRT.MonadRandom m => m ByteString +-- > newSalt = CRT.getRandomBytes 16 -- > --- > -- | Generate a random initialization vector for a given block cipher --- > genRandomIV :: forall m c. (CRT.MonadRandom m, BlockCipher c) => c -> m (Maybe (IV c)) --- > genRandomIV _ = do --- > bytes :: ByteString <- CRT.getRandomBytes $ blockSize (undefined :: c) --- > return $ makeIV bytes +-- > derive :: ByteString -> ByteString -> CryptoFailable ByteString +-- > derive salt password = Argon2.hash options password salt 32 + +-- $key_derivation +-- +-- HKDF turns one secret into as many keys as a protocol needs. It is for +-- material that is already unguessable -- what comes out of a +-- Diffie-Hellman, or a key already agreed -- and it is deliberately cheap, +-- which is exactly what makes it the wrong thing for a password. Those go +-- to the section above. +-- +-- The info string is what keeps the outputs independent: the same secret +-- with a different info gives an unrelated key, so each use of a secret +-- names itself there. +-- +-- > {-# LANGUAGE OverloadedStrings #-} -- > --- > -- | Initialize a block cipher --- > initCipher :: (BlockCipher c, ByteArray a) => Key c a -> Either CryptoError c --- > initCipher (Key k) = case cipherInit k of --- > CryptoFailed e -> Left e --- > CryptoPassed a -> Right a +-- > import Crypto.Hash.Algorithms (SHA256) +-- > import qualified Crypto.KDF.HKDF as HKDF -- > --- > encrypt :: (BlockCipher c, ByteArray a) => Key c a -> IV c -> a -> Either CryptoError a --- > encrypt secretKey initIV msg = --- > case initCipher secretKey of --- > Left e -> Left e --- > Right c -> Right $ ctrCombine c initIV msg +-- > import Data.ByteArray (ScrubbedBytes) +-- > import Data.ByteString (ByteString) -- > --- > decrypt :: (BlockCipher c, ByteArray a) => Key c a -> IV c -> a -> Either CryptoError a --- > decrypt = encrypt +-- > -- | One shared secret in, two unrelated keys out. +-- > directionKeys :: ByteString -> ByteString -> (ScrubbedBytes, ScrubbedBytes) +-- > directionKeys salt shared = (keyFor "client write", keyFor "server write") +-- > where +-- > prk = HKDF.extract salt shared :: HKDF.PRK SHA256 +-- > keyFor info = HKDF.expand prk (info :: ByteString) 32 + +-- $digital_signatures +-- +-- Ed25519 is the one to reach for. The keys are thirty-two bytes, there is +-- nothing to choose and nothing to encode, and signing needs no randomness, +-- so it cannot be ruined by a bad source of it. "Crypto.PubKey.Ed448" is +-- the same shape at a larger size, "Crypto.PubKey.ECDSA" and +-- "Crypto.PubKey.RSA.PSS" are there for protocols that ask for them, and +-- "Crypto.PubKey.MLDSA" is the post-quantum one. +-- +-- > {-# LANGUAGE OverloadedStrings #-} -- > --- > exampleAES256 :: ByteString -> IO () --- > exampleAES256 msg = do --- > -- secret key needs 256 bits (32 * 8) --- > secretKey <- genSecretKey (undefined :: AES256) 32 --- > mInitIV <- genRandomIV (undefined :: AES256) --- > case mInitIV of --- > Nothing -> error "Failed to generate and initialization vector." --- > Just initIV -> do --- > let encryptedMsg = encrypt secretKey initIV msg --- > decryptedMsg = decrypt secretKey initIV =<< encryptedMsg --- > case (,) <$> encryptedMsg <*> decryptedMsg of --- > Left err -> error $ show err --- > Right (eMsg, dMsg) -> do --- > putStrLn $ "Original Message: " ++ show msg --- > putStrLn $ "Message after encryption: " ++ show eMsg --- > putStrLn $ "Message after decryption: " ++ show dMsg +-- > import Crypto.Error (throwCryptoError) +-- > import qualified Crypto.PubKey.Ed25519 as Ed25519 +-- > +-- > import qualified Data.ByteArray as BA +-- > import Data.ByteString (ByteString) +-- > +-- > exampleEd25519 :: ByteString -> IO () +-- > exampleEd25519 msg = do +-- > sk <- Ed25519.generateSecretKey +-- > let pk = Ed25519.toPublic sk +-- > sig = Ed25519.sign sk pk msg +-- > print (Ed25519.verify pk msg sig) +-- > -- The same signature against a message one byte longer: False. +-- > print (Ed25519.verify pk (msg <> "!") sig) +-- > +-- > -- | A secret key on its way to storage. Printing one does not reveal +-- > -- it -- see the first section -- so this is the way out. +-- > store :: Ed25519.SecretKey -> ByteString +-- > store = BA.convert +-- > +-- > -- | And the way back in, which is checked, because the bytes read from +-- > -- a file may be anything at all. +-- > load :: ByteString -> Ed25519.SecretKey +-- > load = throwCryptoError . Ed25519.secretKey -- $combining_primitives -- -- This example shows how to use Curve25519, XSalsa and Poly1305 primitives to -- emulate NaCl's @crypto_box@ construct. -- +-- It is here to show how the pieces fit together, not as something to +-- deploy. An authenticated encryption scheme assembled by hand is the kind +-- of code that is wrong in ways no test notices; for actual use, +-- "Crypto.Cipher.ChaChaPoly1305" does this job with the mistakes already +-- made. +-- -- > import qualified Data.ByteArray as BA -- > import Data.ByteString (ByteString) -- > import qualified Data.ByteString as B @@ -167,6 +376,9 @@ -- > -- > -- | Build a @crypto_box@ packet encrypting the specified content with a -- > -- 192-bit nonce, receiver public key and sender private key. +-- > crypto_box +-- > :: ByteString -> ByteString -> X25519.PublicKey -> X25519.SecretKey +-- > -> ByteString -- > crypto_box content nonce pk sk = BA.convert tag `B.append` c -- > where -- > zero = B.replicate 16 0 @@ -181,6 +393,9 @@ -- > -- > -- | Try to open a @crypto_box@ packet and recover the content using the -- > -- 192-bit nonce, sender public key and receiver private key. +-- > crypto_box_open +-- > :: ByteString -> ByteString -> X25519.PublicKey -> X25519.SecretKey +-- > -> Maybe ByteString -- > crypto_box_open packet nonce pk sk -- > | B.length packet < 16 = Nothing -- > | BA.constEq tag' tag = Just content diff --git a/README.md b/README.md --- a/README.md +++ b/README.md @@ -14,33 +14,49 @@ Side channels ------------- +### AES + AES is where this matters most, and which implementation runs is decided at runtime from what the processor has. -On x86-64 with AES-NI and carry-less multiply, and on AArch64 with the ARMv8 -cryptographic extension, AES and GHASH are instructions rather than tables. -crypton's AES and AES-GCM then make no branch and no memory access that -depends on the key or on the data: the secrets stay in vector registers and -never reach one a branch can test, which the generated code is checked -against. Every x86-64 part since about 2010 and every AArch64 part in -ordinary use has these. +On x86-64 with AES-NI and carry-less multiply, on AArch64 and on 32-bit ARM +with the ARMv8 cryptographic extension, and on ppc64le with the vector +instructions PowerISA 2.07 brought to POWER8, AES and GHASH are instructions +rather than software. crypton's AES and AES-GCM then make no branch and no +memory access that depends on the key or on the data: the secrets stay in +vector registers and never reach one a branch can test. Every x86-64 part +since about 2010 and every AArch64 part in ordinary use has these, and the +ppc64le path is little-endian only. -Where neither is present crypton falls back to a table-driven AES, which -indexes a 256-byte substitution table with data derived from the key and the -input. **That is not constant time**, and on a machine where an attacker can -observe the cache it is open to a timing attack. The fallback exists so that -the library builds and runs everywhere; it is not meant for a setting where -that matters. +Where none of them is there, crypton computes AES and GHASH without a table. +The portable AES is bitsliced -- the S-box is boolean algebra over four +blocks held across eight 64-bit words -- and the GF(2^128) multiply is built +from shifts, masks and integer multiplies. Neither looks anything up, so +neither derives an address from a secret, and **the portable path is constant +time as well**. The code is BearSSL's, under MIT, in `cbits/bearssl`. -`Crypto.System.CPU.processorOptions` says which is in use. `AESNI` in that -list means the instruction path, and `PCLMUL` that GHASH has its instruction -too; without `AESNI` it is the tables. The list also reports `RDRAND`, which -is unrelated to this. +The constant-time harness in `cbits/tests/ct` runs the same driver against +the portable implementation and against each architecture's instructions, and +every one of them is required to report nothing at all. +So the answer does not depend on the machine any more, and what is left to +ask is whether AES is fast on it. `hasAESAcceleration` and +`hasGHASHAcceleration` answer that without naming an architecture. +`processorOptions` says what was found, in the processor's own names -- +`AESNI` and `PCLMUL` on x86, `ARMAES` and `ARMPMULL` on AArch64 and 32-bit +ARM, `PPCAES` and `PPCVPMSUM` on ppc64le -- along with everything else it was +asked about, which is unrelated to this. + ghci> import Crypto.System.CPU - ghci> processorOptions - [AESNI,PCLMUL] + ghci> hasAESAcceleration + True + ghci> processorOptions -- on an x86-64 machine + [AESNI,PCLMUL,SSSE3,AVX,AVX2,SHANI,MOVBE,ADX,VAES] + ghci> processorOptions -- and on an AArch64 one + [NEON,ARMAES,ARMPMULL,ARMSHA1,ARMSHA2,ARMSHA512] +### RSA + RSA is the other place to know about, and there the choice is the caller's. The private key operations in `Crypto.PubKey.RSA.PKCS15`, `.OAEP` and `.PSS` take a `Maybe Blinder`, and `Nothing` is no harder to write than the safe @@ -62,10 +78,14 @@ ones to reach for. Pass `Nothing` only where the input is not attacker controlled and you have decided that it is not. -The RSA rows in the tables below are the unblinded path. A blinder costs one -more exponentiation, by the public exponent, which is the cheap direction: -measured on the M4, signing goes from about 460 to about 476 microseconds, -under four per cent. +The RSA rows in the tables below are the unblinded path. A fresh blinder +costs one more exponentiation, by the public exponent, which is the cheap +direction: measured on the M4, signing goes from about 316 to about 331 +microseconds, under five per cent. `signSafer` and `decryptSafer` pay less +than that from 2.2.4: they keep the blinder of a key they have seen and +square it for the next use, as OpenSSL does, drawing a fresh one every 32 +uses. `Crypto.PubKey.RSA.Blinded` gives a caller the same thing to hold, a +`BlindedKey` passed along from one operation to the next. Performance ----------- @@ -95,31 +115,33 @@ Throughput in MB/s, **higher is better**: -| | crypton 1.1.5 | crypton 2.1.5 | OpenSSL | 2.1.5 / OpenSSL | +| | crypton 1.1.5 | crypton 2.2.4 | OpenSSL | 2.2.4 / OpenSSL | | --- | ---: | ---: | ---: | ---: | -| AES-128-GCM | 1362 | **6038** | 4055 | 1.49 | -| AES-256-GCM | 1093 | **5462** | 3770 | 1.45 | -| ChaCha20-Poly1305 | 399 | 2211 | 2229 | 0.99 | -| SHA-1 | 727 | 1678 | 1673 | 1.00 | -| SHA-256 | 290 | 1585 | 1579 | 1.00 | -| SHA-512 | 463 | 804 | 751 | 1.07 | -| SHA3-256 | 109 | 424 | 425 | 1.00 | +| AES-128-GCM | 1335 | **6041** | 4066 | 1.49 | +| AES-256-GCM | 1093 | **5492** | 3766 | 1.46 | +| ChaCha20-Poly1305 | 399 | 2195 | 2222 | 0.99 | +| SHA-1 | 727 | 1663 | 1672 | 0.99 | +| SHA-256 | 289 | 1585 | 1576 | 1.01 | +| SHA-512 | 448 | 770 | 749 | 1.03 | +| SHA3-256 | 109 | 421 | 429 | 0.98 | Time per operation in microseconds, **lower is better**: -| | crypton 1.1.5 | crypton 2.1.5 | OpenSSL | OpenSSL / 2.1.5 | +| | crypton 1.1.5 | crypton 2.2.4 | OpenSSL | OpenSSL / 2.2.4 | | --- | ---: | ---: | ---: | ---: | -| X25519 | 45.31 | 28.41 | 36.48 | 1.28 | -| ECDH P-256 | 165.4 | 51.14 | 51.65 | 1.01 | -| ECDH P-384 | 2278 | **165.0** | 847.5 | 5.13 | -| Ed25519 sign | 30.03 | 18.62 | 33.71 | 1.81 | -| Ed25519 verify | 48.05 | 47.66 | 110.6 | 2.32 | -| ECDSA P-256 sign | 81.70 | 18.96 | 21.87 | 1.15 | -| ECDSA P-256 verify | 233.3 | 70.47 | 67.52 | 0.96 | -| ECDSA P-384 sign | 2264 | **303.4** | 890.1 | 2.93 | -| ECDSA P-384 verify | 2676 | **471.4** | 721.5 | 1.53 | -| RSA-2048 sign/decrypt | 759.4 | 612.0 | 659.4 | 1.08 | -| RSA-2048 verify/encrypt | 33.56 | 30.21 | 18.86 | 0.62 | +| X25519 | 44.38 | 28.29 | 36.61 | 1.29 | +| ECDH P-256 | 164.0 | 51.31 | 51.70 | 1.01 | +| ECDH P-384 | 2242 | **164.3** | 836.1 | 5.09 | +| Ed25519 sign | 43.86 | 18.50 | 33.74 | 1.82 | +| Ed25519 verify | 48.30 | 48.43 | 109.8 | 2.27 | +| ECDSA P-256 sign | 81.80 | 19.72 | 21.84 | 1.11 | +| ECDSA P-256 verify | 231.8 | 70.47 | 67.55 | 0.96 | +| ECDSA P-384 sign | 2267 | **305.2** | 879.5 | 2.88 | +| ECDSA P-384 verify | 2698 | **473.6** | 725.2 | 1.53 | +| RSA-2048 sign/decrypt | 760.1 | 612.3 | 659.6 | 1.08 | +| RSA-2048 verify/encrypt | 33.61 | 23.50 | 18.88 | 0.80 | +| RSA-3072 sign/decrypt | 2499 | 2324 | 2012 | 0.87 | +| RSA-3072 verify/encrypt | 66.80 | 61.91 | 40.14 | 0.65 | ### AArch64 @@ -128,35 +150,37 @@ Throughput in MB/s, **higher is better**: -| | crypton 1.1.5 | crypton 2.1.5 | OpenSSL | 2.1.5 / OpenSSL | +| | crypton 1.1.5 | crypton 2.2.4 | OpenSSL | 2.2.4 / OpenSSL | | --- | ---: | ---: | ---: | ---: | -| AES-128-GCM | 127 | **12422** | 10846 | 1.15 | -| AES-256-GCM | 98 | **9721** | 9197 | 1.06 | -| ChaCha20-Poly1305 | 771 | 2319 | 2250 | 1.03 | -| SHA-1 | 1209 | 3389 | 3361 | 1.01 | -| SHA-256 | 474 | 3400 | 3362 | 1.01 | -| SHA-512 | 730 | 1880 | 1883 | 1.00 | -| SHA3-256 | 550 | 1075 | 1065 | 1.01 | +| AES-128-GCM | 131 | **12052** | 10922 | 1.10 | +| AES-256-GCM | 99 | **10192** | 9208 | 1.11 | +| ChaCha20-Poly1305 | 783 | 2342 | 2261 | 1.04 | +| SHA-1 | 1234 | 3383 | 3347 | 1.01 | +| SHA-256 | 486 | 3392 | 3354 | 1.01 | +| SHA-512 | 742 | 1869 | 1860 | 1.00 | +| SHA3-256 | 552 | 1096 | 1092 | 1.00 | Time per operation in microseconds, **lower is better**: -| | crypton 1.1.5 | crypton 2.1.5 | OpenSSL | OpenSSL / 2.1.5 | +| | crypton 1.1.5 | crypton 2.2.4 | OpenSSL | OpenSSL / 2.2.4 | | --- | ---: | ---: | ---: | ---: | -| X25519 | 18.27 | **12.22** | 15.53 | 1.27 | -| ECDH P-256 | 68.70 | **20.43** | 24.77 | 1.21 | -| ECDH P-384 | 3328 | **73.50** | 372.6 | 5.07 | -| Ed25519 sign | 13.58 | **7.75** | 13.23 | 1.71 | -| Ed25519 verify | 18.28 | 18.17 | 34.76 | 1.91 | -| ECDSA P-256 sign | 31.97 | **6.55** | 10.92 | 1.67 | -| ECDSA P-256 verify | 95.63 | **26.80** | 32.68 | 1.22 | -| ECDSA P-384 sign | 3219 | **124.1** | 394.2 | 3.18 | -| ECDSA P-384 verify | 3870 | **203.1** | 326.5 | 1.61 | -| RSA-2048 sign/decrypt | 447.9 | 460.1 | 319.9 | 0.70 | -| RSA-2048 verify/encrypt | 18.23 | 15.12 | 8.405 | 0.56 | +| X25519 | 17.92 | **12.02** | 15.32 | 1.28 | +| ECDH P-256 | 67.50 | **19.05** | 24.36 | 1.28 | +| ECDH P-384 | 3256 | **71.78** | 373.1 | 5.20 | +| Ed25519 sign | 13.38 | **7.717** | 13.31 | 1.72 | +| Ed25519 verify | 18.32 | 18.18 | 34.77 | 1.91 | +| ECDSA P-256 sign | 31.08 | **6.500** | 10.66 | 1.64 | +| ECDSA P-256 verify | 94.27 | **26.58** | 32.25 | 1.21 | +| ECDSA P-384 sign | 3225 | **124.0** | 394.2 | 3.18 | +| ECDSA P-384 verify | 3866 | **203.4** | 332.3 | 1.63 | +| RSA-2048 sign/decrypt | 451.3 | 318.4 | 321.5 | 1.01 | +| RSA-2048 verify/encrypt | 18.15 | 11.12 | 8.405 | 0.76 | +| RSA-3072 sign/decrypt | 1473 | 937.4 | 931.1 | 0.99 | +| RSA-3072 verify/encrypt | 47.95 | 43.63 | 17.89 | 0.41 | ### What the numbers say -There are two changes behind the 1.1.5 column and the 2.1.5 one, not a +There are two changes behind the 1.1.5 column and the 2.2.4 one, not a single steady improvement. The first, in 2.0.0, was a rewrite: the bulk algorithms moved into C, the @@ -184,25 +208,34 @@ it is Apache-2.0 only, and Intel and CloudFlare hold copyright in it besides OpenSSL, so nobody is in a position to relicense it. -Where crypton is behind, which is now the RSA rows on both architectures and -ECDSA P-256 verification on x86-64, there is one reason. The AES-GCM rows +crypton is behind in a few rows now: RSA verification on both architectures, +and RSA-3072 signing and ECDSA P-256 verification on x86-64. The AES-GCM rows were the other half of this section until 2.1.5; they are ahead on both machines now, and what the instructions do is still worth setting out. *RSA.* 2.0.0 made signing slower than 1.1.5 on purpose: its modular exponentiation stopped indexing a table with the bits of the exponent, and -hiding the exponent is what the difference bought. On x86-64 that cost is -more than repaid -- s2n-bignum's Montgomery multiplication is twice the C's, -because the C cannot form the two carry chains `ADCX` and `ADOX` give, and -2.1.5 signs in less than 1.1.5 took while keeping what 2.0.0 gained. On -AArch64 there is nothing to use: s2n-bignum has no generic routine for it, -and the same five that help on x86-64 measure level with the C there, so the -C stays and the gap with it. No portable C closes that gap either -- the -measurements are in -[#275](https://github.com/kazu-yamamoto/crypton/issues/275). Verification -does not move much either way: its exponent is 65537, seventeen bits, and -there is no exponentiation to speak of. +hiding the exponent is what the difference bought. 2.2.4 has repaid that +cost on both machines, by different means. On x86-64 s2n-bignum's +Montgomery multiplication is twice the C's, because the C cannot form the two +carry chains `ADCX` and `ADOX` give, and 2.2.4 signs in less than 1.1.5 +took, at 3072 bits as well as 2048. On AArch64 s2n-bignum's routines measure +no better than the C, so the gain there is the C's own: the multiplication +and squaring for each key size are generated and unrolled, with the +reduction folded into each column as in Koç, Acar and Kaliski's FIPS method, +and signing is level with OpenSSL. The measurements are in +[#275](https://github.com/kazu-yamamoto/crypton/issues/275). +On processors with AVX-512 IFMA, Intel Xeons from Ice Lake and AMD from Zen 4, +2.2.4 computes the two halves of the CRT together in 52-bit digits, Gueron and +Krasnov's method and the one OpenSSL takes there. Neither machine above has +IFMA, so neither table shows it. + +Verification's exponent is 65537, seventeen bits, so there is little +exponentiation to speed up and the work around it counts. 2.2.4 keeps R^2, +the Montgomery constant of a modulus, for the moduli it has seen, which is +most of what the RSA-2048 verification rows gained. + *The wide AES instructions.* `VAES` and `VPCLMULQDQ` do two blocks where `AES-NI` and `PCLMULQDQ` do one, and four in their 512-bit form. crypton uses the 256-bit form where the processor has it, which is Zen 3 and Ice Lake @@ -220,7 +253,7 @@ takes a block at a time where crypton takes two. The same idea as theirs, one step further down the feature ladder; not a better one. -The 512-bit path arrived after 2.1.2, so it is in the 2.1.5 column -- but +The 512-bit path arrived after 2.1.2, so it is in the 2.2.4 column -- but neither machine in the tables above has AVX-512, so neither column shows it. On the runners that do, measured over 16 KiB in MB/s: an EPYC 9V45 (Zen 5) goes from 9616 to 14268 with it, a Xeon 6973P-C from 8095 to 9848, a Xeon @@ -245,10 +278,33 @@ key from the secret key every time it signs, so that a caller who passes a public key that does not match cannot be made to leak the private one. That costs a second scalar multiplication, which OpenSSL's signing does not pay -- -and the row is still 1.71 on AArch64 and 1.81 on x86-64, so the safety is had +and the row is still 1.72 on AArch64 and 1.82 on x86-64, so the safety is had for nothing here rather than paid for. +Ed25519 verification wanted one too, for an unhappier reason, and 2.2.1 +settles it. On x86-64 the row was about eight per cent slower from 2.1.8 +onwards: 2.1.5, 2.1.6 and 2.1.7 all measure near 48 microseconds there, +2.1.8, 2.1.10 and 2.2.0 all measure near 52, and 2.2.1 is near 48 again. +Nothing in Ed25519 ever changed -- no commit touches `cbits/ed25519` across +any of it, and AArch64 never moved, the compiled `ed25519.o` there being +byte for byte the same object either side of 2.1.8. What moved was a line +in this package's cabal file. The ML-KEM and ML-DSA backends that 2.1.8 +vendored want `-mavx2 -mbmi2`, and they were given as `cc-options` inside an +`if arch(x86_64)` block -- but cc-options in a conditional belong to the +component and not to the block, so every C file in the library was compiled +with AVX2 allowed, `cbits/ed25519` among them, and what the compiler made of +it that way is slower on this machine than what it replaced. Signing, which +is the same file and a different routine, is where it always was. 2.2.1 +narrowed the flags to the assembly that asked for them, which is also what +stopped a binary built on one x86-64 machine from dying on another, and the +row came back. + SHA-1 is in the tables because a number of protocols and file formats still ask for it, not because it is a good choice for anything new. The algorithms that nothing should ask for any more -- MD5, 3DES, RC4, CBC mode -- are left out. + +ML-KEM and ML-DSA are left out too, though 2.1.8 brought both and they are +fast -- `mlkem-native` and `mldsa-native`, from the PQ Code Package. No +comparison is reported here: the implementations to measure against are +still developing. diff --git a/cbits/aes/generic.c b/cbits/aes/generic.c --- a/cbits/aes/generic.c +++ b/cbits/aes/generic.c @@ -27,420 +27,191 @@ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * - * AES implementation + * AES, portable, for machines whose processor has no AES instructions or + * whose build was not given them. + * + * This was a table-driven implementation: an S-box indexed by a byte of the + * state, in every round and in the key expansion, which is what made it fast + * and what made it variable-time. It is now BearSSL's bitsliced aes_ct64, + * which holds four blocks interleaved across eight 64-bit words and computes + * the S-box as boolean algebra -- no table, and so no address derived from a + * secret. See cbits/bearssl/README.md. + * + * What is here is only the glue. Nothing in this file branches or indexes on + * the key or the data. */ #include -#include +#include #include -#include - -static uint8_t sbox[256] = { - 0x63, 0x7c, 0x77, 0x7b, 0xf2, 0x6b, 0x6f, 0xc5, 0x30, 0x01, 0x67, 0x2b, 0xfe, - 0xd7, 0xab, 0x76, 0xca, 0x82, 0xc9, 0x7d, 0xfa, 0x59, 0x47, 0xf0, 0xad, 0xd4, - 0xa2, 0xaf, 0x9c, 0xa4, 0x72, 0xc0, 0xb7, 0xfd, 0x93, 0x26, 0x36, 0x3f, 0xf7, - 0xcc, 0x34, 0xa5, 0xe5, 0xf1, 0x71, 0xd8, 0x31, 0x15, 0x04, 0xc7, 0x23, 0xc3, - 0x18, 0x96, 0x05, 0x9a, 0x07, 0x12, 0x80, 0xe2, 0xeb, 0x27, 0xb2, 0x75, 0x09, - 0x83, 0x2c, 0x1a, 0x1b, 0x6e, 0x5a, 0xa0, 0x52, 0x3b, 0xd6, 0xb3, 0x29, 0xe3, - 0x2f, 0x84, 0x53, 0xd1, 0x00, 0xed, 0x20, 0xfc, 0xb1, 0x5b, 0x6a, 0xcb, 0xbe, - 0x39, 0x4a, 0x4c, 0x58, 0xcf, 0xd0, 0xef, 0xaa, 0xfb, 0x43, 0x4d, 0x33, 0x85, - 0x45, 0xf9, 0x02, 0x7f, 0x50, 0x3c, 0x9f, 0xa8, 0x51, 0xa3, 0x40, 0x8f, 0x92, - 0x9d, 0x38, 0xf5, 0xbc, 0xb6, 0xda, 0x21, 0x10, 0xff, 0xf3, 0xd2, 0xcd, 0x0c, - 0x13, 0xec, 0x5f, 0x97, 0x44, 0x17, 0xc4, 0xa7, 0x7e, 0x3d, 0x64, 0x5d, 0x19, - 0x73, 0x60, 0x81, 0x4f, 0xdc, 0x22, 0x2a, 0x90, 0x88, 0x46, 0xee, 0xb8, 0x14, - 0xde, 0x5e, 0x0b, 0xdb, 0xe0, 0x32, 0x3a, 0x0a, 0x49, 0x06, 0x24, 0x5c, 0xc2, - 0xd3, 0xac, 0x62, 0x91, 0x95, 0xe4, 0x79, 0xe7, 0xc8, 0x37, 0x6d, 0x8d, 0xd5, - 0x4e, 0xa9, 0x6c, 0x56, 0xf4, 0xea, 0x65, 0x7a, 0xae, 0x08, 0xba, 0x78, 0x25, - 0x2e, 0x1c, 0xa6, 0xb4, 0xc6, 0xe8, 0xdd, 0x74, 0x1f, 0x4b, 0xbd, 0x8b, 0x8a, - 0x70, 0x3e, 0xb5, 0x66, 0x48, 0x03, 0xf6, 0x0e, 0x61, 0x35, 0x57, 0xb9, 0x86, - 0xc1, 0x1d, 0x9e, 0xe1, 0xf8, 0x98, 0x11, 0x69, 0xd9, 0x8e, 0x94, 0x9b, 0x1e, - 0x87, 0xe9, 0xce, 0x55, 0x28, 0xdf, 0x8c, 0xa1, 0x89, 0x0d, 0xbf, 0xe6, 0x42, - 0x68, 0x41, 0x99, 0x2d, 0x0f, 0xb0, 0x54, 0xbb, 0x16 -}; - -static uint8_t rsbox[256] = { - 0x52, 0x09, 0x6a, 0xd5, 0x30, 0x36, 0xa5, 0x38, 0xbf, 0x40, 0xa3, 0x9e, 0x81, - 0xf3, 0xd7, 0xfb, 0x7c, 0xe3, 0x39, 0x82, 0x9b, 0x2f, 0xff, 0x87, 0x34, 0x8e, - 0x43, 0x44, 0xc4, 0xde, 0xe9, 0xcb, 0x54, 0x7b, 0x94, 0x32, 0xa6, 0xc2, 0x23, - 0x3d, 0xee, 0x4c, 0x95, 0x0b, 0x42, 0xfa, 0xc3, 0x4e, 0x08, 0x2e, 0xa1, 0x66, - 0x28, 0xd9, 0x24, 0xb2, 0x76, 0x5b, 0xa2, 0x49, 0x6d, 0x8b, 0xd1, 0x25, 0x72, - 0xf8, 0xf6, 0x64, 0x86, 0x68, 0x98, 0x16, 0xd4, 0xa4, 0x5c, 0xcc, 0x5d, 0x65, - 0xb6, 0x92, 0x6c, 0x70, 0x48, 0x50, 0xfd, 0xed, 0xb9, 0xda, 0x5e, 0x15, 0x46, - 0x57, 0xa7, 0x8d, 0x9d, 0x84, 0x90, 0xd8, 0xab, 0x00, 0x8c, 0xbc, 0xd3, 0x0a, - 0xf7, 0xe4, 0x58, 0x05, 0xb8, 0xb3, 0x45, 0x06, 0xd0, 0x2c, 0x1e, 0x8f, 0xca, - 0x3f, 0x0f, 0x02, 0xc1, 0xaf, 0xbd, 0x03, 0x01, 0x13, 0x8a, 0x6b, 0x3a, 0x91, - 0x11, 0x41, 0x4f, 0x67, 0xdc, 0xea, 0x97, 0xf2, 0xcf, 0xce, 0xf0, 0xb4, 0xe6, - 0x73, 0x96, 0xac, 0x74, 0x22, 0xe7, 0xad, 0x35, 0x85, 0xe2, 0xf9, 0x37, 0xe8, - 0x1c, 0x75, 0xdf, 0x6e, 0x47, 0xf1, 0x1a, 0x71, 0x1d, 0x29, 0xc5, 0x89, 0x6f, - 0xb7, 0x62, 0x0e, 0xaa, 0x18, 0xbe, 0x1b, 0xfc, 0x56, 0x3e, 0x4b, 0xc6, 0xd2, - 0x79, 0x20, 0x9a, 0xdb, 0xc0, 0xfe, 0x78, 0xcd, 0x5a, 0xf4, 0x1f, 0xdd, 0xa8, - 0x33, 0x88, 0x07, 0xc7, 0x31, 0xb1, 0x12, 0x10, 0x59, 0x27, 0x80, 0xec, 0x5f, - 0x60, 0x51, 0x7f, 0xa9, 0x19, 0xb5, 0x4a, 0x0d, 0x2d, 0xe5, 0x7a, 0x9f, 0x93, - 0xc9, 0x9c, 0xef, 0xa0, 0xe0, 0x3b, 0x4d, 0xae, 0x2a, 0xf5, 0xb0, 0xc8, 0xeb, - 0xbb, 0x3c, 0x83, 0x53, 0x99, 0x61, 0x17, 0x2b, 0x04, 0x7e, 0xba, 0x77, 0xd6, - 0x26, 0xe1, 0x69, 0x14, 0x63, 0x55, 0x21, 0x0c, 0x7d -}; - -static uint8_t Rcon[] = { - 0x8d, 0x01, 0x02, 0x04, 0x08, 0x10, 0x20, 0x40, 0x80, 0x1b, 0x36, 0x6c, 0xd8, - 0xab, 0x4d, 0x9a, 0x2f, 0x5e, 0xbc, 0x63, 0xc6, 0x97, 0x35, 0x6a, 0xd4, 0xb3, - 0x7d, 0xfa, 0xef, 0xc5, 0x91, 0x39, 0x72, 0xe4, 0xd3, 0xbd, 0x61, 0xc2, 0x9f, - 0x25, 0x4a, 0x94, 0x33, 0x66, 0xcc, 0x83, 0x1d, 0x3a, 0x74, 0xe8, 0xcb, -}; +#include "bearssl/inner.h" +#include "aes/block128.h" +#include "aes/generic.h" -#define G(a,b,c,d,e,f) { a,b,c,d,e,f } -static uint8_t gmtab[256][6] = -{ - G(0x00, 0x00, 0x00, 0x00, 0x00, 0x00), G(0x02, 0x03, 0x09, 0x0b, 0x0d, 0x0e), - G(0x04, 0x06, 0x12, 0x16, 0x1a, 0x1c), G(0x06, 0x05, 0x1b, 0x1d, 0x17, 0x12), - G(0x08, 0x0c, 0x24, 0x2c, 0x34, 0x38), G(0x0a, 0x0f, 0x2d, 0x27, 0x39, 0x36), - G(0x0c, 0x0a, 0x36, 0x3a, 0x2e, 0x24), G(0x0e, 0x09, 0x3f, 0x31, 0x23, 0x2a), - G(0x10, 0x18, 0x48, 0x58, 0x68, 0x70), G(0x12, 0x1b, 0x41, 0x53, 0x65, 0x7e), - G(0x14, 0x1e, 0x5a, 0x4e, 0x72, 0x6c), G(0x16, 0x1d, 0x53, 0x45, 0x7f, 0x62), - G(0x18, 0x14, 0x6c, 0x74, 0x5c, 0x48), G(0x1a, 0x17, 0x65, 0x7f, 0x51, 0x46), - G(0x1c, 0x12, 0x7e, 0x62, 0x46, 0x54), G(0x1e, 0x11, 0x77, 0x69, 0x4b, 0x5a), - G(0x20, 0x30, 0x90, 0xb0, 0xd0, 0xe0), G(0x22, 0x33, 0x99, 0xbb, 0xdd, 0xee), - G(0x24, 0x36, 0x82, 0xa6, 0xca, 0xfc), G(0x26, 0x35, 0x8b, 0xad, 0xc7, 0xf2), - G(0x28, 0x3c, 0xb4, 0x9c, 0xe4, 0xd8), G(0x2a, 0x3f, 0xbd, 0x97, 0xe9, 0xd6), - G(0x2c, 0x3a, 0xa6, 0x8a, 0xfe, 0xc4), G(0x2e, 0x39, 0xaf, 0x81, 0xf3, 0xca), - G(0x30, 0x28, 0xd8, 0xe8, 0xb8, 0x90), G(0x32, 0x2b, 0xd1, 0xe3, 0xb5, 0x9e), - G(0x34, 0x2e, 0xca, 0xfe, 0xa2, 0x8c), G(0x36, 0x2d, 0xc3, 0xf5, 0xaf, 0x82), - G(0x38, 0x24, 0xfc, 0xc4, 0x8c, 0xa8), G(0x3a, 0x27, 0xf5, 0xcf, 0x81, 0xa6), - G(0x3c, 0x22, 0xee, 0xd2, 0x96, 0xb4), G(0x3e, 0x21, 0xe7, 0xd9, 0x9b, 0xba), - G(0x40, 0x60, 0x3b, 0x7b, 0xbb, 0xdb), G(0x42, 0x63, 0x32, 0x70, 0xb6, 0xd5), - G(0x44, 0x66, 0x29, 0x6d, 0xa1, 0xc7), G(0x46, 0x65, 0x20, 0x66, 0xac, 0xc9), - G(0x48, 0x6c, 0x1f, 0x57, 0x8f, 0xe3), G(0x4a, 0x6f, 0x16, 0x5c, 0x82, 0xed), - G(0x4c, 0x6a, 0x0d, 0x41, 0x95, 0xff), G(0x4e, 0x69, 0x04, 0x4a, 0x98, 0xf1), - G(0x50, 0x78, 0x73, 0x23, 0xd3, 0xab), G(0x52, 0x7b, 0x7a, 0x28, 0xde, 0xa5), - G(0x54, 0x7e, 0x61, 0x35, 0xc9, 0xb7), G(0x56, 0x7d, 0x68, 0x3e, 0xc4, 0xb9), - G(0x58, 0x74, 0x57, 0x0f, 0xe7, 0x93), G(0x5a, 0x77, 0x5e, 0x04, 0xea, 0x9d), - G(0x5c, 0x72, 0x45, 0x19, 0xfd, 0x8f), G(0x5e, 0x71, 0x4c, 0x12, 0xf0, 0x81), - G(0x60, 0x50, 0xab, 0xcb, 0x6b, 0x3b), G(0x62, 0x53, 0xa2, 0xc0, 0x66, 0x35), - G(0x64, 0x56, 0xb9, 0xdd, 0x71, 0x27), G(0x66, 0x55, 0xb0, 0xd6, 0x7c, 0x29), - G(0x68, 0x5c, 0x8f, 0xe7, 0x5f, 0x03), G(0x6a, 0x5f, 0x86, 0xec, 0x52, 0x0d), - G(0x6c, 0x5a, 0x9d, 0xf1, 0x45, 0x1f), G(0x6e, 0x59, 0x94, 0xfa, 0x48, 0x11), - G(0x70, 0x48, 0xe3, 0x93, 0x03, 0x4b), G(0x72, 0x4b, 0xea, 0x98, 0x0e, 0x45), - G(0x74, 0x4e, 0xf1, 0x85, 0x19, 0x57), G(0x76, 0x4d, 0xf8, 0x8e, 0x14, 0x59), - G(0x78, 0x44, 0xc7, 0xbf, 0x37, 0x73), G(0x7a, 0x47, 0xce, 0xb4, 0x3a, 0x7d), - G(0x7c, 0x42, 0xd5, 0xa9, 0x2d, 0x6f), G(0x7e, 0x41, 0xdc, 0xa2, 0x20, 0x61), - G(0x80, 0xc0, 0x76, 0xf6, 0x6d, 0xad), G(0x82, 0xc3, 0x7f, 0xfd, 0x60, 0xa3), - G(0x84, 0xc6, 0x64, 0xe0, 0x77, 0xb1), G(0x86, 0xc5, 0x6d, 0xeb, 0x7a, 0xbf), - G(0x88, 0xcc, 0x52, 0xda, 0x59, 0x95), G(0x8a, 0xcf, 0x5b, 0xd1, 0x54, 0x9b), - G(0x8c, 0xca, 0x40, 0xcc, 0x43, 0x89), G(0x8e, 0xc9, 0x49, 0xc7, 0x4e, 0x87), - G(0x90, 0xd8, 0x3e, 0xae, 0x05, 0xdd), G(0x92, 0xdb, 0x37, 0xa5, 0x08, 0xd3), - G(0x94, 0xde, 0x2c, 0xb8, 0x1f, 0xc1), G(0x96, 0xdd, 0x25, 0xb3, 0x12, 0xcf), - G(0x98, 0xd4, 0x1a, 0x82, 0x31, 0xe5), G(0x9a, 0xd7, 0x13, 0x89, 0x3c, 0xeb), - G(0x9c, 0xd2, 0x08, 0x94, 0x2b, 0xf9), G(0x9e, 0xd1, 0x01, 0x9f, 0x26, 0xf7), - G(0xa0, 0xf0, 0xe6, 0x46, 0xbd, 0x4d), G(0xa2, 0xf3, 0xef, 0x4d, 0xb0, 0x43), - G(0xa4, 0xf6, 0xf4, 0x50, 0xa7, 0x51), G(0xa6, 0xf5, 0xfd, 0x5b, 0xaa, 0x5f), - G(0xa8, 0xfc, 0xc2, 0x6a, 0x89, 0x75), G(0xaa, 0xff, 0xcb, 0x61, 0x84, 0x7b), - G(0xac, 0xfa, 0xd0, 0x7c, 0x93, 0x69), G(0xae, 0xf9, 0xd9, 0x77, 0x9e, 0x67), - G(0xb0, 0xe8, 0xae, 0x1e, 0xd5, 0x3d), G(0xb2, 0xeb, 0xa7, 0x15, 0xd8, 0x33), - G(0xb4, 0xee, 0xbc, 0x08, 0xcf, 0x21), G(0xb6, 0xed, 0xb5, 0x03, 0xc2, 0x2f), - G(0xb8, 0xe4, 0x8a, 0x32, 0xe1, 0x05), G(0xba, 0xe7, 0x83, 0x39, 0xec, 0x0b), - G(0xbc, 0xe2, 0x98, 0x24, 0xfb, 0x19), G(0xbe, 0xe1, 0x91, 0x2f, 0xf6, 0x17), - G(0xc0, 0xa0, 0x4d, 0x8d, 0xd6, 0x76), G(0xc2, 0xa3, 0x44, 0x86, 0xdb, 0x78), - G(0xc4, 0xa6, 0x5f, 0x9b, 0xcc, 0x6a), G(0xc6, 0xa5, 0x56, 0x90, 0xc1, 0x64), - G(0xc8, 0xac, 0x69, 0xa1, 0xe2, 0x4e), G(0xca, 0xaf, 0x60, 0xaa, 0xef, 0x40), - G(0xcc, 0xaa, 0x7b, 0xb7, 0xf8, 0x52), G(0xce, 0xa9, 0x72, 0xbc, 0xf5, 0x5c), - G(0xd0, 0xb8, 0x05, 0xd5, 0xbe, 0x06), G(0xd2, 0xbb, 0x0c, 0xde, 0xb3, 0x08), - G(0xd4, 0xbe, 0x17, 0xc3, 0xa4, 0x1a), G(0xd6, 0xbd, 0x1e, 0xc8, 0xa9, 0x14), - G(0xd8, 0xb4, 0x21, 0xf9, 0x8a, 0x3e), G(0xda, 0xb7, 0x28, 0xf2, 0x87, 0x30), - G(0xdc, 0xb2, 0x33, 0xef, 0x90, 0x22), G(0xde, 0xb1, 0x3a, 0xe4, 0x9d, 0x2c), - G(0xe0, 0x90, 0xdd, 0x3d, 0x06, 0x96), G(0xe2, 0x93, 0xd4, 0x36, 0x0b, 0x98), - G(0xe4, 0x96, 0xcf, 0x2b, 0x1c, 0x8a), G(0xe6, 0x95, 0xc6, 0x20, 0x11, 0x84), - G(0xe8, 0x9c, 0xf9, 0x11, 0x32, 0xae), G(0xea, 0x9f, 0xf0, 0x1a, 0x3f, 0xa0), - G(0xec, 0x9a, 0xeb, 0x07, 0x28, 0xb2), G(0xee, 0x99, 0xe2, 0x0c, 0x25, 0xbc), - G(0xf0, 0x88, 0x95, 0x65, 0x6e, 0xe6), G(0xf2, 0x8b, 0x9c, 0x6e, 0x63, 0xe8), - G(0xf4, 0x8e, 0x87, 0x73, 0x74, 0xfa), G(0xf6, 0x8d, 0x8e, 0x78, 0x79, 0xf4), - G(0xf8, 0x84, 0xb1, 0x49, 0x5a, 0xde), G(0xfa, 0x87, 0xb8, 0x42, 0x57, 0xd0), - G(0xfc, 0x82, 0xa3, 0x5f, 0x40, 0xc2), G(0xfe, 0x81, 0xaa, 0x54, 0x4d, 0xcc), - G(0x1b, 0x9b, 0xec, 0xf7, 0xda, 0x41), G(0x19, 0x98, 0xe5, 0xfc, 0xd7, 0x4f), - G(0x1f, 0x9d, 0xfe, 0xe1, 0xc0, 0x5d), G(0x1d, 0x9e, 0xf7, 0xea, 0xcd, 0x53), - G(0x13, 0x97, 0xc8, 0xdb, 0xee, 0x79), G(0x11, 0x94, 0xc1, 0xd0, 0xe3, 0x77), - G(0x17, 0x91, 0xda, 0xcd, 0xf4, 0x65), G(0x15, 0x92, 0xd3, 0xc6, 0xf9, 0x6b), - G(0x0b, 0x83, 0xa4, 0xaf, 0xb2, 0x31), G(0x09, 0x80, 0xad, 0xa4, 0xbf, 0x3f), - G(0x0f, 0x85, 0xb6, 0xb9, 0xa8, 0x2d), G(0x0d, 0x86, 0xbf, 0xb2, 0xa5, 0x23), - G(0x03, 0x8f, 0x80, 0x83, 0x86, 0x09), G(0x01, 0x8c, 0x89, 0x88, 0x8b, 0x07), - G(0x07, 0x89, 0x92, 0x95, 0x9c, 0x15), G(0x05, 0x8a, 0x9b, 0x9e, 0x91, 0x1b), - G(0x3b, 0xab, 0x7c, 0x47, 0x0a, 0xa1), G(0x39, 0xa8, 0x75, 0x4c, 0x07, 0xaf), - G(0x3f, 0xad, 0x6e, 0x51, 0x10, 0xbd), G(0x3d, 0xae, 0x67, 0x5a, 0x1d, 0xb3), - G(0x33, 0xa7, 0x58, 0x6b, 0x3e, 0x99), G(0x31, 0xa4, 0x51, 0x60, 0x33, 0x97), - G(0x37, 0xa1, 0x4a, 0x7d, 0x24, 0x85), G(0x35, 0xa2, 0x43, 0x76, 0x29, 0x8b), - G(0x2b, 0xb3, 0x34, 0x1f, 0x62, 0xd1), G(0x29, 0xb0, 0x3d, 0x14, 0x6f, 0xdf), - G(0x2f, 0xb5, 0x26, 0x09, 0x78, 0xcd), G(0x2d, 0xb6, 0x2f, 0x02, 0x75, 0xc3), - G(0x23, 0xbf, 0x10, 0x33, 0x56, 0xe9), G(0x21, 0xbc, 0x19, 0x38, 0x5b, 0xe7), - G(0x27, 0xb9, 0x02, 0x25, 0x4c, 0xf5), G(0x25, 0xba, 0x0b, 0x2e, 0x41, 0xfb), - G(0x5b, 0xfb, 0xd7, 0x8c, 0x61, 0x9a), G(0x59, 0xf8, 0xde, 0x87, 0x6c, 0x94), - G(0x5f, 0xfd, 0xc5, 0x9a, 0x7b, 0x86), G(0x5d, 0xfe, 0xcc, 0x91, 0x76, 0x88), - G(0x53, 0xf7, 0xf3, 0xa0, 0x55, 0xa2), G(0x51, 0xf4, 0xfa, 0xab, 0x58, 0xac), - G(0x57, 0xf1, 0xe1, 0xb6, 0x4f, 0xbe), G(0x55, 0xf2, 0xe8, 0xbd, 0x42, 0xb0), - G(0x4b, 0xe3, 0x9f, 0xd4, 0x09, 0xea), G(0x49, 0xe0, 0x96, 0xdf, 0x04, 0xe4), - G(0x4f, 0xe5, 0x8d, 0xc2, 0x13, 0xf6), G(0x4d, 0xe6, 0x84, 0xc9, 0x1e, 0xf8), - G(0x43, 0xef, 0xbb, 0xf8, 0x3d, 0xd2), G(0x41, 0xec, 0xb2, 0xf3, 0x30, 0xdc), - G(0x47, 0xe9, 0xa9, 0xee, 0x27, 0xce), G(0x45, 0xea, 0xa0, 0xe5, 0x2a, 0xc0), - G(0x7b, 0xcb, 0x47, 0x3c, 0xb1, 0x7a), G(0x79, 0xc8, 0x4e, 0x37, 0xbc, 0x74), - G(0x7f, 0xcd, 0x55, 0x2a, 0xab, 0x66), G(0x7d, 0xce, 0x5c, 0x21, 0xa6, 0x68), - G(0x73, 0xc7, 0x63, 0x10, 0x85, 0x42), G(0x71, 0xc4, 0x6a, 0x1b, 0x88, 0x4c), - G(0x77, 0xc1, 0x71, 0x06, 0x9f, 0x5e), G(0x75, 0xc2, 0x78, 0x0d, 0x92, 0x50), - G(0x6b, 0xd3, 0x0f, 0x64, 0xd9, 0x0a), G(0x69, 0xd0, 0x06, 0x6f, 0xd4, 0x04), - G(0x6f, 0xd5, 0x1d, 0x72, 0xc3, 0x16), G(0x6d, 0xd6, 0x14, 0x79, 0xce, 0x18), - G(0x63, 0xdf, 0x2b, 0x48, 0xed, 0x32), G(0x61, 0xdc, 0x22, 0x43, 0xe0, 0x3c), - G(0x67, 0xd9, 0x39, 0x5e, 0xf7, 0x2e), G(0x65, 0xda, 0x30, 0x55, 0xfa, 0x20), - G(0x9b, 0x5b, 0x9a, 0x01, 0xb7, 0xec), G(0x99, 0x58, 0x93, 0x0a, 0xba, 0xe2), - G(0x9f, 0x5d, 0x88, 0x17, 0xad, 0xf0), G(0x9d, 0x5e, 0x81, 0x1c, 0xa0, 0xfe), - G(0x93, 0x57, 0xbe, 0x2d, 0x83, 0xd4), G(0x91, 0x54, 0xb7, 0x26, 0x8e, 0xda), - G(0x97, 0x51, 0xac, 0x3b, 0x99, 0xc8), G(0x95, 0x52, 0xa5, 0x30, 0x94, 0xc6), - G(0x8b, 0x43, 0xd2, 0x59, 0xdf, 0x9c), G(0x89, 0x40, 0xdb, 0x52, 0xd2, 0x92), - G(0x8f, 0x45, 0xc0, 0x4f, 0xc5, 0x80), G(0x8d, 0x46, 0xc9, 0x44, 0xc8, 0x8e), - G(0x83, 0x4f, 0xf6, 0x75, 0xeb, 0xa4), G(0x81, 0x4c, 0xff, 0x7e, 0xe6, 0xaa), - G(0x87, 0x49, 0xe4, 0x63, 0xf1, 0xb8), G(0x85, 0x4a, 0xed, 0x68, 0xfc, 0xb6), - G(0xbb, 0x6b, 0x0a, 0xb1, 0x67, 0x0c), G(0xb9, 0x68, 0x03, 0xba, 0x6a, 0x02), - G(0xbf, 0x6d, 0x18, 0xa7, 0x7d, 0x10), G(0xbd, 0x6e, 0x11, 0xac, 0x70, 0x1e), - G(0xb3, 0x67, 0x2e, 0x9d, 0x53, 0x34), G(0xb1, 0x64, 0x27, 0x96, 0x5e, 0x3a), - G(0xb7, 0x61, 0x3c, 0x8b, 0x49, 0x28), G(0xb5, 0x62, 0x35, 0x80, 0x44, 0x26), - G(0xab, 0x73, 0x42, 0xe9, 0x0f, 0x7c), G(0xa9, 0x70, 0x4b, 0xe2, 0x02, 0x72), - G(0xaf, 0x75, 0x50, 0xff, 0x15, 0x60), G(0xad, 0x76, 0x59, 0xf4, 0x18, 0x6e), - G(0xa3, 0x7f, 0x66, 0xc5, 0x3b, 0x44), G(0xa1, 0x7c, 0x6f, 0xce, 0x36, 0x4a), - G(0xa7, 0x79, 0x74, 0xd3, 0x21, 0x58), G(0xa5, 0x7a, 0x7d, 0xd8, 0x2c, 0x56), - G(0xdb, 0x3b, 0xa1, 0x7a, 0x0c, 0x37), G(0xd9, 0x38, 0xa8, 0x71, 0x01, 0x39), - G(0xdf, 0x3d, 0xb3, 0x6c, 0x16, 0x2b), G(0xdd, 0x3e, 0xba, 0x67, 0x1b, 0x25), - G(0xd3, 0x37, 0x85, 0x56, 0x38, 0x0f), G(0xd1, 0x34, 0x8c, 0x5d, 0x35, 0x01), - G(0xd7, 0x31, 0x97, 0x40, 0x22, 0x13), G(0xd5, 0x32, 0x9e, 0x4b, 0x2f, 0x1d), - G(0xcb, 0x23, 0xe9, 0x22, 0x64, 0x47), G(0xc9, 0x20, 0xe0, 0x29, 0x69, 0x49), - G(0xcf, 0x25, 0xfb, 0x34, 0x7e, 0x5b), G(0xcd, 0x26, 0xf2, 0x3f, 0x73, 0x55), - G(0xc3, 0x2f, 0xcd, 0x0e, 0x50, 0x7f), G(0xc1, 0x2c, 0xc4, 0x05, 0x5d, 0x71), - G(0xc7, 0x29, 0xdf, 0x18, 0x4a, 0x63), G(0xc5, 0x2a, 0xd6, 0x13, 0x47, 0x6d), - G(0xfb, 0x0b, 0x31, 0xca, 0xdc, 0xd7), G(0xf9, 0x08, 0x38, 0xc1, 0xd1, 0xd9), - G(0xff, 0x0d, 0x23, 0xdc, 0xc6, 0xcb), G(0xfd, 0x0e, 0x2a, 0xd7, 0xcb, 0xc5), - G(0xf3, 0x07, 0x15, 0xe6, 0xe8, 0xef), G(0xf1, 0x04, 0x1c, 0xed, 0xe5, 0xe1), - G(0xf7, 0x01, 0x07, 0xf0, 0xf2, 0xf3), G(0xf5, 0x02, 0x0e, 0xfb, 0xff, 0xfd), - G(0xeb, 0x13, 0x79, 0x92, 0xb4, 0xa7), G(0xe9, 0x10, 0x70, 0x99, 0xb9, 0xa9), - G(0xef, 0x15, 0x6b, 0x84, 0xae, 0xbb), G(0xed, 0x16, 0x62, 0x8f, 0xa3, 0xb5), - G(0xe3, 0x1f, 0x5d, 0xbe, 0x80, 0x9f), G(0xe1, 0x1c, 0x54, 0xb5, 0x8d, 0x91), - G(0xe7, 0x19, 0x4f, 0xa8, 0x9a, 0x83), G(0xe5, 0x1a, 0x46, 0xa3, 0x97, 0x8d), -}; -#undef G +/* + * The schedule is kept in its compressed form, 30 words of it, inside the + * aes_key the caller already has -- comfortably inside the 448 bytes that + * held the round keys before. br_aes_ct64_skey_expand blows it up to 120 + * words on the stack once per call, which is how BearSSL's own CTR and CBC + * use it. + * + * It travels through memcpy rather than a cast because aes_key is all + * uint8_t and so carries no alignment of its own, whatever the allocator + * happens to give it. + */ +#define COMP_SKEY_WORDS 30 -static void expand_key(uint8_t *expandedKey, uint8_t *key, int size, size_t expandedKeySize) +void crypton_aes_generic_schedule(aes_sched *sched, const aes_key *key) { - int csz; - int i; - uint8_t t[4] = { 0 }; - - for (i = 0; i < size; i++) - expandedKey[i] = key[i]; - csz = size; + uint64_t comp_skey[COMP_SKEY_WORDS]; - i = 1; - while (csz < expandedKeySize) { - t[0] = expandedKey[(csz - 4) + 0]; - t[1] = expandedKey[(csz - 4) + 1]; - t[2] = expandedKey[(csz - 4) + 2]; - t[3] = expandedKey[(csz - 4) + 3]; + memcpy(comp_skey, key->data, sizeof comp_skey); + sched->nbr = key->nbr; + br_aes_ct64_skey_expand(sched->sk_exp, sched->nbr, comp_skey); +} - if (csz % size == 0) { - uint8_t tmp; +/* + * Up to four blocks through the bitsliced core at once. Fewer than four is + * the same work as four -- the lanes are there whether anything is in them + * -- so a caller with four to offer gets them for what one used to cost. + */ +static void pass(uint8_t *output, const uint8_t *input, unsigned n, + const aes_sched *sched, int decrypt) +{ + uint32_t w[16]; + uint64_t q[8]; + unsigned i; - tmp = t[0]; - t[0] = sbox[t[1]] ^ Rcon[i++ % sizeof(Rcon)]; - t[1] = sbox[t[2]]; - t[2] = sbox[t[3]]; - t[3] = sbox[tmp]; - } + memset(w, 0, sizeof w); + for (i = 0; i < n; i++) { + w[4 * i] = br_dec32le(input + 16 * i); + w[4 * i + 1] = br_dec32le(input + 16 * i + 4); + w[4 * i + 2] = br_dec32le(input + 16 * i + 8); + w[4 * i + 3] = br_dec32le(input + 16 * i + 12); + } - if (size == 32 && ((csz % size) == 16)) { - t[0] = sbox[t[0]]; - t[1] = sbox[t[1]]; - t[2] = sbox[t[2]]; - t[3] = sbox[t[3]]; - } + for (i = 0; i < 4; i++) + br_aes_ct64_interleave_in(&q[i], &q[i + 4], w + 4 * i); + br_aes_ct64_ortho(q); + if (decrypt) + br_aes_ct64_bitslice_decrypt(sched->nbr, sched->sk_exp, q); + else + br_aes_ct64_bitslice_encrypt(sched->nbr, sched->sk_exp, q); + br_aes_ct64_ortho(q); + for (i = 0; i < 4; i++) + br_aes_ct64_interleave_out(w + 4 * i, q[i], q[i + 4]); - expandedKey[csz] = expandedKey[csz - size] ^ t[0]; csz++; - expandedKey[csz] = expandedKey[csz - size] ^ t[1]; csz++; - expandedKey[csz] = expandedKey[csz - size] ^ t[2]; csz++; - expandedKey[csz] = expandedKey[csz - size] ^ t[3]; csz++; + for (i = 0; i < n; i++) { + br_enc32le(output + 16 * i, w[4 * i]); + br_enc32le(output + 16 * i + 4, w[4 * i + 1]); + br_enc32le(output + 16 * i + 8, w[4 * i + 2]); + br_enc32le(output + 16 * i + 12, w[4 * i + 3]); } } -static void shift_rows(uint8_t *state) +void crypton_aes_generic_blocks(uint8_t *output, const uint8_t *input, + uint32_t nb_blocks, const aes_sched *sched, + int decrypt) { - uint32_t *s32; - int i; - - for (i = 0; i < 16; i++) - state[i] = sbox[state[i]]; - s32 = (uint32_t *) state; - s32[1] = rol32_be(s32[1], 8); - s32[2] = rol32_be(s32[2], 16); - s32[3] = rol32_be(s32[3], 24); + while (nb_blocks >= 4) { + pass(output, input, 4, sched, decrypt); + output += 64; + input += 64; + nb_blocks -= 4; + } + if (nb_blocks > 0) + pass(output, input, (unsigned) nb_blocks, sched, decrypt); } -static void add_round_key(uint8_t *state, uint8_t *rk) +static void one(aes_block *output, aes_key *key, aes_block *input, int decrypt) { - uint32_t *s32, *r32; + aes_sched sched; - s32 = (uint32_t *) state; - r32 = (uint32_t *) rk; - s32[0] ^= r32[0]; - s32[1] ^= r32[1]; - s32[2] ^= r32[2]; - s32[3] ^= r32[3]; + crypton_aes_generic_schedule(&sched, key); + crypton_aes_generic_blocks((uint8_t *) output, (const uint8_t *) input, + 1, &sched, decrypt); } -#define gm1(a) (a) -#define gm2(a) gmtab[a][0] -#define gm3(a) gmtab[a][1] -#define gm9(a) gmtab[a][2] -#define gm11(a) gmtab[a][3] -#define gm13(a) gmtab[a][4] -#define gm14(a) gmtab[a][5] - -static void mix_columns(uint8_t *state) +void crypton_aes_generic_encrypt_block(aes_block *output, aes_key *key, aes_block *input) { - int i; - uint8_t cpy[4]; - - for (i = 0; i < 4; i++) { - cpy[0] = state[0 * 4 + i]; - cpy[1] = state[1 * 4 + i]; - cpy[2] = state[2 * 4 + i]; - cpy[3] = state[3 * 4 + i]; - state[i] = gm2(cpy[0]) ^ gm1(cpy[3]) ^ gm1(cpy[2]) ^ gm3(cpy[1]); - state[4+i] = gm2(cpy[1]) ^ gm1(cpy[0]) ^ gm1(cpy[3]) ^ gm3(cpy[2]); - state[8+i] = gm2(cpy[2]) ^ gm1(cpy[1]) ^ gm1(cpy[0]) ^ gm3(cpy[3]); - state[12+i] = gm2(cpy[3]) ^ gm1(cpy[2]) ^ gm1(cpy[1]) ^ gm3(cpy[0]); - } + one(output, key, input, 0); } -static void create_round_key(uint8_t *expandedKey, uint8_t *rk) +void crypton_aes_generic_decrypt_block(aes_block *output, aes_key *key, aes_block *input) { - int i,j; - for (i = 0; i < 4; i++) - for (j = 0; j < 4; j++) - rk[i + j * 4] = expandedKey[i * 4 + j]; + one(output, key, input, 1); } -static void aes_main(aes_key *key, uint8_t *state) +void crypton_aes_generic_init(aes_key *key, uint8_t *origkey, uint8_t size) { - int i = 0; - uint32_t rk[4]; - uint8_t *rkptr = (uint8_t *) rk; - - create_round_key(key->data, rkptr); - add_round_key(state, rkptr); - - for (i = 1; i < key->nbr; i++) { - create_round_key(key->data + 16 * i, rkptr); - shift_rows(state); - mix_columns(state); - add_round_key(state, rkptr); - } - - create_round_key(key->data + 16 * key->nbr, rkptr); - shift_rows(state); - add_round_key(state, rkptr); -} + uint64_t comp_skey[COMP_SKEY_WORDS]; + unsigned nbr; -static void shift_rows_inv(uint8_t *state) -{ - uint32_t *s32; - int i; + /* 0 for a key length that is not 16, 24 or 32; the old code returned + * without touching the key in that case and so does this */ + nbr = br_aes_ct64_keysched(comp_skey, origkey, size); + if (nbr == 0) + return; - s32 = (uint32_t *) state; - s32[1] = ror32_be(s32[1], 8); - s32[2] = ror32_be(s32[2], 16); - s32[3] = ror32_be(s32[3], 24); - for (i = 0; i < 16; i++) - state[i] = rsbox[state[i]]; + key->nbr = (uint8_t) nbr; + memcpy(key->data, comp_skey, sizeof comp_skey); } -static void mix_columns_inv(uint8_t *state) +/* + * CTR, four counter blocks at a time. The counter itself is serial, but + * nothing about it depends on the keystream, so the four blocks it will + * reach next can be written down before any of them is encrypted. + */ +static void ctr(uint8_t *output, aes_key *key, aes_block *iv, + uint8_t *input, uint32_t len, int c32) { - int i; - uint8_t cpy[4]; + aes_sched sched; + aes_block counter; + uint8_t ks[64]; + uint32_t nb_blocks = len / 16; + uint32_t tail = len % 16; + uint32_t i; - for (i = 0; i < 4; i++) { - cpy[0] = state[0 * 4 + i]; - cpy[1] = state[1 * 4 + i]; - cpy[2] = state[2 * 4 + i]; - cpy[3] = state[3 * 4 + i]; - state[i] = gm14(cpy[0]) ^ gm9(cpy[3]) ^ gm13(cpy[2]) ^ gm11(cpy[1]); - state[4+i] = gm14(cpy[1]) ^ gm9(cpy[0]) ^ gm13(cpy[3]) ^ gm11(cpy[2]); - state[8+i] = gm14(cpy[2]) ^ gm9(cpy[1]) ^ gm13(cpy[0]) ^ gm11(cpy[3]); - state[12+i] = gm14(cpy[3]) ^ gm9(cpy[2]) ^ gm13(cpy[1]) ^ gm11(cpy[0]); - } -} + crypton_aes_generic_schedule(&sched, key); + block128_copy(&counter, iv); -static void aes_main_inv(aes_key *key, uint8_t *state) -{ - int i = 0; - uint32_t rk[4]; - uint8_t *rkptr = (uint8_t *) rk; + while (nb_blocks > 0) { + uint32_t n = nb_blocks < 4 ? nb_blocks : 4; - create_round_key(key->data + 16 * key->nbr, rkptr); - add_round_key(state, rkptr); + for (i = 0; i < n; i++) { + block128_copy((block128 *) (ks + 16 * i), &counter); + if (c32) + block128_inc32_le(&counter); + else + block128_inc_be(&counter); + } + crypton_aes_generic_blocks(ks, ks, n, &sched, 0); + for (i = 0; i < n * 16; i++) + output[i] = ks[i] ^ input[i]; - for (i = key->nbr - 1; i > 0; i--) { - create_round_key(key->data + 16 * i, rkptr); - shift_rows_inv(state); - add_round_key(state, rkptr); - mix_columns_inv(state); + output += n * 16; + input += n * 16; + nb_blocks -= n; } - create_round_key(key->data, rkptr); - shift_rows_inv(state); - add_round_key(state, rkptr); -} - -/* Set the block values, for the block: - * a0,0 a0,1 a0,2 a0,3 - * a1,0 a1,1 a1,2 a1,3 -> a0,0 a1,0 a2,0 a3,0 a0,1 a1,1 ... a2,3 a3,3 - * a2,0 a2,1 a2,2 a2,3 - * a3,0 a3,1 a3,2 a3,3 - */ -#define swap_block(t, f) \ - t[0] = f[0]; t[4] = f[1]; t[8] = f[2]; t[12] = f[3]; \ - t[1] = f[4]; t[5] = f[5]; t[9] = f[6]; t[13] = f[7]; \ - t[2] = f[8]; t[6] = f[9]; t[10] = f[10]; t[14] = f[11]; \ - t[3] = f[12]; t[7] = f[13]; t[11] = f[14]; t[15] = f[15] - -void crypton_aes_generic_encrypt_block(aes_block *output, aes_key *key, aes_block *input) -{ - uint32_t block[4]; - uint8_t *iptr, *optr, *bptr; - - iptr = (uint8_t *) input; - optr = (uint8_t *) output; - bptr = (uint8_t *) block; - swap_block(bptr, iptr); - aes_main(key, bptr); - swap_block(optr, bptr); + if (tail != 0) { + block128_copy((block128 *) ks, &counter); + crypton_aes_generic_blocks(ks, ks, 1, &sched, 0); + for (i = 0; i < tail; i++) + output[i] = ks[i] ^ input[i]; + } } -void crypton_aes_generic_decrypt_block(aes_block *output, aes_key *key, aes_block *input) +void crypton_aes_bitsliced_encrypt_ctr(uint8_t *output, aes_key *key, + aes_block *iv, uint8_t *input, + uint32_t len) { - uint32_t block[4]; - uint8_t *iptr, *optr, *bptr; - - iptr = (uint8_t *) input; - optr = (uint8_t *) output; - bptr = (uint8_t *) block; - swap_block(bptr, iptr); - aes_main_inv(key, bptr); - swap_block(optr, bptr); + ctr(output, key, iv, input, len, 0); } -void crypton_aes_generic_init(aes_key *key, uint8_t *origkey, uint8_t size) +void crypton_aes_bitsliced_encrypt_c32(uint8_t *output, aes_key *key, + aes_block *iv, uint8_t *input, + uint32_t len) { - int esz; - - switch (size) { - case 16: key->nbr = 10; esz = 176; break; - case 24: key->nbr = 12; esz = 208; break; - case 32: key->nbr = 14; esz = 240; break; - default: return; - } - expand_key(key->data, origkey, size, esz); - return; + ctr(output, key, iv, input, len, 1); } diff --git a/cbits/aes/generic.h b/cbits/aes/generic.h --- a/cbits/aes/generic.h +++ b/cbits/aes/generic.h @@ -32,3 +32,35 @@ void crypton_aes_generic_encrypt_block(aes_block *output, aes_key *key, aes_block *input); void crypton_aes_generic_decrypt_block(aes_block *output, aes_key *key, aes_block *input); void crypton_aes_generic_init(aes_key *key, uint8_t *origkey, uint8_t size); + +/* + * The bitsliced core takes four blocks at a time, and the schedule it reads + * is the expanded one rather than the compressed form the aes_key holds. + * Expanding costs about what a block costs, so a mode with more than one + * block to do expands once, here, and hands the result to every group. + */ +typedef struct { + uint64_t sk_exp[120]; + unsigned nbr; +} aes_sched; + +void crypton_aes_generic_schedule(aes_sched *sched, const aes_key *key); + +/* nb_blocks of them, four at a pass; decrypt selects the direction */ +void crypton_aes_generic_blocks(uint8_t *output, const uint8_t *input, + uint32_t nb_blocks, const aes_sched *sched, + int decrypt); + +/* + * CTR with the two counters crypton uses. These are not the generic + * entries: those go through the branch table for the block itself and so + * run on accelerated machines too, where the key holds a different + * schedule. crypton_aes.c installs these only when nothing was + * accelerated. + */ +void crypton_aes_bitsliced_encrypt_ctr(uint8_t *output, aes_key *key, + aes_block *iv, uint8_t *input, + uint32_t len); +void crypton_aes_bitsliced_encrypt_c32(uint8_t *output, aes_key *key, + aes_block *iv, uint8_t *input, + uint32_t len); diff --git a/cbits/aes/gf.c b/cbits/aes/gf.c --- a/cbits/aes/gf.c +++ b/cbits/aes/gf.c @@ -33,6 +33,7 @@ #include #include #include +#include "bearssl/inner.h" /* inplace GFMUL for xts mode */ void crypton_aes_generic_gf_mulx(block128 *a) @@ -45,118 +46,43 @@ /* - * GF multiplication with Shoup's method and 4-bit table. + * GHASH, without a table. * - * We precompute the products of H with all 4-bit polynomials and store them in - * a 'table_4bit' array. To avoid unnecessary byte swapping, the 16 blocks are - * written to the table with qwords already converted to CPU order. Table - * indices use the reflected bit ordering, i.e. polynomials X^0, X^1, X^2, X^3 - * map to bit positions 3, 2, 1, 0 respectively. + * This was Shoup's method: the products of H with all sixteen 4-bit + * polynomials, precomputed, and thirty-two lookups a block at indices taken + * from the accumulator -- which is to say, thirty-two addresses derived from + * a secret. It is now BearSSL's ghash_ctmul64, which builds the GF(2^128) + * multiply out of shifts, masks and integer multiplies and looks nothing up. + * See cbits/bearssl/README.md. * - * To multiply an arbitrary block with H, the input block is decomposed in 4-bit - * segments. We get the final result after 32 table lookups and additions, one - * for each segment, interleaving multiplication by P(X)=X^4. + * The table_4bit the interface names is sixteen blocks wide because the + * table needed it. This keeps H in the first and leaves the rest alone; the + * PMULL and PCLMUL implementations keep their own powers of H in that same + * space, so the width stays as it is. */ -/* convert block128 qwords between BE and CPU order */ -static inline void block128_cpu_swap_be(block128 *a, const block128 *b) -{ - a->q[1] = cpu_to_be64(b->q[1]); - a->q[0] = cpu_to_be64(b->q[0]); -} - -/* multiplication by P(X)=X, assuming qwords already in CPU order */ -static inline void cpu_gf_mulx(block128 *a, const block128 *b) -{ - uint64_t v0 = b->q[0]; - uint64_t v1 = b->q[1]; - a->q[1] = v1 >> 1 | v0 << 63; - a->q[0] = v0 >> 1 ^ ((0-(v1 & 1)) & 0xe100000000000000ULL); -} - -static const uint64_t r4_0[] = - { 0x0000000000000000ULL, 0x1c20000000000000ULL - , 0x3840000000000000ULL, 0x2460000000000000ULL - , 0x7080000000000000ULL, 0x6ca0000000000000ULL - , 0x48c0000000000000ULL, 0x54e0000000000000ULL - , 0xe100000000000000ULL, 0xfd20000000000000ULL - , 0xd940000000000000ULL, 0xc560000000000000ULL - , 0x9180000000000000ULL, 0x8da0000000000000ULL - , 0xa9c0000000000000ULL, 0xb5e0000000000000ULL - }; - -/* multiplication by P(X)=X^4, assuming qwords already in CPU order */ -static inline void cpu_gf_mulx4(block128 *a, const block128 *b) -{ - uint64_t v0 = b->q[0]; - uint64_t v1 = b->q[1]; - a->q[1] = v1 >> 4 | v0 << 60; - a->q[0] = v0 >> 4 ^ r4_0[v1 & 0xf]; -} - -/* initialize the 4-bit table given H */ +/* remember H */ void crypton_aes_generic_hinit(table_4bit htable, const block128 *h) { - block128 v, *p; - int i, j; - - /* multiplication by 0 is 0 */ - block128_zero(&htable[0]); - - /* at index 8=2^3 we have H.X^0 = H */ - i = 8; - block128_cpu_swap_be(&htable[i], h); /* in CPU order */ - p = &htable[i]; - - /* for other powers of 2, repeat multiplication by P(X)=X */ - for (i = 4; i > 0; i >>= 1) - { - cpu_gf_mulx(&htable[i], p); - p = &htable[i]; - } - - /* remaining elements are linear combinations */ - for (i = 2; i < 16; i <<= 1) { - p = &htable[i]; - v = *p; - for (j = 1; j < i; j++) { - p[j] = v; - block128_xor_aligned(&p[j], &htable[j]); - } - } + block128_copy(&htable[0], h); } -/* multiply a block with H */ +/* + * br_ghash_ctmul64 computes y = (y ^ x) * H for each block x it is given, so + * a block of zeros is the bare multiply this entry is asked for. + */ void crypton_aes_generic_gf_mul(block128 *a, const table_4bit htable) { - block128 b; - int i; - block128_zero(&b); - for (i = 15; i >= 0; i--) - { - uint8_t v = a->b[i]; - block128_xor_aligned(&b, &htable[v & 0xf]); /* high bits (reflected) */ - cpu_gf_mulx4(&b, &b); - block128_xor_aligned(&b, &htable[v >> 4]); /* low bits (reflected) */ - if (i > 0) - cpu_gf_mulx4(&b, &b); - else - block128_cpu_swap_be(a, &b); /* restore BE order when done */ - } + static const uint8_t zero[16] = { 0 }; + + br_ghash_ctmul64(a, &htable[0], zero, sizeof zero); } /* - * Four GHASH steps at once. The generic table-driven multiply has no cheaper - * way to do this than one block at a time; the point of the entry is that the - * PMULL and PCLMUL versions can fold the four products into one reduction, so - * the GCM loops hand over four blocks whenever they have them. + * Four GHASH steps at once, which here is one call rather than four: the + * loop inside br_ghash_ctmul64 takes the blocks as they come. */ void crypton_aes_generic_gf_mul4(block128 *a, const block128 *blocks, const table_4bit htable) { - int i; - - for (i = 0; i < 4; i++) { - block128_xor(a, &blocks[i]); - crypton_aes_generic_gf_mul(a, htable); - } + br_ghash_ctmul64(a, &htable[0], blocks, 4 * sizeof(block128)); } diff --git a/cbits/aes/ppc8.c b/cbits/aes/ppc8.c new file mode 100644 --- /dev/null +++ b/cbits/aes/ppc8.c @@ -0,0 +1,287 @@ +/* + * AES and GHASH using the PowerISA 2.07 vector instructions, first + * implemented by POWER8. + * + * The instructions themselves come from CRYPTOGAMS, assembled from + * cbits/asm/aesp8-ppc-*.S and cbits/asm/ghashp8-ppc-*.S; what is here is the + * glue that puts them behind crypton's branch table. Nothing in this file + * branches or indexes on a key or on data. + * + * == Where the key schedule lives + * + * The assembly takes OpenSSL's AES_KEY -- sixty round-key words and a round + * count, 244 bytes -- and its set_decrypt_key writes a second, complete one + * rather than sharing ends with the first the way cbits/aes/armv8.c does. + * Two of those are 488 bytes and aes_key.data is 448, so both do not fit. + * + * So the forward schedule is kept there, with the key the caller gave after + * it, and the inverse is built on the stack by the operations that need it. + * Those are all bulk -- ECB, CBC and XTS decryption -- so it is one key + * schedule per call rather than per block. Single-block decryption pays for + * one too, and is reached by nothing that runs in a loop: OCB and CCM drive + * the block function in the encrypting direction. + */ + +#include +#include +#include +#include +#include "aes/block128.h" +#include "aes/gf.h" +#include "aes/ppc8.h" + +/* OpenSSL's AES_KEY, which is what the assembly was written against */ +typedef struct { + unsigned int rd_key[60]; + int rounds; +} p8_key; + +int crypton_aes_p8_set_encrypt_key(const unsigned char *, int, p8_key *); +int crypton_aes_p8_set_decrypt_key(const unsigned char *, int, p8_key *); +void crypton_aes_p8_encrypt(const unsigned char *, unsigned char *, const p8_key *); +void crypton_aes_p8_decrypt(const unsigned char *, unsigned char *, const p8_key *); +void crypton_aes_p8_cbc_encrypt(const unsigned char *, unsigned char *, size_t, + const p8_key *, unsigned char *, int); +void crypton_aes_p8_ctr32_encrypt_blocks(const unsigned char *, unsigned char *, + size_t, const p8_key *, + const unsigned char *); +void crypton_aes_p8_xts_encrypt(const unsigned char *, unsigned char *, size_t, + const p8_key *, const p8_key *, + const unsigned char *); +void crypton_aes_p8_xts_decrypt(const unsigned char *, unsigned char *, size_t, + const p8_key *, const p8_key *, + const unsigned char *); +/* void * rather than uint64_t *: the assembly loads these with lvx_u and so + * does not want the alignment a uint64_t pointer would promise, and + * block128 is packed. */ +void crypton_gcm_init_p8(void *Htable, const void *H); +void crypton_gcm_gmult_p8(void *Xi, const void *Htable); +void crypton_gcm_ghash_p8(void *Xi, const void *Htable, + const unsigned char *inp, size_t len); + +#define FORWARD(k) ((p8_key *) (void *) (k)->data) +#define USERKEY(k) ((uint8_t *) (k)->data + sizeof(p8_key)) +#define USERLEN(k) (*((uint8_t *) (k)->data + sizeof(p8_key) + 32)) + +static void inverse(p8_key *dk, aes_key *key) +{ + crypton_aes_p8_set_decrypt_key(USERKEY(key), USERLEN(key) * 8, dk); +} + +void crypton_aes_ppc8_init(aes_key *key, uint8_t *origkey, uint8_t size) +{ + if (size != 16 && size != 24 && size != 32) + return; + crypton_aes_p8_set_encrypt_key(origkey, size * 8, FORWARD(key)); + memcpy(USERKEY(key), origkey, size); + USERLEN(key) = size; +} + +void crypton_aes_ppc8_encrypt_block(aes_block *output, aes_key *key, aes_block *input) +{ + crypton_aes_p8_encrypt((const unsigned char *) input, + (unsigned char *) output, FORWARD(key)); +} + +void crypton_aes_ppc8_decrypt_block(aes_block *output, aes_key *key, aes_block *input) +{ + p8_key dk; + + inverse(&dk, key); + crypton_aes_p8_decrypt((const unsigned char *) input, + (unsigned char *) output, &dk); + memset(&dk, 0, sizeof dk); +} + +/* The assembly has no ECB entry: it is the block function in a loop, which + * is what the generic implementation does too. */ +void crypton_aes_ppc8_encrypt_ecb(aes_block *output, aes_key *key, aes_block *input, + uint32_t nb_blocks) +{ + for (; nb_blocks-- > 0; input++, output++) + crypton_aes_p8_encrypt((const unsigned char *) input, + (unsigned char *) output, FORWARD(key)); +} + +void crypton_aes_ppc8_decrypt_ecb(aes_block *output, aes_key *key, aes_block *input, + uint32_t nb_blocks) +{ + p8_key dk; + + inverse(&dk, key); + for (; nb_blocks-- > 0; input++, output++) + crypton_aes_p8_decrypt((const unsigned char *) input, + (unsigned char *) output, &dk); + memset(&dk, 0, sizeof dk); +} + +void crypton_aes_ppc8_encrypt_cbc(aes_block *output, aes_key *key, aes_block *iv, + aes_block *input, uint32_t nb_blocks) +{ + uint8_t ivbuf[16]; + + /* a copy, because the assembly writes the last block back through this + * pointer and the callers of this entry do not expect their IV touched */ + memcpy(ivbuf, iv, 16); + crypton_aes_p8_cbc_encrypt((const unsigned char *) input, + (unsigned char *) output, + (size_t) nb_blocks * 16, FORWARD(key), ivbuf, 1); +} + +void crypton_aes_ppc8_decrypt_cbc(aes_block *output, aes_key *key, aes_block *iv, + aes_block *input, uint32_t nb_blocks) +{ + p8_key dk; + uint8_t ivbuf[16]; + + inverse(&dk, key); + memcpy(ivbuf, iv, 16); + crypton_aes_p8_cbc_encrypt((const unsigned char *) input, + (unsigned char *) output, + (size_t) nb_blocks * 16, &dk, ivbuf, 0); + memset(&dk, 0, sizeof dk); +} + +/* + * CTR, which is where the two counters have to be reconciled. + * + * crypton counts with block128_inc_be, over the whole 128 bits; the assembly + * counts over the low 32 only. They agree until that word wraps, so the + * work is handed over in runs that stop there, and the carry into the upper + * bits is done here. This is the same arrangement OpenSSL makes for the + * same reason. + */ +void crypton_aes_ppc8_encrypt_ctr(uint8_t *output, aes_key *key, aes_block *iv, + uint8_t *input, uint32_t len) +{ + block128 ctr; + uint32_t nb_blocks = len / 16; + uint32_t tail = len % 16; + uint32_t i; + + block128_copy(&ctr, iv); + + while (nb_blocks > 0) { + uint64_t low = (uint64_t) be32_to_cpu(ctr.d[3]); + uint64_t room = 0x100000000ULL - low; /* blocks before it wraps */ + uint32_t n; + + /* room is 2^32 when the word is at zero, which does not fit the + * type the comparison would narrow it to -- and a zero n here + * would not advance */ + if (room > (uint64_t) nb_blocks) + room = (uint64_t) nb_blocks; + n = (uint32_t) room; + + crypton_aes_p8_ctr32_encrypt_blocks((const unsigned char *) input, + (unsigned char *) output, + n, FORWARD(key), + (const unsigned char *) &ctr); + /* the assembly does not write the counter back */ + for (i = 0; i < n; i++) + block128_inc_be(&ctr); + + output += (size_t) n * 16; + input += (size_t) n * 16; + nb_blocks -= n; + } + + if (tail != 0) { + block128 ks; + + crypton_aes_p8_encrypt((const unsigned char *) &ctr, + (unsigned char *) &ks, FORWARD(key)); + for (i = 0; i < tail; i++) + output[i] = ks.b[i] ^ input[i]; + } +} + +/* + * XTS. The assembly encrypts the tweak with its second key, unless that key + * is NULL, in which case it takes the tweak already encrypted -- which is + * what this needs, because crypton's entry also carries a starting point and + * the tweak has to be advanced that many doublings before any block is + * enciphered. + */ +static void xts_tweak(block128 *tweak, aes_key *k2, aes_block *dataunit, + uint32_t spoint) +{ + block128_copy(tweak, dataunit); + crypton_aes_p8_encrypt((const unsigned char *) tweak, + (unsigned char *) tweak, FORWARD(k2)); + while (spoint-- > 0) + crypton_aes_generic_gf_mulx(tweak); +} + +void crypton_aes_ppc8_encrypt_xts(aes_block *output, aes_key *k1, aes_key *k2, + aes_block *dataunit, uint32_t spoint, + aes_block *input, uint32_t nb_blocks) +{ + block128 tweak; + + xts_tweak(&tweak, k2, dataunit, spoint); + crypton_aes_p8_xts_encrypt((const unsigned char *) input, + (unsigned char *) output, + (size_t) nb_blocks * 16, FORWARD(k1), NULL, + (const unsigned char *) &tweak); +} + +void crypton_aes_ppc8_decrypt_xts(aes_block *output, aes_key *k1, aes_key *k2, + aes_block *dataunit, uint32_t spoint, + aes_block *input, uint32_t nb_blocks) +{ + block128 tweak; + p8_key dk; + + /* the tweak is enciphered with k2 forwards either way */ + xts_tweak(&tweak, k2, dataunit, spoint); + inverse(&dk, k1); + crypton_aes_p8_xts_decrypt((const unsigned char *) input, + (unsigned char *) output, + (size_t) nb_blocks * 16, &dk, NULL, + (const unsigned char *) &tweak); + memset(&dk, 0, sizeof dk); +} + +/* + * GHASH. The table the assembly builds is 192 bytes, which fits the sixteen + * blocks the interface names. + * + * The two arguments do not take the same convention, which is worth saying + * because it does not look like an accident until it is checked. H arrives + * here as the sixteen bytes GCM defines, and gcm_init_p8 wants those as a + * pair of host-order words -- so they are swapped on a little-endian machine + * and left alone on a big-endian one, which be64_to_cpu does. The + * accumulator is the byte string throughout, in and out, and is passed as it + * stands. OpenSSL makes the same pair of choices. + * + * Checked rather than assumed: against crypton's own GHASH, over two + * different H and with a non-zero accumulator, so that neither a symmetric + * value nor a zero start could hide a wrong one. + */ +void crypton_aes_ppc8_hinit(table_4bit htable, const block128 *h) +{ + uint64_t H[2]; + + memcpy(H, h, sizeof H); + H[0] = be64_to_cpu(H[0]); + H[1] = be64_to_cpu(H[1]); + crypton_gcm_init_p8(htable, H); +} + +void crypton_aes_ppc8_gf_mul(block128 *a, const table_4bit htable) +{ + crypton_gcm_gmult_p8(a, htable); +} + +void crypton_aes_ppc8_gf_mul4(block128 *a, const block128 *blocks, + const table_4bit htable) +{ + crypton_gcm_ghash_p8(a, htable, (const unsigned char *) blocks, + 4 * sizeof(block128)); +} + +int crypton_aes_ppc8_available(void) +{ + return (crypton_ppc_features() & CRYPTON_PPC_VCRYPTO) != 0; +} diff --git a/cbits/aes/ppc8.h b/cbits/aes/ppc8.h new file mode 100644 --- /dev/null +++ b/cbits/aes/ppc8.h @@ -0,0 +1,29 @@ +/* + * AES and GHASH on PowerISA 2.07, as implemented by cbits/aes/ppc8.c over + * the CRYPTOGAMS assembly. crypton_aes.c installs these when + * crypton_aes_ppc8_available says the processor has the instructions. + */ +#ifndef CRYPTON_AES_PPC8_H +#define CRYPTON_AES_PPC8_H + +#include "crypton_aes.h" +#include "aes/gf.h" +#include "crypton_cpu.h" + +int crypton_aes_ppc8_available(void); + +void crypton_aes_ppc8_init(aes_key *key, uint8_t *origkey, uint8_t size); +void crypton_aes_ppc8_encrypt_block(aes_block *output, aes_key *key, aes_block *input); +void crypton_aes_ppc8_decrypt_block(aes_block *output, aes_key *key, aes_block *input); +void crypton_aes_ppc8_encrypt_ecb(aes_block *output, aes_key *key, aes_block *input, uint32_t nb_blocks); +void crypton_aes_ppc8_decrypt_ecb(aes_block *output, aes_key *key, aes_block *input, uint32_t nb_blocks); +void crypton_aes_ppc8_encrypt_cbc(aes_block *output, aes_key *key, aes_block *iv, aes_block *input, uint32_t nb_blocks); +void crypton_aes_ppc8_decrypt_cbc(aes_block *output, aes_key *key, aes_block *iv, aes_block *input, uint32_t nb_blocks); +void crypton_aes_ppc8_encrypt_ctr(uint8_t *output, aes_key *key, aes_block *iv, uint8_t *input, uint32_t len); +void crypton_aes_ppc8_encrypt_xts(aes_block *output, aes_key *k1, aes_key *k2, aes_block *dataunit, uint32_t spoint, aes_block *input, uint32_t nb_blocks); +void crypton_aes_ppc8_decrypt_xts(aes_block *output, aes_key *k1, aes_key *k2, aes_block *dataunit, uint32_t spoint, aes_block *input, uint32_t nb_blocks); +void crypton_aes_ppc8_hinit(table_4bit htable, const block128 *h); +void crypton_aes_ppc8_gf_mul(block128 *a, const table_4bit htable); +void crypton_aes_ppc8_gf_mul4(block128 *a, const block128 *blocks, const table_4bit htable); + +#endif diff --git a/cbits/asm/README.md b/cbits/asm/README.md --- a/cbits/asm/README.md +++ b/cbits/asm/README.md @@ -2,7 +2,7 @@ ## What is here -Two modules from [CRYPTOGAMS](https://github.com/dot-asm/cryptogams), by Andy +Modules from [CRYPTOGAMS](https://github.com/dot-asm/cryptogams), by Andy Polyakov, checked in unmodified together with the translators they need: | generator | what it is | @@ -17,6 +17,8 @@ | `sha1-armv8.pl` | SHA-1 for AArch64 | | `sha512-armv8.pl` | SHA-256 for AArch64 (the generator emits SHA-512 or SHA-256 according to the name it is given, and only the latter is wanted) | | `keccak1600-armv8.pl` | Keccak for AArch64 | +| `aesp8-ppc.pl` | AES for PowerISA 2.07, which POWER8 was the first to implement | +| `ghashp8-ppc.pl` | GHASH for the same, over `vpmsumd` | `x86_64-xlate.pl`, `arm-xlate.pl` and `arm_arch.h` are the machinery those modules use. `generate.sh` runs the generators to produce the `.S` files, which are diff --git a/cbits/asm/aesp8-ppc-linux64le.S b/cbits/asm/aesp8-ppc-linux64le.S new file mode 100644 --- /dev/null +++ b/cbits/asm/aesp8-ppc-linux64le.S @@ -0,0 +1,3658 @@ +.machine "any" + +.abiversion 2 +.text + +.align 7 +rcon: +.byte 0x00,0x00,0x00,0x01,0x00,0x00,0x00,0x01,0x00,0x00,0x00,0x01,0x00,0x00,0x00,0x01 +.byte 0x00,0x00,0x00,0x1b,0x00,0x00,0x00,0x1b,0x00,0x00,0x00,0x1b,0x00,0x00,0x00,0x1b +.byte 0x0c,0x0f,0x0e,0x0d,0x0c,0x0f,0x0e,0x0d,0x0c,0x0f,0x0e,0x0d,0x0c,0x0f,0x0e,0x0d +.byte 0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00 +.Lconsts: + mflr 0 + bcl 20,31,$+4 + mflr 6 + addi 6,6,-0x48 + mtlr 0 + blr +.long 0 +.byte 0,12,0x14,0,0,0,0,0 +.byte 65,69,83,32,102,111,114,32,80,111,119,101,114,73,83,65,32,50,46,48,55,44,67,82,89,80,84,79,71,65,77,83,32,98,121,32,60,97,112,112,114,111,64,111,112,101,110,115,115,108,46,111,114,103,62,0 +.align 2 + +.globl crypton_aes_p8_set_encrypt_key +.type crypton_aes_p8_set_encrypt_key,@function +.align 5 +crypton_aes_p8_set_encrypt_key: +.localentry crypton_aes_p8_set_encrypt_key,0 + +.Lset_encrypt_key: + mflr 11 + std 11,16(1) + + li 6,-1 + cmpldi 3,0 + beq- .Lenc_key_abort + cmpldi 5,0 + beq- .Lenc_key_abort + li 6,-2 + cmpwi 4,128 + blt- .Lenc_key_abort + cmpwi 4,256 + bgt- .Lenc_key_abort + andi. 0,4,0x3f + bne- .Lenc_key_abort + + lis 0,0xfff0 + li 12,-1 + or 0,0,0 + + bl .Lconsts + mtlr 11 + + neg 9,3 + lvx 1,0,3 + addi 3,3,15 + lvsr 3,0,9 + li 8,0x20 + cmpwi 4,192 + lvx 2,0,3 + vspltisb 5,0x0f + lvx 4,0,6 + vxor 3,3,5 + lvx 5,8,6 + addi 6,6,0x10 + vperm 1,1,2,3 + li 7,8 + vxor 0,0,0 + mtctr 7 + + lvsl 8,0,5 + vspltisb 9,-1 + lvx 10,0,5 + vperm 9,9,0,8 + + blt .Loop128 + addi 3,3,8 + beq .L192 + addi 3,3,8 + b .L256 + +.align 4 +.Loop128: + vperm 3,1,1,5 + vsldoi 6,0,1,12 + vperm 11,1,1,8 + vsel 7,10,11,9 + vor 10,11,11 + .long 0x10632509 + stvx 7,0,5 + addi 5,5,16 + + vxor 1,1,6 + vsldoi 6,0,6,12 + vxor 1,1,6 + vsldoi 6,0,6,12 + vxor 1,1,6 + vadduwm 4,4,4 + vxor 1,1,3 + bdnz .Loop128 + + lvx 4,0,6 + + vperm 3,1,1,5 + vsldoi 6,0,1,12 + vperm 11,1,1,8 + vsel 7,10,11,9 + vor 10,11,11 + .long 0x10632509 + stvx 7,0,5 + addi 5,5,16 + + vxor 1,1,6 + vsldoi 6,0,6,12 + vxor 1,1,6 + vsldoi 6,0,6,12 + vxor 1,1,6 + vadduwm 4,4,4 + vxor 1,1,3 + + vperm 3,1,1,5 + vsldoi 6,0,1,12 + vperm 11,1,1,8 + vsel 7,10,11,9 + vor 10,11,11 + .long 0x10632509 + stvx 7,0,5 + addi 5,5,16 + + vxor 1,1,6 + vsldoi 6,0,6,12 + vxor 1,1,6 + vsldoi 6,0,6,12 + vxor 1,1,6 + vxor 1,1,3 + vperm 11,1,1,8 + vsel 7,10,11,9 + vor 10,11,11 + stvx 7,0,5 + + addi 3,5,15 + addi 5,5,0x50 + + li 8,10 + b .Ldone + +.align 4 +.L192: + lvx 6,0,3 + li 7,4 + vperm 11,1,1,8 + vsel 7,10,11,9 + vor 10,11,11 + stvx 7,0,5 + addi 5,5,16 + vperm 2,2,6,3 + vspltisb 3,8 + mtctr 7 + vsububm 5,5,3 + +.Loop192: + vperm 3,2,2,5 + vsldoi 6,0,1,12 + .long 0x10632509 + + vxor 1,1,6 + vsldoi 6,0,6,12 + vxor 1,1,6 + vsldoi 6,0,6,12 + vxor 1,1,6 + + vsldoi 7,0,2,8 + vspltw 6,1,3 + vxor 6,6,2 + vsldoi 2,0,2,12 + vadduwm 4,4,4 + vxor 2,2,6 + vxor 1,1,3 + vxor 2,2,3 + vsldoi 7,7,1,8 + + vperm 3,2,2,5 + vsldoi 6,0,1,12 + vperm 11,7,7,8 + vsel 7,10,11,9 + vor 10,11,11 + .long 0x10632509 + stvx 7,0,5 + addi 5,5,16 + + vsldoi 7,1,2,8 + vxor 1,1,6 + vsldoi 6,0,6,12 + vperm 11,7,7,8 + vsel 7,10,11,9 + vor 10,11,11 + vxor 1,1,6 + vsldoi 6,0,6,12 + vxor 1,1,6 + stvx 7,0,5 + addi 5,5,16 + + vspltw 6,1,3 + vxor 6,6,2 + vsldoi 2,0,2,12 + vadduwm 4,4,4 + vxor 2,2,6 + vxor 1,1,3 + vxor 2,2,3 + vperm 11,1,1,8 + vsel 7,10,11,9 + vor 10,11,11 + stvx 7,0,5 + addi 3,5,15 + addi 5,5,16 + bdnz .Loop192 + + li 8,12 + addi 5,5,0x20 + b .Ldone + +.align 4 +.L256: + lvx 6,0,3 + li 7,7 + li 8,14 + vperm 11,1,1,8 + vsel 7,10,11,9 + vor 10,11,11 + stvx 7,0,5 + addi 5,5,16 + vperm 2,2,6,3 + mtctr 7 + +.Loop256: + vperm 3,2,2,5 + vsldoi 6,0,1,12 + vperm 11,2,2,8 + vsel 7,10,11,9 + vor 10,11,11 + .long 0x10632509 + stvx 7,0,5 + addi 5,5,16 + + vxor 1,1,6 + vsldoi 6,0,6,12 + vxor 1,1,6 + vsldoi 6,0,6,12 + vxor 1,1,6 + vadduwm 4,4,4 + vxor 1,1,3 + vperm 11,1,1,8 + vsel 7,10,11,9 + vor 10,11,11 + stvx 7,0,5 + addi 3,5,15 + addi 5,5,16 + bdz .Ldone + + vspltw 3,1,3 + vsldoi 6,0,2,12 + .long 0x106305C8 + + vxor 2,2,6 + vsldoi 6,0,6,12 + vxor 2,2,6 + vsldoi 6,0,6,12 + vxor 2,2,6 + + vxor 2,2,3 + b .Loop256 + +.align 4 +.Ldone: + lvx 2,0,3 + vsel 2,10,2,9 + stvx 2,0,3 + li 6,0 + or 12,12,12 + stw 8,0(5) + +.Lenc_key_abort: + mr 3,6 + blr +.long 0 +.byte 0,12,0x14,1,0,0,3,0 +.long 0 +.size crypton_aes_p8_set_encrypt_key,.-crypton_aes_p8_set_encrypt_key + +.globl crypton_aes_p8_set_decrypt_key +.type crypton_aes_p8_set_decrypt_key,@function +.align 5 +crypton_aes_p8_set_decrypt_key: +.localentry crypton_aes_p8_set_decrypt_key,0 + + stdu 1,-64(1) + mflr 10 + std 10,64+16(1) + bl .Lset_encrypt_key + mtlr 10 + + cmpwi 3,0 + bne- .Ldec_key_abort + + slwi 7,8,4 + subi 3,5,240 + srwi 8,8,1 + add 5,3,7 + mtctr 8 + +.Ldeckey: + lwz 0, 0(3) + lwz 6, 4(3) + lwz 7, 8(3) + lwz 8, 12(3) + addi 3,3,16 + lwz 9, 0(5) + lwz 10,4(5) + lwz 11,8(5) + lwz 12,12(5) + stw 0, 0(5) + stw 6, 4(5) + stw 7, 8(5) + stw 8, 12(5) + subi 5,5,16 + stw 9, -16(3) + stw 10,-12(3) + stw 11,-8(3) + stw 12,-4(3) + bdnz .Ldeckey + + xor 3,3,3 +.Ldec_key_abort: + addi 1,1,64 + blr +.long 0 +.byte 0,12,4,1,0x80,0,3,0 +.long 0 +.size crypton_aes_p8_set_decrypt_key,.-crypton_aes_p8_set_decrypt_key +.globl crypton_aes_p8_encrypt +.type crypton_aes_p8_encrypt,@function +.align 5 +crypton_aes_p8_encrypt: +.localentry crypton_aes_p8_encrypt,0 + + lwz 6,240(5) + lis 0,0xfc00 + li 12,-1 + li 7,15 + or 0,0,0 + + lvx 0,0,3 + neg 11,4 + lvx 1,7,3 + lvsl 2,0,3 + vspltisb 4,0x0f + lvsr 3,0,11 + vxor 2,2,4 + li 7,16 + vperm 0,0,1,2 + lvx 1,0,5 + lvsr 5,0,5 + srwi 6,6,1 + lvx 2,7,5 + addi 7,7,16 + subi 6,6,1 + vperm 1,2,1,5 + + vxor 0,0,1 + lvx 1,7,5 + addi 7,7,16 + mtctr 6 + +.Loop_enc: + vperm 2,1,2,5 + .long 0x10001508 + lvx 2,7,5 + addi 7,7,16 + vperm 1,2,1,5 + .long 0x10000D08 + lvx 1,7,5 + addi 7,7,16 + bdnz .Loop_enc + + vperm 2,1,2,5 + .long 0x10001508 + lvx 2,7,5 + vperm 1,2,1,5 + .long 0x10000D09 + + vspltisb 2,-1 + vxor 1,1,1 + li 7,15 + vperm 2,2,1,3 + vxor 3,3,4 + lvx 1,0,4 + vperm 0,0,0,3 + vsel 1,1,0,2 + lvx 4,7,4 + stvx 1,0,4 + vsel 0,0,4,2 + stvx 0,7,4 + + or 12,12,12 + blr +.long 0 +.byte 0,12,0x14,0,0,0,3,0 +.long 0 +.size crypton_aes_p8_encrypt,.-crypton_aes_p8_encrypt +.globl crypton_aes_p8_decrypt +.type crypton_aes_p8_decrypt,@function +.align 5 +crypton_aes_p8_decrypt: +.localentry crypton_aes_p8_decrypt,0 + + lwz 6,240(5) + lis 0,0xfc00 + li 12,-1 + li 7,15 + or 0,0,0 + + lvx 0,0,3 + neg 11,4 + lvx 1,7,3 + lvsl 2,0,3 + vspltisb 4,0x0f + lvsr 3,0,11 + vxor 2,2,4 + li 7,16 + vperm 0,0,1,2 + lvx 1,0,5 + lvsr 5,0,5 + srwi 6,6,1 + lvx 2,7,5 + addi 7,7,16 + subi 6,6,1 + vperm 1,2,1,5 + + vxor 0,0,1 + lvx 1,7,5 + addi 7,7,16 + mtctr 6 + +.Loop_dec: + vperm 2,1,2,5 + .long 0x10001548 + lvx 2,7,5 + addi 7,7,16 + vperm 1,2,1,5 + .long 0x10000D48 + lvx 1,7,5 + addi 7,7,16 + bdnz .Loop_dec + + vperm 2,1,2,5 + .long 0x10001548 + lvx 2,7,5 + vperm 1,2,1,5 + .long 0x10000D49 + + vspltisb 2,-1 + vxor 1,1,1 + li 7,15 + vperm 2,2,1,3 + vxor 3,3,4 + lvx 1,0,4 + vperm 0,0,0,3 + vsel 1,1,0,2 + lvx 4,7,4 + stvx 1,0,4 + vsel 0,0,4,2 + stvx 0,7,4 + + or 12,12,12 + blr +.long 0 +.byte 0,12,0x14,0,0,0,3,0 +.long 0 +.size crypton_aes_p8_decrypt,.-crypton_aes_p8_decrypt +.globl crypton_aes_p8_cbc_encrypt +.type crypton_aes_p8_cbc_encrypt,@function +.align 5 +crypton_aes_p8_cbc_encrypt: +.localentry crypton_aes_p8_cbc_encrypt,0 + + cmpldi 5,16 + .long 0x4dc00020 + + cmpwi 8,0 + lis 0,0xffe0 + li 12,-1 + or 0,0,0 + + li 10,15 + vxor 0,0,0 + vspltisb 3,0x0f + + lvx 4,0,7 + lvsl 6,0,7 + lvx 5,10,7 + vxor 6,6,3 + vperm 4,4,5,6 + + neg 11,3 + lvsr 10,0,6 + lwz 9,240(6) + + lvsr 6,0,11 + lvx 5,0,3 + addi 3,3,15 + vxor 6,6,3 + + lvsl 8,0,4 + vspltisb 9,-1 + lvx 7,0,4 + vperm 9,9,0,8 + vxor 8,8,3 + + srwi 9,9,1 + li 10,16 + subi 9,9,1 + beq .Lcbc_dec + +.Lcbc_enc: + vor 2,5,5 + lvx 5,0,3 + addi 3,3,16 + mtctr 9 + subi 5,5,16 + + lvx 0,0,6 + vperm 2,2,5,6 + lvx 1,10,6 + addi 10,10,16 + vperm 0,1,0,10 + vxor 2,2,0 + lvx 0,10,6 + addi 10,10,16 + vxor 2,2,4 + +.Loop_cbc_enc: + vperm 1,0,1,10 + .long 0x10420D08 + lvx 1,10,6 + addi 10,10,16 + vperm 0,1,0,10 + .long 0x10420508 + lvx 0,10,6 + addi 10,10,16 + bdnz .Loop_cbc_enc + + vperm 1,0,1,10 + .long 0x10420D08 + lvx 1,10,6 + li 10,16 + vperm 0,1,0,10 + .long 0x10820509 + cmpldi 5,16 + + vperm 3,4,4,8 + vsel 2,7,3,9 + vor 7,3,3 + stvx 2,0,4 + addi 4,4,16 + bge .Lcbc_enc + + b .Lcbc_done + +.align 4 +.Lcbc_dec: + cmpldi 5,128 + bge _aesp8_cbc_decrypt8x + vor 3,5,5 + lvx 5,0,3 + addi 3,3,16 + mtctr 9 + subi 5,5,16 + + lvx 0,0,6 + vperm 3,3,5,6 + lvx 1,10,6 + addi 10,10,16 + vperm 0,1,0,10 + vxor 2,3,0 + lvx 0,10,6 + addi 10,10,16 + +.Loop_cbc_dec: + vperm 1,0,1,10 + .long 0x10420D48 + lvx 1,10,6 + addi 10,10,16 + vperm 0,1,0,10 + .long 0x10420548 + lvx 0,10,6 + addi 10,10,16 + bdnz .Loop_cbc_dec + + vperm 1,0,1,10 + .long 0x10420D48 + lvx 1,10,6 + li 10,16 + vperm 0,1,0,10 + .long 0x10420549 + cmpldi 5,16 + + vxor 2,2,4 + vor 4,3,3 + vperm 3,2,2,8 + vsel 2,7,3,9 + vor 7,3,3 + stvx 2,0,4 + addi 4,4,16 + bge .Lcbc_dec + +.Lcbc_done: + addi 4,4,-1 + lvx 2,0,4 + vsel 2,7,2,9 + stvx 2,0,4 + + neg 8,7 + li 10,15 + vxor 0,0,0 + vspltisb 9,-1 + vspltisb 3,0x0f + lvsr 8,0,8 + vperm 9,9,0,8 + vxor 8,8,3 + lvx 7,0,7 + vperm 4,4,4,8 + vsel 2,7,4,9 + lvx 5,10,7 + stvx 2,0,7 + vsel 2,4,5,9 + stvx 2,10,7 + + or 12,12,12 + blr +.long 0 +.byte 0,12,0x14,0,0,0,6,0 +.long 0 +.align 5 +_aesp8_cbc_decrypt8x: + stdu 1,-448(1) + li 10,207 + li 11,223 + stvx 20,10,1 + addi 10,10,32 + stvx 21,11,1 + addi 11,11,32 + stvx 22,10,1 + addi 10,10,32 + stvx 23,11,1 + addi 11,11,32 + stvx 24,10,1 + addi 10,10,32 + stvx 25,11,1 + addi 11,11,32 + stvx 26,10,1 + addi 10,10,32 + stvx 27,11,1 + addi 11,11,32 + stvx 28,10,1 + addi 10,10,32 + stvx 29,11,1 + addi 11,11,32 + stvx 30,10,1 + stvx 31,11,1 + li 0,-1 + stw 12,396(1) + li 8,0x10 + std 26,400(1) + li 26,0x20 + std 27,408(1) + li 27,0x30 + std 28,416(1) + li 28,0x40 + std 29,424(1) + li 29,0x50 + std 30,432(1) + li 30,0x60 + std 31,440(1) + li 31,0x70 + or 0,0,0 + + subi 9,9,3 + subi 5,5,128 + + lvx 23,0,6 + lvx 30,8,6 + addi 6,6,0x20 + lvx 31,0,6 + vperm 23,30,23,10 + addi 11,1,64+15 + mtctr 9 + +.Load_cbc_dec_key: + vperm 24,31,30,10 + lvx 30,8,6 + addi 6,6,0x20 + stvx 24,0,11 + vperm 25,30,31,10 + lvx 31,0,6 + stvx 25,8,11 + addi 11,11,0x20 + bdnz .Load_cbc_dec_key + + lvx 26,8,6 + vperm 24,31,30,10 + lvx 27,26,6 + stvx 24,0,11 + vperm 25,26,31,10 + lvx 28,27,6 + stvx 25,8,11 + addi 11,1,64+15 + vperm 26,27,26,10 + lvx 29,28,6 + vperm 27,28,27,10 + lvx 30,29,6 + vperm 28,29,28,10 + lvx 31,30,6 + vperm 29,30,29,10 + lvx 14,31,6 + vperm 30,31,30,10 + lvx 24,0,11 + vperm 31,14,31,10 + lvx 25,8,11 + + + + subi 3,3,15 + + li 10,8 + .long 0x7C001E99 + lvsl 6,0,10 + vspltisb 3,0x0f + .long 0x7C281E99 + vxor 6,6,3 + .long 0x7C5A1E99 + vperm 0,0,0,6 + .long 0x7C7B1E99 + vperm 1,1,1,6 + .long 0x7D5C1E99 + vperm 2,2,2,6 + vxor 14,0,23 + .long 0x7D7D1E99 + vperm 3,3,3,6 + vxor 15,1,23 + .long 0x7D9E1E99 + vperm 10,10,10,6 + vxor 16,2,23 + .long 0x7DBF1E99 + addi 3,3,0x80 + vperm 11,11,11,6 + vxor 17,3,23 + vperm 12,12,12,6 + vxor 18,10,23 + vperm 13,13,13,6 + vxor 19,11,23 + vxor 20,12,23 + vxor 21,13,23 + + mtctr 9 + b .Loop_cbc_dec8x +.align 5 +.Loop_cbc_dec8x: + .long 0x11CEC548 + .long 0x11EFC548 + .long 0x1210C548 + .long 0x1231C548 + .long 0x1252C548 + .long 0x1273C548 + .long 0x1294C548 + .long 0x12B5C548 + lvx 24,26,11 + addi 11,11,0x20 + + .long 0x11CECD48 + .long 0x11EFCD48 + .long 0x1210CD48 + .long 0x1231CD48 + .long 0x1252CD48 + .long 0x1273CD48 + .long 0x1294CD48 + .long 0x12B5CD48 + lvx 25,8,11 + bdnz .Loop_cbc_dec8x + + subic 5,5,128 + .long 0x11CEC548 + .long 0x11EFC548 + .long 0x1210C548 + .long 0x1231C548 + .long 0x1252C548 + .long 0x1273C548 + .long 0x1294C548 + .long 0x12B5C548 + + subfe. 0,0,0 + .long 0x11CECD48 + .long 0x11EFCD48 + .long 0x1210CD48 + .long 0x1231CD48 + .long 0x1252CD48 + .long 0x1273CD48 + .long 0x1294CD48 + .long 0x12B5CD48 + + and 0,0,5 + .long 0x11CED548 + .long 0x11EFD548 + .long 0x1210D548 + .long 0x1231D548 + .long 0x1252D548 + .long 0x1273D548 + .long 0x1294D548 + .long 0x12B5D548 + + add 3,3,0 + + + + .long 0x11CEDD48 + .long 0x11EFDD48 + .long 0x1210DD48 + .long 0x1231DD48 + .long 0x1252DD48 + .long 0x1273DD48 + .long 0x1294DD48 + .long 0x12B5DD48 + + addi 11,1,64+15 + .long 0x11CEE548 + .long 0x11EFE548 + .long 0x1210E548 + .long 0x1231E548 + .long 0x1252E548 + .long 0x1273E548 + .long 0x1294E548 + .long 0x12B5E548 + lvx 24,0,11 + + .long 0x11CEED48 + .long 0x11EFED48 + .long 0x1210ED48 + .long 0x1231ED48 + .long 0x1252ED48 + .long 0x1273ED48 + .long 0x1294ED48 + .long 0x12B5ED48 + lvx 25,8,11 + + .long 0x11CEF548 + vxor 4,4,31 + .long 0x11EFF548 + vxor 0,0,31 + .long 0x1210F548 + vxor 1,1,31 + .long 0x1231F548 + vxor 2,2,31 + .long 0x1252F548 + vxor 3,3,31 + .long 0x1273F548 + vxor 10,10,31 + .long 0x1294F548 + vxor 11,11,31 + .long 0x12B5F548 + vxor 12,12,31 + + .long 0x11CE2549 + .long 0x11EF0549 + .long 0x7C001E99 + .long 0x12100D49 + .long 0x7C281E99 + .long 0x12311549 + vperm 0,0,0,6 + .long 0x7C5A1E99 + .long 0x12521D49 + vperm 1,1,1,6 + .long 0x7C7B1E99 + .long 0x12735549 + vperm 2,2,2,6 + .long 0x7D5C1E99 + .long 0x12945D49 + vperm 3,3,3,6 + .long 0x7D7D1E99 + .long 0x12B56549 + vperm 10,10,10,6 + .long 0x7D9E1E99 + vor 4,13,13 + vperm 11,11,11,6 + .long 0x7DBF1E99 + addi 3,3,0x80 + + vperm 14,14,14,6 + vperm 15,15,15,6 + .long 0x7DC02799 + vperm 12,12,12,6 + vxor 14,0,23 + vperm 16,16,16,6 + .long 0x7DE82799 + vperm 13,13,13,6 + vxor 15,1,23 + vperm 17,17,17,6 + .long 0x7E1A2799 + vxor 16,2,23 + vperm 18,18,18,6 + .long 0x7E3B2799 + vxor 17,3,23 + vperm 19,19,19,6 + .long 0x7E5C2799 + vxor 18,10,23 + vperm 20,20,20,6 + .long 0x7E7D2799 + vxor 19,11,23 + vperm 21,21,21,6 + .long 0x7E9E2799 + vxor 20,12,23 + .long 0x7EBF2799 + addi 4,4,0x80 + vxor 21,13,23 + + mtctr 9 + beq .Loop_cbc_dec8x + + addic. 5,5,128 + beq .Lcbc_dec8x_done + nop + nop + +.Loop_cbc_dec8x_tail: + .long 0x11EFC548 + .long 0x1210C548 + .long 0x1231C548 + .long 0x1252C548 + .long 0x1273C548 + .long 0x1294C548 + .long 0x12B5C548 + lvx 24,26,11 + addi 11,11,0x20 + + .long 0x11EFCD48 + .long 0x1210CD48 + .long 0x1231CD48 + .long 0x1252CD48 + .long 0x1273CD48 + .long 0x1294CD48 + .long 0x12B5CD48 + lvx 25,8,11 + bdnz .Loop_cbc_dec8x_tail + + .long 0x11EFC548 + .long 0x1210C548 + .long 0x1231C548 + .long 0x1252C548 + .long 0x1273C548 + .long 0x1294C548 + .long 0x12B5C548 + + .long 0x11EFCD48 + .long 0x1210CD48 + .long 0x1231CD48 + .long 0x1252CD48 + .long 0x1273CD48 + .long 0x1294CD48 + .long 0x12B5CD48 + + .long 0x11EFD548 + .long 0x1210D548 + .long 0x1231D548 + .long 0x1252D548 + .long 0x1273D548 + .long 0x1294D548 + .long 0x12B5D548 + + .long 0x11EFDD48 + .long 0x1210DD48 + .long 0x1231DD48 + .long 0x1252DD48 + .long 0x1273DD48 + .long 0x1294DD48 + .long 0x12B5DD48 + + .long 0x11EFE548 + .long 0x1210E548 + .long 0x1231E548 + .long 0x1252E548 + .long 0x1273E548 + .long 0x1294E548 + .long 0x12B5E548 + + .long 0x11EFED48 + .long 0x1210ED48 + .long 0x1231ED48 + .long 0x1252ED48 + .long 0x1273ED48 + .long 0x1294ED48 + .long 0x12B5ED48 + + .long 0x11EFF548 + vxor 4,4,31 + .long 0x1210F548 + vxor 1,1,31 + .long 0x1231F548 + vxor 2,2,31 + .long 0x1252F548 + vxor 3,3,31 + .long 0x1273F548 + vxor 10,10,31 + .long 0x1294F548 + vxor 11,11,31 + .long 0x12B5F548 + vxor 12,12,31 + + cmplwi 5,32 + blt .Lcbc_dec8x_one + nop + beq .Lcbc_dec8x_two + cmplwi 5,64 + blt .Lcbc_dec8x_three + nop + beq .Lcbc_dec8x_four + cmplwi 5,96 + blt .Lcbc_dec8x_five + nop + beq .Lcbc_dec8x_six + +.Lcbc_dec8x_seven: + .long 0x11EF2549 + .long 0x12100D49 + .long 0x12311549 + .long 0x12521D49 + .long 0x12735549 + .long 0x12945D49 + .long 0x12B56549 + vor 4,13,13 + + vperm 15,15,15,6 + vperm 16,16,16,6 + .long 0x7DE02799 + vperm 17,17,17,6 + .long 0x7E082799 + vperm 18,18,18,6 + .long 0x7E3A2799 + vperm 19,19,19,6 + .long 0x7E5B2799 + vperm 20,20,20,6 + .long 0x7E7C2799 + vperm 21,21,21,6 + .long 0x7E9D2799 + .long 0x7EBE2799 + addi 4,4,0x70 + b .Lcbc_dec8x_done + +.align 5 +.Lcbc_dec8x_six: + .long 0x12102549 + .long 0x12311549 + .long 0x12521D49 + .long 0x12735549 + .long 0x12945D49 + .long 0x12B56549 + vor 4,13,13 + + vperm 16,16,16,6 + vperm 17,17,17,6 + .long 0x7E002799 + vperm 18,18,18,6 + .long 0x7E282799 + vperm 19,19,19,6 + .long 0x7E5A2799 + vperm 20,20,20,6 + .long 0x7E7B2799 + vperm 21,21,21,6 + .long 0x7E9C2799 + .long 0x7EBD2799 + addi 4,4,0x60 + b .Lcbc_dec8x_done + +.align 5 +.Lcbc_dec8x_five: + .long 0x12312549 + .long 0x12521D49 + .long 0x12735549 + .long 0x12945D49 + .long 0x12B56549 + vor 4,13,13 + + vperm 17,17,17,6 + vperm 18,18,18,6 + .long 0x7E202799 + vperm 19,19,19,6 + .long 0x7E482799 + vperm 20,20,20,6 + .long 0x7E7A2799 + vperm 21,21,21,6 + .long 0x7E9B2799 + .long 0x7EBC2799 + addi 4,4,0x50 + b .Lcbc_dec8x_done + +.align 5 +.Lcbc_dec8x_four: + .long 0x12522549 + .long 0x12735549 + .long 0x12945D49 + .long 0x12B56549 + vor 4,13,13 + + vperm 18,18,18,6 + vperm 19,19,19,6 + .long 0x7E402799 + vperm 20,20,20,6 + .long 0x7E682799 + vperm 21,21,21,6 + .long 0x7E9A2799 + .long 0x7EBB2799 + addi 4,4,0x40 + b .Lcbc_dec8x_done + +.align 5 +.Lcbc_dec8x_three: + .long 0x12732549 + .long 0x12945D49 + .long 0x12B56549 + vor 4,13,13 + + vperm 19,19,19,6 + vperm 20,20,20,6 + .long 0x7E602799 + vperm 21,21,21,6 + .long 0x7E882799 + .long 0x7EBA2799 + addi 4,4,0x30 + b .Lcbc_dec8x_done + +.align 5 +.Lcbc_dec8x_two: + .long 0x12942549 + .long 0x12B56549 + vor 4,13,13 + + vperm 20,20,20,6 + vperm 21,21,21,6 + .long 0x7E802799 + .long 0x7EA82799 + addi 4,4,0x20 + b .Lcbc_dec8x_done + +.align 5 +.Lcbc_dec8x_one: + .long 0x12B52549 + vor 4,13,13 + + vperm 21,21,21,6 + .long 0x7EA02799 + addi 4,4,0x10 + +.Lcbc_dec8x_done: + vperm 4,4,4,6 + .long 0x7C803F99 + + li 10,79 + li 11,95 + stvx 6,10,1 + addi 10,10,32 + stvx 6,11,1 + addi 11,11,32 + stvx 6,10,1 + addi 10,10,32 + stvx 6,11,1 + addi 11,11,32 + stvx 6,10,1 + addi 10,10,32 + stvx 6,11,1 + addi 11,11,32 + stvx 6,10,1 + addi 10,10,32 + stvx 6,11,1 + addi 11,11,32 + + or 12,12,12 + lvx 20,10,1 + addi 10,10,32 + lvx 21,11,1 + addi 11,11,32 + lvx 22,10,1 + addi 10,10,32 + lvx 23,11,1 + addi 11,11,32 + lvx 24,10,1 + addi 10,10,32 + lvx 25,11,1 + addi 11,11,32 + lvx 26,10,1 + addi 10,10,32 + lvx 27,11,1 + addi 11,11,32 + lvx 28,10,1 + addi 10,10,32 + lvx 29,11,1 + addi 11,11,32 + lvx 30,10,1 + lvx 31,11,1 + ld 26,400(1) + ld 27,408(1) + ld 28,416(1) + ld 29,424(1) + ld 30,432(1) + ld 31,440(1) + addi 1,1,448 + blr +.long 0 +.byte 0,12,0x04,0,0x80,6,6,0 +.long 0 +.size crypton_aes_p8_cbc_encrypt,.-crypton_aes_p8_cbc_encrypt +.globl crypton_aes_p8_ctr32_encrypt_blocks +.type crypton_aes_p8_ctr32_encrypt_blocks,@function +.align 5 +crypton_aes_p8_ctr32_encrypt_blocks: +.localentry crypton_aes_p8_ctr32_encrypt_blocks,0 + + cmpldi 5,1 + .long 0x4dc00020 + + lis 0,0xfff0 + li 12,-1 + or 0,0,0 + + li 10,15 + vxor 0,0,0 + vspltisb 3,0x0f + + lvx 4,0,7 + lvsl 6,0,7 + lvx 5,10,7 + vspltisb 11,1 + vxor 6,6,3 + vperm 4,4,5,6 + vsldoi 11,0,11,1 + + neg 11,3 + lvsr 10,0,6 + lwz 9,240(6) + + lvsr 6,0,11 + lvx 5,0,3 + addi 3,3,15 + vxor 6,6,3 + + srwi 9,9,1 + li 10,16 + subi 9,9,1 + + cmpldi 5,8 + bge _aesp8_ctr32_encrypt8x + + lvsl 8,0,4 + vspltisb 9,-1 + lvx 7,0,4 + vperm 9,9,0,8 + vxor 8,8,3 + + lvx 0,0,6 + mtctr 9 + lvx 1,10,6 + addi 10,10,16 + vperm 0,1,0,10 + vxor 2,4,0 + lvx 0,10,6 + addi 10,10,16 + b .Loop_ctr32_enc + +.align 5 +.Loop_ctr32_enc: + vperm 1,0,1,10 + .long 0x10420D08 + lvx 1,10,6 + addi 10,10,16 + vperm 0,1,0,10 + .long 0x10420508 + lvx 0,10,6 + addi 10,10,16 + bdnz .Loop_ctr32_enc + + vadduwm 4,4,11 + vor 3,5,5 + lvx 5,0,3 + addi 3,3,16 + subic. 5,5,1 + + vperm 1,0,1,10 + .long 0x10420D08 + lvx 1,10,6 + vperm 3,3,5,6 + li 10,16 + vperm 1,1,0,10 + lvx 0,0,6 + vxor 3,3,1 + .long 0x10421D09 + + lvx 1,10,6 + addi 10,10,16 + vperm 2,2,2,8 + vsel 3,7,2,9 + mtctr 9 + vperm 0,1,0,10 + vor 7,2,2 + vxor 2,4,0 + lvx 0,10,6 + addi 10,10,16 + stvx 3,0,4 + addi 4,4,16 + bne .Loop_ctr32_enc + + addi 4,4,-1 + lvx 2,0,4 + vsel 2,7,2,9 + stvx 2,0,4 + + or 12,12,12 + blr +.long 0 +.byte 0,12,0x14,0,0,0,6,0 +.long 0 +.align 5 +_aesp8_ctr32_encrypt8x: + stdu 1,-448(1) + li 10,207 + li 11,223 + stvx 20,10,1 + addi 10,10,32 + stvx 21,11,1 + addi 11,11,32 + stvx 22,10,1 + addi 10,10,32 + stvx 23,11,1 + addi 11,11,32 + stvx 24,10,1 + addi 10,10,32 + stvx 25,11,1 + addi 11,11,32 + stvx 26,10,1 + addi 10,10,32 + stvx 27,11,1 + addi 11,11,32 + stvx 28,10,1 + addi 10,10,32 + stvx 29,11,1 + addi 11,11,32 + stvx 30,10,1 + stvx 31,11,1 + li 0,-1 + stw 12,396(1) + li 8,0x10 + std 26,400(1) + li 26,0x20 + std 27,408(1) + li 27,0x30 + std 28,416(1) + li 28,0x40 + std 29,424(1) + li 29,0x50 + std 30,432(1) + li 30,0x60 + std 31,440(1) + li 31,0x70 + or 0,0,0 + + subi 9,9,3 + + lvx 23,0,6 + lvx 30,8,6 + addi 6,6,0x20 + lvx 31,0,6 + vperm 23,30,23,10 + addi 11,1,64+15 + mtctr 9 + +.Load_ctr32_enc_key: + vperm 24,31,30,10 + lvx 30,8,6 + addi 6,6,0x20 + stvx 24,0,11 + vperm 25,30,31,10 + lvx 31,0,6 + stvx 25,8,11 + addi 11,11,0x20 + bdnz .Load_ctr32_enc_key + + lvx 26,8,6 + vperm 24,31,30,10 + lvx 27,26,6 + stvx 24,0,11 + vperm 25,26,31,10 + lvx 28,27,6 + stvx 25,8,11 + addi 11,1,64+15 + vperm 26,27,26,10 + lvx 29,28,6 + vperm 27,28,27,10 + lvx 30,29,6 + vperm 28,29,28,10 + lvx 31,30,6 + vperm 29,30,29,10 + lvx 15,31,6 + vperm 30,31,30,10 + lvx 24,0,11 + vperm 31,15,31,10 + lvx 25,8,11 + + vadduwm 7,11,11 + subi 3,3,15 + sldi 5,5,4 + + vadduwm 16,4,11 + vadduwm 17,4,7 + vxor 15,4,23 + li 10,8 + vadduwm 18,16,7 + vxor 16,16,23 + lvsl 6,0,10 + vadduwm 19,17,7 + vxor 17,17,23 + vspltisb 3,0x0f + vadduwm 20,18,7 + vxor 18,18,23 + vxor 6,6,3 + vadduwm 21,19,7 + vxor 19,19,23 + vadduwm 22,20,7 + vxor 20,20,23 + vadduwm 4,21,7 + vxor 21,21,23 + vxor 22,22,23 + + mtctr 9 + b .Loop_ctr32_enc8x +.align 5 +.Loop_ctr32_enc8x: + .long 0x11EFC508 + .long 0x1210C508 + .long 0x1231C508 + .long 0x1252C508 + .long 0x1273C508 + .long 0x1294C508 + .long 0x12B5C508 + .long 0x12D6C508 +.Loop_ctr32_enc8x_middle: + lvx 24,26,11 + addi 11,11,0x20 + + .long 0x11EFCD08 + .long 0x1210CD08 + .long 0x1231CD08 + .long 0x1252CD08 + .long 0x1273CD08 + .long 0x1294CD08 + .long 0x12B5CD08 + .long 0x12D6CD08 + lvx 25,8,11 + bdnz .Loop_ctr32_enc8x + + subic 11,5,256 + .long 0x11EFC508 + .long 0x1210C508 + .long 0x1231C508 + .long 0x1252C508 + .long 0x1273C508 + .long 0x1294C508 + .long 0x12B5C508 + .long 0x12D6C508 + + subfe 0,0,0 + .long 0x11EFCD08 + .long 0x1210CD08 + .long 0x1231CD08 + .long 0x1252CD08 + .long 0x1273CD08 + .long 0x1294CD08 + .long 0x12B5CD08 + .long 0x12D6CD08 + + and 0,0,11 + addi 11,1,64+15 + .long 0x11EFD508 + .long 0x1210D508 + .long 0x1231D508 + .long 0x1252D508 + .long 0x1273D508 + .long 0x1294D508 + .long 0x12B5D508 + .long 0x12D6D508 + lvx 24,0,11 + + subic 5,5,129 + .long 0x11EFDD08 + addi 5,5,1 + .long 0x1210DD08 + .long 0x1231DD08 + .long 0x1252DD08 + .long 0x1273DD08 + .long 0x1294DD08 + .long 0x12B5DD08 + .long 0x12D6DD08 + lvx 25,8,11 + + .long 0x11EFE508 + .long 0x7C001E99 + .long 0x1210E508 + .long 0x7C281E99 + .long 0x1231E508 + .long 0x7C5A1E99 + .long 0x1252E508 + .long 0x7C7B1E99 + .long 0x1273E508 + .long 0x7D5C1E99 + .long 0x1294E508 + .long 0x7D9D1E99 + .long 0x12B5E508 + .long 0x7DBE1E99 + .long 0x12D6E508 + .long 0x7DDF1E99 + addi 3,3,0x80 + + .long 0x11EFED08 + vperm 0,0,0,6 + .long 0x1210ED08 + vperm 1,1,1,6 + .long 0x1231ED08 + vperm 2,2,2,6 + .long 0x1252ED08 + vperm 3,3,3,6 + .long 0x1273ED08 + vperm 10,10,10,6 + .long 0x1294ED08 + vperm 12,12,12,6 + .long 0x12B5ED08 + vperm 13,13,13,6 + .long 0x12D6ED08 + vperm 14,14,14,6 + + add 3,3,0 + + + + subfe. 0,0,0 + .long 0x11EFF508 + vxor 0,0,31 + .long 0x1210F508 + vxor 1,1,31 + .long 0x1231F508 + vxor 2,2,31 + .long 0x1252F508 + vxor 3,3,31 + .long 0x1273F508 + vxor 10,10,31 + .long 0x1294F508 + vxor 12,12,31 + .long 0x12B5F508 + vxor 13,13,31 + .long 0x12D6F508 + vxor 14,14,31 + + bne .Lctr32_enc8x_break + + .long 0x100F0509 + .long 0x10300D09 + vadduwm 16,4,11 + .long 0x10511509 + vadduwm 17,4,7 + vxor 15,4,23 + .long 0x10721D09 + vadduwm 18,16,7 + vxor 16,16,23 + .long 0x11535509 + vadduwm 19,17,7 + vxor 17,17,23 + .long 0x11946509 + vadduwm 20,18,7 + vxor 18,18,23 + .long 0x11B56D09 + vadduwm 21,19,7 + vxor 19,19,23 + .long 0x11D67509 + vadduwm 22,20,7 + vxor 20,20,23 + vperm 0,0,0,6 + vadduwm 4,21,7 + vxor 21,21,23 + vperm 1,1,1,6 + vxor 22,22,23 + mtctr 9 + + .long 0x11EFC508 + .long 0x7C002799 + vperm 2,2,2,6 + .long 0x1210C508 + .long 0x7C282799 + vperm 3,3,3,6 + .long 0x1231C508 + .long 0x7C5A2799 + vperm 10,10,10,6 + .long 0x1252C508 + .long 0x7C7B2799 + vperm 12,12,12,6 + .long 0x1273C508 + .long 0x7D5C2799 + vperm 13,13,13,6 + .long 0x1294C508 + .long 0x7D9D2799 + vperm 14,14,14,6 + .long 0x12B5C508 + .long 0x7DBE2799 + .long 0x12D6C508 + .long 0x7DDF2799 + addi 4,4,0x80 + + b .Loop_ctr32_enc8x_middle + +.align 5 +.Lctr32_enc8x_break: + cmpwi 5,-0x60 + blt .Lctr32_enc8x_one + nop + beq .Lctr32_enc8x_two + cmpwi 5,-0x40 + blt .Lctr32_enc8x_three + nop + beq .Lctr32_enc8x_four + cmpwi 5,-0x20 + blt .Lctr32_enc8x_five + nop + beq .Lctr32_enc8x_six + cmpwi 5,0x00 + blt .Lctr32_enc8x_seven + +.Lctr32_enc8x_eight: + .long 0x11EF0509 + .long 0x12100D09 + .long 0x12311509 + .long 0x12521D09 + .long 0x12735509 + .long 0x12946509 + .long 0x12B56D09 + .long 0x12D67509 + + vperm 15,15,15,6 + vperm 16,16,16,6 + .long 0x7DE02799 + vperm 17,17,17,6 + .long 0x7E082799 + vperm 18,18,18,6 + .long 0x7E3A2799 + vperm 19,19,19,6 + .long 0x7E5B2799 + vperm 20,20,20,6 + .long 0x7E7C2799 + vperm 21,21,21,6 + .long 0x7E9D2799 + vperm 22,22,22,6 + .long 0x7EBE2799 + .long 0x7EDF2799 + addi 4,4,0x80 + b .Lctr32_enc8x_done + +.align 5 +.Lctr32_enc8x_seven: + .long 0x11EF0D09 + .long 0x12101509 + .long 0x12311D09 + .long 0x12525509 + .long 0x12736509 + .long 0x12946D09 + .long 0x12B57509 + + vperm 15,15,15,6 + vperm 16,16,16,6 + .long 0x7DE02799 + vperm 17,17,17,6 + .long 0x7E082799 + vperm 18,18,18,6 + .long 0x7E3A2799 + vperm 19,19,19,6 + .long 0x7E5B2799 + vperm 20,20,20,6 + .long 0x7E7C2799 + vperm 21,21,21,6 + .long 0x7E9D2799 + .long 0x7EBE2799 + addi 4,4,0x70 + b .Lctr32_enc8x_done + +.align 5 +.Lctr32_enc8x_six: + .long 0x11EF1509 + .long 0x12101D09 + .long 0x12315509 + .long 0x12526509 + .long 0x12736D09 + .long 0x12947509 + + vperm 15,15,15,6 + vperm 16,16,16,6 + .long 0x7DE02799 + vperm 17,17,17,6 + .long 0x7E082799 + vperm 18,18,18,6 + .long 0x7E3A2799 + vperm 19,19,19,6 + .long 0x7E5B2799 + vperm 20,20,20,6 + .long 0x7E7C2799 + .long 0x7E9D2799 + addi 4,4,0x60 + b .Lctr32_enc8x_done + +.align 5 +.Lctr32_enc8x_five: + .long 0x11EF1D09 + .long 0x12105509 + .long 0x12316509 + .long 0x12526D09 + .long 0x12737509 + + vperm 15,15,15,6 + vperm 16,16,16,6 + .long 0x7DE02799 + vperm 17,17,17,6 + .long 0x7E082799 + vperm 18,18,18,6 + .long 0x7E3A2799 + vperm 19,19,19,6 + .long 0x7E5B2799 + .long 0x7E7C2799 + addi 4,4,0x50 + b .Lctr32_enc8x_done + +.align 5 +.Lctr32_enc8x_four: + .long 0x11EF5509 + .long 0x12106509 + .long 0x12316D09 + .long 0x12527509 + + vperm 15,15,15,6 + vperm 16,16,16,6 + .long 0x7DE02799 + vperm 17,17,17,6 + .long 0x7E082799 + vperm 18,18,18,6 + .long 0x7E3A2799 + .long 0x7E5B2799 + addi 4,4,0x40 + b .Lctr32_enc8x_done + +.align 5 +.Lctr32_enc8x_three: + .long 0x11EF6509 + .long 0x12106D09 + .long 0x12317509 + + vperm 15,15,15,6 + vperm 16,16,16,6 + .long 0x7DE02799 + vperm 17,17,17,6 + .long 0x7E082799 + .long 0x7E3A2799 + addi 4,4,0x30 + b .Lctr32_enc8x_done + +.align 5 +.Lctr32_enc8x_two: + .long 0x11EF6D09 + .long 0x12107509 + + vperm 15,15,15,6 + vperm 16,16,16,6 + .long 0x7DE02799 + .long 0x7E082799 + addi 4,4,0x20 + b .Lctr32_enc8x_done + +.align 5 +.Lctr32_enc8x_one: + .long 0x11EF7509 + + vperm 15,15,15,6 + .long 0x7DE02799 + addi 4,4,0x10 + +.Lctr32_enc8x_done: + li 10,79 + li 11,95 + stvx 6,10,1 + addi 10,10,32 + stvx 6,11,1 + addi 11,11,32 + stvx 6,10,1 + addi 10,10,32 + stvx 6,11,1 + addi 11,11,32 + stvx 6,10,1 + addi 10,10,32 + stvx 6,11,1 + addi 11,11,32 + stvx 6,10,1 + addi 10,10,32 + stvx 6,11,1 + addi 11,11,32 + + or 12,12,12 + lvx 20,10,1 + addi 10,10,32 + lvx 21,11,1 + addi 11,11,32 + lvx 22,10,1 + addi 10,10,32 + lvx 23,11,1 + addi 11,11,32 + lvx 24,10,1 + addi 10,10,32 + lvx 25,11,1 + addi 11,11,32 + lvx 26,10,1 + addi 10,10,32 + lvx 27,11,1 + addi 11,11,32 + lvx 28,10,1 + addi 10,10,32 + lvx 29,11,1 + addi 11,11,32 + lvx 30,10,1 + lvx 31,11,1 + ld 26,400(1) + ld 27,408(1) + ld 28,416(1) + ld 29,424(1) + ld 30,432(1) + ld 31,440(1) + addi 1,1,448 + blr +.long 0 +.byte 0,12,0x04,0,0x80,6,6,0 +.long 0 +.size crypton_aes_p8_ctr32_encrypt_blocks,.-crypton_aes_p8_ctr32_encrypt_blocks +.globl crypton_aes_p8_xts_encrypt +.type crypton_aes_p8_xts_encrypt,@function +.align 5 +crypton_aes_p8_xts_encrypt: +.localentry crypton_aes_p8_xts_encrypt,0 + + mr 10,3 + li 3,-1 + cmpldi 5,16 + .long 0x4dc00020 + + lis 0,0xfff0 + li 12,-1 + li 11,0 + or 0,0,0 + + vspltisb 9,0x07 + lvsl 6,11,11 + vspltisb 11,0x0f + vxor 6,6,9 + + li 3,15 + lvx 8,0,8 + lvsl 5,0,8 + lvx 4,3,8 + vxor 5,5,11 + vperm 8,8,4,5 + + neg 11,10 + lvsr 5,0,11 + lvx 2,0,10 + addi 10,10,15 + vxor 5,5,11 + + cmpldi 7,0 + beq .Lxts_enc_no_key2 + + lvsr 7,0,7 + lwz 9,240(7) + srwi 9,9,1 + subi 9,9,1 + li 3,16 + + lvx 0,0,7 + lvx 1,3,7 + addi 3,3,16 + vperm 0,1,0,7 + vxor 8,8,0 + lvx 0,3,7 + addi 3,3,16 + mtctr 9 + +.Ltweak_xts_enc: + vperm 1,0,1,7 + .long 0x11080D08 + lvx 1,3,7 + addi 3,3,16 + vperm 0,1,0,7 + .long 0x11080508 + lvx 0,3,7 + addi 3,3,16 + bdnz .Ltweak_xts_enc + + vperm 1,0,1,7 + .long 0x11080D08 + lvx 1,3,7 + vperm 0,1,0,7 + .long 0x11080509 + + li 8,0 + b .Lxts_enc + +.Lxts_enc_no_key2: + li 3,-16 + and 5,5,3 + + +.Lxts_enc: + lvx 4,0,10 + addi 10,10,16 + + lvsr 7,0,6 + lwz 9,240(6) + srwi 9,9,1 + subi 9,9,1 + li 3,16 + + vslb 10,9,9 + vor 10,10,9 + vspltisb 11,1 + vsldoi 10,10,11,15 + + cmpldi 5,96 + bge _aesp8_xts_encrypt6x + + andi. 7,5,15 + subic 0,5,32 + subi 7,7,16 + subfe 0,0,0 + and 0,0,7 + add 10,10,0 + + lvx 0,0,6 + lvx 1,3,6 + addi 3,3,16 + vperm 2,2,4,5 + vperm 0,1,0,7 + vxor 2,2,8 + vxor 2,2,0 + lvx 0,3,6 + addi 3,3,16 + mtctr 9 + b .Loop_xts_enc + +.align 5 +.Loop_xts_enc: + vperm 1,0,1,7 + .long 0x10420D08 + lvx 1,3,6 + addi 3,3,16 + vperm 0,1,0,7 + .long 0x10420508 + lvx 0,3,6 + addi 3,3,16 + bdnz .Loop_xts_enc + + vperm 1,0,1,7 + .long 0x10420D08 + lvx 1,3,6 + li 3,16 + vperm 0,1,0,7 + vxor 0,0,8 + .long 0x10620509 + + vperm 11,3,3,6 + + .long 0x7D602799 + + addi 4,4,16 + + subic. 5,5,16 + beq .Lxts_enc_done + + vor 2,4,4 + lvx 4,0,10 + addi 10,10,16 + lvx 0,0,6 + lvx 1,3,6 + addi 3,3,16 + + subic 0,5,32 + subfe 0,0,0 + and 0,0,7 + add 10,10,0 + + vsrab 11,8,9 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + vand 11,11,10 + vxor 8,8,11 + + vperm 2,2,4,5 + vperm 0,1,0,7 + vxor 2,2,8 + vxor 3,3,0 + vxor 2,2,0 + lvx 0,3,6 + addi 3,3,16 + + mtctr 9 + cmpldi 5,16 + bge .Loop_xts_enc + + vxor 3,3,8 + lvsr 5,0,5 + vxor 4,4,4 + vspltisb 11,-1 + vperm 4,4,11,5 + vsel 2,2,3,4 + + subi 11,4,17 + subi 4,4,16 + mtctr 5 + li 5,16 +.Loop_xts_enc_steal: + lbzu 0,1(11) + stb 0,16(11) + bdnz .Loop_xts_enc_steal + + mtctr 9 + b .Loop_xts_enc + +.Lxts_enc_done: + cmpldi 8,0 + beq .Lxts_enc_ret + + vsrab 11,8,9 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + vand 11,11,10 + vxor 8,8,11 + + vperm 8,8,8,6 + .long 0x7D004799 + +.Lxts_enc_ret: + or 12,12,12 + li 3,0 + blr +.long 0 +.byte 0,12,0x04,0,0x80,6,6,0 +.long 0 +.size crypton_aes_p8_xts_encrypt,.-crypton_aes_p8_xts_encrypt + +.globl crypton_aes_p8_xts_decrypt +.type crypton_aes_p8_xts_decrypt,@function +.align 5 +crypton_aes_p8_xts_decrypt: +.localentry crypton_aes_p8_xts_decrypt,0 + + mr 10,3 + li 3,-1 + cmpldi 5,16 + .long 0x4dc00020 + + lis 0,0xfff8 + li 12,-1 + li 11,0 + or 0,0,0 + + andi. 0,5,15 + neg 0,0 + andi. 0,0,16 + sub 5,5,0 + + vspltisb 9,0x07 + lvsl 6,11,11 + vspltisb 11,0x0f + vxor 6,6,9 + + li 3,15 + lvx 8,0,8 + lvsl 5,0,8 + lvx 4,3,8 + vxor 5,5,11 + vperm 8,8,4,5 + + neg 11,10 + lvsr 5,0,11 + lvx 2,0,10 + addi 10,10,15 + vxor 5,5,11 + + cmpldi 7,0 + beq .Lxts_dec_no_key2 + + lvsr 7,0,7 + lwz 9,240(7) + srwi 9,9,1 + subi 9,9,1 + li 3,16 + + lvx 0,0,7 + lvx 1,3,7 + addi 3,3,16 + vperm 0,1,0,7 + vxor 8,8,0 + lvx 0,3,7 + addi 3,3,16 + mtctr 9 + +.Ltweak_xts_dec: + vperm 1,0,1,7 + .long 0x11080D08 + lvx 1,3,7 + addi 3,3,16 + vperm 0,1,0,7 + .long 0x11080508 + lvx 0,3,7 + addi 3,3,16 + bdnz .Ltweak_xts_dec + + vperm 1,0,1,7 + .long 0x11080D08 + lvx 1,3,7 + vperm 0,1,0,7 + .long 0x11080509 + + li 8,0 + b .Lxts_dec + +.Lxts_dec_no_key2: + neg 3,5 + andi. 3,3,15 + add 5,5,3 + + +.Lxts_dec: + lvx 4,0,10 + addi 10,10,16 + + lvsr 7,0,6 + lwz 9,240(6) + srwi 9,9,1 + subi 9,9,1 + li 3,16 + + vslb 10,9,9 + vor 10,10,9 + vspltisb 11,1 + vsldoi 10,10,11,15 + + cmpldi 5,96 + bge _aesp8_xts_decrypt6x + + lvx 0,0,6 + lvx 1,3,6 + addi 3,3,16 + vperm 2,2,4,5 + vperm 0,1,0,7 + vxor 2,2,8 + vxor 2,2,0 + lvx 0,3,6 + addi 3,3,16 + mtctr 9 + + cmpldi 5,16 + blt .Ltail_xts_dec + + +.align 5 +.Loop_xts_dec: + vperm 1,0,1,7 + .long 0x10420D48 + lvx 1,3,6 + addi 3,3,16 + vperm 0,1,0,7 + .long 0x10420548 + lvx 0,3,6 + addi 3,3,16 + bdnz .Loop_xts_dec + + vperm 1,0,1,7 + .long 0x10420D48 + lvx 1,3,6 + li 3,16 + vperm 0,1,0,7 + vxor 0,0,8 + .long 0x10620549 + + vperm 11,3,3,6 + + .long 0x7D602799 + + addi 4,4,16 + + subic. 5,5,16 + beq .Lxts_dec_done + + vor 2,4,4 + lvx 4,0,10 + addi 10,10,16 + lvx 0,0,6 + lvx 1,3,6 + addi 3,3,16 + + vsrab 11,8,9 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + vand 11,11,10 + vxor 8,8,11 + + vperm 2,2,4,5 + vperm 0,1,0,7 + vxor 2,2,8 + vxor 2,2,0 + lvx 0,3,6 + addi 3,3,16 + + mtctr 9 + cmpldi 5,16 + bge .Loop_xts_dec + +.Ltail_xts_dec: + vsrab 11,8,9 + vaddubm 12,8,8 + vsldoi 11,11,11,15 + vand 11,11,10 + vxor 12,12,11 + + subi 10,10,16 + add 10,10,5 + + vxor 2,2,8 + vxor 2,2,12 + +.Loop_xts_dec_short: + vperm 1,0,1,7 + .long 0x10420D48 + lvx 1,3,6 + addi 3,3,16 + vperm 0,1,0,7 + .long 0x10420548 + lvx 0,3,6 + addi 3,3,16 + bdnz .Loop_xts_dec_short + + vperm 1,0,1,7 + .long 0x10420D48 + lvx 1,3,6 + li 3,16 + vperm 0,1,0,7 + vxor 0,0,12 + .long 0x10620549 + + vperm 11,3,3,6 + + .long 0x7D602799 + + + vor 2,4,4 + lvx 4,0,10 + + lvx 0,0,6 + lvx 1,3,6 + addi 3,3,16 + vperm 2,2,4,5 + vperm 0,1,0,7 + + lvsr 5,0,5 + vxor 4,4,4 + vspltisb 11,-1 + vperm 4,4,11,5 + vsel 2,2,3,4 + + vxor 0,0,8 + vxor 2,2,0 + lvx 0,3,6 + addi 3,3,16 + + subi 11,4,1 + mtctr 5 + li 5,16 +.Loop_xts_dec_steal: + lbzu 0,1(11) + stb 0,16(11) + bdnz .Loop_xts_dec_steal + + mtctr 9 + b .Loop_xts_dec + +.Lxts_dec_done: + cmpldi 8,0 + beq .Lxts_dec_ret + + vsrab 11,8,9 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + vand 11,11,10 + vxor 8,8,11 + + vperm 8,8,8,6 + .long 0x7D004799 + +.Lxts_dec_ret: + or 12,12,12 + li 3,0 + blr +.long 0 +.byte 0,12,0x04,0,0x80,6,6,0 +.long 0 +.size crypton_aes_p8_xts_decrypt,.-crypton_aes_p8_xts_decrypt +.align 5 +_aesp8_xts_encrypt6x: + stdu 1,-448(1) + mflr 11 + li 7,207 + li 3,223 + std 11,464(1) + stvx 20,7,1 + addi 7,7,32 + stvx 21,3,1 + addi 3,3,32 + stvx 22,7,1 + addi 7,7,32 + stvx 23,3,1 + addi 3,3,32 + stvx 24,7,1 + addi 7,7,32 + stvx 25,3,1 + addi 3,3,32 + stvx 26,7,1 + addi 7,7,32 + stvx 27,3,1 + addi 3,3,32 + stvx 28,7,1 + addi 7,7,32 + stvx 29,3,1 + addi 3,3,32 + stvx 30,7,1 + stvx 31,3,1 + li 0,-1 + stw 12,396(1) + li 3,0x10 + std 26,400(1) + li 26,0x20 + std 27,408(1) + li 27,0x30 + std 28,416(1) + li 28,0x40 + std 29,424(1) + li 29,0x50 + std 30,432(1) + li 30,0x60 + std 31,440(1) + li 31,0x70 + or 0,0,0 + + subi 9,9,3 + + lvx 23,0,6 + lvx 30,3,6 + addi 6,6,0x20 + lvx 31,0,6 + vperm 23,30,23,7 + addi 7,1,64+15 + mtctr 9 + +.Load_xts_enc_key: + vperm 24,31,30,7 + lvx 30,3,6 + addi 6,6,0x20 + stvx 24,0,7 + vperm 25,30,31,7 + lvx 31,0,6 + stvx 25,3,7 + addi 7,7,0x20 + bdnz .Load_xts_enc_key + + lvx 26,3,6 + vperm 24,31,30,7 + lvx 27,26,6 + stvx 24,0,7 + vperm 25,26,31,7 + lvx 28,27,6 + stvx 25,3,7 + addi 7,1,64+15 + vperm 26,27,26,7 + lvx 29,28,6 + vperm 27,28,27,7 + lvx 30,29,6 + vperm 28,29,28,7 + lvx 31,30,6 + vperm 29,30,29,7 + lvx 22,31,6 + vperm 30,31,30,7 + lvx 24,0,7 + vperm 31,22,31,7 + lvx 25,3,7 + + vperm 0,2,4,5 + subi 10,10,31 + vxor 17,8,23 + vsrab 11,8,9 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + vand 11,11,10 + vxor 7,0,17 + vxor 8,8,11 + + .long 0x7C235699 + vxor 18,8,23 + vsrab 11,8,9 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + vperm 1,1,1,6 + vand 11,11,10 + vxor 12,1,18 + vxor 8,8,11 + + .long 0x7C5A5699 + andi. 31,5,15 + vxor 19,8,23 + vsrab 11,8,9 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + vperm 2,2,2,6 + vand 11,11,10 + vxor 13,2,19 + vxor 8,8,11 + + .long 0x7C7B5699 + sub 5,5,31 + vxor 20,8,23 + vsrab 11,8,9 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + vperm 3,3,3,6 + vand 11,11,10 + vxor 14,3,20 + vxor 8,8,11 + + .long 0x7C9C5699 + subi 5,5,0x60 + vxor 21,8,23 + vsrab 11,8,9 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + vperm 4,4,4,6 + vand 11,11,10 + vxor 15,4,21 + vxor 8,8,11 + + .long 0x7CBD5699 + addi 10,10,0x60 + vxor 22,8,23 + vsrab 11,8,9 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + vperm 5,5,5,6 + vand 11,11,10 + vxor 16,5,22 + vxor 8,8,11 + + vxor 31,31,23 + mtctr 9 + b .Loop_xts_enc6x + +.align 5 +.Loop_xts_enc6x: + .long 0x10E7C508 + .long 0x118CC508 + .long 0x11ADC508 + .long 0x11CEC508 + .long 0x11EFC508 + .long 0x1210C508 + lvx 24,26,7 + addi 7,7,0x20 + + .long 0x10E7CD08 + .long 0x118CCD08 + .long 0x11ADCD08 + .long 0x11CECD08 + .long 0x11EFCD08 + .long 0x1210CD08 + lvx 25,3,7 + bdnz .Loop_xts_enc6x + + subic 5,5,96 + vxor 0,17,31 + .long 0x10E7C508 + .long 0x118CC508 + vsrab 11,8,9 + vxor 17,8,23 + vaddubm 8,8,8 + .long 0x11ADC508 + .long 0x11CEC508 + vsldoi 11,11,11,15 + .long 0x11EFC508 + .long 0x1210C508 + + subfe. 0,0,0 + vand 11,11,10 + .long 0x10E7CD08 + .long 0x118CCD08 + vxor 8,8,11 + .long 0x11ADCD08 + .long 0x11CECD08 + vxor 1,18,31 + vsrab 11,8,9 + vxor 18,8,23 + .long 0x11EFCD08 + .long 0x1210CD08 + + and 0,0,5 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + .long 0x10E7D508 + .long 0x118CD508 + vand 11,11,10 + .long 0x11ADD508 + .long 0x11CED508 + vxor 8,8,11 + .long 0x11EFD508 + .long 0x1210D508 + + add 10,10,0 + + + + vxor 2,19,31 + vsrab 11,8,9 + vxor 19,8,23 + vaddubm 8,8,8 + .long 0x10E7DD08 + .long 0x118CDD08 + vsldoi 11,11,11,15 + .long 0x11ADDD08 + .long 0x11CEDD08 + vand 11,11,10 + .long 0x11EFDD08 + .long 0x1210DD08 + + addi 7,1,64+15 + vxor 8,8,11 + .long 0x10E7E508 + .long 0x118CE508 + vxor 3,20,31 + vsrab 11,8,9 + vxor 20,8,23 + .long 0x11ADE508 + .long 0x11CEE508 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + .long 0x11EFE508 + .long 0x1210E508 + lvx 24,0,7 + vand 11,11,10 + + .long 0x10E7ED08 + .long 0x118CED08 + vxor 8,8,11 + .long 0x11ADED08 + .long 0x11CEED08 + vxor 4,21,31 + vsrab 11,8,9 + vxor 21,8,23 + .long 0x11EFED08 + .long 0x1210ED08 + lvx 25,3,7 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + + .long 0x10E7F508 + .long 0x118CF508 + vand 11,11,10 + .long 0x11ADF508 + .long 0x11CEF508 + vxor 8,8,11 + .long 0x11EFF508 + .long 0x1210F508 + vxor 5,22,31 + vsrab 11,8,9 + vxor 22,8,23 + + .long 0x10E70509 + .long 0x7C005699 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + .long 0x118C0D09 + .long 0x7C235699 + .long 0x11AD1509 + vperm 0,0,0,6 + .long 0x7C5A5699 + vand 11,11,10 + .long 0x11CE1D09 + vperm 1,1,1,6 + .long 0x7C7B5699 + .long 0x11EF2509 + vperm 2,2,2,6 + .long 0x7C9C5699 + vxor 8,8,11 + .long 0x11702D09 + + vperm 3,3,3,6 + .long 0x7CBD5699 + addi 10,10,0x60 + vperm 4,4,4,6 + vperm 5,5,5,6 + + vperm 7,7,7,6 + vperm 12,12,12,6 + .long 0x7CE02799 + vxor 7,0,17 + vperm 13,13,13,6 + .long 0x7D832799 + vxor 12,1,18 + vperm 14,14,14,6 + .long 0x7DBA2799 + vxor 13,2,19 + vperm 15,15,15,6 + .long 0x7DDB2799 + vxor 14,3,20 + vperm 16,11,11,6 + .long 0x7DFC2799 + vxor 15,4,21 + .long 0x7E1D2799 + + vxor 16,5,22 + addi 4,4,0x60 + + mtctr 9 + beq .Loop_xts_enc6x + + addic. 5,5,0x60 + beq .Lxts_enc6x_zero + cmpwi 5,0x20 + blt .Lxts_enc6x_one + nop + beq .Lxts_enc6x_two + cmpwi 5,0x40 + blt .Lxts_enc6x_three + nop + beq .Lxts_enc6x_four + +.Lxts_enc6x_five: + vxor 7,1,17 + vxor 12,2,18 + vxor 13,3,19 + vxor 14,4,20 + vxor 15,5,21 + + bl _aesp8_xts_enc5x + + vperm 7,7,7,6 + vor 17,22,22 + vperm 12,12,12,6 + .long 0x7CE02799 + vperm 13,13,13,6 + .long 0x7D832799 + vperm 14,14,14,6 + .long 0x7DBA2799 + vxor 11,15,22 + vperm 15,15,15,6 + .long 0x7DDB2799 + .long 0x7DFC2799 + addi 4,4,0x50 + bne .Lxts_enc6x_steal + b .Lxts_enc6x_done + +.align 4 +.Lxts_enc6x_four: + vxor 7,2,17 + vxor 12,3,18 + vxor 13,4,19 + vxor 14,5,20 + vxor 15,15,15 + + bl _aesp8_xts_enc5x + + vperm 7,7,7,6 + vor 17,21,21 + vperm 12,12,12,6 + .long 0x7CE02799 + vperm 13,13,13,6 + .long 0x7D832799 + vxor 11,14,21 + vperm 14,14,14,6 + .long 0x7DBA2799 + .long 0x7DDB2799 + addi 4,4,0x40 + bne .Lxts_enc6x_steal + b .Lxts_enc6x_done + +.align 4 +.Lxts_enc6x_three: + vxor 7,3,17 + vxor 12,4,18 + vxor 13,5,19 + vxor 14,14,14 + vxor 15,15,15 + + bl _aesp8_xts_enc5x + + vperm 7,7,7,6 + vor 17,20,20 + vperm 12,12,12,6 + .long 0x7CE02799 + vxor 11,13,20 + vperm 13,13,13,6 + .long 0x7D832799 + .long 0x7DBA2799 + addi 4,4,0x30 + bne .Lxts_enc6x_steal + b .Lxts_enc6x_done + +.align 4 +.Lxts_enc6x_two: + vxor 7,4,17 + vxor 12,5,18 + vxor 13,13,13 + vxor 14,14,14 + vxor 15,15,15 + + bl _aesp8_xts_enc5x + + vperm 7,7,7,6 + vor 17,19,19 + vxor 11,12,19 + vperm 12,12,12,6 + .long 0x7CE02799 + .long 0x7D832799 + addi 4,4,0x20 + bne .Lxts_enc6x_steal + b .Lxts_enc6x_done + +.align 4 +.Lxts_enc6x_one: + vxor 7,5,17 + nop +.Loop_xts_enc1x: + .long 0x10E7C508 + lvx 24,26,7 + addi 7,7,0x20 + + .long 0x10E7CD08 + lvx 25,3,7 + bdnz .Loop_xts_enc1x + + add 10,10,31 + cmpwi 31,0 + .long 0x10E7C508 + + subi 10,10,16 + .long 0x10E7CD08 + + lvsr 5,0,31 + .long 0x10E7D508 + + .long 0x7C005699 + .long 0x10E7DD08 + + addi 7,1,64+15 + .long 0x10E7E508 + lvx 24,0,7 + + .long 0x10E7ED08 + lvx 25,3,7 + vxor 17,17,31 + + vperm 0,0,0,6 + .long 0x10E7F508 + + vperm 0,0,0,5 + .long 0x10E78D09 + + vor 17,18,18 + vxor 11,7,18 + vperm 7,7,7,6 + .long 0x7CE02799 + addi 4,4,0x10 + bne .Lxts_enc6x_steal + b .Lxts_enc6x_done + +.align 4 +.Lxts_enc6x_zero: + cmpwi 31,0 + beq .Lxts_enc6x_done + + add 10,10,31 + subi 10,10,16 + .long 0x7C005699 + lvsr 5,0,31 + vperm 0,0,0,6 + vperm 0,0,0,5 + vxor 11,11,17 +.Lxts_enc6x_steal: + vxor 0,0,17 + vxor 7,7,7 + vspltisb 12,-1 + vperm 7,7,12,5 + vsel 7,0,11,7 + + subi 30,4,17 + subi 4,4,16 + mtctr 31 +.Loop_xts_enc6x_steal: + lbzu 0,1(30) + stb 0,16(30) + bdnz .Loop_xts_enc6x_steal + + li 31,0 + mtctr 9 + b .Loop_xts_enc1x + +.align 4 +.Lxts_enc6x_done: + cmpldi 8,0 + beq .Lxts_enc6x_ret + + vxor 8,17,23 + vperm 8,8,8,6 + .long 0x7D004799 + +.Lxts_enc6x_ret: + mtlr 11 + li 10,79 + li 11,95 + stvx 9,10,1 + addi 10,10,32 + stvx 9,11,1 + addi 11,11,32 + stvx 9,10,1 + addi 10,10,32 + stvx 9,11,1 + addi 11,11,32 + stvx 9,10,1 + addi 10,10,32 + stvx 9,11,1 + addi 11,11,32 + stvx 9,10,1 + addi 10,10,32 + stvx 9,11,1 + addi 11,11,32 + + or 12,12,12 + lvx 20,10,1 + addi 10,10,32 + lvx 21,11,1 + addi 11,11,32 + lvx 22,10,1 + addi 10,10,32 + lvx 23,11,1 + addi 11,11,32 + lvx 24,10,1 + addi 10,10,32 + lvx 25,11,1 + addi 11,11,32 + lvx 26,10,1 + addi 10,10,32 + lvx 27,11,1 + addi 11,11,32 + lvx 28,10,1 + addi 10,10,32 + lvx 29,11,1 + addi 11,11,32 + lvx 30,10,1 + lvx 31,11,1 + ld 26,400(1) + ld 27,408(1) + ld 28,416(1) + ld 29,424(1) + ld 30,432(1) + ld 31,440(1) + addi 1,1,448 + blr +.long 0 +.byte 0,12,0x04,1,0x80,6,6,0 +.long 0 + +.align 5 +_aesp8_xts_enc5x: + .long 0x10E7C508 + .long 0x118CC508 + .long 0x11ADC508 + .long 0x11CEC508 + .long 0x11EFC508 + lvx 24,26,7 + addi 7,7,0x20 + + .long 0x10E7CD08 + .long 0x118CCD08 + .long 0x11ADCD08 + .long 0x11CECD08 + .long 0x11EFCD08 + lvx 25,3,7 + bdnz _aesp8_xts_enc5x + + add 10,10,31 + cmpwi 31,0 + .long 0x10E7C508 + .long 0x118CC508 + .long 0x11ADC508 + .long 0x11CEC508 + .long 0x11EFC508 + + subi 10,10,16 + .long 0x10E7CD08 + .long 0x118CCD08 + .long 0x11ADCD08 + .long 0x11CECD08 + .long 0x11EFCD08 + vxor 17,17,31 + + .long 0x10E7D508 + lvsr 5,0,31 + .long 0x118CD508 + .long 0x11ADD508 + .long 0x11CED508 + .long 0x11EFD508 + vxor 1,18,31 + + .long 0x10E7DD08 + .long 0x7C005699 + .long 0x118CDD08 + .long 0x11ADDD08 + .long 0x11CEDD08 + .long 0x11EFDD08 + vxor 2,19,31 + + addi 7,1,64+15 + .long 0x10E7E508 + .long 0x118CE508 + .long 0x11ADE508 + .long 0x11CEE508 + .long 0x11EFE508 + lvx 24,0,7 + vxor 3,20,31 + + .long 0x10E7ED08 + vperm 0,0,0,6 + .long 0x118CED08 + .long 0x11ADED08 + .long 0x11CEED08 + .long 0x11EFED08 + lvx 25,3,7 + vxor 4,21,31 + + .long 0x10E7F508 + vperm 0,0,0,5 + .long 0x118CF508 + .long 0x11ADF508 + .long 0x11CEF508 + .long 0x11EFF508 + + .long 0x10E78D09 + .long 0x118C0D09 + .long 0x11AD1509 + .long 0x11CE1D09 + .long 0x11EF2509 + blr +.long 0 +.byte 0,12,0x14,0,0,0,0,0 + +.align 5 +_aesp8_xts_decrypt6x: + stdu 1,-448(1) + mflr 11 + li 7,207 + li 3,223 + std 11,464(1) + stvx 20,7,1 + addi 7,7,32 + stvx 21,3,1 + addi 3,3,32 + stvx 22,7,1 + addi 7,7,32 + stvx 23,3,1 + addi 3,3,32 + stvx 24,7,1 + addi 7,7,32 + stvx 25,3,1 + addi 3,3,32 + stvx 26,7,1 + addi 7,7,32 + stvx 27,3,1 + addi 3,3,32 + stvx 28,7,1 + addi 7,7,32 + stvx 29,3,1 + addi 3,3,32 + stvx 30,7,1 + stvx 31,3,1 + li 0,-1 + stw 12,396(1) + li 3,0x10 + std 26,400(1) + li 26,0x20 + std 27,408(1) + li 27,0x30 + std 28,416(1) + li 28,0x40 + std 29,424(1) + li 29,0x50 + std 30,432(1) + li 30,0x60 + std 31,440(1) + li 31,0x70 + or 0,0,0 + + subi 9,9,3 + + lvx 23,0,6 + lvx 30,3,6 + addi 6,6,0x20 + lvx 31,0,6 + vperm 23,30,23,7 + addi 7,1,64+15 + mtctr 9 + +.Load_xts_dec_key: + vperm 24,31,30,7 + lvx 30,3,6 + addi 6,6,0x20 + stvx 24,0,7 + vperm 25,30,31,7 + lvx 31,0,6 + stvx 25,3,7 + addi 7,7,0x20 + bdnz .Load_xts_dec_key + + lvx 26,3,6 + vperm 24,31,30,7 + lvx 27,26,6 + stvx 24,0,7 + vperm 25,26,31,7 + lvx 28,27,6 + stvx 25,3,7 + addi 7,1,64+15 + vperm 26,27,26,7 + lvx 29,28,6 + vperm 27,28,27,7 + lvx 30,29,6 + vperm 28,29,28,7 + lvx 31,30,6 + vperm 29,30,29,7 + lvx 22,31,6 + vperm 30,31,30,7 + lvx 24,0,7 + vperm 31,22,31,7 + lvx 25,3,7 + + vperm 0,2,4,5 + subi 10,10,31 + vxor 17,8,23 + vsrab 11,8,9 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + vand 11,11,10 + vxor 7,0,17 + vxor 8,8,11 + + .long 0x7C235699 + vxor 18,8,23 + vsrab 11,8,9 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + vperm 1,1,1,6 + vand 11,11,10 + vxor 12,1,18 + vxor 8,8,11 + + .long 0x7C5A5699 + andi. 31,5,15 + vxor 19,8,23 + vsrab 11,8,9 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + vperm 2,2,2,6 + vand 11,11,10 + vxor 13,2,19 + vxor 8,8,11 + + .long 0x7C7B5699 + sub 5,5,31 + vxor 20,8,23 + vsrab 11,8,9 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + vperm 3,3,3,6 + vand 11,11,10 + vxor 14,3,20 + vxor 8,8,11 + + .long 0x7C9C5699 + subi 5,5,0x60 + vxor 21,8,23 + vsrab 11,8,9 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + vperm 4,4,4,6 + vand 11,11,10 + vxor 15,4,21 + vxor 8,8,11 + + .long 0x7CBD5699 + addi 10,10,0x60 + vxor 22,8,23 + vsrab 11,8,9 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + vperm 5,5,5,6 + vand 11,11,10 + vxor 16,5,22 + vxor 8,8,11 + + vxor 31,31,23 + mtctr 9 + b .Loop_xts_dec6x + +.align 5 +.Loop_xts_dec6x: + .long 0x10E7C548 + .long 0x118CC548 + .long 0x11ADC548 + .long 0x11CEC548 + .long 0x11EFC548 + .long 0x1210C548 + lvx 24,26,7 + addi 7,7,0x20 + + .long 0x10E7CD48 + .long 0x118CCD48 + .long 0x11ADCD48 + .long 0x11CECD48 + .long 0x11EFCD48 + .long 0x1210CD48 + lvx 25,3,7 + bdnz .Loop_xts_dec6x + + subic 5,5,96 + vxor 0,17,31 + .long 0x10E7C548 + .long 0x118CC548 + vsrab 11,8,9 + vxor 17,8,23 + vaddubm 8,8,8 + .long 0x11ADC548 + .long 0x11CEC548 + vsldoi 11,11,11,15 + .long 0x11EFC548 + .long 0x1210C548 + + subfe. 0,0,0 + vand 11,11,10 + .long 0x10E7CD48 + .long 0x118CCD48 + vxor 8,8,11 + .long 0x11ADCD48 + .long 0x11CECD48 + vxor 1,18,31 + vsrab 11,8,9 + vxor 18,8,23 + .long 0x11EFCD48 + .long 0x1210CD48 + + and 0,0,5 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + .long 0x10E7D548 + .long 0x118CD548 + vand 11,11,10 + .long 0x11ADD548 + .long 0x11CED548 + vxor 8,8,11 + .long 0x11EFD548 + .long 0x1210D548 + + add 10,10,0 + + + + vxor 2,19,31 + vsrab 11,8,9 + vxor 19,8,23 + vaddubm 8,8,8 + .long 0x10E7DD48 + .long 0x118CDD48 + vsldoi 11,11,11,15 + .long 0x11ADDD48 + .long 0x11CEDD48 + vand 11,11,10 + .long 0x11EFDD48 + .long 0x1210DD48 + + addi 7,1,64+15 + vxor 8,8,11 + .long 0x10E7E548 + .long 0x118CE548 + vxor 3,20,31 + vsrab 11,8,9 + vxor 20,8,23 + .long 0x11ADE548 + .long 0x11CEE548 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + .long 0x11EFE548 + .long 0x1210E548 + lvx 24,0,7 + vand 11,11,10 + + .long 0x10E7ED48 + .long 0x118CED48 + vxor 8,8,11 + .long 0x11ADED48 + .long 0x11CEED48 + vxor 4,21,31 + vsrab 11,8,9 + vxor 21,8,23 + .long 0x11EFED48 + .long 0x1210ED48 + lvx 25,3,7 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + + .long 0x10E7F548 + .long 0x118CF548 + vand 11,11,10 + .long 0x11ADF548 + .long 0x11CEF548 + vxor 8,8,11 + .long 0x11EFF548 + .long 0x1210F548 + vxor 5,22,31 + vsrab 11,8,9 + vxor 22,8,23 + + .long 0x10E70549 + .long 0x7C005699 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + .long 0x118C0D49 + .long 0x7C235699 + .long 0x11AD1549 + vperm 0,0,0,6 + .long 0x7C5A5699 + vand 11,11,10 + .long 0x11CE1D49 + vperm 1,1,1,6 + .long 0x7C7B5699 + .long 0x11EF2549 + vperm 2,2,2,6 + .long 0x7C9C5699 + vxor 8,8,11 + .long 0x12102D49 + vperm 3,3,3,6 + .long 0x7CBD5699 + addi 10,10,0x60 + vperm 4,4,4,6 + vperm 5,5,5,6 + + vperm 7,7,7,6 + vperm 12,12,12,6 + .long 0x7CE02799 + vxor 7,0,17 + vperm 13,13,13,6 + .long 0x7D832799 + vxor 12,1,18 + vperm 14,14,14,6 + .long 0x7DBA2799 + vxor 13,2,19 + vperm 15,15,15,6 + .long 0x7DDB2799 + vxor 14,3,20 + vperm 16,16,16,6 + .long 0x7DFC2799 + vxor 15,4,21 + .long 0x7E1D2799 + vxor 16,5,22 + addi 4,4,0x60 + + mtctr 9 + beq .Loop_xts_dec6x + + addic. 5,5,0x60 + beq .Lxts_dec6x_zero + cmpwi 5,0x20 + blt .Lxts_dec6x_one + nop + beq .Lxts_dec6x_two + cmpwi 5,0x40 + blt .Lxts_dec6x_three + nop + beq .Lxts_dec6x_four + +.Lxts_dec6x_five: + vxor 7,1,17 + vxor 12,2,18 + vxor 13,3,19 + vxor 14,4,20 + vxor 15,5,21 + + bl _aesp8_xts_dec5x + + vperm 7,7,7,6 + vor 17,22,22 + vxor 18,8,23 + vperm 12,12,12,6 + .long 0x7CE02799 + vxor 7,0,18 + vperm 13,13,13,6 + .long 0x7D832799 + vperm 14,14,14,6 + .long 0x7DBA2799 + vperm 15,15,15,6 + .long 0x7DDB2799 + .long 0x7DFC2799 + addi 4,4,0x50 + bne .Lxts_dec6x_steal + b .Lxts_dec6x_done + +.align 4 +.Lxts_dec6x_four: + vxor 7,2,17 + vxor 12,3,18 + vxor 13,4,19 + vxor 14,5,20 + vxor 15,15,15 + + bl _aesp8_xts_dec5x + + vperm 7,7,7,6 + vor 17,21,21 + vor 18,22,22 + vperm 12,12,12,6 + .long 0x7CE02799 + vxor 7,0,22 + vperm 13,13,13,6 + .long 0x7D832799 + vperm 14,14,14,6 + .long 0x7DBA2799 + .long 0x7DDB2799 + addi 4,4,0x40 + bne .Lxts_dec6x_steal + b .Lxts_dec6x_done + +.align 4 +.Lxts_dec6x_three: + vxor 7,3,17 + vxor 12,4,18 + vxor 13,5,19 + vxor 14,14,14 + vxor 15,15,15 + + bl _aesp8_xts_dec5x + + vperm 7,7,7,6 + vor 17,20,20 + vor 18,21,21 + vperm 12,12,12,6 + .long 0x7CE02799 + vxor 7,0,21 + vperm 13,13,13,6 + .long 0x7D832799 + .long 0x7DBA2799 + addi 4,4,0x30 + bne .Lxts_dec6x_steal + b .Lxts_dec6x_done + +.align 4 +.Lxts_dec6x_two: + vxor 7,4,17 + vxor 12,5,18 + vxor 13,13,13 + vxor 14,14,14 + vxor 15,15,15 + + bl _aesp8_xts_dec5x + + vperm 7,7,7,6 + vor 17,19,19 + vor 18,20,20 + vperm 12,12,12,6 + .long 0x7CE02799 + vxor 7,0,20 + .long 0x7D832799 + addi 4,4,0x20 + bne .Lxts_dec6x_steal + b .Lxts_dec6x_done + +.align 4 +.Lxts_dec6x_one: + vxor 7,5,17 + nop +.Loop_xts_dec1x: + .long 0x10E7C548 + lvx 24,26,7 + addi 7,7,0x20 + + .long 0x10E7CD48 + lvx 25,3,7 + bdnz .Loop_xts_dec1x + + subi 0,31,1 + .long 0x10E7C548 + + andi. 0,0,16 + cmpwi 31,0 + .long 0x10E7CD48 + + sub 10,10,0 + .long 0x10E7D548 + + .long 0x7C005699 + .long 0x10E7DD48 + + addi 7,1,64+15 + .long 0x10E7E548 + lvx 24,0,7 + + .long 0x10E7ED48 + lvx 25,3,7 + vxor 17,17,31 + + vperm 0,0,0,6 + .long 0x10E7F548 + + mtctr 9 + .long 0x10E78D49 + + vor 17,18,18 + vor 18,19,19 + vperm 7,7,7,6 + .long 0x7CE02799 + addi 4,4,0x10 + vxor 7,0,19 + bne .Lxts_dec6x_steal + b .Lxts_dec6x_done + +.align 4 +.Lxts_dec6x_zero: + cmpwi 31,0 + beq .Lxts_dec6x_done + + .long 0x7C005699 + vperm 0,0,0,6 + vxor 7,0,18 +.Lxts_dec6x_steal: + .long 0x10E7C548 + lvx 24,26,7 + addi 7,7,0x20 + + .long 0x10E7CD48 + lvx 25,3,7 + bdnz .Lxts_dec6x_steal + + add 10,10,31 + .long 0x10E7C548 + + cmpwi 31,0 + .long 0x10E7CD48 + + .long 0x7C005699 + .long 0x10E7D548 + + lvsr 5,0,31 + .long 0x10E7DD48 + + addi 7,1,64+15 + .long 0x10E7E548 + lvx 24,0,7 + + .long 0x10E7ED48 + lvx 25,3,7 + vxor 18,18,31 + + vperm 0,0,0,6 + .long 0x10E7F548 + + vperm 0,0,0,5 + .long 0x11679549 + + vperm 7,11,11,6 + .long 0x7CE02799 + + + vxor 7,7,7 + vspltisb 12,-1 + vperm 7,7,12,5 + vsel 7,0,11,7 + vxor 7,7,17 + + subi 30,4,1 + mtctr 31 +.Loop_xts_dec6x_steal: + lbzu 0,1(30) + stb 0,16(30) + bdnz .Loop_xts_dec6x_steal + + li 31,0 + mtctr 9 + b .Loop_xts_dec1x + +.align 4 +.Lxts_dec6x_done: + cmpldi 8,0 + beq .Lxts_dec6x_ret + + vxor 8,17,23 + vperm 8,8,8,6 + .long 0x7D004799 + +.Lxts_dec6x_ret: + mtlr 11 + li 10,79 + li 11,95 + stvx 9,10,1 + addi 10,10,32 + stvx 9,11,1 + addi 11,11,32 + stvx 9,10,1 + addi 10,10,32 + stvx 9,11,1 + addi 11,11,32 + stvx 9,10,1 + addi 10,10,32 + stvx 9,11,1 + addi 11,11,32 + stvx 9,10,1 + addi 10,10,32 + stvx 9,11,1 + addi 11,11,32 + + or 12,12,12 + lvx 20,10,1 + addi 10,10,32 + lvx 21,11,1 + addi 11,11,32 + lvx 22,10,1 + addi 10,10,32 + lvx 23,11,1 + addi 11,11,32 + lvx 24,10,1 + addi 10,10,32 + lvx 25,11,1 + addi 11,11,32 + lvx 26,10,1 + addi 10,10,32 + lvx 27,11,1 + addi 11,11,32 + lvx 28,10,1 + addi 10,10,32 + lvx 29,11,1 + addi 11,11,32 + lvx 30,10,1 + lvx 31,11,1 + ld 26,400(1) + ld 27,408(1) + ld 28,416(1) + ld 29,424(1) + ld 30,432(1) + ld 31,440(1) + addi 1,1,448 + blr +.long 0 +.byte 0,12,0x04,1,0x80,6,6,0 +.long 0 + +.align 5 +_aesp8_xts_dec5x: + .long 0x10E7C548 + .long 0x118CC548 + .long 0x11ADC548 + .long 0x11CEC548 + .long 0x11EFC548 + lvx 24,26,7 + addi 7,7,0x20 + + .long 0x10E7CD48 + .long 0x118CCD48 + .long 0x11ADCD48 + .long 0x11CECD48 + .long 0x11EFCD48 + lvx 25,3,7 + bdnz _aesp8_xts_dec5x + + subi 0,31,1 + .long 0x10E7C548 + .long 0x118CC548 + .long 0x11ADC548 + .long 0x11CEC548 + .long 0x11EFC548 + + andi. 0,0,16 + cmpwi 31,0 + .long 0x10E7CD48 + .long 0x118CCD48 + .long 0x11ADCD48 + .long 0x11CECD48 + .long 0x11EFCD48 + vxor 17,17,31 + + sub 10,10,0 + .long 0x10E7D548 + .long 0x118CD548 + .long 0x11ADD548 + .long 0x11CED548 + .long 0x11EFD548 + vxor 1,18,31 + + .long 0x10E7DD48 + .long 0x7C005699 + .long 0x118CDD48 + .long 0x11ADDD48 + .long 0x11CEDD48 + .long 0x11EFDD48 + vxor 2,19,31 + + addi 7,1,64+15 + .long 0x10E7E548 + .long 0x118CE548 + .long 0x11ADE548 + .long 0x11CEE548 + .long 0x11EFE548 + lvx 24,0,7 + vxor 3,20,31 + + .long 0x10E7ED48 + vperm 0,0,0,6 + .long 0x118CED48 + .long 0x11ADED48 + .long 0x11CEED48 + .long 0x11EFED48 + lvx 25,3,7 + vxor 4,21,31 + + .long 0x10E7F548 + .long 0x118CF548 + .long 0x11ADF548 + .long 0x11CEF548 + .long 0x11EFF548 + + .long 0x10E78D49 + .long 0x118C0D49 + .long 0x11AD1549 + .long 0x11CE1D49 + .long 0x11EF2549 + mtctr 9 + blr +.long 0 +.byte 0,12,0x14,0,0,0,0,0 diff --git a/cbits/asm/aesp8-ppc.pl b/cbits/asm/aesp8-ppc.pl new file mode 100644 --- /dev/null +++ b/cbits/asm/aesp8-ppc.pl @@ -0,0 +1,3801 @@ +#!/usr/bin/env perl +# +# ==================================================================== +# Written by Andy Polyakov for the OpenSSL +# project. The module is, however, dual licensed under OpenSSL and +# CRYPTOGAMS licenses depending on where you obtain it. For further +# details see http://www.openssl.org/~appro/cryptogams/. +# ==================================================================== +# +# This module implements support for AES instructions as per PowerISA +# specification version 2.07, first implemented by POWER8 processor. +# The module is endian-agnostic in sense that it supports both big- +# and little-endian cases. Data alignment in parallelizable modes is +# handled with VSX loads and stores, which implies MSR.VSX flag being +# set. It should also be noted that ISA specification doesn't prohibit +# alignment exceptions for these instructions on page boundaries. +# Initially alignment was handled in pure AltiVec/VMX way [when data +# is aligned programmatically, which in turn guarantees exception- +# free execution], but it turned to hamper performance when vcipher +# instructions are interleaved. It's reckoned that eventual +# misalignment penalties at page boundaries are in average lower +# than additional overhead in pure AltiVec approach. +# +# May 2016 +# +# Add XTS subroutine, 9x on little- and 12x improvement on big-endian +# systems were measured. +# +###################################################################### +# Current large-block performance in cycles per byte processed with +# 128-bit key (less is better). +# +# CBC en-/decrypt CTR XTS +# POWER8[le] 3.96/0.72 0.74 1.1 +# POWER8[be] 3.75/0.65 0.66 1.0 +# POWER9[le] 4.02/0.86 0.84 1.05 +# POWER9[be] 3.99/0.78 0.79 0.97 + + +$flavour = shift; + +if ($flavour =~ /64/) { + $SIZE_T =8; + $LRSAVE =2*$SIZE_T; + $STU ="stdu"; + $POP ="ld"; + $PUSH ="std"; + $UCMP ="cmpld"; + $SHL ="sldi"; +} elsif ($flavour =~ /32/) { + $SIZE_T =4; + $LRSAVE =$SIZE_T; + $STU ="stwu"; + $POP ="lwz"; + $PUSH ="stw"; + $UCMP ="cmplw"; + $SHL ="slwi"; +} else { die "nonsense $flavour"; } + +$LITTLE_ENDIAN = ($flavour=~/le$/) ? $SIZE_T : 0; + +$0 =~ m/(.*[\/\\])[^\/\\]+$/; $dir=$1; +( $xlate="${dir}ppc-xlate.pl" and -f $xlate ) or +( $xlate="${dir}../../perlasm/ppc-xlate.pl" and -f $xlate) or +die "can't locate ppc-xlate.pl"; + +open STDOUT,"| $^X $xlate $flavour ".shift || die "can't call $xlate: $!"; + +$FRAME=8*$SIZE_T; +$prefix="aes_p8"; + +$sp="r1"; +$vrsave="r12"; + +######################################################################### +{{{ # Key setup procedures # +my ($inp,$bits,$out,$ptr,$cnt,$rounds)=map("r$_",(3..8)); +my ($zero,$in0,$in1,$key,$rcon,$mask,$tmp)=map("v$_",(0..6)); +my ($stage,$outperm,$outmask,$outhead,$outtail)=map("v$_",(7..11)); + +$code.=<<___; +.machine "any" + +.text + +.align 7 +rcon: +.long 0x01000000, 0x01000000, 0x01000000, 0x01000000 ?rev +.long 0x1b000000, 0x1b000000, 0x1b000000, 0x1b000000 ?rev +.long 0x0d0e0f0c, 0x0d0e0f0c, 0x0d0e0f0c, 0x0d0e0f0c ?rev +.long 0,0,0,0 ?asis +Lconsts: + mflr r0 + bcl 20,31,\$+4 + mflr $ptr #vvvvv "distance between . and rcon + addi $ptr,$ptr,-0x48 + mtlr r0 + blr + .long 0 + .byte 0,12,0x14,0,0,0,0,0 +.asciz "AES for PowerISA 2.07, CRYPTOGAMS by " + +.globl .${prefix}_set_encrypt_key +.align 5 +.${prefix}_set_encrypt_key: +Lset_encrypt_key: + mflr r11 + $PUSH r11,$LRSAVE($sp) + + li $ptr,-1 + ${UCMP}i $inp,0 + beq- Lenc_key_abort # if ($inp==0) return -1; + ${UCMP}i $out,0 + beq- Lenc_key_abort # if ($out==0) return -1; + li $ptr,-2 + cmpwi $bits,128 + blt- Lenc_key_abort + cmpwi $bits,256 + bgt- Lenc_key_abort + andi. r0,$bits,0x3f + bne- Lenc_key_abort + + lis r0,0xfff0 + mfspr $vrsave,256 + mtspr 256,r0 + + bl Lconsts + mtlr r11 + + neg r9,$inp + lvx $in0,0,$inp + addi $inp,$inp,15 # 15 is not typo + lvsr $key,0,r9 # borrow $key + li r8,0x20 + cmpwi $bits,192 + lvx $in1,0,$inp + le?vspltisb $mask,0x0f # borrow $mask + lvx $rcon,0,$ptr + le?vxor $key,$key,$mask # adjust for byte swap + lvx $mask,r8,$ptr + addi $ptr,$ptr,0x10 + vperm $in0,$in0,$in1,$key # align [and byte swap in LE] + li $cnt,8 + vxor $zero,$zero,$zero + mtctr $cnt + + ?lvsr $outperm,0,$out + vspltisb $outmask,-1 + lvx $outhead,0,$out + ?vperm $outmask,$zero,$outmask,$outperm + + blt Loop128 + addi $inp,$inp,8 + beq L192 + addi $inp,$inp,8 + b L256 + +.align 4 +Loop128: + vperm $key,$in0,$in0,$mask # rotate-n-splat + vsldoi $tmp,$zero,$in0,12 # >>32 + vperm $outtail,$in0,$in0,$outperm # rotate + vsel $stage,$outhead,$outtail,$outmask + vmr $outhead,$outtail + vcipherlast $key,$key,$rcon + stvx $stage,0,$out + addi $out,$out,16 + + vxor $in0,$in0,$tmp + vsldoi $tmp,$zero,$tmp,12 # >>32 + vxor $in0,$in0,$tmp + vsldoi $tmp,$zero,$tmp,12 # >>32 + vxor $in0,$in0,$tmp + vadduwm $rcon,$rcon,$rcon + vxor $in0,$in0,$key + bdnz Loop128 + + lvx $rcon,0,$ptr # last two round keys + + vperm $key,$in0,$in0,$mask # rotate-n-splat + vsldoi $tmp,$zero,$in0,12 # >>32 + vperm $outtail,$in0,$in0,$outperm # rotate + vsel $stage,$outhead,$outtail,$outmask + vmr $outhead,$outtail + vcipherlast $key,$key,$rcon + stvx $stage,0,$out + addi $out,$out,16 + + vxor $in0,$in0,$tmp + vsldoi $tmp,$zero,$tmp,12 # >>32 + vxor $in0,$in0,$tmp + vsldoi $tmp,$zero,$tmp,12 # >>32 + vxor $in0,$in0,$tmp + vadduwm $rcon,$rcon,$rcon + vxor $in0,$in0,$key + + vperm $key,$in0,$in0,$mask # rotate-n-splat + vsldoi $tmp,$zero,$in0,12 # >>32 + vperm $outtail,$in0,$in0,$outperm # rotate + vsel $stage,$outhead,$outtail,$outmask + vmr $outhead,$outtail + vcipherlast $key,$key,$rcon + stvx $stage,0,$out + addi $out,$out,16 + + vxor $in0,$in0,$tmp + vsldoi $tmp,$zero,$tmp,12 # >>32 + vxor $in0,$in0,$tmp + vsldoi $tmp,$zero,$tmp,12 # >>32 + vxor $in0,$in0,$tmp + vxor $in0,$in0,$key + vperm $outtail,$in0,$in0,$outperm # rotate + vsel $stage,$outhead,$outtail,$outmask + vmr $outhead,$outtail + stvx $stage,0,$out + + addi $inp,$out,15 # 15 is not typo + addi $out,$out,0x50 + + li $rounds,10 + b Ldone + +.align 4 +L192: + lvx $tmp,0,$inp + li $cnt,4 + vperm $outtail,$in0,$in0,$outperm # rotate + vsel $stage,$outhead,$outtail,$outmask + vmr $outhead,$outtail + stvx $stage,0,$out + addi $out,$out,16 + vperm $in1,$in1,$tmp,$key # align [and byte swap in LE] + vspltisb $key,8 # borrow $key + mtctr $cnt + vsububm $mask,$mask,$key # adjust the mask + +Loop192: + vperm $key,$in1,$in1,$mask # roate-n-splat + vsldoi $tmp,$zero,$in0,12 # >>32 + vcipherlast $key,$key,$rcon + + vxor $in0,$in0,$tmp + vsldoi $tmp,$zero,$tmp,12 # >>32 + vxor $in0,$in0,$tmp + vsldoi $tmp,$zero,$tmp,12 # >>32 + vxor $in0,$in0,$tmp + + vsldoi $stage,$zero,$in1,8 + vspltw $tmp,$in0,3 + vxor $tmp,$tmp,$in1 + vsldoi $in1,$zero,$in1,12 # >>32 + vadduwm $rcon,$rcon,$rcon + vxor $in1,$in1,$tmp + vxor $in0,$in0,$key + vxor $in1,$in1,$key + vsldoi $stage,$stage,$in0,8 + + vperm $key,$in1,$in1,$mask # rotate-n-splat + vsldoi $tmp,$zero,$in0,12 # >>32 + vperm $outtail,$stage,$stage,$outperm # rotate + vsel $stage,$outhead,$outtail,$outmask + vmr $outhead,$outtail + vcipherlast $key,$key,$rcon + stvx $stage,0,$out + addi $out,$out,16 + + vsldoi $stage,$in0,$in1,8 + vxor $in0,$in0,$tmp + vsldoi $tmp,$zero,$tmp,12 # >>32 + vperm $outtail,$stage,$stage,$outperm # rotate + vsel $stage,$outhead,$outtail,$outmask + vmr $outhead,$outtail + vxor $in0,$in0,$tmp + vsldoi $tmp,$zero,$tmp,12 # >>32 + vxor $in0,$in0,$tmp + stvx $stage,0,$out + addi $out,$out,16 + + vspltw $tmp,$in0,3 + vxor $tmp,$tmp,$in1 + vsldoi $in1,$zero,$in1,12 # >>32 + vadduwm $rcon,$rcon,$rcon + vxor $in1,$in1,$tmp + vxor $in0,$in0,$key + vxor $in1,$in1,$key + vperm $outtail,$in0,$in0,$outperm # rotate + vsel $stage,$outhead,$outtail,$outmask + vmr $outhead,$outtail + stvx $stage,0,$out + addi $inp,$out,15 # 15 is not typo + addi $out,$out,16 + bdnz Loop192 + + li $rounds,12 + addi $out,$out,0x20 + b Ldone + +.align 4 +L256: + lvx $tmp,0,$inp + li $cnt,7 + li $rounds,14 + vperm $outtail,$in0,$in0,$outperm # rotate + vsel $stage,$outhead,$outtail,$outmask + vmr $outhead,$outtail + stvx $stage,0,$out + addi $out,$out,16 + vperm $in1,$in1,$tmp,$key # align [and byte swap in LE] + mtctr $cnt + +Loop256: + vperm $key,$in1,$in1,$mask # rotate-n-splat + vsldoi $tmp,$zero,$in0,12 # >>32 + vperm $outtail,$in1,$in1,$outperm # rotate + vsel $stage,$outhead,$outtail,$outmask + vmr $outhead,$outtail + vcipherlast $key,$key,$rcon + stvx $stage,0,$out + addi $out,$out,16 + + vxor $in0,$in0,$tmp + vsldoi $tmp,$zero,$tmp,12 # >>32 + vxor $in0,$in0,$tmp + vsldoi $tmp,$zero,$tmp,12 # >>32 + vxor $in0,$in0,$tmp + vadduwm $rcon,$rcon,$rcon + vxor $in0,$in0,$key + vperm $outtail,$in0,$in0,$outperm # rotate + vsel $stage,$outhead,$outtail,$outmask + vmr $outhead,$outtail + stvx $stage,0,$out + addi $inp,$out,15 # 15 is not typo + addi $out,$out,16 + bdz Ldone + + vspltw $key,$in0,3 # just splat + vsldoi $tmp,$zero,$in1,12 # >>32 + vsbox $key,$key + + vxor $in1,$in1,$tmp + vsldoi $tmp,$zero,$tmp,12 # >>32 + vxor $in1,$in1,$tmp + vsldoi $tmp,$zero,$tmp,12 # >>32 + vxor $in1,$in1,$tmp + + vxor $in1,$in1,$key + b Loop256 + +.align 4 +Ldone: + lvx $in1,0,$inp # redundant in aligned case + vsel $in1,$outhead,$in1,$outmask + stvx $in1,0,$inp + li $ptr,0 + mtspr 256,$vrsave + stw $rounds,0($out) + +Lenc_key_abort: + mr r3,$ptr + blr + .long 0 + .byte 0,12,0x14,1,0,0,3,0 + .long 0 +.size .${prefix}_set_encrypt_key,.-.${prefix}_set_encrypt_key + +.globl .${prefix}_set_decrypt_key +.align 5 +.${prefix}_set_decrypt_key: + $STU $sp,-$FRAME($sp) + mflr r10 + $PUSH r10,$FRAME+$LRSAVE($sp) + bl Lset_encrypt_key + mtlr r10 + + cmpwi r3,0 + bne- Ldec_key_abort + + slwi $cnt,$rounds,4 + subi $inp,$out,240 # first round key + srwi $rounds,$rounds,1 + add $out,$inp,$cnt # last round key + mtctr $rounds + +Ldeckey: + lwz r0, 0($inp) + lwz r6, 4($inp) + lwz r7, 8($inp) + lwz r8, 12($inp) + addi $inp,$inp,16 + lwz r9, 0($out) + lwz r10,4($out) + lwz r11,8($out) + lwz r12,12($out) + stw r0, 0($out) + stw r6, 4($out) + stw r7, 8($out) + stw r8, 12($out) + subi $out,$out,16 + stw r9, -16($inp) + stw r10,-12($inp) + stw r11,-8($inp) + stw r12,-4($inp) + bdnz Ldeckey + + xor r3,r3,r3 # return value +Ldec_key_abort: + addi $sp,$sp,$FRAME + blr + .long 0 + .byte 0,12,4,1,0x80,0,3,0 + .long 0 +.size .${prefix}_set_decrypt_key,.-.${prefix}_set_decrypt_key +___ +}}} +######################################################################### +{{{ # Single block en- and decrypt procedures # +sub gen_block () { +my $dir = shift; +my $n = $dir eq "de" ? "n" : ""; +my ($inp,$out,$key,$rounds,$idx)=map("r$_",(3..7)); + +$code.=<<___; +.globl .${prefix}_${dir}crypt +.align 5 +.${prefix}_${dir}crypt: + lwz $rounds,240($key) + lis r0,0xfc00 + mfspr $vrsave,256 + li $idx,15 # 15 is not typo + mtspr 256,r0 + + lvx v0,0,$inp + neg r11,$out + lvx v1,$idx,$inp + lvsl v2,0,$inp # inpperm + le?vspltisb v4,0x0f + ?lvsl v3,0,r11 # outperm + le?vxor v2,v2,v4 + li $idx,16 + vperm v0,v0,v1,v2 # align [and byte swap in LE] + lvx v1,0,$key + ?lvsl v5,0,$key # keyperm + srwi $rounds,$rounds,1 + lvx v2,$idx,$key + addi $idx,$idx,16 + subi $rounds,$rounds,1 + ?vperm v1,v1,v2,v5 # align round key + + vxor v0,v0,v1 + lvx v1,$idx,$key + addi $idx,$idx,16 + mtctr $rounds + +Loop_${dir}c: + ?vperm v2,v2,v1,v5 + v${n}cipher v0,v0,v2 + lvx v2,$idx,$key + addi $idx,$idx,16 + ?vperm v1,v1,v2,v5 + v${n}cipher v0,v0,v1 + lvx v1,$idx,$key + addi $idx,$idx,16 + bdnz Loop_${dir}c + + ?vperm v2,v2,v1,v5 + v${n}cipher v0,v0,v2 + lvx v2,$idx,$key + ?vperm v1,v1,v2,v5 + v${n}cipherlast v0,v0,v1 + + vspltisb v2,-1 + vxor v1,v1,v1 + li $idx,15 # 15 is not typo + ?vperm v2,v1,v2,v3 # outmask + le?vxor v3,v3,v4 + lvx v1,0,$out # outhead + vperm v0,v0,v0,v3 # rotate [and byte swap in LE] + vsel v1,v1,v0,v2 + lvx v4,$idx,$out + stvx v1,0,$out + vsel v0,v0,v4,v2 + stvx v0,$idx,$out + + mtspr 256,$vrsave + blr + .long 0 + .byte 0,12,0x14,0,0,0,3,0 + .long 0 +.size .${prefix}_${dir}crypt,.-.${prefix}_${dir}crypt +___ +} +&gen_block("en"); +&gen_block("de"); +}}} +######################################################################### +{{{ # CBC en- and decrypt procedures # +my ($inp,$out,$len,$key,$ivp,$enc,$rounds,$idx)=map("r$_",(3..10)); +my ($rndkey0,$rndkey1,$inout,$tmp)= map("v$_",(0..3)); +my ($ivec,$inptail,$inpperm,$outhead,$outperm,$outmask,$keyperm)= + map("v$_",(4..10)); +$code.=<<___; +.globl .${prefix}_cbc_encrypt +.align 5 +.${prefix}_cbc_encrypt: + ${UCMP}i $len,16 + bltlr- + + cmpwi $enc,0 # test direction + lis r0,0xffe0 + mfspr $vrsave,256 + mtspr 256,r0 + + li $idx,15 + vxor $rndkey0,$rndkey0,$rndkey0 + le?vspltisb $tmp,0x0f + + lvx $ivec,0,$ivp # load [unaligned] iv + lvsl $inpperm,0,$ivp + lvx $inptail,$idx,$ivp + le?vxor $inpperm,$inpperm,$tmp + vperm $ivec,$ivec,$inptail,$inpperm + + neg r11,$inp + ?lvsl $keyperm,0,$key # prepare for unaligned key + lwz $rounds,240($key) + + lvsr $inpperm,0,r11 # prepare for unaligned load + lvx $inptail,0,$inp + addi $inp,$inp,15 # 15 is not typo + le?vxor $inpperm,$inpperm,$tmp + + ?lvsr $outperm,0,$out # prepare for unaligned store + vspltisb $outmask,-1 + lvx $outhead,0,$out + ?vperm $outmask,$rndkey0,$outmask,$outperm + le?vxor $outperm,$outperm,$tmp + + srwi $rounds,$rounds,1 + li $idx,16 + subi $rounds,$rounds,1 + beq Lcbc_dec + +Lcbc_enc: + vmr $inout,$inptail + lvx $inptail,0,$inp + addi $inp,$inp,16 + mtctr $rounds + subi $len,$len,16 # len-=16 + + lvx $rndkey0,0,$key + vperm $inout,$inout,$inptail,$inpperm + lvx $rndkey1,$idx,$key + addi $idx,$idx,16 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vxor $inout,$inout,$rndkey0 + lvx $rndkey0,$idx,$key + addi $idx,$idx,16 + vxor $inout,$inout,$ivec + +Loop_cbc_enc: + ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm + vcipher $inout,$inout,$rndkey1 + lvx $rndkey1,$idx,$key + addi $idx,$idx,16 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vcipher $inout,$inout,$rndkey0 + lvx $rndkey0,$idx,$key + addi $idx,$idx,16 + bdnz Loop_cbc_enc + + ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm + vcipher $inout,$inout,$rndkey1 + lvx $rndkey1,$idx,$key + li $idx,16 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vcipherlast $ivec,$inout,$rndkey0 + ${UCMP}i $len,16 + + vperm $tmp,$ivec,$ivec,$outperm + vsel $inout,$outhead,$tmp,$outmask + vmr $outhead,$tmp + stvx $inout,0,$out + addi $out,$out,16 + bge Lcbc_enc + + b Lcbc_done + +.align 4 +Lcbc_dec: + ${UCMP}i $len,128 + bge _aesp8_cbc_decrypt8x + vmr $tmp,$inptail + lvx $inptail,0,$inp + addi $inp,$inp,16 + mtctr $rounds + subi $len,$len,16 # len-=16 + + lvx $rndkey0,0,$key + vperm $tmp,$tmp,$inptail,$inpperm + lvx $rndkey1,$idx,$key + addi $idx,$idx,16 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vxor $inout,$tmp,$rndkey0 + lvx $rndkey0,$idx,$key + addi $idx,$idx,16 + +Loop_cbc_dec: + ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm + vncipher $inout,$inout,$rndkey1 + lvx $rndkey1,$idx,$key + addi $idx,$idx,16 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vncipher $inout,$inout,$rndkey0 + lvx $rndkey0,$idx,$key + addi $idx,$idx,16 + bdnz Loop_cbc_dec + + ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm + vncipher $inout,$inout,$rndkey1 + lvx $rndkey1,$idx,$key + li $idx,16 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vncipherlast $inout,$inout,$rndkey0 + ${UCMP}i $len,16 + + vxor $inout,$inout,$ivec + vmr $ivec,$tmp + vperm $tmp,$inout,$inout,$outperm + vsel $inout,$outhead,$tmp,$outmask + vmr $outhead,$tmp + stvx $inout,0,$out + addi $out,$out,16 + bge Lcbc_dec + +Lcbc_done: + addi $out,$out,-1 + lvx $inout,0,$out # redundant in aligned case + vsel $inout,$outhead,$inout,$outmask + stvx $inout,0,$out + + neg $enc,$ivp # write [unaligned] iv + li $idx,15 # 15 is not typo + vxor $rndkey0,$rndkey0,$rndkey0 + vspltisb $outmask,-1 + le?vspltisb $tmp,0x0f + ?lvsl $outperm,0,$enc + ?vperm $outmask,$rndkey0,$outmask,$outperm + le?vxor $outperm,$outperm,$tmp + lvx $outhead,0,$ivp + vperm $ivec,$ivec,$ivec,$outperm + vsel $inout,$outhead,$ivec,$outmask + lvx $inptail,$idx,$ivp + stvx $inout,0,$ivp + vsel $inout,$ivec,$inptail,$outmask + stvx $inout,$idx,$ivp + + mtspr 256,$vrsave + blr + .long 0 + .byte 0,12,0x14,0,0,0,6,0 + .long 0 +___ +######################################################################### +{{ # Optimized CBC decrypt procedure # +my $key_="r11"; +my ($x00,$x10,$x20,$x30,$x40,$x50,$x60,$x70)=map("r$_",(0,8,26..31)); + $x00=0 if ($flavour =~ /osx/); +my ($in0, $in1, $in2, $in3, $in4, $in5, $in6, $in7 )=map("v$_",(0..3,10..13)); +my ($out0,$out1,$out2,$out3,$out4,$out5,$out6,$out7)=map("v$_",(14..21)); +my $rndkey0="v23"; # v24-v25 rotating buffer for first found keys + # v26-v31 last 6 round keys +my ($tmp,$keyperm)=($in3,$in4); # aliases with "caller", redundant assignment + +$code.=<<___; +.align 5 +_aesp8_cbc_decrypt8x: + $STU $sp,-`($FRAME+21*16+6*$SIZE_T)`($sp) + li r10,`$FRAME+8*16+15` + li r11,`$FRAME+8*16+31` + stvx v20,r10,$sp # ABI says so + addi r10,r10,32 + stvx v21,r11,$sp + addi r11,r11,32 + stvx v22,r10,$sp + addi r10,r10,32 + stvx v23,r11,$sp + addi r11,r11,32 + stvx v24,r10,$sp + addi r10,r10,32 + stvx v25,r11,$sp + addi r11,r11,32 + stvx v26,r10,$sp + addi r10,r10,32 + stvx v27,r11,$sp + addi r11,r11,32 + stvx v28,r10,$sp + addi r10,r10,32 + stvx v29,r11,$sp + addi r11,r11,32 + stvx v30,r10,$sp + stvx v31,r11,$sp + li r0,-1 + stw $vrsave,`$FRAME+21*16-4`($sp) # save vrsave + li $x10,0x10 + $PUSH r26,`$FRAME+21*16+0*$SIZE_T`($sp) + li $x20,0x20 + $PUSH r27,`$FRAME+21*16+1*$SIZE_T`($sp) + li $x30,0x30 + $PUSH r28,`$FRAME+21*16+2*$SIZE_T`($sp) + li $x40,0x40 + $PUSH r29,`$FRAME+21*16+3*$SIZE_T`($sp) + li $x50,0x50 + $PUSH r30,`$FRAME+21*16+4*$SIZE_T`($sp) + li $x60,0x60 + $PUSH r31,`$FRAME+21*16+5*$SIZE_T`($sp) + li $x70,0x70 + mtspr 256,r0 + + subi $rounds,$rounds,3 # -4 in total + subi $len,$len,128 # bias + + lvx $rndkey0,$x00,$key # load key schedule + lvx v30,$x10,$key + addi $key,$key,0x20 + lvx v31,$x00,$key + ?vperm $rndkey0,$rndkey0,v30,$keyperm + addi $key_,$sp,$FRAME+15 + mtctr $rounds + +Load_cbc_dec_key: + ?vperm v24,v30,v31,$keyperm + lvx v30,$x10,$key + addi $key,$key,0x20 + stvx v24,$x00,$key_ # off-load round[1] + ?vperm v25,v31,v30,$keyperm + lvx v31,$x00,$key + stvx v25,$x10,$key_ # off-load round[2] + addi $key_,$key_,0x20 + bdnz Load_cbc_dec_key + + lvx v26,$x10,$key + ?vperm v24,v30,v31,$keyperm + lvx v27,$x20,$key + stvx v24,$x00,$key_ # off-load round[3] + ?vperm v25,v31,v26,$keyperm + lvx v28,$x30,$key + stvx v25,$x10,$key_ # off-load round[4] + addi $key_,$sp,$FRAME+15 # rewind $key_ + ?vperm v26,v26,v27,$keyperm + lvx v29,$x40,$key + ?vperm v27,v27,v28,$keyperm + lvx v30,$x50,$key + ?vperm v28,v28,v29,$keyperm + lvx v31,$x60,$key + ?vperm v29,v29,v30,$keyperm + lvx $out0,$x70,$key # borrow $out0 + ?vperm v30,v30,v31,$keyperm + lvx v24,$x00,$key_ # pre-load round[1] + ?vperm v31,v31,$out0,$keyperm + lvx v25,$x10,$key_ # pre-load round[2] + + #lvx $inptail,0,$inp # "caller" already did this + #addi $inp,$inp,15 # 15 is not typo + subi $inp,$inp,15 # undo "caller" + + le?li $idx,8 + lvx_u $in0,$x00,$inp # load first 8 "words" + le?lvsl $inpperm,0,$idx + le?vspltisb $tmp,0x0f + lvx_u $in1,$x10,$inp + le?vxor $inpperm,$inpperm,$tmp # transform for lvx_u/stvx_u + lvx_u $in2,$x20,$inp + le?vperm $in0,$in0,$in0,$inpperm + lvx_u $in3,$x30,$inp + le?vperm $in1,$in1,$in1,$inpperm + lvx_u $in4,$x40,$inp + le?vperm $in2,$in2,$in2,$inpperm + vxor $out0,$in0,$rndkey0 + lvx_u $in5,$x50,$inp + le?vperm $in3,$in3,$in3,$inpperm + vxor $out1,$in1,$rndkey0 + lvx_u $in6,$x60,$inp + le?vperm $in4,$in4,$in4,$inpperm + vxor $out2,$in2,$rndkey0 + lvx_u $in7,$x70,$inp + addi $inp,$inp,0x80 + le?vperm $in5,$in5,$in5,$inpperm + vxor $out3,$in3,$rndkey0 + le?vperm $in6,$in6,$in6,$inpperm + vxor $out4,$in4,$rndkey0 + le?vperm $in7,$in7,$in7,$inpperm + vxor $out5,$in5,$rndkey0 + vxor $out6,$in6,$rndkey0 + vxor $out7,$in7,$rndkey0 + + mtctr $rounds + b Loop_cbc_dec8x +.align 5 +Loop_cbc_dec8x: + vncipher $out0,$out0,v24 + vncipher $out1,$out1,v24 + vncipher $out2,$out2,v24 + vncipher $out3,$out3,v24 + vncipher $out4,$out4,v24 + vncipher $out5,$out5,v24 + vncipher $out6,$out6,v24 + vncipher $out7,$out7,v24 + lvx v24,$x20,$key_ # round[3] + addi $key_,$key_,0x20 + + vncipher $out0,$out0,v25 + vncipher $out1,$out1,v25 + vncipher $out2,$out2,v25 + vncipher $out3,$out3,v25 + vncipher $out4,$out4,v25 + vncipher $out5,$out5,v25 + vncipher $out6,$out6,v25 + vncipher $out7,$out7,v25 + lvx v25,$x10,$key_ # round[4] + bdnz Loop_cbc_dec8x + + subic $len,$len,128 # $len-=128 + vncipher $out0,$out0,v24 + vncipher $out1,$out1,v24 + vncipher $out2,$out2,v24 + vncipher $out3,$out3,v24 + vncipher $out4,$out4,v24 + vncipher $out5,$out5,v24 + vncipher $out6,$out6,v24 + vncipher $out7,$out7,v24 + + subfe. r0,r0,r0 # borrow?-1:0 + vncipher $out0,$out0,v25 + vncipher $out1,$out1,v25 + vncipher $out2,$out2,v25 + vncipher $out3,$out3,v25 + vncipher $out4,$out4,v25 + vncipher $out5,$out5,v25 + vncipher $out6,$out6,v25 + vncipher $out7,$out7,v25 + + and r0,r0,$len + vncipher $out0,$out0,v26 + vncipher $out1,$out1,v26 + vncipher $out2,$out2,v26 + vncipher $out3,$out3,v26 + vncipher $out4,$out4,v26 + vncipher $out5,$out5,v26 + vncipher $out6,$out6,v26 + vncipher $out7,$out7,v26 + + add $inp,$inp,r0 # $inp is adjusted in such + # way that at exit from the + # loop inX-in7 are loaded + # with last "words" + vncipher $out0,$out0,v27 + vncipher $out1,$out1,v27 + vncipher $out2,$out2,v27 + vncipher $out3,$out3,v27 + vncipher $out4,$out4,v27 + vncipher $out5,$out5,v27 + vncipher $out6,$out6,v27 + vncipher $out7,$out7,v27 + + addi $key_,$sp,$FRAME+15 # rewind $key_ + vncipher $out0,$out0,v28 + vncipher $out1,$out1,v28 + vncipher $out2,$out2,v28 + vncipher $out3,$out3,v28 + vncipher $out4,$out4,v28 + vncipher $out5,$out5,v28 + vncipher $out6,$out6,v28 + vncipher $out7,$out7,v28 + lvx v24,$x00,$key_ # re-pre-load round[1] + + vncipher $out0,$out0,v29 + vncipher $out1,$out1,v29 + vncipher $out2,$out2,v29 + vncipher $out3,$out3,v29 + vncipher $out4,$out4,v29 + vncipher $out5,$out5,v29 + vncipher $out6,$out6,v29 + vncipher $out7,$out7,v29 + lvx v25,$x10,$key_ # re-pre-load round[2] + + vncipher $out0,$out0,v30 + vxor $ivec,$ivec,v31 # xor with last round key + vncipher $out1,$out1,v30 + vxor $in0,$in0,v31 + vncipher $out2,$out2,v30 + vxor $in1,$in1,v31 + vncipher $out3,$out3,v30 + vxor $in2,$in2,v31 + vncipher $out4,$out4,v30 + vxor $in3,$in3,v31 + vncipher $out5,$out5,v30 + vxor $in4,$in4,v31 + vncipher $out6,$out6,v30 + vxor $in5,$in5,v31 + vncipher $out7,$out7,v30 + vxor $in6,$in6,v31 + + vncipherlast $out0,$out0,$ivec + vncipherlast $out1,$out1,$in0 + lvx_u $in0,$x00,$inp # load next input block + vncipherlast $out2,$out2,$in1 + lvx_u $in1,$x10,$inp + vncipherlast $out3,$out3,$in2 + le?vperm $in0,$in0,$in0,$inpperm + lvx_u $in2,$x20,$inp + vncipherlast $out4,$out4,$in3 + le?vperm $in1,$in1,$in1,$inpperm + lvx_u $in3,$x30,$inp + vncipherlast $out5,$out5,$in4 + le?vperm $in2,$in2,$in2,$inpperm + lvx_u $in4,$x40,$inp + vncipherlast $out6,$out6,$in5 + le?vperm $in3,$in3,$in3,$inpperm + lvx_u $in5,$x50,$inp + vncipherlast $out7,$out7,$in6 + le?vperm $in4,$in4,$in4,$inpperm + lvx_u $in6,$x60,$inp + vmr $ivec,$in7 + le?vperm $in5,$in5,$in5,$inpperm + lvx_u $in7,$x70,$inp + addi $inp,$inp,0x80 + + le?vperm $out0,$out0,$out0,$inpperm + le?vperm $out1,$out1,$out1,$inpperm + stvx_u $out0,$x00,$out + le?vperm $in6,$in6,$in6,$inpperm + vxor $out0,$in0,$rndkey0 + le?vperm $out2,$out2,$out2,$inpperm + stvx_u $out1,$x10,$out + le?vperm $in7,$in7,$in7,$inpperm + vxor $out1,$in1,$rndkey0 + le?vperm $out3,$out3,$out3,$inpperm + stvx_u $out2,$x20,$out + vxor $out2,$in2,$rndkey0 + le?vperm $out4,$out4,$out4,$inpperm + stvx_u $out3,$x30,$out + vxor $out3,$in3,$rndkey0 + le?vperm $out5,$out5,$out5,$inpperm + stvx_u $out4,$x40,$out + vxor $out4,$in4,$rndkey0 + le?vperm $out6,$out6,$out6,$inpperm + stvx_u $out5,$x50,$out + vxor $out5,$in5,$rndkey0 + le?vperm $out7,$out7,$out7,$inpperm + stvx_u $out6,$x60,$out + vxor $out6,$in6,$rndkey0 + stvx_u $out7,$x70,$out + addi $out,$out,0x80 + vxor $out7,$in7,$rndkey0 + + mtctr $rounds + beq Loop_cbc_dec8x # did $len-=128 borrow? + + addic. $len,$len,128 + beq Lcbc_dec8x_done + nop + nop + +Loop_cbc_dec8x_tail: # up to 7 "words" tail... + vncipher $out1,$out1,v24 + vncipher $out2,$out2,v24 + vncipher $out3,$out3,v24 + vncipher $out4,$out4,v24 + vncipher $out5,$out5,v24 + vncipher $out6,$out6,v24 + vncipher $out7,$out7,v24 + lvx v24,$x20,$key_ # round[3] + addi $key_,$key_,0x20 + + vncipher $out1,$out1,v25 + vncipher $out2,$out2,v25 + vncipher $out3,$out3,v25 + vncipher $out4,$out4,v25 + vncipher $out5,$out5,v25 + vncipher $out6,$out6,v25 + vncipher $out7,$out7,v25 + lvx v25,$x10,$key_ # round[4] + bdnz Loop_cbc_dec8x_tail + + vncipher $out1,$out1,v24 + vncipher $out2,$out2,v24 + vncipher $out3,$out3,v24 + vncipher $out4,$out4,v24 + vncipher $out5,$out5,v24 + vncipher $out6,$out6,v24 + vncipher $out7,$out7,v24 + + vncipher $out1,$out1,v25 + vncipher $out2,$out2,v25 + vncipher $out3,$out3,v25 + vncipher $out4,$out4,v25 + vncipher $out5,$out5,v25 + vncipher $out6,$out6,v25 + vncipher $out7,$out7,v25 + + vncipher $out1,$out1,v26 + vncipher $out2,$out2,v26 + vncipher $out3,$out3,v26 + vncipher $out4,$out4,v26 + vncipher $out5,$out5,v26 + vncipher $out6,$out6,v26 + vncipher $out7,$out7,v26 + + vncipher $out1,$out1,v27 + vncipher $out2,$out2,v27 + vncipher $out3,$out3,v27 + vncipher $out4,$out4,v27 + vncipher $out5,$out5,v27 + vncipher $out6,$out6,v27 + vncipher $out7,$out7,v27 + + vncipher $out1,$out1,v28 + vncipher $out2,$out2,v28 + vncipher $out3,$out3,v28 + vncipher $out4,$out4,v28 + vncipher $out5,$out5,v28 + vncipher $out6,$out6,v28 + vncipher $out7,$out7,v28 + + vncipher $out1,$out1,v29 + vncipher $out2,$out2,v29 + vncipher $out3,$out3,v29 + vncipher $out4,$out4,v29 + vncipher $out5,$out5,v29 + vncipher $out6,$out6,v29 + vncipher $out7,$out7,v29 + + vncipher $out1,$out1,v30 + vxor $ivec,$ivec,v31 # last round key + vncipher $out2,$out2,v30 + vxor $in1,$in1,v31 + vncipher $out3,$out3,v30 + vxor $in2,$in2,v31 + vncipher $out4,$out4,v30 + vxor $in3,$in3,v31 + vncipher $out5,$out5,v30 + vxor $in4,$in4,v31 + vncipher $out6,$out6,v30 + vxor $in5,$in5,v31 + vncipher $out7,$out7,v30 + vxor $in6,$in6,v31 + + cmplwi $len,32 # switch($len) + blt Lcbc_dec8x_one + nop + beq Lcbc_dec8x_two + cmplwi $len,64 + blt Lcbc_dec8x_three + nop + beq Lcbc_dec8x_four + cmplwi $len,96 + blt Lcbc_dec8x_five + nop + beq Lcbc_dec8x_six + +Lcbc_dec8x_seven: + vncipherlast $out1,$out1,$ivec + vncipherlast $out2,$out2,$in1 + vncipherlast $out3,$out3,$in2 + vncipherlast $out4,$out4,$in3 + vncipherlast $out5,$out5,$in4 + vncipherlast $out6,$out6,$in5 + vncipherlast $out7,$out7,$in6 + vmr $ivec,$in7 + + le?vperm $out1,$out1,$out1,$inpperm + le?vperm $out2,$out2,$out2,$inpperm + stvx_u $out1,$x00,$out + le?vperm $out3,$out3,$out3,$inpperm + stvx_u $out2,$x10,$out + le?vperm $out4,$out4,$out4,$inpperm + stvx_u $out3,$x20,$out + le?vperm $out5,$out5,$out5,$inpperm + stvx_u $out4,$x30,$out + le?vperm $out6,$out6,$out6,$inpperm + stvx_u $out5,$x40,$out + le?vperm $out7,$out7,$out7,$inpperm + stvx_u $out6,$x50,$out + stvx_u $out7,$x60,$out + addi $out,$out,0x70 + b Lcbc_dec8x_done + +.align 5 +Lcbc_dec8x_six: + vncipherlast $out2,$out2,$ivec + vncipherlast $out3,$out3,$in2 + vncipherlast $out4,$out4,$in3 + vncipherlast $out5,$out5,$in4 + vncipherlast $out6,$out6,$in5 + vncipherlast $out7,$out7,$in6 + vmr $ivec,$in7 + + le?vperm $out2,$out2,$out2,$inpperm + le?vperm $out3,$out3,$out3,$inpperm + stvx_u $out2,$x00,$out + le?vperm $out4,$out4,$out4,$inpperm + stvx_u $out3,$x10,$out + le?vperm $out5,$out5,$out5,$inpperm + stvx_u $out4,$x20,$out + le?vperm $out6,$out6,$out6,$inpperm + stvx_u $out5,$x30,$out + le?vperm $out7,$out7,$out7,$inpperm + stvx_u $out6,$x40,$out + stvx_u $out7,$x50,$out + addi $out,$out,0x60 + b Lcbc_dec8x_done + +.align 5 +Lcbc_dec8x_five: + vncipherlast $out3,$out3,$ivec + vncipherlast $out4,$out4,$in3 + vncipherlast $out5,$out5,$in4 + vncipherlast $out6,$out6,$in5 + vncipherlast $out7,$out7,$in6 + vmr $ivec,$in7 + + le?vperm $out3,$out3,$out3,$inpperm + le?vperm $out4,$out4,$out4,$inpperm + stvx_u $out3,$x00,$out + le?vperm $out5,$out5,$out5,$inpperm + stvx_u $out4,$x10,$out + le?vperm $out6,$out6,$out6,$inpperm + stvx_u $out5,$x20,$out + le?vperm $out7,$out7,$out7,$inpperm + stvx_u $out6,$x30,$out + stvx_u $out7,$x40,$out + addi $out,$out,0x50 + b Lcbc_dec8x_done + +.align 5 +Lcbc_dec8x_four: + vncipherlast $out4,$out4,$ivec + vncipherlast $out5,$out5,$in4 + vncipherlast $out6,$out6,$in5 + vncipherlast $out7,$out7,$in6 + vmr $ivec,$in7 + + le?vperm $out4,$out4,$out4,$inpperm + le?vperm $out5,$out5,$out5,$inpperm + stvx_u $out4,$x00,$out + le?vperm $out6,$out6,$out6,$inpperm + stvx_u $out5,$x10,$out + le?vperm $out7,$out7,$out7,$inpperm + stvx_u $out6,$x20,$out + stvx_u $out7,$x30,$out + addi $out,$out,0x40 + b Lcbc_dec8x_done + +.align 5 +Lcbc_dec8x_three: + vncipherlast $out5,$out5,$ivec + vncipherlast $out6,$out6,$in5 + vncipherlast $out7,$out7,$in6 + vmr $ivec,$in7 + + le?vperm $out5,$out5,$out5,$inpperm + le?vperm $out6,$out6,$out6,$inpperm + stvx_u $out5,$x00,$out + le?vperm $out7,$out7,$out7,$inpperm + stvx_u $out6,$x10,$out + stvx_u $out7,$x20,$out + addi $out,$out,0x30 + b Lcbc_dec8x_done + +.align 5 +Lcbc_dec8x_two: + vncipherlast $out6,$out6,$ivec + vncipherlast $out7,$out7,$in6 + vmr $ivec,$in7 + + le?vperm $out6,$out6,$out6,$inpperm + le?vperm $out7,$out7,$out7,$inpperm + stvx_u $out6,$x00,$out + stvx_u $out7,$x10,$out + addi $out,$out,0x20 + b Lcbc_dec8x_done + +.align 5 +Lcbc_dec8x_one: + vncipherlast $out7,$out7,$ivec + vmr $ivec,$in7 + + le?vperm $out7,$out7,$out7,$inpperm + stvx_u $out7,0,$out + addi $out,$out,0x10 + +Lcbc_dec8x_done: + le?vperm $ivec,$ivec,$ivec,$inpperm + stvx_u $ivec,0,$ivp # write [unaligned] iv + + li r10,`$FRAME+15` + li r11,`$FRAME+31` + stvx $inpperm,r10,$sp # wipe copies of round keys + addi r10,r10,32 + stvx $inpperm,r11,$sp + addi r11,r11,32 + stvx $inpperm,r10,$sp + addi r10,r10,32 + stvx $inpperm,r11,$sp + addi r11,r11,32 + stvx $inpperm,r10,$sp + addi r10,r10,32 + stvx $inpperm,r11,$sp + addi r11,r11,32 + stvx $inpperm,r10,$sp + addi r10,r10,32 + stvx $inpperm,r11,$sp + addi r11,r11,32 + + mtspr 256,$vrsave + lvx v20,r10,$sp # ABI says so + addi r10,r10,32 + lvx v21,r11,$sp + addi r11,r11,32 + lvx v22,r10,$sp + addi r10,r10,32 + lvx v23,r11,$sp + addi r11,r11,32 + lvx v24,r10,$sp + addi r10,r10,32 + lvx v25,r11,$sp + addi r11,r11,32 + lvx v26,r10,$sp + addi r10,r10,32 + lvx v27,r11,$sp + addi r11,r11,32 + lvx v28,r10,$sp + addi r10,r10,32 + lvx v29,r11,$sp + addi r11,r11,32 + lvx v30,r10,$sp + lvx v31,r11,$sp + $POP r26,`$FRAME+21*16+0*$SIZE_T`($sp) + $POP r27,`$FRAME+21*16+1*$SIZE_T`($sp) + $POP r28,`$FRAME+21*16+2*$SIZE_T`($sp) + $POP r29,`$FRAME+21*16+3*$SIZE_T`($sp) + $POP r30,`$FRAME+21*16+4*$SIZE_T`($sp) + $POP r31,`$FRAME+21*16+5*$SIZE_T`($sp) + addi $sp,$sp,`$FRAME+21*16+6*$SIZE_T` + blr + .long 0 + .byte 0,12,0x04,0,0x80,6,6,0 + .long 0 +.size .${prefix}_cbc_encrypt,.-.${prefix}_cbc_encrypt +___ +}} }}} + +######################################################################### +{{{ # CTR procedure[s] # +my ($inp,$out,$len,$key,$ivp,$x10,$rounds,$idx)=map("r$_",(3..10)); +my ($rndkey0,$rndkey1,$inout,$tmp)= map("v$_",(0..3)); +my ($ivec,$inptail,$inpperm,$outhead,$outperm,$outmask,$keyperm,$one)= + map("v$_",(4..11)); +my $dat=$tmp; + +$code.=<<___; +.globl .${prefix}_ctr32_encrypt_blocks +.align 5 +.${prefix}_ctr32_encrypt_blocks: + ${UCMP}i $len,1 + bltlr- + + lis r0,0xfff0 + mfspr $vrsave,256 + mtspr 256,r0 + + li $idx,15 + vxor $rndkey0,$rndkey0,$rndkey0 + le?vspltisb $tmp,0x0f + + lvx $ivec,0,$ivp # load [unaligned] iv + lvsl $inpperm,0,$ivp + lvx $inptail,$idx,$ivp + vspltisb $one,1 + le?vxor $inpperm,$inpperm,$tmp + vperm $ivec,$ivec,$inptail,$inpperm + vsldoi $one,$rndkey0,$one,1 + + neg r11,$inp + ?lvsl $keyperm,0,$key # prepare for unaligned key + lwz $rounds,240($key) + + lvsr $inpperm,0,r11 # prepare for unaligned load + lvx $inptail,0,$inp + addi $inp,$inp,15 # 15 is not typo + le?vxor $inpperm,$inpperm,$tmp + + srwi $rounds,$rounds,1 + li $idx,16 + subi $rounds,$rounds,1 + + ${UCMP}i $len,8 + bge _aesp8_ctr32_encrypt8x + + ?lvsr $outperm,0,$out # prepare for unaligned store + vspltisb $outmask,-1 + lvx $outhead,0,$out + ?vperm $outmask,$rndkey0,$outmask,$outperm + le?vxor $outperm,$outperm,$tmp + + lvx $rndkey0,0,$key + mtctr $rounds + lvx $rndkey1,$idx,$key + addi $idx,$idx,16 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vxor $inout,$ivec,$rndkey0 + lvx $rndkey0,$idx,$key + addi $idx,$idx,16 + b Loop_ctr32_enc + +.align 5 +Loop_ctr32_enc: + ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm + vcipher $inout,$inout,$rndkey1 + lvx $rndkey1,$idx,$key + addi $idx,$idx,16 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vcipher $inout,$inout,$rndkey0 + lvx $rndkey0,$idx,$key + addi $idx,$idx,16 + bdnz Loop_ctr32_enc + + vadduwm $ivec,$ivec,$one + vmr $dat,$inptail + lvx $inptail,0,$inp + addi $inp,$inp,16 + subic. $len,$len,1 # blocks-- + + ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm + vcipher $inout,$inout,$rndkey1 + lvx $rndkey1,$idx,$key + vperm $dat,$dat,$inptail,$inpperm + li $idx,16 + ?vperm $rndkey1,$rndkey0,$rndkey1,$keyperm + lvx $rndkey0,0,$key + vxor $dat,$dat,$rndkey1 # last round key + vcipherlast $inout,$inout,$dat + + lvx $rndkey1,$idx,$key + addi $idx,$idx,16 + vperm $inout,$inout,$inout,$outperm + vsel $dat,$outhead,$inout,$outmask + mtctr $rounds + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vmr $outhead,$inout + vxor $inout,$ivec,$rndkey0 + lvx $rndkey0,$idx,$key + addi $idx,$idx,16 + stvx $dat,0,$out + addi $out,$out,16 + bne Loop_ctr32_enc + + addi $out,$out,-1 + lvx $inout,0,$out # redundant in aligned case + vsel $inout,$outhead,$inout,$outmask + stvx $inout,0,$out + + mtspr 256,$vrsave + blr + .long 0 + .byte 0,12,0x14,0,0,0,6,0 + .long 0 +___ +######################################################################### +{{ # Optimized CTR procedure # +my $key_="r11"; +my ($x00,$x10,$x20,$x30,$x40,$x50,$x60,$x70)=map("r$_",(0,8,26..31)); + $x00=0 if ($flavour =~ /osx/); +my ($in0, $in1, $in2, $in3, $in4, $in5, $in6, $in7 )=map("v$_",(0..3,10,12..14)); +my ($out0,$out1,$out2,$out3,$out4,$out5,$out6,$out7)=map("v$_",(15..22)); +my $rndkey0="v23"; # v24-v25 rotating buffer for first found keys + # v26-v31 last 6 round keys +my ($tmp,$keyperm)=($in3,$in4); # aliases with "caller", redundant assignment +my ($two,$three,$four)=($outhead,$outperm,$outmask); + +$code.=<<___; +.align 5 +_aesp8_ctr32_encrypt8x: + $STU $sp,-`($FRAME+21*16+6*$SIZE_T)`($sp) + li r10,`$FRAME+8*16+15` + li r11,`$FRAME+8*16+31` + stvx v20,r10,$sp # ABI says so + addi r10,r10,32 + stvx v21,r11,$sp + addi r11,r11,32 + stvx v22,r10,$sp + addi r10,r10,32 + stvx v23,r11,$sp + addi r11,r11,32 + stvx v24,r10,$sp + addi r10,r10,32 + stvx v25,r11,$sp + addi r11,r11,32 + stvx v26,r10,$sp + addi r10,r10,32 + stvx v27,r11,$sp + addi r11,r11,32 + stvx v28,r10,$sp + addi r10,r10,32 + stvx v29,r11,$sp + addi r11,r11,32 + stvx v30,r10,$sp + stvx v31,r11,$sp + li r0,-1 + stw $vrsave,`$FRAME+21*16-4`($sp) # save vrsave + li $x10,0x10 + $PUSH r26,`$FRAME+21*16+0*$SIZE_T`($sp) + li $x20,0x20 + $PUSH r27,`$FRAME+21*16+1*$SIZE_T`($sp) + li $x30,0x30 + $PUSH r28,`$FRAME+21*16+2*$SIZE_T`($sp) + li $x40,0x40 + $PUSH r29,`$FRAME+21*16+3*$SIZE_T`($sp) + li $x50,0x50 + $PUSH r30,`$FRAME+21*16+4*$SIZE_T`($sp) + li $x60,0x60 + $PUSH r31,`$FRAME+21*16+5*$SIZE_T`($sp) + li $x70,0x70 + mtspr 256,r0 + + subi $rounds,$rounds,3 # -4 in total + + lvx $rndkey0,$x00,$key # load key schedule + lvx v30,$x10,$key + addi $key,$key,0x20 + lvx v31,$x00,$key + ?vperm $rndkey0,$rndkey0,v30,$keyperm + addi $key_,$sp,$FRAME+15 + mtctr $rounds + +Load_ctr32_enc_key: + ?vperm v24,v30,v31,$keyperm + lvx v30,$x10,$key + addi $key,$key,0x20 + stvx v24,$x00,$key_ # off-load round[1] + ?vperm v25,v31,v30,$keyperm + lvx v31,$x00,$key + stvx v25,$x10,$key_ # off-load round[2] + addi $key_,$key_,0x20 + bdnz Load_ctr32_enc_key + + lvx v26,$x10,$key + ?vperm v24,v30,v31,$keyperm + lvx v27,$x20,$key + stvx v24,$x00,$key_ # off-load round[3] + ?vperm v25,v31,v26,$keyperm + lvx v28,$x30,$key + stvx v25,$x10,$key_ # off-load round[4] + addi $key_,$sp,$FRAME+15 # rewind $key_ + ?vperm v26,v26,v27,$keyperm + lvx v29,$x40,$key + ?vperm v27,v27,v28,$keyperm + lvx v30,$x50,$key + ?vperm v28,v28,v29,$keyperm + lvx v31,$x60,$key + ?vperm v29,v29,v30,$keyperm + lvx $out0,$x70,$key # borrow $out0 + ?vperm v30,v30,v31,$keyperm + lvx v24,$x00,$key_ # pre-load round[1] + ?vperm v31,v31,$out0,$keyperm + lvx v25,$x10,$key_ # pre-load round[2] + + vadduwm $two,$one,$one + subi $inp,$inp,15 # undo "caller" + $SHL $len,$len,4 + + vadduwm $out1,$ivec,$one # counter values ... + vadduwm $out2,$ivec,$two + vxor $out0,$ivec,$rndkey0 # ... xored with rndkey[0] + le?li $idx,8 + vadduwm $out3,$out1,$two + vxor $out1,$out1,$rndkey0 + le?lvsl $inpperm,0,$idx + vadduwm $out4,$out2,$two + vxor $out2,$out2,$rndkey0 + le?vspltisb $tmp,0x0f + vadduwm $out5,$out3,$two + vxor $out3,$out3,$rndkey0 + le?vxor $inpperm,$inpperm,$tmp # transform for lvx_u/stvx_u + vadduwm $out6,$out4,$two + vxor $out4,$out4,$rndkey0 + vadduwm $out7,$out5,$two + vxor $out5,$out5,$rndkey0 + vadduwm $ivec,$out6,$two # next counter value + vxor $out6,$out6,$rndkey0 + vxor $out7,$out7,$rndkey0 + + mtctr $rounds + b Loop_ctr32_enc8x +.align 5 +Loop_ctr32_enc8x: + vcipher $out0,$out0,v24 + vcipher $out1,$out1,v24 + vcipher $out2,$out2,v24 + vcipher $out3,$out3,v24 + vcipher $out4,$out4,v24 + vcipher $out5,$out5,v24 + vcipher $out6,$out6,v24 + vcipher $out7,$out7,v24 +Loop_ctr32_enc8x_middle: + lvx v24,$x20,$key_ # round[3] + addi $key_,$key_,0x20 + + vcipher $out0,$out0,v25 + vcipher $out1,$out1,v25 + vcipher $out2,$out2,v25 + vcipher $out3,$out3,v25 + vcipher $out4,$out4,v25 + vcipher $out5,$out5,v25 + vcipher $out6,$out6,v25 + vcipher $out7,$out7,v25 + lvx v25,$x10,$key_ # round[4] + bdnz Loop_ctr32_enc8x + + subic r11,$len,256 # $len-256, borrow $key_ + vcipher $out0,$out0,v24 + vcipher $out1,$out1,v24 + vcipher $out2,$out2,v24 + vcipher $out3,$out3,v24 + vcipher $out4,$out4,v24 + vcipher $out5,$out5,v24 + vcipher $out6,$out6,v24 + vcipher $out7,$out7,v24 + + subfe r0,r0,r0 # borrow?-1:0 + vcipher $out0,$out0,v25 + vcipher $out1,$out1,v25 + vcipher $out2,$out2,v25 + vcipher $out3,$out3,v25 + vcipher $out4,$out4,v25 + vcipher $out5,$out5,v25 + vcipher $out6,$out6,v25 + vcipher $out7,$out7,v25 + + and r0,r0,r11 + addi $key_,$sp,$FRAME+15 # rewind $key_ + vcipher $out0,$out0,v26 + vcipher $out1,$out1,v26 + vcipher $out2,$out2,v26 + vcipher $out3,$out3,v26 + vcipher $out4,$out4,v26 + vcipher $out5,$out5,v26 + vcipher $out6,$out6,v26 + vcipher $out7,$out7,v26 + lvx v24,$x00,$key_ # re-pre-load round[1] + + subic $len,$len,129 # $len-=129 + vcipher $out0,$out0,v27 + addi $len,$len,1 # $len-=128 really + vcipher $out1,$out1,v27 + vcipher $out2,$out2,v27 + vcipher $out3,$out3,v27 + vcipher $out4,$out4,v27 + vcipher $out5,$out5,v27 + vcipher $out6,$out6,v27 + vcipher $out7,$out7,v27 + lvx v25,$x10,$key_ # re-pre-load round[2] + + vcipher $out0,$out0,v28 + lvx_u $in0,$x00,$inp # load input + vcipher $out1,$out1,v28 + lvx_u $in1,$x10,$inp + vcipher $out2,$out2,v28 + lvx_u $in2,$x20,$inp + vcipher $out3,$out3,v28 + lvx_u $in3,$x30,$inp + vcipher $out4,$out4,v28 + lvx_u $in4,$x40,$inp + vcipher $out5,$out5,v28 + lvx_u $in5,$x50,$inp + vcipher $out6,$out6,v28 + lvx_u $in6,$x60,$inp + vcipher $out7,$out7,v28 + lvx_u $in7,$x70,$inp + addi $inp,$inp,0x80 + + vcipher $out0,$out0,v29 + le?vperm $in0,$in0,$in0,$inpperm + vcipher $out1,$out1,v29 + le?vperm $in1,$in1,$in1,$inpperm + vcipher $out2,$out2,v29 + le?vperm $in2,$in2,$in2,$inpperm + vcipher $out3,$out3,v29 + le?vperm $in3,$in3,$in3,$inpperm + vcipher $out4,$out4,v29 + le?vperm $in4,$in4,$in4,$inpperm + vcipher $out5,$out5,v29 + le?vperm $in5,$in5,$in5,$inpperm + vcipher $out6,$out6,v29 + le?vperm $in6,$in6,$in6,$inpperm + vcipher $out7,$out7,v29 + le?vperm $in7,$in7,$in7,$inpperm + + add $inp,$inp,r0 # $inp is adjusted in such + # way that at exit from the + # loop inX-in7 are loaded + # with last "words" + subfe. r0,r0,r0 # borrow?-1:0 + vcipher $out0,$out0,v30 + vxor $in0,$in0,v31 # xor with last round key + vcipher $out1,$out1,v30 + vxor $in1,$in1,v31 + vcipher $out2,$out2,v30 + vxor $in2,$in2,v31 + vcipher $out3,$out3,v30 + vxor $in3,$in3,v31 + vcipher $out4,$out4,v30 + vxor $in4,$in4,v31 + vcipher $out5,$out5,v30 + vxor $in5,$in5,v31 + vcipher $out6,$out6,v30 + vxor $in6,$in6,v31 + vcipher $out7,$out7,v30 + vxor $in7,$in7,v31 + + bne Lctr32_enc8x_break # did $len-129 borrow? + + vcipherlast $in0,$out0,$in0 + vcipherlast $in1,$out1,$in1 + vadduwm $out1,$ivec,$one # counter values ... + vcipherlast $in2,$out2,$in2 + vadduwm $out2,$ivec,$two + vxor $out0,$ivec,$rndkey0 # ... xored with rndkey[0] + vcipherlast $in3,$out3,$in3 + vadduwm $out3,$out1,$two + vxor $out1,$out1,$rndkey0 + vcipherlast $in4,$out4,$in4 + vadduwm $out4,$out2,$two + vxor $out2,$out2,$rndkey0 + vcipherlast $in5,$out5,$in5 + vadduwm $out5,$out3,$two + vxor $out3,$out3,$rndkey0 + vcipherlast $in6,$out6,$in6 + vadduwm $out6,$out4,$two + vxor $out4,$out4,$rndkey0 + vcipherlast $in7,$out7,$in7 + vadduwm $out7,$out5,$two + vxor $out5,$out5,$rndkey0 + le?vperm $in0,$in0,$in0,$inpperm + vadduwm $ivec,$out6,$two # next counter value + vxor $out6,$out6,$rndkey0 + le?vperm $in1,$in1,$in1,$inpperm + vxor $out7,$out7,$rndkey0 + mtctr $rounds + + vcipher $out0,$out0,v24 + stvx_u $in0,$x00,$out + le?vperm $in2,$in2,$in2,$inpperm + vcipher $out1,$out1,v24 + stvx_u $in1,$x10,$out + le?vperm $in3,$in3,$in3,$inpperm + vcipher $out2,$out2,v24 + stvx_u $in2,$x20,$out + le?vperm $in4,$in4,$in4,$inpperm + vcipher $out3,$out3,v24 + stvx_u $in3,$x30,$out + le?vperm $in5,$in5,$in5,$inpperm + vcipher $out4,$out4,v24 + stvx_u $in4,$x40,$out + le?vperm $in6,$in6,$in6,$inpperm + vcipher $out5,$out5,v24 + stvx_u $in5,$x50,$out + le?vperm $in7,$in7,$in7,$inpperm + vcipher $out6,$out6,v24 + stvx_u $in6,$x60,$out + vcipher $out7,$out7,v24 + stvx_u $in7,$x70,$out + addi $out,$out,0x80 + + b Loop_ctr32_enc8x_middle + +.align 5 +Lctr32_enc8x_break: + cmpwi $len,-0x60 + blt Lctr32_enc8x_one + nop + beq Lctr32_enc8x_two + cmpwi $len,-0x40 + blt Lctr32_enc8x_three + nop + beq Lctr32_enc8x_four + cmpwi $len,-0x20 + blt Lctr32_enc8x_five + nop + beq Lctr32_enc8x_six + cmpwi $len,0x00 + blt Lctr32_enc8x_seven + +Lctr32_enc8x_eight: + vcipherlast $out0,$out0,$in0 + vcipherlast $out1,$out1,$in1 + vcipherlast $out2,$out2,$in2 + vcipherlast $out3,$out3,$in3 + vcipherlast $out4,$out4,$in4 + vcipherlast $out5,$out5,$in5 + vcipherlast $out6,$out6,$in6 + vcipherlast $out7,$out7,$in7 + + le?vperm $out0,$out0,$out0,$inpperm + le?vperm $out1,$out1,$out1,$inpperm + stvx_u $out0,$x00,$out + le?vperm $out2,$out2,$out2,$inpperm + stvx_u $out1,$x10,$out + le?vperm $out3,$out3,$out3,$inpperm + stvx_u $out2,$x20,$out + le?vperm $out4,$out4,$out4,$inpperm + stvx_u $out3,$x30,$out + le?vperm $out5,$out5,$out5,$inpperm + stvx_u $out4,$x40,$out + le?vperm $out6,$out6,$out6,$inpperm + stvx_u $out5,$x50,$out + le?vperm $out7,$out7,$out7,$inpperm + stvx_u $out6,$x60,$out + stvx_u $out7,$x70,$out + addi $out,$out,0x80 + b Lctr32_enc8x_done + +.align 5 +Lctr32_enc8x_seven: + vcipherlast $out0,$out0,$in1 + vcipherlast $out1,$out1,$in2 + vcipherlast $out2,$out2,$in3 + vcipherlast $out3,$out3,$in4 + vcipherlast $out4,$out4,$in5 + vcipherlast $out5,$out5,$in6 + vcipherlast $out6,$out6,$in7 + + le?vperm $out0,$out0,$out0,$inpperm + le?vperm $out1,$out1,$out1,$inpperm + stvx_u $out0,$x00,$out + le?vperm $out2,$out2,$out2,$inpperm + stvx_u $out1,$x10,$out + le?vperm $out3,$out3,$out3,$inpperm + stvx_u $out2,$x20,$out + le?vperm $out4,$out4,$out4,$inpperm + stvx_u $out3,$x30,$out + le?vperm $out5,$out5,$out5,$inpperm + stvx_u $out4,$x40,$out + le?vperm $out6,$out6,$out6,$inpperm + stvx_u $out5,$x50,$out + stvx_u $out6,$x60,$out + addi $out,$out,0x70 + b Lctr32_enc8x_done + +.align 5 +Lctr32_enc8x_six: + vcipherlast $out0,$out0,$in2 + vcipherlast $out1,$out1,$in3 + vcipherlast $out2,$out2,$in4 + vcipherlast $out3,$out3,$in5 + vcipherlast $out4,$out4,$in6 + vcipherlast $out5,$out5,$in7 + + le?vperm $out0,$out0,$out0,$inpperm + le?vperm $out1,$out1,$out1,$inpperm + stvx_u $out0,$x00,$out + le?vperm $out2,$out2,$out2,$inpperm + stvx_u $out1,$x10,$out + le?vperm $out3,$out3,$out3,$inpperm + stvx_u $out2,$x20,$out + le?vperm $out4,$out4,$out4,$inpperm + stvx_u $out3,$x30,$out + le?vperm $out5,$out5,$out5,$inpperm + stvx_u $out4,$x40,$out + stvx_u $out5,$x50,$out + addi $out,$out,0x60 + b Lctr32_enc8x_done + +.align 5 +Lctr32_enc8x_five: + vcipherlast $out0,$out0,$in3 + vcipherlast $out1,$out1,$in4 + vcipherlast $out2,$out2,$in5 + vcipherlast $out3,$out3,$in6 + vcipherlast $out4,$out4,$in7 + + le?vperm $out0,$out0,$out0,$inpperm + le?vperm $out1,$out1,$out1,$inpperm + stvx_u $out0,$x00,$out + le?vperm $out2,$out2,$out2,$inpperm + stvx_u $out1,$x10,$out + le?vperm $out3,$out3,$out3,$inpperm + stvx_u $out2,$x20,$out + le?vperm $out4,$out4,$out4,$inpperm + stvx_u $out3,$x30,$out + stvx_u $out4,$x40,$out + addi $out,$out,0x50 + b Lctr32_enc8x_done + +.align 5 +Lctr32_enc8x_four: + vcipherlast $out0,$out0,$in4 + vcipherlast $out1,$out1,$in5 + vcipherlast $out2,$out2,$in6 + vcipherlast $out3,$out3,$in7 + + le?vperm $out0,$out0,$out0,$inpperm + le?vperm $out1,$out1,$out1,$inpperm + stvx_u $out0,$x00,$out + le?vperm $out2,$out2,$out2,$inpperm + stvx_u $out1,$x10,$out + le?vperm $out3,$out3,$out3,$inpperm + stvx_u $out2,$x20,$out + stvx_u $out3,$x30,$out + addi $out,$out,0x40 + b Lctr32_enc8x_done + +.align 5 +Lctr32_enc8x_three: + vcipherlast $out0,$out0,$in5 + vcipherlast $out1,$out1,$in6 + vcipherlast $out2,$out2,$in7 + + le?vperm $out0,$out0,$out0,$inpperm + le?vperm $out1,$out1,$out1,$inpperm + stvx_u $out0,$x00,$out + le?vperm $out2,$out2,$out2,$inpperm + stvx_u $out1,$x10,$out + stvx_u $out2,$x20,$out + addi $out,$out,0x30 + b Lctr32_enc8x_done + +.align 5 +Lctr32_enc8x_two: + vcipherlast $out0,$out0,$in6 + vcipherlast $out1,$out1,$in7 + + le?vperm $out0,$out0,$out0,$inpperm + le?vperm $out1,$out1,$out1,$inpperm + stvx_u $out0,$x00,$out + stvx_u $out1,$x10,$out + addi $out,$out,0x20 + b Lctr32_enc8x_done + +.align 5 +Lctr32_enc8x_one: + vcipherlast $out0,$out0,$in7 + + le?vperm $out0,$out0,$out0,$inpperm + stvx_u $out0,0,$out + addi $out,$out,0x10 + +Lctr32_enc8x_done: + li r10,`$FRAME+15` + li r11,`$FRAME+31` + stvx $inpperm,r10,$sp # wipe copies of round keys + addi r10,r10,32 + stvx $inpperm,r11,$sp + addi r11,r11,32 + stvx $inpperm,r10,$sp + addi r10,r10,32 + stvx $inpperm,r11,$sp + addi r11,r11,32 + stvx $inpperm,r10,$sp + addi r10,r10,32 + stvx $inpperm,r11,$sp + addi r11,r11,32 + stvx $inpperm,r10,$sp + addi r10,r10,32 + stvx $inpperm,r11,$sp + addi r11,r11,32 + + mtspr 256,$vrsave + lvx v20,r10,$sp # ABI says so + addi r10,r10,32 + lvx v21,r11,$sp + addi r11,r11,32 + lvx v22,r10,$sp + addi r10,r10,32 + lvx v23,r11,$sp + addi r11,r11,32 + lvx v24,r10,$sp + addi r10,r10,32 + lvx v25,r11,$sp + addi r11,r11,32 + lvx v26,r10,$sp + addi r10,r10,32 + lvx v27,r11,$sp + addi r11,r11,32 + lvx v28,r10,$sp + addi r10,r10,32 + lvx v29,r11,$sp + addi r11,r11,32 + lvx v30,r10,$sp + lvx v31,r11,$sp + $POP r26,`$FRAME+21*16+0*$SIZE_T`($sp) + $POP r27,`$FRAME+21*16+1*$SIZE_T`($sp) + $POP r28,`$FRAME+21*16+2*$SIZE_T`($sp) + $POP r29,`$FRAME+21*16+3*$SIZE_T`($sp) + $POP r30,`$FRAME+21*16+4*$SIZE_T`($sp) + $POP r31,`$FRAME+21*16+5*$SIZE_T`($sp) + addi $sp,$sp,`$FRAME+21*16+6*$SIZE_T` + blr + .long 0 + .byte 0,12,0x04,0,0x80,6,6,0 + .long 0 +.size .${prefix}_ctr32_encrypt_blocks,.-.${prefix}_ctr32_encrypt_blocks +___ +}} }}} + +######################################################################### +{{{ # XTS procedures # +# int aes_p8_xts_[en|de]crypt(const char *inp, char *out, size_t len, # +# const AES_KEY *key1, const AES_KEY *key2, # +# [const] unsigned char iv[16]); # +# If $key2 is NULL, then a "tweak chaining" mode is engaged, in which # +# input tweak value is assumed to be encrypted already, and last tweak # +# value, one suitable for consecutive call on same chunk of data, is # +# written back to original buffer. In addition, in "tweak chaining" # +# mode only complete input blocks are processed. # + +my ($inp,$out,$len,$key1,$key2,$ivp,$rounds,$idx) = map("r$_",(3..10)); +my ($rndkey0,$rndkey1,$inout) = map("v$_",(0..2)); +my ($output,$inptail,$inpperm,$leperm,$keyperm) = map("v$_",(3..7)); +my ($tweak,$seven,$eighty7,$tmp,$tweak1) = map("v$_",(8..12)); +my $taillen = $key2; + + ($inp,$idx) = ($idx,$inp); # reassign + +$code.=<<___; +.globl .${prefix}_xts_encrypt +.align 5 +.${prefix}_xts_encrypt: + mr $inp,r3 # reassign + li r3,-1 + ${UCMP}i $len,16 + bltlr- + + lis r0,0xfff0 + mfspr r12,256 # save vrsave + li r11,0 + mtspr 256,r0 + + vspltisb $seven,0x07 # 0x070707..07 + le?lvsl $leperm,r11,r11 + le?vspltisb $tmp,0x0f + le?vxor $leperm,$leperm,$seven + + li $idx,15 + lvx $tweak,0,$ivp # load [unaligned] iv + lvsl $inpperm,0,$ivp + lvx $inptail,$idx,$ivp + le?vxor $inpperm,$inpperm,$tmp + vperm $tweak,$tweak,$inptail,$inpperm + + neg r11,$inp + lvsr $inpperm,0,r11 # prepare for unaligned load + lvx $inout,0,$inp + addi $inp,$inp,15 # 15 is not typo + le?vxor $inpperm,$inpperm,$tmp + + ${UCMP}i $key2,0 # key2==NULL? + beq Lxts_enc_no_key2 + + ?lvsl $keyperm,0,$key2 # prepare for unaligned key + lwz $rounds,240($key2) + srwi $rounds,$rounds,1 + subi $rounds,$rounds,1 + li $idx,16 + + lvx $rndkey0,0,$key2 + lvx $rndkey1,$idx,$key2 + addi $idx,$idx,16 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vxor $tweak,$tweak,$rndkey0 + lvx $rndkey0,$idx,$key2 + addi $idx,$idx,16 + mtctr $rounds + +Ltweak_xts_enc: + ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm + vcipher $tweak,$tweak,$rndkey1 + lvx $rndkey1,$idx,$key2 + addi $idx,$idx,16 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vcipher $tweak,$tweak,$rndkey0 + lvx $rndkey0,$idx,$key2 + addi $idx,$idx,16 + bdnz Ltweak_xts_enc + + ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm + vcipher $tweak,$tweak,$rndkey1 + lvx $rndkey1,$idx,$key2 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vcipherlast $tweak,$tweak,$rndkey0 + + li $ivp,0 # don't chain the tweak + b Lxts_enc + +Lxts_enc_no_key2: + li $idx,-16 + and $len,$len,$idx # in "tweak chaining" + # mode only complete + # blocks are processed +Lxts_enc: + lvx $inptail,0,$inp + addi $inp,$inp,16 + + ?lvsl $keyperm,0,$key1 # prepare for unaligned key + lwz $rounds,240($key1) + srwi $rounds,$rounds,1 + subi $rounds,$rounds,1 + li $idx,16 + + vslb $eighty7,$seven,$seven # 0x808080..80 + vor $eighty7,$eighty7,$seven # 0x878787..87 + vspltisb $tmp,1 # 0x010101..01 + vsldoi $eighty7,$eighty7,$tmp,15 # 0x870101..01 + + ${UCMP}i $len,96 + bge _aesp8_xts_encrypt6x + + andi. $taillen,$len,15 + subic r0,$len,32 + subi $taillen,$taillen,16 + subfe r0,r0,r0 + and r0,r0,$taillen + add $inp,$inp,r0 + + lvx $rndkey0,0,$key1 + lvx $rndkey1,$idx,$key1 + addi $idx,$idx,16 + vperm $inout,$inout,$inptail,$inpperm + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vxor $inout,$inout,$tweak + vxor $inout,$inout,$rndkey0 + lvx $rndkey0,$idx,$key1 + addi $idx,$idx,16 + mtctr $rounds + b Loop_xts_enc + +.align 5 +Loop_xts_enc: + ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm + vcipher $inout,$inout,$rndkey1 + lvx $rndkey1,$idx,$key1 + addi $idx,$idx,16 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vcipher $inout,$inout,$rndkey0 + lvx $rndkey0,$idx,$key1 + addi $idx,$idx,16 + bdnz Loop_xts_enc + + ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm + vcipher $inout,$inout,$rndkey1 + lvx $rndkey1,$idx,$key1 + li $idx,16 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vxor $rndkey0,$rndkey0,$tweak + vcipherlast $output,$inout,$rndkey0 + + le?vperm $tmp,$output,$output,$leperm + be?nop + le?stvx_u $tmp,0,$out + be?stvx_u $output,0,$out + addi $out,$out,16 + + subic. $len,$len,16 + beq Lxts_enc_done + + vmr $inout,$inptail + lvx $inptail,0,$inp + addi $inp,$inp,16 + lvx $rndkey0,0,$key1 + lvx $rndkey1,$idx,$key1 + addi $idx,$idx,16 + + subic r0,$len,32 + subfe r0,r0,r0 + and r0,r0,$taillen + add $inp,$inp,r0 + + vsrab $tmp,$tweak,$seven # next tweak value + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + vand $tmp,$tmp,$eighty7 + vxor $tweak,$tweak,$tmp + + vperm $inout,$inout,$inptail,$inpperm + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vxor $inout,$inout,$tweak + vxor $output,$output,$rndkey0 # just in case $len<16 + vxor $inout,$inout,$rndkey0 + lvx $rndkey0,$idx,$key1 + addi $idx,$idx,16 + + mtctr $rounds + ${UCMP}i $len,16 + bge Loop_xts_enc + + vxor $output,$output,$tweak + lvsr $inpperm,0,$len # $inpperm is no longer needed + vxor $inptail,$inptail,$inptail # $inptail is no longer needed + vspltisb $tmp,-1 + vperm $inptail,$inptail,$tmp,$inpperm + vsel $inout,$inout,$output,$inptail + + subi r11,$out,17 + subi $out,$out,16 + mtctr $len + li $len,16 +Loop_xts_enc_steal: + lbzu r0,1(r11) + stb r0,16(r11) + bdnz Loop_xts_enc_steal + + mtctr $rounds + b Loop_xts_enc # one more time... + +Lxts_enc_done: + ${UCMP}i $ivp,0 + beq Lxts_enc_ret + + vsrab $tmp,$tweak,$seven # next tweak value + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + vand $tmp,$tmp,$eighty7 + vxor $tweak,$tweak,$tmp + + le?vperm $tweak,$tweak,$tweak,$leperm + stvx_u $tweak,0,$ivp + +Lxts_enc_ret: + mtspr 256,r12 # restore vrsave + li r3,0 + blr + .long 0 + .byte 0,12,0x04,0,0x80,6,6,0 + .long 0 +.size .${prefix}_xts_encrypt,.-.${prefix}_xts_encrypt + +.globl .${prefix}_xts_decrypt +.align 5 +.${prefix}_xts_decrypt: + mr $inp,r3 # reassign + li r3,-1 + ${UCMP}i $len,16 + bltlr- + + lis r0,0xfff8 + mfspr r12,256 # save vrsave + li r11,0 + mtspr 256,r0 + + andi. r0,$len,15 + neg r0,r0 + andi. r0,r0,16 + sub $len,$len,r0 + + vspltisb $seven,0x07 # 0x070707..07 + le?lvsl $leperm,r11,r11 + le?vspltisb $tmp,0x0f + le?vxor $leperm,$leperm,$seven + + li $idx,15 + lvx $tweak,0,$ivp # load [unaligned] iv + lvsl $inpperm,0,$ivp + lvx $inptail,$idx,$ivp + le?vxor $inpperm,$inpperm,$tmp + vperm $tweak,$tweak,$inptail,$inpperm + + neg r11,$inp + lvsr $inpperm,0,r11 # prepare for unaligned load + lvx $inout,0,$inp + addi $inp,$inp,15 # 15 is not typo + le?vxor $inpperm,$inpperm,$tmp + + ${UCMP}i $key2,0 # key2==NULL? + beq Lxts_dec_no_key2 + + ?lvsl $keyperm,0,$key2 # prepare for unaligned key + lwz $rounds,240($key2) + srwi $rounds,$rounds,1 + subi $rounds,$rounds,1 + li $idx,16 + + lvx $rndkey0,0,$key2 + lvx $rndkey1,$idx,$key2 + addi $idx,$idx,16 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vxor $tweak,$tweak,$rndkey0 + lvx $rndkey0,$idx,$key2 + addi $idx,$idx,16 + mtctr $rounds + +Ltweak_xts_dec: + ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm + vcipher $tweak,$tweak,$rndkey1 + lvx $rndkey1,$idx,$key2 + addi $idx,$idx,16 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vcipher $tweak,$tweak,$rndkey0 + lvx $rndkey0,$idx,$key2 + addi $idx,$idx,16 + bdnz Ltweak_xts_dec + + ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm + vcipher $tweak,$tweak,$rndkey1 + lvx $rndkey1,$idx,$key2 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vcipherlast $tweak,$tweak,$rndkey0 + + li $ivp,0 # don't chain the tweak + b Lxts_dec + +Lxts_dec_no_key2: + neg $idx,$len + andi. $idx,$idx,15 + add $len,$len,$idx # in "tweak chaining" + # mode only complete + # blocks are processed +Lxts_dec: + lvx $inptail,0,$inp + addi $inp,$inp,16 + + ?lvsl $keyperm,0,$key1 # prepare for unaligned key + lwz $rounds,240($key1) + srwi $rounds,$rounds,1 + subi $rounds,$rounds,1 + li $idx,16 + + vslb $eighty7,$seven,$seven # 0x808080..80 + vor $eighty7,$eighty7,$seven # 0x878787..87 + vspltisb $tmp,1 # 0x010101..01 + vsldoi $eighty7,$eighty7,$tmp,15 # 0x870101..01 + + ${UCMP}i $len,96 + bge _aesp8_xts_decrypt6x + + lvx $rndkey0,0,$key1 + lvx $rndkey1,$idx,$key1 + addi $idx,$idx,16 + vperm $inout,$inout,$inptail,$inpperm + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vxor $inout,$inout,$tweak + vxor $inout,$inout,$rndkey0 + lvx $rndkey0,$idx,$key1 + addi $idx,$idx,16 + mtctr $rounds + + ${UCMP}i $len,16 + blt Ltail_xts_dec + be?b Loop_xts_dec + +.align 5 +Loop_xts_dec: + ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm + vncipher $inout,$inout,$rndkey1 + lvx $rndkey1,$idx,$key1 + addi $idx,$idx,16 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vncipher $inout,$inout,$rndkey0 + lvx $rndkey0,$idx,$key1 + addi $idx,$idx,16 + bdnz Loop_xts_dec + + ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm + vncipher $inout,$inout,$rndkey1 + lvx $rndkey1,$idx,$key1 + li $idx,16 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vxor $rndkey0,$rndkey0,$tweak + vncipherlast $output,$inout,$rndkey0 + + le?vperm $tmp,$output,$output,$leperm + be?nop + le?stvx_u $tmp,0,$out + be?stvx_u $output,0,$out + addi $out,$out,16 + + subic. $len,$len,16 + beq Lxts_dec_done + + vmr $inout,$inptail + lvx $inptail,0,$inp + addi $inp,$inp,16 + lvx $rndkey0,0,$key1 + lvx $rndkey1,$idx,$key1 + addi $idx,$idx,16 + + vsrab $tmp,$tweak,$seven # next tweak value + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + vand $tmp,$tmp,$eighty7 + vxor $tweak,$tweak,$tmp + + vperm $inout,$inout,$inptail,$inpperm + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vxor $inout,$inout,$tweak + vxor $inout,$inout,$rndkey0 + lvx $rndkey0,$idx,$key1 + addi $idx,$idx,16 + + mtctr $rounds + ${UCMP}i $len,16 + bge Loop_xts_dec + +Ltail_xts_dec: + vsrab $tmp,$tweak,$seven # next tweak value + vaddubm $tweak1,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + vand $tmp,$tmp,$eighty7 + vxor $tweak1,$tweak1,$tmp + + subi $inp,$inp,16 + add $inp,$inp,$len + + vxor $inout,$inout,$tweak # :-( + vxor $inout,$inout,$tweak1 # :-) + +Loop_xts_dec_short: + ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm + vncipher $inout,$inout,$rndkey1 + lvx $rndkey1,$idx,$key1 + addi $idx,$idx,16 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vncipher $inout,$inout,$rndkey0 + lvx $rndkey0,$idx,$key1 + addi $idx,$idx,16 + bdnz Loop_xts_dec_short + + ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm + vncipher $inout,$inout,$rndkey1 + lvx $rndkey1,$idx,$key1 + li $idx,16 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vxor $rndkey0,$rndkey0,$tweak1 + vncipherlast $output,$inout,$rndkey0 + + le?vperm $tmp,$output,$output,$leperm + be?nop + le?stvx_u $tmp,0,$out + be?stvx_u $output,0,$out + + vmr $inout,$inptail + lvx $inptail,0,$inp + #addi $inp,$inp,16 + lvx $rndkey0,0,$key1 + lvx $rndkey1,$idx,$key1 + addi $idx,$idx,16 + vperm $inout,$inout,$inptail,$inpperm + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + + lvsr $inpperm,0,$len # $inpperm is no longer needed + vxor $inptail,$inptail,$inptail # $inptail is no longer needed + vspltisb $tmp,-1 + vperm $inptail,$inptail,$tmp,$inpperm + vsel $inout,$inout,$output,$inptail + + vxor $rndkey0,$rndkey0,$tweak + vxor $inout,$inout,$rndkey0 + lvx $rndkey0,$idx,$key1 + addi $idx,$idx,16 + + subi r11,$out,1 + mtctr $len + li $len,16 +Loop_xts_dec_steal: + lbzu r0,1(r11) + stb r0,16(r11) + bdnz Loop_xts_dec_steal + + mtctr $rounds + b Loop_xts_dec # one more time... + +Lxts_dec_done: + ${UCMP}i $ivp,0 + beq Lxts_dec_ret + + vsrab $tmp,$tweak,$seven # next tweak value + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + vand $tmp,$tmp,$eighty7 + vxor $tweak,$tweak,$tmp + + le?vperm $tweak,$tweak,$tweak,$leperm + stvx_u $tweak,0,$ivp + +Lxts_dec_ret: + mtspr 256,r12 # restore vrsave + li r3,0 + blr + .long 0 + .byte 0,12,0x04,0,0x80,6,6,0 + .long 0 +.size .${prefix}_xts_decrypt,.-.${prefix}_xts_decrypt +___ +######################################################################### +{{ # Optimized XTS procedures # +my $key_=$key2; +my ($x00,$x10,$x20,$x30,$x40,$x50,$x60,$x70)=map("r$_",(0,3,26..31)); + $x00=0 if ($flavour =~ /osx/); +my ($in0, $in1, $in2, $in3, $in4, $in5 )=map("v$_",(0..5)); +my ($out0, $out1, $out2, $out3, $out4, $out5)=map("v$_",(7,12..16)); +my ($twk0, $twk1, $twk2, $twk3, $twk4, $twk5)=map("v$_",(17..22)); +my $rndkey0="v23"; # v24-v25 rotating buffer for first found keys + # v26-v31 last 6 round keys +my ($keyperm)=($out0); # aliases with "caller", redundant assignment +my $taillen=$x70; + +$code.=<<___; +.align 5 +_aesp8_xts_encrypt6x: + $STU $sp,-`($FRAME+21*16+6*$SIZE_T)`($sp) + mflr r11 + li r7,`$FRAME+8*16+15` + li r3,`$FRAME+8*16+31` + $PUSH r11,`$FRAME+21*16+6*$SIZE_T+$LRSAVE`($sp) + stvx v20,r7,$sp # ABI says so + addi r7,r7,32 + stvx v21,r3,$sp + addi r3,r3,32 + stvx v22,r7,$sp + addi r7,r7,32 + stvx v23,r3,$sp + addi r3,r3,32 + stvx v24,r7,$sp + addi r7,r7,32 + stvx v25,r3,$sp + addi r3,r3,32 + stvx v26,r7,$sp + addi r7,r7,32 + stvx v27,r3,$sp + addi r3,r3,32 + stvx v28,r7,$sp + addi r7,r7,32 + stvx v29,r3,$sp + addi r3,r3,32 + stvx v30,r7,$sp + stvx v31,r3,$sp + li r0,-1 + stw $vrsave,`$FRAME+21*16-4`($sp) # save vrsave + li $x10,0x10 + $PUSH r26,`$FRAME+21*16+0*$SIZE_T`($sp) + li $x20,0x20 + $PUSH r27,`$FRAME+21*16+1*$SIZE_T`($sp) + li $x30,0x30 + $PUSH r28,`$FRAME+21*16+2*$SIZE_T`($sp) + li $x40,0x40 + $PUSH r29,`$FRAME+21*16+3*$SIZE_T`($sp) + li $x50,0x50 + $PUSH r30,`$FRAME+21*16+4*$SIZE_T`($sp) + li $x60,0x60 + $PUSH r31,`$FRAME+21*16+5*$SIZE_T`($sp) + li $x70,0x70 + mtspr 256,r0 + + subi $rounds,$rounds,3 # -4 in total + + lvx $rndkey0,$x00,$key1 # load key schedule + lvx v30,$x10,$key1 + addi $key1,$key1,0x20 + lvx v31,$x00,$key1 + ?vperm $rndkey0,$rndkey0,v30,$keyperm + addi $key_,$sp,$FRAME+15 + mtctr $rounds + +Load_xts_enc_key: + ?vperm v24,v30,v31,$keyperm + lvx v30,$x10,$key1 + addi $key1,$key1,0x20 + stvx v24,$x00,$key_ # off-load round[1] + ?vperm v25,v31,v30,$keyperm + lvx v31,$x00,$key1 + stvx v25,$x10,$key_ # off-load round[2] + addi $key_,$key_,0x20 + bdnz Load_xts_enc_key + + lvx v26,$x10,$key1 + ?vperm v24,v30,v31,$keyperm + lvx v27,$x20,$key1 + stvx v24,$x00,$key_ # off-load round[3] + ?vperm v25,v31,v26,$keyperm + lvx v28,$x30,$key1 + stvx v25,$x10,$key_ # off-load round[4] + addi $key_,$sp,$FRAME+15 # rewind $key_ + ?vperm v26,v26,v27,$keyperm + lvx v29,$x40,$key1 + ?vperm v27,v27,v28,$keyperm + lvx v30,$x50,$key1 + ?vperm v28,v28,v29,$keyperm + lvx v31,$x60,$key1 + ?vperm v29,v29,v30,$keyperm + lvx $twk5,$x70,$key1 # borrow $twk5 + ?vperm v30,v30,v31,$keyperm + lvx v24,$x00,$key_ # pre-load round[1] + ?vperm v31,v31,$twk5,$keyperm + lvx v25,$x10,$key_ # pre-load round[2] + + vperm $in0,$inout,$inptail,$inpperm + subi $inp,$inp,31 # undo "caller" + vxor $twk0,$tweak,$rndkey0 + vsrab $tmp,$tweak,$seven # next tweak value + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + vand $tmp,$tmp,$eighty7 + vxor $out0,$in0,$twk0 + vxor $tweak,$tweak,$tmp + + lvx_u $in1,$x10,$inp + vxor $twk1,$tweak,$rndkey0 + vsrab $tmp,$tweak,$seven # next tweak value + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + le?vperm $in1,$in1,$in1,$leperm + vand $tmp,$tmp,$eighty7 + vxor $out1,$in1,$twk1 + vxor $tweak,$tweak,$tmp + + lvx_u $in2,$x20,$inp + andi. $taillen,$len,15 + vxor $twk2,$tweak,$rndkey0 + vsrab $tmp,$tweak,$seven # next tweak value + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + le?vperm $in2,$in2,$in2,$leperm + vand $tmp,$tmp,$eighty7 + vxor $out2,$in2,$twk2 + vxor $tweak,$tweak,$tmp + + lvx_u $in3,$x30,$inp + sub $len,$len,$taillen + vxor $twk3,$tweak,$rndkey0 + vsrab $tmp,$tweak,$seven # next tweak value + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + le?vperm $in3,$in3,$in3,$leperm + vand $tmp,$tmp,$eighty7 + vxor $out3,$in3,$twk3 + vxor $tweak,$tweak,$tmp + + lvx_u $in4,$x40,$inp + subi $len,$len,0x60 + vxor $twk4,$tweak,$rndkey0 + vsrab $tmp,$tweak,$seven # next tweak value + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + le?vperm $in4,$in4,$in4,$leperm + vand $tmp,$tmp,$eighty7 + vxor $out4,$in4,$twk4 + vxor $tweak,$tweak,$tmp + + lvx_u $in5,$x50,$inp + addi $inp,$inp,0x60 + vxor $twk5,$tweak,$rndkey0 + vsrab $tmp,$tweak,$seven # next tweak value + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + le?vperm $in5,$in5,$in5,$leperm + vand $tmp,$tmp,$eighty7 + vxor $out5,$in5,$twk5 + vxor $tweak,$tweak,$tmp + + vxor v31,v31,$rndkey0 + mtctr $rounds + b Loop_xts_enc6x + +.align 5 +Loop_xts_enc6x: + vcipher $out0,$out0,v24 + vcipher $out1,$out1,v24 + vcipher $out2,$out2,v24 + vcipher $out3,$out3,v24 + vcipher $out4,$out4,v24 + vcipher $out5,$out5,v24 + lvx v24,$x20,$key_ # round[3] + addi $key_,$key_,0x20 + + vcipher $out0,$out0,v25 + vcipher $out1,$out1,v25 + vcipher $out2,$out2,v25 + vcipher $out3,$out3,v25 + vcipher $out4,$out4,v25 + vcipher $out5,$out5,v25 + lvx v25,$x10,$key_ # round[4] + bdnz Loop_xts_enc6x + + subic $len,$len,96 # $len-=96 + vxor $in0,$twk0,v31 # xor with last round key + vcipher $out0,$out0,v24 + vcipher $out1,$out1,v24 + vsrab $tmp,$tweak,$seven # next tweak value + vxor $twk0,$tweak,$rndkey0 + vaddubm $tweak,$tweak,$tweak + vcipher $out2,$out2,v24 + vcipher $out3,$out3,v24 + vsldoi $tmp,$tmp,$tmp,15 + vcipher $out4,$out4,v24 + vcipher $out5,$out5,v24 + + subfe. r0,r0,r0 # borrow?-1:0 + vand $tmp,$tmp,$eighty7 + vcipher $out0,$out0,v25 + vcipher $out1,$out1,v25 + vxor $tweak,$tweak,$tmp + vcipher $out2,$out2,v25 + vcipher $out3,$out3,v25 + vxor $in1,$twk1,v31 + vsrab $tmp,$tweak,$seven # next tweak value + vxor $twk1,$tweak,$rndkey0 + vcipher $out4,$out4,v25 + vcipher $out5,$out5,v25 + + and r0,r0,$len + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + vcipher $out0,$out0,v26 + vcipher $out1,$out1,v26 + vand $tmp,$tmp,$eighty7 + vcipher $out2,$out2,v26 + vcipher $out3,$out3,v26 + vxor $tweak,$tweak,$tmp + vcipher $out4,$out4,v26 + vcipher $out5,$out5,v26 + + add $inp,$inp,r0 # $inp is adjusted in such + # way that at exit from the + # loop inX-in5 are loaded + # with last "words" + vxor $in2,$twk2,v31 + vsrab $tmp,$tweak,$seven # next tweak value + vxor $twk2,$tweak,$rndkey0 + vaddubm $tweak,$tweak,$tweak + vcipher $out0,$out0,v27 + vcipher $out1,$out1,v27 + vsldoi $tmp,$tmp,$tmp,15 + vcipher $out2,$out2,v27 + vcipher $out3,$out3,v27 + vand $tmp,$tmp,$eighty7 + vcipher $out4,$out4,v27 + vcipher $out5,$out5,v27 + + addi $key_,$sp,$FRAME+15 # rewind $key_ + vxor $tweak,$tweak,$tmp + vcipher $out0,$out0,v28 + vcipher $out1,$out1,v28 + vxor $in3,$twk3,v31 + vsrab $tmp,$tweak,$seven # next tweak value + vxor $twk3,$tweak,$rndkey0 + vcipher $out2,$out2,v28 + vcipher $out3,$out3,v28 + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + vcipher $out4,$out4,v28 + vcipher $out5,$out5,v28 + lvx v24,$x00,$key_ # re-pre-load round[1] + vand $tmp,$tmp,$eighty7 + + vcipher $out0,$out0,v29 + vcipher $out1,$out1,v29 + vxor $tweak,$tweak,$tmp + vcipher $out2,$out2,v29 + vcipher $out3,$out3,v29 + vxor $in4,$twk4,v31 + vsrab $tmp,$tweak,$seven # next tweak value + vxor $twk4,$tweak,$rndkey0 + vcipher $out4,$out4,v29 + vcipher $out5,$out5,v29 + lvx v25,$x10,$key_ # re-pre-load round[2] + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + + vcipher $out0,$out0,v30 + vcipher $out1,$out1,v30 + vand $tmp,$tmp,$eighty7 + vcipher $out2,$out2,v30 + vcipher $out3,$out3,v30 + vxor $tweak,$tweak,$tmp + vcipher $out4,$out4,v30 + vcipher $out5,$out5,v30 + vxor $in5,$twk5,v31 + vsrab $tmp,$tweak,$seven # next tweak value + vxor $twk5,$tweak,$rndkey0 + + vcipherlast $out0,$out0,$in0 + lvx_u $in0,$x00,$inp # load next input block + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + vcipherlast $out1,$out1,$in1 + lvx_u $in1,$x10,$inp + vcipherlast $out2,$out2,$in2 + le?vperm $in0,$in0,$in0,$leperm + lvx_u $in2,$x20,$inp + vand $tmp,$tmp,$eighty7 + vcipherlast $out3,$out3,$in3 + le?vperm $in1,$in1,$in1,$leperm + lvx_u $in3,$x30,$inp + vcipherlast $out4,$out4,$in4 + le?vperm $in2,$in2,$in2,$leperm + lvx_u $in4,$x40,$inp + vxor $tweak,$tweak,$tmp + vcipherlast $tmp,$out5,$in5 # last block might be needed + # in stealing mode + le?vperm $in3,$in3,$in3,$leperm + lvx_u $in5,$x50,$inp + addi $inp,$inp,0x60 + le?vperm $in4,$in4,$in4,$leperm + le?vperm $in5,$in5,$in5,$leperm + + le?vperm $out0,$out0,$out0,$leperm + le?vperm $out1,$out1,$out1,$leperm + stvx_u $out0,$x00,$out # store output + vxor $out0,$in0,$twk0 + le?vperm $out2,$out2,$out2,$leperm + stvx_u $out1,$x10,$out + vxor $out1,$in1,$twk1 + le?vperm $out3,$out3,$out3,$leperm + stvx_u $out2,$x20,$out + vxor $out2,$in2,$twk2 + le?vperm $out4,$out4,$out4,$leperm + stvx_u $out3,$x30,$out + vxor $out3,$in3,$twk3 + le?vperm $out5,$tmp,$tmp,$leperm + stvx_u $out4,$x40,$out + vxor $out4,$in4,$twk4 + le?stvx_u $out5,$x50,$out + be?stvx_u $tmp, $x50,$out + vxor $out5,$in5,$twk5 + addi $out,$out,0x60 + + mtctr $rounds + beq Loop_xts_enc6x # did $len-=96 borrow? + + addic. $len,$len,0x60 + beq Lxts_enc6x_zero + cmpwi $len,0x20 + blt Lxts_enc6x_one + nop + beq Lxts_enc6x_two + cmpwi $len,0x40 + blt Lxts_enc6x_three + nop + beq Lxts_enc6x_four + +Lxts_enc6x_five: + vxor $out0,$in1,$twk0 + vxor $out1,$in2,$twk1 + vxor $out2,$in3,$twk2 + vxor $out3,$in4,$twk3 + vxor $out4,$in5,$twk4 + + bl _aesp8_xts_enc5x + + le?vperm $out0,$out0,$out0,$leperm + vmr $twk0,$twk5 # unused tweak + le?vperm $out1,$out1,$out1,$leperm + stvx_u $out0,$x00,$out # store output + le?vperm $out2,$out2,$out2,$leperm + stvx_u $out1,$x10,$out + le?vperm $out3,$out3,$out3,$leperm + stvx_u $out2,$x20,$out + vxor $tmp,$out4,$twk5 # last block prep for stealing + le?vperm $out4,$out4,$out4,$leperm + stvx_u $out3,$x30,$out + stvx_u $out4,$x40,$out + addi $out,$out,0x50 + bne Lxts_enc6x_steal + b Lxts_enc6x_done + +.align 4 +Lxts_enc6x_four: + vxor $out0,$in2,$twk0 + vxor $out1,$in3,$twk1 + vxor $out2,$in4,$twk2 + vxor $out3,$in5,$twk3 + vxor $out4,$out4,$out4 + + bl _aesp8_xts_enc5x + + le?vperm $out0,$out0,$out0,$leperm + vmr $twk0,$twk4 # unused tweak + le?vperm $out1,$out1,$out1,$leperm + stvx_u $out0,$x00,$out # store output + le?vperm $out2,$out2,$out2,$leperm + stvx_u $out1,$x10,$out + vxor $tmp,$out3,$twk4 # last block prep for stealing + le?vperm $out3,$out3,$out3,$leperm + stvx_u $out2,$x20,$out + stvx_u $out3,$x30,$out + addi $out,$out,0x40 + bne Lxts_enc6x_steal + b Lxts_enc6x_done + +.align 4 +Lxts_enc6x_three: + vxor $out0,$in3,$twk0 + vxor $out1,$in4,$twk1 + vxor $out2,$in5,$twk2 + vxor $out3,$out3,$out3 + vxor $out4,$out4,$out4 + + bl _aesp8_xts_enc5x + + le?vperm $out0,$out0,$out0,$leperm + vmr $twk0,$twk3 # unused tweak + le?vperm $out1,$out1,$out1,$leperm + stvx_u $out0,$x00,$out # store output + vxor $tmp,$out2,$twk3 # last block prep for stealing + le?vperm $out2,$out2,$out2,$leperm + stvx_u $out1,$x10,$out + stvx_u $out2,$x20,$out + addi $out,$out,0x30 + bne Lxts_enc6x_steal + b Lxts_enc6x_done + +.align 4 +Lxts_enc6x_two: + vxor $out0,$in4,$twk0 + vxor $out1,$in5,$twk1 + vxor $out2,$out2,$out2 + vxor $out3,$out3,$out3 + vxor $out4,$out4,$out4 + + bl _aesp8_xts_enc5x + + le?vperm $out0,$out0,$out0,$leperm + vmr $twk0,$twk2 # unused tweak + vxor $tmp,$out1,$twk2 # last block prep for stealing + le?vperm $out1,$out1,$out1,$leperm + stvx_u $out0,$x00,$out # store output + stvx_u $out1,$x10,$out + addi $out,$out,0x20 + bne Lxts_enc6x_steal + b Lxts_enc6x_done + +.align 4 +Lxts_enc6x_one: + vxor $out0,$in5,$twk0 + nop +Loop_xts_enc1x: + vcipher $out0,$out0,v24 + lvx v24,$x20,$key_ # round[3] + addi $key_,$key_,0x20 + + vcipher $out0,$out0,v25 + lvx v25,$x10,$key_ # round[4] + bdnz Loop_xts_enc1x + + add $inp,$inp,$taillen + cmpwi $taillen,0 + vcipher $out0,$out0,v24 + + subi $inp,$inp,16 + vcipher $out0,$out0,v25 + + lvsr $inpperm,0,$taillen + vcipher $out0,$out0,v26 + + lvx_u $in0,0,$inp + vcipher $out0,$out0,v27 + + addi $key_,$sp,$FRAME+15 # rewind $key_ + vcipher $out0,$out0,v28 + lvx v24,$x00,$key_ # re-pre-load round[1] + + vcipher $out0,$out0,v29 + lvx v25,$x10,$key_ # re-pre-load round[2] + vxor $twk0,$twk0,v31 + + le?vperm $in0,$in0,$in0,$leperm + vcipher $out0,$out0,v30 + + vperm $in0,$in0,$in0,$inpperm + vcipherlast $out0,$out0,$twk0 + + vmr $twk0,$twk1 # unused tweak + vxor $tmp,$out0,$twk1 # last block prep for stealing + le?vperm $out0,$out0,$out0,$leperm + stvx_u $out0,$x00,$out # store output + addi $out,$out,0x10 + bne Lxts_enc6x_steal + b Lxts_enc6x_done + +.align 4 +Lxts_enc6x_zero: + cmpwi $taillen,0 + beq Lxts_enc6x_done + + add $inp,$inp,$taillen + subi $inp,$inp,16 + lvx_u $in0,0,$inp + lvsr $inpperm,0,$taillen # $in5 is no more + le?vperm $in0,$in0,$in0,$leperm + vperm $in0,$in0,$in0,$inpperm + vxor $tmp,$tmp,$twk0 +Lxts_enc6x_steal: + vxor $in0,$in0,$twk0 + vxor $out0,$out0,$out0 + vspltisb $out1,-1 + vperm $out0,$out0,$out1,$inpperm + vsel $out0,$in0,$tmp,$out0 # $tmp is last block, remember? + + subi r30,$out,17 + subi $out,$out,16 + mtctr $taillen +Loop_xts_enc6x_steal: + lbzu r0,1(r30) + stb r0,16(r30) + bdnz Loop_xts_enc6x_steal + + li $taillen,0 + mtctr $rounds + b Loop_xts_enc1x # one more time... + +.align 4 +Lxts_enc6x_done: + ${UCMP}i $ivp,0 + beq Lxts_enc6x_ret + + vxor $tweak,$twk0,$rndkey0 + le?vperm $tweak,$tweak,$tweak,$leperm + stvx_u $tweak,0,$ivp + +Lxts_enc6x_ret: + mtlr r11 + li r10,`$FRAME+15` + li r11,`$FRAME+31` + stvx $seven,r10,$sp # wipe copies of round keys + addi r10,r10,32 + stvx $seven,r11,$sp + addi r11,r11,32 + stvx $seven,r10,$sp + addi r10,r10,32 + stvx $seven,r11,$sp + addi r11,r11,32 + stvx $seven,r10,$sp + addi r10,r10,32 + stvx $seven,r11,$sp + addi r11,r11,32 + stvx $seven,r10,$sp + addi r10,r10,32 + stvx $seven,r11,$sp + addi r11,r11,32 + + mtspr 256,$vrsave + lvx v20,r10,$sp # ABI says so + addi r10,r10,32 + lvx v21,r11,$sp + addi r11,r11,32 + lvx v22,r10,$sp + addi r10,r10,32 + lvx v23,r11,$sp + addi r11,r11,32 + lvx v24,r10,$sp + addi r10,r10,32 + lvx v25,r11,$sp + addi r11,r11,32 + lvx v26,r10,$sp + addi r10,r10,32 + lvx v27,r11,$sp + addi r11,r11,32 + lvx v28,r10,$sp + addi r10,r10,32 + lvx v29,r11,$sp + addi r11,r11,32 + lvx v30,r10,$sp + lvx v31,r11,$sp + $POP r26,`$FRAME+21*16+0*$SIZE_T`($sp) + $POP r27,`$FRAME+21*16+1*$SIZE_T`($sp) + $POP r28,`$FRAME+21*16+2*$SIZE_T`($sp) + $POP r29,`$FRAME+21*16+3*$SIZE_T`($sp) + $POP r30,`$FRAME+21*16+4*$SIZE_T`($sp) + $POP r31,`$FRAME+21*16+5*$SIZE_T`($sp) + addi $sp,$sp,`$FRAME+21*16+6*$SIZE_T` + blr + .long 0 + .byte 0,12,0x04,1,0x80,6,6,0 + .long 0 + +.align 5 +_aesp8_xts_enc5x: + vcipher $out0,$out0,v24 + vcipher $out1,$out1,v24 + vcipher $out2,$out2,v24 + vcipher $out3,$out3,v24 + vcipher $out4,$out4,v24 + lvx v24,$x20,$key_ # round[3] + addi $key_,$key_,0x20 + + vcipher $out0,$out0,v25 + vcipher $out1,$out1,v25 + vcipher $out2,$out2,v25 + vcipher $out3,$out3,v25 + vcipher $out4,$out4,v25 + lvx v25,$x10,$key_ # round[4] + bdnz _aesp8_xts_enc5x + + add $inp,$inp,$taillen + cmpwi $taillen,0 + vcipher $out0,$out0,v24 + vcipher $out1,$out1,v24 + vcipher $out2,$out2,v24 + vcipher $out3,$out3,v24 + vcipher $out4,$out4,v24 + + subi $inp,$inp,16 + vcipher $out0,$out0,v25 + vcipher $out1,$out1,v25 + vcipher $out2,$out2,v25 + vcipher $out3,$out3,v25 + vcipher $out4,$out4,v25 + vxor $twk0,$twk0,v31 + + vcipher $out0,$out0,v26 + lvsr $inpperm,0,$taillen # $in5 is no more + vcipher $out1,$out1,v26 + vcipher $out2,$out2,v26 + vcipher $out3,$out3,v26 + vcipher $out4,$out4,v26 + vxor $in1,$twk1,v31 + + vcipher $out0,$out0,v27 + lvx_u $in0,0,$inp + vcipher $out1,$out1,v27 + vcipher $out2,$out2,v27 + vcipher $out3,$out3,v27 + vcipher $out4,$out4,v27 + vxor $in2,$twk2,v31 + + addi $key_,$sp,$FRAME+15 # rewind $key_ + vcipher $out0,$out0,v28 + vcipher $out1,$out1,v28 + vcipher $out2,$out2,v28 + vcipher $out3,$out3,v28 + vcipher $out4,$out4,v28 + lvx v24,$x00,$key_ # re-pre-load round[1] + vxor $in3,$twk3,v31 + + vcipher $out0,$out0,v29 + le?vperm $in0,$in0,$in0,$leperm + vcipher $out1,$out1,v29 + vcipher $out2,$out2,v29 + vcipher $out3,$out3,v29 + vcipher $out4,$out4,v29 + lvx v25,$x10,$key_ # re-pre-load round[2] + vxor $in4,$twk4,v31 + + vcipher $out0,$out0,v30 + vperm $in0,$in0,$in0,$inpperm + vcipher $out1,$out1,v30 + vcipher $out2,$out2,v30 + vcipher $out3,$out3,v30 + vcipher $out4,$out4,v30 + + vcipherlast $out0,$out0,$twk0 + vcipherlast $out1,$out1,$in1 + vcipherlast $out2,$out2,$in2 + vcipherlast $out3,$out3,$in3 + vcipherlast $out4,$out4,$in4 + blr + .long 0 + .byte 0,12,0x14,0,0,0,0,0 + +.align 5 +_aesp8_xts_decrypt6x: + $STU $sp,-`($FRAME+21*16+6*$SIZE_T)`($sp) + mflr r11 + li r7,`$FRAME+8*16+15` + li r3,`$FRAME+8*16+31` + $PUSH r11,`$FRAME+21*16+6*$SIZE_T+$LRSAVE`($sp) + stvx v20,r7,$sp # ABI says so + addi r7,r7,32 + stvx v21,r3,$sp + addi r3,r3,32 + stvx v22,r7,$sp + addi r7,r7,32 + stvx v23,r3,$sp + addi r3,r3,32 + stvx v24,r7,$sp + addi r7,r7,32 + stvx v25,r3,$sp + addi r3,r3,32 + stvx v26,r7,$sp + addi r7,r7,32 + stvx v27,r3,$sp + addi r3,r3,32 + stvx v28,r7,$sp + addi r7,r7,32 + stvx v29,r3,$sp + addi r3,r3,32 + stvx v30,r7,$sp + stvx v31,r3,$sp + li r0,-1 + stw $vrsave,`$FRAME+21*16-4`($sp) # save vrsave + li $x10,0x10 + $PUSH r26,`$FRAME+21*16+0*$SIZE_T`($sp) + li $x20,0x20 + $PUSH r27,`$FRAME+21*16+1*$SIZE_T`($sp) + li $x30,0x30 + $PUSH r28,`$FRAME+21*16+2*$SIZE_T`($sp) + li $x40,0x40 + $PUSH r29,`$FRAME+21*16+3*$SIZE_T`($sp) + li $x50,0x50 + $PUSH r30,`$FRAME+21*16+4*$SIZE_T`($sp) + li $x60,0x60 + $PUSH r31,`$FRAME+21*16+5*$SIZE_T`($sp) + li $x70,0x70 + mtspr 256,r0 + + subi $rounds,$rounds,3 # -4 in total + + lvx $rndkey0,$x00,$key1 # load key schedule + lvx v30,$x10,$key1 + addi $key1,$key1,0x20 + lvx v31,$x00,$key1 + ?vperm $rndkey0,$rndkey0,v30,$keyperm + addi $key_,$sp,$FRAME+15 + mtctr $rounds + +Load_xts_dec_key: + ?vperm v24,v30,v31,$keyperm + lvx v30,$x10,$key1 + addi $key1,$key1,0x20 + stvx v24,$x00,$key_ # off-load round[1] + ?vperm v25,v31,v30,$keyperm + lvx v31,$x00,$key1 + stvx v25,$x10,$key_ # off-load round[2] + addi $key_,$key_,0x20 + bdnz Load_xts_dec_key + + lvx v26,$x10,$key1 + ?vperm v24,v30,v31,$keyperm + lvx v27,$x20,$key1 + stvx v24,$x00,$key_ # off-load round[3] + ?vperm v25,v31,v26,$keyperm + lvx v28,$x30,$key1 + stvx v25,$x10,$key_ # off-load round[4] + addi $key_,$sp,$FRAME+15 # rewind $key_ + ?vperm v26,v26,v27,$keyperm + lvx v29,$x40,$key1 + ?vperm v27,v27,v28,$keyperm + lvx v30,$x50,$key1 + ?vperm v28,v28,v29,$keyperm + lvx v31,$x60,$key1 + ?vperm v29,v29,v30,$keyperm + lvx $twk5,$x70,$key1 # borrow $twk5 + ?vperm v30,v30,v31,$keyperm + lvx v24,$x00,$key_ # pre-load round[1] + ?vperm v31,v31,$twk5,$keyperm + lvx v25,$x10,$key_ # pre-load round[2] + + vperm $in0,$inout,$inptail,$inpperm + subi $inp,$inp,31 # undo "caller" + vxor $twk0,$tweak,$rndkey0 + vsrab $tmp,$tweak,$seven # next tweak value + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + vand $tmp,$tmp,$eighty7 + vxor $out0,$in0,$twk0 + vxor $tweak,$tweak,$tmp + + lvx_u $in1,$x10,$inp + vxor $twk1,$tweak,$rndkey0 + vsrab $tmp,$tweak,$seven # next tweak value + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + le?vperm $in1,$in1,$in1,$leperm + vand $tmp,$tmp,$eighty7 + vxor $out1,$in1,$twk1 + vxor $tweak,$tweak,$tmp + + lvx_u $in2,$x20,$inp + andi. $taillen,$len,15 + vxor $twk2,$tweak,$rndkey0 + vsrab $tmp,$tweak,$seven # next tweak value + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + le?vperm $in2,$in2,$in2,$leperm + vand $tmp,$tmp,$eighty7 + vxor $out2,$in2,$twk2 + vxor $tweak,$tweak,$tmp + + lvx_u $in3,$x30,$inp + sub $len,$len,$taillen + vxor $twk3,$tweak,$rndkey0 + vsrab $tmp,$tweak,$seven # next tweak value + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + le?vperm $in3,$in3,$in3,$leperm + vand $tmp,$tmp,$eighty7 + vxor $out3,$in3,$twk3 + vxor $tweak,$tweak,$tmp + + lvx_u $in4,$x40,$inp + subi $len,$len,0x60 + vxor $twk4,$tweak,$rndkey0 + vsrab $tmp,$tweak,$seven # next tweak value + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + le?vperm $in4,$in4,$in4,$leperm + vand $tmp,$tmp,$eighty7 + vxor $out4,$in4,$twk4 + vxor $tweak,$tweak,$tmp + + lvx_u $in5,$x50,$inp + addi $inp,$inp,0x60 + vxor $twk5,$tweak,$rndkey0 + vsrab $tmp,$tweak,$seven # next tweak value + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + le?vperm $in5,$in5,$in5,$leperm + vand $tmp,$tmp,$eighty7 + vxor $out5,$in5,$twk5 + vxor $tweak,$tweak,$tmp + + vxor v31,v31,$rndkey0 + mtctr $rounds + b Loop_xts_dec6x + +.align 5 +Loop_xts_dec6x: + vncipher $out0,$out0,v24 + vncipher $out1,$out1,v24 + vncipher $out2,$out2,v24 + vncipher $out3,$out3,v24 + vncipher $out4,$out4,v24 + vncipher $out5,$out5,v24 + lvx v24,$x20,$key_ # round[3] + addi $key_,$key_,0x20 + + vncipher $out0,$out0,v25 + vncipher $out1,$out1,v25 + vncipher $out2,$out2,v25 + vncipher $out3,$out3,v25 + vncipher $out4,$out4,v25 + vncipher $out5,$out5,v25 + lvx v25,$x10,$key_ # round[4] + bdnz Loop_xts_dec6x + + subic $len,$len,96 # $len-=96 + vxor $in0,$twk0,v31 # xor with last round key + vncipher $out0,$out0,v24 + vncipher $out1,$out1,v24 + vsrab $tmp,$tweak,$seven # next tweak value + vxor $twk0,$tweak,$rndkey0 + vaddubm $tweak,$tweak,$tweak + vncipher $out2,$out2,v24 + vncipher $out3,$out3,v24 + vsldoi $tmp,$tmp,$tmp,15 + vncipher $out4,$out4,v24 + vncipher $out5,$out5,v24 + + subfe. r0,r0,r0 # borrow?-1:0 + vand $tmp,$tmp,$eighty7 + vncipher $out0,$out0,v25 + vncipher $out1,$out1,v25 + vxor $tweak,$tweak,$tmp + vncipher $out2,$out2,v25 + vncipher $out3,$out3,v25 + vxor $in1,$twk1,v31 + vsrab $tmp,$tweak,$seven # next tweak value + vxor $twk1,$tweak,$rndkey0 + vncipher $out4,$out4,v25 + vncipher $out5,$out5,v25 + + and r0,r0,$len + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + vncipher $out0,$out0,v26 + vncipher $out1,$out1,v26 + vand $tmp,$tmp,$eighty7 + vncipher $out2,$out2,v26 + vncipher $out3,$out3,v26 + vxor $tweak,$tweak,$tmp + vncipher $out4,$out4,v26 + vncipher $out5,$out5,v26 + + add $inp,$inp,r0 # $inp is adjusted in such + # way that at exit from the + # loop inX-in5 are loaded + # with last "words" + vxor $in2,$twk2,v31 + vsrab $tmp,$tweak,$seven # next tweak value + vxor $twk2,$tweak,$rndkey0 + vaddubm $tweak,$tweak,$tweak + vncipher $out0,$out0,v27 + vncipher $out1,$out1,v27 + vsldoi $tmp,$tmp,$tmp,15 + vncipher $out2,$out2,v27 + vncipher $out3,$out3,v27 + vand $tmp,$tmp,$eighty7 + vncipher $out4,$out4,v27 + vncipher $out5,$out5,v27 + + addi $key_,$sp,$FRAME+15 # rewind $key_ + vxor $tweak,$tweak,$tmp + vncipher $out0,$out0,v28 + vncipher $out1,$out1,v28 + vxor $in3,$twk3,v31 + vsrab $tmp,$tweak,$seven # next tweak value + vxor $twk3,$tweak,$rndkey0 + vncipher $out2,$out2,v28 + vncipher $out3,$out3,v28 + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + vncipher $out4,$out4,v28 + vncipher $out5,$out5,v28 + lvx v24,$x00,$key_ # re-pre-load round[1] + vand $tmp,$tmp,$eighty7 + + vncipher $out0,$out0,v29 + vncipher $out1,$out1,v29 + vxor $tweak,$tweak,$tmp + vncipher $out2,$out2,v29 + vncipher $out3,$out3,v29 + vxor $in4,$twk4,v31 + vsrab $tmp,$tweak,$seven # next tweak value + vxor $twk4,$tweak,$rndkey0 + vncipher $out4,$out4,v29 + vncipher $out5,$out5,v29 + lvx v25,$x10,$key_ # re-pre-load round[2] + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + + vncipher $out0,$out0,v30 + vncipher $out1,$out1,v30 + vand $tmp,$tmp,$eighty7 + vncipher $out2,$out2,v30 + vncipher $out3,$out3,v30 + vxor $tweak,$tweak,$tmp + vncipher $out4,$out4,v30 + vncipher $out5,$out5,v30 + vxor $in5,$twk5,v31 + vsrab $tmp,$tweak,$seven # next tweak value + vxor $twk5,$tweak,$rndkey0 + + vncipherlast $out0,$out0,$in0 + lvx_u $in0,$x00,$inp # load next input block + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + vncipherlast $out1,$out1,$in1 + lvx_u $in1,$x10,$inp + vncipherlast $out2,$out2,$in2 + le?vperm $in0,$in0,$in0,$leperm + lvx_u $in2,$x20,$inp + vand $tmp,$tmp,$eighty7 + vncipherlast $out3,$out3,$in3 + le?vperm $in1,$in1,$in1,$leperm + lvx_u $in3,$x30,$inp + vncipherlast $out4,$out4,$in4 + le?vperm $in2,$in2,$in2,$leperm + lvx_u $in4,$x40,$inp + vxor $tweak,$tweak,$tmp + vncipherlast $out5,$out5,$in5 + le?vperm $in3,$in3,$in3,$leperm + lvx_u $in5,$x50,$inp + addi $inp,$inp,0x60 + le?vperm $in4,$in4,$in4,$leperm + le?vperm $in5,$in5,$in5,$leperm + + le?vperm $out0,$out0,$out0,$leperm + le?vperm $out1,$out1,$out1,$leperm + stvx_u $out0,$x00,$out # store output + vxor $out0,$in0,$twk0 + le?vperm $out2,$out2,$out2,$leperm + stvx_u $out1,$x10,$out + vxor $out1,$in1,$twk1 + le?vperm $out3,$out3,$out3,$leperm + stvx_u $out2,$x20,$out + vxor $out2,$in2,$twk2 + le?vperm $out4,$out4,$out4,$leperm + stvx_u $out3,$x30,$out + vxor $out3,$in3,$twk3 + le?vperm $out5,$out5,$out5,$leperm + stvx_u $out4,$x40,$out + vxor $out4,$in4,$twk4 + stvx_u $out5,$x50,$out + vxor $out5,$in5,$twk5 + addi $out,$out,0x60 + + mtctr $rounds + beq Loop_xts_dec6x # did $len-=96 borrow? + + addic. $len,$len,0x60 + beq Lxts_dec6x_zero + cmpwi $len,0x20 + blt Lxts_dec6x_one + nop + beq Lxts_dec6x_two + cmpwi $len,0x40 + blt Lxts_dec6x_three + nop + beq Lxts_dec6x_four + +Lxts_dec6x_five: + vxor $out0,$in1,$twk0 + vxor $out1,$in2,$twk1 + vxor $out2,$in3,$twk2 + vxor $out3,$in4,$twk3 + vxor $out4,$in5,$twk4 + + bl _aesp8_xts_dec5x + + le?vperm $out0,$out0,$out0,$leperm + vmr $twk0,$twk5 # unused tweak + vxor $twk1,$tweak,$rndkey0 + le?vperm $out1,$out1,$out1,$leperm + stvx_u $out0,$x00,$out # store output + vxor $out0,$in0,$twk1 + le?vperm $out2,$out2,$out2,$leperm + stvx_u $out1,$x10,$out + le?vperm $out3,$out3,$out3,$leperm + stvx_u $out2,$x20,$out + le?vperm $out4,$out4,$out4,$leperm + stvx_u $out3,$x30,$out + stvx_u $out4,$x40,$out + addi $out,$out,0x50 + bne Lxts_dec6x_steal + b Lxts_dec6x_done + +.align 4 +Lxts_dec6x_four: + vxor $out0,$in2,$twk0 + vxor $out1,$in3,$twk1 + vxor $out2,$in4,$twk2 + vxor $out3,$in5,$twk3 + vxor $out4,$out4,$out4 + + bl _aesp8_xts_dec5x + + le?vperm $out0,$out0,$out0,$leperm + vmr $twk0,$twk4 # unused tweak + vmr $twk1,$twk5 + le?vperm $out1,$out1,$out1,$leperm + stvx_u $out0,$x00,$out # store output + vxor $out0,$in0,$twk5 + le?vperm $out2,$out2,$out2,$leperm + stvx_u $out1,$x10,$out + le?vperm $out3,$out3,$out3,$leperm + stvx_u $out2,$x20,$out + stvx_u $out3,$x30,$out + addi $out,$out,0x40 + bne Lxts_dec6x_steal + b Lxts_dec6x_done + +.align 4 +Lxts_dec6x_three: + vxor $out0,$in3,$twk0 + vxor $out1,$in4,$twk1 + vxor $out2,$in5,$twk2 + vxor $out3,$out3,$out3 + vxor $out4,$out4,$out4 + + bl _aesp8_xts_dec5x + + le?vperm $out0,$out0,$out0,$leperm + vmr $twk0,$twk3 # unused tweak + vmr $twk1,$twk4 + le?vperm $out1,$out1,$out1,$leperm + stvx_u $out0,$x00,$out # store output + vxor $out0,$in0,$twk4 + le?vperm $out2,$out2,$out2,$leperm + stvx_u $out1,$x10,$out + stvx_u $out2,$x20,$out + addi $out,$out,0x30 + bne Lxts_dec6x_steal + b Lxts_dec6x_done + +.align 4 +Lxts_dec6x_two: + vxor $out0,$in4,$twk0 + vxor $out1,$in5,$twk1 + vxor $out2,$out2,$out2 + vxor $out3,$out3,$out3 + vxor $out4,$out4,$out4 + + bl _aesp8_xts_dec5x + + le?vperm $out0,$out0,$out0,$leperm + vmr $twk0,$twk2 # unused tweak + vmr $twk1,$twk3 + le?vperm $out1,$out1,$out1,$leperm + stvx_u $out0,$x00,$out # store output + vxor $out0,$in0,$twk3 + stvx_u $out1,$x10,$out + addi $out,$out,0x20 + bne Lxts_dec6x_steal + b Lxts_dec6x_done + +.align 4 +Lxts_dec6x_one: + vxor $out0,$in5,$twk0 + nop +Loop_xts_dec1x: + vncipher $out0,$out0,v24 + lvx v24,$x20,$key_ # round[3] + addi $key_,$key_,0x20 + + vncipher $out0,$out0,v25 + lvx v25,$x10,$key_ # round[4] + bdnz Loop_xts_dec1x + + subi r0,$taillen,1 + vncipher $out0,$out0,v24 + + andi. r0,r0,16 + cmpwi $taillen,0 + vncipher $out0,$out0,v25 + + sub $inp,$inp,r0 + vncipher $out0,$out0,v26 + + lvx_u $in0,0,$inp + vncipher $out0,$out0,v27 + + addi $key_,$sp,$FRAME+15 # rewind $key_ + vncipher $out0,$out0,v28 + lvx v24,$x00,$key_ # re-pre-load round[1] + + vncipher $out0,$out0,v29 + lvx v25,$x10,$key_ # re-pre-load round[2] + vxor $twk0,$twk0,v31 + + le?vperm $in0,$in0,$in0,$leperm + vncipher $out0,$out0,v30 + + mtctr $rounds + vncipherlast $out0,$out0,$twk0 + + vmr $twk0,$twk1 # unused tweak + vmr $twk1,$twk2 + le?vperm $out0,$out0,$out0,$leperm + stvx_u $out0,$x00,$out # store output + addi $out,$out,0x10 + vxor $out0,$in0,$twk2 + bne Lxts_dec6x_steal + b Lxts_dec6x_done + +.align 4 +Lxts_dec6x_zero: + cmpwi $taillen,0 + beq Lxts_dec6x_done + + lvx_u $in0,0,$inp + le?vperm $in0,$in0,$in0,$leperm + vxor $out0,$in0,$twk1 +Lxts_dec6x_steal: + vncipher $out0,$out0,v24 + lvx v24,$x20,$key_ # round[3] + addi $key_,$key_,0x20 + + vncipher $out0,$out0,v25 + lvx v25,$x10,$key_ # round[4] + bdnz Lxts_dec6x_steal + + add $inp,$inp,$taillen + vncipher $out0,$out0,v24 + + cmpwi $taillen,0 + vncipher $out0,$out0,v25 + + lvx_u $in0,0,$inp + vncipher $out0,$out0,v26 + + lvsr $inpperm,0,$taillen # $in5 is no more + vncipher $out0,$out0,v27 + + addi $key_,$sp,$FRAME+15 # rewind $key_ + vncipher $out0,$out0,v28 + lvx v24,$x00,$key_ # re-pre-load round[1] + + vncipher $out0,$out0,v29 + lvx v25,$x10,$key_ # re-pre-load round[2] + vxor $twk1,$twk1,v31 + + le?vperm $in0,$in0,$in0,$leperm + vncipher $out0,$out0,v30 + + vperm $in0,$in0,$in0,$inpperm + vncipherlast $tmp,$out0,$twk1 + + le?vperm $out0,$tmp,$tmp,$leperm + le?stvx_u $out0,0,$out + be?stvx_u $tmp,0,$out + + vxor $out0,$out0,$out0 + vspltisb $out1,-1 + vperm $out0,$out0,$out1,$inpperm + vsel $out0,$in0,$tmp,$out0 + vxor $out0,$out0,$twk0 + + subi r30,$out,1 + mtctr $taillen +Loop_xts_dec6x_steal: + lbzu r0,1(r30) + stb r0,16(r30) + bdnz Loop_xts_dec6x_steal + + li $taillen,0 + mtctr $rounds + b Loop_xts_dec1x # one more time... + +.align 4 +Lxts_dec6x_done: + ${UCMP}i $ivp,0 + beq Lxts_dec6x_ret + + vxor $tweak,$twk0,$rndkey0 + le?vperm $tweak,$tweak,$tweak,$leperm + stvx_u $tweak,0,$ivp + +Lxts_dec6x_ret: + mtlr r11 + li r10,`$FRAME+15` + li r11,`$FRAME+31` + stvx $seven,r10,$sp # wipe copies of round keys + addi r10,r10,32 + stvx $seven,r11,$sp + addi r11,r11,32 + stvx $seven,r10,$sp + addi r10,r10,32 + stvx $seven,r11,$sp + addi r11,r11,32 + stvx $seven,r10,$sp + addi r10,r10,32 + stvx $seven,r11,$sp + addi r11,r11,32 + stvx $seven,r10,$sp + addi r10,r10,32 + stvx $seven,r11,$sp + addi r11,r11,32 + + mtspr 256,$vrsave + lvx v20,r10,$sp # ABI says so + addi r10,r10,32 + lvx v21,r11,$sp + addi r11,r11,32 + lvx v22,r10,$sp + addi r10,r10,32 + lvx v23,r11,$sp + addi r11,r11,32 + lvx v24,r10,$sp + addi r10,r10,32 + lvx v25,r11,$sp + addi r11,r11,32 + lvx v26,r10,$sp + addi r10,r10,32 + lvx v27,r11,$sp + addi r11,r11,32 + lvx v28,r10,$sp + addi r10,r10,32 + lvx v29,r11,$sp + addi r11,r11,32 + lvx v30,r10,$sp + lvx v31,r11,$sp + $POP r26,`$FRAME+21*16+0*$SIZE_T`($sp) + $POP r27,`$FRAME+21*16+1*$SIZE_T`($sp) + $POP r28,`$FRAME+21*16+2*$SIZE_T`($sp) + $POP r29,`$FRAME+21*16+3*$SIZE_T`($sp) + $POP r30,`$FRAME+21*16+4*$SIZE_T`($sp) + $POP r31,`$FRAME+21*16+5*$SIZE_T`($sp) + addi $sp,$sp,`$FRAME+21*16+6*$SIZE_T` + blr + .long 0 + .byte 0,12,0x04,1,0x80,6,6,0 + .long 0 + +.align 5 +_aesp8_xts_dec5x: + vncipher $out0,$out0,v24 + vncipher $out1,$out1,v24 + vncipher $out2,$out2,v24 + vncipher $out3,$out3,v24 + vncipher $out4,$out4,v24 + lvx v24,$x20,$key_ # round[3] + addi $key_,$key_,0x20 + + vncipher $out0,$out0,v25 + vncipher $out1,$out1,v25 + vncipher $out2,$out2,v25 + vncipher $out3,$out3,v25 + vncipher $out4,$out4,v25 + lvx v25,$x10,$key_ # round[4] + bdnz _aesp8_xts_dec5x + + subi r0,$taillen,1 + vncipher $out0,$out0,v24 + vncipher $out1,$out1,v24 + vncipher $out2,$out2,v24 + vncipher $out3,$out3,v24 + vncipher $out4,$out4,v24 + + andi. r0,r0,16 + cmpwi $taillen,0 + vncipher $out0,$out0,v25 + vncipher $out1,$out1,v25 + vncipher $out2,$out2,v25 + vncipher $out3,$out3,v25 + vncipher $out4,$out4,v25 + vxor $twk0,$twk0,v31 + + sub $inp,$inp,r0 + vncipher $out0,$out0,v26 + vncipher $out1,$out1,v26 + vncipher $out2,$out2,v26 + vncipher $out3,$out3,v26 + vncipher $out4,$out4,v26 + vxor $in1,$twk1,v31 + + vncipher $out0,$out0,v27 + lvx_u $in0,0,$inp + vncipher $out1,$out1,v27 + vncipher $out2,$out2,v27 + vncipher $out3,$out3,v27 + vncipher $out4,$out4,v27 + vxor $in2,$twk2,v31 + + addi $key_,$sp,$FRAME+15 # rewind $key_ + vncipher $out0,$out0,v28 + vncipher $out1,$out1,v28 + vncipher $out2,$out2,v28 + vncipher $out3,$out3,v28 + vncipher $out4,$out4,v28 + lvx v24,$x00,$key_ # re-pre-load round[1] + vxor $in3,$twk3,v31 + + vncipher $out0,$out0,v29 + le?vperm $in0,$in0,$in0,$leperm + vncipher $out1,$out1,v29 + vncipher $out2,$out2,v29 + vncipher $out3,$out3,v29 + vncipher $out4,$out4,v29 + lvx v25,$x10,$key_ # re-pre-load round[2] + vxor $in4,$twk4,v31 + + vncipher $out0,$out0,v30 + vncipher $out1,$out1,v30 + vncipher $out2,$out2,v30 + vncipher $out3,$out3,v30 + vncipher $out4,$out4,v30 + + vncipherlast $out0,$out0,$twk0 + vncipherlast $out1,$out1,$in1 + vncipherlast $out2,$out2,$in2 + vncipherlast $out3,$out3,$in3 + vncipherlast $out4,$out4,$in4 + mtctr $rounds + blr + .long 0 + .byte 0,12,0x14,0,0,0,0,0 +___ +}} }}} + +my $consts=1; +foreach(split("\n",$code)) { + s/\`([^\`]*)\`/eval($1)/geo; + + # constants table endian-specific conversion + if ($consts && m/\.(long|byte)\s+(.+)\s+(\?[a-z]*)$/o) { + my $conv=$3; + my @bytes=(); + + # convert to endian-agnostic format + if ($1 eq "long") { + foreach (split(/,\s*/,$2)) { + my $l = /^0/?oct:int; + push @bytes,($l>>24)&0xff,($l>>16)&0xff,($l>>8)&0xff,$l&0xff; + } + } else { + @bytes = map(/^0/?oct:int,split(/,\s*/,$2)); + } + + # little-endian conversion + if ($flavour =~ /le$/o) { + SWITCH: for($conv) { + /\?inv/ && do { @bytes=map($_^0xf,@bytes); last; }; + /\?rev/ && do { @bytes=reverse(@bytes); last; }; + } + } + + #emit + print ".byte\t",join(',',map (sprintf("0x%02x",$_),@bytes)),"\n"; + next; + } + $consts=0 if (m/Lconsts:/o); # end of table + + # instructions prefixed with '?' are endian-specific and need + # to be adjusted accordingly... + if ($flavour =~ /le$/o) { # little-endian + s/le\?//o or + s/be\?/#be#/o or + s/\?lvsr/lvsl/o or + s/\?lvsl/lvsr/o or + s/\?(vperm\s+v[0-9]+,\s*)(v[0-9]+,\s*)(v[0-9]+,\s*)(v[0-9]+)/$1$3$2$4/o or + s/\?(vsldoi\s+v[0-9]+,\s*)(v[0-9]+,)\s*(v[0-9]+,\s*)([0-9]+)/$1$3$2 16-$4/o or + s/\?(vspltw\s+v[0-9]+,\s*)(v[0-9]+,)\s*([0-9])/$1$2 3-$3/o; + } else { # big-endian + s/le\?/#le#/o or + s/be\?//o or + s/\?([a-z]+)/$1/o; + } + + print $_,"\n"; +} + +close STDOUT; diff --git a/cbits/asm/generate.sh b/cbits/asm/generate.sh --- a/cbits/asm/generate.sh +++ b/cbits/asm/generate.sh @@ -12,6 +12,8 @@ # arm/sha1-armv8.pl arm/sha512-armv8.pl # arm/keccak1600-armv8.pl arm/arm-xlate.pl # arm/arm_arch.h +# ppc/aesp8-ppc.pl ppc/ghashp8-ppc.pl +# ppc/ppc-xlate.pl # # The .pl files are the generator, not the product: each one emits # assembly for a given "flavour", which is the calling convention and the @@ -150,4 +152,25 @@ .section .note.GNU-stack,"",%progbits NOTE +done + +# POWER8's AES and GHASH, little-endian only. The generator emits a +# big-endian flavour from the same source, and crypton does not build for +# big-endian POWER: nothing here has been able to run that code, and an AES +# path that has never been executed is not one to check in. Adding +# "linux64" to the list below is what it would take. +# +# These two read no capability word of their own -- unlike the ARM and x86-64 +# modules above -- so the entry points are all that is renamed. +for flavour in linux64le; do + perl aesp8-ppc.pl $flavour tmp-$flavour.S + sed -e 's/\baes_p8_/crypton_aes_p8_/g' \ + tmp-$flavour.S > aesp8-ppc-$flavour.S + + perl ghashp8-ppc.pl $flavour tmp-$flavour.S + sed -e 's/\bgcm_init_p8/crypton_gcm_init_p8/g' \ + -e 's/\bgcm_gmult_p8/crypton_gcm_gmult_p8/g' \ + -e 's/\bgcm_ghash_p8/crypton_gcm_ghash_p8/g' \ + tmp-$flavour.S > ghashp8-ppc-$flavour.S + rm -f tmp-$flavour.S done diff --git a/cbits/asm/ghashp8-ppc-linux64le.S b/cbits/asm/ghashp8-ppc-linux64le.S new file mode 100644 --- /dev/null +++ b/cbits/asm/ghashp8-ppc-linux64le.S @@ -0,0 +1,575 @@ +.machine "any" + +.abiversion 2 +.text + +.globl crypton_gcm_init_p8 +.type crypton_gcm_init_p8,@function +.align 5 +crypton_gcm_init_p8: +.localentry crypton_gcm_init_p8,0 + + li 0,-4096 + li 8,0x10 + li 12,-1 + li 9,0x20 + or 0,0,0 + li 10,0x30 + .long 0x7D202699 + + vspltisb 8,-16 + vspltisb 5,1 + vaddubm 8,8,8 + vxor 4,4,4 + vor 8,8,5 + vsldoi 8,8,4,15 + vsldoi 6,4,5,1 + vaddubm 8,8,8 + vspltisb 7,7 + vor 8,8,6 + vspltb 6,9,0 + vsl 9,9,5 + vsrab 6,6,7 + vand 6,6,8 + vxor 3,9,6 + + vsldoi 9,3,3,8 + vsldoi 8,4,8,8 + vsldoi 11,4,9,8 + vsldoi 10,9,4,8 + + .long 0x7D001F99 + .long 0x7D681F99 + li 8,0x40 + .long 0x7D291F99 + li 9,0x50 + .long 0x7D4A1F99 + li 10,0x60 + + .long 0x10035CC8 + .long 0x10234CC8 + .long 0x104354C8 + + .long 0x10E044C8 + + vsldoi 5,1,4,8 + vsldoi 6,4,1,8 + vxor 0,0,5 + vxor 2,2,6 + + vsldoi 0,0,0,8 + vxor 0,0,7 + + vsldoi 6,0,0,8 + .long 0x100044C8 + vxor 6,6,2 + vxor 16,0,6 + + vsldoi 17,16,16,8 + vsldoi 19,4,17,8 + vsldoi 18,17,4,8 + + .long 0x7E681F99 + li 8,0x70 + .long 0x7E291F99 + li 9,0x80 + .long 0x7E4A1F99 + li 10,0x90 + .long 0x10039CC8 + .long 0x11B09CC8 + .long 0x10238CC8 + .long 0x11D08CC8 + .long 0x104394C8 + .long 0x11F094C8 + + .long 0x10E044C8 + .long 0x114D44C8 + + vsldoi 5,1,4,8 + vsldoi 6,4,1,8 + vsldoi 11,14,4,8 + vsldoi 9,4,14,8 + vxor 0,0,5 + vxor 2,2,6 + vxor 13,13,11 + vxor 15,15,9 + + vsldoi 0,0,0,8 + vsldoi 13,13,13,8 + vxor 0,0,7 + vxor 13,13,10 + + vsldoi 6,0,0,8 + vsldoi 9,13,13,8 + .long 0x100044C8 + .long 0x11AD44C8 + vxor 6,6,2 + vxor 9,9,15 + vxor 0,0,6 + vxor 13,13,9 + + vsldoi 9,0,0,8 + vsldoi 17,13,13,8 + vsldoi 11,4,9,8 + vsldoi 10,9,4,8 + vsldoi 19,4,17,8 + vsldoi 18,17,4,8 + + .long 0x7D681F99 + li 8,0xa0 + .long 0x7D291F99 + li 9,0xb0 + .long 0x7D4A1F99 + li 10,0xc0 + .long 0x7E681F99 + .long 0x7E291F99 + .long 0x7E4A1F99 + + or 12,12,12 + blr +.long 0 +.byte 0,12,0x14,0,0,0,2,0 +.long 0 +.size crypton_gcm_init_p8,.-crypton_gcm_init_p8 +.globl crypton_gcm_gmult_p8 +.type crypton_gcm_gmult_p8,@function +.align 5 +crypton_gcm_gmult_p8: +.localentry crypton_gcm_gmult_p8,0 + + lis 0,0xfff8 + li 8,0x10 + li 12,-1 + li 9,0x20 + or 0,0,0 + li 10,0x30 + .long 0x7C601E99 + + .long 0x7D682699 + lvsl 12,0,0 + .long 0x7D292699 + vspltisb 5,0x07 + .long 0x7D4A2699 + vxor 12,12,5 + .long 0x7D002699 + vperm 3,3,3,12 + vxor 4,4,4 + + .long 0x10035CC8 + .long 0x10234CC8 + .long 0x104354C8 + + .long 0x10E044C8 + + vsldoi 5,1,4,8 + vsldoi 6,4,1,8 + vxor 0,0,5 + vxor 2,2,6 + + vsldoi 0,0,0,8 + vxor 0,0,7 + + vsldoi 6,0,0,8 + .long 0x100044C8 + vxor 6,6,2 + vxor 0,0,6 + + vperm 0,0,0,12 + .long 0x7C001F99 + + or 12,12,12 + blr +.long 0 +.byte 0,12,0x14,0,0,0,2,0 +.long 0 +.size crypton_gcm_gmult_p8,.-crypton_gcm_gmult_p8 + +.globl crypton_gcm_ghash_p8 +.type crypton_gcm_ghash_p8,@function +.align 5 +crypton_gcm_ghash_p8: +.localentry crypton_gcm_ghash_p8,0 + + li 0,-4096 + li 8,0x10 + li 12,-1 + li 9,0x20 + or 0,0,0 + li 10,0x30 + .long 0x7C001E99 + + .long 0x7D682699 + li 8,0x40 + lvsl 12,0,0 + .long 0x7D292699 + li 9,0x50 + vspltisb 5,0x07 + .long 0x7D4A2699 + li 10,0x60 + vxor 12,12,5 + .long 0x7D002699 + vperm 0,0,0,12 + vxor 4,4,4 + + cmpldi 6,64 + bge .Lgcm_ghash_p8_4x + + .long 0x7C602E99 + addi 5,5,16 + subic. 6,6,16 + vperm 3,3,3,12 + vxor 3,3,0 + beq .Lshort + + .long 0x7E682699 + li 8,16 + .long 0x7E292699 + add 9,5,6 + .long 0x7E4A2699 + + +.align 5 +.Loop_2x: + .long 0x7E002E99 + vperm 16,16,16,12 + + subic 6,6,32 + .long 0x10039CC8 + .long 0x11B05CC8 + subfe 0,0,0 + .long 0x10238CC8 + .long 0x11D04CC8 + and 0,0,6 + .long 0x104394C8 + .long 0x11F054C8 + add 5,5,0 + + vxor 0,0,13 + vxor 1,1,14 + + .long 0x10E044C8 + + vsldoi 5,1,4,8 + vsldoi 6,4,1,8 + vxor 2,2,15 + vxor 0,0,5 + vxor 2,2,6 + + vsldoi 0,0,0,8 + vxor 0,0,7 + .long 0x7C682E99 + addi 5,5,32 + + vsldoi 6,0,0,8 + .long 0x100044C8 + vperm 3,3,3,12 + vxor 6,6,2 + vxor 3,3,6 + vxor 3,3,0 + cmpld 9,5 + bgt .Loop_2x + + cmplwi 6,0 + bne .Leven + +.Lshort: + .long 0x10035CC8 + .long 0x10234CC8 + .long 0x104354C8 + + .long 0x10E044C8 + + vsldoi 5,1,4,8 + vsldoi 6,4,1,8 + vxor 0,0,5 + vxor 2,2,6 + + vsldoi 0,0,0,8 + vxor 0,0,7 + + vsldoi 6,0,0,8 + .long 0x100044C8 + vxor 6,6,2 + +.Leven: + vxor 0,0,6 + vperm 0,0,0,12 + .long 0x7C001F99 + + or 12,12,12 + blr +.long 0 +.byte 0,12,0x14,0,0,0,4,0 +.long 0 +.align 5 +.crypton_gcm_ghash_p8_4x: +.Lgcm_ghash_p8_4x: + stdu 1,-256(1) + li 10,63 + li 11,79 + stvx 20,10,1 + addi 10,10,32 + stvx 21,11,1 + addi 11,11,32 + stvx 22,10,1 + addi 10,10,32 + stvx 23,11,1 + addi 11,11,32 + stvx 24,10,1 + addi 10,10,32 + stvx 25,11,1 + addi 11,11,32 + stvx 26,10,1 + addi 10,10,32 + stvx 27,11,1 + addi 11,11,32 + stvx 28,10,1 + addi 10,10,32 + stvx 29,11,1 + addi 11,11,32 + stvx 30,10,1 + li 10,0x60 + stvx 31,11,1 + li 0,-1 + stw 12,252(1) + or 0,0,0 + + lvsl 5,0,8 + + li 8,0x70 + .long 0x7E292699 + li 9,0x80 + vspltisb 6,8 + + li 10,0x90 + .long 0x7EE82699 + li 8,0xa0 + .long 0x7F092699 + li 9,0xb0 + .long 0x7F2A2699 + li 10,0xc0 + .long 0x7FA82699 + li 8,0x10 + .long 0x7FC92699 + li 9,0x20 + .long 0x7FEA2699 + li 10,0x30 + + vsldoi 7,4,6,8 + vaddubm 18,5,7 + vaddubm 19,6,18 + + srdi 6,6,4 + + .long 0x7C602E99 + .long 0x7E082E99 + subic. 6,6,8 + .long 0x7EC92E99 + .long 0x7F8A2E99 + addi 5,5,0x40 + vperm 3,3,3,12 + vperm 16,16,16,12 + vperm 22,22,22,12 + vperm 28,28,28,12 + + vxor 2,3,0 + + .long 0x11B0BCC8 + .long 0x11D0C4C8 + .long 0x11F0CCC8 + + vperm 11,17,9,18 + vperm 5,22,28,19 + vperm 10,17,9,19 + vperm 6,22,28,18 + .long 0x12B68CC8 + .long 0x12855CC8 + .long 0x137C4CC8 + .long 0x134654C8 + + vxor 21,21,14 + vxor 20,20,13 + vxor 27,27,21 + vxor 26,26,15 + + blt .Ltail_4x + +.Loop_4x: + .long 0x7C602E99 + .long 0x7E082E99 + subic. 6,6,4 + .long 0x7EC92E99 + .long 0x7F8A2E99 + addi 5,5,0x40 + vperm 16,16,16,12 + vperm 22,22,22,12 + vperm 28,28,28,12 + vperm 3,3,3,12 + + .long 0x1002ECC8 + .long 0x1022F4C8 + .long 0x1042FCC8 + .long 0x11B0BCC8 + .long 0x11D0C4C8 + .long 0x11F0CCC8 + + vxor 0,0,20 + vxor 1,1,27 + vxor 2,2,26 + vperm 5,22,28,19 + vperm 6,22,28,18 + + .long 0x10E044C8 + .long 0x12855CC8 + .long 0x134654C8 + + vsldoi 5,1,4,8 + vsldoi 6,4,1,8 + vxor 0,0,5 + vxor 2,2,6 + + vsldoi 0,0,0,8 + vxor 0,0,7 + + vsldoi 6,0,0,8 + .long 0x12B68CC8 + .long 0x137C4CC8 + .long 0x100044C8 + + vxor 20,20,13 + vxor 26,26,15 + vxor 2,2,3 + vxor 21,21,14 + vxor 2,2,6 + vxor 27,27,21 + vxor 2,2,0 + bge .Loop_4x + +.Ltail_4x: + .long 0x1002ECC8 + .long 0x1022F4C8 + .long 0x1042FCC8 + + vxor 0,0,20 + vxor 1,1,27 + + .long 0x10E044C8 + + vsldoi 5,1,4,8 + vsldoi 6,4,1,8 + vxor 2,2,26 + vxor 0,0,5 + vxor 2,2,6 + + vsldoi 0,0,0,8 + vxor 0,0,7 + + vsldoi 6,0,0,8 + .long 0x100044C8 + vxor 6,6,2 + vxor 0,0,6 + + addic. 6,6,4 + beq .Ldone_4x + + .long 0x7C602E99 + cmpldi 6,2 + li 6,-4 + blt .Lone + .long 0x7E082E99 + beq .Ltwo + +.Lthree: + .long 0x7EC92E99 + vperm 3,3,3,12 + vperm 16,16,16,12 + vperm 22,22,22,12 + + vxor 2,3,0 + vor 29,23,23 + vor 30,24,24 + vor 31,25,25 + + vperm 5,16,22,19 + vperm 6,16,22,18 + .long 0x12B08CC8 + .long 0x13764CC8 + .long 0x12855CC8 + .long 0x134654C8 + + vxor 27,27,21 + b .Ltail_4x + +.align 4 +.Ltwo: + vperm 3,3,3,12 + vperm 16,16,16,12 + + vxor 2,3,0 + vperm 5,4,16,19 + vperm 6,4,16,18 + + vsldoi 29,4,17,8 + vor 30,17,17 + vsldoi 31,17,4,8 + + .long 0x12855CC8 + .long 0x13704CC8 + .long 0x134654C8 + + b .Ltail_4x + +.align 4 +.Lone: + vperm 3,3,3,12 + + vsldoi 29,4,9,8 + vor 30,9,9 + vsldoi 31,9,4,8 + + vxor 2,3,0 + vxor 20,20,20 + vxor 27,27,27 + vxor 26,26,26 + + b .Ltail_4x + +.Ldone_4x: + vperm 0,0,0,12 + .long 0x7C001F99 + + li 10,63 + li 11,79 + or 12,12,12 + lvx 20,10,1 + addi 10,10,32 + lvx 21,11,1 + addi 11,11,32 + lvx 22,10,1 + addi 10,10,32 + lvx 23,11,1 + addi 11,11,32 + lvx 24,10,1 + addi 10,10,32 + lvx 25,11,1 + addi 11,11,32 + lvx 26,10,1 + addi 10,10,32 + lvx 27,11,1 + addi 11,11,32 + lvx 28,10,1 + addi 10,10,32 + lvx 29,11,1 + addi 11,11,32 + lvx 30,10,1 + lvx 31,11,1 + addi 1,1,256 + blr +.long 0 +.byte 0,12,0x04,0,0x80,0,4,0 +.long 0 +.size crypton_gcm_ghash_p8,.-crypton_gcm_ghash_p8 + +.byte 71,72,65,83,72,32,102,111,114,32,80,111,119,101,114,73,83,65,32,50,46,48,55,44,67,82,89,80,84,79,71,65,77,83,32,98,121,32,60,97,112,112,114,111,64,111,112,101,110,115,115,108,46,111,114,103,62,0 +.align 2 +.align 2 diff --git a/cbits/asm/ghashp8-ppc.pl b/cbits/asm/ghashp8-ppc.pl new file mode 100644 --- /dev/null +++ b/cbits/asm/ghashp8-ppc.pl @@ -0,0 +1,664 @@ +#!/usr/bin/env perl +# +# ==================================================================== +# Written by Andy Polyakov, @dot-asm, initially for use in the OpenSSL +# project. The module is dual licensed under OpenSSL and CRYPTOGAMS +# licenses depending on where you obtain it. For further details see +# https://github.com/dot-asm/cryptogams/. +# ==================================================================== +# +# GHASH for for PowerISA v2.07. +# +# July 2014 +# +# Accurate performance measurements are problematic, because it's +# always virtualized setup with possibly throttled processor. +# Relative comparison is therefore more informative. This initial +# version is ~2.1x slower than hardware-assisted AES-128-CTR, ~12x +# faster than "4-bit" integer-only compiler-generated 64-bit code. +# "Initial version" means that there is room for further improvement. + +# May 2016 +# +# 2x aggregated reduction improves performance by 50% (resulting +# performance on POWER8 is 1 cycle per processed byte), and 4x +# aggregated reduction - by 170% or 2.7x (resulting in 0.55 cpb). +# POWER9 delivers 0.51 cpb. + +$flavour=shift; +$output =shift; + +if ($flavour =~ /64/) { + $SIZE_T=8; + $LRSAVE=2*$SIZE_T; + $STU="stdu"; + $POP="ld"; + $PUSH="std"; + $UCMP="cmpld"; + $SHRI="srdi"; +} elsif ($flavour =~ /32/) { + $SIZE_T=4; + $LRSAVE=$SIZE_T; + $STU="stwu"; + $POP="lwz"; + $PUSH="stw"; + $UCMP="cmplw"; + $SHRI="srwi"; +} else { die "nonsense $flavour"; } + +$sp="r1"; +$FRAME=6*$SIZE_T+13*16; # 13*16 is for v20-v31 offload + +$0 =~ m/(.*[\/\\])[^\/\\]+$/; $dir=$1; +( $xlate="${dir}ppc-xlate.pl" and -f $xlate ) or +( $xlate="${dir}../../perlasm/ppc-xlate.pl" and -f $xlate) or +die "can't locate ppc-xlate.pl"; + +open STDOUT,"| $^X $xlate $flavour $output" || die "can't call $xlate: $!"; + +my ($Xip,$Htbl,$inp,$len)=map("r$_",(3..6)); # argument block + +my ($Xl,$Xm,$Xh,$IN)=map("v$_",(0..3)); +my ($zero,$t0,$t1,$t2,$xC2,$H,$Hh,$Hl,$lemask)=map("v$_",(4..12)); +my ($Xl1,$Xm1,$Xh1,$IN1,$H2,$H2h,$H2l)=map("v$_",(13..19)); +my $vrsave="r12"; + +$code=<<___; +.machine "any" + +.text + +.globl .gcm_init_p8 +.align 5 +.gcm_init_p8: + li r0,-4096 + li r8,0x10 + mfspr $vrsave,256 + li r9,0x20 + mtspr 256,r0 + li r10,0x30 + lvx_u $H,0,r4 # load H + + vspltisb $xC2,-16 # 0xf0 + vspltisb $t0,1 # one + vaddubm $xC2,$xC2,$xC2 # 0xe0 + vxor $zero,$zero,$zero + vor $xC2,$xC2,$t0 # 0xe1 + vsldoi $xC2,$xC2,$zero,15 # 0xe1... + vsldoi $t1,$zero,$t0,1 # ...1 + vaddubm $xC2,$xC2,$xC2 # 0xc2... + vspltisb $t2,7 + vor $xC2,$xC2,$t1 # 0xc2....01 + vspltb $t1,$H,0 # most significant byte + vsl $H,$H,$t0 # H<<=1 + vsrab $t1,$t1,$t2 # broadcast carry bit + vand $t1,$t1,$xC2 + vxor $IN,$H,$t1 # twisted H + + vsldoi $H,$IN,$IN,8 # twist even more ... + vsldoi $xC2,$zero,$xC2,8 # 0xc2.0 + vsldoi $Hl,$zero,$H,8 # ... and split + vsldoi $Hh,$H,$zero,8 + + stvx_u $xC2,0,r3 # save pre-computed table + stvx_u $Hl,r8,r3 + li r8,0x40 + stvx_u $H, r9,r3 + li r9,0x50 + stvx_u $Hh,r10,r3 + li r10,0x60 + + vpmsumd $Xl,$IN,$Hl # H.lo·H.lo + vpmsumd $Xm,$IN,$H # H.hi·H.lo+H.lo·H.hi + vpmsumd $Xh,$IN,$Hh # H.hi·H.hi + + vpmsumd $t2,$Xl,$xC2 # 1st reduction phase + + vsldoi $t0,$Xm,$zero,8 + vsldoi $t1,$zero,$Xm,8 + vxor $Xl,$Xl,$t0 + vxor $Xh,$Xh,$t1 + + vsldoi $Xl,$Xl,$Xl,8 + vxor $Xl,$Xl,$t2 + + vsldoi $t1,$Xl,$Xl,8 # 2nd reduction phase + vpmsumd $Xl,$Xl,$xC2 + vxor $t1,$t1,$Xh + vxor $IN1,$Xl,$t1 + + vsldoi $H2,$IN1,$IN1,8 + vsldoi $H2l,$zero,$H2,8 + vsldoi $H2h,$H2,$zero,8 + + stvx_u $H2l,r8,r3 # save H^2 + li r8,0x70 + stvx_u $H2,r9,r3 + li r9,0x80 + stvx_u $H2h,r10,r3 + li r10,0x90 +___ +{ +my ($t4,$t5,$t6) = ($Hl,$H,$Hh); +$code.=<<___; + vpmsumd $Xl,$IN,$H2l # H.lo·H^2.lo + vpmsumd $Xl1,$IN1,$H2l # H^2.lo·H^2.lo + vpmsumd $Xm,$IN,$H2 # H.hi·H^2.lo+H.lo·H^2.hi + vpmsumd $Xm1,$IN1,$H2 # H^2.hi·H^2.lo+H^2.lo·H^2.hi + vpmsumd $Xh,$IN,$H2h # H.hi·H^2.hi + vpmsumd $Xh1,$IN1,$H2h # H^2.hi·H^2.hi + + vpmsumd $t2,$Xl,$xC2 # 1st reduction phase + vpmsumd $t6,$Xl1,$xC2 # 1st reduction phase + + vsldoi $t0,$Xm,$zero,8 + vsldoi $t1,$zero,$Xm,8 + vsldoi $t4,$Xm1,$zero,8 + vsldoi $t5,$zero,$Xm1,8 + vxor $Xl,$Xl,$t0 + vxor $Xh,$Xh,$t1 + vxor $Xl1,$Xl1,$t4 + vxor $Xh1,$Xh1,$t5 + + vsldoi $Xl,$Xl,$Xl,8 + vsldoi $Xl1,$Xl1,$Xl1,8 + vxor $Xl,$Xl,$t2 + vxor $Xl1,$Xl1,$t6 + + vsldoi $t1,$Xl,$Xl,8 # 2nd reduction phase + vsldoi $t5,$Xl1,$Xl1,8 # 2nd reduction phase + vpmsumd $Xl,$Xl,$xC2 + vpmsumd $Xl1,$Xl1,$xC2 + vxor $t1,$t1,$Xh + vxor $t5,$t5,$Xh1 + vxor $Xl,$Xl,$t1 + vxor $Xl1,$Xl1,$t5 + + vsldoi $H,$Xl,$Xl,8 + vsldoi $H2,$Xl1,$Xl1,8 + vsldoi $Hl,$zero,$H,8 + vsldoi $Hh,$H,$zero,8 + vsldoi $H2l,$zero,$H2,8 + vsldoi $H2h,$H2,$zero,8 + + stvx_u $Hl,r8,r3 # save H^3 + li r8,0xa0 + stvx_u $H,r9,r3 + li r9,0xb0 + stvx_u $Hh,r10,r3 + li r10,0xc0 + stvx_u $H2l,r8,r3 # save H^4 + stvx_u $H2,r9,r3 + stvx_u $H2h,r10,r3 + + mtspr 256,$vrsave + blr + .long 0 + .byte 0,12,0x14,0,0,0,2,0 + .long 0 +.size .gcm_init_p8,.-.gcm_init_p8 +___ +} +$code.=<<___; +.globl .gcm_gmult_p8 +.align 5 +.gcm_gmult_p8: + lis r0,0xfff8 + li r8,0x10 + mfspr $vrsave,256 + li r9,0x20 + mtspr 256,r0 + li r10,0x30 + lvx_u $IN,0,$Xip # load Xi + + lvx_u $Hl,r8,$Htbl # load pre-computed table + le?lvsl $lemask,r0,r0 + lvx_u $H, r9,$Htbl + le?vspltisb $t0,0x07 + lvx_u $Hh,r10,$Htbl + le?vxor $lemask,$lemask,$t0 + lvx_u $xC2,0,$Htbl + le?vperm $IN,$IN,$IN,$lemask + vxor $zero,$zero,$zero + + vpmsumd $Xl,$IN,$Hl # H.lo·Xi.lo + vpmsumd $Xm,$IN,$H # H.hi·Xi.lo+H.lo·Xi.hi + vpmsumd $Xh,$IN,$Hh # H.hi·Xi.hi + + vpmsumd $t2,$Xl,$xC2 # 1st reduction phase + + vsldoi $t0,$Xm,$zero,8 + vsldoi $t1,$zero,$Xm,8 + vxor $Xl,$Xl,$t0 + vxor $Xh,$Xh,$t1 + + vsldoi $Xl,$Xl,$Xl,8 + vxor $Xl,$Xl,$t2 + + vsldoi $t1,$Xl,$Xl,8 # 2nd reduction phase + vpmsumd $Xl,$Xl,$xC2 + vxor $t1,$t1,$Xh + vxor $Xl,$Xl,$t1 + + le?vperm $Xl,$Xl,$Xl,$lemask + stvx_u $Xl,0,$Xip # write out Xi + + mtspr 256,$vrsave + blr + .long 0 + .byte 0,12,0x14,0,0,0,2,0 + .long 0 +.size .gcm_gmult_p8,.-.gcm_gmult_p8 + +.globl .gcm_ghash_p8 +.align 5 +.gcm_ghash_p8: + li r0,-4096 + li r8,0x10 + mfspr $vrsave,256 + li r9,0x20 + mtspr 256,r0 + li r10,0x30 + lvx_u $Xl,0,$Xip # load Xi + + lvx_u $Hl,r8,$Htbl # load pre-computed table + li r8,0x40 + le?lvsl $lemask,r0,r0 + lvx_u $H, r9,$Htbl + li r9,0x50 + le?vspltisb $t0,0x07 + lvx_u $Hh,r10,$Htbl + li r10,0x60 + le?vxor $lemask,$lemask,$t0 + lvx_u $xC2,0,$Htbl + le?vperm $Xl,$Xl,$Xl,$lemask + vxor $zero,$zero,$zero + + ${UCMP}i $len,64 + bge Lgcm_ghash_p8_4x + + lvx_u $IN,0,$inp + addi $inp,$inp,16 + subic. $len,$len,16 + le?vperm $IN,$IN,$IN,$lemask + vxor $IN,$IN,$Xl + beq Lshort + + lvx_u $H2l,r8,$Htbl # load H^2 + li r8,16 + lvx_u $H2, r9,$Htbl + add r9,$inp,$len # end of input + lvx_u $H2h,r10,$Htbl + be?b Loop_2x + +.align 5 +Loop_2x: + lvx_u $IN1,0,$inp + le?vperm $IN1,$IN1,$IN1,$lemask + + subic $len,$len,32 + vpmsumd $Xl,$IN,$H2l # H^2.lo·Xi.lo + vpmsumd $Xl1,$IN1,$Hl # H.lo·Xi+1.lo + subfe r0,r0,r0 # borrow?-1:0 + vpmsumd $Xm,$IN,$H2 # H^2.hi·Xi.lo+H^2.lo·Xi.hi + vpmsumd $Xm1,$IN1,$H # H.hi·Xi+1.lo+H.lo·Xi+1.hi + and r0,r0,$len + vpmsumd $Xh,$IN,$H2h # H^2.hi·Xi.hi + vpmsumd $Xh1,$IN1,$Hh # H.hi·Xi+1.hi + add $inp,$inp,r0 + + vxor $Xl,$Xl,$Xl1 + vxor $Xm,$Xm,$Xm1 + + vpmsumd $t2,$Xl,$xC2 # 1st reduction phase + + vsldoi $t0,$Xm,$zero,8 + vsldoi $t1,$zero,$Xm,8 + vxor $Xh,$Xh,$Xh1 + vxor $Xl,$Xl,$t0 + vxor $Xh,$Xh,$t1 + + vsldoi $Xl,$Xl,$Xl,8 + vxor $Xl,$Xl,$t2 + lvx_u $IN,r8,$inp + addi $inp,$inp,32 + + vsldoi $t1,$Xl,$Xl,8 # 2nd reduction phase + vpmsumd $Xl,$Xl,$xC2 + le?vperm $IN,$IN,$IN,$lemask + vxor $t1,$t1,$Xh + vxor $IN,$IN,$t1 + vxor $IN,$IN,$Xl + $UCMP r9,$inp + bgt Loop_2x # done yet? + + cmplwi $len,0 + bne Leven + +Lshort: + vpmsumd $Xl,$IN,$Hl # H.lo·Xi.lo + vpmsumd $Xm,$IN,$H # H.hi·Xi.lo+H.lo·Xi.hi + vpmsumd $Xh,$IN,$Hh # H.hi·Xi.hi + + vpmsumd $t2,$Xl,$xC2 # 1st reduction phase + + vsldoi $t0,$Xm,$zero,8 + vsldoi $t1,$zero,$Xm,8 + vxor $Xl,$Xl,$t0 + vxor $Xh,$Xh,$t1 + + vsldoi $Xl,$Xl,$Xl,8 + vxor $Xl,$Xl,$t2 + + vsldoi $t1,$Xl,$Xl,8 # 2nd reduction phase + vpmsumd $Xl,$Xl,$xC2 + vxor $t1,$t1,$Xh + +Leven: + vxor $Xl,$Xl,$t1 + le?vperm $Xl,$Xl,$Xl,$lemask + stvx_u $Xl,0,$Xip # write out Xi + + mtspr 256,$vrsave + blr + .long 0 + .byte 0,12,0x14,0,0,0,4,0 + .long 0 +___ +{ +my ($Xl3,$Xm2,$IN2,$H3l,$H3,$H3h, + $Xh3,$Xm3,$IN3,$H4l,$H4,$H4h) = map("v$_",(20..31)); +my $IN0=$IN; +my ($H21l,$H21h,$loperm,$hiperm) = ($Hl,$Hh,$H2l,$H2h); + +$code.=<<___; +.align 5 +.gcm_ghash_p8_4x: +Lgcm_ghash_p8_4x: + $STU $sp,-$FRAME($sp) + li r10,`15+6*$SIZE_T` + li r11,`31+6*$SIZE_T` + stvx v20,r10,$sp + addi r10,r10,32 + stvx v21,r11,$sp + addi r11,r11,32 + stvx v22,r10,$sp + addi r10,r10,32 + stvx v23,r11,$sp + addi r11,r11,32 + stvx v24,r10,$sp + addi r10,r10,32 + stvx v25,r11,$sp + addi r11,r11,32 + stvx v26,r10,$sp + addi r10,r10,32 + stvx v27,r11,$sp + addi r11,r11,32 + stvx v28,r10,$sp + addi r10,r10,32 + stvx v29,r11,$sp + addi r11,r11,32 + stvx v30,r10,$sp + li r10,0x60 + stvx v31,r11,$sp + li r0,-1 + stw $vrsave,`$FRAME-4`($sp) # save vrsave + mtspr 256,r0 # preserve all AltiVec registers + + lvsl $t0,0,r8 # 0x0001..0e0f + #lvx_u $H2l,r8,$Htbl # load H^2 + li r8,0x70 + lvx_u $H2, r9,$Htbl + li r9,0x80 + vspltisb $t1,8 # 0x0808..0808 + #lvx_u $H2h,r10,$Htbl + li r10,0x90 + lvx_u $H3l,r8,$Htbl # load H^3 + li r8,0xa0 + lvx_u $H3, r9,$Htbl + li r9,0xb0 + lvx_u $H3h,r10,$Htbl + li r10,0xc0 + lvx_u $H4l,r8,$Htbl # load H^4 + li r8,0x10 + lvx_u $H4, r9,$Htbl + li r9,0x20 + lvx_u $H4h,r10,$Htbl + li r10,0x30 + + vsldoi $t2,$zero,$t1,8 # 0x0000..0808 + vaddubm $hiperm,$t0,$t2 # 0x0001..1617 + vaddubm $loperm,$t1,$hiperm # 0x0809..1e1f + + $SHRI $len,$len,4 # this allows to use sign bit + # as carry + lvx_u $IN0,0,$inp # load input + lvx_u $IN1,r8,$inp + subic. $len,$len,8 + lvx_u $IN2,r9,$inp + lvx_u $IN3,r10,$inp + addi $inp,$inp,0x40 + le?vperm $IN0,$IN0,$IN0,$lemask + le?vperm $IN1,$IN1,$IN1,$lemask + le?vperm $IN2,$IN2,$IN2,$lemask + le?vperm $IN3,$IN3,$IN3,$lemask + + vxor $Xh,$IN0,$Xl + + vpmsumd $Xl1,$IN1,$H3l + vpmsumd $Xm1,$IN1,$H3 + vpmsumd $Xh1,$IN1,$H3h + + vperm $H21l,$H2,$H,$hiperm + vperm $t0,$IN2,$IN3,$loperm + vperm $H21h,$H2,$H,$loperm + vperm $t1,$IN2,$IN3,$hiperm + vpmsumd $Xm2,$IN2,$H2 # H^2.lo·Xi+2.hi+H^2.hi·Xi+2.lo + vpmsumd $Xl3,$t0,$H21l # H^2.lo·Xi+2.lo+H.lo·Xi+3.lo + vpmsumd $Xm3,$IN3,$H # H.hi·Xi+3.lo +H.lo·Xi+3.hi + vpmsumd $Xh3,$t1,$H21h # H^2.hi·Xi+2.hi+H.hi·Xi+3.hi + + vxor $Xm2,$Xm2,$Xm1 + vxor $Xl3,$Xl3,$Xl1 + vxor $Xm3,$Xm3,$Xm2 + vxor $Xh3,$Xh3,$Xh1 + + blt Ltail_4x + +Loop_4x: + lvx_u $IN0,0,$inp + lvx_u $IN1,r8,$inp + subic. $len,$len,4 + lvx_u $IN2,r9,$inp + lvx_u $IN3,r10,$inp + addi $inp,$inp,0x40 + le?vperm $IN1,$IN1,$IN1,$lemask + le?vperm $IN2,$IN2,$IN2,$lemask + le?vperm $IN3,$IN3,$IN3,$lemask + le?vperm $IN0,$IN0,$IN0,$lemask + + vpmsumd $Xl,$Xh,$H4l # H^4.lo·Xi.lo + vpmsumd $Xm,$Xh,$H4 # H^4.hi·Xi.lo+H^4.lo·Xi.hi + vpmsumd $Xh,$Xh,$H4h # H^4.hi·Xi.hi + vpmsumd $Xl1,$IN1,$H3l + vpmsumd $Xm1,$IN1,$H3 + vpmsumd $Xh1,$IN1,$H3h + + vxor $Xl,$Xl,$Xl3 + vxor $Xm,$Xm,$Xm3 + vxor $Xh,$Xh,$Xh3 + vperm $t0,$IN2,$IN3,$loperm + vperm $t1,$IN2,$IN3,$hiperm + + vpmsumd $t2,$Xl,$xC2 # 1st reduction phase + vpmsumd $Xl3,$t0,$H21l # H.lo·Xi+3.lo +H^2.lo·Xi+2.lo + vpmsumd $Xh3,$t1,$H21h # H.hi·Xi+3.hi +H^2.hi·Xi+2.hi + + vsldoi $t0,$Xm,$zero,8 + vsldoi $t1,$zero,$Xm,8 + vxor $Xl,$Xl,$t0 + vxor $Xh,$Xh,$t1 + + vsldoi $Xl,$Xl,$Xl,8 + vxor $Xl,$Xl,$t2 + + vsldoi $t1,$Xl,$Xl,8 # 2nd reduction phase + vpmsumd $Xm2,$IN2,$H2 # H^2.hi·Xi+2.lo+H^2.lo·Xi+2.hi + vpmsumd $Xm3,$IN3,$H # H.hi·Xi+3.lo +H.lo·Xi+3.hi + vpmsumd $Xl,$Xl,$xC2 + + vxor $Xl3,$Xl3,$Xl1 + vxor $Xh3,$Xh3,$Xh1 + vxor $Xh,$Xh,$IN0 + vxor $Xm2,$Xm2,$Xm1 + vxor $Xh,$Xh,$t1 + vxor $Xm3,$Xm3,$Xm2 + vxor $Xh,$Xh,$Xl + bge Loop_4x + +Ltail_4x: + vpmsumd $Xl,$Xh,$H4l # H^4.lo·Xi.lo + vpmsumd $Xm,$Xh,$H4 # H^4.hi·Xi.lo+H^4.lo·Xi.hi + vpmsumd $Xh,$Xh,$H4h # H^4.hi·Xi.hi + + vxor $Xl,$Xl,$Xl3 + vxor $Xm,$Xm,$Xm3 + + vpmsumd $t2,$Xl,$xC2 # 1st reduction phase + + vsldoi $t0,$Xm,$zero,8 + vsldoi $t1,$zero,$Xm,8 + vxor $Xh,$Xh,$Xh3 + vxor $Xl,$Xl,$t0 + vxor $Xh,$Xh,$t1 + + vsldoi $Xl,$Xl,$Xl,8 + vxor $Xl,$Xl,$t2 + + vsldoi $t1,$Xl,$Xl,8 # 2nd reduction phase + vpmsumd $Xl,$Xl,$xC2 + vxor $t1,$t1,$Xh + vxor $Xl,$Xl,$t1 + + addic. $len,$len,4 + beq Ldone_4x + + lvx_u $IN0,0,$inp + ${UCMP}i $len,2 + li $len,-4 + blt Lone + lvx_u $IN1,r8,$inp + beq Ltwo + +Lthree: + lvx_u $IN2,r9,$inp + le?vperm $IN0,$IN0,$IN0,$lemask + le?vperm $IN1,$IN1,$IN1,$lemask + le?vperm $IN2,$IN2,$IN2,$lemask + + vxor $Xh,$IN0,$Xl + vmr $H4l,$H3l + vmr $H4, $H3 + vmr $H4h,$H3h + + vperm $t0,$IN1,$IN2,$loperm + vperm $t1,$IN1,$IN2,$hiperm + vpmsumd $Xm2,$IN1,$H2 # H^2.lo·Xi+1.hi+H^2.hi·Xi+1.lo + vpmsumd $Xm3,$IN2,$H # H.hi·Xi+2.lo +H.lo·Xi+2.hi + vpmsumd $Xl3,$t0,$H21l # H^2.lo·Xi+1.lo+H.lo·Xi+2.lo + vpmsumd $Xh3,$t1,$H21h # H^2.hi·Xi+1.hi+H.hi·Xi+2.hi + + vxor $Xm3,$Xm3,$Xm2 + b Ltail_4x + +.align 4 +Ltwo: + le?vperm $IN0,$IN0,$IN0,$lemask + le?vperm $IN1,$IN1,$IN1,$lemask + + vxor $Xh,$IN0,$Xl + vperm $t0,$zero,$IN1,$loperm + vperm $t1,$zero,$IN1,$hiperm + + vsldoi $H4l,$zero,$H2,8 + vmr $H4, $H2 + vsldoi $H4h,$H2,$zero,8 + + vpmsumd $Xl3,$t0, $H21l # H.lo·Xi+1.lo + vpmsumd $Xm3,$IN1,$H # H.hi·Xi+1.lo+H.lo·Xi+2.hi + vpmsumd $Xh3,$t1, $H21h # H.hi·Xi+1.hi + + b Ltail_4x + +.align 4 +Lone: + le?vperm $IN0,$IN0,$IN0,$lemask + + vsldoi $H4l,$zero,$H,8 + vmr $H4, $H + vsldoi $H4h,$H,$zero,8 + + vxor $Xh,$IN0,$Xl + vxor $Xl3,$Xl3,$Xl3 + vxor $Xm3,$Xm3,$Xm3 + vxor $Xh3,$Xh3,$Xh3 + + b Ltail_4x + +Ldone_4x: + le?vperm $Xl,$Xl,$Xl,$lemask + stvx_u $Xl,0,$Xip # write out Xi + + li r10,`15+6*$SIZE_T` + li r11,`31+6*$SIZE_T` + mtspr 256,$vrsave + lvx v20,r10,$sp + addi r10,r10,32 + lvx v21,r11,$sp + addi r11,r11,32 + lvx v22,r10,$sp + addi r10,r10,32 + lvx v23,r11,$sp + addi r11,r11,32 + lvx v24,r10,$sp + addi r10,r10,32 + lvx v25,r11,$sp + addi r11,r11,32 + lvx v26,r10,$sp + addi r10,r10,32 + lvx v27,r11,$sp + addi r11,r11,32 + lvx v28,r10,$sp + addi r10,r10,32 + lvx v29,r11,$sp + addi r11,r11,32 + lvx v30,r10,$sp + lvx v31,r11,$sp + addi $sp,$sp,$FRAME + blr + .long 0 + .byte 0,12,0x04,0,0x80,0,4,0 + .long 0 +___ +} +$code.=<<___; +.size .gcm_ghash_p8,.-.gcm_ghash_p8 + +.asciz "GHASH for PowerISA 2.07, CRYPTOGAMS by " +.align 2 +___ + +foreach (split("\n",$code)) { + s/\`([^\`]*)\`/eval $1/geo; + + if ($flavour =~ /le$/o) { # little-endian + s/le\?//o or + s/be\?/#be#/o; + } else { + s/le\?/#le#/o or + s/be\?//o; + } + print $_,"\n"; +} + +close STDOUT; # enforce flush diff --git a/cbits/asm/ppc-xlate.pl b/cbits/asm/ppc-xlate.pl new file mode 100644 --- /dev/null +++ b/cbits/asm/ppc-xlate.pl @@ -0,0 +1,352 @@ +#!/usr/bin/env perl + +# PowerPC assembler distiller by \@dot-asm. + +################################################################ +# Recognized "flavour"-s are: +# +# linux{32|64}[le] GNU assembler and ELF symbol decorations, +# with little-endian option +# linux64v2 GNU asssembler and big-endian instantiation +# of latest ELF specification +# aix{32|64} AIX assembler and symbol decorations +# osx{32|64} Mac OS X assembler and symbol decoratons + +my $flavour = shift; +my $output = shift; +open STDOUT,">$output" || die "can't open $output: $!"; + +my %GLOBALS; +my %TYPES; +my $dotinlocallabels=($flavour=~/linux/)?1:0; + +################################################################ +# directives which need special treatment on different platforms +################################################################ +my $type = sub { + my ($dir,$name,$type) = @_; + + $TYPES{$name} = $type; + if ($flavour =~ /linux/) { + $name =~ s|^\.||; + ".type $name,$type"; + } else { + ""; + } +}; +my $globl = sub { + my $junk = shift; + my $name = shift; + my $global = \$GLOBALS{$name}; + my $type = \$TYPES{$name}; + my $ret; + + $name =~ s|^\.||; + + SWITCH: for ($flavour) { + /aix/ && do { if (!$$type) { + $$type = "\@function"; + } + if ($$type =~ /function/) { + $name = ".$name"; + } + last; + }; + /osx/ && do { $name = "_$name"; + last; + }; + /linux.*(32|64(le|v2))/ + && do { $ret .= ".globl $name"; + if (!$$type) { + $ret .= "\n.type $name,\@function"; + $$type = "\@function"; + } + last; + }; + /linux.*64/ && do { $ret .= ".globl $name"; + if (!$$type) { + $ret .= "\n.type $name,\@function"; + $$type = "\@function"; + } + if ($$type =~ /function/) { + $ret .= "\n.section \".opd\",\"aw\""; + $ret .= "\n.align 3"; + $ret .= "\n$name:"; + $ret .= "\n.quad .$name,.TOC.\@tocbase,0"; + $ret .= "\n.previous"; + $name = ".$name"; + } + last; + }; + } + + $ret = ".globl $name" if (!$ret); + $$global = $name; + $ret; +}; +my $text = sub { + my $ret = ($flavour =~ /aix/) ? ".csect\t.text[PR],7" : ".text"; + $ret = ".abiversion 2\n".$ret if ($flavour =~ /linux.*64(le|v2)/); + $ret; +}; +my $machine = sub { + my $junk = shift; + my $arch = shift; + if ($flavour =~ /osx/) + { $arch =~ s/\"//g; + $arch = ($flavour=~/64/) ? "ppc970-64" : "ppc970" if ($arch eq "any"); + } + ".machine $arch"; +}; +my $size = sub { + if ($flavour =~ /linux/) + { shift; + my $name = shift; + my $real = $GLOBALS{$name} ? \$GLOBALS{$name} : \$name; + my $ret = ".size $$real,.-$$real"; + $name =~ s|^\.||; + if ($$real ne $name) { + $ret .= "\n.size $name,.-$$real"; + } + $ret; + } + else + { ""; } +}; +my $asciz = sub { + shift; + my $line = join(",",@_); + if ($line =~ /^"(.*)"$/) + { ".byte " . join(",",unpack("C*",$1),0) . "\n.align 2"; } + else + { ""; } +}; +my $quad = sub { + shift; + my @ret; + my ($hi,$lo); + for (@_) { + if (/^0x([0-9a-f]*?)([0-9a-f]{1,8})$/io) + { $hi=$1?"0x$1":"0"; $lo="0x$2"; } + elsif (/^([0-9]+)$/o) + { $hi=$1>>32; $lo=$1&0xffffffff; } # error-prone with 32-bit perl + else + { $hi=undef; $lo=$_; } + + if (defined($hi)) + { push(@ret,$flavour=~/le$/o?".long\t$lo,$hi":".long\t$hi,$lo"); } + else + { push(@ret,".quad $lo"); } + } + join("\n",@ret); +}; + +################################################################ +# simplified mnemonics not handled by at least one assembler +################################################################ +my $cmplw = sub { + my $f = shift; + my $cr = 0; $cr = shift if ($#_>1); + # Some out-of-date 32-bit GNU assembler just can't handle cmplw... + ($flavour =~ /linux.*32/) ? + " .long ".sprintf "0x%x",31<<26|$cr<<23|$_[0]<<16|$_[1]<<11|64 : + " cmplw ".join(',',$cr,@_); +}; +my $bdnz = sub { + my $f = shift; + my $bo = $f=~/[\+\-]/ ? 16+9 : 16; # optional "to be taken" hint + " bc $bo,0,".shift; +} if ($flavour!~/linux/); +my $bltlr = sub { + my $f = shift; + my $bo = $f=~/\-/ ? 12+2 : 12; # optional "not to be taken" hint + ($flavour =~ /linux/) ? # GNU as doesn't allow most recent hints + " .long ".sprintf "0x%x",19<<26|$bo<<21|16<<1 : + " bclr $bo,0"; +}; +my $bnelr = sub { + my $f = shift; + my $bo = $f=~/\-/ ? 4+2 : 4; # optional "not to be taken" hint + ($flavour =~ /linux/) ? # GNU as doesn't allow most recent hints + " .long ".sprintf "0x%x",19<<26|$bo<<21|2<<16|16<<1 : + " bclr $bo,2"; +}; +my $beqlr = sub { + my $f = shift; + my $bo = $f=~/-/ ? 12+2 : 12; # optional "not to be taken" hint + ($flavour =~ /linux/) ? # GNU as doesn't allow most recent hints + " .long ".sprintf "0x%X",19<<26|$bo<<21|2<<16|16<<1 : + " bclr $bo,2"; +}; +# GNU assembler can't handle extrdi rA,rS,16,48, or when sum of last two +# arguments is 64, with "operand out of range" error. +my $extrdi = sub { + my ($f,$ra,$rs,$n,$b) = @_; + $b = ($b+$n)&63; $n = 64-$n; + " rldicl $ra,$rs,$b,$n"; +}; +my $vmr = sub { + my ($f,$vx,$vy) = @_; + " vor $vx,$vy,$vy"; +}; + +# Some ABIs specify vrsave, special-purpose register #256, as reserved +# for system use. +my $no_vrsave = ($flavour =~ /aix|linux64(le|v2)/); +my $mtspr = sub { + my ($f,$idx,$ra) = @_; + if ($idx == 256 && $no_vrsave) { + " or $ra,$ra,$ra"; + } else { + " mtspr $idx,$ra"; + } +}; +my $mfspr = sub { + my ($f,$rd,$idx) = @_; + if ($idx == 256 && $no_vrsave) { + " li $rd,-1"; + } else { + " mfspr $rd,$idx"; + } +}; + +# PowerISA 2.06 stuff +sub vsxmem_op { + my ($f, $vrt, $ra, $rb, $op) = @_; + " .long ".sprintf "0x%X",(31<<26)|($vrt<<21)|($ra<<16)|($rb<<11)|($op*2+1); +} +# made-up unaligned memory reference AltiVec/VMX instructions +my $lvx_u = sub { vsxmem_op(@_, 844); }; # lxvd2x +my $stvx_u = sub { vsxmem_op(@_, 972); }; # stxvd2x +my $lvdx_u = sub { vsxmem_op(@_, 588); }; # lxsdx +my $stvdx_u = sub { vsxmem_op(@_, 716); }; # stxsdx +my $lvx_4w = sub { vsxmem_op(@_, 780); }; # lxvw4x +my $stvx_4w = sub { vsxmem_op(@_, 908); }; # stxvw4x +my $lvx_splt = sub { vsxmem_op(@_, 332); }; # lxvdsx +# VSX instruction[s] masqueraded as made-up AltiVec/VMX +my $vpermdi = sub { # xxpermdi + my ($f, $vrt, $vra, $vrb, $dm) = @_; + $dm = oct($dm) if ($dm =~ /^0/); + " .long ".sprintf "0x%X",(60<<26)|($vrt<<21)|($vra<<16)|($vrb<<11)|($dm<<8)|(10<<3)|7; +}; + +# PowerISA 2.07 stuff +sub vcrypto_op { + my ($f, $vrt, $vra, $vrb, $op) = @_; + " .long ".sprintf "0x%X",(4<<26)|($vrt<<21)|($vra<<16)|($vrb<<11)|$op; +} +sub vfour { + my ($f, $vrt, $vra, $vrb, $vrc, $op) = @_; + " .long ".sprintf "0x%X",(4<<26)|($vrt<<21)|($vra<<16)|($vrb<<11)|($vrc<<6)|$op; +}; +my $vcipher = sub { vcrypto_op(@_, 1288); }; +my $vcipherlast = sub { vcrypto_op(@_, 1289); }; +my $vncipher = sub { vcrypto_op(@_, 1352); }; +my $vncipherlast= sub { vcrypto_op(@_, 1353); }; +my $vsbox = sub { vcrypto_op(@_, 0, 1480); }; +my $vshasigmad = sub { my ($st,$six)=splice(@_,-2); vcrypto_op(@_, $st<<4|$six, 1730); }; +my $vshasigmaw = sub { my ($st,$six)=splice(@_,-2); vcrypto_op(@_, $st<<4|$six, 1666); }; +my $vpmsumb = sub { vcrypto_op(@_, 1032); }; +my $vpmsumd = sub { vcrypto_op(@_, 1224); }; +my $vpmsubh = sub { vcrypto_op(@_, 1096); }; +my $vpmsumw = sub { vcrypto_op(@_, 1160); }; +# These are not really crypto, but vcrypto_op template works +my $vaddudm = sub { vcrypto_op(@_, 192); }; +my $vadduqm = sub { vcrypto_op(@_, 256); }; +my $vmuleuw = sub { vcrypto_op(@_, 648); }; +my $vmulouw = sub { vcrypto_op(@_, 136); }; +my $vrld = sub { vcrypto_op(@_, 196); }; +my $vsld = sub { vcrypto_op(@_, 1476); }; +my $vsrd = sub { vcrypto_op(@_, 1732); }; +my $vsubudm = sub { vcrypto_op(@_, 1216); }; +my $vaddcuq = sub { vcrypto_op(@_, 320); }; +my $vaddeuqm = sub { vfour(@_,60); }; +my $vaddecuq = sub { vfour(@_,61); }; +my $vmrgew = sub { vfour(@_,0,1932); }; +my $vmrgow = sub { vfour(@_,0,1676); }; + +my $mtsle = sub { + my ($f, $arg) = @_; + " .long ".sprintf "0x%X",(31<<26)|($arg<<21)|(147*2); +}; + +# VSX instructions masqueraded as AltiVec/VMX +my $mtvrd = sub { + my ($f, $vrt, $ra) = @_; + " .long ".sprintf "0x%X",(31<<26)|($vrt<<21)|($ra<<16)|(179<<1)|1; +}; +my $mtvrwz = sub { + my ($f, $vrt, $ra) = @_; + " .long ".sprintf "0x%X",(31<<26)|($vrt<<21)|($ra<<16)|(243<<1)|1; +}; +my $lvwzx_u = sub { vsxmem_op(@_, 12); }; # lxsiwzx +my $stvwx_u = sub { vsxmem_op(@_, 140); }; # stxsiwx + +# PowerISA 3.0 stuff +my $maddhdu = sub { vfour(@_,49); }; +my $maddld = sub { vfour(@_,51); }; +my $darn = sub { + my ($f, $rt, $l) = @_; + " .long ".sprintf "0x%X",(31<<26)|($rt<<21)|($l<<16)|(755<<1); +}; +my $iseleq = sub { + my ($f, $rt, $ra, $rb) = @_; + " .long ".sprintf "0x%X",(31<<26)|($rt<<21)|($ra<<16)|($rb<<11)|(2<<6)|30; +}; +# VSX instruction[s] masqueraded as made-up AltiVec/VMX +my $vspltib = sub { # xxspltib + my ($f, $vrt, $imm8) = @_; + $imm8 = oct($imm8) if ($imm8 =~ /^0/); + $imm8 &= 0xff; + " .long ".sprintf "0x%X",(60<<26)|($vrt<<21)|($imm8<<11)|(360<<1)|1; +}; + +# PowerISA 3.0B stuff +my $addex = sub { + my ($f, $rt, $ra, $rb, $cy) = @_; # only cy==0 is specified in 3.0B + " .long ".sprintf "0x%X",(31<<26)|($rt<<21)|($ra<<16)|($rb<<11)|($cy<<9)|(170<<1); +}; +my $vmsumudm = sub { vfour(@_,35); }; + +while($line=<>) { + + $line =~ s|[#!;].*$||; # get rid of asm-style comments... + $line =~ s|/\*.*\*/||; # ... and C-style comments... + $line =~ s|^\s+||; # ... and skip white spaces in beginning... + $line =~ s|\s+$||; # ... and at the end + + { + $line =~ s|\.L(\w+)|L$1|g; # common denominator for Locallabel + $line =~ s|\bL(\w+)|\.L$1|g if ($dotinlocallabels); + } + + { + $line =~ s|(^[\.\w]+)\:\s*||; + my $label = $1; + if ($label) { + my $xlated = ($GLOBALS{$label} or $label); + print "$xlated:"; + if ($flavour =~ /linux.*64(le|v2)/) { + if ($TYPES{$label} =~ /function/) { + printf "\n.localentry %s,0\n",$xlated; + } + } + } + } + + { + $line =~ s|^\s*(\.?)(\w+)([\.\+\-]?)\s*||; + my $c = $1; $c = "\t" if ($c eq ""); + my $mnemonic = $2; + my $f = $3; + my $opcode = eval("\$$mnemonic"); + $line =~ s/\b(c?[rf]|v|vs)([0-9]+)\b/$2/g if ($c ne "." and $flavour !~ /osx/); + if (ref($opcode) eq 'CODE') { $line = &$opcode($f,split(/,\s*/,$line)); } + elsif ($mnemonic) { $line = $c.$mnemonic.$f."\t".$line; } + } + + print $line if ($line); + print "\n"; +} + +close STDOUT; diff --git a/cbits/bearssl/LICENSE b/cbits/bearssl/LICENSE new file mode 100644 --- /dev/null +++ b/cbits/bearssl/LICENSE @@ -0,0 +1,21 @@ +Copyright (c) 2016 Thomas Pornin + +Permission is hereby granted, free of charge, to any person obtaining +a copy of this software and associated documentation files (the +"Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS +BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN +ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/cbits/bearssl/README.md b/cbits/bearssl/README.md new file mode 100644 --- /dev/null +++ b/cbits/bearssl/README.md @@ -0,0 +1,57 @@ +# BearSSL + +Constant-time AES and GHASH from [BearSSL](https://bearssl.org/), vendored +here. `VERSION` holds the upstream release these files came from, and +`import.sh` fetches them again. + +## Why + +crypton's portable AES and GHASH are table-driven, and both index their +tables with a secret: `cbits/aes/generic.c` with a byte of the state, in +every round and in the key expansion, and `cbits/aes/gf.c` with a nibble of +the GHASH accumulator. Both are variable-time by construction, and the +cache-timing attacks on that shape of code are old and well documented. + +The processors that have AES instructions do not run any of it -- crypton +asks them first. What runs it is everything else: ppc64le and s390x, where +the instructions exist but crypton has no path to them; 32-bit ARM, where +the same is true; riscv64 and loongarch64; and the boards that genuinely +have no AES instructions at all, of which the Raspberry Pi 3 and 4 are by +some distance the largest population. + +BearSSL's answer is bitslicing. `aes_ct64` holds four blocks interleaved +across eight 64-bit words and computes the S-box as boolean algebra, so +there is no table and no address derived from a secret; `ghash_ctmul64` +builds the GF(2^128) multiply out of shifts, masks and integer multiplies +rather than a table of H. Both are plain C99 and assume nothing beyond +`uint64_t`. + +## What is here + +| file | from | +| --- | --- | +| `aes_ct64.c` | `src/symcipher/aes_ct64.c` | +| `aes_ct64_enc.c` | `src/symcipher/aes_ct64_enc.c` | +| `aes_ct64_dec.c` | `src/symcipher/aes_ct64_dec.c` | +| `ghash_ctmul64.c` | `src/hash/ghash_ctmul64.c` | +| `dec32le.c` | `src/codec/dec32le.c`, for `br_range_dec32le` | +| `LICENSE` | `LICENSE.txt` -- MIT, (c) 2016 Thomas Pornin | + +`inner.h` is **crypton's, not upstream's**. Each of those five opens with +`#include "inner.h"`, and upstream's is some two thousand lines declaring +the whole library; these five want six things from it. So this one gives +those six and nothing else, which is what lets the five stay byte for byte +what upstream ships. `import.sh` does not overwrite it. + +The byte-order helpers in it are written rather than copied, in their plain +portable form without upstream's unaligned-access fast paths, so that no +platform configuration comes with them. + +## Keeping it honest + +`cbits/tests/bearssl_diff.c` checks this against the implementation it +replaces: the FIPS-197 vectors first, so that agreement means AES and not +merely that both sides compute the same wrong thing, then a run of random +keys and blocks, then GHASH against the 4-bit table. Given any argument it +corrupts three results on purpose and the comparison has to notice -- a +differential test that cannot fail has said nothing. diff --git a/cbits/bearssl/VERSION b/cbits/bearssl/VERSION new file mode 100644 --- /dev/null +++ b/cbits/bearssl/VERSION @@ -0,0 +1,1 @@ +0.6 diff --git a/cbits/bearssl/aes_ct64.c b/cbits/bearssl/aes_ct64.c new file mode 100644 --- /dev/null +++ b/cbits/bearssl/aes_ct64.c @@ -0,0 +1,398 @@ +/* + * Copyright (c) 2016 Thomas Pornin + * + * Permission is hereby granted, free of charge, to any person obtaining + * a copy of this software and associated documentation files (the + * "Software"), to deal in the Software without restriction, including + * without limitation the rights to use, copy, modify, merge, publish, + * distribute, sublicense, and/or sell copies of the Software, and to + * permit persons to whom the Software is furnished to do so, subject to + * the following conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#include "inner.h" + +/* see inner.h */ +void +br_aes_ct64_bitslice_Sbox(uint64_t *q) +{ + /* + * This S-box implementation is a straightforward translation of + * the circuit described by Boyar and Peralta in "A new + * combinational logic minimization technique with applications + * to cryptology" (https://eprint.iacr.org/2009/191.pdf). + * + * Note that variables x* (input) and s* (output) are numbered + * in "reverse" order (x0 is the high bit, x7 is the low bit). + */ + + uint64_t x0, x1, x2, x3, x4, x5, x6, x7; + uint64_t y1, y2, y3, y4, y5, y6, y7, y8, y9; + uint64_t y10, y11, y12, y13, y14, y15, y16, y17, y18, y19; + uint64_t y20, y21; + uint64_t z0, z1, z2, z3, z4, z5, z6, z7, z8, z9; + uint64_t z10, z11, z12, z13, z14, z15, z16, z17; + uint64_t t0, t1, t2, t3, t4, t5, t6, t7, t8, t9; + uint64_t t10, t11, t12, t13, t14, t15, t16, t17, t18, t19; + uint64_t t20, t21, t22, t23, t24, t25, t26, t27, t28, t29; + uint64_t t30, t31, t32, t33, t34, t35, t36, t37, t38, t39; + uint64_t t40, t41, t42, t43, t44, t45, t46, t47, t48, t49; + uint64_t t50, t51, t52, t53, t54, t55, t56, t57, t58, t59; + uint64_t t60, t61, t62, t63, t64, t65, t66, t67; + uint64_t s0, s1, s2, s3, s4, s5, s6, s7; + + x0 = q[7]; + x1 = q[6]; + x2 = q[5]; + x3 = q[4]; + x4 = q[3]; + x5 = q[2]; + x6 = q[1]; + x7 = q[0]; + + /* + * Top linear transformation. + */ + y14 = x3 ^ x5; + y13 = x0 ^ x6; + y9 = x0 ^ x3; + y8 = x0 ^ x5; + t0 = x1 ^ x2; + y1 = t0 ^ x7; + y4 = y1 ^ x3; + y12 = y13 ^ y14; + y2 = y1 ^ x0; + y5 = y1 ^ x6; + y3 = y5 ^ y8; + t1 = x4 ^ y12; + y15 = t1 ^ x5; + y20 = t1 ^ x1; + y6 = y15 ^ x7; + y10 = y15 ^ t0; + y11 = y20 ^ y9; + y7 = x7 ^ y11; + y17 = y10 ^ y11; + y19 = y10 ^ y8; + y16 = t0 ^ y11; + y21 = y13 ^ y16; + y18 = x0 ^ y16; + + /* + * Non-linear section. + */ + t2 = y12 & y15; + t3 = y3 & y6; + t4 = t3 ^ t2; + t5 = y4 & x7; + t6 = t5 ^ t2; + t7 = y13 & y16; + t8 = y5 & y1; + t9 = t8 ^ t7; + t10 = y2 & y7; + t11 = t10 ^ t7; + t12 = y9 & y11; + t13 = y14 & y17; + t14 = t13 ^ t12; + t15 = y8 & y10; + t16 = t15 ^ t12; + t17 = t4 ^ t14; + t18 = t6 ^ t16; + t19 = t9 ^ t14; + t20 = t11 ^ t16; + t21 = t17 ^ y20; + t22 = t18 ^ y19; + t23 = t19 ^ y21; + t24 = t20 ^ y18; + + t25 = t21 ^ t22; + t26 = t21 & t23; + t27 = t24 ^ t26; + t28 = t25 & t27; + t29 = t28 ^ t22; + t30 = t23 ^ t24; + t31 = t22 ^ t26; + t32 = t31 & t30; + t33 = t32 ^ t24; + t34 = t23 ^ t33; + t35 = t27 ^ t33; + t36 = t24 & t35; + t37 = t36 ^ t34; + t38 = t27 ^ t36; + t39 = t29 & t38; + t40 = t25 ^ t39; + + t41 = t40 ^ t37; + t42 = t29 ^ t33; + t43 = t29 ^ t40; + t44 = t33 ^ t37; + t45 = t42 ^ t41; + z0 = t44 & y15; + z1 = t37 & y6; + z2 = t33 & x7; + z3 = t43 & y16; + z4 = t40 & y1; + z5 = t29 & y7; + z6 = t42 & y11; + z7 = t45 & y17; + z8 = t41 & y10; + z9 = t44 & y12; + z10 = t37 & y3; + z11 = t33 & y4; + z12 = t43 & y13; + z13 = t40 & y5; + z14 = t29 & y2; + z15 = t42 & y9; + z16 = t45 & y14; + z17 = t41 & y8; + + /* + * Bottom linear transformation. + */ + t46 = z15 ^ z16; + t47 = z10 ^ z11; + t48 = z5 ^ z13; + t49 = z9 ^ z10; + t50 = z2 ^ z12; + t51 = z2 ^ z5; + t52 = z7 ^ z8; + t53 = z0 ^ z3; + t54 = z6 ^ z7; + t55 = z16 ^ z17; + t56 = z12 ^ t48; + t57 = t50 ^ t53; + t58 = z4 ^ t46; + t59 = z3 ^ t54; + t60 = t46 ^ t57; + t61 = z14 ^ t57; + t62 = t52 ^ t58; + t63 = t49 ^ t58; + t64 = z4 ^ t59; + t65 = t61 ^ t62; + t66 = z1 ^ t63; + s0 = t59 ^ t63; + s6 = t56 ^ ~t62; + s7 = t48 ^ ~t60; + t67 = t64 ^ t65; + s3 = t53 ^ t66; + s4 = t51 ^ t66; + s5 = t47 ^ t65; + s1 = t64 ^ ~s3; + s2 = t55 ^ ~t67; + + q[7] = s0; + q[6] = s1; + q[5] = s2; + q[4] = s3; + q[3] = s4; + q[2] = s5; + q[1] = s6; + q[0] = s7; +} + +/* see inner.h */ +void +br_aes_ct64_ortho(uint64_t *q) +{ +#define SWAPN(cl, ch, s, x, y) do { \ + uint64_t a, b; \ + a = (x); \ + b = (y); \ + (x) = (a & (uint64_t)cl) | ((b & (uint64_t)cl) << (s)); \ + (y) = ((a & (uint64_t)ch) >> (s)) | (b & (uint64_t)ch); \ + } while (0) + +#define SWAP2(x, y) SWAPN(0x5555555555555555, 0xAAAAAAAAAAAAAAAA, 1, x, y) +#define SWAP4(x, y) SWAPN(0x3333333333333333, 0xCCCCCCCCCCCCCCCC, 2, x, y) +#define SWAP8(x, y) SWAPN(0x0F0F0F0F0F0F0F0F, 0xF0F0F0F0F0F0F0F0, 4, x, y) + + SWAP2(q[0], q[1]); + SWAP2(q[2], q[3]); + SWAP2(q[4], q[5]); + SWAP2(q[6], q[7]); + + SWAP4(q[0], q[2]); + SWAP4(q[1], q[3]); + SWAP4(q[4], q[6]); + SWAP4(q[5], q[7]); + + SWAP8(q[0], q[4]); + SWAP8(q[1], q[5]); + SWAP8(q[2], q[6]); + SWAP8(q[3], q[7]); +} + +/* see inner.h */ +void +br_aes_ct64_interleave_in(uint64_t *q0, uint64_t *q1, const uint32_t *w) +{ + uint64_t x0, x1, x2, x3; + + x0 = w[0]; + x1 = w[1]; + x2 = w[2]; + x3 = w[3]; + x0 |= (x0 << 16); + x1 |= (x1 << 16); + x2 |= (x2 << 16); + x3 |= (x3 << 16); + x0 &= (uint64_t)0x0000FFFF0000FFFF; + x1 &= (uint64_t)0x0000FFFF0000FFFF; + x2 &= (uint64_t)0x0000FFFF0000FFFF; + x3 &= (uint64_t)0x0000FFFF0000FFFF; + x0 |= (x0 << 8); + x1 |= (x1 << 8); + x2 |= (x2 << 8); + x3 |= (x3 << 8); + x0 &= (uint64_t)0x00FF00FF00FF00FF; + x1 &= (uint64_t)0x00FF00FF00FF00FF; + x2 &= (uint64_t)0x00FF00FF00FF00FF; + x3 &= (uint64_t)0x00FF00FF00FF00FF; + *q0 = x0 | (x2 << 8); + *q1 = x1 | (x3 << 8); +} + +/* see inner.h */ +void +br_aes_ct64_interleave_out(uint32_t *w, uint64_t q0, uint64_t q1) +{ + uint64_t x0, x1, x2, x3; + + x0 = q0 & (uint64_t)0x00FF00FF00FF00FF; + x1 = q1 & (uint64_t)0x00FF00FF00FF00FF; + x2 = (q0 >> 8) & (uint64_t)0x00FF00FF00FF00FF; + x3 = (q1 >> 8) & (uint64_t)0x00FF00FF00FF00FF; + x0 |= (x0 >> 8); + x1 |= (x1 >> 8); + x2 |= (x2 >> 8); + x3 |= (x3 >> 8); + x0 &= (uint64_t)0x0000FFFF0000FFFF; + x1 &= (uint64_t)0x0000FFFF0000FFFF; + x2 &= (uint64_t)0x0000FFFF0000FFFF; + x3 &= (uint64_t)0x0000FFFF0000FFFF; + w[0] = (uint32_t)x0 | (uint32_t)(x0 >> 16); + w[1] = (uint32_t)x1 | (uint32_t)(x1 >> 16); + w[2] = (uint32_t)x2 | (uint32_t)(x2 >> 16); + w[3] = (uint32_t)x3 | (uint32_t)(x3 >> 16); +} + +static const unsigned char Rcon[] = { + 0x01, 0x02, 0x04, 0x08, 0x10, 0x20, 0x40, 0x80, 0x1B, 0x36 +}; + +static uint32_t +sub_word(uint32_t x) +{ + uint64_t q[8]; + + memset(q, 0, sizeof q); + q[0] = x; + br_aes_ct64_ortho(q); + br_aes_ct64_bitslice_Sbox(q); + br_aes_ct64_ortho(q); + return (uint32_t)q[0]; +} + +/* see inner.h */ +unsigned +br_aes_ct64_keysched(uint64_t *comp_skey, const void *key, size_t key_len) +{ + unsigned num_rounds; + int i, j, k, nk, nkf; + uint32_t tmp; + uint32_t skey[60]; + + switch (key_len) { + case 16: + num_rounds = 10; + break; + case 24: + num_rounds = 12; + break; + case 32: + num_rounds = 14; + break; + default: + /* abort(); */ + return 0; + } + nk = (int)(key_len >> 2); + nkf = (int)((num_rounds + 1) << 2); + br_range_dec32le(skey, (key_len >> 2), key); + tmp = skey[(key_len >> 2) - 1]; + for (i = nk, j = 0, k = 0; i < nkf; i ++) { + if (j == 0) { + tmp = (tmp << 24) | (tmp >> 8); + tmp = sub_word(tmp) ^ Rcon[k]; + } else if (nk > 6 && j == 4) { + tmp = sub_word(tmp); + } + tmp ^= skey[i - nk]; + skey[i] = tmp; + if (++ j == nk) { + j = 0; + k ++; + } + } + + for (i = 0, j = 0; i < nkf; i += 4, j += 2) { + uint64_t q[8]; + + br_aes_ct64_interleave_in(&q[0], &q[4], skey + i); + q[1] = q[0]; + q[2] = q[0]; + q[3] = q[0]; + q[5] = q[4]; + q[6] = q[4]; + q[7] = q[4]; + br_aes_ct64_ortho(q); + comp_skey[j + 0] = + (q[0] & (uint64_t)0x1111111111111111) + | (q[1] & (uint64_t)0x2222222222222222) + | (q[2] & (uint64_t)0x4444444444444444) + | (q[3] & (uint64_t)0x8888888888888888); + comp_skey[j + 1] = + (q[4] & (uint64_t)0x1111111111111111) + | (q[5] & (uint64_t)0x2222222222222222) + | (q[6] & (uint64_t)0x4444444444444444) + | (q[7] & (uint64_t)0x8888888888888888); + } + return num_rounds; +} + +/* see inner.h */ +void +br_aes_ct64_skey_expand(uint64_t *skey, + unsigned num_rounds, const uint64_t *comp_skey) +{ + unsigned u, v, n; + + n = (num_rounds + 1) << 1; + for (u = 0, v = 0; u < n; u ++, v += 4) { + uint64_t x0, x1, x2, x3; + + x0 = x1 = x2 = x3 = comp_skey[u]; + x0 &= (uint64_t)0x1111111111111111; + x1 &= (uint64_t)0x2222222222222222; + x2 &= (uint64_t)0x4444444444444444; + x3 &= (uint64_t)0x8888888888888888; + x1 >>= 1; + x2 >>= 2; + x3 >>= 3; + skey[v + 0] = (x0 << 4) - x0; + skey[v + 1] = (x1 << 4) - x1; + skey[v + 2] = (x2 << 4) - x2; + skey[v + 3] = (x3 << 4) - x3; + } +} diff --git a/cbits/bearssl/aes_ct64_dec.c b/cbits/bearssl/aes_ct64_dec.c new file mode 100644 --- /dev/null +++ b/cbits/bearssl/aes_ct64_dec.c @@ -0,0 +1,159 @@ +/* + * Copyright (c) 2016 Thomas Pornin + * + * Permission is hereby granted, free of charge, to any person obtaining + * a copy of this software and associated documentation files (the + * "Software"), to deal in the Software without restriction, including + * without limitation the rights to use, copy, modify, merge, publish, + * distribute, sublicense, and/or sell copies of the Software, and to + * permit persons to whom the Software is furnished to do so, subject to + * the following conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#include "inner.h" + +/* see inner.h */ +void +br_aes_ct64_bitslice_invSbox(uint64_t *q) +{ + /* + * See br_aes_ct_bitslice_invSbox(). This is the natural extension + * to 64-bit registers. + */ + uint64_t q0, q1, q2, q3, q4, q5, q6, q7; + + q0 = ~q[0]; + q1 = ~q[1]; + q2 = q[2]; + q3 = q[3]; + q4 = q[4]; + q5 = ~q[5]; + q6 = ~q[6]; + q7 = q[7]; + q[7] = q1 ^ q4 ^ q6; + q[6] = q0 ^ q3 ^ q5; + q[5] = q7 ^ q2 ^ q4; + q[4] = q6 ^ q1 ^ q3; + q[3] = q5 ^ q0 ^ q2; + q[2] = q4 ^ q7 ^ q1; + q[1] = q3 ^ q6 ^ q0; + q[0] = q2 ^ q5 ^ q7; + + br_aes_ct64_bitslice_Sbox(q); + + q0 = ~q[0]; + q1 = ~q[1]; + q2 = q[2]; + q3 = q[3]; + q4 = q[4]; + q5 = ~q[5]; + q6 = ~q[6]; + q7 = q[7]; + q[7] = q1 ^ q4 ^ q6; + q[6] = q0 ^ q3 ^ q5; + q[5] = q7 ^ q2 ^ q4; + q[4] = q6 ^ q1 ^ q3; + q[3] = q5 ^ q0 ^ q2; + q[2] = q4 ^ q7 ^ q1; + q[1] = q3 ^ q6 ^ q0; + q[0] = q2 ^ q5 ^ q7; +} + +static void +add_round_key(uint64_t *q, const uint64_t *sk) +{ + int i; + + for (i = 0; i < 8; i ++) { + q[i] ^= sk[i]; + } +} + +static void +inv_shift_rows(uint64_t *q) +{ + int i; + + for (i = 0; i < 8; i ++) { + uint64_t x; + + x = q[i]; + q[i] = (x & (uint64_t)0x000000000000FFFF) + | ((x & (uint64_t)0x000000000FFF0000) << 4) + | ((x & (uint64_t)0x00000000F0000000) >> 12) + | ((x & (uint64_t)0x000000FF00000000) << 8) + | ((x & (uint64_t)0x0000FF0000000000) >> 8) + | ((x & (uint64_t)0x000F000000000000) << 12) + | ((x & (uint64_t)0xFFF0000000000000) >> 4); + } +} + +static inline uint64_t +rotr32(uint64_t x) +{ + return (x << 32) | (x >> 32); +} + +static void +inv_mix_columns(uint64_t *q) +{ + uint64_t q0, q1, q2, q3, q4, q5, q6, q7; + uint64_t r0, r1, r2, r3, r4, r5, r6, r7; + + q0 = q[0]; + q1 = q[1]; + q2 = q[2]; + q3 = q[3]; + q4 = q[4]; + q5 = q[5]; + q6 = q[6]; + q7 = q[7]; + r0 = (q0 >> 16) | (q0 << 48); + r1 = (q1 >> 16) | (q1 << 48); + r2 = (q2 >> 16) | (q2 << 48); + r3 = (q3 >> 16) | (q3 << 48); + r4 = (q4 >> 16) | (q4 << 48); + r5 = (q5 >> 16) | (q5 << 48); + r6 = (q6 >> 16) | (q6 << 48); + r7 = (q7 >> 16) | (q7 << 48); + + q[0] = q5 ^ q6 ^ q7 ^ r0 ^ r5 ^ r7 ^ rotr32(q0 ^ q5 ^ q6 ^ r0 ^ r5); + q[1] = q0 ^ q5 ^ r0 ^ r1 ^ r5 ^ r6 ^ r7 ^ rotr32(q1 ^ q5 ^ q7 ^ r1 ^ r5 ^ r6); + q[2] = q0 ^ q1 ^ q6 ^ r1 ^ r2 ^ r6 ^ r7 ^ rotr32(q0 ^ q2 ^ q6 ^ r2 ^ r6 ^ r7); + q[3] = q0 ^ q1 ^ q2 ^ q5 ^ q6 ^ r0 ^ r2 ^ r3 ^ r5 ^ rotr32(q0 ^ q1 ^ q3 ^ q5 ^ q6 ^ q7 ^ r0 ^ r3 ^ r5 ^ r7); + q[4] = q1 ^ q2 ^ q3 ^ q5 ^ r1 ^ r3 ^ r4 ^ r5 ^ r6 ^ r7 ^ rotr32(q1 ^ q2 ^ q4 ^ q5 ^ q7 ^ r1 ^ r4 ^ r5 ^ r6); + q[5] = q2 ^ q3 ^ q4 ^ q6 ^ r2 ^ r4 ^ r5 ^ r6 ^ r7 ^ rotr32(q2 ^ q3 ^ q5 ^ q6 ^ r2 ^ r5 ^ r6 ^ r7); + q[6] = q3 ^ q4 ^ q5 ^ q7 ^ r3 ^ r5 ^ r6 ^ r7 ^ rotr32(q3 ^ q4 ^ q6 ^ q7 ^ r3 ^ r6 ^ r7); + q[7] = q4 ^ q5 ^ q6 ^ r4 ^ r6 ^ r7 ^ rotr32(q4 ^ q5 ^ q7 ^ r4 ^ r7); +} + +/* see inner.h */ +void +br_aes_ct64_bitslice_decrypt(unsigned num_rounds, + const uint64_t *skey, uint64_t *q) +{ + unsigned u; + + add_round_key(q, skey + (num_rounds << 3)); + for (u = num_rounds - 1; u > 0; u --) { + inv_shift_rows(q); + br_aes_ct64_bitslice_invSbox(q); + add_round_key(q, skey + (u << 3)); + inv_mix_columns(q); + } + inv_shift_rows(q); + br_aes_ct64_bitslice_invSbox(q); + add_round_key(q, skey); +} diff --git a/cbits/bearssl/aes_ct64_enc.c b/cbits/bearssl/aes_ct64_enc.c new file mode 100644 --- /dev/null +++ b/cbits/bearssl/aes_ct64_enc.c @@ -0,0 +1,115 @@ +/* + * Copyright (c) 2016 Thomas Pornin + * + * Permission is hereby granted, free of charge, to any person obtaining + * a copy of this software and associated documentation files (the + * "Software"), to deal in the Software without restriction, including + * without limitation the rights to use, copy, modify, merge, publish, + * distribute, sublicense, and/or sell copies of the Software, and to + * permit persons to whom the Software is furnished to do so, subject to + * the following conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#include "inner.h" + +static inline void +add_round_key(uint64_t *q, const uint64_t *sk) +{ + q[0] ^= sk[0]; + q[1] ^= sk[1]; + q[2] ^= sk[2]; + q[3] ^= sk[3]; + q[4] ^= sk[4]; + q[5] ^= sk[5]; + q[6] ^= sk[6]; + q[7] ^= sk[7]; +} + +static inline void +shift_rows(uint64_t *q) +{ + int i; + + for (i = 0; i < 8; i ++) { + uint64_t x; + + x = q[i]; + q[i] = (x & (uint64_t)0x000000000000FFFF) + | ((x & (uint64_t)0x00000000FFF00000) >> 4) + | ((x & (uint64_t)0x00000000000F0000) << 12) + | ((x & (uint64_t)0x0000FF0000000000) >> 8) + | ((x & (uint64_t)0x000000FF00000000) << 8) + | ((x & (uint64_t)0xF000000000000000) >> 12) + | ((x & (uint64_t)0x0FFF000000000000) << 4); + } +} + +static inline uint64_t +rotr32(uint64_t x) +{ + return (x << 32) | (x >> 32); +} + +static inline void +mix_columns(uint64_t *q) +{ + uint64_t q0, q1, q2, q3, q4, q5, q6, q7; + uint64_t r0, r1, r2, r3, r4, r5, r6, r7; + + q0 = q[0]; + q1 = q[1]; + q2 = q[2]; + q3 = q[3]; + q4 = q[4]; + q5 = q[5]; + q6 = q[6]; + q7 = q[7]; + r0 = (q0 >> 16) | (q0 << 48); + r1 = (q1 >> 16) | (q1 << 48); + r2 = (q2 >> 16) | (q2 << 48); + r3 = (q3 >> 16) | (q3 << 48); + r4 = (q4 >> 16) | (q4 << 48); + r5 = (q5 >> 16) | (q5 << 48); + r6 = (q6 >> 16) | (q6 << 48); + r7 = (q7 >> 16) | (q7 << 48); + + q[0] = q7 ^ r7 ^ r0 ^ rotr32(q0 ^ r0); + q[1] = q0 ^ r0 ^ q7 ^ r7 ^ r1 ^ rotr32(q1 ^ r1); + q[2] = q1 ^ r1 ^ r2 ^ rotr32(q2 ^ r2); + q[3] = q2 ^ r2 ^ q7 ^ r7 ^ r3 ^ rotr32(q3 ^ r3); + q[4] = q3 ^ r3 ^ q7 ^ r7 ^ r4 ^ rotr32(q4 ^ r4); + q[5] = q4 ^ r4 ^ r5 ^ rotr32(q5 ^ r5); + q[6] = q5 ^ r5 ^ r6 ^ rotr32(q6 ^ r6); + q[7] = q6 ^ r6 ^ r7 ^ rotr32(q7 ^ r7); +} + +/* see inner.h */ +void +br_aes_ct64_bitslice_encrypt(unsigned num_rounds, + const uint64_t *skey, uint64_t *q) +{ + unsigned u; + + add_round_key(q, skey); + for (u = 1; u < num_rounds; u ++) { + br_aes_ct64_bitslice_Sbox(q); + shift_rows(q); + mix_columns(q); + add_round_key(q, skey + (u << 3)); + } + br_aes_ct64_bitslice_Sbox(q); + shift_rows(q); + add_round_key(q, skey + (num_rounds << 3)); +} diff --git a/cbits/bearssl/dec32le.c b/cbits/bearssl/dec32le.c new file mode 100644 --- /dev/null +++ b/cbits/bearssl/dec32le.c @@ -0,0 +1,38 @@ +/* + * Copyright (c) 2016 Thomas Pornin + * + * Permission is hereby granted, free of charge, to any person obtaining + * a copy of this software and associated documentation files (the + * "Software"), to deal in the Software without restriction, including + * without limitation the rights to use, copy, modify, merge, publish, + * distribute, sublicense, and/or sell copies of the Software, and to + * permit persons to whom the Software is furnished to do so, subject to + * the following conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#include "inner.h" + +/* see inner.h */ +void +br_range_dec32le(uint32_t *v, size_t num, const void *src) +{ + const unsigned char *buf; + + buf = src; + while (num -- > 0) { + *v ++ = br_dec32le(buf); + buf += 4; + } +} diff --git a/cbits/bearssl/ghash_ctmul64.c b/cbits/bearssl/ghash_ctmul64.c new file mode 100644 --- /dev/null +++ b/cbits/bearssl/ghash_ctmul64.c @@ -0,0 +1,154 @@ +/* + * Copyright (c) 2016 Thomas Pornin + * + * Permission is hereby granted, free of charge, to any person obtaining + * a copy of this software and associated documentation files (the + * "Software"), to deal in the Software without restriction, including + * without limitation the rights to use, copy, modify, merge, publish, + * distribute, sublicense, and/or sell copies of the Software, and to + * permit persons to whom the Software is furnished to do so, subject to + * the following conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#include "inner.h" + +/* + * This is the 64-bit variant of br_ghash_ctmul32(), with 64-bit operands + * and bit reversal of 64-bit words. + */ + +static inline uint64_t +bmul64(uint64_t x, uint64_t y) +{ + uint64_t x0, x1, x2, x3; + uint64_t y0, y1, y2, y3; + uint64_t z0, z1, z2, z3; + + x0 = x & (uint64_t)0x1111111111111111; + x1 = x & (uint64_t)0x2222222222222222; + x2 = x & (uint64_t)0x4444444444444444; + x3 = x & (uint64_t)0x8888888888888888; + y0 = y & (uint64_t)0x1111111111111111; + y1 = y & (uint64_t)0x2222222222222222; + y2 = y & (uint64_t)0x4444444444444444; + y3 = y & (uint64_t)0x8888888888888888; + z0 = (x0 * y0) ^ (x1 * y3) ^ (x2 * y2) ^ (x3 * y1); + z1 = (x0 * y1) ^ (x1 * y0) ^ (x2 * y3) ^ (x3 * y2); + z2 = (x0 * y2) ^ (x1 * y1) ^ (x2 * y0) ^ (x3 * y3); + z3 = (x0 * y3) ^ (x1 * y2) ^ (x2 * y1) ^ (x3 * y0); + z0 &= (uint64_t)0x1111111111111111; + z1 &= (uint64_t)0x2222222222222222; + z2 &= (uint64_t)0x4444444444444444; + z3 &= (uint64_t)0x8888888888888888; + return z0 | z1 | z2 | z3; +} + +static uint64_t +rev64(uint64_t x) +{ +#define RMS(m, s) do { \ + x = ((x & (uint64_t)(m)) << (s)) \ + | ((x >> (s)) & (uint64_t)(m)); \ + } while (0) + + RMS(0x5555555555555555, 1); + RMS(0x3333333333333333, 2); + RMS(0x0F0F0F0F0F0F0F0F, 4); + RMS(0x00FF00FF00FF00FF, 8); + RMS(0x0000FFFF0000FFFF, 16); + return (x << 32) | (x >> 32); + +#undef RMS +} + +/* see bearssl_ghash.h */ +void +br_ghash_ctmul64(void *y, const void *h, const void *data, size_t len) +{ + const unsigned char *buf, *hb; + unsigned char *yb; + uint64_t y0, y1; + uint64_t h0, h1, h2, h0r, h1r, h2r; + + buf = data; + yb = y; + hb = h; + y1 = br_dec64be(yb); + y0 = br_dec64be(yb + 8); + h1 = br_dec64be(hb); + h0 = br_dec64be(hb + 8); + h0r = rev64(h0); + h1r = rev64(h1); + h2 = h0 ^ h1; + h2r = h0r ^ h1r; + while (len > 0) { + const unsigned char *src; + unsigned char tmp[16]; + uint64_t y0r, y1r, y2, y2r; + uint64_t z0, z1, z2, z0h, z1h, z2h; + uint64_t v0, v1, v2, v3; + + if (len >= 16) { + src = buf; + buf += 16; + len -= 16; + } else { + memcpy(tmp, buf, len); + memset(tmp + len, 0, (sizeof tmp) - len); + src = tmp; + len = 0; + } + y1 ^= br_dec64be(src); + y0 ^= br_dec64be(src + 8); + + y0r = rev64(y0); + y1r = rev64(y1); + y2 = y0 ^ y1; + y2r = y0r ^ y1r; + + z0 = bmul64(y0, h0); + z1 = bmul64(y1, h1); + z2 = bmul64(y2, h2); + z0h = bmul64(y0r, h0r); + z1h = bmul64(y1r, h1r); + z2h = bmul64(y2r, h2r); + z2 ^= z0 ^ z1; + z2h ^= z0h ^ z1h; + z0h = rev64(z0h) >> 1; + z1h = rev64(z1h) >> 1; + z2h = rev64(z2h) >> 1; + + v0 = z0; + v1 = z0h ^ z2; + v2 = z1 ^ z2h; + v3 = z1h; + + v3 = (v3 << 1) | (v2 >> 63); + v2 = (v2 << 1) | (v1 >> 63); + v1 = (v1 << 1) | (v0 >> 63); + v0 = (v0 << 1); + + v2 ^= v0 ^ (v0 >> 1) ^ (v0 >> 2) ^ (v0 >> 7); + v1 ^= (v0 << 63) ^ (v0 << 62) ^ (v0 << 57); + v3 ^= v1 ^ (v1 >> 1) ^ (v1 >> 2) ^ (v1 >> 7); + v2 ^= (v1 << 63) ^ (v1 << 62) ^ (v1 << 57); + + y0 = v2; + y1 = v3; + } + + br_enc64be(yb, y1); + br_enc64be(yb + 8, y0); +} diff --git a/cbits/bearssl/import.sh b/cbits/bearssl/import.sh new file mode 100644 --- /dev/null +++ b/cbits/bearssl/import.sh @@ -0,0 +1,34 @@ +#!/bin/sh +# Re-import the vendored parts of BearSSL. +# +# Only the five files crypton calls are kept, and they are kept unmodified, +# so that `diff` against a new release is readable. What they want from +# upstream's two-thousand-line src/inner.h is supplied by the inner.h here, +# which is crypton's and is NOT overwritten by this script. Run this from +# cbits/bearssl: +# +# ./import.sh [version] +# +# and commit the result together with the VERSION line it writes, so that +# the tree always says which upstream release it holds. +set -eu + +VER=${1:-0.6} +HERE=$(cd "$(dirname "$0")" && pwd) +TMP=$(mktemp -d) +trap 'rm -rf "$TMP"' EXIT + +curl -sSL "https://bearssl.org/bearssl-$VER.tar.gz" -o "$TMP/b.tgz" +tar xzf "$TMP/b.tgz" -C "$TMP" +SRC="$TMP/bearssl-$VER" + +cp "$SRC/LICENSE.txt" "$HERE/LICENSE" +cp "$SRC/src/symcipher/aes_ct64.c" "$HERE/" +cp "$SRC/src/symcipher/aes_ct64_enc.c" "$HERE/" +cp "$SRC/src/symcipher/aes_ct64_dec.c" "$HERE/" +cp "$SRC/src/hash/ghash_ctmul64.c" "$HERE/" +cp "$SRC/src/codec/dec32le.c" "$HERE/" +echo "$VER" > "$HERE/VERSION" + +echo "imported BearSSL $VER" +echo "now run the differential test in cbits/tests/bearssl_diff.c" diff --git a/cbits/bearssl/inner.h b/cbits/bearssl/inner.h new file mode 100644 --- /dev/null +++ b/cbits/bearssl/inner.h @@ -0,0 +1,92 @@ +/* + * Stands in for BearSSL's own src/inner.h. + * + * The five .c files beside this one are upstream's, byte for byte, and each + * of them opens with #include "inner.h". Upstream's is some two thousand + * lines and declares the whole library; these five want six things from it. + * So this header gives those six and nothing else, and the upstream files + * stay unmodified -- which is what makes `diff` against a new BearSSL + * release readable. See README.md. + * + * The byte-order helpers are written here rather than copied, in their plain + * portable form without upstream's unaligned-access fast paths, so that they + * carry no platform configuration with them. cbits/tests/bearssl_diff.c + * checks the whole thing against crypton's existing implementation. + */ +#ifndef CRYPTON_BEARSSL_INNER_H +#define CRYPTON_BEARSSL_INNER_H + +#include +#include +#include + +static inline uint32_t +br_dec32le(const void *src) +{ + const unsigned char *b = src; + + return (uint32_t)b[0] + | ((uint32_t)b[1] << 8) + | ((uint32_t)b[2] << 16) + | ((uint32_t)b[3] << 24); +} + +static inline void +br_enc32le(void *dst, uint32_t x) +{ + unsigned char *b = dst; + + b[0] = (unsigned char)x; + b[1] = (unsigned char)(x >> 8); + b[2] = (unsigned char)(x >> 16); + b[3] = (unsigned char)(x >> 24); +} + +static inline uint64_t +br_dec64be(const void *src) +{ + const unsigned char *b = src; + uint64_t x = 0; + int i; + + for (i = 0; i < 8; i++) + x = (x << 8) | (uint64_t)b[i]; + return x; +} + +static inline void +br_enc64be(void *dst, uint64_t x) +{ + unsigned char *b = dst; + int i; + + for (i = 7; i >= 0; i--) { + b[i] = (unsigned char)(x & 0xff); + x >>= 8; + } +} + +/* dec32le.c */ +void br_range_dec32le(uint32_t *v, size_t num, const void *src); + +/* aes_ct64.c */ +void br_aes_ct64_bitslice_Sbox(uint64_t *q); +void br_aes_ct64_ortho(uint64_t *q); +void br_aes_ct64_interleave_in(uint64_t *q0, uint64_t *q1, const uint32_t *w); +void br_aes_ct64_interleave_out(uint32_t *w, uint64_t q0, uint64_t q1); +unsigned br_aes_ct64_keysched(uint64_t *comp_skey, const void *key, + size_t key_len); +void br_aes_ct64_skey_expand(uint64_t *skey, unsigned num_rounds, + const uint64_t *comp_skey); + +/* aes_ct64_enc.c, aes_ct64_dec.c */ +void br_aes_ct64_bitslice_encrypt(unsigned num_rounds, const uint64_t *skey, + uint64_t *q); +void br_aes_ct64_bitslice_invSbox(uint64_t *q); +void br_aes_ct64_bitslice_decrypt(unsigned num_rounds, const uint64_t *skey, + uint64_t *q); + +/* ghash_ctmul64.c */ +void br_ghash_ctmul64(void *y, const void *h, const void *data, size_t len); + +#endif diff --git a/cbits/crypton_aes.c b/cbits/crypton_aes.c --- a/cbits/crypton_aes.c +++ b/cbits/crypton_aes.c @@ -38,6 +38,9 @@ #include #include #include +#ifdef WITH_PPC8_CRYPTO +#include +#endif #ifdef WITH_GCM_FUSED #include #endif @@ -54,6 +57,8 @@ uint32_t spoint, aes_block *input, uint32_t nb_blocks); void crypton_aes_generic_gcm_encrypt(uint8_t *output, aes_gcm *gcm, aes_key *key, uint8_t *input, uint32_t length); void crypton_aes_generic_gcm_decrypt(uint8_t *output, aes_gcm *gcm, aes_key *key, uint8_t *input, uint32_t length); +void crypton_aes_bitsliced_gcm_encrypt(uint8_t *output, aes_gcm *gcm, aes_key *key, uint8_t *input, uint32_t length); +void crypton_aes_bitsliced_gcm_decrypt(uint8_t *output, aes_gcm *gcm, aes_key *key, uint8_t *input, uint32_t length); void crypton_aes_generic_ocb_encrypt(uint8_t *output, aes_ocb *ocb, aes_key *key, uint8_t *input, uint32_t length); void crypton_aes_generic_ocb_decrypt(uint8_t *output, aes_ocb *ocb, aes_key *key, uint8_t *input, uint32_t length); void crypton_aes_generic_ccm_encrypt(uint8_t *output, aes_ccm *ccm, aes_key *key, uint8_t *input, uint32_t length); @@ -208,7 +213,7 @@ typedef void (*gf_mul_f)(block128 *a, const table_4bit htable); typedef void (*gf_mul4_f)(block128 *a, const block128 *blocks, const table_4bit htable); -#if defined(WITH_AESNI) || defined(WITH_ARMV8_CRYPTO) +#if defined(WITH_AESNI) || defined(WITH_ARMV8_CRYPTO) || defined(WITH_PPC8_CRYPTO) #define GET_INIT(strength) \ ((init_f) (crypton_aes_branch_table[INIT_128 + strength])) #define GET_ECB_ENCRYPT(strength) \ @@ -255,12 +260,16 @@ #define GET_ECB_DECRYPT(strength) crypton_aes_generic_decrypt_ecb #define GET_CBC_ENCRYPT(strength) crypton_aes_generic_encrypt_cbc #define GET_CBC_DECRYPT(strength) crypton_aes_generic_decrypt_cbc -#define GET_CTR_ENCRYPT(strength) crypton_aes_generic_encrypt_ctr -#define GET_C32_ENCRYPT(strength) crypton_aes_generic_encrypt_c32 +/* No accelerator is compiled in, so the portable schedule is the only one + * a key can hold and the four-block CTR is named here rather than installed + * at run time. This is the build ppc64le, s390x, riscv64, 32-bit ARM and + * -f-support_aesni all get, and nothing in it reads the branch table. */ +#define GET_CTR_ENCRYPT(strength) crypton_aes_bitsliced_encrypt_ctr +#define GET_C32_ENCRYPT(strength) crypton_aes_bitsliced_encrypt_c32 #define GET_XTS_ENCRYPT(strength) crypton_aes_generic_encrypt_xts #define GET_XTS_DECRYPT(strength) crypton_aes_generic_decrypt_xts -#define GET_GCM_ENCRYPT(strength) crypton_aes_generic_gcm_encrypt -#define GET_GCM_DECRYPT(strength) crypton_aes_generic_gcm_decrypt +#define GET_GCM_ENCRYPT(strength) crypton_aes_bitsliced_gcm_encrypt +#define GET_GCM_DECRYPT(strength) crypton_aes_bitsliced_gcm_decrypt #define GET_OCB_ENCRYPT(strength) crypton_aes_generic_ocb_encrypt #define GET_OCB_DECRYPT(strength) crypton_aes_generic_ocb_decrypt #define GET_CCM_ENCRYPT(strength) crypton_aes_generic_ccm_encrypt @@ -272,10 +281,6 @@ #define crypton_gf_mul4(a,b,t) crypton_aes_generic_gf_mul4(a,b,t) #endif -#define CPU_AESNI 0 -#define CPU_PCLMUL 1 -#define CPU_OPTION_COUNT 2 - static uint8_t crypton_aes_cpu_options[CPU_OPTION_COUNT] = {}; #if defined(ARCH_X86) && defined(WITH_AESNI) @@ -440,6 +445,74 @@ * A constructor runs while there is one thread, which is the cheapest way to * have no race at all: no flag to test, no lock to take, and one less thing * for crypton_aes_initkey to do per key. */ +/* + * The two CTR entries, where the portable implementation is the one that + * will run. This is for the build that has an accelerator compiled in and + * did not find it on the processor; a build with no accelerator at all + * names them directly in the GET_ macros above and never reads this table. + * + * They cannot be the defaults. crypton_aes_generic_encrypt_c32 is not + * overridden on AArch64 -- the ARMv8 table has no C32 of its own -- and + * neither CCM nor OCB is overridden anywhere, so those reach the block + * function through the branch table and work whatever is installed. A CTR + * that read the portable schedule directly would be wrong on exactly those + * machines. So these go in only once nothing has claimed AES. + */ +static void initialize_table_bitsliced(void) +{ + crypton_aes_branch_table[ENCRYPT_CTR_128] = crypton_aes_bitsliced_encrypt_ctr; + crypton_aes_branch_table[ENCRYPT_CTR_192] = crypton_aes_bitsliced_encrypt_ctr; + crypton_aes_branch_table[ENCRYPT_CTR_256] = crypton_aes_bitsliced_encrypt_ctr; + crypton_aes_branch_table[ENCRYPT_C32_128] = crypton_aes_bitsliced_encrypt_c32; + crypton_aes_branch_table[ENCRYPT_C32_192] = crypton_aes_bitsliced_encrypt_c32; + crypton_aes_branch_table[ENCRYPT_C32_256] = crypton_aes_bitsliced_encrypt_c32; + crypton_aes_branch_table[ENCRYPT_GCM_128] = crypton_aes_bitsliced_gcm_encrypt; + crypton_aes_branch_table[ENCRYPT_GCM_192] = crypton_aes_bitsliced_gcm_encrypt; + crypton_aes_branch_table[ENCRYPT_GCM_256] = crypton_aes_bitsliced_gcm_encrypt; + crypton_aes_branch_table[DECRYPT_GCM_128] = crypton_aes_bitsliced_gcm_decrypt; + crypton_aes_branch_table[DECRYPT_GCM_192] = crypton_aes_bitsliced_gcm_decrypt; + crypton_aes_branch_table[DECRYPT_GCM_256] = crypton_aes_bitsliced_gcm_decrypt; +} + +#ifdef WITH_PPC8_CRYPTO +/* + * POWER8. One function per operation rather than three: the assembly reads + * the round count out of the key, so the same entry serves every key size. + * + * GCM and the 32-bit counter are left where they are. Neither has an entry + * in the assembly, and the generic loops reach the block function and the + * GHASH through this table, so both get the instructions anyway. + */ +static void initialize_table_ppc8(void) +{ + int sz; + + if (!crypton_aes_ppc8_available()) + return; + /* what stops initialize_table_bitsliced below from putting its own CTR + * in, which would read this key as a bitsliced schedule */ + crypton_aes_cpu_options[CPU_AESNI] = 1; + crypton_aes_cpu_options[CPU_PCLMUL] = 1; + + for (sz = 0; sz < 3; sz++) { + crypton_aes_branch_table[INIT_128 + sz] = crypton_aes_ppc8_init; + crypton_aes_branch_table[ENCRYPT_BLOCK_128 + sz] = crypton_aes_ppc8_encrypt_block; + crypton_aes_branch_table[DECRYPT_BLOCK_128 + sz] = crypton_aes_ppc8_decrypt_block; + crypton_aes_branch_table[ENCRYPT_ECB_128 + sz] = crypton_aes_ppc8_encrypt_ecb; + crypton_aes_branch_table[DECRYPT_ECB_128 + sz] = crypton_aes_ppc8_decrypt_ecb; + crypton_aes_branch_table[ENCRYPT_CBC_128 + sz] = crypton_aes_ppc8_encrypt_cbc; + crypton_aes_branch_table[DECRYPT_CBC_128 + sz] = crypton_aes_ppc8_decrypt_cbc; + crypton_aes_branch_table[ENCRYPT_CTR_128 + sz] = crypton_aes_ppc8_encrypt_ctr; + crypton_aes_branch_table[ENCRYPT_XTS_128 + sz] = crypton_aes_ppc8_encrypt_xts; + crypton_aes_branch_table[DECRYPT_XTS_128 + sz] = crypton_aes_ppc8_decrypt_xts; + } + + crypton_aes_branch_table[GHASH_HINIT] = crypton_aes_ppc8_hinit; + crypton_aes_branch_table[GHASH_GF_MUL] = crypton_aes_ppc8_gf_mul; + crypton_aes_branch_table[GHASH_GF_MUL4] = crypton_aes_ppc8_gf_mul4; +} +#endif + static void crypton_aes_cpu_setup(void) { #if defined(ARCH_X86) && defined(WITH_AESNI) @@ -448,6 +521,11 @@ #ifdef WITH_ARMV8_CRYPTO initialize_table_armv8(); #endif +#ifdef WITH_PPC8_CRYPTO + initialize_table_ppc8(); +#endif + if (crypton_aes_cpu_options[CPU_AESNI] == 0) + initialize_table_bitsliced(); } __attribute__((constructor)) @@ -1136,18 +1214,30 @@ block128_xor((block128 *) tag, &ocb->sum_aad); } +/* + * These three reach cbits/aes/generic.c directly rather than through the + * branch table, so they run only where the portable implementation was the + * one installed -- which is what lets them read its schedule and hand its + * core four blocks at a time. The CTR entries below cannot do the same: + * they go through the branch table for the block, and so run on accelerated + * machines too. + */ void crypton_aes_generic_encrypt_ecb(aes_block *output, aes_key *key, aes_block *input, uint32_t nb_blocks) { - for ( ; nb_blocks-- > 0; input++, output++) { - crypton_aes_generic_encrypt_block(output, key, input); - } + aes_sched sched; + + crypton_aes_generic_schedule(&sched, key); + crypton_aes_generic_blocks((uint8_t *) output, (const uint8_t *) input, + nb_blocks, &sched, 0); } void crypton_aes_generic_decrypt_ecb(aes_block *output, aes_key *key, aes_block *input, uint32_t nb_blocks) { - for ( ; nb_blocks-- > 0; input++, output++) { - crypton_aes_generic_decrypt_block(output, key, input); - } + aes_sched sched; + + crypton_aes_generic_schedule(&sched, key); + crypton_aes_generic_blocks((uint8_t *) output, (const uint8_t *) input, + nb_blocks, &sched, 1); } void crypton_aes_generic_encrypt_cbc(aes_block *output, aes_key *key, aes_block *iv, aes_block *input, uint32_t nb_blocks) @@ -1163,18 +1253,35 @@ } } +/* + * Decryption, unlike encryption, does not wait for the block before it: + * every ciphertext block is ready at once and the chaining is a XOR + * afterwards. So four at a pass, with the ciphertext copied aside first, + * since the caller is allowed to decrypt in place and the next group's IV + * is the last ciphertext block of this one. + */ void crypton_aes_generic_decrypt_cbc(aes_block *output, aes_key *key, aes_block *ivini, aes_block *input, uint32_t nb_blocks) { - aes_block block, blocko; + aes_sched sched; aes_block iv; + uint8_t ct[64], plain[64]; - /* preload IV in block */ + crypton_aes_generic_schedule(&sched, key); block128_copy(&iv, ivini); - for ( ; nb_blocks-- > 0; input++, output++) { - block128_copy(&block, (block128 *) input); - crypton_aes_generic_decrypt_block(&blocko, key, &block); - block128_vxor((block128 *) output, &blocko, &iv); - block128_copy(&iv, &block); + while (nb_blocks > 0) { + uint32_t n = nb_blocks < 4 ? nb_blocks : 4; + uint32_t i; + + memcpy(ct, input, n * 16); + crypton_aes_generic_blocks(plain, ct, n, &sched, 1); + for (i = 0; i < n; i++) { + block128_vxor((block128 *) (output + i), + (block128 *) (plain + 16 * i), &iv); + block128_copy(&iv, (block128 *) (ct + 16 * i)); + } + input += n; + output += n; + nb_blocks -= n; } } @@ -1229,7 +1336,9 @@ void crypton_aes_generic_encrypt_xts(aes_block *output, aes_key *k1, aes_key *k2, aes_block *dataunit, uint32_t spoint, aes_block *input, uint32_t nb_blocks) { - aes_block block, tweak; + aes_sched sched; + aes_block tweak; + uint8_t buf[64], tw[64]; /* load IV and encrypt it using k2 as the tweak */ block128_copy(&tweak, dataunit); @@ -1239,17 +1348,34 @@ while (spoint-- > 0) crypton_aes_generic_gf_mulx(&tweak); - for ( ; nb_blocks-- > 0; input++, output++, crypton_aes_generic_gf_mulx(&tweak)) { - block128_vxor(&block, input, &tweak); - crypton_aes_encrypt_block(&block, k1, &block); - block128_vxor(output, &block, &tweak); + crypton_aes_generic_schedule(&sched, k1); + while (nb_blocks > 0) { + uint32_t n = nb_blocks < 4 ? nb_blocks : 4; + uint32_t i; + + /* the tweaks for a group depend on nothing but each other, so + * all four are known before any block is enciphered */ + for (i = 0; i < n; i++) { + block128_copy((block128 *) (tw + 16 * i), &tweak); + block128_vxor((block128 *) (buf + 16 * i), input + i, &tweak); + crypton_aes_generic_gf_mulx(&tweak); + } + crypton_aes_generic_blocks(buf, buf, n, &sched, 0); + for (i = 0; i < n; i++) + block128_vxor(output + i, (block128 *) (buf + 16 * i), + (block128 *) (tw + 16 * i)); + input += n; + output += n; + nb_blocks -= n; } } void crypton_aes_generic_decrypt_xts(aes_block *output, aes_key *k1, aes_key *k2, aes_block *dataunit, uint32_t spoint, aes_block *input, uint32_t nb_blocks) { - aes_block block, tweak; + aes_sched sched; + aes_block tweak; + uint8_t buf[64], tw[64]; /* load IV and encrypt it using k2 as the tweak */ block128_copy(&tweak, dataunit); @@ -1259,10 +1385,25 @@ while (spoint-- > 0) crypton_aes_generic_gf_mulx(&tweak); - for ( ; nb_blocks-- > 0; input++, output++, crypton_aes_generic_gf_mulx(&tweak)) { - block128_vxor(&block, input, &tweak); - crypton_aes_decrypt_block(&block, k1, &block); - block128_vxor(output, &block, &tweak); + crypton_aes_generic_schedule(&sched, k1); + while (nb_blocks > 0) { + uint32_t n = nb_blocks < 4 ? nb_blocks : 4; + uint32_t i; + + /* the tweaks for a group depend on nothing but each other, so + * all four are known before any block is enciphered */ + for (i = 0; i < n; i++) { + block128_copy((block128 *) (tw + 16 * i), &tweak); + block128_vxor((block128 *) (buf + 16 * i), input + i, &tweak); + crypton_aes_generic_gf_mulx(&tweak); + } + crypton_aes_generic_blocks(buf, buf, n, &sched, 1); + for (i = 0; i < n; i++) + block128_vxor(output + i, (block128 *) (buf + 16 * i), + (block128 *) (tw + 16 * i)); + input += n; + output += n; + nb_blocks -= n; } } @@ -1491,6 +1632,116 @@ block128_zero(&tmp); block128_copy_bytes(&tmp, input, length); ccm_cbcmac_add(ccm, key, &tmp); + } +} + +/* + * GCM where the portable implementation is the one that will run. + * + * The same shape as the two above, with two differences: the four counter + * blocks of a group are encrypted together, which is what the bitsliced + * core is for, and the schedule is expanded once for the message rather + * than once per block. + * + * These cannot replace the generic pair. That pair also runs on an x86 + * machine that has AES-NI and no carry-less multiply, where crypton_aes.c + * installs the AES-NI block function and leaves GCM alone -- and there the + * key holds the AES-NI schedule, not this one. So the choice is made where + * the rest of the portable entries are chosen. + */ +void crypton_aes_bitsliced_gcm_encrypt(uint8_t *output, aes_gcm *gcm, aes_key *key, uint8_t *input, uint32_t length) +{ + aes_sched sched; + aes_block out; + + crypton_aes_generic_schedule(&sched, key); + gcm->length_input += length; + for (; length >= 64; input += 64, output += 64, length -= 64) { + aes_block buf[4]; + int i; + + for (i = 0; i < 4; i++) { + block128_inc32_be(&gcm->civ); + block128_copy(&buf[i], &gcm->civ); + } + crypton_aes_generic_blocks((uint8_t *) buf, (const uint8_t *) buf, + 4, &sched, 0); + for (i = 0; i < 4; i++) + block128_xor(&buf[i], (block128 *) (input + 16 * i)); + gcm_ghash_add4(gcm, buf); + for (i = 0; i < 4; i++) + block128_copy((block128 *) (output + 16 * i), &buf[i]); + } + for (; length >= 16; input += 16, output += 16, length -= 16) { + block128_inc32_be(&gcm->civ); + crypton_aes_generic_blocks((uint8_t *) &out, + (const uint8_t *) &gcm->civ, 1, &sched, 0); + block128_xor(&out, (block128 *) input); + gcm_ghash_add(gcm, &out); + block128_copy((block128 *) output, &out); + } + if (length > 0) { + aes_block tmp; + uint32_t i; + + block128_inc32_be(&gcm->civ); + crypton_aes_generic_blocks((uint8_t *) &out, + (const uint8_t *) &gcm->civ, 1, &sched, 0); + block128_zero(&tmp); + block128_copy_bytes(&tmp, input, length); + block128_xor_bytes(&tmp, out.b, length); + gcm_ghash_add(gcm, &tmp); + for (i = 0; i < length; i++) + output[i] = tmp.b[i]; + } +} + +void crypton_aes_bitsliced_gcm_decrypt(uint8_t *output, aes_gcm *gcm, aes_key *key, uint8_t *input, uint32_t length) +{ + aes_sched sched; + aes_block out; + + crypton_aes_generic_schedule(&sched, key); + gcm->length_input += length; + /* GHASH all four ciphertext blocks before writing any plaintext, since + * output may be input */ + for (; length >= 64; input += 64, output += 64, length -= 64) { + aes_block buf[4]; + int i; + + gcm_ghash_add4(gcm, (const block128 *) input); + for (i = 0; i < 4; i++) { + block128_inc32_be(&gcm->civ); + block128_copy(&buf[i], &gcm->civ); + } + crypton_aes_generic_blocks((uint8_t *) buf, (const uint8_t *) buf, + 4, &sched, 0); + for (i = 0; i < 4; i++) { + block128_xor(&buf[i], (block128 *) (input + 16 * i)); + block128_copy((block128 *) (output + 16 * i), &buf[i]); + } + } + for (; length >= 16; input += 16, output += 16, length -= 16) { + block128_inc32_be(&gcm->civ); + crypton_aes_generic_blocks((uint8_t *) &out, + (const uint8_t *) &gcm->civ, 1, &sched, 0); + gcm_ghash_add(gcm, (block128 *) input); + block128_xor(&out, (block128 *) input); + block128_copy((block128 *) output, &out); + } + if (length > 0) { + aes_block tmp; + uint32_t i; + + block128_inc32_be(&gcm->civ); + crypton_aes_generic_blocks((uint8_t *) &out, + (const uint8_t *) &gcm->civ, 1, &sched, 0); + block128_zero(&tmp); + block128_copy_bytes(&tmp, input, length); + gcm_ghash_add(gcm, &tmp); + block128_xor_bytes(&tmp, out.b, length); + for (i = 0; i < length; i++) + output[i] = tmp.b[i]; } } diff --git a/cbits/crypton_armv8_target.h b/cbits/crypton_armv8_target.h --- a/cbits/crypton_armv8_target.h +++ b/cbits/crypton_armv8_target.h @@ -23,7 +23,19 @@ #define CRYPTON_ARMV8_TARGET_H #ifdef WITH_TARGET_ATTRIBUTES -#if defined(__clang__) +#if defined(__arm__) +/* + * AArch32 asks for an FPU rather than an architecture extension, and the + * spelling differs between the compilers in ways that cannot be checked from + * here. Rather than guess, the cabal file passes -mfpu=crypto-neon-fp-armv8 + * for the whole component on this architecture whatever this flag says, and + * these expand to nothing. A global target option is safe because a + * compiler emits these instructions where an intrinsic asks for them and + * nowhere else. + */ +#define CRYPTON_TARGET_ARMV8_CRYPTO +#define CRYPTON_TARGET_ARMV8_SHA3 +#elif defined(__clang__) #define CRYPTON_TARGET_ARMV8_CRYPTO __attribute__((target("+crypto"))) #define CRYPTON_TARGET_ARMV8_SHA3 __attribute__((target("+sha3"))) #else diff --git a/cbits/crypton_bignum.h b/cbits/crypton_bignum.h --- a/cbits/crypton_bignum.h +++ b/cbits/crypton_bignum.h @@ -29,11 +29,6 @@ #define LIMB_BYTES (LIMB_BITS / 8) -/* four bits of exponent per window, so a table of sixteen and no leftover - * bits: a byte holds exactly two windows */ -#define WINDOW_BITS 4 -#define TABLE_SIZE (1 << WINDOW_BITS) - /* r = a - b, returning the borrow out of the top */ static inline limb_t sub_n(limb_t *r, const limb_t *a, const limb_t *b, uint32_t n) { @@ -444,7 +439,20 @@ * What the trip counts depend on is the modulus' length, which is what the * doubling loop showed as well, and nothing else about it. */ -static inline void mont_r2(limb_t *r2, const limb_t *m, limb_t n0, uint32_t n, +/* s2n-bignum's bignum_montifier computes the same constant in assembly, on + * both architectures, and is the faster of the two by a good margin -- the + * C cannot form the carry chains it does. It is used where it is built, + * and the C below is what everywhere else gets and what it is checked + * against. It needs a scratch buffer of n limbs, which every caller + * already passes. */ +#if defined(CRYPTON_S2N_BIGNUM) && !defined(CRYPTON_NO_MONTIFIER) \ + && (defined(__x86_64__) || defined(__aarch64__)) && LIMB_BITS == 64 +#define CRYPTON_MONTIFIER 1 +extern void bignum_montifier(uint64_t k, uint64_t *z, const uint64_t *m, + uint64_t *t); +#endif + +static inline void mont_r2_c(limb_t *r2, const limb_t *m, limb_t n0, uint32_t n, limb_t *t) { uint32_t lgr = n * LIMB_BITS; @@ -475,6 +483,17 @@ if ((lgr >> (i - 1)) & 1) dbl_mod(r2, m, n, t); } +} + +static inline void mont_r2(limb_t *r2, const limb_t *m, limb_t n0, uint32_t n, + limb_t *t) +{ +#ifdef CRYPTON_MONTIFIER + (void) n0; + bignum_montifier(n, r2, m, t); +#else + mont_r2_c(r2, m, n0, n, t); +#endif } /* big-endian bytes into limbs, least significant limb first; anything above diff --git a/cbits/crypton_bitfn.h b/cbits/crypton_bitfn.h --- a/cbits/crypton_bitfn.h +++ b/cbits/crypton_bitfn.h @@ -127,23 +127,6 @@ } #endif -#ifndef ARCH_HAS_MEMORY_ZERO -static inline void memory_zero(void *ptr, uint32_t len) -{ - uint32_t *ptr32 = ptr; - uint8_t *ptr8; - int i; - - for (i = 0; i < len / 4; i++) - *ptr32++ = 0; - if (len % 4) { - ptr8 = (uint8_t *) ptr32; - for (i = len % 4; i >= 0; i--) - ptr8[i] = 0; - } -} -#endif - #ifndef ARCH_HAS_ARRAY_COPY32 static inline void array_copy32(uint32_t *d, uint32_t *s, uint32_t nb) { diff --git a/cbits/crypton_chacha.c b/cbits/crypton_chacha.c --- a/cbits/crypton_chacha.c +++ b/cbits/crypton_chacha.c @@ -109,8 +109,9 @@ #define ALIGNED64(PTR) \ (((uintptr_t)(const void *)(PTR)) % 8 == 0) -static const uint8_t sigma[16] = "expand 32-byte k"; -static const uint8_t tau[16] = "expand 16-byte k"; +/* sixteen bytes are read; the seventeenth is the string's terminator */ +static const uint8_t sigma[17] = "expand 32-byte k"; +static const uint8_t tau[17] = "expand 16-byte k"; static void chacha_core(int rounds, crypton_chacha_block *out, const crypton_chacha_state *in) { @@ -384,7 +385,7 @@ } /* xor as much as needed */ - for (i = 0; i < bytes; i++) + for (i = 0; i < (int) bytes; i++) dst[i] = src[i] ^ out.b[i]; /* copy the left over in the buffer */ @@ -510,7 +511,7 @@ } /* xor as much as needed */ - for (i = 0; i < bytes; i++) + for (i = 0; i < (int) bytes; i++) dst[i] = out.b[i]; /* copy the left over in the buffer */ diff --git a/cbits/crypton_cpu.c b/cbits/crypton_cpu.c --- a/cbits/crypton_cpu.c +++ b/cbits/crypton_cpu.c @@ -60,7 +60,7 @@ CRYPTON_ARMCAP_NEON; #endif -#if defined(__aarch64__) +#if defined(__aarch64__) || defined(__arm__) #if defined(__APPLE__) #include #elif defined(__linux__) @@ -71,14 +71,19 @@ #endif /* - * The HWCAP bit positions are the ARM ELF ABI's, so every system that - * reports through the auxiliary vector agrees on them. What differs is - * AT_HWCAP itself -- 16 on Linux, 25 on FreeBSD -- and that comes from each - * system's own header, which is why the tag is never written out here. - * Defined only where the system's headers did not define them, the way - * compiler-rt does it, so that a system which reports through the auxiliary - * vector without shipping the ARM names still compiles. + * The bit positions are the ARM ELF ABI's, so every system that reports + * through the auxiliary vector agrees on them. What differs is the tag -- + * AT_HWCAP is 16 on Linux and 25 on FreeBSD -- and that comes from each + * system's own header, which is why no tag is written out here. Defined + * only where the system's headers did not define them, the way compiler-rt + * does it, so that a system which reports through the auxiliary vector + * without shipping the ARM names still compiles. + * + * The two execution states do not share a word or an order. AArch64 puts + * these in AT_HWCAP; AArch32 has filled that word with older features and + * puts the cryptographic ones in AT_HWCAP2, starting again from bit zero. */ +#if defined(__aarch64__) #ifndef HWCAP_AES #define HWCAP_AES (1 << 3) #endif @@ -97,6 +102,20 @@ #ifndef HWCAP_SHA512 #define HWCAP_SHA512 (1 << 21) #endif +#else +#ifndef HWCAP2_AES +#define HWCAP2_AES (1 << 0) +#endif +#ifndef HWCAP2_PMULL +#define HWCAP2_PMULL (1 << 1) +#endif +#ifndef HWCAP2_SHA1 +#define HWCAP2_SHA1 (1 << 2) +#endif +#ifndef HWCAP2_SHA2 +#define HWCAP2_SHA2 (1 << 3) +#endif +#endif #if defined(__APPLE__) static int apple_has(const char *name) @@ -145,7 +164,7 @@ f |= CRYPTON_ARM_SHA512; if (apple_has("hw.optional.arm.FEAT_SHA3")) f |= CRYPTON_ARM_SHA3; -#else +#elif defined(__aarch64__) unsigned long cap = 0; #if defined(__linux__) @@ -160,14 +179,103 @@ if (cap & HWCAP_SHA2) f |= CRYPTON_ARM_SHA2; if (cap & HWCAP_SHA512) f |= CRYPTON_ARM_SHA512; if (cap & HWCAP_SHA3) f |= CRYPTON_ARM_SHA3; +#else + /* AArch32: the second word, and nothing later than SHA-256 to + * ask about */ + unsigned long cap = 0; + +#if defined(__linux__) + cap = getauxval(AT_HWCAP2); +#elif defined(__FreeBSD__) + if (elf_aux_info(AT_HWCAP2, &cap, sizeof(cap)) != 0) + cap = 0; #endif + if (cap & HWCAP2_AES) f |= CRYPTON_ARM_AES; + if (cap & HWCAP2_PMULL) f |= CRYPTON_ARM_PMULL; + if (cap & HWCAP2_SHA1) f |= CRYPTON_ARM_SHA1; + if (cap & HWCAP2_SHA2) f |= CRYPTON_ARM_SHA2; +#endif features = f; resolved = 1; } return features; } -#endif /* __aarch64__ */ +#endif /* __aarch64__ || __arm__ */ +#if defined(__powerpc64__) || defined(__PPC64__) +#if defined(__linux__) +#include +#elif defined(__FreeBSD__) +#include + +#endif + +/* + * PPC_FEATURE2_VEC_CRYPTO, from the Power ELF ABI. Defined here only where + * the system's headers did not, so that a system reporting through the + * auxiliary vector without shipping the name still compiles. + */ +#ifndef PPC_FEATURE2_VEC_CRYPTO +#define PPC_FEATURE2_VEC_CRYPTO 0x02000000 +#endif + +unsigned int crypton_ppc_features(void) +{ + static unsigned int features; + static int resolved; + + if (!resolved) { + unsigned long cap = 0; + +#if defined(__linux__) + cap = getauxval(AT_HWCAP2); +#elif defined(__FreeBSD__) + if (elf_aux_info(AT_HWCAP2, &cap, sizeof(cap)) != 0) + cap = 0; +#endif + features = (cap & PPC_FEATURE2_VEC_CRYPTO) + ? CRYPTON_PPC_VCRYPTO : 0; + resolved = 1; + } + return features; +} +#endif /* __powerpc64__ */ + +/* + * For testing on a machine whose CPUID says less than its core executes, and + * for nothing else: each of these makes the detection below report a + * feature it did not find. + * + * CRYPTON_CPU_TESTING_ADX BMI2 and ADX + * CRYPTON_CPU_TESTING_SHA_NI the SHA extensions + * CRYPTON_CPU_TESTING_VAES VAES and VPCLMULQDQ, 256-bit + * + * A hypervisor may show a guest an older processor model than the host's so + * that the guest can be moved to another host, and the instructions it hides + * then still run. That is the only place this is for: forcing a path that + * CPUID would not choose, to measure it or to test it, on a machine where the + * instructions have been seen to execute. Anywhere they do not, the first + * use is a SIGILL -- which is the bug #317 fixed, made on purpose. + * + * What the operating system has to agree to is still asked. VAES works in + * the wide registers, so it is assumed only where AVX2 was found, which + * means XCR0 says those are saved; the SHA extensions still want SSSE3 and + * SSE4.1 to have been found, as the detection asks. Nothing 512-bit can be + * assumed: whether the operating system saves that state is not something a + * build can promise. + * + * The library is never built with these, and a build that has one says so. + */ +#if defined(CRYPTON_CPU_TESTING_ADX) || defined(CRYPTON_CPU_TESTING_SHA_NI) \ + || defined(CRYPTON_CPU_TESTING_VAES) +/* The code below asks for each switch by its own name, which comes from the + * command line and so is seen everywhere in the file. It once asked for one + * umbrella name defined beside the first function that read it -- and the + * capability vector the assembly reads is filled earlier in the file than + * that, so the SHA bit never reached it, without a word from the compiler. */ +#warning "CRYPTON_CPU_TESTING_*: features assumed, not detected" +#endif + #ifdef ARCH_X86 static void cpuid(uint32_t info, uint32_t *eax, uint32_t *ebx, uint32_t *ecx, uint32_t *edx) { @@ -294,6 +402,16 @@ } if (!(f & CRYPTON_X86_AVX2)) leaf7_ebx &= ~(1u << 5); + /* what the testing switches assumed, told to the assembly as + * well: its SHA-256 reads the SHA bit here, not the word */ +#ifdef CRYPTON_CPU_TESTING_SHA_NI + if (f & CRYPTON_X86_SHA_NI) + leaf7_ebx |= 1u << 29; +#endif +#ifdef CRYPTON_CPU_TESTING_ADX + if (f & CRYPTON_X86_ADX) + leaf7_ebx |= (1u << 8) | (1u << 19); +#endif crypton_ia32cap_P[2] = leaf7_ebx & ~IA32CAP_AVX512; resolved = 1; @@ -394,7 +512,26 @@ * AVX-512 at all and never reaches here. */ && !(amd && family == 0x19)) f |= CRYPTON_X86_VAES512; + /* IFMA (bit 21) with F (16) and VL (31), and the same + * AVX-512 state. Zen 4 is not left out here: RSA's + * multiplication uses the 256-bit form. */ + if ((f & CRYPTON_X86_AVX2) + && (ebx & (1 << 16)) && (ebx & (1 << 21)) + && (ebx & (1u << 31)) + && ((xcr0() & 0xe6) == 0xe6)) + f |= CRYPTON_X86_IFMA; } +#ifdef CRYPTON_CPU_TESTING_ADX + f |= CRYPTON_X86_ADX; +#endif +#ifdef CRYPTON_CPU_TESTING_SHA_NI + if ((leaf1 & (1 << 9)) && (leaf1 & (1 << 19))) + f |= CRYPTON_X86_SHA_NI; +#endif +#ifdef CRYPTON_CPU_TESTING_VAES + if (f & CRYPTON_X86_AVX2) + f |= CRYPTON_X86_VAES; +#endif features = f; resolved = 1; } @@ -421,3 +558,110 @@ #endif #endif + +/* + * What Crypto.System.CPU reports. + * + * The numbers are that module's, and this answers for them. The old shape + * was the other way round -- Haskell read crypton_aes.c's two-entry array by + * the Enum index of its own constructors -- which tied the list of names to + * the indices of an array that had no reason to grow, and so it never did. + * + * Nothing here is cached: every answer is either a static array filled by a + * constructor, a value resolved once and remembered by the function that + * owns it, or a runtime check that does its own remembering. Haskell calls + * this once, for a CAF. + */ +uint8_t *crypton_aes_cpu_init(void); + +#ifdef WITH_ARMV8_CRYPTO +int crypton_aes_armv8_available(void); +int crypton_aes_armv8_pmull_available(void); +#endif +#ifdef WITH_ARMV8_SHA1 +extern int crypton_sha1_armv8_available(void); +#endif +#ifdef WITH_ARMV8_SHA2 +extern int crypton_sha256_armv8_available(void); +#endif +#ifdef WITH_ARMV8_SHA512 +extern int crypton_sha512_armv8_available(void); +#endif + +/* Keep in step with Crypto/System/CPU.hs, which is where these are named. + * 2 is missing: it was RDRAND, which crypton no longer dispatches on. The + * number is left out rather than reused, so that the rest keep the values + * they had. */ +#define OPT_AESNI 0 +#define OPT_PCLMUL 1 +#define OPT_SSSE3 3 +#define OPT_AVX 4 +#define OPT_AVX2 5 +#define OPT_SHANI 6 +#define OPT_MOVBE 7 +#define OPT_ADX 8 +#define OPT_VAES 9 +#define OPT_VAES512 10 +#define OPT_NEON 11 +#define OPT_ARMAES 12 +#define OPT_ARMPMULL 13 +#define OPT_ARMSHA1 14 +#define OPT_ARMSHA2 15 +#define OPT_ARMSHA512 16 +#define OPT_PPCAES 17 +#define OPT_PPCVPMSUM 18 +#define OPT_IFMA 19 + +int crypton_cpu_option(unsigned int option) +{ +#ifdef ARCH_X86 + uint32_t f = crypton_x86_simd_features(); + + switch (option) { + case OPT_AESNI: return crypton_aes_cpu_init()[CPU_AESNI] != 0; + case OPT_PCLMUL: return crypton_aes_cpu_init()[CPU_PCLMUL] != 0; + case OPT_SSSE3: return (f & CRYPTON_X86_SSSE3) != 0; + case OPT_AVX: return (f & CRYPTON_X86_AVX) != 0; + case OPT_AVX2: return (f & CRYPTON_X86_AVX2) != 0; + case OPT_SHANI: return (f & CRYPTON_X86_SHA_NI) != 0; + case OPT_MOVBE: return (f & CRYPTON_X86_MOVBE) != 0; + case OPT_ADX: return (f & CRYPTON_X86_ADX) != 0; + case OPT_VAES: return (f & CRYPTON_X86_VAES) != 0; + case OPT_VAES512: return (f & CRYPTON_X86_VAES512) != 0; + case OPT_IFMA: return (f & CRYPTON_X86_IFMA) != 0; + default: return 0; + } +#else + switch (option) { + /* NEON is not optional on AArch64, and crypton_armcap_P is set from + * the start for it. Where there is no ARM assembly at all there is + * nothing to say. */ +#ifdef CRYPTON_ARM_ASM + case OPT_NEON: return (crypton_armcap_P & CRYPTON_ARMCAP_NEON) != 0; +#endif +#ifdef WITH_ARMV8_CRYPTO + /* The array rather than the check: it is what the branch table was + * actually built from, so it says what will run, not merely what the + * processor has. */ + case OPT_ARMAES: return crypton_aes_cpu_init()[CPU_AESNI] != 0; + case OPT_ARMPMULL: return crypton_aes_cpu_init()[CPU_PCLMUL] != 0; +#endif +#ifdef WITH_ARMV8_SHA1 + case OPT_ARMSHA1: return crypton_sha1_armv8_available() != 0; +#endif +#ifdef WITH_ARMV8_SHA2 + case OPT_ARMSHA2: return crypton_sha256_armv8_available() != 0; +#endif +#ifdef WITH_ARMV8_SHA512 + case OPT_ARMSHA512: return crypton_sha512_armv8_available() != 0; +#endif +#ifdef WITH_PPC8_CRYPTO + /* The array rather than the check, for the same reason as the ARM pair + * above: it says what the branch table was built from. */ + case OPT_PPCAES: return crypton_aes_cpu_init()[CPU_AESNI] != 0; + case OPT_PPCVPMSUM: return crypton_aes_cpu_init()[CPU_PCLMUL] != 0; +#endif + default: return 0; + } +#endif +} diff --git a/cbits/crypton_cpu.h b/cbits/crypton_cpu.h --- a/cbits/crypton_cpu.h +++ b/cbits/crypton_cpu.h @@ -65,6 +65,9 @@ * bits of XCR0 than AVX2 wants: the mask registers and the two upper halves * of the vector registers. */ #define CRYPTON_X86_VAES512 256 +/* AVX-512 IFMA, the 52-bit multiply-adds, with F and VL for the 256-bit form + * RSA uses them in: EVEX-encoded, so they want the AVX-512 state saved too. */ +#define CRYPTON_X86_IFMA 512 #ifdef ARCH_X86 uint32_t crypton_x86_simd_features(void); #endif @@ -107,8 +110,13 @@ * which systems someone had thought of. It is what left FreeBSD running * the table-driven AES and the C SHA on hardware that has the * instructions. In one place the next system is added once. + * + * AArch32 answers here too. It has the same AES, PMULL, SHA-1 and SHA-256 + * instructions, reports them in a different word of the auxiliary vector, + * and has no SHA-512 or SHA-3 to report at all -- those two are AArch64's + * alone, so the bits exist and are never set. */ -#if defined(__aarch64__) +#if defined(__aarch64__) || defined(__arm__) #define CRYPTON_ARM_AES (1u << 0) #define CRYPTON_ARM_PMULL (1u << 1) #define CRYPTON_ARM_SHA1 (1u << 2) @@ -117,6 +125,40 @@ #define CRYPTON_ARM_SHA3 (1u << 5) unsigned int crypton_arm_features(void); #endif + +/* + * Which of the optional PowerISA instruction sets this processor has. + * + * One bit so far: the vector AES and the vector carry-less multiply that + * PowerISA 2.07 brought and POWER8 was the first to implement. The system + * reports it in the second capability word, as AArch32 does, and with its + * own numbering again. + */ +#if defined(__powerpc64__) || defined(__PPC64__) +#define CRYPTON_PPC_VCRYPTO (1u << 0) +unsigned int crypton_ppc_features(void); +#endif + +/* + * The two slots of the array crypton_aes.c fills and crypton_aes_cpu_init + * hands out. Here rather than in that file because crypton_cpu.c reads + * them too, to answer for Crypto.System.CPU. + */ +#define CPU_AESNI 0 +#define CPU_PCLMUL 1 +#define CPU_OPTION_COUNT 2 + +/* + * One question at a time, numbered the way Crypto.System.CPU numbers its + * ProcessorOption rather than the way any array here is indexed. The + * numbering is the module's to choose and this answers for it; reading an + * array by the Haskell constructor's Enum index is what kept that type at + * three names. Returns 1 if the processor has it and crypton was built to + * use it, 0 otherwise -- including for anything this does not know, so an + * older library answers a newer caller rather than failing to link. + */ +int crypton_cpu_option(unsigned int option); + #ifdef USE_AESNI void crypton_aesni_initialize_hw(void (*init_table)(int, int)); #else diff --git a/cbits/crypton_mont.h b/cbits/crypton_mont.h new file mode 100644 --- /dev/null +++ b/cbits/crypton_mont.h @@ -0,0 +1,13445 @@ +/* Generated by cbits/crypton_mont.py; do not edit. + * + * Montgomery multiplication and squaring for AArch64 at 16, 24 and 32 limbs, + * the halves of RSA-2048, -3072 and -4096, and squaring at 48, RSA-3072's + * modulus. Each answers what mont_mul and mont_sqr answer. */ +#ifndef CRYPTON_MONT_H +#define CRYPTON_MONT_H + +#include "crypton_bignum.h" + +/* (t2:t1:t0) += x * y */ +#define MONT_MAC(t0, t1, t2, x, y) \ + do { \ + limb_t lo_, hi_; \ + __asm__("mul\t%0, %5, %6\n\t" \ + "umulh\t%1, %5, %6\n\t" \ + "adds\t%2, %2, %0\n\t" \ + "adcs\t%3, %3, %1\n\t" \ + "adc\t%4, %4, xzr" \ + : "=&r"(lo_), "=&r"(hi_), "+r"(t0), "+r"(t1), "+r"(t2) \ + : "r"(x), "r"(y) \ + : "cc"); \ + } while (0) + +/* (s2:s1:s0) = x * y */ +#define MONT_MUL(s0, s1, s2, x, y) \ + __asm__("mul\t%0, %3, %4\n\t" \ + "umulh\t%1, %3, %4\n\t" \ + "mov\t%2, xzr" \ + : "=&r"(s0), "=&r"(s1), "=r"(s2) \ + : "r"(x), "r"(y)) + +/* (t2:t1:t0) += (s2:s1:s0) */ +#define MONT_ADD(t0, t1, t2, s0, s1, s2) \ + __asm__("adds\t%0, %0, %3\n\t" \ + "adcs\t%1, %1, %4\n\t" \ + "adc\t%2, %2, %5" \ + : "+r"(t0), "+r"(t1), "+r"(t2) \ + : "r"(s0), "r"(s1), "r"(s2) \ + : "cc") + +/* (s2:s1:s0) *= 2 */ +#define MONT_DBL(s0, s1, s2) \ + do { \ + s2 = (s2 << 1) | (s1 >> 63); \ + s1 = (s1 << 1) | (s0 >> 63); \ + s0 <<= 1; \ + } while (0) + +/* r = a * b * R^-1 mod m, for a and b under m; t is 2n limbs of scratch */ +static void crypton_mont_mul16(limb_t *r, const limb_t *a, const limb_t *b, + const limb_t *m, limb_t n0, limb_t *t) +{ + limb_t *u = t, *z = t + 16; + limb_t t0 = 0, t1 = 0, t2 = 0, v0, v1, v2, w0, w1, w2, borrow, take; + + /* column 0 */ + MONT_MAC(t0, t1, t2, a[0], b[0]); + u[0] = t0 * n0; + MONT_MAC(t0, t1, t2, u[0], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 1 */ + MONT_MAC(t0, t1, t2, a[0], b[1]); + MONT_MUL(v0, v1, v2, a[1], b[0]); + MONT_MUL(w0, w1, w2, u[0], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[1] = t0 * n0; + MONT_MAC(t0, t1, t2, u[1], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 2 */ + MONT_MAC(t0, t1, t2, a[0], b[2]); + MONT_MUL(v0, v1, v2, a[1], b[1]); + MONT_MUL(w0, w1, w2, a[2], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[2]); + MONT_MAC(v0, v1, v2, u[1], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[2] = t0 * n0; + MONT_MAC(t0, t1, t2, u[2], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 3 */ + MONT_MAC(t0, t1, t2, a[0], b[3]); + MONT_MUL(v0, v1, v2, a[1], b[2]); + MONT_MUL(w0, w1, w2, a[2], b[1]); + MONT_MAC(t0, t1, t2, a[3], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[3]); + MONT_MAC(w0, w1, w2, u[1], m[2]); + MONT_MAC(t0, t1, t2, u[2], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[3] = t0 * n0; + MONT_MAC(t0, t1, t2, u[3], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 4 */ + MONT_MAC(t0, t1, t2, a[0], b[4]); + MONT_MUL(v0, v1, v2, a[1], b[3]); + MONT_MUL(w0, w1, w2, a[2], b[2]); + MONT_MAC(t0, t1, t2, a[3], b[1]); + MONT_MAC(v0, v1, v2, a[4], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[4]); + MONT_MAC(t0, t1, t2, u[1], m[3]); + MONT_MAC(v0, v1, v2, u[2], m[2]); + MONT_MAC(w0, w1, w2, u[3], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[4] = t0 * n0; + MONT_MAC(t0, t1, t2, u[4], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 5 */ + MONT_MAC(t0, t1, t2, a[0], b[5]); + MONT_MUL(v0, v1, v2, a[1], b[4]); + MONT_MUL(w0, w1, w2, a[2], b[3]); + MONT_MAC(t0, t1, t2, a[3], b[2]); + MONT_MAC(v0, v1, v2, a[4], b[1]); + MONT_MAC(w0, w1, w2, a[5], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[5]); + MONT_MAC(v0, v1, v2, u[1], m[4]); + MONT_MAC(w0, w1, w2, u[2], m[3]); + MONT_MAC(t0, t1, t2, u[3], m[2]); + MONT_MAC(v0, v1, v2, u[4], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[5] = t0 * n0; + MONT_MAC(t0, t1, t2, u[5], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 6 */ + MONT_MAC(t0, t1, t2, a[0], b[6]); + MONT_MUL(v0, v1, v2, a[1], b[5]); + MONT_MUL(w0, w1, w2, a[2], b[4]); + MONT_MAC(t0, t1, t2, a[3], b[3]); + MONT_MAC(v0, v1, v2, a[4], b[2]); + MONT_MAC(w0, w1, w2, a[5], b[1]); + MONT_MAC(t0, t1, t2, a[6], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[6]); + MONT_MAC(w0, w1, w2, u[1], m[5]); + MONT_MAC(t0, t1, t2, u[2], m[4]); + MONT_MAC(v0, v1, v2, u[3], m[3]); + MONT_MAC(w0, w1, w2, u[4], m[2]); + MONT_MAC(t0, t1, t2, u[5], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[6] = t0 * n0; + MONT_MAC(t0, t1, t2, u[6], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 7 */ + MONT_MAC(t0, t1, t2, a[0], b[7]); + MONT_MUL(v0, v1, v2, a[1], b[6]); + MONT_MUL(w0, w1, w2, a[2], b[5]); + MONT_MAC(t0, t1, t2, a[3], b[4]); + MONT_MAC(v0, v1, v2, a[4], b[3]); + MONT_MAC(w0, w1, w2, a[5], b[2]); + MONT_MAC(t0, t1, t2, a[6], b[1]); + MONT_MAC(v0, v1, v2, a[7], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[7]); + MONT_MAC(t0, t1, t2, u[1], m[6]); + MONT_MAC(v0, v1, v2, u[2], m[5]); + MONT_MAC(w0, w1, w2, u[3], m[4]); + MONT_MAC(t0, t1, t2, u[4], m[3]); + MONT_MAC(v0, v1, v2, u[5], m[2]); + MONT_MAC(w0, w1, w2, u[6], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[7] = t0 * n0; + MONT_MAC(t0, t1, t2, u[7], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 8 */ + MONT_MAC(t0, t1, t2, a[0], b[8]); + MONT_MUL(v0, v1, v2, a[1], b[7]); + MONT_MUL(w0, w1, w2, a[2], b[6]); + MONT_MAC(t0, t1, t2, a[3], b[5]); + MONT_MAC(v0, v1, v2, a[4], b[4]); + MONT_MAC(w0, w1, w2, a[5], b[3]); + MONT_MAC(t0, t1, t2, a[6], b[2]); + MONT_MAC(v0, v1, v2, a[7], b[1]); + MONT_MAC(w0, w1, w2, a[8], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[8]); + MONT_MAC(v0, v1, v2, u[1], m[7]); + MONT_MAC(w0, w1, w2, u[2], m[6]); + MONT_MAC(t0, t1, t2, u[3], m[5]); + MONT_MAC(v0, v1, v2, u[4], m[4]); + MONT_MAC(w0, w1, w2, u[5], m[3]); + MONT_MAC(t0, t1, t2, u[6], m[2]); + MONT_MAC(v0, v1, v2, u[7], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[8] = t0 * n0; + MONT_MAC(t0, t1, t2, u[8], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 9 */ + MONT_MAC(t0, t1, t2, a[0], b[9]); + MONT_MUL(v0, v1, v2, a[1], b[8]); + MONT_MUL(w0, w1, w2, a[2], b[7]); + MONT_MAC(t0, t1, t2, a[3], b[6]); + MONT_MAC(v0, v1, v2, a[4], b[5]); + MONT_MAC(w0, w1, w2, a[5], b[4]); + MONT_MAC(t0, t1, t2, a[6], b[3]); + MONT_MAC(v0, v1, v2, a[7], b[2]); + MONT_MAC(w0, w1, w2, a[8], b[1]); + MONT_MAC(t0, t1, t2, a[9], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[9]); + MONT_MAC(w0, w1, w2, u[1], m[8]); + MONT_MAC(t0, t1, t2, u[2], m[7]); + MONT_MAC(v0, v1, v2, u[3], m[6]); + MONT_MAC(w0, w1, w2, u[4], m[5]); + MONT_MAC(t0, t1, t2, u[5], m[4]); + MONT_MAC(v0, v1, v2, u[6], m[3]); + MONT_MAC(w0, w1, w2, u[7], m[2]); + MONT_MAC(t0, t1, t2, u[8], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[9] = t0 * n0; + MONT_MAC(t0, t1, t2, u[9], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 10 */ + MONT_MAC(t0, t1, t2, a[0], b[10]); + MONT_MUL(v0, v1, v2, a[1], b[9]); + MONT_MUL(w0, w1, w2, a[2], b[8]); + MONT_MAC(t0, t1, t2, a[3], b[7]); + MONT_MAC(v0, v1, v2, a[4], b[6]); + MONT_MAC(w0, w1, w2, a[5], b[5]); + MONT_MAC(t0, t1, t2, a[6], b[4]); + MONT_MAC(v0, v1, v2, a[7], b[3]); + MONT_MAC(w0, w1, w2, a[8], b[2]); + MONT_MAC(t0, t1, t2, a[9], b[1]); + MONT_MAC(v0, v1, v2, a[10], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[10]); + MONT_MAC(t0, t1, t2, u[1], m[9]); + MONT_MAC(v0, v1, v2, u[2], m[8]); + MONT_MAC(w0, w1, w2, u[3], m[7]); + MONT_MAC(t0, t1, t2, u[4], m[6]); + MONT_MAC(v0, v1, v2, u[5], m[5]); + MONT_MAC(w0, w1, w2, u[6], m[4]); + MONT_MAC(t0, t1, t2, u[7], m[3]); + MONT_MAC(v0, v1, v2, u[8], m[2]); + MONT_MAC(w0, w1, w2, u[9], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[10] = t0 * n0; + MONT_MAC(t0, t1, t2, u[10], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 11 */ + MONT_MAC(t0, t1, t2, a[0], b[11]); + MONT_MUL(v0, v1, v2, a[1], b[10]); + MONT_MUL(w0, w1, w2, a[2], b[9]); + MONT_MAC(t0, t1, t2, a[3], b[8]); + MONT_MAC(v0, v1, v2, a[4], b[7]); + MONT_MAC(w0, w1, w2, a[5], b[6]); + MONT_MAC(t0, t1, t2, a[6], b[5]); + MONT_MAC(v0, v1, v2, a[7], b[4]); + MONT_MAC(w0, w1, w2, a[8], b[3]); + MONT_MAC(t0, t1, t2, a[9], b[2]); + MONT_MAC(v0, v1, v2, a[10], b[1]); + MONT_MAC(w0, w1, w2, a[11], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[11]); + MONT_MAC(v0, v1, v2, u[1], m[10]); + MONT_MAC(w0, w1, w2, u[2], m[9]); + MONT_MAC(t0, t1, t2, u[3], m[8]); + MONT_MAC(v0, v1, v2, u[4], m[7]); + MONT_MAC(w0, w1, w2, u[5], m[6]); + MONT_MAC(t0, t1, t2, u[6], m[5]); + MONT_MAC(v0, v1, v2, u[7], m[4]); + MONT_MAC(w0, w1, w2, u[8], m[3]); + MONT_MAC(t0, t1, t2, u[9], m[2]); + MONT_MAC(v0, v1, v2, u[10], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[11] = t0 * n0; + MONT_MAC(t0, t1, t2, u[11], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 12 */ + MONT_MAC(t0, t1, t2, a[0], b[12]); + MONT_MUL(v0, v1, v2, a[1], b[11]); + MONT_MUL(w0, w1, w2, a[2], b[10]); + MONT_MAC(t0, t1, t2, a[3], b[9]); + MONT_MAC(v0, v1, v2, a[4], b[8]); + MONT_MAC(w0, w1, w2, a[5], b[7]); + MONT_MAC(t0, t1, t2, a[6], b[6]); + MONT_MAC(v0, v1, v2, a[7], b[5]); + MONT_MAC(w0, w1, w2, a[8], b[4]); + MONT_MAC(t0, t1, t2, a[9], b[3]); + MONT_MAC(v0, v1, v2, a[10], b[2]); + MONT_MAC(w0, w1, w2, a[11], b[1]); + MONT_MAC(t0, t1, t2, a[12], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[12]); + MONT_MAC(w0, w1, w2, u[1], m[11]); + MONT_MAC(t0, t1, t2, u[2], m[10]); + MONT_MAC(v0, v1, v2, u[3], m[9]); + MONT_MAC(w0, w1, w2, u[4], m[8]); + MONT_MAC(t0, t1, t2, u[5], m[7]); + MONT_MAC(v0, v1, v2, u[6], m[6]); + MONT_MAC(w0, w1, w2, u[7], m[5]); + MONT_MAC(t0, t1, t2, u[8], m[4]); + MONT_MAC(v0, v1, v2, u[9], m[3]); + MONT_MAC(w0, w1, w2, u[10], m[2]); + MONT_MAC(t0, t1, t2, u[11], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[12] = t0 * n0; + MONT_MAC(t0, t1, t2, u[12], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 13 */ + MONT_MAC(t0, t1, t2, a[0], b[13]); + MONT_MUL(v0, v1, v2, a[1], b[12]); + MONT_MUL(w0, w1, w2, a[2], b[11]); + MONT_MAC(t0, t1, t2, a[3], b[10]); + MONT_MAC(v0, v1, v2, a[4], b[9]); + MONT_MAC(w0, w1, w2, a[5], b[8]); + MONT_MAC(t0, t1, t2, a[6], b[7]); + MONT_MAC(v0, v1, v2, a[7], b[6]); + MONT_MAC(w0, w1, w2, a[8], b[5]); + MONT_MAC(t0, t1, t2, a[9], b[4]); + MONT_MAC(v0, v1, v2, a[10], b[3]); + MONT_MAC(w0, w1, w2, a[11], b[2]); + MONT_MAC(t0, t1, t2, a[12], b[1]); + MONT_MAC(v0, v1, v2, a[13], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[13]); + MONT_MAC(t0, t1, t2, u[1], m[12]); + MONT_MAC(v0, v1, v2, u[2], m[11]); + MONT_MAC(w0, w1, w2, u[3], m[10]); + MONT_MAC(t0, t1, t2, u[4], m[9]); + MONT_MAC(v0, v1, v2, u[5], m[8]); + MONT_MAC(w0, w1, w2, u[6], m[7]); + MONT_MAC(t0, t1, t2, u[7], m[6]); + MONT_MAC(v0, v1, v2, u[8], m[5]); + MONT_MAC(w0, w1, w2, u[9], m[4]); + MONT_MAC(t0, t1, t2, u[10], m[3]); + MONT_MAC(v0, v1, v2, u[11], m[2]); + MONT_MAC(w0, w1, w2, u[12], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[13] = t0 * n0; + MONT_MAC(t0, t1, t2, u[13], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 14 */ + MONT_MAC(t0, t1, t2, a[0], b[14]); + MONT_MUL(v0, v1, v2, a[1], b[13]); + MONT_MUL(w0, w1, w2, a[2], b[12]); + MONT_MAC(t0, t1, t2, a[3], b[11]); + MONT_MAC(v0, v1, v2, a[4], b[10]); + MONT_MAC(w0, w1, w2, a[5], b[9]); + MONT_MAC(t0, t1, t2, a[6], b[8]); + MONT_MAC(v0, v1, v2, a[7], b[7]); + MONT_MAC(w0, w1, w2, a[8], b[6]); + MONT_MAC(t0, t1, t2, a[9], b[5]); + MONT_MAC(v0, v1, v2, a[10], b[4]); + MONT_MAC(w0, w1, w2, a[11], b[3]); + MONT_MAC(t0, t1, t2, a[12], b[2]); + MONT_MAC(v0, v1, v2, a[13], b[1]); + MONT_MAC(w0, w1, w2, a[14], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[14]); + MONT_MAC(v0, v1, v2, u[1], m[13]); + MONT_MAC(w0, w1, w2, u[2], m[12]); + MONT_MAC(t0, t1, t2, u[3], m[11]); + MONT_MAC(v0, v1, v2, u[4], m[10]); + MONT_MAC(w0, w1, w2, u[5], m[9]); + MONT_MAC(t0, t1, t2, u[6], m[8]); + MONT_MAC(v0, v1, v2, u[7], m[7]); + MONT_MAC(w0, w1, w2, u[8], m[6]); + MONT_MAC(t0, t1, t2, u[9], m[5]); + MONT_MAC(v0, v1, v2, u[10], m[4]); + MONT_MAC(w0, w1, w2, u[11], m[3]); + MONT_MAC(t0, t1, t2, u[12], m[2]); + MONT_MAC(v0, v1, v2, u[13], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[14] = t0 * n0; + MONT_MAC(t0, t1, t2, u[14], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 15 */ + MONT_MAC(t0, t1, t2, a[0], b[15]); + MONT_MUL(v0, v1, v2, a[1], b[14]); + MONT_MUL(w0, w1, w2, a[2], b[13]); + MONT_MAC(t0, t1, t2, a[3], b[12]); + MONT_MAC(v0, v1, v2, a[4], b[11]); + MONT_MAC(w0, w1, w2, a[5], b[10]); + MONT_MAC(t0, t1, t2, a[6], b[9]); + MONT_MAC(v0, v1, v2, a[7], b[8]); + MONT_MAC(w0, w1, w2, a[8], b[7]); + MONT_MAC(t0, t1, t2, a[9], b[6]); + MONT_MAC(v0, v1, v2, a[10], b[5]); + MONT_MAC(w0, w1, w2, a[11], b[4]); + MONT_MAC(t0, t1, t2, a[12], b[3]); + MONT_MAC(v0, v1, v2, a[13], b[2]); + MONT_MAC(w0, w1, w2, a[14], b[1]); + MONT_MAC(t0, t1, t2, a[15], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[15]); + MONT_MAC(w0, w1, w2, u[1], m[14]); + MONT_MAC(t0, t1, t2, u[2], m[13]); + MONT_MAC(v0, v1, v2, u[3], m[12]); + MONT_MAC(w0, w1, w2, u[4], m[11]); + MONT_MAC(t0, t1, t2, u[5], m[10]); + MONT_MAC(v0, v1, v2, u[6], m[9]); + MONT_MAC(w0, w1, w2, u[7], m[8]); + MONT_MAC(t0, t1, t2, u[8], m[7]); + MONT_MAC(v0, v1, v2, u[9], m[6]); + MONT_MAC(w0, w1, w2, u[10], m[5]); + MONT_MAC(t0, t1, t2, u[11], m[4]); + MONT_MAC(v0, v1, v2, u[12], m[3]); + MONT_MAC(w0, w1, w2, u[13], m[2]); + MONT_MAC(t0, t1, t2, u[14], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[15] = t0 * n0; + MONT_MAC(t0, t1, t2, u[15], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 16 */ + MONT_MAC(t0, t1, t2, a[1], b[15]); + MONT_MUL(v0, v1, v2, a[2], b[14]); + MONT_MUL(w0, w1, w2, a[3], b[13]); + MONT_MAC(t0, t1, t2, a[4], b[12]); + MONT_MAC(v0, v1, v2, a[5], b[11]); + MONT_MAC(w0, w1, w2, a[6], b[10]); + MONT_MAC(t0, t1, t2, a[7], b[9]); + MONT_MAC(v0, v1, v2, a[8], b[8]); + MONT_MAC(w0, w1, w2, a[9], b[7]); + MONT_MAC(t0, t1, t2, a[10], b[6]); + MONT_MAC(v0, v1, v2, a[11], b[5]); + MONT_MAC(w0, w1, w2, a[12], b[4]); + MONT_MAC(t0, t1, t2, a[13], b[3]); + MONT_MAC(v0, v1, v2, a[14], b[2]); + MONT_MAC(w0, w1, w2, a[15], b[1]); + MONT_MAC(t0, t1, t2, u[1], m[15]); + MONT_MAC(v0, v1, v2, u[2], m[14]); + MONT_MAC(w0, w1, w2, u[3], m[13]); + MONT_MAC(t0, t1, t2, u[4], m[12]); + MONT_MAC(v0, v1, v2, u[5], m[11]); + MONT_MAC(w0, w1, w2, u[6], m[10]); + MONT_MAC(t0, t1, t2, u[7], m[9]); + MONT_MAC(v0, v1, v2, u[8], m[8]); + MONT_MAC(w0, w1, w2, u[9], m[7]); + MONT_MAC(t0, t1, t2, u[10], m[6]); + MONT_MAC(v0, v1, v2, u[11], m[5]); + MONT_MAC(w0, w1, w2, u[12], m[4]); + MONT_MAC(t0, t1, t2, u[13], m[3]); + MONT_MAC(v0, v1, v2, u[14], m[2]); + MONT_MAC(w0, w1, w2, u[15], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[0] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 17 */ + MONT_MAC(t0, t1, t2, a[2], b[15]); + MONT_MUL(v0, v1, v2, a[3], b[14]); + MONT_MUL(w0, w1, w2, a[4], b[13]); + MONT_MAC(t0, t1, t2, a[5], b[12]); + MONT_MAC(v0, v1, v2, a[6], b[11]); + MONT_MAC(w0, w1, w2, a[7], b[10]); + MONT_MAC(t0, t1, t2, a[8], b[9]); + MONT_MAC(v0, v1, v2, a[9], b[8]); + MONT_MAC(w0, w1, w2, a[10], b[7]); + MONT_MAC(t0, t1, t2, a[11], b[6]); + MONT_MAC(v0, v1, v2, a[12], b[5]); + MONT_MAC(w0, w1, w2, a[13], b[4]); + MONT_MAC(t0, t1, t2, a[14], b[3]); + MONT_MAC(v0, v1, v2, a[15], b[2]); + MONT_MAC(w0, w1, w2, u[2], m[15]); + MONT_MAC(t0, t1, t2, u[3], m[14]); + MONT_MAC(v0, v1, v2, u[4], m[13]); + MONT_MAC(w0, w1, w2, u[5], m[12]); + MONT_MAC(t0, t1, t2, u[6], m[11]); + MONT_MAC(v0, v1, v2, u[7], m[10]); + MONT_MAC(w0, w1, w2, u[8], m[9]); + MONT_MAC(t0, t1, t2, u[9], m[8]); + MONT_MAC(v0, v1, v2, u[10], m[7]); + MONT_MAC(w0, w1, w2, u[11], m[6]); + MONT_MAC(t0, t1, t2, u[12], m[5]); + MONT_MAC(v0, v1, v2, u[13], m[4]); + MONT_MAC(w0, w1, w2, u[14], m[3]); + MONT_MAC(t0, t1, t2, u[15], m[2]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[1] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 18 */ + MONT_MAC(t0, t1, t2, a[3], b[15]); + MONT_MUL(v0, v1, v2, a[4], b[14]); + MONT_MUL(w0, w1, w2, a[5], b[13]); + MONT_MAC(t0, t1, t2, a[6], b[12]); + MONT_MAC(v0, v1, v2, a[7], b[11]); + MONT_MAC(w0, w1, w2, a[8], b[10]); + MONT_MAC(t0, t1, t2, a[9], b[9]); + MONT_MAC(v0, v1, v2, a[10], b[8]); + MONT_MAC(w0, w1, w2, a[11], b[7]); + MONT_MAC(t0, t1, t2, a[12], b[6]); + MONT_MAC(v0, v1, v2, a[13], b[5]); + MONT_MAC(w0, w1, w2, a[14], b[4]); + MONT_MAC(t0, t1, t2, a[15], b[3]); + MONT_MAC(v0, v1, v2, u[3], m[15]); + MONT_MAC(w0, w1, w2, u[4], m[14]); + MONT_MAC(t0, t1, t2, u[5], m[13]); + MONT_MAC(v0, v1, v2, u[6], m[12]); + MONT_MAC(w0, w1, w2, u[7], m[11]); + MONT_MAC(t0, t1, t2, u[8], m[10]); + MONT_MAC(v0, v1, v2, u[9], m[9]); + MONT_MAC(w0, w1, w2, u[10], m[8]); + MONT_MAC(t0, t1, t2, u[11], m[7]); + MONT_MAC(v0, v1, v2, u[12], m[6]); + MONT_MAC(w0, w1, w2, u[13], m[5]); + MONT_MAC(t0, t1, t2, u[14], m[4]); + MONT_MAC(v0, v1, v2, u[15], m[3]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[2] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 19 */ + MONT_MAC(t0, t1, t2, a[4], b[15]); + MONT_MUL(v0, v1, v2, a[5], b[14]); + MONT_MUL(w0, w1, w2, a[6], b[13]); + MONT_MAC(t0, t1, t2, a[7], b[12]); + MONT_MAC(v0, v1, v2, a[8], b[11]); + MONT_MAC(w0, w1, w2, a[9], b[10]); + MONT_MAC(t0, t1, t2, a[10], b[9]); + MONT_MAC(v0, v1, v2, a[11], b[8]); + MONT_MAC(w0, w1, w2, a[12], b[7]); + MONT_MAC(t0, t1, t2, a[13], b[6]); + MONT_MAC(v0, v1, v2, a[14], b[5]); + MONT_MAC(w0, w1, w2, a[15], b[4]); + MONT_MAC(t0, t1, t2, u[4], m[15]); + MONT_MAC(v0, v1, v2, u[5], m[14]); + MONT_MAC(w0, w1, w2, u[6], m[13]); + MONT_MAC(t0, t1, t2, u[7], m[12]); + MONT_MAC(v0, v1, v2, u[8], m[11]); + MONT_MAC(w0, w1, w2, u[9], m[10]); + MONT_MAC(t0, t1, t2, u[10], m[9]); + MONT_MAC(v0, v1, v2, u[11], m[8]); + MONT_MAC(w0, w1, w2, u[12], m[7]); + MONT_MAC(t0, t1, t2, u[13], m[6]); + MONT_MAC(v0, v1, v2, u[14], m[5]); + MONT_MAC(w0, w1, w2, u[15], m[4]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[3] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 20 */ + MONT_MAC(t0, t1, t2, a[5], b[15]); + MONT_MUL(v0, v1, v2, a[6], b[14]); + MONT_MUL(w0, w1, w2, a[7], b[13]); + MONT_MAC(t0, t1, t2, a[8], b[12]); + MONT_MAC(v0, v1, v2, a[9], b[11]); + MONT_MAC(w0, w1, w2, a[10], b[10]); + MONT_MAC(t0, t1, t2, a[11], b[9]); + MONT_MAC(v0, v1, v2, a[12], b[8]); + MONT_MAC(w0, w1, w2, a[13], b[7]); + MONT_MAC(t0, t1, t2, a[14], b[6]); + MONT_MAC(v0, v1, v2, a[15], b[5]); + MONT_MAC(w0, w1, w2, u[5], m[15]); + MONT_MAC(t0, t1, t2, u[6], m[14]); + MONT_MAC(v0, v1, v2, u[7], m[13]); + MONT_MAC(w0, w1, w2, u[8], m[12]); + MONT_MAC(t0, t1, t2, u[9], m[11]); + MONT_MAC(v0, v1, v2, u[10], m[10]); + MONT_MAC(w0, w1, w2, u[11], m[9]); + MONT_MAC(t0, t1, t2, u[12], m[8]); + MONT_MAC(v0, v1, v2, u[13], m[7]); + MONT_MAC(w0, w1, w2, u[14], m[6]); + MONT_MAC(t0, t1, t2, u[15], m[5]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[4] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 21 */ + MONT_MAC(t0, t1, t2, a[6], b[15]); + MONT_MUL(v0, v1, v2, a[7], b[14]); + MONT_MUL(w0, w1, w2, a[8], b[13]); + MONT_MAC(t0, t1, t2, a[9], b[12]); + MONT_MAC(v0, v1, v2, a[10], b[11]); + MONT_MAC(w0, w1, w2, a[11], b[10]); + MONT_MAC(t0, t1, t2, a[12], b[9]); + MONT_MAC(v0, v1, v2, a[13], b[8]); + MONT_MAC(w0, w1, w2, a[14], b[7]); + MONT_MAC(t0, t1, t2, a[15], b[6]); + MONT_MAC(v0, v1, v2, u[6], m[15]); + MONT_MAC(w0, w1, w2, u[7], m[14]); + MONT_MAC(t0, t1, t2, u[8], m[13]); + MONT_MAC(v0, v1, v2, u[9], m[12]); + MONT_MAC(w0, w1, w2, u[10], m[11]); + MONT_MAC(t0, t1, t2, u[11], m[10]); + MONT_MAC(v0, v1, v2, u[12], m[9]); + MONT_MAC(w0, w1, w2, u[13], m[8]); + MONT_MAC(t0, t1, t2, u[14], m[7]); + MONT_MAC(v0, v1, v2, u[15], m[6]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[5] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 22 */ + MONT_MAC(t0, t1, t2, a[7], b[15]); + MONT_MUL(v0, v1, v2, a[8], b[14]); + MONT_MUL(w0, w1, w2, a[9], b[13]); + MONT_MAC(t0, t1, t2, a[10], b[12]); + MONT_MAC(v0, v1, v2, a[11], b[11]); + MONT_MAC(w0, w1, w2, a[12], b[10]); + MONT_MAC(t0, t1, t2, a[13], b[9]); + MONT_MAC(v0, v1, v2, a[14], b[8]); + MONT_MAC(w0, w1, w2, a[15], b[7]); + MONT_MAC(t0, t1, t2, u[7], m[15]); + MONT_MAC(v0, v1, v2, u[8], m[14]); + MONT_MAC(w0, w1, w2, u[9], m[13]); + MONT_MAC(t0, t1, t2, u[10], m[12]); + MONT_MAC(v0, v1, v2, u[11], m[11]); + MONT_MAC(w0, w1, w2, u[12], m[10]); + MONT_MAC(t0, t1, t2, u[13], m[9]); + MONT_MAC(v0, v1, v2, u[14], m[8]); + MONT_MAC(w0, w1, w2, u[15], m[7]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[6] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 23 */ + MONT_MAC(t0, t1, t2, a[8], b[15]); + MONT_MUL(v0, v1, v2, a[9], b[14]); + MONT_MUL(w0, w1, w2, a[10], b[13]); + MONT_MAC(t0, t1, t2, a[11], b[12]); + MONT_MAC(v0, v1, v2, a[12], b[11]); + MONT_MAC(w0, w1, w2, a[13], b[10]); + MONT_MAC(t0, t1, t2, a[14], b[9]); + MONT_MAC(v0, v1, v2, a[15], b[8]); + MONT_MAC(w0, w1, w2, u[8], m[15]); + MONT_MAC(t0, t1, t2, u[9], m[14]); + MONT_MAC(v0, v1, v2, u[10], m[13]); + MONT_MAC(w0, w1, w2, u[11], m[12]); + MONT_MAC(t0, t1, t2, u[12], m[11]); + MONT_MAC(v0, v1, v2, u[13], m[10]); + MONT_MAC(w0, w1, w2, u[14], m[9]); + MONT_MAC(t0, t1, t2, u[15], m[8]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[7] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 24 */ + MONT_MAC(t0, t1, t2, a[9], b[15]); + MONT_MUL(v0, v1, v2, a[10], b[14]); + MONT_MUL(w0, w1, w2, a[11], b[13]); + MONT_MAC(t0, t1, t2, a[12], b[12]); + MONT_MAC(v0, v1, v2, a[13], b[11]); + MONT_MAC(w0, w1, w2, a[14], b[10]); + MONT_MAC(t0, t1, t2, a[15], b[9]); + MONT_MAC(v0, v1, v2, u[9], m[15]); + MONT_MAC(w0, w1, w2, u[10], m[14]); + MONT_MAC(t0, t1, t2, u[11], m[13]); + MONT_MAC(v0, v1, v2, u[12], m[12]); + MONT_MAC(w0, w1, w2, u[13], m[11]); + MONT_MAC(t0, t1, t2, u[14], m[10]); + MONT_MAC(v0, v1, v2, u[15], m[9]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[8] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 25 */ + MONT_MAC(t0, t1, t2, a[10], b[15]); + MONT_MUL(v0, v1, v2, a[11], b[14]); + MONT_MUL(w0, w1, w2, a[12], b[13]); + MONT_MAC(t0, t1, t2, a[13], b[12]); + MONT_MAC(v0, v1, v2, a[14], b[11]); + MONT_MAC(w0, w1, w2, a[15], b[10]); + MONT_MAC(t0, t1, t2, u[10], m[15]); + MONT_MAC(v0, v1, v2, u[11], m[14]); + MONT_MAC(w0, w1, w2, u[12], m[13]); + MONT_MAC(t0, t1, t2, u[13], m[12]); + MONT_MAC(v0, v1, v2, u[14], m[11]); + MONT_MAC(w0, w1, w2, u[15], m[10]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[9] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 26 */ + MONT_MAC(t0, t1, t2, a[11], b[15]); + MONT_MUL(v0, v1, v2, a[12], b[14]); + MONT_MUL(w0, w1, w2, a[13], b[13]); + MONT_MAC(t0, t1, t2, a[14], b[12]); + MONT_MAC(v0, v1, v2, a[15], b[11]); + MONT_MAC(w0, w1, w2, u[11], m[15]); + MONT_MAC(t0, t1, t2, u[12], m[14]); + MONT_MAC(v0, v1, v2, u[13], m[13]); + MONT_MAC(w0, w1, w2, u[14], m[12]); + MONT_MAC(t0, t1, t2, u[15], m[11]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[10] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 27 */ + MONT_MAC(t0, t1, t2, a[12], b[15]); + MONT_MUL(v0, v1, v2, a[13], b[14]); + MONT_MUL(w0, w1, w2, a[14], b[13]); + MONT_MAC(t0, t1, t2, a[15], b[12]); + MONT_MAC(v0, v1, v2, u[12], m[15]); + MONT_MAC(w0, w1, w2, u[13], m[14]); + MONT_MAC(t0, t1, t2, u[14], m[13]); + MONT_MAC(v0, v1, v2, u[15], m[12]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[11] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 28 */ + MONT_MAC(t0, t1, t2, a[13], b[15]); + MONT_MUL(v0, v1, v2, a[14], b[14]); + MONT_MUL(w0, w1, w2, a[15], b[13]); + MONT_MAC(t0, t1, t2, u[13], m[15]); + MONT_MAC(v0, v1, v2, u[14], m[14]); + MONT_MAC(w0, w1, w2, u[15], m[13]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[12] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 29 */ + MONT_MAC(t0, t1, t2, a[14], b[15]); + MONT_MUL(v0, v1, v2, a[15], b[14]); + MONT_MUL(w0, w1, w2, u[14], m[15]); + MONT_MAC(t0, t1, t2, u[15], m[14]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[13] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 30 */ + MONT_MAC(t0, t1, t2, a[15], b[15]); + MONT_MUL(v0, v1, v2, u[15], m[15]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + z[14] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + z[15] = t0; + + /* under 2m, with t1 the bit above n limbs: at most one subtraction */ + borrow = sub_n(r, z, m, 16); + take = t1 | (borrow ^ 1); + select_n(r, r, z, take & 1, 16); +} + +/* r = a * a * R^-1 mod m, for a under m; t is 2n limbs of scratch */ +static void crypton_mont_sqr16(limb_t *r, const limb_t *a, + const limb_t *m, limb_t n0, limb_t *t) +{ + limb_t *u = t, *z = t + 16; + limb_t t0 = 0, t1 = 0, t2 = 0, v0, v1, v2, w0, w1, w2, borrow, take; + limb_t s0, s1, s2, c0, c1, c2; + + /* column 0 */ + MONT_MAC(t0, t1, t2, a[0], a[0]); + u[0] = t0 * n0; + MONT_MAC(t0, t1, t2, u[0], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 1 */ + MONT_MUL(s0, s1, s2, a[0], a[1]); + MONT_MAC(t0, t1, t2, u[0], m[1]); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[1] = t0 * n0; + MONT_MAC(t0, t1, t2, u[1], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 2 */ + MONT_MUL(s0, s1, s2, a[0], a[2]); + MONT_MAC(t0, t1, t2, a[1], a[1]); + MONT_MUL(v0, v1, v2, u[0], m[2]); + MONT_MUL(w0, w1, w2, u[1], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[2] = t0 * n0; + MONT_MAC(t0, t1, t2, u[2], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 3 */ + MONT_MUL(s0, s1, s2, a[0], a[3]); + MONT_MUL(c0, c1, c2, a[1], a[2]); + MONT_MAC(t0, t1, t2, u[0], m[3]); + MONT_MUL(v0, v1, v2, u[1], m[2]); + MONT_MUL(w0, w1, w2, u[2], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[3] = t0 * n0; + MONT_MAC(t0, t1, t2, u[3], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 4 */ + MONT_MUL(s0, s1, s2, a[0], a[4]); + MONT_MUL(c0, c1, c2, a[1], a[3]); + MONT_MAC(t0, t1, t2, a[2], a[2]); + MONT_MUL(v0, v1, v2, u[0], m[4]); + MONT_MUL(w0, w1, w2, u[1], m[3]); + MONT_MAC(t0, t1, t2, u[2], m[2]); + MONT_MAC(v0, v1, v2, u[3], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[4] = t0 * n0; + MONT_MAC(t0, t1, t2, u[4], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 5 */ + MONT_MUL(s0, s1, s2, a[0], a[5]); + MONT_MUL(c0, c1, c2, a[1], a[4]); + MONT_MAC(s0, s1, s2, a[2], a[3]); + MONT_MAC(t0, t1, t2, u[0], m[5]); + MONT_MUL(v0, v1, v2, u[1], m[4]); + MONT_MUL(w0, w1, w2, u[2], m[3]); + MONT_MAC(t0, t1, t2, u[3], m[2]); + MONT_MAC(v0, v1, v2, u[4], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[5] = t0 * n0; + MONT_MAC(t0, t1, t2, u[5], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 6 */ + MONT_MUL(s0, s1, s2, a[0], a[6]); + MONT_MUL(c0, c1, c2, a[1], a[5]); + MONT_MAC(s0, s1, s2, a[2], a[4]); + MONT_MAC(t0, t1, t2, a[3], a[3]); + MONT_MUL(v0, v1, v2, u[0], m[6]); + MONT_MUL(w0, w1, w2, u[1], m[5]); + MONT_MAC(t0, t1, t2, u[2], m[4]); + MONT_MAC(v0, v1, v2, u[3], m[3]); + MONT_MAC(w0, w1, w2, u[4], m[2]); + MONT_MAC(t0, t1, t2, u[5], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[6] = t0 * n0; + MONT_MAC(t0, t1, t2, u[6], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 7 */ + MONT_MUL(s0, s1, s2, a[0], a[7]); + MONT_MUL(c0, c1, c2, a[1], a[6]); + MONT_MAC(s0, s1, s2, a[2], a[5]); + MONT_MAC(c0, c1, c2, a[3], a[4]); + MONT_MAC(t0, t1, t2, u[0], m[7]); + MONT_MUL(v0, v1, v2, u[1], m[6]); + MONT_MUL(w0, w1, w2, u[2], m[5]); + MONT_MAC(t0, t1, t2, u[3], m[4]); + MONT_MAC(v0, v1, v2, u[4], m[3]); + MONT_MAC(w0, w1, w2, u[5], m[2]); + MONT_MAC(t0, t1, t2, u[6], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[7] = t0 * n0; + MONT_MAC(t0, t1, t2, u[7], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 8 */ + MONT_MUL(s0, s1, s2, a[0], a[8]); + MONT_MUL(c0, c1, c2, a[1], a[7]); + MONT_MAC(s0, s1, s2, a[2], a[6]); + MONT_MAC(c0, c1, c2, a[3], a[5]); + MONT_MAC(t0, t1, t2, a[4], a[4]); + MONT_MUL(v0, v1, v2, u[0], m[8]); + MONT_MUL(w0, w1, w2, u[1], m[7]); + MONT_MAC(t0, t1, t2, u[2], m[6]); + MONT_MAC(v0, v1, v2, u[3], m[5]); + MONT_MAC(w0, w1, w2, u[4], m[4]); + MONT_MAC(t0, t1, t2, u[5], m[3]); + MONT_MAC(v0, v1, v2, u[6], m[2]); + MONT_MAC(w0, w1, w2, u[7], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[8] = t0 * n0; + MONT_MAC(t0, t1, t2, u[8], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 9 */ + MONT_MUL(s0, s1, s2, a[0], a[9]); + MONT_MUL(c0, c1, c2, a[1], a[8]); + MONT_MAC(s0, s1, s2, a[2], a[7]); + MONT_MAC(c0, c1, c2, a[3], a[6]); + MONT_MAC(s0, s1, s2, a[4], a[5]); + MONT_MAC(t0, t1, t2, u[0], m[9]); + MONT_MUL(v0, v1, v2, u[1], m[8]); + MONT_MUL(w0, w1, w2, u[2], m[7]); + MONT_MAC(t0, t1, t2, u[3], m[6]); + MONT_MAC(v0, v1, v2, u[4], m[5]); + MONT_MAC(w0, w1, w2, u[5], m[4]); + MONT_MAC(t0, t1, t2, u[6], m[3]); + MONT_MAC(v0, v1, v2, u[7], m[2]); + MONT_MAC(w0, w1, w2, u[8], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[9] = t0 * n0; + MONT_MAC(t0, t1, t2, u[9], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 10 */ + MONT_MUL(s0, s1, s2, a[0], a[10]); + MONT_MUL(c0, c1, c2, a[1], a[9]); + MONT_MAC(s0, s1, s2, a[2], a[8]); + MONT_MAC(c0, c1, c2, a[3], a[7]); + MONT_MAC(s0, s1, s2, a[4], a[6]); + MONT_MAC(t0, t1, t2, a[5], a[5]); + MONT_MUL(v0, v1, v2, u[0], m[10]); + MONT_MUL(w0, w1, w2, u[1], m[9]); + MONT_MAC(t0, t1, t2, u[2], m[8]); + MONT_MAC(v0, v1, v2, u[3], m[7]); + MONT_MAC(w0, w1, w2, u[4], m[6]); + MONT_MAC(t0, t1, t2, u[5], m[5]); + MONT_MAC(v0, v1, v2, u[6], m[4]); + MONT_MAC(w0, w1, w2, u[7], m[3]); + MONT_MAC(t0, t1, t2, u[8], m[2]); + MONT_MAC(v0, v1, v2, u[9], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[10] = t0 * n0; + MONT_MAC(t0, t1, t2, u[10], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 11 */ + MONT_MUL(s0, s1, s2, a[0], a[11]); + MONT_MUL(c0, c1, c2, a[1], a[10]); + MONT_MAC(s0, s1, s2, a[2], a[9]); + MONT_MAC(c0, c1, c2, a[3], a[8]); + MONT_MAC(s0, s1, s2, a[4], a[7]); + MONT_MAC(c0, c1, c2, a[5], a[6]); + MONT_MAC(t0, t1, t2, u[0], m[11]); + MONT_MUL(v0, v1, v2, u[1], m[10]); + MONT_MUL(w0, w1, w2, u[2], m[9]); + MONT_MAC(t0, t1, t2, u[3], m[8]); + MONT_MAC(v0, v1, v2, u[4], m[7]); + MONT_MAC(w0, w1, w2, u[5], m[6]); + MONT_MAC(t0, t1, t2, u[6], m[5]); + MONT_MAC(v0, v1, v2, u[7], m[4]); + MONT_MAC(w0, w1, w2, u[8], m[3]); + MONT_MAC(t0, t1, t2, u[9], m[2]); + MONT_MAC(v0, v1, v2, u[10], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[11] = t0 * n0; + MONT_MAC(t0, t1, t2, u[11], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 12 */ + MONT_MUL(s0, s1, s2, a[0], a[12]); + MONT_MUL(c0, c1, c2, a[1], a[11]); + MONT_MAC(s0, s1, s2, a[2], a[10]); + MONT_MAC(c0, c1, c2, a[3], a[9]); + MONT_MAC(s0, s1, s2, a[4], a[8]); + MONT_MAC(c0, c1, c2, a[5], a[7]); + MONT_MAC(t0, t1, t2, a[6], a[6]); + MONT_MUL(v0, v1, v2, u[0], m[12]); + MONT_MUL(w0, w1, w2, u[1], m[11]); + MONT_MAC(t0, t1, t2, u[2], m[10]); + MONT_MAC(v0, v1, v2, u[3], m[9]); + MONT_MAC(w0, w1, w2, u[4], m[8]); + MONT_MAC(t0, t1, t2, u[5], m[7]); + MONT_MAC(v0, v1, v2, u[6], m[6]); + MONT_MAC(w0, w1, w2, u[7], m[5]); + MONT_MAC(t0, t1, t2, u[8], m[4]); + MONT_MAC(v0, v1, v2, u[9], m[3]); + MONT_MAC(w0, w1, w2, u[10], m[2]); + MONT_MAC(t0, t1, t2, u[11], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[12] = t0 * n0; + MONT_MAC(t0, t1, t2, u[12], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 13 */ + MONT_MUL(s0, s1, s2, a[0], a[13]); + MONT_MUL(c0, c1, c2, a[1], a[12]); + MONT_MAC(s0, s1, s2, a[2], a[11]); + MONT_MAC(c0, c1, c2, a[3], a[10]); + MONT_MAC(s0, s1, s2, a[4], a[9]); + MONT_MAC(c0, c1, c2, a[5], a[8]); + MONT_MAC(s0, s1, s2, a[6], a[7]); + MONT_MAC(t0, t1, t2, u[0], m[13]); + MONT_MUL(v0, v1, v2, u[1], m[12]); + MONT_MUL(w0, w1, w2, u[2], m[11]); + MONT_MAC(t0, t1, t2, u[3], m[10]); + MONT_MAC(v0, v1, v2, u[4], m[9]); + MONT_MAC(w0, w1, w2, u[5], m[8]); + MONT_MAC(t0, t1, t2, u[6], m[7]); + MONT_MAC(v0, v1, v2, u[7], m[6]); + MONT_MAC(w0, w1, w2, u[8], m[5]); + MONT_MAC(t0, t1, t2, u[9], m[4]); + MONT_MAC(v0, v1, v2, u[10], m[3]); + MONT_MAC(w0, w1, w2, u[11], m[2]); + MONT_MAC(t0, t1, t2, u[12], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[13] = t0 * n0; + MONT_MAC(t0, t1, t2, u[13], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 14 */ + MONT_MUL(s0, s1, s2, a[0], a[14]); + MONT_MUL(c0, c1, c2, a[1], a[13]); + MONT_MAC(s0, s1, s2, a[2], a[12]); + MONT_MAC(c0, c1, c2, a[3], a[11]); + MONT_MAC(s0, s1, s2, a[4], a[10]); + MONT_MAC(c0, c1, c2, a[5], a[9]); + MONT_MAC(s0, s1, s2, a[6], a[8]); + MONT_MAC(t0, t1, t2, a[7], a[7]); + MONT_MUL(v0, v1, v2, u[0], m[14]); + MONT_MUL(w0, w1, w2, u[1], m[13]); + MONT_MAC(t0, t1, t2, u[2], m[12]); + MONT_MAC(v0, v1, v2, u[3], m[11]); + MONT_MAC(w0, w1, w2, u[4], m[10]); + MONT_MAC(t0, t1, t2, u[5], m[9]); + MONT_MAC(v0, v1, v2, u[6], m[8]); + MONT_MAC(w0, w1, w2, u[7], m[7]); + MONT_MAC(t0, t1, t2, u[8], m[6]); + MONT_MAC(v0, v1, v2, u[9], m[5]); + MONT_MAC(w0, w1, w2, u[10], m[4]); + MONT_MAC(t0, t1, t2, u[11], m[3]); + MONT_MAC(v0, v1, v2, u[12], m[2]); + MONT_MAC(w0, w1, w2, u[13], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[14] = t0 * n0; + MONT_MAC(t0, t1, t2, u[14], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 15 */ + MONT_MUL(s0, s1, s2, a[0], a[15]); + MONT_MUL(c0, c1, c2, a[1], a[14]); + MONT_MAC(s0, s1, s2, a[2], a[13]); + MONT_MAC(c0, c1, c2, a[3], a[12]); + MONT_MAC(s0, s1, s2, a[4], a[11]); + MONT_MAC(c0, c1, c2, a[5], a[10]); + MONT_MAC(s0, s1, s2, a[6], a[9]); + MONT_MAC(c0, c1, c2, a[7], a[8]); + MONT_MAC(t0, t1, t2, u[0], m[15]); + MONT_MUL(v0, v1, v2, u[1], m[14]); + MONT_MUL(w0, w1, w2, u[2], m[13]); + MONT_MAC(t0, t1, t2, u[3], m[12]); + MONT_MAC(v0, v1, v2, u[4], m[11]); + MONT_MAC(w0, w1, w2, u[5], m[10]); + MONT_MAC(t0, t1, t2, u[6], m[9]); + MONT_MAC(v0, v1, v2, u[7], m[8]); + MONT_MAC(w0, w1, w2, u[8], m[7]); + MONT_MAC(t0, t1, t2, u[9], m[6]); + MONT_MAC(v0, v1, v2, u[10], m[5]); + MONT_MAC(w0, w1, w2, u[11], m[4]); + MONT_MAC(t0, t1, t2, u[12], m[3]); + MONT_MAC(v0, v1, v2, u[13], m[2]); + MONT_MAC(w0, w1, w2, u[14], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[15] = t0 * n0; + MONT_MAC(t0, t1, t2, u[15], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 16 */ + MONT_MUL(s0, s1, s2, a[1], a[15]); + MONT_MUL(c0, c1, c2, a[2], a[14]); + MONT_MAC(s0, s1, s2, a[3], a[13]); + MONT_MAC(c0, c1, c2, a[4], a[12]); + MONT_MAC(s0, s1, s2, a[5], a[11]); + MONT_MAC(c0, c1, c2, a[6], a[10]); + MONT_MAC(s0, s1, s2, a[7], a[9]); + MONT_MAC(t0, t1, t2, a[8], a[8]); + MONT_MUL(v0, v1, v2, u[1], m[15]); + MONT_MUL(w0, w1, w2, u[2], m[14]); + MONT_MAC(t0, t1, t2, u[3], m[13]); + MONT_MAC(v0, v1, v2, u[4], m[12]); + MONT_MAC(w0, w1, w2, u[5], m[11]); + MONT_MAC(t0, t1, t2, u[6], m[10]); + MONT_MAC(v0, v1, v2, u[7], m[9]); + MONT_MAC(w0, w1, w2, u[8], m[8]); + MONT_MAC(t0, t1, t2, u[9], m[7]); + MONT_MAC(v0, v1, v2, u[10], m[6]); + MONT_MAC(w0, w1, w2, u[11], m[5]); + MONT_MAC(t0, t1, t2, u[12], m[4]); + MONT_MAC(v0, v1, v2, u[13], m[3]); + MONT_MAC(w0, w1, w2, u[14], m[2]); + MONT_MAC(t0, t1, t2, u[15], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[0] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 17 */ + MONT_MUL(s0, s1, s2, a[2], a[15]); + MONT_MUL(c0, c1, c2, a[3], a[14]); + MONT_MAC(s0, s1, s2, a[4], a[13]); + MONT_MAC(c0, c1, c2, a[5], a[12]); + MONT_MAC(s0, s1, s2, a[6], a[11]); + MONT_MAC(c0, c1, c2, a[7], a[10]); + MONT_MAC(s0, s1, s2, a[8], a[9]); + MONT_MAC(t0, t1, t2, u[2], m[15]); + MONT_MUL(v0, v1, v2, u[3], m[14]); + MONT_MUL(w0, w1, w2, u[4], m[13]); + MONT_MAC(t0, t1, t2, u[5], m[12]); + MONT_MAC(v0, v1, v2, u[6], m[11]); + MONT_MAC(w0, w1, w2, u[7], m[10]); + MONT_MAC(t0, t1, t2, u[8], m[9]); + MONT_MAC(v0, v1, v2, u[9], m[8]); + MONT_MAC(w0, w1, w2, u[10], m[7]); + MONT_MAC(t0, t1, t2, u[11], m[6]); + MONT_MAC(v0, v1, v2, u[12], m[5]); + MONT_MAC(w0, w1, w2, u[13], m[4]); + MONT_MAC(t0, t1, t2, u[14], m[3]); + MONT_MAC(v0, v1, v2, u[15], m[2]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[1] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 18 */ + MONT_MUL(s0, s1, s2, a[3], a[15]); + MONT_MUL(c0, c1, c2, a[4], a[14]); + MONT_MAC(s0, s1, s2, a[5], a[13]); + MONT_MAC(c0, c1, c2, a[6], a[12]); + MONT_MAC(s0, s1, s2, a[7], a[11]); + MONT_MAC(c0, c1, c2, a[8], a[10]); + MONT_MAC(t0, t1, t2, a[9], a[9]); + MONT_MUL(v0, v1, v2, u[3], m[15]); + MONT_MUL(w0, w1, w2, u[4], m[14]); + MONT_MAC(t0, t1, t2, u[5], m[13]); + MONT_MAC(v0, v1, v2, u[6], m[12]); + MONT_MAC(w0, w1, w2, u[7], m[11]); + MONT_MAC(t0, t1, t2, u[8], m[10]); + MONT_MAC(v0, v1, v2, u[9], m[9]); + MONT_MAC(w0, w1, w2, u[10], m[8]); + MONT_MAC(t0, t1, t2, u[11], m[7]); + MONT_MAC(v0, v1, v2, u[12], m[6]); + MONT_MAC(w0, w1, w2, u[13], m[5]); + MONT_MAC(t0, t1, t2, u[14], m[4]); + MONT_MAC(v0, v1, v2, u[15], m[3]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[2] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 19 */ + MONT_MUL(s0, s1, s2, a[4], a[15]); + MONT_MUL(c0, c1, c2, a[5], a[14]); + MONT_MAC(s0, s1, s2, a[6], a[13]); + MONT_MAC(c0, c1, c2, a[7], a[12]); + MONT_MAC(s0, s1, s2, a[8], a[11]); + MONT_MAC(c0, c1, c2, a[9], a[10]); + MONT_MAC(t0, t1, t2, u[4], m[15]); + MONT_MUL(v0, v1, v2, u[5], m[14]); + MONT_MUL(w0, w1, w2, u[6], m[13]); + MONT_MAC(t0, t1, t2, u[7], m[12]); + MONT_MAC(v0, v1, v2, u[8], m[11]); + MONT_MAC(w0, w1, w2, u[9], m[10]); + MONT_MAC(t0, t1, t2, u[10], m[9]); + MONT_MAC(v0, v1, v2, u[11], m[8]); + MONT_MAC(w0, w1, w2, u[12], m[7]); + MONT_MAC(t0, t1, t2, u[13], m[6]); + MONT_MAC(v0, v1, v2, u[14], m[5]); + MONT_MAC(w0, w1, w2, u[15], m[4]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[3] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 20 */ + MONT_MUL(s0, s1, s2, a[5], a[15]); + MONT_MUL(c0, c1, c2, a[6], a[14]); + MONT_MAC(s0, s1, s2, a[7], a[13]); + MONT_MAC(c0, c1, c2, a[8], a[12]); + MONT_MAC(s0, s1, s2, a[9], a[11]); + MONT_MAC(t0, t1, t2, a[10], a[10]); + MONT_MUL(v0, v1, v2, u[5], m[15]); + MONT_MUL(w0, w1, w2, u[6], m[14]); + MONT_MAC(t0, t1, t2, u[7], m[13]); + MONT_MAC(v0, v1, v2, u[8], m[12]); + MONT_MAC(w0, w1, w2, u[9], m[11]); + MONT_MAC(t0, t1, t2, u[10], m[10]); + MONT_MAC(v0, v1, v2, u[11], m[9]); + MONT_MAC(w0, w1, w2, u[12], m[8]); + MONT_MAC(t0, t1, t2, u[13], m[7]); + MONT_MAC(v0, v1, v2, u[14], m[6]); + MONT_MAC(w0, w1, w2, u[15], m[5]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[4] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 21 */ + MONT_MUL(s0, s1, s2, a[6], a[15]); + MONT_MUL(c0, c1, c2, a[7], a[14]); + MONT_MAC(s0, s1, s2, a[8], a[13]); + MONT_MAC(c0, c1, c2, a[9], a[12]); + MONT_MAC(s0, s1, s2, a[10], a[11]); + MONT_MAC(t0, t1, t2, u[6], m[15]); + MONT_MUL(v0, v1, v2, u[7], m[14]); + MONT_MUL(w0, w1, w2, u[8], m[13]); + MONT_MAC(t0, t1, t2, u[9], m[12]); + MONT_MAC(v0, v1, v2, u[10], m[11]); + MONT_MAC(w0, w1, w2, u[11], m[10]); + MONT_MAC(t0, t1, t2, u[12], m[9]); + MONT_MAC(v0, v1, v2, u[13], m[8]); + MONT_MAC(w0, w1, w2, u[14], m[7]); + MONT_MAC(t0, t1, t2, u[15], m[6]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[5] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 22 */ + MONT_MUL(s0, s1, s2, a[7], a[15]); + MONT_MUL(c0, c1, c2, a[8], a[14]); + MONT_MAC(s0, s1, s2, a[9], a[13]); + MONT_MAC(c0, c1, c2, a[10], a[12]); + MONT_MAC(t0, t1, t2, a[11], a[11]); + MONT_MUL(v0, v1, v2, u[7], m[15]); + MONT_MUL(w0, w1, w2, u[8], m[14]); + MONT_MAC(t0, t1, t2, u[9], m[13]); + MONT_MAC(v0, v1, v2, u[10], m[12]); + MONT_MAC(w0, w1, w2, u[11], m[11]); + MONT_MAC(t0, t1, t2, u[12], m[10]); + MONT_MAC(v0, v1, v2, u[13], m[9]); + MONT_MAC(w0, w1, w2, u[14], m[8]); + MONT_MAC(t0, t1, t2, u[15], m[7]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[6] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 23 */ + MONT_MUL(s0, s1, s2, a[8], a[15]); + MONT_MUL(c0, c1, c2, a[9], a[14]); + MONT_MAC(s0, s1, s2, a[10], a[13]); + MONT_MAC(c0, c1, c2, a[11], a[12]); + MONT_MAC(t0, t1, t2, u[8], m[15]); + MONT_MUL(v0, v1, v2, u[9], m[14]); + MONT_MUL(w0, w1, w2, u[10], m[13]); + MONT_MAC(t0, t1, t2, u[11], m[12]); + MONT_MAC(v0, v1, v2, u[12], m[11]); + MONT_MAC(w0, w1, w2, u[13], m[10]); + MONT_MAC(t0, t1, t2, u[14], m[9]); + MONT_MAC(v0, v1, v2, u[15], m[8]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[7] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 24 */ + MONT_MUL(s0, s1, s2, a[9], a[15]); + MONT_MUL(c0, c1, c2, a[10], a[14]); + MONT_MAC(s0, s1, s2, a[11], a[13]); + MONT_MAC(t0, t1, t2, a[12], a[12]); + MONT_MUL(v0, v1, v2, u[9], m[15]); + MONT_MUL(w0, w1, w2, u[10], m[14]); + MONT_MAC(t0, t1, t2, u[11], m[13]); + MONT_MAC(v0, v1, v2, u[12], m[12]); + MONT_MAC(w0, w1, w2, u[13], m[11]); + MONT_MAC(t0, t1, t2, u[14], m[10]); + MONT_MAC(v0, v1, v2, u[15], m[9]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[8] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 25 */ + MONT_MUL(s0, s1, s2, a[10], a[15]); + MONT_MUL(c0, c1, c2, a[11], a[14]); + MONT_MAC(s0, s1, s2, a[12], a[13]); + MONT_MAC(t0, t1, t2, u[10], m[15]); + MONT_MUL(v0, v1, v2, u[11], m[14]); + MONT_MUL(w0, w1, w2, u[12], m[13]); + MONT_MAC(t0, t1, t2, u[13], m[12]); + MONT_MAC(v0, v1, v2, u[14], m[11]); + MONT_MAC(w0, w1, w2, u[15], m[10]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[9] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 26 */ + MONT_MUL(s0, s1, s2, a[11], a[15]); + MONT_MUL(c0, c1, c2, a[12], a[14]); + MONT_MAC(t0, t1, t2, a[13], a[13]); + MONT_MUL(v0, v1, v2, u[11], m[15]); + MONT_MUL(w0, w1, w2, u[12], m[14]); + MONT_MAC(t0, t1, t2, u[13], m[13]); + MONT_MAC(v0, v1, v2, u[14], m[12]); + MONT_MAC(w0, w1, w2, u[15], m[11]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[10] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 27 */ + MONT_MUL(s0, s1, s2, a[12], a[15]); + MONT_MUL(c0, c1, c2, a[13], a[14]); + MONT_MAC(t0, t1, t2, u[12], m[15]); + MONT_MUL(v0, v1, v2, u[13], m[14]); + MONT_MUL(w0, w1, w2, u[14], m[13]); + MONT_MAC(t0, t1, t2, u[15], m[12]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[11] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 28 */ + MONT_MUL(s0, s1, s2, a[13], a[15]); + MONT_MAC(t0, t1, t2, a[14], a[14]); + MONT_MUL(v0, v1, v2, u[13], m[15]); + MONT_MUL(w0, w1, w2, u[14], m[14]); + MONT_MAC(t0, t1, t2, u[15], m[13]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[12] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 29 */ + MONT_MUL(s0, s1, s2, a[14], a[15]); + MONT_MAC(t0, t1, t2, u[14], m[15]); + MONT_MUL(v0, v1, v2, u[15], m[14]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[13] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 30 */ + MONT_MAC(t0, t1, t2, a[15], a[15]); + MONT_MUL(v0, v1, v2, u[15], m[15]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + z[14] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + z[15] = t0; + + /* under 2m, with t1 the bit above n limbs: at most one subtraction */ + borrow = sub_n(r, z, m, 16); + take = t1 | (borrow ^ 1); + select_n(r, r, z, take & 1, 16); +} + +/* r = a * b * R^-1 mod m, for a and b under m; t is 2n limbs of scratch */ +static void crypton_mont_mul24(limb_t *r, const limb_t *a, const limb_t *b, + const limb_t *m, limb_t n0, limb_t *t) +{ + limb_t *u = t, *z = t + 24; + limb_t t0 = 0, t1 = 0, t2 = 0, v0, v1, v2, w0, w1, w2, borrow, take; + + /* column 0 */ + MONT_MAC(t0, t1, t2, a[0], b[0]); + u[0] = t0 * n0; + MONT_MAC(t0, t1, t2, u[0], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 1 */ + MONT_MAC(t0, t1, t2, a[0], b[1]); + MONT_MUL(v0, v1, v2, a[1], b[0]); + MONT_MUL(w0, w1, w2, u[0], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[1] = t0 * n0; + MONT_MAC(t0, t1, t2, u[1], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 2 */ + MONT_MAC(t0, t1, t2, a[0], b[2]); + MONT_MUL(v0, v1, v2, a[1], b[1]); + MONT_MUL(w0, w1, w2, a[2], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[2]); + MONT_MAC(v0, v1, v2, u[1], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[2] = t0 * n0; + MONT_MAC(t0, t1, t2, u[2], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 3 */ + MONT_MAC(t0, t1, t2, a[0], b[3]); + MONT_MUL(v0, v1, v2, a[1], b[2]); + MONT_MUL(w0, w1, w2, a[2], b[1]); + MONT_MAC(t0, t1, t2, a[3], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[3]); + MONT_MAC(w0, w1, w2, u[1], m[2]); + MONT_MAC(t0, t1, t2, u[2], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[3] = t0 * n0; + MONT_MAC(t0, t1, t2, u[3], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 4 */ + MONT_MAC(t0, t1, t2, a[0], b[4]); + MONT_MUL(v0, v1, v2, a[1], b[3]); + MONT_MUL(w0, w1, w2, a[2], b[2]); + MONT_MAC(t0, t1, t2, a[3], b[1]); + MONT_MAC(v0, v1, v2, a[4], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[4]); + MONT_MAC(t0, t1, t2, u[1], m[3]); + MONT_MAC(v0, v1, v2, u[2], m[2]); + MONT_MAC(w0, w1, w2, u[3], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[4] = t0 * n0; + MONT_MAC(t0, t1, t2, u[4], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 5 */ + MONT_MAC(t0, t1, t2, a[0], b[5]); + MONT_MUL(v0, v1, v2, a[1], b[4]); + MONT_MUL(w0, w1, w2, a[2], b[3]); + MONT_MAC(t0, t1, t2, a[3], b[2]); + MONT_MAC(v0, v1, v2, a[4], b[1]); + MONT_MAC(w0, w1, w2, a[5], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[5]); + MONT_MAC(v0, v1, v2, u[1], m[4]); + MONT_MAC(w0, w1, w2, u[2], m[3]); + MONT_MAC(t0, t1, t2, u[3], m[2]); + MONT_MAC(v0, v1, v2, u[4], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[5] = t0 * n0; + MONT_MAC(t0, t1, t2, u[5], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 6 */ + MONT_MAC(t0, t1, t2, a[0], b[6]); + MONT_MUL(v0, v1, v2, a[1], b[5]); + MONT_MUL(w0, w1, w2, a[2], b[4]); + MONT_MAC(t0, t1, t2, a[3], b[3]); + MONT_MAC(v0, v1, v2, a[4], b[2]); + MONT_MAC(w0, w1, w2, a[5], b[1]); + MONT_MAC(t0, t1, t2, a[6], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[6]); + MONT_MAC(w0, w1, w2, u[1], m[5]); + MONT_MAC(t0, t1, t2, u[2], m[4]); + MONT_MAC(v0, v1, v2, u[3], m[3]); + MONT_MAC(w0, w1, w2, u[4], m[2]); + MONT_MAC(t0, t1, t2, u[5], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[6] = t0 * n0; + MONT_MAC(t0, t1, t2, u[6], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 7 */ + MONT_MAC(t0, t1, t2, a[0], b[7]); + MONT_MUL(v0, v1, v2, a[1], b[6]); + MONT_MUL(w0, w1, w2, a[2], b[5]); + MONT_MAC(t0, t1, t2, a[3], b[4]); + MONT_MAC(v0, v1, v2, a[4], b[3]); + MONT_MAC(w0, w1, w2, a[5], b[2]); + MONT_MAC(t0, t1, t2, a[6], b[1]); + MONT_MAC(v0, v1, v2, a[7], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[7]); + MONT_MAC(t0, t1, t2, u[1], m[6]); + MONT_MAC(v0, v1, v2, u[2], m[5]); + MONT_MAC(w0, w1, w2, u[3], m[4]); + MONT_MAC(t0, t1, t2, u[4], m[3]); + MONT_MAC(v0, v1, v2, u[5], m[2]); + MONT_MAC(w0, w1, w2, u[6], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[7] = t0 * n0; + MONT_MAC(t0, t1, t2, u[7], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 8 */ + MONT_MAC(t0, t1, t2, a[0], b[8]); + MONT_MUL(v0, v1, v2, a[1], b[7]); + MONT_MUL(w0, w1, w2, a[2], b[6]); + MONT_MAC(t0, t1, t2, a[3], b[5]); + MONT_MAC(v0, v1, v2, a[4], b[4]); + MONT_MAC(w0, w1, w2, a[5], b[3]); + MONT_MAC(t0, t1, t2, a[6], b[2]); + MONT_MAC(v0, v1, v2, a[7], b[1]); + MONT_MAC(w0, w1, w2, a[8], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[8]); + MONT_MAC(v0, v1, v2, u[1], m[7]); + MONT_MAC(w0, w1, w2, u[2], m[6]); + MONT_MAC(t0, t1, t2, u[3], m[5]); + MONT_MAC(v0, v1, v2, u[4], m[4]); + MONT_MAC(w0, w1, w2, u[5], m[3]); + MONT_MAC(t0, t1, t2, u[6], m[2]); + MONT_MAC(v0, v1, v2, u[7], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[8] = t0 * n0; + MONT_MAC(t0, t1, t2, u[8], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 9 */ + MONT_MAC(t0, t1, t2, a[0], b[9]); + MONT_MUL(v0, v1, v2, a[1], b[8]); + MONT_MUL(w0, w1, w2, a[2], b[7]); + MONT_MAC(t0, t1, t2, a[3], b[6]); + MONT_MAC(v0, v1, v2, a[4], b[5]); + MONT_MAC(w0, w1, w2, a[5], b[4]); + MONT_MAC(t0, t1, t2, a[6], b[3]); + MONT_MAC(v0, v1, v2, a[7], b[2]); + MONT_MAC(w0, w1, w2, a[8], b[1]); + MONT_MAC(t0, t1, t2, a[9], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[9]); + MONT_MAC(w0, w1, w2, u[1], m[8]); + MONT_MAC(t0, t1, t2, u[2], m[7]); + MONT_MAC(v0, v1, v2, u[3], m[6]); + MONT_MAC(w0, w1, w2, u[4], m[5]); + MONT_MAC(t0, t1, t2, u[5], m[4]); + MONT_MAC(v0, v1, v2, u[6], m[3]); + MONT_MAC(w0, w1, w2, u[7], m[2]); + MONT_MAC(t0, t1, t2, u[8], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[9] = t0 * n0; + MONT_MAC(t0, t1, t2, u[9], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 10 */ + MONT_MAC(t0, t1, t2, a[0], b[10]); + MONT_MUL(v0, v1, v2, a[1], b[9]); + MONT_MUL(w0, w1, w2, a[2], b[8]); + MONT_MAC(t0, t1, t2, a[3], b[7]); + MONT_MAC(v0, v1, v2, a[4], b[6]); + MONT_MAC(w0, w1, w2, a[5], b[5]); + MONT_MAC(t0, t1, t2, a[6], b[4]); + MONT_MAC(v0, v1, v2, a[7], b[3]); + MONT_MAC(w0, w1, w2, a[8], b[2]); + MONT_MAC(t0, t1, t2, a[9], b[1]); + MONT_MAC(v0, v1, v2, a[10], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[10]); + MONT_MAC(t0, t1, t2, u[1], m[9]); + MONT_MAC(v0, v1, v2, u[2], m[8]); + MONT_MAC(w0, w1, w2, u[3], m[7]); + MONT_MAC(t0, t1, t2, u[4], m[6]); + MONT_MAC(v0, v1, v2, u[5], m[5]); + MONT_MAC(w0, w1, w2, u[6], m[4]); + MONT_MAC(t0, t1, t2, u[7], m[3]); + MONT_MAC(v0, v1, v2, u[8], m[2]); + MONT_MAC(w0, w1, w2, u[9], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[10] = t0 * n0; + MONT_MAC(t0, t1, t2, u[10], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 11 */ + MONT_MAC(t0, t1, t2, a[0], b[11]); + MONT_MUL(v0, v1, v2, a[1], b[10]); + MONT_MUL(w0, w1, w2, a[2], b[9]); + MONT_MAC(t0, t1, t2, a[3], b[8]); + MONT_MAC(v0, v1, v2, a[4], b[7]); + MONT_MAC(w0, w1, w2, a[5], b[6]); + MONT_MAC(t0, t1, t2, a[6], b[5]); + MONT_MAC(v0, v1, v2, a[7], b[4]); + MONT_MAC(w0, w1, w2, a[8], b[3]); + MONT_MAC(t0, t1, t2, a[9], b[2]); + MONT_MAC(v0, v1, v2, a[10], b[1]); + MONT_MAC(w0, w1, w2, a[11], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[11]); + MONT_MAC(v0, v1, v2, u[1], m[10]); + MONT_MAC(w0, w1, w2, u[2], m[9]); + MONT_MAC(t0, t1, t2, u[3], m[8]); + MONT_MAC(v0, v1, v2, u[4], m[7]); + MONT_MAC(w0, w1, w2, u[5], m[6]); + MONT_MAC(t0, t1, t2, u[6], m[5]); + MONT_MAC(v0, v1, v2, u[7], m[4]); + MONT_MAC(w0, w1, w2, u[8], m[3]); + MONT_MAC(t0, t1, t2, u[9], m[2]); + MONT_MAC(v0, v1, v2, u[10], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[11] = t0 * n0; + MONT_MAC(t0, t1, t2, u[11], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 12 */ + MONT_MAC(t0, t1, t2, a[0], b[12]); + MONT_MUL(v0, v1, v2, a[1], b[11]); + MONT_MUL(w0, w1, w2, a[2], b[10]); + MONT_MAC(t0, t1, t2, a[3], b[9]); + MONT_MAC(v0, v1, v2, a[4], b[8]); + MONT_MAC(w0, w1, w2, a[5], b[7]); + MONT_MAC(t0, t1, t2, a[6], b[6]); + MONT_MAC(v0, v1, v2, a[7], b[5]); + MONT_MAC(w0, w1, w2, a[8], b[4]); + MONT_MAC(t0, t1, t2, a[9], b[3]); + MONT_MAC(v0, v1, v2, a[10], b[2]); + MONT_MAC(w0, w1, w2, a[11], b[1]); + MONT_MAC(t0, t1, t2, a[12], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[12]); + MONT_MAC(w0, w1, w2, u[1], m[11]); + MONT_MAC(t0, t1, t2, u[2], m[10]); + MONT_MAC(v0, v1, v2, u[3], m[9]); + MONT_MAC(w0, w1, w2, u[4], m[8]); + MONT_MAC(t0, t1, t2, u[5], m[7]); + MONT_MAC(v0, v1, v2, u[6], m[6]); + MONT_MAC(w0, w1, w2, u[7], m[5]); + MONT_MAC(t0, t1, t2, u[8], m[4]); + MONT_MAC(v0, v1, v2, u[9], m[3]); + MONT_MAC(w0, w1, w2, u[10], m[2]); + MONT_MAC(t0, t1, t2, u[11], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[12] = t0 * n0; + MONT_MAC(t0, t1, t2, u[12], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 13 */ + MONT_MAC(t0, t1, t2, a[0], b[13]); + MONT_MUL(v0, v1, v2, a[1], b[12]); + MONT_MUL(w0, w1, w2, a[2], b[11]); + MONT_MAC(t0, t1, t2, a[3], b[10]); + MONT_MAC(v0, v1, v2, a[4], b[9]); + MONT_MAC(w0, w1, w2, a[5], b[8]); + MONT_MAC(t0, t1, t2, a[6], b[7]); + MONT_MAC(v0, v1, v2, a[7], b[6]); + MONT_MAC(w0, w1, w2, a[8], b[5]); + MONT_MAC(t0, t1, t2, a[9], b[4]); + MONT_MAC(v0, v1, v2, a[10], b[3]); + MONT_MAC(w0, w1, w2, a[11], b[2]); + MONT_MAC(t0, t1, t2, a[12], b[1]); + MONT_MAC(v0, v1, v2, a[13], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[13]); + MONT_MAC(t0, t1, t2, u[1], m[12]); + MONT_MAC(v0, v1, v2, u[2], m[11]); + MONT_MAC(w0, w1, w2, u[3], m[10]); + MONT_MAC(t0, t1, t2, u[4], m[9]); + MONT_MAC(v0, v1, v2, u[5], m[8]); + MONT_MAC(w0, w1, w2, u[6], m[7]); + MONT_MAC(t0, t1, t2, u[7], m[6]); + MONT_MAC(v0, v1, v2, u[8], m[5]); + MONT_MAC(w0, w1, w2, u[9], m[4]); + MONT_MAC(t0, t1, t2, u[10], m[3]); + MONT_MAC(v0, v1, v2, u[11], m[2]); + MONT_MAC(w0, w1, w2, u[12], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[13] = t0 * n0; + MONT_MAC(t0, t1, t2, u[13], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 14 */ + MONT_MAC(t0, t1, t2, a[0], b[14]); + MONT_MUL(v0, v1, v2, a[1], b[13]); + MONT_MUL(w0, w1, w2, a[2], b[12]); + MONT_MAC(t0, t1, t2, a[3], b[11]); + MONT_MAC(v0, v1, v2, a[4], b[10]); + MONT_MAC(w0, w1, w2, a[5], b[9]); + MONT_MAC(t0, t1, t2, a[6], b[8]); + MONT_MAC(v0, v1, v2, a[7], b[7]); + MONT_MAC(w0, w1, w2, a[8], b[6]); + MONT_MAC(t0, t1, t2, a[9], b[5]); + MONT_MAC(v0, v1, v2, a[10], b[4]); + MONT_MAC(w0, w1, w2, a[11], b[3]); + MONT_MAC(t0, t1, t2, a[12], b[2]); + MONT_MAC(v0, v1, v2, a[13], b[1]); + MONT_MAC(w0, w1, w2, a[14], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[14]); + MONT_MAC(v0, v1, v2, u[1], m[13]); + MONT_MAC(w0, w1, w2, u[2], m[12]); + MONT_MAC(t0, t1, t2, u[3], m[11]); + MONT_MAC(v0, v1, v2, u[4], m[10]); + MONT_MAC(w0, w1, w2, u[5], m[9]); + MONT_MAC(t0, t1, t2, u[6], m[8]); + MONT_MAC(v0, v1, v2, u[7], m[7]); + MONT_MAC(w0, w1, w2, u[8], m[6]); + MONT_MAC(t0, t1, t2, u[9], m[5]); + MONT_MAC(v0, v1, v2, u[10], m[4]); + MONT_MAC(w0, w1, w2, u[11], m[3]); + MONT_MAC(t0, t1, t2, u[12], m[2]); + MONT_MAC(v0, v1, v2, u[13], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[14] = t0 * n0; + MONT_MAC(t0, t1, t2, u[14], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 15 */ + MONT_MAC(t0, t1, t2, a[0], b[15]); + MONT_MUL(v0, v1, v2, a[1], b[14]); + MONT_MUL(w0, w1, w2, a[2], b[13]); + MONT_MAC(t0, t1, t2, a[3], b[12]); + MONT_MAC(v0, v1, v2, a[4], b[11]); + MONT_MAC(w0, w1, w2, a[5], b[10]); + MONT_MAC(t0, t1, t2, a[6], b[9]); + MONT_MAC(v0, v1, v2, a[7], b[8]); + MONT_MAC(w0, w1, w2, a[8], b[7]); + MONT_MAC(t0, t1, t2, a[9], b[6]); + MONT_MAC(v0, v1, v2, a[10], b[5]); + MONT_MAC(w0, w1, w2, a[11], b[4]); + MONT_MAC(t0, t1, t2, a[12], b[3]); + MONT_MAC(v0, v1, v2, a[13], b[2]); + MONT_MAC(w0, w1, w2, a[14], b[1]); + MONT_MAC(t0, t1, t2, a[15], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[15]); + MONT_MAC(w0, w1, w2, u[1], m[14]); + MONT_MAC(t0, t1, t2, u[2], m[13]); + MONT_MAC(v0, v1, v2, u[3], m[12]); + MONT_MAC(w0, w1, w2, u[4], m[11]); + MONT_MAC(t0, t1, t2, u[5], m[10]); + MONT_MAC(v0, v1, v2, u[6], m[9]); + MONT_MAC(w0, w1, w2, u[7], m[8]); + MONT_MAC(t0, t1, t2, u[8], m[7]); + MONT_MAC(v0, v1, v2, u[9], m[6]); + MONT_MAC(w0, w1, w2, u[10], m[5]); + MONT_MAC(t0, t1, t2, u[11], m[4]); + MONT_MAC(v0, v1, v2, u[12], m[3]); + MONT_MAC(w0, w1, w2, u[13], m[2]); + MONT_MAC(t0, t1, t2, u[14], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[15] = t0 * n0; + MONT_MAC(t0, t1, t2, u[15], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 16 */ + MONT_MAC(t0, t1, t2, a[0], b[16]); + MONT_MUL(v0, v1, v2, a[1], b[15]); + MONT_MUL(w0, w1, w2, a[2], b[14]); + MONT_MAC(t0, t1, t2, a[3], b[13]); + MONT_MAC(v0, v1, v2, a[4], b[12]); + MONT_MAC(w0, w1, w2, a[5], b[11]); + MONT_MAC(t0, t1, t2, a[6], b[10]); + MONT_MAC(v0, v1, v2, a[7], b[9]); + MONT_MAC(w0, w1, w2, a[8], b[8]); + MONT_MAC(t0, t1, t2, a[9], b[7]); + MONT_MAC(v0, v1, v2, a[10], b[6]); + MONT_MAC(w0, w1, w2, a[11], b[5]); + MONT_MAC(t0, t1, t2, a[12], b[4]); + MONT_MAC(v0, v1, v2, a[13], b[3]); + MONT_MAC(w0, w1, w2, a[14], b[2]); + MONT_MAC(t0, t1, t2, a[15], b[1]); + MONT_MAC(v0, v1, v2, a[16], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[16]); + MONT_MAC(t0, t1, t2, u[1], m[15]); + MONT_MAC(v0, v1, v2, u[2], m[14]); + MONT_MAC(w0, w1, w2, u[3], m[13]); + MONT_MAC(t0, t1, t2, u[4], m[12]); + MONT_MAC(v0, v1, v2, u[5], m[11]); + MONT_MAC(w0, w1, w2, u[6], m[10]); + MONT_MAC(t0, t1, t2, u[7], m[9]); + MONT_MAC(v0, v1, v2, u[8], m[8]); + MONT_MAC(w0, w1, w2, u[9], m[7]); + MONT_MAC(t0, t1, t2, u[10], m[6]); + MONT_MAC(v0, v1, v2, u[11], m[5]); + MONT_MAC(w0, w1, w2, u[12], m[4]); + MONT_MAC(t0, t1, t2, u[13], m[3]); + MONT_MAC(v0, v1, v2, u[14], m[2]); + MONT_MAC(w0, w1, w2, u[15], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[16] = t0 * n0; + MONT_MAC(t0, t1, t2, u[16], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 17 */ + MONT_MAC(t0, t1, t2, a[0], b[17]); + MONT_MUL(v0, v1, v2, a[1], b[16]); + MONT_MUL(w0, w1, w2, a[2], b[15]); + MONT_MAC(t0, t1, t2, a[3], b[14]); + MONT_MAC(v0, v1, v2, a[4], b[13]); + MONT_MAC(w0, w1, w2, a[5], b[12]); + MONT_MAC(t0, t1, t2, a[6], b[11]); + MONT_MAC(v0, v1, v2, a[7], b[10]); + MONT_MAC(w0, w1, w2, a[8], b[9]); + MONT_MAC(t0, t1, t2, a[9], b[8]); + MONT_MAC(v0, v1, v2, a[10], b[7]); + MONT_MAC(w0, w1, w2, a[11], b[6]); + MONT_MAC(t0, t1, t2, a[12], b[5]); + MONT_MAC(v0, v1, v2, a[13], b[4]); + MONT_MAC(w0, w1, w2, a[14], b[3]); + MONT_MAC(t0, t1, t2, a[15], b[2]); + MONT_MAC(v0, v1, v2, a[16], b[1]); + MONT_MAC(w0, w1, w2, a[17], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[17]); + MONT_MAC(v0, v1, v2, u[1], m[16]); + MONT_MAC(w0, w1, w2, u[2], m[15]); + MONT_MAC(t0, t1, t2, u[3], m[14]); + MONT_MAC(v0, v1, v2, u[4], m[13]); + MONT_MAC(w0, w1, w2, u[5], m[12]); + MONT_MAC(t0, t1, t2, u[6], m[11]); + MONT_MAC(v0, v1, v2, u[7], m[10]); + MONT_MAC(w0, w1, w2, u[8], m[9]); + MONT_MAC(t0, t1, t2, u[9], m[8]); + MONT_MAC(v0, v1, v2, u[10], m[7]); + MONT_MAC(w0, w1, w2, u[11], m[6]); + MONT_MAC(t0, t1, t2, u[12], m[5]); + MONT_MAC(v0, v1, v2, u[13], m[4]); + MONT_MAC(w0, w1, w2, u[14], m[3]); + MONT_MAC(t0, t1, t2, u[15], m[2]); + MONT_MAC(v0, v1, v2, u[16], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[17] = t0 * n0; + MONT_MAC(t0, t1, t2, u[17], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 18 */ + MONT_MAC(t0, t1, t2, a[0], b[18]); + MONT_MUL(v0, v1, v2, a[1], b[17]); + MONT_MUL(w0, w1, w2, a[2], b[16]); + MONT_MAC(t0, t1, t2, a[3], b[15]); + MONT_MAC(v0, v1, v2, a[4], b[14]); + MONT_MAC(w0, w1, w2, a[5], b[13]); + MONT_MAC(t0, t1, t2, a[6], b[12]); + MONT_MAC(v0, v1, v2, a[7], b[11]); + MONT_MAC(w0, w1, w2, a[8], b[10]); + MONT_MAC(t0, t1, t2, a[9], b[9]); + MONT_MAC(v0, v1, v2, a[10], b[8]); + MONT_MAC(w0, w1, w2, a[11], b[7]); + MONT_MAC(t0, t1, t2, a[12], b[6]); + MONT_MAC(v0, v1, v2, a[13], b[5]); + MONT_MAC(w0, w1, w2, a[14], b[4]); + MONT_MAC(t0, t1, t2, a[15], b[3]); + MONT_MAC(v0, v1, v2, a[16], b[2]); + MONT_MAC(w0, w1, w2, a[17], b[1]); + MONT_MAC(t0, t1, t2, a[18], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[18]); + MONT_MAC(w0, w1, w2, u[1], m[17]); + MONT_MAC(t0, t1, t2, u[2], m[16]); + MONT_MAC(v0, v1, v2, u[3], m[15]); + MONT_MAC(w0, w1, w2, u[4], m[14]); + MONT_MAC(t0, t1, t2, u[5], m[13]); + MONT_MAC(v0, v1, v2, u[6], m[12]); + MONT_MAC(w0, w1, w2, u[7], m[11]); + MONT_MAC(t0, t1, t2, u[8], m[10]); + MONT_MAC(v0, v1, v2, u[9], m[9]); + MONT_MAC(w0, w1, w2, u[10], m[8]); + MONT_MAC(t0, t1, t2, u[11], m[7]); + MONT_MAC(v0, v1, v2, u[12], m[6]); + MONT_MAC(w0, w1, w2, u[13], m[5]); + MONT_MAC(t0, t1, t2, u[14], m[4]); + MONT_MAC(v0, v1, v2, u[15], m[3]); + MONT_MAC(w0, w1, w2, u[16], m[2]); + MONT_MAC(t0, t1, t2, u[17], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[18] = t0 * n0; + MONT_MAC(t0, t1, t2, u[18], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 19 */ + MONT_MAC(t0, t1, t2, a[0], b[19]); + MONT_MUL(v0, v1, v2, a[1], b[18]); + MONT_MUL(w0, w1, w2, a[2], b[17]); + MONT_MAC(t0, t1, t2, a[3], b[16]); + MONT_MAC(v0, v1, v2, a[4], b[15]); + MONT_MAC(w0, w1, w2, a[5], b[14]); + MONT_MAC(t0, t1, t2, a[6], b[13]); + MONT_MAC(v0, v1, v2, a[7], b[12]); + MONT_MAC(w0, w1, w2, a[8], b[11]); + MONT_MAC(t0, t1, t2, a[9], b[10]); + MONT_MAC(v0, v1, v2, a[10], b[9]); + MONT_MAC(w0, w1, w2, a[11], b[8]); + MONT_MAC(t0, t1, t2, a[12], b[7]); + MONT_MAC(v0, v1, v2, a[13], b[6]); + MONT_MAC(w0, w1, w2, a[14], b[5]); + MONT_MAC(t0, t1, t2, a[15], b[4]); + MONT_MAC(v0, v1, v2, a[16], b[3]); + MONT_MAC(w0, w1, w2, a[17], b[2]); + MONT_MAC(t0, t1, t2, a[18], b[1]); + MONT_MAC(v0, v1, v2, a[19], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[19]); + MONT_MAC(t0, t1, t2, u[1], m[18]); + MONT_MAC(v0, v1, v2, u[2], m[17]); + MONT_MAC(w0, w1, w2, u[3], m[16]); + MONT_MAC(t0, t1, t2, u[4], m[15]); + MONT_MAC(v0, v1, v2, u[5], m[14]); + MONT_MAC(w0, w1, w2, u[6], m[13]); + MONT_MAC(t0, t1, t2, u[7], m[12]); + MONT_MAC(v0, v1, v2, u[8], m[11]); + MONT_MAC(w0, w1, w2, u[9], m[10]); + MONT_MAC(t0, t1, t2, u[10], m[9]); + MONT_MAC(v0, v1, v2, u[11], m[8]); + MONT_MAC(w0, w1, w2, u[12], m[7]); + MONT_MAC(t0, t1, t2, u[13], m[6]); + MONT_MAC(v0, v1, v2, u[14], m[5]); + MONT_MAC(w0, w1, w2, u[15], m[4]); + MONT_MAC(t0, t1, t2, u[16], m[3]); + MONT_MAC(v0, v1, v2, u[17], m[2]); + MONT_MAC(w0, w1, w2, u[18], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[19] = t0 * n0; + MONT_MAC(t0, t1, t2, u[19], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 20 */ + MONT_MAC(t0, t1, t2, a[0], b[20]); + MONT_MUL(v0, v1, v2, a[1], b[19]); + MONT_MUL(w0, w1, w2, a[2], b[18]); + MONT_MAC(t0, t1, t2, a[3], b[17]); + MONT_MAC(v0, v1, v2, a[4], b[16]); + MONT_MAC(w0, w1, w2, a[5], b[15]); + MONT_MAC(t0, t1, t2, a[6], b[14]); + MONT_MAC(v0, v1, v2, a[7], b[13]); + MONT_MAC(w0, w1, w2, a[8], b[12]); + MONT_MAC(t0, t1, t2, a[9], b[11]); + MONT_MAC(v0, v1, v2, a[10], b[10]); + MONT_MAC(w0, w1, w2, a[11], b[9]); + MONT_MAC(t0, t1, t2, a[12], b[8]); + MONT_MAC(v0, v1, v2, a[13], b[7]); + MONT_MAC(w0, w1, w2, a[14], b[6]); + MONT_MAC(t0, t1, t2, a[15], b[5]); + MONT_MAC(v0, v1, v2, a[16], b[4]); + MONT_MAC(w0, w1, w2, a[17], b[3]); + MONT_MAC(t0, t1, t2, a[18], b[2]); + MONT_MAC(v0, v1, v2, a[19], b[1]); + MONT_MAC(w0, w1, w2, a[20], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[20]); + MONT_MAC(v0, v1, v2, u[1], m[19]); + MONT_MAC(w0, w1, w2, u[2], m[18]); + MONT_MAC(t0, t1, t2, u[3], m[17]); + MONT_MAC(v0, v1, v2, u[4], m[16]); + MONT_MAC(w0, w1, w2, u[5], m[15]); + MONT_MAC(t0, t1, t2, u[6], m[14]); + MONT_MAC(v0, v1, v2, u[7], m[13]); + MONT_MAC(w0, w1, w2, u[8], m[12]); + MONT_MAC(t0, t1, t2, u[9], m[11]); + MONT_MAC(v0, v1, v2, u[10], m[10]); + MONT_MAC(w0, w1, w2, u[11], m[9]); + MONT_MAC(t0, t1, t2, u[12], m[8]); + MONT_MAC(v0, v1, v2, u[13], m[7]); + MONT_MAC(w0, w1, w2, u[14], m[6]); + MONT_MAC(t0, t1, t2, u[15], m[5]); + MONT_MAC(v0, v1, v2, u[16], m[4]); + MONT_MAC(w0, w1, w2, u[17], m[3]); + MONT_MAC(t0, t1, t2, u[18], m[2]); + MONT_MAC(v0, v1, v2, u[19], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[20] = t0 * n0; + MONT_MAC(t0, t1, t2, u[20], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 21 */ + MONT_MAC(t0, t1, t2, a[0], b[21]); + MONT_MUL(v0, v1, v2, a[1], b[20]); + MONT_MUL(w0, w1, w2, a[2], b[19]); + MONT_MAC(t0, t1, t2, a[3], b[18]); + MONT_MAC(v0, v1, v2, a[4], b[17]); + MONT_MAC(w0, w1, w2, a[5], b[16]); + MONT_MAC(t0, t1, t2, a[6], b[15]); + MONT_MAC(v0, v1, v2, a[7], b[14]); + MONT_MAC(w0, w1, w2, a[8], b[13]); + MONT_MAC(t0, t1, t2, a[9], b[12]); + MONT_MAC(v0, v1, v2, a[10], b[11]); + MONT_MAC(w0, w1, w2, a[11], b[10]); + MONT_MAC(t0, t1, t2, a[12], b[9]); + MONT_MAC(v0, v1, v2, a[13], b[8]); + MONT_MAC(w0, w1, w2, a[14], b[7]); + MONT_MAC(t0, t1, t2, a[15], b[6]); + MONT_MAC(v0, v1, v2, a[16], b[5]); + MONT_MAC(w0, w1, w2, a[17], b[4]); + MONT_MAC(t0, t1, t2, a[18], b[3]); + MONT_MAC(v0, v1, v2, a[19], b[2]); + MONT_MAC(w0, w1, w2, a[20], b[1]); + MONT_MAC(t0, t1, t2, a[21], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[21]); + MONT_MAC(w0, w1, w2, u[1], m[20]); + MONT_MAC(t0, t1, t2, u[2], m[19]); + MONT_MAC(v0, v1, v2, u[3], m[18]); + MONT_MAC(w0, w1, w2, u[4], m[17]); + MONT_MAC(t0, t1, t2, u[5], m[16]); + MONT_MAC(v0, v1, v2, u[6], m[15]); + MONT_MAC(w0, w1, w2, u[7], m[14]); + MONT_MAC(t0, t1, t2, u[8], m[13]); + MONT_MAC(v0, v1, v2, u[9], m[12]); + MONT_MAC(w0, w1, w2, u[10], m[11]); + MONT_MAC(t0, t1, t2, u[11], m[10]); + MONT_MAC(v0, v1, v2, u[12], m[9]); + MONT_MAC(w0, w1, w2, u[13], m[8]); + MONT_MAC(t0, t1, t2, u[14], m[7]); + MONT_MAC(v0, v1, v2, u[15], m[6]); + MONT_MAC(w0, w1, w2, u[16], m[5]); + MONT_MAC(t0, t1, t2, u[17], m[4]); + MONT_MAC(v0, v1, v2, u[18], m[3]); + MONT_MAC(w0, w1, w2, u[19], m[2]); + MONT_MAC(t0, t1, t2, u[20], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[21] = t0 * n0; + MONT_MAC(t0, t1, t2, u[21], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 22 */ + MONT_MAC(t0, t1, t2, a[0], b[22]); + MONT_MUL(v0, v1, v2, a[1], b[21]); + MONT_MUL(w0, w1, w2, a[2], b[20]); + MONT_MAC(t0, t1, t2, a[3], b[19]); + MONT_MAC(v0, v1, v2, a[4], b[18]); + MONT_MAC(w0, w1, w2, a[5], b[17]); + MONT_MAC(t0, t1, t2, a[6], b[16]); + MONT_MAC(v0, v1, v2, a[7], b[15]); + MONT_MAC(w0, w1, w2, a[8], b[14]); + MONT_MAC(t0, t1, t2, a[9], b[13]); + MONT_MAC(v0, v1, v2, a[10], b[12]); + MONT_MAC(w0, w1, w2, a[11], b[11]); + MONT_MAC(t0, t1, t2, a[12], b[10]); + MONT_MAC(v0, v1, v2, a[13], b[9]); + MONT_MAC(w0, w1, w2, a[14], b[8]); + MONT_MAC(t0, t1, t2, a[15], b[7]); + MONT_MAC(v0, v1, v2, a[16], b[6]); + MONT_MAC(w0, w1, w2, a[17], b[5]); + MONT_MAC(t0, t1, t2, a[18], b[4]); + MONT_MAC(v0, v1, v2, a[19], b[3]); + MONT_MAC(w0, w1, w2, a[20], b[2]); + MONT_MAC(t0, t1, t2, a[21], b[1]); + MONT_MAC(v0, v1, v2, a[22], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[22]); + MONT_MAC(t0, t1, t2, u[1], m[21]); + MONT_MAC(v0, v1, v2, u[2], m[20]); + MONT_MAC(w0, w1, w2, u[3], m[19]); + MONT_MAC(t0, t1, t2, u[4], m[18]); + MONT_MAC(v0, v1, v2, u[5], m[17]); + MONT_MAC(w0, w1, w2, u[6], m[16]); + MONT_MAC(t0, t1, t2, u[7], m[15]); + MONT_MAC(v0, v1, v2, u[8], m[14]); + MONT_MAC(w0, w1, w2, u[9], m[13]); + MONT_MAC(t0, t1, t2, u[10], m[12]); + MONT_MAC(v0, v1, v2, u[11], m[11]); + MONT_MAC(w0, w1, w2, u[12], m[10]); + MONT_MAC(t0, t1, t2, u[13], m[9]); + MONT_MAC(v0, v1, v2, u[14], m[8]); + MONT_MAC(w0, w1, w2, u[15], m[7]); + MONT_MAC(t0, t1, t2, u[16], m[6]); + MONT_MAC(v0, v1, v2, u[17], m[5]); + MONT_MAC(w0, w1, w2, u[18], m[4]); + MONT_MAC(t0, t1, t2, u[19], m[3]); + MONT_MAC(v0, v1, v2, u[20], m[2]); + MONT_MAC(w0, w1, w2, u[21], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[22] = t0 * n0; + MONT_MAC(t0, t1, t2, u[22], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 23 */ + MONT_MAC(t0, t1, t2, a[0], b[23]); + MONT_MUL(v0, v1, v2, a[1], b[22]); + MONT_MUL(w0, w1, w2, a[2], b[21]); + MONT_MAC(t0, t1, t2, a[3], b[20]); + MONT_MAC(v0, v1, v2, a[4], b[19]); + MONT_MAC(w0, w1, w2, a[5], b[18]); + MONT_MAC(t0, t1, t2, a[6], b[17]); + MONT_MAC(v0, v1, v2, a[7], b[16]); + MONT_MAC(w0, w1, w2, a[8], b[15]); + MONT_MAC(t0, t1, t2, a[9], b[14]); + MONT_MAC(v0, v1, v2, a[10], b[13]); + MONT_MAC(w0, w1, w2, a[11], b[12]); + MONT_MAC(t0, t1, t2, a[12], b[11]); + MONT_MAC(v0, v1, v2, a[13], b[10]); + MONT_MAC(w0, w1, w2, a[14], b[9]); + MONT_MAC(t0, t1, t2, a[15], b[8]); + MONT_MAC(v0, v1, v2, a[16], b[7]); + MONT_MAC(w0, w1, w2, a[17], b[6]); + MONT_MAC(t0, t1, t2, a[18], b[5]); + MONT_MAC(v0, v1, v2, a[19], b[4]); + MONT_MAC(w0, w1, w2, a[20], b[3]); + MONT_MAC(t0, t1, t2, a[21], b[2]); + MONT_MAC(v0, v1, v2, a[22], b[1]); + MONT_MAC(w0, w1, w2, a[23], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[23]); + MONT_MAC(v0, v1, v2, u[1], m[22]); + MONT_MAC(w0, w1, w2, u[2], m[21]); + MONT_MAC(t0, t1, t2, u[3], m[20]); + MONT_MAC(v0, v1, v2, u[4], m[19]); + MONT_MAC(w0, w1, w2, u[5], m[18]); + MONT_MAC(t0, t1, t2, u[6], m[17]); + MONT_MAC(v0, v1, v2, u[7], m[16]); + MONT_MAC(w0, w1, w2, u[8], m[15]); + MONT_MAC(t0, t1, t2, u[9], m[14]); + MONT_MAC(v0, v1, v2, u[10], m[13]); + MONT_MAC(w0, w1, w2, u[11], m[12]); + MONT_MAC(t0, t1, t2, u[12], m[11]); + MONT_MAC(v0, v1, v2, u[13], m[10]); + MONT_MAC(w0, w1, w2, u[14], m[9]); + MONT_MAC(t0, t1, t2, u[15], m[8]); + MONT_MAC(v0, v1, v2, u[16], m[7]); + MONT_MAC(w0, w1, w2, u[17], m[6]); + MONT_MAC(t0, t1, t2, u[18], m[5]); + MONT_MAC(v0, v1, v2, u[19], m[4]); + MONT_MAC(w0, w1, w2, u[20], m[3]); + MONT_MAC(t0, t1, t2, u[21], m[2]); + MONT_MAC(v0, v1, v2, u[22], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[23] = t0 * n0; + MONT_MAC(t0, t1, t2, u[23], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 24 */ + MONT_MAC(t0, t1, t2, a[1], b[23]); + MONT_MUL(v0, v1, v2, a[2], b[22]); + MONT_MUL(w0, w1, w2, a[3], b[21]); + MONT_MAC(t0, t1, t2, a[4], b[20]); + MONT_MAC(v0, v1, v2, a[5], b[19]); + MONT_MAC(w0, w1, w2, a[6], b[18]); + MONT_MAC(t0, t1, t2, a[7], b[17]); + MONT_MAC(v0, v1, v2, a[8], b[16]); + MONT_MAC(w0, w1, w2, a[9], b[15]); + MONT_MAC(t0, t1, t2, a[10], b[14]); + MONT_MAC(v0, v1, v2, a[11], b[13]); + MONT_MAC(w0, w1, w2, a[12], b[12]); + MONT_MAC(t0, t1, t2, a[13], b[11]); + MONT_MAC(v0, v1, v2, a[14], b[10]); + MONT_MAC(w0, w1, w2, a[15], b[9]); + MONT_MAC(t0, t1, t2, a[16], b[8]); + MONT_MAC(v0, v1, v2, a[17], b[7]); + MONT_MAC(w0, w1, w2, a[18], b[6]); + MONT_MAC(t0, t1, t2, a[19], b[5]); + MONT_MAC(v0, v1, v2, a[20], b[4]); + MONT_MAC(w0, w1, w2, a[21], b[3]); + MONT_MAC(t0, t1, t2, a[22], b[2]); + MONT_MAC(v0, v1, v2, a[23], b[1]); + MONT_MAC(w0, w1, w2, u[1], m[23]); + MONT_MAC(t0, t1, t2, u[2], m[22]); + MONT_MAC(v0, v1, v2, u[3], m[21]); + MONT_MAC(w0, w1, w2, u[4], m[20]); + MONT_MAC(t0, t1, t2, u[5], m[19]); + MONT_MAC(v0, v1, v2, u[6], m[18]); + MONT_MAC(w0, w1, w2, u[7], m[17]); + MONT_MAC(t0, t1, t2, u[8], m[16]); + MONT_MAC(v0, v1, v2, u[9], m[15]); + MONT_MAC(w0, w1, w2, u[10], m[14]); + MONT_MAC(t0, t1, t2, u[11], m[13]); + MONT_MAC(v0, v1, v2, u[12], m[12]); + MONT_MAC(w0, w1, w2, u[13], m[11]); + MONT_MAC(t0, t1, t2, u[14], m[10]); + MONT_MAC(v0, v1, v2, u[15], m[9]); + MONT_MAC(w0, w1, w2, u[16], m[8]); + MONT_MAC(t0, t1, t2, u[17], m[7]); + MONT_MAC(v0, v1, v2, u[18], m[6]); + MONT_MAC(w0, w1, w2, u[19], m[5]); + MONT_MAC(t0, t1, t2, u[20], m[4]); + MONT_MAC(v0, v1, v2, u[21], m[3]); + MONT_MAC(w0, w1, w2, u[22], m[2]); + MONT_MAC(t0, t1, t2, u[23], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[0] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 25 */ + MONT_MAC(t0, t1, t2, a[2], b[23]); + MONT_MUL(v0, v1, v2, a[3], b[22]); + MONT_MUL(w0, w1, w2, a[4], b[21]); + MONT_MAC(t0, t1, t2, a[5], b[20]); + MONT_MAC(v0, v1, v2, a[6], b[19]); + MONT_MAC(w0, w1, w2, a[7], b[18]); + MONT_MAC(t0, t1, t2, a[8], b[17]); + MONT_MAC(v0, v1, v2, a[9], b[16]); + MONT_MAC(w0, w1, w2, a[10], b[15]); + MONT_MAC(t0, t1, t2, a[11], b[14]); + MONT_MAC(v0, v1, v2, a[12], b[13]); + MONT_MAC(w0, w1, w2, a[13], b[12]); + MONT_MAC(t0, t1, t2, a[14], b[11]); + MONT_MAC(v0, v1, v2, a[15], b[10]); + MONT_MAC(w0, w1, w2, a[16], b[9]); + MONT_MAC(t0, t1, t2, a[17], b[8]); + MONT_MAC(v0, v1, v2, a[18], b[7]); + MONT_MAC(w0, w1, w2, a[19], b[6]); + MONT_MAC(t0, t1, t2, a[20], b[5]); + MONT_MAC(v0, v1, v2, a[21], b[4]); + MONT_MAC(w0, w1, w2, a[22], b[3]); + MONT_MAC(t0, t1, t2, a[23], b[2]); + MONT_MAC(v0, v1, v2, u[2], m[23]); + MONT_MAC(w0, w1, w2, u[3], m[22]); + MONT_MAC(t0, t1, t2, u[4], m[21]); + MONT_MAC(v0, v1, v2, u[5], m[20]); + MONT_MAC(w0, w1, w2, u[6], m[19]); + MONT_MAC(t0, t1, t2, u[7], m[18]); + MONT_MAC(v0, v1, v2, u[8], m[17]); + MONT_MAC(w0, w1, w2, u[9], m[16]); + MONT_MAC(t0, t1, t2, u[10], m[15]); + MONT_MAC(v0, v1, v2, u[11], m[14]); + MONT_MAC(w0, w1, w2, u[12], m[13]); + MONT_MAC(t0, t1, t2, u[13], m[12]); + MONT_MAC(v0, v1, v2, u[14], m[11]); + MONT_MAC(w0, w1, w2, u[15], m[10]); + MONT_MAC(t0, t1, t2, u[16], m[9]); + MONT_MAC(v0, v1, v2, u[17], m[8]); + MONT_MAC(w0, w1, w2, u[18], m[7]); + MONT_MAC(t0, t1, t2, u[19], m[6]); + MONT_MAC(v0, v1, v2, u[20], m[5]); + MONT_MAC(w0, w1, w2, u[21], m[4]); + MONT_MAC(t0, t1, t2, u[22], m[3]); + MONT_MAC(v0, v1, v2, u[23], m[2]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[1] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 26 */ + MONT_MAC(t0, t1, t2, a[3], b[23]); + MONT_MUL(v0, v1, v2, a[4], b[22]); + MONT_MUL(w0, w1, w2, a[5], b[21]); + MONT_MAC(t0, t1, t2, a[6], b[20]); + MONT_MAC(v0, v1, v2, a[7], b[19]); + MONT_MAC(w0, w1, w2, a[8], b[18]); + MONT_MAC(t0, t1, t2, a[9], b[17]); + MONT_MAC(v0, v1, v2, a[10], b[16]); + MONT_MAC(w0, w1, w2, a[11], b[15]); + MONT_MAC(t0, t1, t2, a[12], b[14]); + MONT_MAC(v0, v1, v2, a[13], b[13]); + MONT_MAC(w0, w1, w2, a[14], b[12]); + MONT_MAC(t0, t1, t2, a[15], b[11]); + MONT_MAC(v0, v1, v2, a[16], b[10]); + MONT_MAC(w0, w1, w2, a[17], b[9]); + MONT_MAC(t0, t1, t2, a[18], b[8]); + MONT_MAC(v0, v1, v2, a[19], b[7]); + MONT_MAC(w0, w1, w2, a[20], b[6]); + MONT_MAC(t0, t1, t2, a[21], b[5]); + MONT_MAC(v0, v1, v2, a[22], b[4]); + MONT_MAC(w0, w1, w2, a[23], b[3]); + MONT_MAC(t0, t1, t2, u[3], m[23]); + MONT_MAC(v0, v1, v2, u[4], m[22]); + MONT_MAC(w0, w1, w2, u[5], m[21]); + MONT_MAC(t0, t1, t2, u[6], m[20]); + MONT_MAC(v0, v1, v2, u[7], m[19]); + MONT_MAC(w0, w1, w2, u[8], m[18]); + MONT_MAC(t0, t1, t2, u[9], m[17]); + MONT_MAC(v0, v1, v2, u[10], m[16]); + MONT_MAC(w0, w1, w2, u[11], m[15]); + MONT_MAC(t0, t1, t2, u[12], m[14]); + MONT_MAC(v0, v1, v2, u[13], m[13]); + MONT_MAC(w0, w1, w2, u[14], m[12]); + MONT_MAC(t0, t1, t2, u[15], m[11]); + MONT_MAC(v0, v1, v2, u[16], m[10]); + MONT_MAC(w0, w1, w2, u[17], m[9]); + MONT_MAC(t0, t1, t2, u[18], m[8]); + MONT_MAC(v0, v1, v2, u[19], m[7]); + MONT_MAC(w0, w1, w2, u[20], m[6]); + MONT_MAC(t0, t1, t2, u[21], m[5]); + MONT_MAC(v0, v1, v2, u[22], m[4]); + MONT_MAC(w0, w1, w2, u[23], m[3]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[2] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 27 */ + MONT_MAC(t0, t1, t2, a[4], b[23]); + MONT_MUL(v0, v1, v2, a[5], b[22]); + MONT_MUL(w0, w1, w2, a[6], b[21]); + MONT_MAC(t0, t1, t2, a[7], b[20]); + MONT_MAC(v0, v1, v2, a[8], b[19]); + MONT_MAC(w0, w1, w2, a[9], b[18]); + MONT_MAC(t0, t1, t2, a[10], b[17]); + MONT_MAC(v0, v1, v2, a[11], b[16]); + MONT_MAC(w0, w1, w2, a[12], b[15]); + MONT_MAC(t0, t1, t2, a[13], b[14]); + MONT_MAC(v0, v1, v2, a[14], b[13]); + MONT_MAC(w0, w1, w2, a[15], b[12]); + MONT_MAC(t0, t1, t2, a[16], b[11]); + MONT_MAC(v0, v1, v2, a[17], b[10]); + MONT_MAC(w0, w1, w2, a[18], b[9]); + MONT_MAC(t0, t1, t2, a[19], b[8]); + MONT_MAC(v0, v1, v2, a[20], b[7]); + MONT_MAC(w0, w1, w2, a[21], b[6]); + MONT_MAC(t0, t1, t2, a[22], b[5]); + MONT_MAC(v0, v1, v2, a[23], b[4]); + MONT_MAC(w0, w1, w2, u[4], m[23]); + MONT_MAC(t0, t1, t2, u[5], m[22]); + MONT_MAC(v0, v1, v2, u[6], m[21]); + MONT_MAC(w0, w1, w2, u[7], m[20]); + MONT_MAC(t0, t1, t2, u[8], m[19]); + MONT_MAC(v0, v1, v2, u[9], m[18]); + MONT_MAC(w0, w1, w2, u[10], m[17]); + MONT_MAC(t0, t1, t2, u[11], m[16]); + MONT_MAC(v0, v1, v2, u[12], m[15]); + MONT_MAC(w0, w1, w2, u[13], m[14]); + MONT_MAC(t0, t1, t2, u[14], m[13]); + MONT_MAC(v0, v1, v2, u[15], m[12]); + MONT_MAC(w0, w1, w2, u[16], m[11]); + MONT_MAC(t0, t1, t2, u[17], m[10]); + MONT_MAC(v0, v1, v2, u[18], m[9]); + MONT_MAC(w0, w1, w2, u[19], m[8]); + MONT_MAC(t0, t1, t2, u[20], m[7]); + MONT_MAC(v0, v1, v2, u[21], m[6]); + MONT_MAC(w0, w1, w2, u[22], m[5]); + MONT_MAC(t0, t1, t2, u[23], m[4]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[3] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 28 */ + MONT_MAC(t0, t1, t2, a[5], b[23]); + MONT_MUL(v0, v1, v2, a[6], b[22]); + MONT_MUL(w0, w1, w2, a[7], b[21]); + MONT_MAC(t0, t1, t2, a[8], b[20]); + MONT_MAC(v0, v1, v2, a[9], b[19]); + MONT_MAC(w0, w1, w2, a[10], b[18]); + MONT_MAC(t0, t1, t2, a[11], b[17]); + MONT_MAC(v0, v1, v2, a[12], b[16]); + MONT_MAC(w0, w1, w2, a[13], b[15]); + MONT_MAC(t0, t1, t2, a[14], b[14]); + MONT_MAC(v0, v1, v2, a[15], b[13]); + MONT_MAC(w0, w1, w2, a[16], b[12]); + MONT_MAC(t0, t1, t2, a[17], b[11]); + MONT_MAC(v0, v1, v2, a[18], b[10]); + MONT_MAC(w0, w1, w2, a[19], b[9]); + MONT_MAC(t0, t1, t2, a[20], b[8]); + MONT_MAC(v0, v1, v2, a[21], b[7]); + MONT_MAC(w0, w1, w2, a[22], b[6]); + MONT_MAC(t0, t1, t2, a[23], b[5]); + MONT_MAC(v0, v1, v2, u[5], m[23]); + MONT_MAC(w0, w1, w2, u[6], m[22]); + MONT_MAC(t0, t1, t2, u[7], m[21]); + MONT_MAC(v0, v1, v2, u[8], m[20]); + MONT_MAC(w0, w1, w2, u[9], m[19]); + MONT_MAC(t0, t1, t2, u[10], m[18]); + MONT_MAC(v0, v1, v2, u[11], m[17]); + MONT_MAC(w0, w1, w2, u[12], m[16]); + MONT_MAC(t0, t1, t2, u[13], m[15]); + MONT_MAC(v0, v1, v2, u[14], m[14]); + MONT_MAC(w0, w1, w2, u[15], m[13]); + MONT_MAC(t0, t1, t2, u[16], m[12]); + MONT_MAC(v0, v1, v2, u[17], m[11]); + MONT_MAC(w0, w1, w2, u[18], m[10]); + MONT_MAC(t0, t1, t2, u[19], m[9]); + MONT_MAC(v0, v1, v2, u[20], m[8]); + MONT_MAC(w0, w1, w2, u[21], m[7]); + MONT_MAC(t0, t1, t2, u[22], m[6]); + MONT_MAC(v0, v1, v2, u[23], m[5]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[4] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 29 */ + MONT_MAC(t0, t1, t2, a[6], b[23]); + MONT_MUL(v0, v1, v2, a[7], b[22]); + MONT_MUL(w0, w1, w2, a[8], b[21]); + MONT_MAC(t0, t1, t2, a[9], b[20]); + MONT_MAC(v0, v1, v2, a[10], b[19]); + MONT_MAC(w0, w1, w2, a[11], b[18]); + MONT_MAC(t0, t1, t2, a[12], b[17]); + MONT_MAC(v0, v1, v2, a[13], b[16]); + MONT_MAC(w0, w1, w2, a[14], b[15]); + MONT_MAC(t0, t1, t2, a[15], b[14]); + MONT_MAC(v0, v1, v2, a[16], b[13]); + MONT_MAC(w0, w1, w2, a[17], b[12]); + MONT_MAC(t0, t1, t2, a[18], b[11]); + MONT_MAC(v0, v1, v2, a[19], b[10]); + MONT_MAC(w0, w1, w2, a[20], b[9]); + MONT_MAC(t0, t1, t2, a[21], b[8]); + MONT_MAC(v0, v1, v2, a[22], b[7]); + MONT_MAC(w0, w1, w2, a[23], b[6]); + MONT_MAC(t0, t1, t2, u[6], m[23]); + MONT_MAC(v0, v1, v2, u[7], m[22]); + MONT_MAC(w0, w1, w2, u[8], m[21]); + MONT_MAC(t0, t1, t2, u[9], m[20]); + MONT_MAC(v0, v1, v2, u[10], m[19]); + MONT_MAC(w0, w1, w2, u[11], m[18]); + MONT_MAC(t0, t1, t2, u[12], m[17]); + MONT_MAC(v0, v1, v2, u[13], m[16]); + MONT_MAC(w0, w1, w2, u[14], m[15]); + MONT_MAC(t0, t1, t2, u[15], m[14]); + MONT_MAC(v0, v1, v2, u[16], m[13]); + MONT_MAC(w0, w1, w2, u[17], m[12]); + MONT_MAC(t0, t1, t2, u[18], m[11]); + MONT_MAC(v0, v1, v2, u[19], m[10]); + MONT_MAC(w0, w1, w2, u[20], m[9]); + MONT_MAC(t0, t1, t2, u[21], m[8]); + MONT_MAC(v0, v1, v2, u[22], m[7]); + MONT_MAC(w0, w1, w2, u[23], m[6]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[5] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 30 */ + MONT_MAC(t0, t1, t2, a[7], b[23]); + MONT_MUL(v0, v1, v2, a[8], b[22]); + MONT_MUL(w0, w1, w2, a[9], b[21]); + MONT_MAC(t0, t1, t2, a[10], b[20]); + MONT_MAC(v0, v1, v2, a[11], b[19]); + MONT_MAC(w0, w1, w2, a[12], b[18]); + MONT_MAC(t0, t1, t2, a[13], b[17]); + MONT_MAC(v0, v1, v2, a[14], b[16]); + MONT_MAC(w0, w1, w2, a[15], b[15]); + MONT_MAC(t0, t1, t2, a[16], b[14]); + MONT_MAC(v0, v1, v2, a[17], b[13]); + MONT_MAC(w0, w1, w2, a[18], b[12]); + MONT_MAC(t0, t1, t2, a[19], b[11]); + MONT_MAC(v0, v1, v2, a[20], b[10]); + MONT_MAC(w0, w1, w2, a[21], b[9]); + MONT_MAC(t0, t1, t2, a[22], b[8]); + MONT_MAC(v0, v1, v2, a[23], b[7]); + MONT_MAC(w0, w1, w2, u[7], m[23]); + MONT_MAC(t0, t1, t2, u[8], m[22]); + MONT_MAC(v0, v1, v2, u[9], m[21]); + MONT_MAC(w0, w1, w2, u[10], m[20]); + MONT_MAC(t0, t1, t2, u[11], m[19]); + MONT_MAC(v0, v1, v2, u[12], m[18]); + MONT_MAC(w0, w1, w2, u[13], m[17]); + MONT_MAC(t0, t1, t2, u[14], m[16]); + MONT_MAC(v0, v1, v2, u[15], m[15]); + MONT_MAC(w0, w1, w2, u[16], m[14]); + MONT_MAC(t0, t1, t2, u[17], m[13]); + MONT_MAC(v0, v1, v2, u[18], m[12]); + MONT_MAC(w0, w1, w2, u[19], m[11]); + MONT_MAC(t0, t1, t2, u[20], m[10]); + MONT_MAC(v0, v1, v2, u[21], m[9]); + MONT_MAC(w0, w1, w2, u[22], m[8]); + MONT_MAC(t0, t1, t2, u[23], m[7]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[6] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 31 */ + MONT_MAC(t0, t1, t2, a[8], b[23]); + MONT_MUL(v0, v1, v2, a[9], b[22]); + MONT_MUL(w0, w1, w2, a[10], b[21]); + MONT_MAC(t0, t1, t2, a[11], b[20]); + MONT_MAC(v0, v1, v2, a[12], b[19]); + MONT_MAC(w0, w1, w2, a[13], b[18]); + MONT_MAC(t0, t1, t2, a[14], b[17]); + MONT_MAC(v0, v1, v2, a[15], b[16]); + MONT_MAC(w0, w1, w2, a[16], b[15]); + MONT_MAC(t0, t1, t2, a[17], b[14]); + MONT_MAC(v0, v1, v2, a[18], b[13]); + MONT_MAC(w0, w1, w2, a[19], b[12]); + MONT_MAC(t0, t1, t2, a[20], b[11]); + MONT_MAC(v0, v1, v2, a[21], b[10]); + MONT_MAC(w0, w1, w2, a[22], b[9]); + MONT_MAC(t0, t1, t2, a[23], b[8]); + MONT_MAC(v0, v1, v2, u[8], m[23]); + MONT_MAC(w0, w1, w2, u[9], m[22]); + MONT_MAC(t0, t1, t2, u[10], m[21]); + MONT_MAC(v0, v1, v2, u[11], m[20]); + MONT_MAC(w0, w1, w2, u[12], m[19]); + MONT_MAC(t0, t1, t2, u[13], m[18]); + MONT_MAC(v0, v1, v2, u[14], m[17]); + MONT_MAC(w0, w1, w2, u[15], m[16]); + MONT_MAC(t0, t1, t2, u[16], m[15]); + MONT_MAC(v0, v1, v2, u[17], m[14]); + MONT_MAC(w0, w1, w2, u[18], m[13]); + MONT_MAC(t0, t1, t2, u[19], m[12]); + MONT_MAC(v0, v1, v2, u[20], m[11]); + MONT_MAC(w0, w1, w2, u[21], m[10]); + MONT_MAC(t0, t1, t2, u[22], m[9]); + MONT_MAC(v0, v1, v2, u[23], m[8]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[7] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 32 */ + MONT_MAC(t0, t1, t2, a[9], b[23]); + MONT_MUL(v0, v1, v2, a[10], b[22]); + MONT_MUL(w0, w1, w2, a[11], b[21]); + MONT_MAC(t0, t1, t2, a[12], b[20]); + MONT_MAC(v0, v1, v2, a[13], b[19]); + MONT_MAC(w0, w1, w2, a[14], b[18]); + MONT_MAC(t0, t1, t2, a[15], b[17]); + MONT_MAC(v0, v1, v2, a[16], b[16]); + MONT_MAC(w0, w1, w2, a[17], b[15]); + MONT_MAC(t0, t1, t2, a[18], b[14]); + MONT_MAC(v0, v1, v2, a[19], b[13]); + MONT_MAC(w0, w1, w2, a[20], b[12]); + MONT_MAC(t0, t1, t2, a[21], b[11]); + MONT_MAC(v0, v1, v2, a[22], b[10]); + MONT_MAC(w0, w1, w2, a[23], b[9]); + MONT_MAC(t0, t1, t2, u[9], m[23]); + MONT_MAC(v0, v1, v2, u[10], m[22]); + MONT_MAC(w0, w1, w2, u[11], m[21]); + MONT_MAC(t0, t1, t2, u[12], m[20]); + MONT_MAC(v0, v1, v2, u[13], m[19]); + MONT_MAC(w0, w1, w2, u[14], m[18]); + MONT_MAC(t0, t1, t2, u[15], m[17]); + MONT_MAC(v0, v1, v2, u[16], m[16]); + MONT_MAC(w0, w1, w2, u[17], m[15]); + MONT_MAC(t0, t1, t2, u[18], m[14]); + MONT_MAC(v0, v1, v2, u[19], m[13]); + MONT_MAC(w0, w1, w2, u[20], m[12]); + MONT_MAC(t0, t1, t2, u[21], m[11]); + MONT_MAC(v0, v1, v2, u[22], m[10]); + MONT_MAC(w0, w1, w2, u[23], m[9]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[8] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 33 */ + MONT_MAC(t0, t1, t2, a[10], b[23]); + MONT_MUL(v0, v1, v2, a[11], b[22]); + MONT_MUL(w0, w1, w2, a[12], b[21]); + MONT_MAC(t0, t1, t2, a[13], b[20]); + MONT_MAC(v0, v1, v2, a[14], b[19]); + MONT_MAC(w0, w1, w2, a[15], b[18]); + MONT_MAC(t0, t1, t2, a[16], b[17]); + MONT_MAC(v0, v1, v2, a[17], b[16]); + MONT_MAC(w0, w1, w2, a[18], b[15]); + MONT_MAC(t0, t1, t2, a[19], b[14]); + MONT_MAC(v0, v1, v2, a[20], b[13]); + MONT_MAC(w0, w1, w2, a[21], b[12]); + MONT_MAC(t0, t1, t2, a[22], b[11]); + MONT_MAC(v0, v1, v2, a[23], b[10]); + MONT_MAC(w0, w1, w2, u[10], m[23]); + MONT_MAC(t0, t1, t2, u[11], m[22]); + MONT_MAC(v0, v1, v2, u[12], m[21]); + MONT_MAC(w0, w1, w2, u[13], m[20]); + MONT_MAC(t0, t1, t2, u[14], m[19]); + MONT_MAC(v0, v1, v2, u[15], m[18]); + MONT_MAC(w0, w1, w2, u[16], m[17]); + MONT_MAC(t0, t1, t2, u[17], m[16]); + MONT_MAC(v0, v1, v2, u[18], m[15]); + MONT_MAC(w0, w1, w2, u[19], m[14]); + MONT_MAC(t0, t1, t2, u[20], m[13]); + MONT_MAC(v0, v1, v2, u[21], m[12]); + MONT_MAC(w0, w1, w2, u[22], m[11]); + MONT_MAC(t0, t1, t2, u[23], m[10]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[9] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 34 */ + MONT_MAC(t0, t1, t2, a[11], b[23]); + MONT_MUL(v0, v1, v2, a[12], b[22]); + MONT_MUL(w0, w1, w2, a[13], b[21]); + MONT_MAC(t0, t1, t2, a[14], b[20]); + MONT_MAC(v0, v1, v2, a[15], b[19]); + MONT_MAC(w0, w1, w2, a[16], b[18]); + MONT_MAC(t0, t1, t2, a[17], b[17]); + MONT_MAC(v0, v1, v2, a[18], b[16]); + MONT_MAC(w0, w1, w2, a[19], b[15]); + MONT_MAC(t0, t1, t2, a[20], b[14]); + MONT_MAC(v0, v1, v2, a[21], b[13]); + MONT_MAC(w0, w1, w2, a[22], b[12]); + MONT_MAC(t0, t1, t2, a[23], b[11]); + MONT_MAC(v0, v1, v2, u[11], m[23]); + MONT_MAC(w0, w1, w2, u[12], m[22]); + MONT_MAC(t0, t1, t2, u[13], m[21]); + MONT_MAC(v0, v1, v2, u[14], m[20]); + MONT_MAC(w0, w1, w2, u[15], m[19]); + MONT_MAC(t0, t1, t2, u[16], m[18]); + MONT_MAC(v0, v1, v2, u[17], m[17]); + MONT_MAC(w0, w1, w2, u[18], m[16]); + MONT_MAC(t0, t1, t2, u[19], m[15]); + MONT_MAC(v0, v1, v2, u[20], m[14]); + MONT_MAC(w0, w1, w2, u[21], m[13]); + MONT_MAC(t0, t1, t2, u[22], m[12]); + MONT_MAC(v0, v1, v2, u[23], m[11]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[10] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 35 */ + MONT_MAC(t0, t1, t2, a[12], b[23]); + MONT_MUL(v0, v1, v2, a[13], b[22]); + MONT_MUL(w0, w1, w2, a[14], b[21]); + MONT_MAC(t0, t1, t2, a[15], b[20]); + MONT_MAC(v0, v1, v2, a[16], b[19]); + MONT_MAC(w0, w1, w2, a[17], b[18]); + MONT_MAC(t0, t1, t2, a[18], b[17]); + MONT_MAC(v0, v1, v2, a[19], b[16]); + MONT_MAC(w0, w1, w2, a[20], b[15]); + MONT_MAC(t0, t1, t2, a[21], b[14]); + MONT_MAC(v0, v1, v2, a[22], b[13]); + MONT_MAC(w0, w1, w2, a[23], b[12]); + MONT_MAC(t0, t1, t2, u[12], m[23]); + MONT_MAC(v0, v1, v2, u[13], m[22]); + MONT_MAC(w0, w1, w2, u[14], m[21]); + MONT_MAC(t0, t1, t2, u[15], m[20]); + MONT_MAC(v0, v1, v2, u[16], m[19]); + MONT_MAC(w0, w1, w2, u[17], m[18]); + MONT_MAC(t0, t1, t2, u[18], m[17]); + MONT_MAC(v0, v1, v2, u[19], m[16]); + MONT_MAC(w0, w1, w2, u[20], m[15]); + MONT_MAC(t0, t1, t2, u[21], m[14]); + MONT_MAC(v0, v1, v2, u[22], m[13]); + MONT_MAC(w0, w1, w2, u[23], m[12]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[11] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 36 */ + MONT_MAC(t0, t1, t2, a[13], b[23]); + MONT_MUL(v0, v1, v2, a[14], b[22]); + MONT_MUL(w0, w1, w2, a[15], b[21]); + MONT_MAC(t0, t1, t2, a[16], b[20]); + MONT_MAC(v0, v1, v2, a[17], b[19]); + MONT_MAC(w0, w1, w2, a[18], b[18]); + MONT_MAC(t0, t1, t2, a[19], b[17]); + MONT_MAC(v0, v1, v2, a[20], b[16]); + MONT_MAC(w0, w1, w2, a[21], b[15]); + MONT_MAC(t0, t1, t2, a[22], b[14]); + MONT_MAC(v0, v1, v2, a[23], b[13]); + MONT_MAC(w0, w1, w2, u[13], m[23]); + MONT_MAC(t0, t1, t2, u[14], m[22]); + MONT_MAC(v0, v1, v2, u[15], m[21]); + MONT_MAC(w0, w1, w2, u[16], m[20]); + MONT_MAC(t0, t1, t2, u[17], m[19]); + MONT_MAC(v0, v1, v2, u[18], m[18]); + MONT_MAC(w0, w1, w2, u[19], m[17]); + MONT_MAC(t0, t1, t2, u[20], m[16]); + MONT_MAC(v0, v1, v2, u[21], m[15]); + MONT_MAC(w0, w1, w2, u[22], m[14]); + MONT_MAC(t0, t1, t2, u[23], m[13]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[12] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 37 */ + MONT_MAC(t0, t1, t2, a[14], b[23]); + MONT_MUL(v0, v1, v2, a[15], b[22]); + MONT_MUL(w0, w1, w2, a[16], b[21]); + MONT_MAC(t0, t1, t2, a[17], b[20]); + MONT_MAC(v0, v1, v2, a[18], b[19]); + MONT_MAC(w0, w1, w2, a[19], b[18]); + MONT_MAC(t0, t1, t2, a[20], b[17]); + MONT_MAC(v0, v1, v2, a[21], b[16]); + MONT_MAC(w0, w1, w2, a[22], b[15]); + MONT_MAC(t0, t1, t2, a[23], b[14]); + MONT_MAC(v0, v1, v2, u[14], m[23]); + MONT_MAC(w0, w1, w2, u[15], m[22]); + MONT_MAC(t0, t1, t2, u[16], m[21]); + MONT_MAC(v0, v1, v2, u[17], m[20]); + MONT_MAC(w0, w1, w2, u[18], m[19]); + MONT_MAC(t0, t1, t2, u[19], m[18]); + MONT_MAC(v0, v1, v2, u[20], m[17]); + MONT_MAC(w0, w1, w2, u[21], m[16]); + MONT_MAC(t0, t1, t2, u[22], m[15]); + MONT_MAC(v0, v1, v2, u[23], m[14]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[13] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 38 */ + MONT_MAC(t0, t1, t2, a[15], b[23]); + MONT_MUL(v0, v1, v2, a[16], b[22]); + MONT_MUL(w0, w1, w2, a[17], b[21]); + MONT_MAC(t0, t1, t2, a[18], b[20]); + MONT_MAC(v0, v1, v2, a[19], b[19]); + MONT_MAC(w0, w1, w2, a[20], b[18]); + MONT_MAC(t0, t1, t2, a[21], b[17]); + MONT_MAC(v0, v1, v2, a[22], b[16]); + MONT_MAC(w0, w1, w2, a[23], b[15]); + MONT_MAC(t0, t1, t2, u[15], m[23]); + MONT_MAC(v0, v1, v2, u[16], m[22]); + MONT_MAC(w0, w1, w2, u[17], m[21]); + MONT_MAC(t0, t1, t2, u[18], m[20]); + MONT_MAC(v0, v1, v2, u[19], m[19]); + MONT_MAC(w0, w1, w2, u[20], m[18]); + MONT_MAC(t0, t1, t2, u[21], m[17]); + MONT_MAC(v0, v1, v2, u[22], m[16]); + MONT_MAC(w0, w1, w2, u[23], m[15]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[14] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 39 */ + MONT_MAC(t0, t1, t2, a[16], b[23]); + MONT_MUL(v0, v1, v2, a[17], b[22]); + MONT_MUL(w0, w1, w2, a[18], b[21]); + MONT_MAC(t0, t1, t2, a[19], b[20]); + MONT_MAC(v0, v1, v2, a[20], b[19]); + MONT_MAC(w0, w1, w2, a[21], b[18]); + MONT_MAC(t0, t1, t2, a[22], b[17]); + MONT_MAC(v0, v1, v2, a[23], b[16]); + MONT_MAC(w0, w1, w2, u[16], m[23]); + MONT_MAC(t0, t1, t2, u[17], m[22]); + MONT_MAC(v0, v1, v2, u[18], m[21]); + MONT_MAC(w0, w1, w2, u[19], m[20]); + MONT_MAC(t0, t1, t2, u[20], m[19]); + MONT_MAC(v0, v1, v2, u[21], m[18]); + MONT_MAC(w0, w1, w2, u[22], m[17]); + MONT_MAC(t0, t1, t2, u[23], m[16]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[15] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 40 */ + MONT_MAC(t0, t1, t2, a[17], b[23]); + MONT_MUL(v0, v1, v2, a[18], b[22]); + MONT_MUL(w0, w1, w2, a[19], b[21]); + MONT_MAC(t0, t1, t2, a[20], b[20]); + MONT_MAC(v0, v1, v2, a[21], b[19]); + MONT_MAC(w0, w1, w2, a[22], b[18]); + MONT_MAC(t0, t1, t2, a[23], b[17]); + MONT_MAC(v0, v1, v2, u[17], m[23]); + MONT_MAC(w0, w1, w2, u[18], m[22]); + MONT_MAC(t0, t1, t2, u[19], m[21]); + MONT_MAC(v0, v1, v2, u[20], m[20]); + MONT_MAC(w0, w1, w2, u[21], m[19]); + MONT_MAC(t0, t1, t2, u[22], m[18]); + MONT_MAC(v0, v1, v2, u[23], m[17]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[16] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 41 */ + MONT_MAC(t0, t1, t2, a[18], b[23]); + MONT_MUL(v0, v1, v2, a[19], b[22]); + MONT_MUL(w0, w1, w2, a[20], b[21]); + MONT_MAC(t0, t1, t2, a[21], b[20]); + MONT_MAC(v0, v1, v2, a[22], b[19]); + MONT_MAC(w0, w1, w2, a[23], b[18]); + MONT_MAC(t0, t1, t2, u[18], m[23]); + MONT_MAC(v0, v1, v2, u[19], m[22]); + MONT_MAC(w0, w1, w2, u[20], m[21]); + MONT_MAC(t0, t1, t2, u[21], m[20]); + MONT_MAC(v0, v1, v2, u[22], m[19]); + MONT_MAC(w0, w1, w2, u[23], m[18]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[17] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 42 */ + MONT_MAC(t0, t1, t2, a[19], b[23]); + MONT_MUL(v0, v1, v2, a[20], b[22]); + MONT_MUL(w0, w1, w2, a[21], b[21]); + MONT_MAC(t0, t1, t2, a[22], b[20]); + MONT_MAC(v0, v1, v2, a[23], b[19]); + MONT_MAC(w0, w1, w2, u[19], m[23]); + MONT_MAC(t0, t1, t2, u[20], m[22]); + MONT_MAC(v0, v1, v2, u[21], m[21]); + MONT_MAC(w0, w1, w2, u[22], m[20]); + MONT_MAC(t0, t1, t2, u[23], m[19]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[18] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 43 */ + MONT_MAC(t0, t1, t2, a[20], b[23]); + MONT_MUL(v0, v1, v2, a[21], b[22]); + MONT_MUL(w0, w1, w2, a[22], b[21]); + MONT_MAC(t0, t1, t2, a[23], b[20]); + MONT_MAC(v0, v1, v2, u[20], m[23]); + MONT_MAC(w0, w1, w2, u[21], m[22]); + MONT_MAC(t0, t1, t2, u[22], m[21]); + MONT_MAC(v0, v1, v2, u[23], m[20]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[19] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 44 */ + MONT_MAC(t0, t1, t2, a[21], b[23]); + MONT_MUL(v0, v1, v2, a[22], b[22]); + MONT_MUL(w0, w1, w2, a[23], b[21]); + MONT_MAC(t0, t1, t2, u[21], m[23]); + MONT_MAC(v0, v1, v2, u[22], m[22]); + MONT_MAC(w0, w1, w2, u[23], m[21]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[20] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 45 */ + MONT_MAC(t0, t1, t2, a[22], b[23]); + MONT_MUL(v0, v1, v2, a[23], b[22]); + MONT_MUL(w0, w1, w2, u[22], m[23]); + MONT_MAC(t0, t1, t2, u[23], m[22]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[21] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 46 */ + MONT_MAC(t0, t1, t2, a[23], b[23]); + MONT_MUL(v0, v1, v2, u[23], m[23]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + z[22] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + z[23] = t0; + + /* under 2m, with t1 the bit above n limbs: at most one subtraction */ + borrow = sub_n(r, z, m, 24); + take = t1 | (borrow ^ 1); + select_n(r, r, z, take & 1, 24); +} + +/* r = a * a * R^-1 mod m, for a under m; t is 2n limbs of scratch */ +static void crypton_mont_sqr24(limb_t *r, const limb_t *a, + const limb_t *m, limb_t n0, limb_t *t) +{ + limb_t *u = t, *z = t + 24; + limb_t t0 = 0, t1 = 0, t2 = 0, v0, v1, v2, w0, w1, w2, borrow, take; + limb_t s0, s1, s2, c0, c1, c2; + + /* column 0 */ + MONT_MAC(t0, t1, t2, a[0], a[0]); + u[0] = t0 * n0; + MONT_MAC(t0, t1, t2, u[0], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 1 */ + MONT_MUL(s0, s1, s2, a[0], a[1]); + MONT_MAC(t0, t1, t2, u[0], m[1]); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[1] = t0 * n0; + MONT_MAC(t0, t1, t2, u[1], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 2 */ + MONT_MUL(s0, s1, s2, a[0], a[2]); + MONT_MAC(t0, t1, t2, a[1], a[1]); + MONT_MUL(v0, v1, v2, u[0], m[2]); + MONT_MUL(w0, w1, w2, u[1], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[2] = t0 * n0; + MONT_MAC(t0, t1, t2, u[2], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 3 */ + MONT_MUL(s0, s1, s2, a[0], a[3]); + MONT_MUL(c0, c1, c2, a[1], a[2]); + MONT_MAC(t0, t1, t2, u[0], m[3]); + MONT_MUL(v0, v1, v2, u[1], m[2]); + MONT_MUL(w0, w1, w2, u[2], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[3] = t0 * n0; + MONT_MAC(t0, t1, t2, u[3], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 4 */ + MONT_MUL(s0, s1, s2, a[0], a[4]); + MONT_MUL(c0, c1, c2, a[1], a[3]); + MONT_MAC(t0, t1, t2, a[2], a[2]); + MONT_MUL(v0, v1, v2, u[0], m[4]); + MONT_MUL(w0, w1, w2, u[1], m[3]); + MONT_MAC(t0, t1, t2, u[2], m[2]); + MONT_MAC(v0, v1, v2, u[3], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[4] = t0 * n0; + MONT_MAC(t0, t1, t2, u[4], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 5 */ + MONT_MUL(s0, s1, s2, a[0], a[5]); + MONT_MUL(c0, c1, c2, a[1], a[4]); + MONT_MAC(s0, s1, s2, a[2], a[3]); + MONT_MAC(t0, t1, t2, u[0], m[5]); + MONT_MUL(v0, v1, v2, u[1], m[4]); + MONT_MUL(w0, w1, w2, u[2], m[3]); + MONT_MAC(t0, t1, t2, u[3], m[2]); + MONT_MAC(v0, v1, v2, u[4], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[5] = t0 * n0; + MONT_MAC(t0, t1, t2, u[5], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 6 */ + MONT_MUL(s0, s1, s2, a[0], a[6]); + MONT_MUL(c0, c1, c2, a[1], a[5]); + MONT_MAC(s0, s1, s2, a[2], a[4]); + MONT_MAC(t0, t1, t2, a[3], a[3]); + MONT_MUL(v0, v1, v2, u[0], m[6]); + MONT_MUL(w0, w1, w2, u[1], m[5]); + MONT_MAC(t0, t1, t2, u[2], m[4]); + MONT_MAC(v0, v1, v2, u[3], m[3]); + MONT_MAC(w0, w1, w2, u[4], m[2]); + MONT_MAC(t0, t1, t2, u[5], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[6] = t0 * n0; + MONT_MAC(t0, t1, t2, u[6], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 7 */ + MONT_MUL(s0, s1, s2, a[0], a[7]); + MONT_MUL(c0, c1, c2, a[1], a[6]); + MONT_MAC(s0, s1, s2, a[2], a[5]); + MONT_MAC(c0, c1, c2, a[3], a[4]); + MONT_MAC(t0, t1, t2, u[0], m[7]); + MONT_MUL(v0, v1, v2, u[1], m[6]); + MONT_MUL(w0, w1, w2, u[2], m[5]); + MONT_MAC(t0, t1, t2, u[3], m[4]); + MONT_MAC(v0, v1, v2, u[4], m[3]); + MONT_MAC(w0, w1, w2, u[5], m[2]); + MONT_MAC(t0, t1, t2, u[6], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[7] = t0 * n0; + MONT_MAC(t0, t1, t2, u[7], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 8 */ + MONT_MUL(s0, s1, s2, a[0], a[8]); + MONT_MUL(c0, c1, c2, a[1], a[7]); + MONT_MAC(s0, s1, s2, a[2], a[6]); + MONT_MAC(c0, c1, c2, a[3], a[5]); + MONT_MAC(t0, t1, t2, a[4], a[4]); + MONT_MUL(v0, v1, v2, u[0], m[8]); + MONT_MUL(w0, w1, w2, u[1], m[7]); + MONT_MAC(t0, t1, t2, u[2], m[6]); + MONT_MAC(v0, v1, v2, u[3], m[5]); + MONT_MAC(w0, w1, w2, u[4], m[4]); + MONT_MAC(t0, t1, t2, u[5], m[3]); + MONT_MAC(v0, v1, v2, u[6], m[2]); + MONT_MAC(w0, w1, w2, u[7], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[8] = t0 * n0; + MONT_MAC(t0, t1, t2, u[8], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 9 */ + MONT_MUL(s0, s1, s2, a[0], a[9]); + MONT_MUL(c0, c1, c2, a[1], a[8]); + MONT_MAC(s0, s1, s2, a[2], a[7]); + MONT_MAC(c0, c1, c2, a[3], a[6]); + MONT_MAC(s0, s1, s2, a[4], a[5]); + MONT_MAC(t0, t1, t2, u[0], m[9]); + MONT_MUL(v0, v1, v2, u[1], m[8]); + MONT_MUL(w0, w1, w2, u[2], m[7]); + MONT_MAC(t0, t1, t2, u[3], m[6]); + MONT_MAC(v0, v1, v2, u[4], m[5]); + MONT_MAC(w0, w1, w2, u[5], m[4]); + MONT_MAC(t0, t1, t2, u[6], m[3]); + MONT_MAC(v0, v1, v2, u[7], m[2]); + MONT_MAC(w0, w1, w2, u[8], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[9] = t0 * n0; + MONT_MAC(t0, t1, t2, u[9], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 10 */ + MONT_MUL(s0, s1, s2, a[0], a[10]); + MONT_MUL(c0, c1, c2, a[1], a[9]); + MONT_MAC(s0, s1, s2, a[2], a[8]); + MONT_MAC(c0, c1, c2, a[3], a[7]); + MONT_MAC(s0, s1, s2, a[4], a[6]); + MONT_MAC(t0, t1, t2, a[5], a[5]); + MONT_MUL(v0, v1, v2, u[0], m[10]); + MONT_MUL(w0, w1, w2, u[1], m[9]); + MONT_MAC(t0, t1, t2, u[2], m[8]); + MONT_MAC(v0, v1, v2, u[3], m[7]); + MONT_MAC(w0, w1, w2, u[4], m[6]); + MONT_MAC(t0, t1, t2, u[5], m[5]); + MONT_MAC(v0, v1, v2, u[6], m[4]); + MONT_MAC(w0, w1, w2, u[7], m[3]); + MONT_MAC(t0, t1, t2, u[8], m[2]); + MONT_MAC(v0, v1, v2, u[9], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[10] = t0 * n0; + MONT_MAC(t0, t1, t2, u[10], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 11 */ + MONT_MUL(s0, s1, s2, a[0], a[11]); + MONT_MUL(c0, c1, c2, a[1], a[10]); + MONT_MAC(s0, s1, s2, a[2], a[9]); + MONT_MAC(c0, c1, c2, a[3], a[8]); + MONT_MAC(s0, s1, s2, a[4], a[7]); + MONT_MAC(c0, c1, c2, a[5], a[6]); + MONT_MAC(t0, t1, t2, u[0], m[11]); + MONT_MUL(v0, v1, v2, u[1], m[10]); + MONT_MUL(w0, w1, w2, u[2], m[9]); + MONT_MAC(t0, t1, t2, u[3], m[8]); + MONT_MAC(v0, v1, v2, u[4], m[7]); + MONT_MAC(w0, w1, w2, u[5], m[6]); + MONT_MAC(t0, t1, t2, u[6], m[5]); + MONT_MAC(v0, v1, v2, u[7], m[4]); + MONT_MAC(w0, w1, w2, u[8], m[3]); + MONT_MAC(t0, t1, t2, u[9], m[2]); + MONT_MAC(v0, v1, v2, u[10], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[11] = t0 * n0; + MONT_MAC(t0, t1, t2, u[11], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 12 */ + MONT_MUL(s0, s1, s2, a[0], a[12]); + MONT_MUL(c0, c1, c2, a[1], a[11]); + MONT_MAC(s0, s1, s2, a[2], a[10]); + MONT_MAC(c0, c1, c2, a[3], a[9]); + MONT_MAC(s0, s1, s2, a[4], a[8]); + MONT_MAC(c0, c1, c2, a[5], a[7]); + MONT_MAC(t0, t1, t2, a[6], a[6]); + MONT_MUL(v0, v1, v2, u[0], m[12]); + MONT_MUL(w0, w1, w2, u[1], m[11]); + MONT_MAC(t0, t1, t2, u[2], m[10]); + MONT_MAC(v0, v1, v2, u[3], m[9]); + MONT_MAC(w0, w1, w2, u[4], m[8]); + MONT_MAC(t0, t1, t2, u[5], m[7]); + MONT_MAC(v0, v1, v2, u[6], m[6]); + MONT_MAC(w0, w1, w2, u[7], m[5]); + MONT_MAC(t0, t1, t2, u[8], m[4]); + MONT_MAC(v0, v1, v2, u[9], m[3]); + MONT_MAC(w0, w1, w2, u[10], m[2]); + MONT_MAC(t0, t1, t2, u[11], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[12] = t0 * n0; + MONT_MAC(t0, t1, t2, u[12], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 13 */ + MONT_MUL(s0, s1, s2, a[0], a[13]); + MONT_MUL(c0, c1, c2, a[1], a[12]); + MONT_MAC(s0, s1, s2, a[2], a[11]); + MONT_MAC(c0, c1, c2, a[3], a[10]); + MONT_MAC(s0, s1, s2, a[4], a[9]); + MONT_MAC(c0, c1, c2, a[5], a[8]); + MONT_MAC(s0, s1, s2, a[6], a[7]); + MONT_MAC(t0, t1, t2, u[0], m[13]); + MONT_MUL(v0, v1, v2, u[1], m[12]); + MONT_MUL(w0, w1, w2, u[2], m[11]); + MONT_MAC(t0, t1, t2, u[3], m[10]); + MONT_MAC(v0, v1, v2, u[4], m[9]); + MONT_MAC(w0, w1, w2, u[5], m[8]); + MONT_MAC(t0, t1, t2, u[6], m[7]); + MONT_MAC(v0, v1, v2, u[7], m[6]); + MONT_MAC(w0, w1, w2, u[8], m[5]); + MONT_MAC(t0, t1, t2, u[9], m[4]); + MONT_MAC(v0, v1, v2, u[10], m[3]); + MONT_MAC(w0, w1, w2, u[11], m[2]); + MONT_MAC(t0, t1, t2, u[12], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[13] = t0 * n0; + MONT_MAC(t0, t1, t2, u[13], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 14 */ + MONT_MUL(s0, s1, s2, a[0], a[14]); + MONT_MUL(c0, c1, c2, a[1], a[13]); + MONT_MAC(s0, s1, s2, a[2], a[12]); + MONT_MAC(c0, c1, c2, a[3], a[11]); + MONT_MAC(s0, s1, s2, a[4], a[10]); + MONT_MAC(c0, c1, c2, a[5], a[9]); + MONT_MAC(s0, s1, s2, a[6], a[8]); + MONT_MAC(t0, t1, t2, a[7], a[7]); + MONT_MUL(v0, v1, v2, u[0], m[14]); + MONT_MUL(w0, w1, w2, u[1], m[13]); + MONT_MAC(t0, t1, t2, u[2], m[12]); + MONT_MAC(v0, v1, v2, u[3], m[11]); + MONT_MAC(w0, w1, w2, u[4], m[10]); + MONT_MAC(t0, t1, t2, u[5], m[9]); + MONT_MAC(v0, v1, v2, u[6], m[8]); + MONT_MAC(w0, w1, w2, u[7], m[7]); + MONT_MAC(t0, t1, t2, u[8], m[6]); + MONT_MAC(v0, v1, v2, u[9], m[5]); + MONT_MAC(w0, w1, w2, u[10], m[4]); + MONT_MAC(t0, t1, t2, u[11], m[3]); + MONT_MAC(v0, v1, v2, u[12], m[2]); + MONT_MAC(w0, w1, w2, u[13], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[14] = t0 * n0; + MONT_MAC(t0, t1, t2, u[14], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 15 */ + MONT_MUL(s0, s1, s2, a[0], a[15]); + MONT_MUL(c0, c1, c2, a[1], a[14]); + MONT_MAC(s0, s1, s2, a[2], a[13]); + MONT_MAC(c0, c1, c2, a[3], a[12]); + MONT_MAC(s0, s1, s2, a[4], a[11]); + MONT_MAC(c0, c1, c2, a[5], a[10]); + MONT_MAC(s0, s1, s2, a[6], a[9]); + MONT_MAC(c0, c1, c2, a[7], a[8]); + MONT_MAC(t0, t1, t2, u[0], m[15]); + MONT_MUL(v0, v1, v2, u[1], m[14]); + MONT_MUL(w0, w1, w2, u[2], m[13]); + MONT_MAC(t0, t1, t2, u[3], m[12]); + MONT_MAC(v0, v1, v2, u[4], m[11]); + MONT_MAC(w0, w1, w2, u[5], m[10]); + MONT_MAC(t0, t1, t2, u[6], m[9]); + MONT_MAC(v0, v1, v2, u[7], m[8]); + MONT_MAC(w0, w1, w2, u[8], m[7]); + MONT_MAC(t0, t1, t2, u[9], m[6]); + MONT_MAC(v0, v1, v2, u[10], m[5]); + MONT_MAC(w0, w1, w2, u[11], m[4]); + MONT_MAC(t0, t1, t2, u[12], m[3]); + MONT_MAC(v0, v1, v2, u[13], m[2]); + MONT_MAC(w0, w1, w2, u[14], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[15] = t0 * n0; + MONT_MAC(t0, t1, t2, u[15], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 16 */ + MONT_MUL(s0, s1, s2, a[0], a[16]); + MONT_MUL(c0, c1, c2, a[1], a[15]); + MONT_MAC(s0, s1, s2, a[2], a[14]); + MONT_MAC(c0, c1, c2, a[3], a[13]); + MONT_MAC(s0, s1, s2, a[4], a[12]); + MONT_MAC(c0, c1, c2, a[5], a[11]); + MONT_MAC(s0, s1, s2, a[6], a[10]); + MONT_MAC(c0, c1, c2, a[7], a[9]); + MONT_MAC(t0, t1, t2, a[8], a[8]); + MONT_MUL(v0, v1, v2, u[0], m[16]); + MONT_MUL(w0, w1, w2, u[1], m[15]); + MONT_MAC(t0, t1, t2, u[2], m[14]); + MONT_MAC(v0, v1, v2, u[3], m[13]); + MONT_MAC(w0, w1, w2, u[4], m[12]); + MONT_MAC(t0, t1, t2, u[5], m[11]); + MONT_MAC(v0, v1, v2, u[6], m[10]); + MONT_MAC(w0, w1, w2, u[7], m[9]); + MONT_MAC(t0, t1, t2, u[8], m[8]); + MONT_MAC(v0, v1, v2, u[9], m[7]); + MONT_MAC(w0, w1, w2, u[10], m[6]); + MONT_MAC(t0, t1, t2, u[11], m[5]); + MONT_MAC(v0, v1, v2, u[12], m[4]); + MONT_MAC(w0, w1, w2, u[13], m[3]); + MONT_MAC(t0, t1, t2, u[14], m[2]); + MONT_MAC(v0, v1, v2, u[15], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[16] = t0 * n0; + MONT_MAC(t0, t1, t2, u[16], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 17 */ + MONT_MUL(s0, s1, s2, a[0], a[17]); + MONT_MUL(c0, c1, c2, a[1], a[16]); + MONT_MAC(s0, s1, s2, a[2], a[15]); + MONT_MAC(c0, c1, c2, a[3], a[14]); + MONT_MAC(s0, s1, s2, a[4], a[13]); + MONT_MAC(c0, c1, c2, a[5], a[12]); + MONT_MAC(s0, s1, s2, a[6], a[11]); + MONT_MAC(c0, c1, c2, a[7], a[10]); + MONT_MAC(s0, s1, s2, a[8], a[9]); + MONT_MAC(t0, t1, t2, u[0], m[17]); + MONT_MUL(v0, v1, v2, u[1], m[16]); + MONT_MUL(w0, w1, w2, u[2], m[15]); + MONT_MAC(t0, t1, t2, u[3], m[14]); + MONT_MAC(v0, v1, v2, u[4], m[13]); + MONT_MAC(w0, w1, w2, u[5], m[12]); + MONT_MAC(t0, t1, t2, u[6], m[11]); + MONT_MAC(v0, v1, v2, u[7], m[10]); + MONT_MAC(w0, w1, w2, u[8], m[9]); + MONT_MAC(t0, t1, t2, u[9], m[8]); + MONT_MAC(v0, v1, v2, u[10], m[7]); + MONT_MAC(w0, w1, w2, u[11], m[6]); + MONT_MAC(t0, t1, t2, u[12], m[5]); + MONT_MAC(v0, v1, v2, u[13], m[4]); + MONT_MAC(w0, w1, w2, u[14], m[3]); + MONT_MAC(t0, t1, t2, u[15], m[2]); + MONT_MAC(v0, v1, v2, u[16], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[17] = t0 * n0; + MONT_MAC(t0, t1, t2, u[17], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 18 */ + MONT_MUL(s0, s1, s2, a[0], a[18]); + MONT_MUL(c0, c1, c2, a[1], a[17]); + MONT_MAC(s0, s1, s2, a[2], a[16]); + MONT_MAC(c0, c1, c2, a[3], a[15]); + MONT_MAC(s0, s1, s2, a[4], a[14]); + MONT_MAC(c0, c1, c2, a[5], a[13]); + MONT_MAC(s0, s1, s2, a[6], a[12]); + MONT_MAC(c0, c1, c2, a[7], a[11]); + MONT_MAC(s0, s1, s2, a[8], a[10]); + MONT_MAC(t0, t1, t2, a[9], a[9]); + MONT_MUL(v0, v1, v2, u[0], m[18]); + MONT_MUL(w0, w1, w2, u[1], m[17]); + MONT_MAC(t0, t1, t2, u[2], m[16]); + MONT_MAC(v0, v1, v2, u[3], m[15]); + MONT_MAC(w0, w1, w2, u[4], m[14]); + MONT_MAC(t0, t1, t2, u[5], m[13]); + MONT_MAC(v0, v1, v2, u[6], m[12]); + MONT_MAC(w0, w1, w2, u[7], m[11]); + MONT_MAC(t0, t1, t2, u[8], m[10]); + MONT_MAC(v0, v1, v2, u[9], m[9]); + MONT_MAC(w0, w1, w2, u[10], m[8]); + MONT_MAC(t0, t1, t2, u[11], m[7]); + MONT_MAC(v0, v1, v2, u[12], m[6]); + MONT_MAC(w0, w1, w2, u[13], m[5]); + MONT_MAC(t0, t1, t2, u[14], m[4]); + MONT_MAC(v0, v1, v2, u[15], m[3]); + MONT_MAC(w0, w1, w2, u[16], m[2]); + MONT_MAC(t0, t1, t2, u[17], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[18] = t0 * n0; + MONT_MAC(t0, t1, t2, u[18], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 19 */ + MONT_MUL(s0, s1, s2, a[0], a[19]); + MONT_MUL(c0, c1, c2, a[1], a[18]); + MONT_MAC(s0, s1, s2, a[2], a[17]); + MONT_MAC(c0, c1, c2, a[3], a[16]); + MONT_MAC(s0, s1, s2, a[4], a[15]); + MONT_MAC(c0, c1, c2, a[5], a[14]); + MONT_MAC(s0, s1, s2, a[6], a[13]); + MONT_MAC(c0, c1, c2, a[7], a[12]); + MONT_MAC(s0, s1, s2, a[8], a[11]); + MONT_MAC(c0, c1, c2, a[9], a[10]); + MONT_MAC(t0, t1, t2, u[0], m[19]); + MONT_MUL(v0, v1, v2, u[1], m[18]); + MONT_MUL(w0, w1, w2, u[2], m[17]); + MONT_MAC(t0, t1, t2, u[3], m[16]); + MONT_MAC(v0, v1, v2, u[4], m[15]); + MONT_MAC(w0, w1, w2, u[5], m[14]); + MONT_MAC(t0, t1, t2, u[6], m[13]); + MONT_MAC(v0, v1, v2, u[7], m[12]); + MONT_MAC(w0, w1, w2, u[8], m[11]); + MONT_MAC(t0, t1, t2, u[9], m[10]); + MONT_MAC(v0, v1, v2, u[10], m[9]); + MONT_MAC(w0, w1, w2, u[11], m[8]); + MONT_MAC(t0, t1, t2, u[12], m[7]); + MONT_MAC(v0, v1, v2, u[13], m[6]); + MONT_MAC(w0, w1, w2, u[14], m[5]); + MONT_MAC(t0, t1, t2, u[15], m[4]); + MONT_MAC(v0, v1, v2, u[16], m[3]); + MONT_MAC(w0, w1, w2, u[17], m[2]); + MONT_MAC(t0, t1, t2, u[18], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[19] = t0 * n0; + MONT_MAC(t0, t1, t2, u[19], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 20 */ + MONT_MUL(s0, s1, s2, a[0], a[20]); + MONT_MUL(c0, c1, c2, a[1], a[19]); + MONT_MAC(s0, s1, s2, a[2], a[18]); + MONT_MAC(c0, c1, c2, a[3], a[17]); + MONT_MAC(s0, s1, s2, a[4], a[16]); + MONT_MAC(c0, c1, c2, a[5], a[15]); + MONT_MAC(s0, s1, s2, a[6], a[14]); + MONT_MAC(c0, c1, c2, a[7], a[13]); + MONT_MAC(s0, s1, s2, a[8], a[12]); + MONT_MAC(c0, c1, c2, a[9], a[11]); + MONT_MAC(t0, t1, t2, a[10], a[10]); + MONT_MUL(v0, v1, v2, u[0], m[20]); + MONT_MUL(w0, w1, w2, u[1], m[19]); + MONT_MAC(t0, t1, t2, u[2], m[18]); + MONT_MAC(v0, v1, v2, u[3], m[17]); + MONT_MAC(w0, w1, w2, u[4], m[16]); + MONT_MAC(t0, t1, t2, u[5], m[15]); + MONT_MAC(v0, v1, v2, u[6], m[14]); + MONT_MAC(w0, w1, w2, u[7], m[13]); + MONT_MAC(t0, t1, t2, u[8], m[12]); + MONT_MAC(v0, v1, v2, u[9], m[11]); + MONT_MAC(w0, w1, w2, u[10], m[10]); + MONT_MAC(t0, t1, t2, u[11], m[9]); + MONT_MAC(v0, v1, v2, u[12], m[8]); + MONT_MAC(w0, w1, w2, u[13], m[7]); + MONT_MAC(t0, t1, t2, u[14], m[6]); + MONT_MAC(v0, v1, v2, u[15], m[5]); + MONT_MAC(w0, w1, w2, u[16], m[4]); + MONT_MAC(t0, t1, t2, u[17], m[3]); + MONT_MAC(v0, v1, v2, u[18], m[2]); + MONT_MAC(w0, w1, w2, u[19], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[20] = t0 * n0; + MONT_MAC(t0, t1, t2, u[20], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 21 */ + MONT_MUL(s0, s1, s2, a[0], a[21]); + MONT_MUL(c0, c1, c2, a[1], a[20]); + MONT_MAC(s0, s1, s2, a[2], a[19]); + MONT_MAC(c0, c1, c2, a[3], a[18]); + MONT_MAC(s0, s1, s2, a[4], a[17]); + MONT_MAC(c0, c1, c2, a[5], a[16]); + MONT_MAC(s0, s1, s2, a[6], a[15]); + MONT_MAC(c0, c1, c2, a[7], a[14]); + MONT_MAC(s0, s1, s2, a[8], a[13]); + MONT_MAC(c0, c1, c2, a[9], a[12]); + MONT_MAC(s0, s1, s2, a[10], a[11]); + MONT_MAC(t0, t1, t2, u[0], m[21]); + MONT_MUL(v0, v1, v2, u[1], m[20]); + MONT_MUL(w0, w1, w2, u[2], m[19]); + MONT_MAC(t0, t1, t2, u[3], m[18]); + MONT_MAC(v0, v1, v2, u[4], m[17]); + MONT_MAC(w0, w1, w2, u[5], m[16]); + MONT_MAC(t0, t1, t2, u[6], m[15]); + MONT_MAC(v0, v1, v2, u[7], m[14]); + MONT_MAC(w0, w1, w2, u[8], m[13]); + MONT_MAC(t0, t1, t2, u[9], m[12]); + MONT_MAC(v0, v1, v2, u[10], m[11]); + MONT_MAC(w0, w1, w2, u[11], m[10]); + MONT_MAC(t0, t1, t2, u[12], m[9]); + MONT_MAC(v0, v1, v2, u[13], m[8]); + MONT_MAC(w0, w1, w2, u[14], m[7]); + MONT_MAC(t0, t1, t2, u[15], m[6]); + MONT_MAC(v0, v1, v2, u[16], m[5]); + MONT_MAC(w0, w1, w2, u[17], m[4]); + MONT_MAC(t0, t1, t2, u[18], m[3]); + MONT_MAC(v0, v1, v2, u[19], m[2]); + MONT_MAC(w0, w1, w2, u[20], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[21] = t0 * n0; + MONT_MAC(t0, t1, t2, u[21], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 22 */ + MONT_MUL(s0, s1, s2, a[0], a[22]); + MONT_MUL(c0, c1, c2, a[1], a[21]); + MONT_MAC(s0, s1, s2, a[2], a[20]); + MONT_MAC(c0, c1, c2, a[3], a[19]); + MONT_MAC(s0, s1, s2, a[4], a[18]); + MONT_MAC(c0, c1, c2, a[5], a[17]); + MONT_MAC(s0, s1, s2, a[6], a[16]); + MONT_MAC(c0, c1, c2, a[7], a[15]); + MONT_MAC(s0, s1, s2, a[8], a[14]); + MONT_MAC(c0, c1, c2, a[9], a[13]); + MONT_MAC(s0, s1, s2, a[10], a[12]); + MONT_MAC(t0, t1, t2, a[11], a[11]); + MONT_MUL(v0, v1, v2, u[0], m[22]); + MONT_MUL(w0, w1, w2, u[1], m[21]); + MONT_MAC(t0, t1, t2, u[2], m[20]); + MONT_MAC(v0, v1, v2, u[3], m[19]); + MONT_MAC(w0, w1, w2, u[4], m[18]); + MONT_MAC(t0, t1, t2, u[5], m[17]); + MONT_MAC(v0, v1, v2, u[6], m[16]); + MONT_MAC(w0, w1, w2, u[7], m[15]); + MONT_MAC(t0, t1, t2, u[8], m[14]); + MONT_MAC(v0, v1, v2, u[9], m[13]); + MONT_MAC(w0, w1, w2, u[10], m[12]); + MONT_MAC(t0, t1, t2, u[11], m[11]); + MONT_MAC(v0, v1, v2, u[12], m[10]); + MONT_MAC(w0, w1, w2, u[13], m[9]); + MONT_MAC(t0, t1, t2, u[14], m[8]); + MONT_MAC(v0, v1, v2, u[15], m[7]); + MONT_MAC(w0, w1, w2, u[16], m[6]); + MONT_MAC(t0, t1, t2, u[17], m[5]); + MONT_MAC(v0, v1, v2, u[18], m[4]); + MONT_MAC(w0, w1, w2, u[19], m[3]); + MONT_MAC(t0, t1, t2, u[20], m[2]); + MONT_MAC(v0, v1, v2, u[21], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[22] = t0 * n0; + MONT_MAC(t0, t1, t2, u[22], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 23 */ + MONT_MUL(s0, s1, s2, a[0], a[23]); + MONT_MUL(c0, c1, c2, a[1], a[22]); + MONT_MAC(s0, s1, s2, a[2], a[21]); + MONT_MAC(c0, c1, c2, a[3], a[20]); + MONT_MAC(s0, s1, s2, a[4], a[19]); + MONT_MAC(c0, c1, c2, a[5], a[18]); + MONT_MAC(s0, s1, s2, a[6], a[17]); + MONT_MAC(c0, c1, c2, a[7], a[16]); + MONT_MAC(s0, s1, s2, a[8], a[15]); + MONT_MAC(c0, c1, c2, a[9], a[14]); + MONT_MAC(s0, s1, s2, a[10], a[13]); + MONT_MAC(c0, c1, c2, a[11], a[12]); + MONT_MAC(t0, t1, t2, u[0], m[23]); + MONT_MUL(v0, v1, v2, u[1], m[22]); + MONT_MUL(w0, w1, w2, u[2], m[21]); + MONT_MAC(t0, t1, t2, u[3], m[20]); + MONT_MAC(v0, v1, v2, u[4], m[19]); + MONT_MAC(w0, w1, w2, u[5], m[18]); + MONT_MAC(t0, t1, t2, u[6], m[17]); + MONT_MAC(v0, v1, v2, u[7], m[16]); + MONT_MAC(w0, w1, w2, u[8], m[15]); + MONT_MAC(t0, t1, t2, u[9], m[14]); + MONT_MAC(v0, v1, v2, u[10], m[13]); + MONT_MAC(w0, w1, w2, u[11], m[12]); + MONT_MAC(t0, t1, t2, u[12], m[11]); + MONT_MAC(v0, v1, v2, u[13], m[10]); + MONT_MAC(w0, w1, w2, u[14], m[9]); + MONT_MAC(t0, t1, t2, u[15], m[8]); + MONT_MAC(v0, v1, v2, u[16], m[7]); + MONT_MAC(w0, w1, w2, u[17], m[6]); + MONT_MAC(t0, t1, t2, u[18], m[5]); + MONT_MAC(v0, v1, v2, u[19], m[4]); + MONT_MAC(w0, w1, w2, u[20], m[3]); + MONT_MAC(t0, t1, t2, u[21], m[2]); + MONT_MAC(v0, v1, v2, u[22], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[23] = t0 * n0; + MONT_MAC(t0, t1, t2, u[23], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 24 */ + MONT_MUL(s0, s1, s2, a[1], a[23]); + MONT_MUL(c0, c1, c2, a[2], a[22]); + MONT_MAC(s0, s1, s2, a[3], a[21]); + MONT_MAC(c0, c1, c2, a[4], a[20]); + MONT_MAC(s0, s1, s2, a[5], a[19]); + MONT_MAC(c0, c1, c2, a[6], a[18]); + MONT_MAC(s0, s1, s2, a[7], a[17]); + MONT_MAC(c0, c1, c2, a[8], a[16]); + MONT_MAC(s0, s1, s2, a[9], a[15]); + MONT_MAC(c0, c1, c2, a[10], a[14]); + MONT_MAC(s0, s1, s2, a[11], a[13]); + MONT_MAC(t0, t1, t2, a[12], a[12]); + MONT_MUL(v0, v1, v2, u[1], m[23]); + MONT_MUL(w0, w1, w2, u[2], m[22]); + MONT_MAC(t0, t1, t2, u[3], m[21]); + MONT_MAC(v0, v1, v2, u[4], m[20]); + MONT_MAC(w0, w1, w2, u[5], m[19]); + MONT_MAC(t0, t1, t2, u[6], m[18]); + MONT_MAC(v0, v1, v2, u[7], m[17]); + MONT_MAC(w0, w1, w2, u[8], m[16]); + MONT_MAC(t0, t1, t2, u[9], m[15]); + MONT_MAC(v0, v1, v2, u[10], m[14]); + MONT_MAC(w0, w1, w2, u[11], m[13]); + MONT_MAC(t0, t1, t2, u[12], m[12]); + MONT_MAC(v0, v1, v2, u[13], m[11]); + MONT_MAC(w0, w1, w2, u[14], m[10]); + MONT_MAC(t0, t1, t2, u[15], m[9]); + MONT_MAC(v0, v1, v2, u[16], m[8]); + MONT_MAC(w0, w1, w2, u[17], m[7]); + MONT_MAC(t0, t1, t2, u[18], m[6]); + MONT_MAC(v0, v1, v2, u[19], m[5]); + MONT_MAC(w0, w1, w2, u[20], m[4]); + MONT_MAC(t0, t1, t2, u[21], m[3]); + MONT_MAC(v0, v1, v2, u[22], m[2]); + MONT_MAC(w0, w1, w2, u[23], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[0] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 25 */ + MONT_MUL(s0, s1, s2, a[2], a[23]); + MONT_MUL(c0, c1, c2, a[3], a[22]); + MONT_MAC(s0, s1, s2, a[4], a[21]); + MONT_MAC(c0, c1, c2, a[5], a[20]); + MONT_MAC(s0, s1, s2, a[6], a[19]); + MONT_MAC(c0, c1, c2, a[7], a[18]); + MONT_MAC(s0, s1, s2, a[8], a[17]); + MONT_MAC(c0, c1, c2, a[9], a[16]); + MONT_MAC(s0, s1, s2, a[10], a[15]); + MONT_MAC(c0, c1, c2, a[11], a[14]); + MONT_MAC(s0, s1, s2, a[12], a[13]); + MONT_MAC(t0, t1, t2, u[2], m[23]); + MONT_MUL(v0, v1, v2, u[3], m[22]); + MONT_MUL(w0, w1, w2, u[4], m[21]); + MONT_MAC(t0, t1, t2, u[5], m[20]); + MONT_MAC(v0, v1, v2, u[6], m[19]); + MONT_MAC(w0, w1, w2, u[7], m[18]); + MONT_MAC(t0, t1, t2, u[8], m[17]); + MONT_MAC(v0, v1, v2, u[9], m[16]); + MONT_MAC(w0, w1, w2, u[10], m[15]); + MONT_MAC(t0, t1, t2, u[11], m[14]); + MONT_MAC(v0, v1, v2, u[12], m[13]); + MONT_MAC(w0, w1, w2, u[13], m[12]); + MONT_MAC(t0, t1, t2, u[14], m[11]); + MONT_MAC(v0, v1, v2, u[15], m[10]); + MONT_MAC(w0, w1, w2, u[16], m[9]); + MONT_MAC(t0, t1, t2, u[17], m[8]); + MONT_MAC(v0, v1, v2, u[18], m[7]); + MONT_MAC(w0, w1, w2, u[19], m[6]); + MONT_MAC(t0, t1, t2, u[20], m[5]); + MONT_MAC(v0, v1, v2, u[21], m[4]); + MONT_MAC(w0, w1, w2, u[22], m[3]); + MONT_MAC(t0, t1, t2, u[23], m[2]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[1] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 26 */ + MONT_MUL(s0, s1, s2, a[3], a[23]); + MONT_MUL(c0, c1, c2, a[4], a[22]); + MONT_MAC(s0, s1, s2, a[5], a[21]); + MONT_MAC(c0, c1, c2, a[6], a[20]); + MONT_MAC(s0, s1, s2, a[7], a[19]); + MONT_MAC(c0, c1, c2, a[8], a[18]); + MONT_MAC(s0, s1, s2, a[9], a[17]); + MONT_MAC(c0, c1, c2, a[10], a[16]); + MONT_MAC(s0, s1, s2, a[11], a[15]); + MONT_MAC(c0, c1, c2, a[12], a[14]); + MONT_MAC(t0, t1, t2, a[13], a[13]); + MONT_MUL(v0, v1, v2, u[3], m[23]); + MONT_MUL(w0, w1, w2, u[4], m[22]); + MONT_MAC(t0, t1, t2, u[5], m[21]); + MONT_MAC(v0, v1, v2, u[6], m[20]); + MONT_MAC(w0, w1, w2, u[7], m[19]); + MONT_MAC(t0, t1, t2, u[8], m[18]); + MONT_MAC(v0, v1, v2, u[9], m[17]); + MONT_MAC(w0, w1, w2, u[10], m[16]); + MONT_MAC(t0, t1, t2, u[11], m[15]); + MONT_MAC(v0, v1, v2, u[12], m[14]); + MONT_MAC(w0, w1, w2, u[13], m[13]); + MONT_MAC(t0, t1, t2, u[14], m[12]); + MONT_MAC(v0, v1, v2, u[15], m[11]); + MONT_MAC(w0, w1, w2, u[16], m[10]); + MONT_MAC(t0, t1, t2, u[17], m[9]); + MONT_MAC(v0, v1, v2, u[18], m[8]); + MONT_MAC(w0, w1, w2, u[19], m[7]); + MONT_MAC(t0, t1, t2, u[20], m[6]); + MONT_MAC(v0, v1, v2, u[21], m[5]); + MONT_MAC(w0, w1, w2, u[22], m[4]); + MONT_MAC(t0, t1, t2, u[23], m[3]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[2] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 27 */ + MONT_MUL(s0, s1, s2, a[4], a[23]); + MONT_MUL(c0, c1, c2, a[5], a[22]); + MONT_MAC(s0, s1, s2, a[6], a[21]); + MONT_MAC(c0, c1, c2, a[7], a[20]); + MONT_MAC(s0, s1, s2, a[8], a[19]); + MONT_MAC(c0, c1, c2, a[9], a[18]); + MONT_MAC(s0, s1, s2, a[10], a[17]); + MONT_MAC(c0, c1, c2, a[11], a[16]); + MONT_MAC(s0, s1, s2, a[12], a[15]); + MONT_MAC(c0, c1, c2, a[13], a[14]); + MONT_MAC(t0, t1, t2, u[4], m[23]); + MONT_MUL(v0, v1, v2, u[5], m[22]); + MONT_MUL(w0, w1, w2, u[6], m[21]); + MONT_MAC(t0, t1, t2, u[7], m[20]); + MONT_MAC(v0, v1, v2, u[8], m[19]); + MONT_MAC(w0, w1, w2, u[9], m[18]); + MONT_MAC(t0, t1, t2, u[10], m[17]); + MONT_MAC(v0, v1, v2, u[11], m[16]); + MONT_MAC(w0, w1, w2, u[12], m[15]); + MONT_MAC(t0, t1, t2, u[13], m[14]); + MONT_MAC(v0, v1, v2, u[14], m[13]); + MONT_MAC(w0, w1, w2, u[15], m[12]); + MONT_MAC(t0, t1, t2, u[16], m[11]); + MONT_MAC(v0, v1, v2, u[17], m[10]); + MONT_MAC(w0, w1, w2, u[18], m[9]); + MONT_MAC(t0, t1, t2, u[19], m[8]); + MONT_MAC(v0, v1, v2, u[20], m[7]); + MONT_MAC(w0, w1, w2, u[21], m[6]); + MONT_MAC(t0, t1, t2, u[22], m[5]); + MONT_MAC(v0, v1, v2, u[23], m[4]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[3] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 28 */ + MONT_MUL(s0, s1, s2, a[5], a[23]); + MONT_MUL(c0, c1, c2, a[6], a[22]); + MONT_MAC(s0, s1, s2, a[7], a[21]); + MONT_MAC(c0, c1, c2, a[8], a[20]); + MONT_MAC(s0, s1, s2, a[9], a[19]); + MONT_MAC(c0, c1, c2, a[10], a[18]); + MONT_MAC(s0, s1, s2, a[11], a[17]); + MONT_MAC(c0, c1, c2, a[12], a[16]); + MONT_MAC(s0, s1, s2, a[13], a[15]); + MONT_MAC(t0, t1, t2, a[14], a[14]); + MONT_MUL(v0, v1, v2, u[5], m[23]); + MONT_MUL(w0, w1, w2, u[6], m[22]); + MONT_MAC(t0, t1, t2, u[7], m[21]); + MONT_MAC(v0, v1, v2, u[8], m[20]); + MONT_MAC(w0, w1, w2, u[9], m[19]); + MONT_MAC(t0, t1, t2, u[10], m[18]); + MONT_MAC(v0, v1, v2, u[11], m[17]); + MONT_MAC(w0, w1, w2, u[12], m[16]); + MONT_MAC(t0, t1, t2, u[13], m[15]); + MONT_MAC(v0, v1, v2, u[14], m[14]); + MONT_MAC(w0, w1, w2, u[15], m[13]); + MONT_MAC(t0, t1, t2, u[16], m[12]); + MONT_MAC(v0, v1, v2, u[17], m[11]); + MONT_MAC(w0, w1, w2, u[18], m[10]); + MONT_MAC(t0, t1, t2, u[19], m[9]); + MONT_MAC(v0, v1, v2, u[20], m[8]); + MONT_MAC(w0, w1, w2, u[21], m[7]); + MONT_MAC(t0, t1, t2, u[22], m[6]); + MONT_MAC(v0, v1, v2, u[23], m[5]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[4] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 29 */ + MONT_MUL(s0, s1, s2, a[6], a[23]); + MONT_MUL(c0, c1, c2, a[7], a[22]); + MONT_MAC(s0, s1, s2, a[8], a[21]); + MONT_MAC(c0, c1, c2, a[9], a[20]); + MONT_MAC(s0, s1, s2, a[10], a[19]); + MONT_MAC(c0, c1, c2, a[11], a[18]); + MONT_MAC(s0, s1, s2, a[12], a[17]); + MONT_MAC(c0, c1, c2, a[13], a[16]); + MONT_MAC(s0, s1, s2, a[14], a[15]); + MONT_MAC(t0, t1, t2, u[6], m[23]); + MONT_MUL(v0, v1, v2, u[7], m[22]); + MONT_MUL(w0, w1, w2, u[8], m[21]); + MONT_MAC(t0, t1, t2, u[9], m[20]); + MONT_MAC(v0, v1, v2, u[10], m[19]); + MONT_MAC(w0, w1, w2, u[11], m[18]); + MONT_MAC(t0, t1, t2, u[12], m[17]); + MONT_MAC(v0, v1, v2, u[13], m[16]); + MONT_MAC(w0, w1, w2, u[14], m[15]); + MONT_MAC(t0, t1, t2, u[15], m[14]); + MONT_MAC(v0, v1, v2, u[16], m[13]); + MONT_MAC(w0, w1, w2, u[17], m[12]); + MONT_MAC(t0, t1, t2, u[18], m[11]); + MONT_MAC(v0, v1, v2, u[19], m[10]); + MONT_MAC(w0, w1, w2, u[20], m[9]); + MONT_MAC(t0, t1, t2, u[21], m[8]); + MONT_MAC(v0, v1, v2, u[22], m[7]); + MONT_MAC(w0, w1, w2, u[23], m[6]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[5] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 30 */ + MONT_MUL(s0, s1, s2, a[7], a[23]); + MONT_MUL(c0, c1, c2, a[8], a[22]); + MONT_MAC(s0, s1, s2, a[9], a[21]); + MONT_MAC(c0, c1, c2, a[10], a[20]); + MONT_MAC(s0, s1, s2, a[11], a[19]); + MONT_MAC(c0, c1, c2, a[12], a[18]); + MONT_MAC(s0, s1, s2, a[13], a[17]); + MONT_MAC(c0, c1, c2, a[14], a[16]); + MONT_MAC(t0, t1, t2, a[15], a[15]); + MONT_MUL(v0, v1, v2, u[7], m[23]); + MONT_MUL(w0, w1, w2, u[8], m[22]); + MONT_MAC(t0, t1, t2, u[9], m[21]); + MONT_MAC(v0, v1, v2, u[10], m[20]); + MONT_MAC(w0, w1, w2, u[11], m[19]); + MONT_MAC(t0, t1, t2, u[12], m[18]); + MONT_MAC(v0, v1, v2, u[13], m[17]); + MONT_MAC(w0, w1, w2, u[14], m[16]); + MONT_MAC(t0, t1, t2, u[15], m[15]); + MONT_MAC(v0, v1, v2, u[16], m[14]); + MONT_MAC(w0, w1, w2, u[17], m[13]); + MONT_MAC(t0, t1, t2, u[18], m[12]); + MONT_MAC(v0, v1, v2, u[19], m[11]); + MONT_MAC(w0, w1, w2, u[20], m[10]); + MONT_MAC(t0, t1, t2, u[21], m[9]); + MONT_MAC(v0, v1, v2, u[22], m[8]); + MONT_MAC(w0, w1, w2, u[23], m[7]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[6] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 31 */ + MONT_MUL(s0, s1, s2, a[8], a[23]); + MONT_MUL(c0, c1, c2, a[9], a[22]); + MONT_MAC(s0, s1, s2, a[10], a[21]); + MONT_MAC(c0, c1, c2, a[11], a[20]); + MONT_MAC(s0, s1, s2, a[12], a[19]); + MONT_MAC(c0, c1, c2, a[13], a[18]); + MONT_MAC(s0, s1, s2, a[14], a[17]); + MONT_MAC(c0, c1, c2, a[15], a[16]); + MONT_MAC(t0, t1, t2, u[8], m[23]); + MONT_MUL(v0, v1, v2, u[9], m[22]); + MONT_MUL(w0, w1, w2, u[10], m[21]); + MONT_MAC(t0, t1, t2, u[11], m[20]); + MONT_MAC(v0, v1, v2, u[12], m[19]); + MONT_MAC(w0, w1, w2, u[13], m[18]); + MONT_MAC(t0, t1, t2, u[14], m[17]); + MONT_MAC(v0, v1, v2, u[15], m[16]); + MONT_MAC(w0, w1, w2, u[16], m[15]); + MONT_MAC(t0, t1, t2, u[17], m[14]); + MONT_MAC(v0, v1, v2, u[18], m[13]); + MONT_MAC(w0, w1, w2, u[19], m[12]); + MONT_MAC(t0, t1, t2, u[20], m[11]); + MONT_MAC(v0, v1, v2, u[21], m[10]); + MONT_MAC(w0, w1, w2, u[22], m[9]); + MONT_MAC(t0, t1, t2, u[23], m[8]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[7] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 32 */ + MONT_MUL(s0, s1, s2, a[9], a[23]); + MONT_MUL(c0, c1, c2, a[10], a[22]); + MONT_MAC(s0, s1, s2, a[11], a[21]); + MONT_MAC(c0, c1, c2, a[12], a[20]); + MONT_MAC(s0, s1, s2, a[13], a[19]); + MONT_MAC(c0, c1, c2, a[14], a[18]); + MONT_MAC(s0, s1, s2, a[15], a[17]); + MONT_MAC(t0, t1, t2, a[16], a[16]); + MONT_MUL(v0, v1, v2, u[9], m[23]); + MONT_MUL(w0, w1, w2, u[10], m[22]); + MONT_MAC(t0, t1, t2, u[11], m[21]); + MONT_MAC(v0, v1, v2, u[12], m[20]); + MONT_MAC(w0, w1, w2, u[13], m[19]); + MONT_MAC(t0, t1, t2, u[14], m[18]); + MONT_MAC(v0, v1, v2, u[15], m[17]); + MONT_MAC(w0, w1, w2, u[16], m[16]); + MONT_MAC(t0, t1, t2, u[17], m[15]); + MONT_MAC(v0, v1, v2, u[18], m[14]); + MONT_MAC(w0, w1, w2, u[19], m[13]); + MONT_MAC(t0, t1, t2, u[20], m[12]); + MONT_MAC(v0, v1, v2, u[21], m[11]); + MONT_MAC(w0, w1, w2, u[22], m[10]); + MONT_MAC(t0, t1, t2, u[23], m[9]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[8] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 33 */ + MONT_MUL(s0, s1, s2, a[10], a[23]); + MONT_MUL(c0, c1, c2, a[11], a[22]); + MONT_MAC(s0, s1, s2, a[12], a[21]); + MONT_MAC(c0, c1, c2, a[13], a[20]); + MONT_MAC(s0, s1, s2, a[14], a[19]); + MONT_MAC(c0, c1, c2, a[15], a[18]); + MONT_MAC(s0, s1, s2, a[16], a[17]); + MONT_MAC(t0, t1, t2, u[10], m[23]); + MONT_MUL(v0, v1, v2, u[11], m[22]); + MONT_MUL(w0, w1, w2, u[12], m[21]); + MONT_MAC(t0, t1, t2, u[13], m[20]); + MONT_MAC(v0, v1, v2, u[14], m[19]); + MONT_MAC(w0, w1, w2, u[15], m[18]); + MONT_MAC(t0, t1, t2, u[16], m[17]); + MONT_MAC(v0, v1, v2, u[17], m[16]); + MONT_MAC(w0, w1, w2, u[18], m[15]); + MONT_MAC(t0, t1, t2, u[19], m[14]); + MONT_MAC(v0, v1, v2, u[20], m[13]); + MONT_MAC(w0, w1, w2, u[21], m[12]); + MONT_MAC(t0, t1, t2, u[22], m[11]); + MONT_MAC(v0, v1, v2, u[23], m[10]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[9] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 34 */ + MONT_MUL(s0, s1, s2, a[11], a[23]); + MONT_MUL(c0, c1, c2, a[12], a[22]); + MONT_MAC(s0, s1, s2, a[13], a[21]); + MONT_MAC(c0, c1, c2, a[14], a[20]); + MONT_MAC(s0, s1, s2, a[15], a[19]); + MONT_MAC(c0, c1, c2, a[16], a[18]); + MONT_MAC(t0, t1, t2, a[17], a[17]); + MONT_MUL(v0, v1, v2, u[11], m[23]); + MONT_MUL(w0, w1, w2, u[12], m[22]); + MONT_MAC(t0, t1, t2, u[13], m[21]); + MONT_MAC(v0, v1, v2, u[14], m[20]); + MONT_MAC(w0, w1, w2, u[15], m[19]); + MONT_MAC(t0, t1, t2, u[16], m[18]); + MONT_MAC(v0, v1, v2, u[17], m[17]); + MONT_MAC(w0, w1, w2, u[18], m[16]); + MONT_MAC(t0, t1, t2, u[19], m[15]); + MONT_MAC(v0, v1, v2, u[20], m[14]); + MONT_MAC(w0, w1, w2, u[21], m[13]); + MONT_MAC(t0, t1, t2, u[22], m[12]); + MONT_MAC(v0, v1, v2, u[23], m[11]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[10] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 35 */ + MONT_MUL(s0, s1, s2, a[12], a[23]); + MONT_MUL(c0, c1, c2, a[13], a[22]); + MONT_MAC(s0, s1, s2, a[14], a[21]); + MONT_MAC(c0, c1, c2, a[15], a[20]); + MONT_MAC(s0, s1, s2, a[16], a[19]); + MONT_MAC(c0, c1, c2, a[17], a[18]); + MONT_MAC(t0, t1, t2, u[12], m[23]); + MONT_MUL(v0, v1, v2, u[13], m[22]); + MONT_MUL(w0, w1, w2, u[14], m[21]); + MONT_MAC(t0, t1, t2, u[15], m[20]); + MONT_MAC(v0, v1, v2, u[16], m[19]); + MONT_MAC(w0, w1, w2, u[17], m[18]); + MONT_MAC(t0, t1, t2, u[18], m[17]); + MONT_MAC(v0, v1, v2, u[19], m[16]); + MONT_MAC(w0, w1, w2, u[20], m[15]); + MONT_MAC(t0, t1, t2, u[21], m[14]); + MONT_MAC(v0, v1, v2, u[22], m[13]); + MONT_MAC(w0, w1, w2, u[23], m[12]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[11] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 36 */ + MONT_MUL(s0, s1, s2, a[13], a[23]); + MONT_MUL(c0, c1, c2, a[14], a[22]); + MONT_MAC(s0, s1, s2, a[15], a[21]); + MONT_MAC(c0, c1, c2, a[16], a[20]); + MONT_MAC(s0, s1, s2, a[17], a[19]); + MONT_MAC(t0, t1, t2, a[18], a[18]); + MONT_MUL(v0, v1, v2, u[13], m[23]); + MONT_MUL(w0, w1, w2, u[14], m[22]); + MONT_MAC(t0, t1, t2, u[15], m[21]); + MONT_MAC(v0, v1, v2, u[16], m[20]); + MONT_MAC(w0, w1, w2, u[17], m[19]); + MONT_MAC(t0, t1, t2, u[18], m[18]); + MONT_MAC(v0, v1, v2, u[19], m[17]); + MONT_MAC(w0, w1, w2, u[20], m[16]); + MONT_MAC(t0, t1, t2, u[21], m[15]); + MONT_MAC(v0, v1, v2, u[22], m[14]); + MONT_MAC(w0, w1, w2, u[23], m[13]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[12] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 37 */ + MONT_MUL(s0, s1, s2, a[14], a[23]); + MONT_MUL(c0, c1, c2, a[15], a[22]); + MONT_MAC(s0, s1, s2, a[16], a[21]); + MONT_MAC(c0, c1, c2, a[17], a[20]); + MONT_MAC(s0, s1, s2, a[18], a[19]); + MONT_MAC(t0, t1, t2, u[14], m[23]); + MONT_MUL(v0, v1, v2, u[15], m[22]); + MONT_MUL(w0, w1, w2, u[16], m[21]); + MONT_MAC(t0, t1, t2, u[17], m[20]); + MONT_MAC(v0, v1, v2, u[18], m[19]); + MONT_MAC(w0, w1, w2, u[19], m[18]); + MONT_MAC(t0, t1, t2, u[20], m[17]); + MONT_MAC(v0, v1, v2, u[21], m[16]); + MONT_MAC(w0, w1, w2, u[22], m[15]); + MONT_MAC(t0, t1, t2, u[23], m[14]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[13] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 38 */ + MONT_MUL(s0, s1, s2, a[15], a[23]); + MONT_MUL(c0, c1, c2, a[16], a[22]); + MONT_MAC(s0, s1, s2, a[17], a[21]); + MONT_MAC(c0, c1, c2, a[18], a[20]); + MONT_MAC(t0, t1, t2, a[19], a[19]); + MONT_MUL(v0, v1, v2, u[15], m[23]); + MONT_MUL(w0, w1, w2, u[16], m[22]); + MONT_MAC(t0, t1, t2, u[17], m[21]); + MONT_MAC(v0, v1, v2, u[18], m[20]); + MONT_MAC(w0, w1, w2, u[19], m[19]); + MONT_MAC(t0, t1, t2, u[20], m[18]); + MONT_MAC(v0, v1, v2, u[21], m[17]); + MONT_MAC(w0, w1, w2, u[22], m[16]); + MONT_MAC(t0, t1, t2, u[23], m[15]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[14] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 39 */ + MONT_MUL(s0, s1, s2, a[16], a[23]); + MONT_MUL(c0, c1, c2, a[17], a[22]); + MONT_MAC(s0, s1, s2, a[18], a[21]); + MONT_MAC(c0, c1, c2, a[19], a[20]); + MONT_MAC(t0, t1, t2, u[16], m[23]); + MONT_MUL(v0, v1, v2, u[17], m[22]); + MONT_MUL(w0, w1, w2, u[18], m[21]); + MONT_MAC(t0, t1, t2, u[19], m[20]); + MONT_MAC(v0, v1, v2, u[20], m[19]); + MONT_MAC(w0, w1, w2, u[21], m[18]); + MONT_MAC(t0, t1, t2, u[22], m[17]); + MONT_MAC(v0, v1, v2, u[23], m[16]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[15] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 40 */ + MONT_MUL(s0, s1, s2, a[17], a[23]); + MONT_MUL(c0, c1, c2, a[18], a[22]); + MONT_MAC(s0, s1, s2, a[19], a[21]); + MONT_MAC(t0, t1, t2, a[20], a[20]); + MONT_MUL(v0, v1, v2, u[17], m[23]); + MONT_MUL(w0, w1, w2, u[18], m[22]); + MONT_MAC(t0, t1, t2, u[19], m[21]); + MONT_MAC(v0, v1, v2, u[20], m[20]); + MONT_MAC(w0, w1, w2, u[21], m[19]); + MONT_MAC(t0, t1, t2, u[22], m[18]); + MONT_MAC(v0, v1, v2, u[23], m[17]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[16] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 41 */ + MONT_MUL(s0, s1, s2, a[18], a[23]); + MONT_MUL(c0, c1, c2, a[19], a[22]); + MONT_MAC(s0, s1, s2, a[20], a[21]); + MONT_MAC(t0, t1, t2, u[18], m[23]); + MONT_MUL(v0, v1, v2, u[19], m[22]); + MONT_MUL(w0, w1, w2, u[20], m[21]); + MONT_MAC(t0, t1, t2, u[21], m[20]); + MONT_MAC(v0, v1, v2, u[22], m[19]); + MONT_MAC(w0, w1, w2, u[23], m[18]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[17] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 42 */ + MONT_MUL(s0, s1, s2, a[19], a[23]); + MONT_MUL(c0, c1, c2, a[20], a[22]); + MONT_MAC(t0, t1, t2, a[21], a[21]); + MONT_MUL(v0, v1, v2, u[19], m[23]); + MONT_MUL(w0, w1, w2, u[20], m[22]); + MONT_MAC(t0, t1, t2, u[21], m[21]); + MONT_MAC(v0, v1, v2, u[22], m[20]); + MONT_MAC(w0, w1, w2, u[23], m[19]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[18] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 43 */ + MONT_MUL(s0, s1, s2, a[20], a[23]); + MONT_MUL(c0, c1, c2, a[21], a[22]); + MONT_MAC(t0, t1, t2, u[20], m[23]); + MONT_MUL(v0, v1, v2, u[21], m[22]); + MONT_MUL(w0, w1, w2, u[22], m[21]); + MONT_MAC(t0, t1, t2, u[23], m[20]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[19] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 44 */ + MONT_MUL(s0, s1, s2, a[21], a[23]); + MONT_MAC(t0, t1, t2, a[22], a[22]); + MONT_MUL(v0, v1, v2, u[21], m[23]); + MONT_MUL(w0, w1, w2, u[22], m[22]); + MONT_MAC(t0, t1, t2, u[23], m[21]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[20] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 45 */ + MONT_MUL(s0, s1, s2, a[22], a[23]); + MONT_MAC(t0, t1, t2, u[22], m[23]); + MONT_MUL(v0, v1, v2, u[23], m[22]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[21] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 46 */ + MONT_MAC(t0, t1, t2, a[23], a[23]); + MONT_MUL(v0, v1, v2, u[23], m[23]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + z[22] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + z[23] = t0; + + /* under 2m, with t1 the bit above n limbs: at most one subtraction */ + borrow = sub_n(r, z, m, 24); + take = t1 | (borrow ^ 1); + select_n(r, r, z, take & 1, 24); +} + +/* r = a * b * R^-1 mod m, for a and b under m; t is 2n limbs of scratch */ +static void crypton_mont_mul32(limb_t *r, const limb_t *a, const limb_t *b, + const limb_t *m, limb_t n0, limb_t *t) +{ + limb_t *u = t, *z = t + 32; + limb_t t0 = 0, t1 = 0, t2 = 0, v0, v1, v2, w0, w1, w2, borrow, take; + + /* column 0 */ + MONT_MAC(t0, t1, t2, a[0], b[0]); + u[0] = t0 * n0; + MONT_MAC(t0, t1, t2, u[0], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 1 */ + MONT_MAC(t0, t1, t2, a[0], b[1]); + MONT_MUL(v0, v1, v2, a[1], b[0]); + MONT_MUL(w0, w1, w2, u[0], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[1] = t0 * n0; + MONT_MAC(t0, t1, t2, u[1], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 2 */ + MONT_MAC(t0, t1, t2, a[0], b[2]); + MONT_MUL(v0, v1, v2, a[1], b[1]); + MONT_MUL(w0, w1, w2, a[2], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[2]); + MONT_MAC(v0, v1, v2, u[1], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[2] = t0 * n0; + MONT_MAC(t0, t1, t2, u[2], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 3 */ + MONT_MAC(t0, t1, t2, a[0], b[3]); + MONT_MUL(v0, v1, v2, a[1], b[2]); + MONT_MUL(w0, w1, w2, a[2], b[1]); + MONT_MAC(t0, t1, t2, a[3], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[3]); + MONT_MAC(w0, w1, w2, u[1], m[2]); + MONT_MAC(t0, t1, t2, u[2], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[3] = t0 * n0; + MONT_MAC(t0, t1, t2, u[3], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 4 */ + MONT_MAC(t0, t1, t2, a[0], b[4]); + MONT_MUL(v0, v1, v2, a[1], b[3]); + MONT_MUL(w0, w1, w2, a[2], b[2]); + MONT_MAC(t0, t1, t2, a[3], b[1]); + MONT_MAC(v0, v1, v2, a[4], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[4]); + MONT_MAC(t0, t1, t2, u[1], m[3]); + MONT_MAC(v0, v1, v2, u[2], m[2]); + MONT_MAC(w0, w1, w2, u[3], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[4] = t0 * n0; + MONT_MAC(t0, t1, t2, u[4], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 5 */ + MONT_MAC(t0, t1, t2, a[0], b[5]); + MONT_MUL(v0, v1, v2, a[1], b[4]); + MONT_MUL(w0, w1, w2, a[2], b[3]); + MONT_MAC(t0, t1, t2, a[3], b[2]); + MONT_MAC(v0, v1, v2, a[4], b[1]); + MONT_MAC(w0, w1, w2, a[5], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[5]); + MONT_MAC(v0, v1, v2, u[1], m[4]); + MONT_MAC(w0, w1, w2, u[2], m[3]); + MONT_MAC(t0, t1, t2, u[3], m[2]); + MONT_MAC(v0, v1, v2, u[4], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[5] = t0 * n0; + MONT_MAC(t0, t1, t2, u[5], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 6 */ + MONT_MAC(t0, t1, t2, a[0], b[6]); + MONT_MUL(v0, v1, v2, a[1], b[5]); + MONT_MUL(w0, w1, w2, a[2], b[4]); + MONT_MAC(t0, t1, t2, a[3], b[3]); + MONT_MAC(v0, v1, v2, a[4], b[2]); + MONT_MAC(w0, w1, w2, a[5], b[1]); + MONT_MAC(t0, t1, t2, a[6], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[6]); + MONT_MAC(w0, w1, w2, u[1], m[5]); + MONT_MAC(t0, t1, t2, u[2], m[4]); + MONT_MAC(v0, v1, v2, u[3], m[3]); + MONT_MAC(w0, w1, w2, u[4], m[2]); + MONT_MAC(t0, t1, t2, u[5], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[6] = t0 * n0; + MONT_MAC(t0, t1, t2, u[6], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 7 */ + MONT_MAC(t0, t1, t2, a[0], b[7]); + MONT_MUL(v0, v1, v2, a[1], b[6]); + MONT_MUL(w0, w1, w2, a[2], b[5]); + MONT_MAC(t0, t1, t2, a[3], b[4]); + MONT_MAC(v0, v1, v2, a[4], b[3]); + MONT_MAC(w0, w1, w2, a[5], b[2]); + MONT_MAC(t0, t1, t2, a[6], b[1]); + MONT_MAC(v0, v1, v2, a[7], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[7]); + MONT_MAC(t0, t1, t2, u[1], m[6]); + MONT_MAC(v0, v1, v2, u[2], m[5]); + MONT_MAC(w0, w1, w2, u[3], m[4]); + MONT_MAC(t0, t1, t2, u[4], m[3]); + MONT_MAC(v0, v1, v2, u[5], m[2]); + MONT_MAC(w0, w1, w2, u[6], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[7] = t0 * n0; + MONT_MAC(t0, t1, t2, u[7], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 8 */ + MONT_MAC(t0, t1, t2, a[0], b[8]); + MONT_MUL(v0, v1, v2, a[1], b[7]); + MONT_MUL(w0, w1, w2, a[2], b[6]); + MONT_MAC(t0, t1, t2, a[3], b[5]); + MONT_MAC(v0, v1, v2, a[4], b[4]); + MONT_MAC(w0, w1, w2, a[5], b[3]); + MONT_MAC(t0, t1, t2, a[6], b[2]); + MONT_MAC(v0, v1, v2, a[7], b[1]); + MONT_MAC(w0, w1, w2, a[8], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[8]); + MONT_MAC(v0, v1, v2, u[1], m[7]); + MONT_MAC(w0, w1, w2, u[2], m[6]); + MONT_MAC(t0, t1, t2, u[3], m[5]); + MONT_MAC(v0, v1, v2, u[4], m[4]); + MONT_MAC(w0, w1, w2, u[5], m[3]); + MONT_MAC(t0, t1, t2, u[6], m[2]); + MONT_MAC(v0, v1, v2, u[7], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[8] = t0 * n0; + MONT_MAC(t0, t1, t2, u[8], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 9 */ + MONT_MAC(t0, t1, t2, a[0], b[9]); + MONT_MUL(v0, v1, v2, a[1], b[8]); + MONT_MUL(w0, w1, w2, a[2], b[7]); + MONT_MAC(t0, t1, t2, a[3], b[6]); + MONT_MAC(v0, v1, v2, a[4], b[5]); + MONT_MAC(w0, w1, w2, a[5], b[4]); + MONT_MAC(t0, t1, t2, a[6], b[3]); + MONT_MAC(v0, v1, v2, a[7], b[2]); + MONT_MAC(w0, w1, w2, a[8], b[1]); + MONT_MAC(t0, t1, t2, a[9], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[9]); + MONT_MAC(w0, w1, w2, u[1], m[8]); + MONT_MAC(t0, t1, t2, u[2], m[7]); + MONT_MAC(v0, v1, v2, u[3], m[6]); + MONT_MAC(w0, w1, w2, u[4], m[5]); + MONT_MAC(t0, t1, t2, u[5], m[4]); + MONT_MAC(v0, v1, v2, u[6], m[3]); + MONT_MAC(w0, w1, w2, u[7], m[2]); + MONT_MAC(t0, t1, t2, u[8], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[9] = t0 * n0; + MONT_MAC(t0, t1, t2, u[9], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 10 */ + MONT_MAC(t0, t1, t2, a[0], b[10]); + MONT_MUL(v0, v1, v2, a[1], b[9]); + MONT_MUL(w0, w1, w2, a[2], b[8]); + MONT_MAC(t0, t1, t2, a[3], b[7]); + MONT_MAC(v0, v1, v2, a[4], b[6]); + MONT_MAC(w0, w1, w2, a[5], b[5]); + MONT_MAC(t0, t1, t2, a[6], b[4]); + MONT_MAC(v0, v1, v2, a[7], b[3]); + MONT_MAC(w0, w1, w2, a[8], b[2]); + MONT_MAC(t0, t1, t2, a[9], b[1]); + MONT_MAC(v0, v1, v2, a[10], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[10]); + MONT_MAC(t0, t1, t2, u[1], m[9]); + MONT_MAC(v0, v1, v2, u[2], m[8]); + MONT_MAC(w0, w1, w2, u[3], m[7]); + MONT_MAC(t0, t1, t2, u[4], m[6]); + MONT_MAC(v0, v1, v2, u[5], m[5]); + MONT_MAC(w0, w1, w2, u[6], m[4]); + MONT_MAC(t0, t1, t2, u[7], m[3]); + MONT_MAC(v0, v1, v2, u[8], m[2]); + MONT_MAC(w0, w1, w2, u[9], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[10] = t0 * n0; + MONT_MAC(t0, t1, t2, u[10], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 11 */ + MONT_MAC(t0, t1, t2, a[0], b[11]); + MONT_MUL(v0, v1, v2, a[1], b[10]); + MONT_MUL(w0, w1, w2, a[2], b[9]); + MONT_MAC(t0, t1, t2, a[3], b[8]); + MONT_MAC(v0, v1, v2, a[4], b[7]); + MONT_MAC(w0, w1, w2, a[5], b[6]); + MONT_MAC(t0, t1, t2, a[6], b[5]); + MONT_MAC(v0, v1, v2, a[7], b[4]); + MONT_MAC(w0, w1, w2, a[8], b[3]); + MONT_MAC(t0, t1, t2, a[9], b[2]); + MONT_MAC(v0, v1, v2, a[10], b[1]); + MONT_MAC(w0, w1, w2, a[11], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[11]); + MONT_MAC(v0, v1, v2, u[1], m[10]); + MONT_MAC(w0, w1, w2, u[2], m[9]); + MONT_MAC(t0, t1, t2, u[3], m[8]); + MONT_MAC(v0, v1, v2, u[4], m[7]); + MONT_MAC(w0, w1, w2, u[5], m[6]); + MONT_MAC(t0, t1, t2, u[6], m[5]); + MONT_MAC(v0, v1, v2, u[7], m[4]); + MONT_MAC(w0, w1, w2, u[8], m[3]); + MONT_MAC(t0, t1, t2, u[9], m[2]); + MONT_MAC(v0, v1, v2, u[10], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[11] = t0 * n0; + MONT_MAC(t0, t1, t2, u[11], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 12 */ + MONT_MAC(t0, t1, t2, a[0], b[12]); + MONT_MUL(v0, v1, v2, a[1], b[11]); + MONT_MUL(w0, w1, w2, a[2], b[10]); + MONT_MAC(t0, t1, t2, a[3], b[9]); + MONT_MAC(v0, v1, v2, a[4], b[8]); + MONT_MAC(w0, w1, w2, a[5], b[7]); + MONT_MAC(t0, t1, t2, a[6], b[6]); + MONT_MAC(v0, v1, v2, a[7], b[5]); + MONT_MAC(w0, w1, w2, a[8], b[4]); + MONT_MAC(t0, t1, t2, a[9], b[3]); + MONT_MAC(v0, v1, v2, a[10], b[2]); + MONT_MAC(w0, w1, w2, a[11], b[1]); + MONT_MAC(t0, t1, t2, a[12], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[12]); + MONT_MAC(w0, w1, w2, u[1], m[11]); + MONT_MAC(t0, t1, t2, u[2], m[10]); + MONT_MAC(v0, v1, v2, u[3], m[9]); + MONT_MAC(w0, w1, w2, u[4], m[8]); + MONT_MAC(t0, t1, t2, u[5], m[7]); + MONT_MAC(v0, v1, v2, u[6], m[6]); + MONT_MAC(w0, w1, w2, u[7], m[5]); + MONT_MAC(t0, t1, t2, u[8], m[4]); + MONT_MAC(v0, v1, v2, u[9], m[3]); + MONT_MAC(w0, w1, w2, u[10], m[2]); + MONT_MAC(t0, t1, t2, u[11], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[12] = t0 * n0; + MONT_MAC(t0, t1, t2, u[12], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 13 */ + MONT_MAC(t0, t1, t2, a[0], b[13]); + MONT_MUL(v0, v1, v2, a[1], b[12]); + MONT_MUL(w0, w1, w2, a[2], b[11]); + MONT_MAC(t0, t1, t2, a[3], b[10]); + MONT_MAC(v0, v1, v2, a[4], b[9]); + MONT_MAC(w0, w1, w2, a[5], b[8]); + MONT_MAC(t0, t1, t2, a[6], b[7]); + MONT_MAC(v0, v1, v2, a[7], b[6]); + MONT_MAC(w0, w1, w2, a[8], b[5]); + MONT_MAC(t0, t1, t2, a[9], b[4]); + MONT_MAC(v0, v1, v2, a[10], b[3]); + MONT_MAC(w0, w1, w2, a[11], b[2]); + MONT_MAC(t0, t1, t2, a[12], b[1]); + MONT_MAC(v0, v1, v2, a[13], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[13]); + MONT_MAC(t0, t1, t2, u[1], m[12]); + MONT_MAC(v0, v1, v2, u[2], m[11]); + MONT_MAC(w0, w1, w2, u[3], m[10]); + MONT_MAC(t0, t1, t2, u[4], m[9]); + MONT_MAC(v0, v1, v2, u[5], m[8]); + MONT_MAC(w0, w1, w2, u[6], m[7]); + MONT_MAC(t0, t1, t2, u[7], m[6]); + MONT_MAC(v0, v1, v2, u[8], m[5]); + MONT_MAC(w0, w1, w2, u[9], m[4]); + MONT_MAC(t0, t1, t2, u[10], m[3]); + MONT_MAC(v0, v1, v2, u[11], m[2]); + MONT_MAC(w0, w1, w2, u[12], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[13] = t0 * n0; + MONT_MAC(t0, t1, t2, u[13], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 14 */ + MONT_MAC(t0, t1, t2, a[0], b[14]); + MONT_MUL(v0, v1, v2, a[1], b[13]); + MONT_MUL(w0, w1, w2, a[2], b[12]); + MONT_MAC(t0, t1, t2, a[3], b[11]); + MONT_MAC(v0, v1, v2, a[4], b[10]); + MONT_MAC(w0, w1, w2, a[5], b[9]); + MONT_MAC(t0, t1, t2, a[6], b[8]); + MONT_MAC(v0, v1, v2, a[7], b[7]); + MONT_MAC(w0, w1, w2, a[8], b[6]); + MONT_MAC(t0, t1, t2, a[9], b[5]); + MONT_MAC(v0, v1, v2, a[10], b[4]); + MONT_MAC(w0, w1, w2, a[11], b[3]); + MONT_MAC(t0, t1, t2, a[12], b[2]); + MONT_MAC(v0, v1, v2, a[13], b[1]); + MONT_MAC(w0, w1, w2, a[14], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[14]); + MONT_MAC(v0, v1, v2, u[1], m[13]); + MONT_MAC(w0, w1, w2, u[2], m[12]); + MONT_MAC(t0, t1, t2, u[3], m[11]); + MONT_MAC(v0, v1, v2, u[4], m[10]); + MONT_MAC(w0, w1, w2, u[5], m[9]); + MONT_MAC(t0, t1, t2, u[6], m[8]); + MONT_MAC(v0, v1, v2, u[7], m[7]); + MONT_MAC(w0, w1, w2, u[8], m[6]); + MONT_MAC(t0, t1, t2, u[9], m[5]); + MONT_MAC(v0, v1, v2, u[10], m[4]); + MONT_MAC(w0, w1, w2, u[11], m[3]); + MONT_MAC(t0, t1, t2, u[12], m[2]); + MONT_MAC(v0, v1, v2, u[13], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[14] = t0 * n0; + MONT_MAC(t0, t1, t2, u[14], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 15 */ + MONT_MAC(t0, t1, t2, a[0], b[15]); + MONT_MUL(v0, v1, v2, a[1], b[14]); + MONT_MUL(w0, w1, w2, a[2], b[13]); + MONT_MAC(t0, t1, t2, a[3], b[12]); + MONT_MAC(v0, v1, v2, a[4], b[11]); + MONT_MAC(w0, w1, w2, a[5], b[10]); + MONT_MAC(t0, t1, t2, a[6], b[9]); + MONT_MAC(v0, v1, v2, a[7], b[8]); + MONT_MAC(w0, w1, w2, a[8], b[7]); + MONT_MAC(t0, t1, t2, a[9], b[6]); + MONT_MAC(v0, v1, v2, a[10], b[5]); + MONT_MAC(w0, w1, w2, a[11], b[4]); + MONT_MAC(t0, t1, t2, a[12], b[3]); + MONT_MAC(v0, v1, v2, a[13], b[2]); + MONT_MAC(w0, w1, w2, a[14], b[1]); + MONT_MAC(t0, t1, t2, a[15], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[15]); + MONT_MAC(w0, w1, w2, u[1], m[14]); + MONT_MAC(t0, t1, t2, u[2], m[13]); + MONT_MAC(v0, v1, v2, u[3], m[12]); + MONT_MAC(w0, w1, w2, u[4], m[11]); + MONT_MAC(t0, t1, t2, u[5], m[10]); + MONT_MAC(v0, v1, v2, u[6], m[9]); + MONT_MAC(w0, w1, w2, u[7], m[8]); + MONT_MAC(t0, t1, t2, u[8], m[7]); + MONT_MAC(v0, v1, v2, u[9], m[6]); + MONT_MAC(w0, w1, w2, u[10], m[5]); + MONT_MAC(t0, t1, t2, u[11], m[4]); + MONT_MAC(v0, v1, v2, u[12], m[3]); + MONT_MAC(w0, w1, w2, u[13], m[2]); + MONT_MAC(t0, t1, t2, u[14], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[15] = t0 * n0; + MONT_MAC(t0, t1, t2, u[15], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 16 */ + MONT_MAC(t0, t1, t2, a[0], b[16]); + MONT_MUL(v0, v1, v2, a[1], b[15]); + MONT_MUL(w0, w1, w2, a[2], b[14]); + MONT_MAC(t0, t1, t2, a[3], b[13]); + MONT_MAC(v0, v1, v2, a[4], b[12]); + MONT_MAC(w0, w1, w2, a[5], b[11]); + MONT_MAC(t0, t1, t2, a[6], b[10]); + MONT_MAC(v0, v1, v2, a[7], b[9]); + MONT_MAC(w0, w1, w2, a[8], b[8]); + MONT_MAC(t0, t1, t2, a[9], b[7]); + MONT_MAC(v0, v1, v2, a[10], b[6]); + MONT_MAC(w0, w1, w2, a[11], b[5]); + MONT_MAC(t0, t1, t2, a[12], b[4]); + MONT_MAC(v0, v1, v2, a[13], b[3]); + MONT_MAC(w0, w1, w2, a[14], b[2]); + MONT_MAC(t0, t1, t2, a[15], b[1]); + MONT_MAC(v0, v1, v2, a[16], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[16]); + MONT_MAC(t0, t1, t2, u[1], m[15]); + MONT_MAC(v0, v1, v2, u[2], m[14]); + MONT_MAC(w0, w1, w2, u[3], m[13]); + MONT_MAC(t0, t1, t2, u[4], m[12]); + MONT_MAC(v0, v1, v2, u[5], m[11]); + MONT_MAC(w0, w1, w2, u[6], m[10]); + MONT_MAC(t0, t1, t2, u[7], m[9]); + MONT_MAC(v0, v1, v2, u[8], m[8]); + MONT_MAC(w0, w1, w2, u[9], m[7]); + MONT_MAC(t0, t1, t2, u[10], m[6]); + MONT_MAC(v0, v1, v2, u[11], m[5]); + MONT_MAC(w0, w1, w2, u[12], m[4]); + MONT_MAC(t0, t1, t2, u[13], m[3]); + MONT_MAC(v0, v1, v2, u[14], m[2]); + MONT_MAC(w0, w1, w2, u[15], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[16] = t0 * n0; + MONT_MAC(t0, t1, t2, u[16], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 17 */ + MONT_MAC(t0, t1, t2, a[0], b[17]); + MONT_MUL(v0, v1, v2, a[1], b[16]); + MONT_MUL(w0, w1, w2, a[2], b[15]); + MONT_MAC(t0, t1, t2, a[3], b[14]); + MONT_MAC(v0, v1, v2, a[4], b[13]); + MONT_MAC(w0, w1, w2, a[5], b[12]); + MONT_MAC(t0, t1, t2, a[6], b[11]); + MONT_MAC(v0, v1, v2, a[7], b[10]); + MONT_MAC(w0, w1, w2, a[8], b[9]); + MONT_MAC(t0, t1, t2, a[9], b[8]); + MONT_MAC(v0, v1, v2, a[10], b[7]); + MONT_MAC(w0, w1, w2, a[11], b[6]); + MONT_MAC(t0, t1, t2, a[12], b[5]); + MONT_MAC(v0, v1, v2, a[13], b[4]); + MONT_MAC(w0, w1, w2, a[14], b[3]); + MONT_MAC(t0, t1, t2, a[15], b[2]); + MONT_MAC(v0, v1, v2, a[16], b[1]); + MONT_MAC(w0, w1, w2, a[17], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[17]); + MONT_MAC(v0, v1, v2, u[1], m[16]); + MONT_MAC(w0, w1, w2, u[2], m[15]); + MONT_MAC(t0, t1, t2, u[3], m[14]); + MONT_MAC(v0, v1, v2, u[4], m[13]); + MONT_MAC(w0, w1, w2, u[5], m[12]); + MONT_MAC(t0, t1, t2, u[6], m[11]); + MONT_MAC(v0, v1, v2, u[7], m[10]); + MONT_MAC(w0, w1, w2, u[8], m[9]); + MONT_MAC(t0, t1, t2, u[9], m[8]); + MONT_MAC(v0, v1, v2, u[10], m[7]); + MONT_MAC(w0, w1, w2, u[11], m[6]); + MONT_MAC(t0, t1, t2, u[12], m[5]); + MONT_MAC(v0, v1, v2, u[13], m[4]); + MONT_MAC(w0, w1, w2, u[14], m[3]); + MONT_MAC(t0, t1, t2, u[15], m[2]); + MONT_MAC(v0, v1, v2, u[16], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[17] = t0 * n0; + MONT_MAC(t0, t1, t2, u[17], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 18 */ + MONT_MAC(t0, t1, t2, a[0], b[18]); + MONT_MUL(v0, v1, v2, a[1], b[17]); + MONT_MUL(w0, w1, w2, a[2], b[16]); + MONT_MAC(t0, t1, t2, a[3], b[15]); + MONT_MAC(v0, v1, v2, a[4], b[14]); + MONT_MAC(w0, w1, w2, a[5], b[13]); + MONT_MAC(t0, t1, t2, a[6], b[12]); + MONT_MAC(v0, v1, v2, a[7], b[11]); + MONT_MAC(w0, w1, w2, a[8], b[10]); + MONT_MAC(t0, t1, t2, a[9], b[9]); + MONT_MAC(v0, v1, v2, a[10], b[8]); + MONT_MAC(w0, w1, w2, a[11], b[7]); + MONT_MAC(t0, t1, t2, a[12], b[6]); + MONT_MAC(v0, v1, v2, a[13], b[5]); + MONT_MAC(w0, w1, w2, a[14], b[4]); + MONT_MAC(t0, t1, t2, a[15], b[3]); + MONT_MAC(v0, v1, v2, a[16], b[2]); + MONT_MAC(w0, w1, w2, a[17], b[1]); + MONT_MAC(t0, t1, t2, a[18], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[18]); + MONT_MAC(w0, w1, w2, u[1], m[17]); + MONT_MAC(t0, t1, t2, u[2], m[16]); + MONT_MAC(v0, v1, v2, u[3], m[15]); + MONT_MAC(w0, w1, w2, u[4], m[14]); + MONT_MAC(t0, t1, t2, u[5], m[13]); + MONT_MAC(v0, v1, v2, u[6], m[12]); + MONT_MAC(w0, w1, w2, u[7], m[11]); + MONT_MAC(t0, t1, t2, u[8], m[10]); + MONT_MAC(v0, v1, v2, u[9], m[9]); + MONT_MAC(w0, w1, w2, u[10], m[8]); + MONT_MAC(t0, t1, t2, u[11], m[7]); + MONT_MAC(v0, v1, v2, u[12], m[6]); + MONT_MAC(w0, w1, w2, u[13], m[5]); + MONT_MAC(t0, t1, t2, u[14], m[4]); + MONT_MAC(v0, v1, v2, u[15], m[3]); + MONT_MAC(w0, w1, w2, u[16], m[2]); + MONT_MAC(t0, t1, t2, u[17], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[18] = t0 * n0; + MONT_MAC(t0, t1, t2, u[18], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 19 */ + MONT_MAC(t0, t1, t2, a[0], b[19]); + MONT_MUL(v0, v1, v2, a[1], b[18]); + MONT_MUL(w0, w1, w2, a[2], b[17]); + MONT_MAC(t0, t1, t2, a[3], b[16]); + MONT_MAC(v0, v1, v2, a[4], b[15]); + MONT_MAC(w0, w1, w2, a[5], b[14]); + MONT_MAC(t0, t1, t2, a[6], b[13]); + MONT_MAC(v0, v1, v2, a[7], b[12]); + MONT_MAC(w0, w1, w2, a[8], b[11]); + MONT_MAC(t0, t1, t2, a[9], b[10]); + MONT_MAC(v0, v1, v2, a[10], b[9]); + MONT_MAC(w0, w1, w2, a[11], b[8]); + MONT_MAC(t0, t1, t2, a[12], b[7]); + MONT_MAC(v0, v1, v2, a[13], b[6]); + MONT_MAC(w0, w1, w2, a[14], b[5]); + MONT_MAC(t0, t1, t2, a[15], b[4]); + MONT_MAC(v0, v1, v2, a[16], b[3]); + MONT_MAC(w0, w1, w2, a[17], b[2]); + MONT_MAC(t0, t1, t2, a[18], b[1]); + MONT_MAC(v0, v1, v2, a[19], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[19]); + MONT_MAC(t0, t1, t2, u[1], m[18]); + MONT_MAC(v0, v1, v2, u[2], m[17]); + MONT_MAC(w0, w1, w2, u[3], m[16]); + MONT_MAC(t0, t1, t2, u[4], m[15]); + MONT_MAC(v0, v1, v2, u[5], m[14]); + MONT_MAC(w0, w1, w2, u[6], m[13]); + MONT_MAC(t0, t1, t2, u[7], m[12]); + MONT_MAC(v0, v1, v2, u[8], m[11]); + MONT_MAC(w0, w1, w2, u[9], m[10]); + MONT_MAC(t0, t1, t2, u[10], m[9]); + MONT_MAC(v0, v1, v2, u[11], m[8]); + MONT_MAC(w0, w1, w2, u[12], m[7]); + MONT_MAC(t0, t1, t2, u[13], m[6]); + MONT_MAC(v0, v1, v2, u[14], m[5]); + MONT_MAC(w0, w1, w2, u[15], m[4]); + MONT_MAC(t0, t1, t2, u[16], m[3]); + MONT_MAC(v0, v1, v2, u[17], m[2]); + MONT_MAC(w0, w1, w2, u[18], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[19] = t0 * n0; + MONT_MAC(t0, t1, t2, u[19], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 20 */ + MONT_MAC(t0, t1, t2, a[0], b[20]); + MONT_MUL(v0, v1, v2, a[1], b[19]); + MONT_MUL(w0, w1, w2, a[2], b[18]); + MONT_MAC(t0, t1, t2, a[3], b[17]); + MONT_MAC(v0, v1, v2, a[4], b[16]); + MONT_MAC(w0, w1, w2, a[5], b[15]); + MONT_MAC(t0, t1, t2, a[6], b[14]); + MONT_MAC(v0, v1, v2, a[7], b[13]); + MONT_MAC(w0, w1, w2, a[8], b[12]); + MONT_MAC(t0, t1, t2, a[9], b[11]); + MONT_MAC(v0, v1, v2, a[10], b[10]); + MONT_MAC(w0, w1, w2, a[11], b[9]); + MONT_MAC(t0, t1, t2, a[12], b[8]); + MONT_MAC(v0, v1, v2, a[13], b[7]); + MONT_MAC(w0, w1, w2, a[14], b[6]); + MONT_MAC(t0, t1, t2, a[15], b[5]); + MONT_MAC(v0, v1, v2, a[16], b[4]); + MONT_MAC(w0, w1, w2, a[17], b[3]); + MONT_MAC(t0, t1, t2, a[18], b[2]); + MONT_MAC(v0, v1, v2, a[19], b[1]); + MONT_MAC(w0, w1, w2, a[20], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[20]); + MONT_MAC(v0, v1, v2, u[1], m[19]); + MONT_MAC(w0, w1, w2, u[2], m[18]); + MONT_MAC(t0, t1, t2, u[3], m[17]); + MONT_MAC(v0, v1, v2, u[4], m[16]); + MONT_MAC(w0, w1, w2, u[5], m[15]); + MONT_MAC(t0, t1, t2, u[6], m[14]); + MONT_MAC(v0, v1, v2, u[7], m[13]); + MONT_MAC(w0, w1, w2, u[8], m[12]); + MONT_MAC(t0, t1, t2, u[9], m[11]); + MONT_MAC(v0, v1, v2, u[10], m[10]); + MONT_MAC(w0, w1, w2, u[11], m[9]); + MONT_MAC(t0, t1, t2, u[12], m[8]); + MONT_MAC(v0, v1, v2, u[13], m[7]); + MONT_MAC(w0, w1, w2, u[14], m[6]); + MONT_MAC(t0, t1, t2, u[15], m[5]); + MONT_MAC(v0, v1, v2, u[16], m[4]); + MONT_MAC(w0, w1, w2, u[17], m[3]); + MONT_MAC(t0, t1, t2, u[18], m[2]); + MONT_MAC(v0, v1, v2, u[19], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[20] = t0 * n0; + MONT_MAC(t0, t1, t2, u[20], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 21 */ + MONT_MAC(t0, t1, t2, a[0], b[21]); + MONT_MUL(v0, v1, v2, a[1], b[20]); + MONT_MUL(w0, w1, w2, a[2], b[19]); + MONT_MAC(t0, t1, t2, a[3], b[18]); + MONT_MAC(v0, v1, v2, a[4], b[17]); + MONT_MAC(w0, w1, w2, a[5], b[16]); + MONT_MAC(t0, t1, t2, a[6], b[15]); + MONT_MAC(v0, v1, v2, a[7], b[14]); + MONT_MAC(w0, w1, w2, a[8], b[13]); + MONT_MAC(t0, t1, t2, a[9], b[12]); + MONT_MAC(v0, v1, v2, a[10], b[11]); + MONT_MAC(w0, w1, w2, a[11], b[10]); + MONT_MAC(t0, t1, t2, a[12], b[9]); + MONT_MAC(v0, v1, v2, a[13], b[8]); + MONT_MAC(w0, w1, w2, a[14], b[7]); + MONT_MAC(t0, t1, t2, a[15], b[6]); + MONT_MAC(v0, v1, v2, a[16], b[5]); + MONT_MAC(w0, w1, w2, a[17], b[4]); + MONT_MAC(t0, t1, t2, a[18], b[3]); + MONT_MAC(v0, v1, v2, a[19], b[2]); + MONT_MAC(w0, w1, w2, a[20], b[1]); + MONT_MAC(t0, t1, t2, a[21], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[21]); + MONT_MAC(w0, w1, w2, u[1], m[20]); + MONT_MAC(t0, t1, t2, u[2], m[19]); + MONT_MAC(v0, v1, v2, u[3], m[18]); + MONT_MAC(w0, w1, w2, u[4], m[17]); + MONT_MAC(t0, t1, t2, u[5], m[16]); + MONT_MAC(v0, v1, v2, u[6], m[15]); + MONT_MAC(w0, w1, w2, u[7], m[14]); + MONT_MAC(t0, t1, t2, u[8], m[13]); + MONT_MAC(v0, v1, v2, u[9], m[12]); + MONT_MAC(w0, w1, w2, u[10], m[11]); + MONT_MAC(t0, t1, t2, u[11], m[10]); + MONT_MAC(v0, v1, v2, u[12], m[9]); + MONT_MAC(w0, w1, w2, u[13], m[8]); + MONT_MAC(t0, t1, t2, u[14], m[7]); + MONT_MAC(v0, v1, v2, u[15], m[6]); + MONT_MAC(w0, w1, w2, u[16], m[5]); + MONT_MAC(t0, t1, t2, u[17], m[4]); + MONT_MAC(v0, v1, v2, u[18], m[3]); + MONT_MAC(w0, w1, w2, u[19], m[2]); + MONT_MAC(t0, t1, t2, u[20], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[21] = t0 * n0; + MONT_MAC(t0, t1, t2, u[21], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 22 */ + MONT_MAC(t0, t1, t2, a[0], b[22]); + MONT_MUL(v0, v1, v2, a[1], b[21]); + MONT_MUL(w0, w1, w2, a[2], b[20]); + MONT_MAC(t0, t1, t2, a[3], b[19]); + MONT_MAC(v0, v1, v2, a[4], b[18]); + MONT_MAC(w0, w1, w2, a[5], b[17]); + MONT_MAC(t0, t1, t2, a[6], b[16]); + MONT_MAC(v0, v1, v2, a[7], b[15]); + MONT_MAC(w0, w1, w2, a[8], b[14]); + MONT_MAC(t0, t1, t2, a[9], b[13]); + MONT_MAC(v0, v1, v2, a[10], b[12]); + MONT_MAC(w0, w1, w2, a[11], b[11]); + MONT_MAC(t0, t1, t2, a[12], b[10]); + MONT_MAC(v0, v1, v2, a[13], b[9]); + MONT_MAC(w0, w1, w2, a[14], b[8]); + MONT_MAC(t0, t1, t2, a[15], b[7]); + MONT_MAC(v0, v1, v2, a[16], b[6]); + MONT_MAC(w0, w1, w2, a[17], b[5]); + MONT_MAC(t0, t1, t2, a[18], b[4]); + MONT_MAC(v0, v1, v2, a[19], b[3]); + MONT_MAC(w0, w1, w2, a[20], b[2]); + MONT_MAC(t0, t1, t2, a[21], b[1]); + MONT_MAC(v0, v1, v2, a[22], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[22]); + MONT_MAC(t0, t1, t2, u[1], m[21]); + MONT_MAC(v0, v1, v2, u[2], m[20]); + MONT_MAC(w0, w1, w2, u[3], m[19]); + MONT_MAC(t0, t1, t2, u[4], m[18]); + MONT_MAC(v0, v1, v2, u[5], m[17]); + MONT_MAC(w0, w1, w2, u[6], m[16]); + MONT_MAC(t0, t1, t2, u[7], m[15]); + MONT_MAC(v0, v1, v2, u[8], m[14]); + MONT_MAC(w0, w1, w2, u[9], m[13]); + MONT_MAC(t0, t1, t2, u[10], m[12]); + MONT_MAC(v0, v1, v2, u[11], m[11]); + MONT_MAC(w0, w1, w2, u[12], m[10]); + MONT_MAC(t0, t1, t2, u[13], m[9]); + MONT_MAC(v0, v1, v2, u[14], m[8]); + MONT_MAC(w0, w1, w2, u[15], m[7]); + MONT_MAC(t0, t1, t2, u[16], m[6]); + MONT_MAC(v0, v1, v2, u[17], m[5]); + MONT_MAC(w0, w1, w2, u[18], m[4]); + MONT_MAC(t0, t1, t2, u[19], m[3]); + MONT_MAC(v0, v1, v2, u[20], m[2]); + MONT_MAC(w0, w1, w2, u[21], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[22] = t0 * n0; + MONT_MAC(t0, t1, t2, u[22], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 23 */ + MONT_MAC(t0, t1, t2, a[0], b[23]); + MONT_MUL(v0, v1, v2, a[1], b[22]); + MONT_MUL(w0, w1, w2, a[2], b[21]); + MONT_MAC(t0, t1, t2, a[3], b[20]); + MONT_MAC(v0, v1, v2, a[4], b[19]); + MONT_MAC(w0, w1, w2, a[5], b[18]); + MONT_MAC(t0, t1, t2, a[6], b[17]); + MONT_MAC(v0, v1, v2, a[7], b[16]); + MONT_MAC(w0, w1, w2, a[8], b[15]); + MONT_MAC(t0, t1, t2, a[9], b[14]); + MONT_MAC(v0, v1, v2, a[10], b[13]); + MONT_MAC(w0, w1, w2, a[11], b[12]); + MONT_MAC(t0, t1, t2, a[12], b[11]); + MONT_MAC(v0, v1, v2, a[13], b[10]); + MONT_MAC(w0, w1, w2, a[14], b[9]); + MONT_MAC(t0, t1, t2, a[15], b[8]); + MONT_MAC(v0, v1, v2, a[16], b[7]); + MONT_MAC(w0, w1, w2, a[17], b[6]); + MONT_MAC(t0, t1, t2, a[18], b[5]); + MONT_MAC(v0, v1, v2, a[19], b[4]); + MONT_MAC(w0, w1, w2, a[20], b[3]); + MONT_MAC(t0, t1, t2, a[21], b[2]); + MONT_MAC(v0, v1, v2, a[22], b[1]); + MONT_MAC(w0, w1, w2, a[23], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[23]); + MONT_MAC(v0, v1, v2, u[1], m[22]); + MONT_MAC(w0, w1, w2, u[2], m[21]); + MONT_MAC(t0, t1, t2, u[3], m[20]); + MONT_MAC(v0, v1, v2, u[4], m[19]); + MONT_MAC(w0, w1, w2, u[5], m[18]); + MONT_MAC(t0, t1, t2, u[6], m[17]); + MONT_MAC(v0, v1, v2, u[7], m[16]); + MONT_MAC(w0, w1, w2, u[8], m[15]); + MONT_MAC(t0, t1, t2, u[9], m[14]); + MONT_MAC(v0, v1, v2, u[10], m[13]); + MONT_MAC(w0, w1, w2, u[11], m[12]); + MONT_MAC(t0, t1, t2, u[12], m[11]); + MONT_MAC(v0, v1, v2, u[13], m[10]); + MONT_MAC(w0, w1, w2, u[14], m[9]); + MONT_MAC(t0, t1, t2, u[15], m[8]); + MONT_MAC(v0, v1, v2, u[16], m[7]); + MONT_MAC(w0, w1, w2, u[17], m[6]); + MONT_MAC(t0, t1, t2, u[18], m[5]); + MONT_MAC(v0, v1, v2, u[19], m[4]); + MONT_MAC(w0, w1, w2, u[20], m[3]); + MONT_MAC(t0, t1, t2, u[21], m[2]); + MONT_MAC(v0, v1, v2, u[22], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[23] = t0 * n0; + MONT_MAC(t0, t1, t2, u[23], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 24 */ + MONT_MAC(t0, t1, t2, a[0], b[24]); + MONT_MUL(v0, v1, v2, a[1], b[23]); + MONT_MUL(w0, w1, w2, a[2], b[22]); + MONT_MAC(t0, t1, t2, a[3], b[21]); + MONT_MAC(v0, v1, v2, a[4], b[20]); + MONT_MAC(w0, w1, w2, a[5], b[19]); + MONT_MAC(t0, t1, t2, a[6], b[18]); + MONT_MAC(v0, v1, v2, a[7], b[17]); + MONT_MAC(w0, w1, w2, a[8], b[16]); + MONT_MAC(t0, t1, t2, a[9], b[15]); + MONT_MAC(v0, v1, v2, a[10], b[14]); + MONT_MAC(w0, w1, w2, a[11], b[13]); + MONT_MAC(t0, t1, t2, a[12], b[12]); + MONT_MAC(v0, v1, v2, a[13], b[11]); + MONT_MAC(w0, w1, w2, a[14], b[10]); + MONT_MAC(t0, t1, t2, a[15], b[9]); + MONT_MAC(v0, v1, v2, a[16], b[8]); + MONT_MAC(w0, w1, w2, a[17], b[7]); + MONT_MAC(t0, t1, t2, a[18], b[6]); + MONT_MAC(v0, v1, v2, a[19], b[5]); + MONT_MAC(w0, w1, w2, a[20], b[4]); + MONT_MAC(t0, t1, t2, a[21], b[3]); + MONT_MAC(v0, v1, v2, a[22], b[2]); + MONT_MAC(w0, w1, w2, a[23], b[1]); + MONT_MAC(t0, t1, t2, a[24], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[24]); + MONT_MAC(w0, w1, w2, u[1], m[23]); + MONT_MAC(t0, t1, t2, u[2], m[22]); + MONT_MAC(v0, v1, v2, u[3], m[21]); + MONT_MAC(w0, w1, w2, u[4], m[20]); + MONT_MAC(t0, t1, t2, u[5], m[19]); + MONT_MAC(v0, v1, v2, u[6], m[18]); + MONT_MAC(w0, w1, w2, u[7], m[17]); + MONT_MAC(t0, t1, t2, u[8], m[16]); + MONT_MAC(v0, v1, v2, u[9], m[15]); + MONT_MAC(w0, w1, w2, u[10], m[14]); + MONT_MAC(t0, t1, t2, u[11], m[13]); + MONT_MAC(v0, v1, v2, u[12], m[12]); + MONT_MAC(w0, w1, w2, u[13], m[11]); + MONT_MAC(t0, t1, t2, u[14], m[10]); + MONT_MAC(v0, v1, v2, u[15], m[9]); + MONT_MAC(w0, w1, w2, u[16], m[8]); + MONT_MAC(t0, t1, t2, u[17], m[7]); + MONT_MAC(v0, v1, v2, u[18], m[6]); + MONT_MAC(w0, w1, w2, u[19], m[5]); + MONT_MAC(t0, t1, t2, u[20], m[4]); + MONT_MAC(v0, v1, v2, u[21], m[3]); + MONT_MAC(w0, w1, w2, u[22], m[2]); + MONT_MAC(t0, t1, t2, u[23], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[24] = t0 * n0; + MONT_MAC(t0, t1, t2, u[24], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 25 */ + MONT_MAC(t0, t1, t2, a[0], b[25]); + MONT_MUL(v0, v1, v2, a[1], b[24]); + MONT_MUL(w0, w1, w2, a[2], b[23]); + MONT_MAC(t0, t1, t2, a[3], b[22]); + MONT_MAC(v0, v1, v2, a[4], b[21]); + MONT_MAC(w0, w1, w2, a[5], b[20]); + MONT_MAC(t0, t1, t2, a[6], b[19]); + MONT_MAC(v0, v1, v2, a[7], b[18]); + MONT_MAC(w0, w1, w2, a[8], b[17]); + MONT_MAC(t0, t1, t2, a[9], b[16]); + MONT_MAC(v0, v1, v2, a[10], b[15]); + MONT_MAC(w0, w1, w2, a[11], b[14]); + MONT_MAC(t0, t1, t2, a[12], b[13]); + MONT_MAC(v0, v1, v2, a[13], b[12]); + MONT_MAC(w0, w1, w2, a[14], b[11]); + MONT_MAC(t0, t1, t2, a[15], b[10]); + MONT_MAC(v0, v1, v2, a[16], b[9]); + MONT_MAC(w0, w1, w2, a[17], b[8]); + MONT_MAC(t0, t1, t2, a[18], b[7]); + MONT_MAC(v0, v1, v2, a[19], b[6]); + MONT_MAC(w0, w1, w2, a[20], b[5]); + MONT_MAC(t0, t1, t2, a[21], b[4]); + MONT_MAC(v0, v1, v2, a[22], b[3]); + MONT_MAC(w0, w1, w2, a[23], b[2]); + MONT_MAC(t0, t1, t2, a[24], b[1]); + MONT_MAC(v0, v1, v2, a[25], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[25]); + MONT_MAC(t0, t1, t2, u[1], m[24]); + MONT_MAC(v0, v1, v2, u[2], m[23]); + MONT_MAC(w0, w1, w2, u[3], m[22]); + MONT_MAC(t0, t1, t2, u[4], m[21]); + MONT_MAC(v0, v1, v2, u[5], m[20]); + MONT_MAC(w0, w1, w2, u[6], m[19]); + MONT_MAC(t0, t1, t2, u[7], m[18]); + MONT_MAC(v0, v1, v2, u[8], m[17]); + MONT_MAC(w0, w1, w2, u[9], m[16]); + MONT_MAC(t0, t1, t2, u[10], m[15]); + MONT_MAC(v0, v1, v2, u[11], m[14]); + MONT_MAC(w0, w1, w2, u[12], m[13]); + MONT_MAC(t0, t1, t2, u[13], m[12]); + MONT_MAC(v0, v1, v2, u[14], m[11]); + MONT_MAC(w0, w1, w2, u[15], m[10]); + MONT_MAC(t0, t1, t2, u[16], m[9]); + MONT_MAC(v0, v1, v2, u[17], m[8]); + MONT_MAC(w0, w1, w2, u[18], m[7]); + MONT_MAC(t0, t1, t2, u[19], m[6]); + MONT_MAC(v0, v1, v2, u[20], m[5]); + MONT_MAC(w0, w1, w2, u[21], m[4]); + MONT_MAC(t0, t1, t2, u[22], m[3]); + MONT_MAC(v0, v1, v2, u[23], m[2]); + MONT_MAC(w0, w1, w2, u[24], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[25] = t0 * n0; + MONT_MAC(t0, t1, t2, u[25], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 26 */ + MONT_MAC(t0, t1, t2, a[0], b[26]); + MONT_MUL(v0, v1, v2, a[1], b[25]); + MONT_MUL(w0, w1, w2, a[2], b[24]); + MONT_MAC(t0, t1, t2, a[3], b[23]); + MONT_MAC(v0, v1, v2, a[4], b[22]); + MONT_MAC(w0, w1, w2, a[5], b[21]); + MONT_MAC(t0, t1, t2, a[6], b[20]); + MONT_MAC(v0, v1, v2, a[7], b[19]); + MONT_MAC(w0, w1, w2, a[8], b[18]); + MONT_MAC(t0, t1, t2, a[9], b[17]); + MONT_MAC(v0, v1, v2, a[10], b[16]); + MONT_MAC(w0, w1, w2, a[11], b[15]); + MONT_MAC(t0, t1, t2, a[12], b[14]); + MONT_MAC(v0, v1, v2, a[13], b[13]); + MONT_MAC(w0, w1, w2, a[14], b[12]); + MONT_MAC(t0, t1, t2, a[15], b[11]); + MONT_MAC(v0, v1, v2, a[16], b[10]); + MONT_MAC(w0, w1, w2, a[17], b[9]); + MONT_MAC(t0, t1, t2, a[18], b[8]); + MONT_MAC(v0, v1, v2, a[19], b[7]); + MONT_MAC(w0, w1, w2, a[20], b[6]); + MONT_MAC(t0, t1, t2, a[21], b[5]); + MONT_MAC(v0, v1, v2, a[22], b[4]); + MONT_MAC(w0, w1, w2, a[23], b[3]); + MONT_MAC(t0, t1, t2, a[24], b[2]); + MONT_MAC(v0, v1, v2, a[25], b[1]); + MONT_MAC(w0, w1, w2, a[26], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[26]); + MONT_MAC(v0, v1, v2, u[1], m[25]); + MONT_MAC(w0, w1, w2, u[2], m[24]); + MONT_MAC(t0, t1, t2, u[3], m[23]); + MONT_MAC(v0, v1, v2, u[4], m[22]); + MONT_MAC(w0, w1, w2, u[5], m[21]); + MONT_MAC(t0, t1, t2, u[6], m[20]); + MONT_MAC(v0, v1, v2, u[7], m[19]); + MONT_MAC(w0, w1, w2, u[8], m[18]); + MONT_MAC(t0, t1, t2, u[9], m[17]); + MONT_MAC(v0, v1, v2, u[10], m[16]); + MONT_MAC(w0, w1, w2, u[11], m[15]); + MONT_MAC(t0, t1, t2, u[12], m[14]); + MONT_MAC(v0, v1, v2, u[13], m[13]); + MONT_MAC(w0, w1, w2, u[14], m[12]); + MONT_MAC(t0, t1, t2, u[15], m[11]); + MONT_MAC(v0, v1, v2, u[16], m[10]); + MONT_MAC(w0, w1, w2, u[17], m[9]); + MONT_MAC(t0, t1, t2, u[18], m[8]); + MONT_MAC(v0, v1, v2, u[19], m[7]); + MONT_MAC(w0, w1, w2, u[20], m[6]); + MONT_MAC(t0, t1, t2, u[21], m[5]); + MONT_MAC(v0, v1, v2, u[22], m[4]); + MONT_MAC(w0, w1, w2, u[23], m[3]); + MONT_MAC(t0, t1, t2, u[24], m[2]); + MONT_MAC(v0, v1, v2, u[25], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[26] = t0 * n0; + MONT_MAC(t0, t1, t2, u[26], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 27 */ + MONT_MAC(t0, t1, t2, a[0], b[27]); + MONT_MUL(v0, v1, v2, a[1], b[26]); + MONT_MUL(w0, w1, w2, a[2], b[25]); + MONT_MAC(t0, t1, t2, a[3], b[24]); + MONT_MAC(v0, v1, v2, a[4], b[23]); + MONT_MAC(w0, w1, w2, a[5], b[22]); + MONT_MAC(t0, t1, t2, a[6], b[21]); + MONT_MAC(v0, v1, v2, a[7], b[20]); + MONT_MAC(w0, w1, w2, a[8], b[19]); + MONT_MAC(t0, t1, t2, a[9], b[18]); + MONT_MAC(v0, v1, v2, a[10], b[17]); + MONT_MAC(w0, w1, w2, a[11], b[16]); + MONT_MAC(t0, t1, t2, a[12], b[15]); + MONT_MAC(v0, v1, v2, a[13], b[14]); + MONT_MAC(w0, w1, w2, a[14], b[13]); + MONT_MAC(t0, t1, t2, a[15], b[12]); + MONT_MAC(v0, v1, v2, a[16], b[11]); + MONT_MAC(w0, w1, w2, a[17], b[10]); + MONT_MAC(t0, t1, t2, a[18], b[9]); + MONT_MAC(v0, v1, v2, a[19], b[8]); + MONT_MAC(w0, w1, w2, a[20], b[7]); + MONT_MAC(t0, t1, t2, a[21], b[6]); + MONT_MAC(v0, v1, v2, a[22], b[5]); + MONT_MAC(w0, w1, w2, a[23], b[4]); + MONT_MAC(t0, t1, t2, a[24], b[3]); + MONT_MAC(v0, v1, v2, a[25], b[2]); + MONT_MAC(w0, w1, w2, a[26], b[1]); + MONT_MAC(t0, t1, t2, a[27], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[27]); + MONT_MAC(w0, w1, w2, u[1], m[26]); + MONT_MAC(t0, t1, t2, u[2], m[25]); + MONT_MAC(v0, v1, v2, u[3], m[24]); + MONT_MAC(w0, w1, w2, u[4], m[23]); + MONT_MAC(t0, t1, t2, u[5], m[22]); + MONT_MAC(v0, v1, v2, u[6], m[21]); + MONT_MAC(w0, w1, w2, u[7], m[20]); + MONT_MAC(t0, t1, t2, u[8], m[19]); + MONT_MAC(v0, v1, v2, u[9], m[18]); + MONT_MAC(w0, w1, w2, u[10], m[17]); + MONT_MAC(t0, t1, t2, u[11], m[16]); + MONT_MAC(v0, v1, v2, u[12], m[15]); + MONT_MAC(w0, w1, w2, u[13], m[14]); + MONT_MAC(t0, t1, t2, u[14], m[13]); + MONT_MAC(v0, v1, v2, u[15], m[12]); + MONT_MAC(w0, w1, w2, u[16], m[11]); + MONT_MAC(t0, t1, t2, u[17], m[10]); + MONT_MAC(v0, v1, v2, u[18], m[9]); + MONT_MAC(w0, w1, w2, u[19], m[8]); + MONT_MAC(t0, t1, t2, u[20], m[7]); + MONT_MAC(v0, v1, v2, u[21], m[6]); + MONT_MAC(w0, w1, w2, u[22], m[5]); + MONT_MAC(t0, t1, t2, u[23], m[4]); + MONT_MAC(v0, v1, v2, u[24], m[3]); + MONT_MAC(w0, w1, w2, u[25], m[2]); + MONT_MAC(t0, t1, t2, u[26], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[27] = t0 * n0; + MONT_MAC(t0, t1, t2, u[27], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 28 */ + MONT_MAC(t0, t1, t2, a[0], b[28]); + MONT_MUL(v0, v1, v2, a[1], b[27]); + MONT_MUL(w0, w1, w2, a[2], b[26]); + MONT_MAC(t0, t1, t2, a[3], b[25]); + MONT_MAC(v0, v1, v2, a[4], b[24]); + MONT_MAC(w0, w1, w2, a[5], b[23]); + MONT_MAC(t0, t1, t2, a[6], b[22]); + MONT_MAC(v0, v1, v2, a[7], b[21]); + MONT_MAC(w0, w1, w2, a[8], b[20]); + MONT_MAC(t0, t1, t2, a[9], b[19]); + MONT_MAC(v0, v1, v2, a[10], b[18]); + MONT_MAC(w0, w1, w2, a[11], b[17]); + MONT_MAC(t0, t1, t2, a[12], b[16]); + MONT_MAC(v0, v1, v2, a[13], b[15]); + MONT_MAC(w0, w1, w2, a[14], b[14]); + MONT_MAC(t0, t1, t2, a[15], b[13]); + MONT_MAC(v0, v1, v2, a[16], b[12]); + MONT_MAC(w0, w1, w2, a[17], b[11]); + MONT_MAC(t0, t1, t2, a[18], b[10]); + MONT_MAC(v0, v1, v2, a[19], b[9]); + MONT_MAC(w0, w1, w2, a[20], b[8]); + MONT_MAC(t0, t1, t2, a[21], b[7]); + MONT_MAC(v0, v1, v2, a[22], b[6]); + MONT_MAC(w0, w1, w2, a[23], b[5]); + MONT_MAC(t0, t1, t2, a[24], b[4]); + MONT_MAC(v0, v1, v2, a[25], b[3]); + MONT_MAC(w0, w1, w2, a[26], b[2]); + MONT_MAC(t0, t1, t2, a[27], b[1]); + MONT_MAC(v0, v1, v2, a[28], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[28]); + MONT_MAC(t0, t1, t2, u[1], m[27]); + MONT_MAC(v0, v1, v2, u[2], m[26]); + MONT_MAC(w0, w1, w2, u[3], m[25]); + MONT_MAC(t0, t1, t2, u[4], m[24]); + MONT_MAC(v0, v1, v2, u[5], m[23]); + MONT_MAC(w0, w1, w2, u[6], m[22]); + MONT_MAC(t0, t1, t2, u[7], m[21]); + MONT_MAC(v0, v1, v2, u[8], m[20]); + MONT_MAC(w0, w1, w2, u[9], m[19]); + MONT_MAC(t0, t1, t2, u[10], m[18]); + MONT_MAC(v0, v1, v2, u[11], m[17]); + MONT_MAC(w0, w1, w2, u[12], m[16]); + MONT_MAC(t0, t1, t2, u[13], m[15]); + MONT_MAC(v0, v1, v2, u[14], m[14]); + MONT_MAC(w0, w1, w2, u[15], m[13]); + MONT_MAC(t0, t1, t2, u[16], m[12]); + MONT_MAC(v0, v1, v2, u[17], m[11]); + MONT_MAC(w0, w1, w2, u[18], m[10]); + MONT_MAC(t0, t1, t2, u[19], m[9]); + MONT_MAC(v0, v1, v2, u[20], m[8]); + MONT_MAC(w0, w1, w2, u[21], m[7]); + MONT_MAC(t0, t1, t2, u[22], m[6]); + MONT_MAC(v0, v1, v2, u[23], m[5]); + MONT_MAC(w0, w1, w2, u[24], m[4]); + MONT_MAC(t0, t1, t2, u[25], m[3]); + MONT_MAC(v0, v1, v2, u[26], m[2]); + MONT_MAC(w0, w1, w2, u[27], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[28] = t0 * n0; + MONT_MAC(t0, t1, t2, u[28], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 29 */ + MONT_MAC(t0, t1, t2, a[0], b[29]); + MONT_MUL(v0, v1, v2, a[1], b[28]); + MONT_MUL(w0, w1, w2, a[2], b[27]); + MONT_MAC(t0, t1, t2, a[3], b[26]); + MONT_MAC(v0, v1, v2, a[4], b[25]); + MONT_MAC(w0, w1, w2, a[5], b[24]); + MONT_MAC(t0, t1, t2, a[6], b[23]); + MONT_MAC(v0, v1, v2, a[7], b[22]); + MONT_MAC(w0, w1, w2, a[8], b[21]); + MONT_MAC(t0, t1, t2, a[9], b[20]); + MONT_MAC(v0, v1, v2, a[10], b[19]); + MONT_MAC(w0, w1, w2, a[11], b[18]); + MONT_MAC(t0, t1, t2, a[12], b[17]); + MONT_MAC(v0, v1, v2, a[13], b[16]); + MONT_MAC(w0, w1, w2, a[14], b[15]); + MONT_MAC(t0, t1, t2, a[15], b[14]); + MONT_MAC(v0, v1, v2, a[16], b[13]); + MONT_MAC(w0, w1, w2, a[17], b[12]); + MONT_MAC(t0, t1, t2, a[18], b[11]); + MONT_MAC(v0, v1, v2, a[19], b[10]); + MONT_MAC(w0, w1, w2, a[20], b[9]); + MONT_MAC(t0, t1, t2, a[21], b[8]); + MONT_MAC(v0, v1, v2, a[22], b[7]); + MONT_MAC(w0, w1, w2, a[23], b[6]); + MONT_MAC(t0, t1, t2, a[24], b[5]); + MONT_MAC(v0, v1, v2, a[25], b[4]); + MONT_MAC(w0, w1, w2, a[26], b[3]); + MONT_MAC(t0, t1, t2, a[27], b[2]); + MONT_MAC(v0, v1, v2, a[28], b[1]); + MONT_MAC(w0, w1, w2, a[29], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[29]); + MONT_MAC(v0, v1, v2, u[1], m[28]); + MONT_MAC(w0, w1, w2, u[2], m[27]); + MONT_MAC(t0, t1, t2, u[3], m[26]); + MONT_MAC(v0, v1, v2, u[4], m[25]); + MONT_MAC(w0, w1, w2, u[5], m[24]); + MONT_MAC(t0, t1, t2, u[6], m[23]); + MONT_MAC(v0, v1, v2, u[7], m[22]); + MONT_MAC(w0, w1, w2, u[8], m[21]); + MONT_MAC(t0, t1, t2, u[9], m[20]); + MONT_MAC(v0, v1, v2, u[10], m[19]); + MONT_MAC(w0, w1, w2, u[11], m[18]); + MONT_MAC(t0, t1, t2, u[12], m[17]); + MONT_MAC(v0, v1, v2, u[13], m[16]); + MONT_MAC(w0, w1, w2, u[14], m[15]); + MONT_MAC(t0, t1, t2, u[15], m[14]); + MONT_MAC(v0, v1, v2, u[16], m[13]); + MONT_MAC(w0, w1, w2, u[17], m[12]); + MONT_MAC(t0, t1, t2, u[18], m[11]); + MONT_MAC(v0, v1, v2, u[19], m[10]); + MONT_MAC(w0, w1, w2, u[20], m[9]); + MONT_MAC(t0, t1, t2, u[21], m[8]); + MONT_MAC(v0, v1, v2, u[22], m[7]); + MONT_MAC(w0, w1, w2, u[23], m[6]); + MONT_MAC(t0, t1, t2, u[24], m[5]); + MONT_MAC(v0, v1, v2, u[25], m[4]); + MONT_MAC(w0, w1, w2, u[26], m[3]); + MONT_MAC(t0, t1, t2, u[27], m[2]); + MONT_MAC(v0, v1, v2, u[28], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[29] = t0 * n0; + MONT_MAC(t0, t1, t2, u[29], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 30 */ + MONT_MAC(t0, t1, t2, a[0], b[30]); + MONT_MUL(v0, v1, v2, a[1], b[29]); + MONT_MUL(w0, w1, w2, a[2], b[28]); + MONT_MAC(t0, t1, t2, a[3], b[27]); + MONT_MAC(v0, v1, v2, a[4], b[26]); + MONT_MAC(w0, w1, w2, a[5], b[25]); + MONT_MAC(t0, t1, t2, a[6], b[24]); + MONT_MAC(v0, v1, v2, a[7], b[23]); + MONT_MAC(w0, w1, w2, a[8], b[22]); + MONT_MAC(t0, t1, t2, a[9], b[21]); + MONT_MAC(v0, v1, v2, a[10], b[20]); + MONT_MAC(w0, w1, w2, a[11], b[19]); + MONT_MAC(t0, t1, t2, a[12], b[18]); + MONT_MAC(v0, v1, v2, a[13], b[17]); + MONT_MAC(w0, w1, w2, a[14], b[16]); + MONT_MAC(t0, t1, t2, a[15], b[15]); + MONT_MAC(v0, v1, v2, a[16], b[14]); + MONT_MAC(w0, w1, w2, a[17], b[13]); + MONT_MAC(t0, t1, t2, a[18], b[12]); + MONT_MAC(v0, v1, v2, a[19], b[11]); + MONT_MAC(w0, w1, w2, a[20], b[10]); + MONT_MAC(t0, t1, t2, a[21], b[9]); + MONT_MAC(v0, v1, v2, a[22], b[8]); + MONT_MAC(w0, w1, w2, a[23], b[7]); + MONT_MAC(t0, t1, t2, a[24], b[6]); + MONT_MAC(v0, v1, v2, a[25], b[5]); + MONT_MAC(w0, w1, w2, a[26], b[4]); + MONT_MAC(t0, t1, t2, a[27], b[3]); + MONT_MAC(v0, v1, v2, a[28], b[2]); + MONT_MAC(w0, w1, w2, a[29], b[1]); + MONT_MAC(t0, t1, t2, a[30], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[30]); + MONT_MAC(w0, w1, w2, u[1], m[29]); + MONT_MAC(t0, t1, t2, u[2], m[28]); + MONT_MAC(v0, v1, v2, u[3], m[27]); + MONT_MAC(w0, w1, w2, u[4], m[26]); + MONT_MAC(t0, t1, t2, u[5], m[25]); + MONT_MAC(v0, v1, v2, u[6], m[24]); + MONT_MAC(w0, w1, w2, u[7], m[23]); + MONT_MAC(t0, t1, t2, u[8], m[22]); + MONT_MAC(v0, v1, v2, u[9], m[21]); + MONT_MAC(w0, w1, w2, u[10], m[20]); + MONT_MAC(t0, t1, t2, u[11], m[19]); + MONT_MAC(v0, v1, v2, u[12], m[18]); + MONT_MAC(w0, w1, w2, u[13], m[17]); + MONT_MAC(t0, t1, t2, u[14], m[16]); + MONT_MAC(v0, v1, v2, u[15], m[15]); + MONT_MAC(w0, w1, w2, u[16], m[14]); + MONT_MAC(t0, t1, t2, u[17], m[13]); + MONT_MAC(v0, v1, v2, u[18], m[12]); + MONT_MAC(w0, w1, w2, u[19], m[11]); + MONT_MAC(t0, t1, t2, u[20], m[10]); + MONT_MAC(v0, v1, v2, u[21], m[9]); + MONT_MAC(w0, w1, w2, u[22], m[8]); + MONT_MAC(t0, t1, t2, u[23], m[7]); + MONT_MAC(v0, v1, v2, u[24], m[6]); + MONT_MAC(w0, w1, w2, u[25], m[5]); + MONT_MAC(t0, t1, t2, u[26], m[4]); + MONT_MAC(v0, v1, v2, u[27], m[3]); + MONT_MAC(w0, w1, w2, u[28], m[2]); + MONT_MAC(t0, t1, t2, u[29], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[30] = t0 * n0; + MONT_MAC(t0, t1, t2, u[30], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 31 */ + MONT_MAC(t0, t1, t2, a[0], b[31]); + MONT_MUL(v0, v1, v2, a[1], b[30]); + MONT_MUL(w0, w1, w2, a[2], b[29]); + MONT_MAC(t0, t1, t2, a[3], b[28]); + MONT_MAC(v0, v1, v2, a[4], b[27]); + MONT_MAC(w0, w1, w2, a[5], b[26]); + MONT_MAC(t0, t1, t2, a[6], b[25]); + MONT_MAC(v0, v1, v2, a[7], b[24]); + MONT_MAC(w0, w1, w2, a[8], b[23]); + MONT_MAC(t0, t1, t2, a[9], b[22]); + MONT_MAC(v0, v1, v2, a[10], b[21]); + MONT_MAC(w0, w1, w2, a[11], b[20]); + MONT_MAC(t0, t1, t2, a[12], b[19]); + MONT_MAC(v0, v1, v2, a[13], b[18]); + MONT_MAC(w0, w1, w2, a[14], b[17]); + MONT_MAC(t0, t1, t2, a[15], b[16]); + MONT_MAC(v0, v1, v2, a[16], b[15]); + MONT_MAC(w0, w1, w2, a[17], b[14]); + MONT_MAC(t0, t1, t2, a[18], b[13]); + MONT_MAC(v0, v1, v2, a[19], b[12]); + MONT_MAC(w0, w1, w2, a[20], b[11]); + MONT_MAC(t0, t1, t2, a[21], b[10]); + MONT_MAC(v0, v1, v2, a[22], b[9]); + MONT_MAC(w0, w1, w2, a[23], b[8]); + MONT_MAC(t0, t1, t2, a[24], b[7]); + MONT_MAC(v0, v1, v2, a[25], b[6]); + MONT_MAC(w0, w1, w2, a[26], b[5]); + MONT_MAC(t0, t1, t2, a[27], b[4]); + MONT_MAC(v0, v1, v2, a[28], b[3]); + MONT_MAC(w0, w1, w2, a[29], b[2]); + MONT_MAC(t0, t1, t2, a[30], b[1]); + MONT_MAC(v0, v1, v2, a[31], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[31]); + MONT_MAC(t0, t1, t2, u[1], m[30]); + MONT_MAC(v0, v1, v2, u[2], m[29]); + MONT_MAC(w0, w1, w2, u[3], m[28]); + MONT_MAC(t0, t1, t2, u[4], m[27]); + MONT_MAC(v0, v1, v2, u[5], m[26]); + MONT_MAC(w0, w1, w2, u[6], m[25]); + MONT_MAC(t0, t1, t2, u[7], m[24]); + MONT_MAC(v0, v1, v2, u[8], m[23]); + MONT_MAC(w0, w1, w2, u[9], m[22]); + MONT_MAC(t0, t1, t2, u[10], m[21]); + MONT_MAC(v0, v1, v2, u[11], m[20]); + MONT_MAC(w0, w1, w2, u[12], m[19]); + MONT_MAC(t0, t1, t2, u[13], m[18]); + MONT_MAC(v0, v1, v2, u[14], m[17]); + MONT_MAC(w0, w1, w2, u[15], m[16]); + MONT_MAC(t0, t1, t2, u[16], m[15]); + MONT_MAC(v0, v1, v2, u[17], m[14]); + MONT_MAC(w0, w1, w2, u[18], m[13]); + MONT_MAC(t0, t1, t2, u[19], m[12]); + MONT_MAC(v0, v1, v2, u[20], m[11]); + MONT_MAC(w0, w1, w2, u[21], m[10]); + MONT_MAC(t0, t1, t2, u[22], m[9]); + MONT_MAC(v0, v1, v2, u[23], m[8]); + MONT_MAC(w0, w1, w2, u[24], m[7]); + MONT_MAC(t0, t1, t2, u[25], m[6]); + MONT_MAC(v0, v1, v2, u[26], m[5]); + MONT_MAC(w0, w1, w2, u[27], m[4]); + MONT_MAC(t0, t1, t2, u[28], m[3]); + MONT_MAC(v0, v1, v2, u[29], m[2]); + MONT_MAC(w0, w1, w2, u[30], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[31] = t0 * n0; + MONT_MAC(t0, t1, t2, u[31], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 32 */ + MONT_MAC(t0, t1, t2, a[1], b[31]); + MONT_MUL(v0, v1, v2, a[2], b[30]); + MONT_MUL(w0, w1, w2, a[3], b[29]); + MONT_MAC(t0, t1, t2, a[4], b[28]); + MONT_MAC(v0, v1, v2, a[5], b[27]); + MONT_MAC(w0, w1, w2, a[6], b[26]); + MONT_MAC(t0, t1, t2, a[7], b[25]); + MONT_MAC(v0, v1, v2, a[8], b[24]); + MONT_MAC(w0, w1, w2, a[9], b[23]); + MONT_MAC(t0, t1, t2, a[10], b[22]); + MONT_MAC(v0, v1, v2, a[11], b[21]); + MONT_MAC(w0, w1, w2, a[12], b[20]); + MONT_MAC(t0, t1, t2, a[13], b[19]); + MONT_MAC(v0, v1, v2, a[14], b[18]); + MONT_MAC(w0, w1, w2, a[15], b[17]); + MONT_MAC(t0, t1, t2, a[16], b[16]); + MONT_MAC(v0, v1, v2, a[17], b[15]); + MONT_MAC(w0, w1, w2, a[18], b[14]); + MONT_MAC(t0, t1, t2, a[19], b[13]); + MONT_MAC(v0, v1, v2, a[20], b[12]); + MONT_MAC(w0, w1, w2, a[21], b[11]); + MONT_MAC(t0, t1, t2, a[22], b[10]); + MONT_MAC(v0, v1, v2, a[23], b[9]); + MONT_MAC(w0, w1, w2, a[24], b[8]); + MONT_MAC(t0, t1, t2, a[25], b[7]); + MONT_MAC(v0, v1, v2, a[26], b[6]); + MONT_MAC(w0, w1, w2, a[27], b[5]); + MONT_MAC(t0, t1, t2, a[28], b[4]); + MONT_MAC(v0, v1, v2, a[29], b[3]); + MONT_MAC(w0, w1, w2, a[30], b[2]); + MONT_MAC(t0, t1, t2, a[31], b[1]); + MONT_MAC(v0, v1, v2, u[1], m[31]); + MONT_MAC(w0, w1, w2, u[2], m[30]); + MONT_MAC(t0, t1, t2, u[3], m[29]); + MONT_MAC(v0, v1, v2, u[4], m[28]); + MONT_MAC(w0, w1, w2, u[5], m[27]); + MONT_MAC(t0, t1, t2, u[6], m[26]); + MONT_MAC(v0, v1, v2, u[7], m[25]); + MONT_MAC(w0, w1, w2, u[8], m[24]); + MONT_MAC(t0, t1, t2, u[9], m[23]); + MONT_MAC(v0, v1, v2, u[10], m[22]); + MONT_MAC(w0, w1, w2, u[11], m[21]); + MONT_MAC(t0, t1, t2, u[12], m[20]); + MONT_MAC(v0, v1, v2, u[13], m[19]); + MONT_MAC(w0, w1, w2, u[14], m[18]); + MONT_MAC(t0, t1, t2, u[15], m[17]); + MONT_MAC(v0, v1, v2, u[16], m[16]); + MONT_MAC(w0, w1, w2, u[17], m[15]); + MONT_MAC(t0, t1, t2, u[18], m[14]); + MONT_MAC(v0, v1, v2, u[19], m[13]); + MONT_MAC(w0, w1, w2, u[20], m[12]); + MONT_MAC(t0, t1, t2, u[21], m[11]); + MONT_MAC(v0, v1, v2, u[22], m[10]); + MONT_MAC(w0, w1, w2, u[23], m[9]); + MONT_MAC(t0, t1, t2, u[24], m[8]); + MONT_MAC(v0, v1, v2, u[25], m[7]); + MONT_MAC(w0, w1, w2, u[26], m[6]); + MONT_MAC(t0, t1, t2, u[27], m[5]); + MONT_MAC(v0, v1, v2, u[28], m[4]); + MONT_MAC(w0, w1, w2, u[29], m[3]); + MONT_MAC(t0, t1, t2, u[30], m[2]); + MONT_MAC(v0, v1, v2, u[31], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[0] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 33 */ + MONT_MAC(t0, t1, t2, a[2], b[31]); + MONT_MUL(v0, v1, v2, a[3], b[30]); + MONT_MUL(w0, w1, w2, a[4], b[29]); + MONT_MAC(t0, t1, t2, a[5], b[28]); + MONT_MAC(v0, v1, v2, a[6], b[27]); + MONT_MAC(w0, w1, w2, a[7], b[26]); + MONT_MAC(t0, t1, t2, a[8], b[25]); + MONT_MAC(v0, v1, v2, a[9], b[24]); + MONT_MAC(w0, w1, w2, a[10], b[23]); + MONT_MAC(t0, t1, t2, a[11], b[22]); + MONT_MAC(v0, v1, v2, a[12], b[21]); + MONT_MAC(w0, w1, w2, a[13], b[20]); + MONT_MAC(t0, t1, t2, a[14], b[19]); + MONT_MAC(v0, v1, v2, a[15], b[18]); + MONT_MAC(w0, w1, w2, a[16], b[17]); + MONT_MAC(t0, t1, t2, a[17], b[16]); + MONT_MAC(v0, v1, v2, a[18], b[15]); + MONT_MAC(w0, w1, w2, a[19], b[14]); + MONT_MAC(t0, t1, t2, a[20], b[13]); + MONT_MAC(v0, v1, v2, a[21], b[12]); + MONT_MAC(w0, w1, w2, a[22], b[11]); + MONT_MAC(t0, t1, t2, a[23], b[10]); + MONT_MAC(v0, v1, v2, a[24], b[9]); + MONT_MAC(w0, w1, w2, a[25], b[8]); + MONT_MAC(t0, t1, t2, a[26], b[7]); + MONT_MAC(v0, v1, v2, a[27], b[6]); + MONT_MAC(w0, w1, w2, a[28], b[5]); + MONT_MAC(t0, t1, t2, a[29], b[4]); + MONT_MAC(v0, v1, v2, a[30], b[3]); + MONT_MAC(w0, w1, w2, a[31], b[2]); + MONT_MAC(t0, t1, t2, u[2], m[31]); + MONT_MAC(v0, v1, v2, u[3], m[30]); + MONT_MAC(w0, w1, w2, u[4], m[29]); + MONT_MAC(t0, t1, t2, u[5], m[28]); + MONT_MAC(v0, v1, v2, u[6], m[27]); + MONT_MAC(w0, w1, w2, u[7], m[26]); + MONT_MAC(t0, t1, t2, u[8], m[25]); + MONT_MAC(v0, v1, v2, u[9], m[24]); + MONT_MAC(w0, w1, w2, u[10], m[23]); + MONT_MAC(t0, t1, t2, u[11], m[22]); + MONT_MAC(v0, v1, v2, u[12], m[21]); + MONT_MAC(w0, w1, w2, u[13], m[20]); + MONT_MAC(t0, t1, t2, u[14], m[19]); + MONT_MAC(v0, v1, v2, u[15], m[18]); + MONT_MAC(w0, w1, w2, u[16], m[17]); + MONT_MAC(t0, t1, t2, u[17], m[16]); + MONT_MAC(v0, v1, v2, u[18], m[15]); + MONT_MAC(w0, w1, w2, u[19], m[14]); + MONT_MAC(t0, t1, t2, u[20], m[13]); + MONT_MAC(v0, v1, v2, u[21], m[12]); + MONT_MAC(w0, w1, w2, u[22], m[11]); + MONT_MAC(t0, t1, t2, u[23], m[10]); + MONT_MAC(v0, v1, v2, u[24], m[9]); + MONT_MAC(w0, w1, w2, u[25], m[8]); + MONT_MAC(t0, t1, t2, u[26], m[7]); + MONT_MAC(v0, v1, v2, u[27], m[6]); + MONT_MAC(w0, w1, w2, u[28], m[5]); + MONT_MAC(t0, t1, t2, u[29], m[4]); + MONT_MAC(v0, v1, v2, u[30], m[3]); + MONT_MAC(w0, w1, w2, u[31], m[2]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[1] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 34 */ + MONT_MAC(t0, t1, t2, a[3], b[31]); + MONT_MUL(v0, v1, v2, a[4], b[30]); + MONT_MUL(w0, w1, w2, a[5], b[29]); + MONT_MAC(t0, t1, t2, a[6], b[28]); + MONT_MAC(v0, v1, v2, a[7], b[27]); + MONT_MAC(w0, w1, w2, a[8], b[26]); + MONT_MAC(t0, t1, t2, a[9], b[25]); + MONT_MAC(v0, v1, v2, a[10], b[24]); + MONT_MAC(w0, w1, w2, a[11], b[23]); + MONT_MAC(t0, t1, t2, a[12], b[22]); + MONT_MAC(v0, v1, v2, a[13], b[21]); + MONT_MAC(w0, w1, w2, a[14], b[20]); + MONT_MAC(t0, t1, t2, a[15], b[19]); + MONT_MAC(v0, v1, v2, a[16], b[18]); + MONT_MAC(w0, w1, w2, a[17], b[17]); + MONT_MAC(t0, t1, t2, a[18], b[16]); + MONT_MAC(v0, v1, v2, a[19], b[15]); + MONT_MAC(w0, w1, w2, a[20], b[14]); + MONT_MAC(t0, t1, t2, a[21], b[13]); + MONT_MAC(v0, v1, v2, a[22], b[12]); + MONT_MAC(w0, w1, w2, a[23], b[11]); + MONT_MAC(t0, t1, t2, a[24], b[10]); + MONT_MAC(v0, v1, v2, a[25], b[9]); + MONT_MAC(w0, w1, w2, a[26], b[8]); + MONT_MAC(t0, t1, t2, a[27], b[7]); + MONT_MAC(v0, v1, v2, a[28], b[6]); + MONT_MAC(w0, w1, w2, a[29], b[5]); + MONT_MAC(t0, t1, t2, a[30], b[4]); + MONT_MAC(v0, v1, v2, a[31], b[3]); + MONT_MAC(w0, w1, w2, u[3], m[31]); + MONT_MAC(t0, t1, t2, u[4], m[30]); + MONT_MAC(v0, v1, v2, u[5], m[29]); + MONT_MAC(w0, w1, w2, u[6], m[28]); + MONT_MAC(t0, t1, t2, u[7], m[27]); + MONT_MAC(v0, v1, v2, u[8], m[26]); + MONT_MAC(w0, w1, w2, u[9], m[25]); + MONT_MAC(t0, t1, t2, u[10], m[24]); + MONT_MAC(v0, v1, v2, u[11], m[23]); + MONT_MAC(w0, w1, w2, u[12], m[22]); + MONT_MAC(t0, t1, t2, u[13], m[21]); + MONT_MAC(v0, v1, v2, u[14], m[20]); + MONT_MAC(w0, w1, w2, u[15], m[19]); + MONT_MAC(t0, t1, t2, u[16], m[18]); + MONT_MAC(v0, v1, v2, u[17], m[17]); + MONT_MAC(w0, w1, w2, u[18], m[16]); + MONT_MAC(t0, t1, t2, u[19], m[15]); + MONT_MAC(v0, v1, v2, u[20], m[14]); + MONT_MAC(w0, w1, w2, u[21], m[13]); + MONT_MAC(t0, t1, t2, u[22], m[12]); + MONT_MAC(v0, v1, v2, u[23], m[11]); + MONT_MAC(w0, w1, w2, u[24], m[10]); + MONT_MAC(t0, t1, t2, u[25], m[9]); + MONT_MAC(v0, v1, v2, u[26], m[8]); + MONT_MAC(w0, w1, w2, u[27], m[7]); + MONT_MAC(t0, t1, t2, u[28], m[6]); + MONT_MAC(v0, v1, v2, u[29], m[5]); + MONT_MAC(w0, w1, w2, u[30], m[4]); + MONT_MAC(t0, t1, t2, u[31], m[3]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[2] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 35 */ + MONT_MAC(t0, t1, t2, a[4], b[31]); + MONT_MUL(v0, v1, v2, a[5], b[30]); + MONT_MUL(w0, w1, w2, a[6], b[29]); + MONT_MAC(t0, t1, t2, a[7], b[28]); + MONT_MAC(v0, v1, v2, a[8], b[27]); + MONT_MAC(w0, w1, w2, a[9], b[26]); + MONT_MAC(t0, t1, t2, a[10], b[25]); + MONT_MAC(v0, v1, v2, a[11], b[24]); + MONT_MAC(w0, w1, w2, a[12], b[23]); + MONT_MAC(t0, t1, t2, a[13], b[22]); + MONT_MAC(v0, v1, v2, a[14], b[21]); + MONT_MAC(w0, w1, w2, a[15], b[20]); + MONT_MAC(t0, t1, t2, a[16], b[19]); + MONT_MAC(v0, v1, v2, a[17], b[18]); + MONT_MAC(w0, w1, w2, a[18], b[17]); + MONT_MAC(t0, t1, t2, a[19], b[16]); + MONT_MAC(v0, v1, v2, a[20], b[15]); + MONT_MAC(w0, w1, w2, a[21], b[14]); + MONT_MAC(t0, t1, t2, a[22], b[13]); + MONT_MAC(v0, v1, v2, a[23], b[12]); + MONT_MAC(w0, w1, w2, a[24], b[11]); + MONT_MAC(t0, t1, t2, a[25], b[10]); + MONT_MAC(v0, v1, v2, a[26], b[9]); + MONT_MAC(w0, w1, w2, a[27], b[8]); + MONT_MAC(t0, t1, t2, a[28], b[7]); + MONT_MAC(v0, v1, v2, a[29], b[6]); + MONT_MAC(w0, w1, w2, a[30], b[5]); + MONT_MAC(t0, t1, t2, a[31], b[4]); + MONT_MAC(v0, v1, v2, u[4], m[31]); + MONT_MAC(w0, w1, w2, u[5], m[30]); + MONT_MAC(t0, t1, t2, u[6], m[29]); + MONT_MAC(v0, v1, v2, u[7], m[28]); + MONT_MAC(w0, w1, w2, u[8], m[27]); + MONT_MAC(t0, t1, t2, u[9], m[26]); + MONT_MAC(v0, v1, v2, u[10], m[25]); + MONT_MAC(w0, w1, w2, u[11], m[24]); + MONT_MAC(t0, t1, t2, u[12], m[23]); + MONT_MAC(v0, v1, v2, u[13], m[22]); + MONT_MAC(w0, w1, w2, u[14], m[21]); + MONT_MAC(t0, t1, t2, u[15], m[20]); + MONT_MAC(v0, v1, v2, u[16], m[19]); + MONT_MAC(w0, w1, w2, u[17], m[18]); + MONT_MAC(t0, t1, t2, u[18], m[17]); + MONT_MAC(v0, v1, v2, u[19], m[16]); + MONT_MAC(w0, w1, w2, u[20], m[15]); + MONT_MAC(t0, t1, t2, u[21], m[14]); + MONT_MAC(v0, v1, v2, u[22], m[13]); + MONT_MAC(w0, w1, w2, u[23], m[12]); + MONT_MAC(t0, t1, t2, u[24], m[11]); + MONT_MAC(v0, v1, v2, u[25], m[10]); + MONT_MAC(w0, w1, w2, u[26], m[9]); + MONT_MAC(t0, t1, t2, u[27], m[8]); + MONT_MAC(v0, v1, v2, u[28], m[7]); + MONT_MAC(w0, w1, w2, u[29], m[6]); + MONT_MAC(t0, t1, t2, u[30], m[5]); + MONT_MAC(v0, v1, v2, u[31], m[4]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[3] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 36 */ + MONT_MAC(t0, t1, t2, a[5], b[31]); + MONT_MUL(v0, v1, v2, a[6], b[30]); + MONT_MUL(w0, w1, w2, a[7], b[29]); + MONT_MAC(t0, t1, t2, a[8], b[28]); + MONT_MAC(v0, v1, v2, a[9], b[27]); + MONT_MAC(w0, w1, w2, a[10], b[26]); + MONT_MAC(t0, t1, t2, a[11], b[25]); + MONT_MAC(v0, v1, v2, a[12], b[24]); + MONT_MAC(w0, w1, w2, a[13], b[23]); + MONT_MAC(t0, t1, t2, a[14], b[22]); + MONT_MAC(v0, v1, v2, a[15], b[21]); + MONT_MAC(w0, w1, w2, a[16], b[20]); + MONT_MAC(t0, t1, t2, a[17], b[19]); + MONT_MAC(v0, v1, v2, a[18], b[18]); + MONT_MAC(w0, w1, w2, a[19], b[17]); + MONT_MAC(t0, t1, t2, a[20], b[16]); + MONT_MAC(v0, v1, v2, a[21], b[15]); + MONT_MAC(w0, w1, w2, a[22], b[14]); + MONT_MAC(t0, t1, t2, a[23], b[13]); + MONT_MAC(v0, v1, v2, a[24], b[12]); + MONT_MAC(w0, w1, w2, a[25], b[11]); + MONT_MAC(t0, t1, t2, a[26], b[10]); + MONT_MAC(v0, v1, v2, a[27], b[9]); + MONT_MAC(w0, w1, w2, a[28], b[8]); + MONT_MAC(t0, t1, t2, a[29], b[7]); + MONT_MAC(v0, v1, v2, a[30], b[6]); + MONT_MAC(w0, w1, w2, a[31], b[5]); + MONT_MAC(t0, t1, t2, u[5], m[31]); + MONT_MAC(v0, v1, v2, u[6], m[30]); + MONT_MAC(w0, w1, w2, u[7], m[29]); + MONT_MAC(t0, t1, t2, u[8], m[28]); + MONT_MAC(v0, v1, v2, u[9], m[27]); + MONT_MAC(w0, w1, w2, u[10], m[26]); + MONT_MAC(t0, t1, t2, u[11], m[25]); + MONT_MAC(v0, v1, v2, u[12], m[24]); + MONT_MAC(w0, w1, w2, u[13], m[23]); + MONT_MAC(t0, t1, t2, u[14], m[22]); + MONT_MAC(v0, v1, v2, u[15], m[21]); + MONT_MAC(w0, w1, w2, u[16], m[20]); + MONT_MAC(t0, t1, t2, u[17], m[19]); + MONT_MAC(v0, v1, v2, u[18], m[18]); + MONT_MAC(w0, w1, w2, u[19], m[17]); + MONT_MAC(t0, t1, t2, u[20], m[16]); + MONT_MAC(v0, v1, v2, u[21], m[15]); + MONT_MAC(w0, w1, w2, u[22], m[14]); + MONT_MAC(t0, t1, t2, u[23], m[13]); + MONT_MAC(v0, v1, v2, u[24], m[12]); + MONT_MAC(w0, w1, w2, u[25], m[11]); + MONT_MAC(t0, t1, t2, u[26], m[10]); + MONT_MAC(v0, v1, v2, u[27], m[9]); + MONT_MAC(w0, w1, w2, u[28], m[8]); + MONT_MAC(t0, t1, t2, u[29], m[7]); + MONT_MAC(v0, v1, v2, u[30], m[6]); + MONT_MAC(w0, w1, w2, u[31], m[5]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[4] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 37 */ + MONT_MAC(t0, t1, t2, a[6], b[31]); + MONT_MUL(v0, v1, v2, a[7], b[30]); + MONT_MUL(w0, w1, w2, a[8], b[29]); + MONT_MAC(t0, t1, t2, a[9], b[28]); + MONT_MAC(v0, v1, v2, a[10], b[27]); + MONT_MAC(w0, w1, w2, a[11], b[26]); + MONT_MAC(t0, t1, t2, a[12], b[25]); + MONT_MAC(v0, v1, v2, a[13], b[24]); + MONT_MAC(w0, w1, w2, a[14], b[23]); + MONT_MAC(t0, t1, t2, a[15], b[22]); + MONT_MAC(v0, v1, v2, a[16], b[21]); + MONT_MAC(w0, w1, w2, a[17], b[20]); + MONT_MAC(t0, t1, t2, a[18], b[19]); + MONT_MAC(v0, v1, v2, a[19], b[18]); + MONT_MAC(w0, w1, w2, a[20], b[17]); + MONT_MAC(t0, t1, t2, a[21], b[16]); + MONT_MAC(v0, v1, v2, a[22], b[15]); + MONT_MAC(w0, w1, w2, a[23], b[14]); + MONT_MAC(t0, t1, t2, a[24], b[13]); + MONT_MAC(v0, v1, v2, a[25], b[12]); + MONT_MAC(w0, w1, w2, a[26], b[11]); + MONT_MAC(t0, t1, t2, a[27], b[10]); + MONT_MAC(v0, v1, v2, a[28], b[9]); + MONT_MAC(w0, w1, w2, a[29], b[8]); + MONT_MAC(t0, t1, t2, a[30], b[7]); + MONT_MAC(v0, v1, v2, a[31], b[6]); + MONT_MAC(w0, w1, w2, u[6], m[31]); + MONT_MAC(t0, t1, t2, u[7], m[30]); + MONT_MAC(v0, v1, v2, u[8], m[29]); + MONT_MAC(w0, w1, w2, u[9], m[28]); + MONT_MAC(t0, t1, t2, u[10], m[27]); + MONT_MAC(v0, v1, v2, u[11], m[26]); + MONT_MAC(w0, w1, w2, u[12], m[25]); + MONT_MAC(t0, t1, t2, u[13], m[24]); + MONT_MAC(v0, v1, v2, u[14], m[23]); + MONT_MAC(w0, w1, w2, u[15], m[22]); + MONT_MAC(t0, t1, t2, u[16], m[21]); + MONT_MAC(v0, v1, v2, u[17], m[20]); + MONT_MAC(w0, w1, w2, u[18], m[19]); + MONT_MAC(t0, t1, t2, u[19], m[18]); + MONT_MAC(v0, v1, v2, u[20], m[17]); + MONT_MAC(w0, w1, w2, u[21], m[16]); + MONT_MAC(t0, t1, t2, u[22], m[15]); + MONT_MAC(v0, v1, v2, u[23], m[14]); + MONT_MAC(w0, w1, w2, u[24], m[13]); + MONT_MAC(t0, t1, t2, u[25], m[12]); + MONT_MAC(v0, v1, v2, u[26], m[11]); + MONT_MAC(w0, w1, w2, u[27], m[10]); + MONT_MAC(t0, t1, t2, u[28], m[9]); + MONT_MAC(v0, v1, v2, u[29], m[8]); + MONT_MAC(w0, w1, w2, u[30], m[7]); + MONT_MAC(t0, t1, t2, u[31], m[6]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[5] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 38 */ + MONT_MAC(t0, t1, t2, a[7], b[31]); + MONT_MUL(v0, v1, v2, a[8], b[30]); + MONT_MUL(w0, w1, w2, a[9], b[29]); + MONT_MAC(t0, t1, t2, a[10], b[28]); + MONT_MAC(v0, v1, v2, a[11], b[27]); + MONT_MAC(w0, w1, w2, a[12], b[26]); + MONT_MAC(t0, t1, t2, a[13], b[25]); + MONT_MAC(v0, v1, v2, a[14], b[24]); + MONT_MAC(w0, w1, w2, a[15], b[23]); + MONT_MAC(t0, t1, t2, a[16], b[22]); + MONT_MAC(v0, v1, v2, a[17], b[21]); + MONT_MAC(w0, w1, w2, a[18], b[20]); + MONT_MAC(t0, t1, t2, a[19], b[19]); + MONT_MAC(v0, v1, v2, a[20], b[18]); + MONT_MAC(w0, w1, w2, a[21], b[17]); + MONT_MAC(t0, t1, t2, a[22], b[16]); + MONT_MAC(v0, v1, v2, a[23], b[15]); + MONT_MAC(w0, w1, w2, a[24], b[14]); + MONT_MAC(t0, t1, t2, a[25], b[13]); + MONT_MAC(v0, v1, v2, a[26], b[12]); + MONT_MAC(w0, w1, w2, a[27], b[11]); + MONT_MAC(t0, t1, t2, a[28], b[10]); + MONT_MAC(v0, v1, v2, a[29], b[9]); + MONT_MAC(w0, w1, w2, a[30], b[8]); + MONT_MAC(t0, t1, t2, a[31], b[7]); + MONT_MAC(v0, v1, v2, u[7], m[31]); + MONT_MAC(w0, w1, w2, u[8], m[30]); + MONT_MAC(t0, t1, t2, u[9], m[29]); + MONT_MAC(v0, v1, v2, u[10], m[28]); + MONT_MAC(w0, w1, w2, u[11], m[27]); + MONT_MAC(t0, t1, t2, u[12], m[26]); + MONT_MAC(v0, v1, v2, u[13], m[25]); + MONT_MAC(w0, w1, w2, u[14], m[24]); + MONT_MAC(t0, t1, t2, u[15], m[23]); + MONT_MAC(v0, v1, v2, u[16], m[22]); + MONT_MAC(w0, w1, w2, u[17], m[21]); + MONT_MAC(t0, t1, t2, u[18], m[20]); + MONT_MAC(v0, v1, v2, u[19], m[19]); + MONT_MAC(w0, w1, w2, u[20], m[18]); + MONT_MAC(t0, t1, t2, u[21], m[17]); + MONT_MAC(v0, v1, v2, u[22], m[16]); + MONT_MAC(w0, w1, w2, u[23], m[15]); + MONT_MAC(t0, t1, t2, u[24], m[14]); + MONT_MAC(v0, v1, v2, u[25], m[13]); + MONT_MAC(w0, w1, w2, u[26], m[12]); + MONT_MAC(t0, t1, t2, u[27], m[11]); + MONT_MAC(v0, v1, v2, u[28], m[10]); + MONT_MAC(w0, w1, w2, u[29], m[9]); + MONT_MAC(t0, t1, t2, u[30], m[8]); + MONT_MAC(v0, v1, v2, u[31], m[7]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[6] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 39 */ + MONT_MAC(t0, t1, t2, a[8], b[31]); + MONT_MUL(v0, v1, v2, a[9], b[30]); + MONT_MUL(w0, w1, w2, a[10], b[29]); + MONT_MAC(t0, t1, t2, a[11], b[28]); + MONT_MAC(v0, v1, v2, a[12], b[27]); + MONT_MAC(w0, w1, w2, a[13], b[26]); + MONT_MAC(t0, t1, t2, a[14], b[25]); + MONT_MAC(v0, v1, v2, a[15], b[24]); + MONT_MAC(w0, w1, w2, a[16], b[23]); + MONT_MAC(t0, t1, t2, a[17], b[22]); + MONT_MAC(v0, v1, v2, a[18], b[21]); + MONT_MAC(w0, w1, w2, a[19], b[20]); + MONT_MAC(t0, t1, t2, a[20], b[19]); + MONT_MAC(v0, v1, v2, a[21], b[18]); + MONT_MAC(w0, w1, w2, a[22], b[17]); + MONT_MAC(t0, t1, t2, a[23], b[16]); + MONT_MAC(v0, v1, v2, a[24], b[15]); + MONT_MAC(w0, w1, w2, a[25], b[14]); + MONT_MAC(t0, t1, t2, a[26], b[13]); + MONT_MAC(v0, v1, v2, a[27], b[12]); + MONT_MAC(w0, w1, w2, a[28], b[11]); + MONT_MAC(t0, t1, t2, a[29], b[10]); + MONT_MAC(v0, v1, v2, a[30], b[9]); + MONT_MAC(w0, w1, w2, a[31], b[8]); + MONT_MAC(t0, t1, t2, u[8], m[31]); + MONT_MAC(v0, v1, v2, u[9], m[30]); + MONT_MAC(w0, w1, w2, u[10], m[29]); + MONT_MAC(t0, t1, t2, u[11], m[28]); + MONT_MAC(v0, v1, v2, u[12], m[27]); + MONT_MAC(w0, w1, w2, u[13], m[26]); + MONT_MAC(t0, t1, t2, u[14], m[25]); + MONT_MAC(v0, v1, v2, u[15], m[24]); + MONT_MAC(w0, w1, w2, u[16], m[23]); + MONT_MAC(t0, t1, t2, u[17], m[22]); + MONT_MAC(v0, v1, v2, u[18], m[21]); + MONT_MAC(w0, w1, w2, u[19], m[20]); + MONT_MAC(t0, t1, t2, u[20], m[19]); + MONT_MAC(v0, v1, v2, u[21], m[18]); + MONT_MAC(w0, w1, w2, u[22], m[17]); + MONT_MAC(t0, t1, t2, u[23], m[16]); + MONT_MAC(v0, v1, v2, u[24], m[15]); + MONT_MAC(w0, w1, w2, u[25], m[14]); + MONT_MAC(t0, t1, t2, u[26], m[13]); + MONT_MAC(v0, v1, v2, u[27], m[12]); + MONT_MAC(w0, w1, w2, u[28], m[11]); + MONT_MAC(t0, t1, t2, u[29], m[10]); + MONT_MAC(v0, v1, v2, u[30], m[9]); + MONT_MAC(w0, w1, w2, u[31], m[8]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[7] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 40 */ + MONT_MAC(t0, t1, t2, a[9], b[31]); + MONT_MUL(v0, v1, v2, a[10], b[30]); + MONT_MUL(w0, w1, w2, a[11], b[29]); + MONT_MAC(t0, t1, t2, a[12], b[28]); + MONT_MAC(v0, v1, v2, a[13], b[27]); + MONT_MAC(w0, w1, w2, a[14], b[26]); + MONT_MAC(t0, t1, t2, a[15], b[25]); + MONT_MAC(v0, v1, v2, a[16], b[24]); + MONT_MAC(w0, w1, w2, a[17], b[23]); + MONT_MAC(t0, t1, t2, a[18], b[22]); + MONT_MAC(v0, v1, v2, a[19], b[21]); + MONT_MAC(w0, w1, w2, a[20], b[20]); + MONT_MAC(t0, t1, t2, a[21], b[19]); + MONT_MAC(v0, v1, v2, a[22], b[18]); + MONT_MAC(w0, w1, w2, a[23], b[17]); + MONT_MAC(t0, t1, t2, a[24], b[16]); + MONT_MAC(v0, v1, v2, a[25], b[15]); + MONT_MAC(w0, w1, w2, a[26], b[14]); + MONT_MAC(t0, t1, t2, a[27], b[13]); + MONT_MAC(v0, v1, v2, a[28], b[12]); + MONT_MAC(w0, w1, w2, a[29], b[11]); + MONT_MAC(t0, t1, t2, a[30], b[10]); + MONT_MAC(v0, v1, v2, a[31], b[9]); + MONT_MAC(w0, w1, w2, u[9], m[31]); + MONT_MAC(t0, t1, t2, u[10], m[30]); + MONT_MAC(v0, v1, v2, u[11], m[29]); + MONT_MAC(w0, w1, w2, u[12], m[28]); + MONT_MAC(t0, t1, t2, u[13], m[27]); + MONT_MAC(v0, v1, v2, u[14], m[26]); + MONT_MAC(w0, w1, w2, u[15], m[25]); + MONT_MAC(t0, t1, t2, u[16], m[24]); + MONT_MAC(v0, v1, v2, u[17], m[23]); + MONT_MAC(w0, w1, w2, u[18], m[22]); + MONT_MAC(t0, t1, t2, u[19], m[21]); + MONT_MAC(v0, v1, v2, u[20], m[20]); + MONT_MAC(w0, w1, w2, u[21], m[19]); + MONT_MAC(t0, t1, t2, u[22], m[18]); + MONT_MAC(v0, v1, v2, u[23], m[17]); + MONT_MAC(w0, w1, w2, u[24], m[16]); + MONT_MAC(t0, t1, t2, u[25], m[15]); + MONT_MAC(v0, v1, v2, u[26], m[14]); + MONT_MAC(w0, w1, w2, u[27], m[13]); + MONT_MAC(t0, t1, t2, u[28], m[12]); + MONT_MAC(v0, v1, v2, u[29], m[11]); + MONT_MAC(w0, w1, w2, u[30], m[10]); + MONT_MAC(t0, t1, t2, u[31], m[9]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[8] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 41 */ + MONT_MAC(t0, t1, t2, a[10], b[31]); + MONT_MUL(v0, v1, v2, a[11], b[30]); + MONT_MUL(w0, w1, w2, a[12], b[29]); + MONT_MAC(t0, t1, t2, a[13], b[28]); + MONT_MAC(v0, v1, v2, a[14], b[27]); + MONT_MAC(w0, w1, w2, a[15], b[26]); + MONT_MAC(t0, t1, t2, a[16], b[25]); + MONT_MAC(v0, v1, v2, a[17], b[24]); + MONT_MAC(w0, w1, w2, a[18], b[23]); + MONT_MAC(t0, t1, t2, a[19], b[22]); + MONT_MAC(v0, v1, v2, a[20], b[21]); + MONT_MAC(w0, w1, w2, a[21], b[20]); + MONT_MAC(t0, t1, t2, a[22], b[19]); + MONT_MAC(v0, v1, v2, a[23], b[18]); + MONT_MAC(w0, w1, w2, a[24], b[17]); + MONT_MAC(t0, t1, t2, a[25], b[16]); + MONT_MAC(v0, v1, v2, a[26], b[15]); + MONT_MAC(w0, w1, w2, a[27], b[14]); + MONT_MAC(t0, t1, t2, a[28], b[13]); + MONT_MAC(v0, v1, v2, a[29], b[12]); + MONT_MAC(w0, w1, w2, a[30], b[11]); + MONT_MAC(t0, t1, t2, a[31], b[10]); + MONT_MAC(v0, v1, v2, u[10], m[31]); + MONT_MAC(w0, w1, w2, u[11], m[30]); + MONT_MAC(t0, t1, t2, u[12], m[29]); + MONT_MAC(v0, v1, v2, u[13], m[28]); + MONT_MAC(w0, w1, w2, u[14], m[27]); + MONT_MAC(t0, t1, t2, u[15], m[26]); + MONT_MAC(v0, v1, v2, u[16], m[25]); + MONT_MAC(w0, w1, w2, u[17], m[24]); + MONT_MAC(t0, t1, t2, u[18], m[23]); + MONT_MAC(v0, v1, v2, u[19], m[22]); + MONT_MAC(w0, w1, w2, u[20], m[21]); + MONT_MAC(t0, t1, t2, u[21], m[20]); + MONT_MAC(v0, v1, v2, u[22], m[19]); + MONT_MAC(w0, w1, w2, u[23], m[18]); + MONT_MAC(t0, t1, t2, u[24], m[17]); + MONT_MAC(v0, v1, v2, u[25], m[16]); + MONT_MAC(w0, w1, w2, u[26], m[15]); + MONT_MAC(t0, t1, t2, u[27], m[14]); + MONT_MAC(v0, v1, v2, u[28], m[13]); + MONT_MAC(w0, w1, w2, u[29], m[12]); + MONT_MAC(t0, t1, t2, u[30], m[11]); + MONT_MAC(v0, v1, v2, u[31], m[10]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[9] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 42 */ + MONT_MAC(t0, t1, t2, a[11], b[31]); + MONT_MUL(v0, v1, v2, a[12], b[30]); + MONT_MUL(w0, w1, w2, a[13], b[29]); + MONT_MAC(t0, t1, t2, a[14], b[28]); + MONT_MAC(v0, v1, v2, a[15], b[27]); + MONT_MAC(w0, w1, w2, a[16], b[26]); + MONT_MAC(t0, t1, t2, a[17], b[25]); + MONT_MAC(v0, v1, v2, a[18], b[24]); + MONT_MAC(w0, w1, w2, a[19], b[23]); + MONT_MAC(t0, t1, t2, a[20], b[22]); + MONT_MAC(v0, v1, v2, a[21], b[21]); + MONT_MAC(w0, w1, w2, a[22], b[20]); + MONT_MAC(t0, t1, t2, a[23], b[19]); + MONT_MAC(v0, v1, v2, a[24], b[18]); + MONT_MAC(w0, w1, w2, a[25], b[17]); + MONT_MAC(t0, t1, t2, a[26], b[16]); + MONT_MAC(v0, v1, v2, a[27], b[15]); + MONT_MAC(w0, w1, w2, a[28], b[14]); + MONT_MAC(t0, t1, t2, a[29], b[13]); + MONT_MAC(v0, v1, v2, a[30], b[12]); + MONT_MAC(w0, w1, w2, a[31], b[11]); + MONT_MAC(t0, t1, t2, u[11], m[31]); + MONT_MAC(v0, v1, v2, u[12], m[30]); + MONT_MAC(w0, w1, w2, u[13], m[29]); + MONT_MAC(t0, t1, t2, u[14], m[28]); + MONT_MAC(v0, v1, v2, u[15], m[27]); + MONT_MAC(w0, w1, w2, u[16], m[26]); + MONT_MAC(t0, t1, t2, u[17], m[25]); + MONT_MAC(v0, v1, v2, u[18], m[24]); + MONT_MAC(w0, w1, w2, u[19], m[23]); + MONT_MAC(t0, t1, t2, u[20], m[22]); + MONT_MAC(v0, v1, v2, u[21], m[21]); + MONT_MAC(w0, w1, w2, u[22], m[20]); + MONT_MAC(t0, t1, t2, u[23], m[19]); + MONT_MAC(v0, v1, v2, u[24], m[18]); + MONT_MAC(w0, w1, w2, u[25], m[17]); + MONT_MAC(t0, t1, t2, u[26], m[16]); + MONT_MAC(v0, v1, v2, u[27], m[15]); + MONT_MAC(w0, w1, w2, u[28], m[14]); + MONT_MAC(t0, t1, t2, u[29], m[13]); + MONT_MAC(v0, v1, v2, u[30], m[12]); + MONT_MAC(w0, w1, w2, u[31], m[11]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[10] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 43 */ + MONT_MAC(t0, t1, t2, a[12], b[31]); + MONT_MUL(v0, v1, v2, a[13], b[30]); + MONT_MUL(w0, w1, w2, a[14], b[29]); + MONT_MAC(t0, t1, t2, a[15], b[28]); + MONT_MAC(v0, v1, v2, a[16], b[27]); + MONT_MAC(w0, w1, w2, a[17], b[26]); + MONT_MAC(t0, t1, t2, a[18], b[25]); + MONT_MAC(v0, v1, v2, a[19], b[24]); + MONT_MAC(w0, w1, w2, a[20], b[23]); + MONT_MAC(t0, t1, t2, a[21], b[22]); + MONT_MAC(v0, v1, v2, a[22], b[21]); + MONT_MAC(w0, w1, w2, a[23], b[20]); + MONT_MAC(t0, t1, t2, a[24], b[19]); + MONT_MAC(v0, v1, v2, a[25], b[18]); + MONT_MAC(w0, w1, w2, a[26], b[17]); + MONT_MAC(t0, t1, t2, a[27], b[16]); + MONT_MAC(v0, v1, v2, a[28], b[15]); + MONT_MAC(w0, w1, w2, a[29], b[14]); + MONT_MAC(t0, t1, t2, a[30], b[13]); + MONT_MAC(v0, v1, v2, a[31], b[12]); + MONT_MAC(w0, w1, w2, u[12], m[31]); + MONT_MAC(t0, t1, t2, u[13], m[30]); + MONT_MAC(v0, v1, v2, u[14], m[29]); + MONT_MAC(w0, w1, w2, u[15], m[28]); + MONT_MAC(t0, t1, t2, u[16], m[27]); + MONT_MAC(v0, v1, v2, u[17], m[26]); + MONT_MAC(w0, w1, w2, u[18], m[25]); + MONT_MAC(t0, t1, t2, u[19], m[24]); + MONT_MAC(v0, v1, v2, u[20], m[23]); + MONT_MAC(w0, w1, w2, u[21], m[22]); + MONT_MAC(t0, t1, t2, u[22], m[21]); + MONT_MAC(v0, v1, v2, u[23], m[20]); + MONT_MAC(w0, w1, w2, u[24], m[19]); + MONT_MAC(t0, t1, t2, u[25], m[18]); + MONT_MAC(v0, v1, v2, u[26], m[17]); + MONT_MAC(w0, w1, w2, u[27], m[16]); + MONT_MAC(t0, t1, t2, u[28], m[15]); + MONT_MAC(v0, v1, v2, u[29], m[14]); + MONT_MAC(w0, w1, w2, u[30], m[13]); + MONT_MAC(t0, t1, t2, u[31], m[12]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[11] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 44 */ + MONT_MAC(t0, t1, t2, a[13], b[31]); + MONT_MUL(v0, v1, v2, a[14], b[30]); + MONT_MUL(w0, w1, w2, a[15], b[29]); + MONT_MAC(t0, t1, t2, a[16], b[28]); + MONT_MAC(v0, v1, v2, a[17], b[27]); + MONT_MAC(w0, w1, w2, a[18], b[26]); + MONT_MAC(t0, t1, t2, a[19], b[25]); + MONT_MAC(v0, v1, v2, a[20], b[24]); + MONT_MAC(w0, w1, w2, a[21], b[23]); + MONT_MAC(t0, t1, t2, a[22], b[22]); + MONT_MAC(v0, v1, v2, a[23], b[21]); + MONT_MAC(w0, w1, w2, a[24], b[20]); + MONT_MAC(t0, t1, t2, a[25], b[19]); + MONT_MAC(v0, v1, v2, a[26], b[18]); + MONT_MAC(w0, w1, w2, a[27], b[17]); + MONT_MAC(t0, t1, t2, a[28], b[16]); + MONT_MAC(v0, v1, v2, a[29], b[15]); + MONT_MAC(w0, w1, w2, a[30], b[14]); + MONT_MAC(t0, t1, t2, a[31], b[13]); + MONT_MAC(v0, v1, v2, u[13], m[31]); + MONT_MAC(w0, w1, w2, u[14], m[30]); + MONT_MAC(t0, t1, t2, u[15], m[29]); + MONT_MAC(v0, v1, v2, u[16], m[28]); + MONT_MAC(w0, w1, w2, u[17], m[27]); + MONT_MAC(t0, t1, t2, u[18], m[26]); + MONT_MAC(v0, v1, v2, u[19], m[25]); + MONT_MAC(w0, w1, w2, u[20], m[24]); + MONT_MAC(t0, t1, t2, u[21], m[23]); + MONT_MAC(v0, v1, v2, u[22], m[22]); + MONT_MAC(w0, w1, w2, u[23], m[21]); + MONT_MAC(t0, t1, t2, u[24], m[20]); + MONT_MAC(v0, v1, v2, u[25], m[19]); + MONT_MAC(w0, w1, w2, u[26], m[18]); + MONT_MAC(t0, t1, t2, u[27], m[17]); + MONT_MAC(v0, v1, v2, u[28], m[16]); + MONT_MAC(w0, w1, w2, u[29], m[15]); + MONT_MAC(t0, t1, t2, u[30], m[14]); + MONT_MAC(v0, v1, v2, u[31], m[13]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[12] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 45 */ + MONT_MAC(t0, t1, t2, a[14], b[31]); + MONT_MUL(v0, v1, v2, a[15], b[30]); + MONT_MUL(w0, w1, w2, a[16], b[29]); + MONT_MAC(t0, t1, t2, a[17], b[28]); + MONT_MAC(v0, v1, v2, a[18], b[27]); + MONT_MAC(w0, w1, w2, a[19], b[26]); + MONT_MAC(t0, t1, t2, a[20], b[25]); + MONT_MAC(v0, v1, v2, a[21], b[24]); + MONT_MAC(w0, w1, w2, a[22], b[23]); + MONT_MAC(t0, t1, t2, a[23], b[22]); + MONT_MAC(v0, v1, v2, a[24], b[21]); + MONT_MAC(w0, w1, w2, a[25], b[20]); + MONT_MAC(t0, t1, t2, a[26], b[19]); + MONT_MAC(v0, v1, v2, a[27], b[18]); + MONT_MAC(w0, w1, w2, a[28], b[17]); + MONT_MAC(t0, t1, t2, a[29], b[16]); + MONT_MAC(v0, v1, v2, a[30], b[15]); + MONT_MAC(w0, w1, w2, a[31], b[14]); + MONT_MAC(t0, t1, t2, u[14], m[31]); + MONT_MAC(v0, v1, v2, u[15], m[30]); + MONT_MAC(w0, w1, w2, u[16], m[29]); + MONT_MAC(t0, t1, t2, u[17], m[28]); + MONT_MAC(v0, v1, v2, u[18], m[27]); + MONT_MAC(w0, w1, w2, u[19], m[26]); + MONT_MAC(t0, t1, t2, u[20], m[25]); + MONT_MAC(v0, v1, v2, u[21], m[24]); + MONT_MAC(w0, w1, w2, u[22], m[23]); + MONT_MAC(t0, t1, t2, u[23], m[22]); + MONT_MAC(v0, v1, v2, u[24], m[21]); + MONT_MAC(w0, w1, w2, u[25], m[20]); + MONT_MAC(t0, t1, t2, u[26], m[19]); + MONT_MAC(v0, v1, v2, u[27], m[18]); + MONT_MAC(w0, w1, w2, u[28], m[17]); + MONT_MAC(t0, t1, t2, u[29], m[16]); + MONT_MAC(v0, v1, v2, u[30], m[15]); + MONT_MAC(w0, w1, w2, u[31], m[14]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[13] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 46 */ + MONT_MAC(t0, t1, t2, a[15], b[31]); + MONT_MUL(v0, v1, v2, a[16], b[30]); + MONT_MUL(w0, w1, w2, a[17], b[29]); + MONT_MAC(t0, t1, t2, a[18], b[28]); + MONT_MAC(v0, v1, v2, a[19], b[27]); + MONT_MAC(w0, w1, w2, a[20], b[26]); + MONT_MAC(t0, t1, t2, a[21], b[25]); + MONT_MAC(v0, v1, v2, a[22], b[24]); + MONT_MAC(w0, w1, w2, a[23], b[23]); + MONT_MAC(t0, t1, t2, a[24], b[22]); + MONT_MAC(v0, v1, v2, a[25], b[21]); + MONT_MAC(w0, w1, w2, a[26], b[20]); + MONT_MAC(t0, t1, t2, a[27], b[19]); + MONT_MAC(v0, v1, v2, a[28], b[18]); + MONT_MAC(w0, w1, w2, a[29], b[17]); + MONT_MAC(t0, t1, t2, a[30], b[16]); + MONT_MAC(v0, v1, v2, a[31], b[15]); + MONT_MAC(w0, w1, w2, u[15], m[31]); + MONT_MAC(t0, t1, t2, u[16], m[30]); + MONT_MAC(v0, v1, v2, u[17], m[29]); + MONT_MAC(w0, w1, w2, u[18], m[28]); + MONT_MAC(t0, t1, t2, u[19], m[27]); + MONT_MAC(v0, v1, v2, u[20], m[26]); + MONT_MAC(w0, w1, w2, u[21], m[25]); + MONT_MAC(t0, t1, t2, u[22], m[24]); + MONT_MAC(v0, v1, v2, u[23], m[23]); + MONT_MAC(w0, w1, w2, u[24], m[22]); + MONT_MAC(t0, t1, t2, u[25], m[21]); + MONT_MAC(v0, v1, v2, u[26], m[20]); + MONT_MAC(w0, w1, w2, u[27], m[19]); + MONT_MAC(t0, t1, t2, u[28], m[18]); + MONT_MAC(v0, v1, v2, u[29], m[17]); + MONT_MAC(w0, w1, w2, u[30], m[16]); + MONT_MAC(t0, t1, t2, u[31], m[15]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[14] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 47 */ + MONT_MAC(t0, t1, t2, a[16], b[31]); + MONT_MUL(v0, v1, v2, a[17], b[30]); + MONT_MUL(w0, w1, w2, a[18], b[29]); + MONT_MAC(t0, t1, t2, a[19], b[28]); + MONT_MAC(v0, v1, v2, a[20], b[27]); + MONT_MAC(w0, w1, w2, a[21], b[26]); + MONT_MAC(t0, t1, t2, a[22], b[25]); + MONT_MAC(v0, v1, v2, a[23], b[24]); + MONT_MAC(w0, w1, w2, a[24], b[23]); + MONT_MAC(t0, t1, t2, a[25], b[22]); + MONT_MAC(v0, v1, v2, a[26], b[21]); + MONT_MAC(w0, w1, w2, a[27], b[20]); + MONT_MAC(t0, t1, t2, a[28], b[19]); + MONT_MAC(v0, v1, v2, a[29], b[18]); + MONT_MAC(w0, w1, w2, a[30], b[17]); + MONT_MAC(t0, t1, t2, a[31], b[16]); + MONT_MAC(v0, v1, v2, u[16], m[31]); + MONT_MAC(w0, w1, w2, u[17], m[30]); + MONT_MAC(t0, t1, t2, u[18], m[29]); + MONT_MAC(v0, v1, v2, u[19], m[28]); + MONT_MAC(w0, w1, w2, u[20], m[27]); + MONT_MAC(t0, t1, t2, u[21], m[26]); + MONT_MAC(v0, v1, v2, u[22], m[25]); + MONT_MAC(w0, w1, w2, u[23], m[24]); + MONT_MAC(t0, t1, t2, u[24], m[23]); + MONT_MAC(v0, v1, v2, u[25], m[22]); + MONT_MAC(w0, w1, w2, u[26], m[21]); + MONT_MAC(t0, t1, t2, u[27], m[20]); + MONT_MAC(v0, v1, v2, u[28], m[19]); + MONT_MAC(w0, w1, w2, u[29], m[18]); + MONT_MAC(t0, t1, t2, u[30], m[17]); + MONT_MAC(v0, v1, v2, u[31], m[16]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[15] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 48 */ + MONT_MAC(t0, t1, t2, a[17], b[31]); + MONT_MUL(v0, v1, v2, a[18], b[30]); + MONT_MUL(w0, w1, w2, a[19], b[29]); + MONT_MAC(t0, t1, t2, a[20], b[28]); + MONT_MAC(v0, v1, v2, a[21], b[27]); + MONT_MAC(w0, w1, w2, a[22], b[26]); + MONT_MAC(t0, t1, t2, a[23], b[25]); + MONT_MAC(v0, v1, v2, a[24], b[24]); + MONT_MAC(w0, w1, w2, a[25], b[23]); + MONT_MAC(t0, t1, t2, a[26], b[22]); + MONT_MAC(v0, v1, v2, a[27], b[21]); + MONT_MAC(w0, w1, w2, a[28], b[20]); + MONT_MAC(t0, t1, t2, a[29], b[19]); + MONT_MAC(v0, v1, v2, a[30], b[18]); + MONT_MAC(w0, w1, w2, a[31], b[17]); + MONT_MAC(t0, t1, t2, u[17], m[31]); + MONT_MAC(v0, v1, v2, u[18], m[30]); + MONT_MAC(w0, w1, w2, u[19], m[29]); + MONT_MAC(t0, t1, t2, u[20], m[28]); + MONT_MAC(v0, v1, v2, u[21], m[27]); + MONT_MAC(w0, w1, w2, u[22], m[26]); + MONT_MAC(t0, t1, t2, u[23], m[25]); + MONT_MAC(v0, v1, v2, u[24], m[24]); + MONT_MAC(w0, w1, w2, u[25], m[23]); + MONT_MAC(t0, t1, t2, u[26], m[22]); + MONT_MAC(v0, v1, v2, u[27], m[21]); + MONT_MAC(w0, w1, w2, u[28], m[20]); + MONT_MAC(t0, t1, t2, u[29], m[19]); + MONT_MAC(v0, v1, v2, u[30], m[18]); + MONT_MAC(w0, w1, w2, u[31], m[17]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[16] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 49 */ + MONT_MAC(t0, t1, t2, a[18], b[31]); + MONT_MUL(v0, v1, v2, a[19], b[30]); + MONT_MUL(w0, w1, w2, a[20], b[29]); + MONT_MAC(t0, t1, t2, a[21], b[28]); + MONT_MAC(v0, v1, v2, a[22], b[27]); + MONT_MAC(w0, w1, w2, a[23], b[26]); + MONT_MAC(t0, t1, t2, a[24], b[25]); + MONT_MAC(v0, v1, v2, a[25], b[24]); + MONT_MAC(w0, w1, w2, a[26], b[23]); + MONT_MAC(t0, t1, t2, a[27], b[22]); + MONT_MAC(v0, v1, v2, a[28], b[21]); + MONT_MAC(w0, w1, w2, a[29], b[20]); + MONT_MAC(t0, t1, t2, a[30], b[19]); + MONT_MAC(v0, v1, v2, a[31], b[18]); + MONT_MAC(w0, w1, w2, u[18], m[31]); + MONT_MAC(t0, t1, t2, u[19], m[30]); + MONT_MAC(v0, v1, v2, u[20], m[29]); + MONT_MAC(w0, w1, w2, u[21], m[28]); + MONT_MAC(t0, t1, t2, u[22], m[27]); + MONT_MAC(v0, v1, v2, u[23], m[26]); + MONT_MAC(w0, w1, w2, u[24], m[25]); + MONT_MAC(t0, t1, t2, u[25], m[24]); + MONT_MAC(v0, v1, v2, u[26], m[23]); + MONT_MAC(w0, w1, w2, u[27], m[22]); + MONT_MAC(t0, t1, t2, u[28], m[21]); + MONT_MAC(v0, v1, v2, u[29], m[20]); + MONT_MAC(w0, w1, w2, u[30], m[19]); + MONT_MAC(t0, t1, t2, u[31], m[18]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[17] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 50 */ + MONT_MAC(t0, t1, t2, a[19], b[31]); + MONT_MUL(v0, v1, v2, a[20], b[30]); + MONT_MUL(w0, w1, w2, a[21], b[29]); + MONT_MAC(t0, t1, t2, a[22], b[28]); + MONT_MAC(v0, v1, v2, a[23], b[27]); + MONT_MAC(w0, w1, w2, a[24], b[26]); + MONT_MAC(t0, t1, t2, a[25], b[25]); + MONT_MAC(v0, v1, v2, a[26], b[24]); + MONT_MAC(w0, w1, w2, a[27], b[23]); + MONT_MAC(t0, t1, t2, a[28], b[22]); + MONT_MAC(v0, v1, v2, a[29], b[21]); + MONT_MAC(w0, w1, w2, a[30], b[20]); + MONT_MAC(t0, t1, t2, a[31], b[19]); + MONT_MAC(v0, v1, v2, u[19], m[31]); + MONT_MAC(w0, w1, w2, u[20], m[30]); + MONT_MAC(t0, t1, t2, u[21], m[29]); + MONT_MAC(v0, v1, v2, u[22], m[28]); + MONT_MAC(w0, w1, w2, u[23], m[27]); + MONT_MAC(t0, t1, t2, u[24], m[26]); + MONT_MAC(v0, v1, v2, u[25], m[25]); + MONT_MAC(w0, w1, w2, u[26], m[24]); + MONT_MAC(t0, t1, t2, u[27], m[23]); + MONT_MAC(v0, v1, v2, u[28], m[22]); + MONT_MAC(w0, w1, w2, u[29], m[21]); + MONT_MAC(t0, t1, t2, u[30], m[20]); + MONT_MAC(v0, v1, v2, u[31], m[19]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[18] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 51 */ + MONT_MAC(t0, t1, t2, a[20], b[31]); + MONT_MUL(v0, v1, v2, a[21], b[30]); + MONT_MUL(w0, w1, w2, a[22], b[29]); + MONT_MAC(t0, t1, t2, a[23], b[28]); + MONT_MAC(v0, v1, v2, a[24], b[27]); + MONT_MAC(w0, w1, w2, a[25], b[26]); + MONT_MAC(t0, t1, t2, a[26], b[25]); + MONT_MAC(v0, v1, v2, a[27], b[24]); + MONT_MAC(w0, w1, w2, a[28], b[23]); + MONT_MAC(t0, t1, t2, a[29], b[22]); + MONT_MAC(v0, v1, v2, a[30], b[21]); + MONT_MAC(w0, w1, w2, a[31], b[20]); + MONT_MAC(t0, t1, t2, u[20], m[31]); + MONT_MAC(v0, v1, v2, u[21], m[30]); + MONT_MAC(w0, w1, w2, u[22], m[29]); + MONT_MAC(t0, t1, t2, u[23], m[28]); + MONT_MAC(v0, v1, v2, u[24], m[27]); + MONT_MAC(w0, w1, w2, u[25], m[26]); + MONT_MAC(t0, t1, t2, u[26], m[25]); + MONT_MAC(v0, v1, v2, u[27], m[24]); + MONT_MAC(w0, w1, w2, u[28], m[23]); + MONT_MAC(t0, t1, t2, u[29], m[22]); + MONT_MAC(v0, v1, v2, u[30], m[21]); + MONT_MAC(w0, w1, w2, u[31], m[20]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[19] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 52 */ + MONT_MAC(t0, t1, t2, a[21], b[31]); + MONT_MUL(v0, v1, v2, a[22], b[30]); + MONT_MUL(w0, w1, w2, a[23], b[29]); + MONT_MAC(t0, t1, t2, a[24], b[28]); + MONT_MAC(v0, v1, v2, a[25], b[27]); + MONT_MAC(w0, w1, w2, a[26], b[26]); + MONT_MAC(t0, t1, t2, a[27], b[25]); + MONT_MAC(v0, v1, v2, a[28], b[24]); + MONT_MAC(w0, w1, w2, a[29], b[23]); + MONT_MAC(t0, t1, t2, a[30], b[22]); + MONT_MAC(v0, v1, v2, a[31], b[21]); + MONT_MAC(w0, w1, w2, u[21], m[31]); + MONT_MAC(t0, t1, t2, u[22], m[30]); + MONT_MAC(v0, v1, v2, u[23], m[29]); + MONT_MAC(w0, w1, w2, u[24], m[28]); + MONT_MAC(t0, t1, t2, u[25], m[27]); + MONT_MAC(v0, v1, v2, u[26], m[26]); + MONT_MAC(w0, w1, w2, u[27], m[25]); + MONT_MAC(t0, t1, t2, u[28], m[24]); + MONT_MAC(v0, v1, v2, u[29], m[23]); + MONT_MAC(w0, w1, w2, u[30], m[22]); + MONT_MAC(t0, t1, t2, u[31], m[21]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[20] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 53 */ + MONT_MAC(t0, t1, t2, a[22], b[31]); + MONT_MUL(v0, v1, v2, a[23], b[30]); + MONT_MUL(w0, w1, w2, a[24], b[29]); + MONT_MAC(t0, t1, t2, a[25], b[28]); + MONT_MAC(v0, v1, v2, a[26], b[27]); + MONT_MAC(w0, w1, w2, a[27], b[26]); + MONT_MAC(t0, t1, t2, a[28], b[25]); + MONT_MAC(v0, v1, v2, a[29], b[24]); + MONT_MAC(w0, w1, w2, a[30], b[23]); + MONT_MAC(t0, t1, t2, a[31], b[22]); + MONT_MAC(v0, v1, v2, u[22], m[31]); + MONT_MAC(w0, w1, w2, u[23], m[30]); + MONT_MAC(t0, t1, t2, u[24], m[29]); + MONT_MAC(v0, v1, v2, u[25], m[28]); + MONT_MAC(w0, w1, w2, u[26], m[27]); + MONT_MAC(t0, t1, t2, u[27], m[26]); + MONT_MAC(v0, v1, v2, u[28], m[25]); + MONT_MAC(w0, w1, w2, u[29], m[24]); + MONT_MAC(t0, t1, t2, u[30], m[23]); + MONT_MAC(v0, v1, v2, u[31], m[22]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[21] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 54 */ + MONT_MAC(t0, t1, t2, a[23], b[31]); + MONT_MUL(v0, v1, v2, a[24], b[30]); + MONT_MUL(w0, w1, w2, a[25], b[29]); + MONT_MAC(t0, t1, t2, a[26], b[28]); + MONT_MAC(v0, v1, v2, a[27], b[27]); + MONT_MAC(w0, w1, w2, a[28], b[26]); + MONT_MAC(t0, t1, t2, a[29], b[25]); + MONT_MAC(v0, v1, v2, a[30], b[24]); + MONT_MAC(w0, w1, w2, a[31], b[23]); + MONT_MAC(t0, t1, t2, u[23], m[31]); + MONT_MAC(v0, v1, v2, u[24], m[30]); + MONT_MAC(w0, w1, w2, u[25], m[29]); + MONT_MAC(t0, t1, t2, u[26], m[28]); + MONT_MAC(v0, v1, v2, u[27], m[27]); + MONT_MAC(w0, w1, w2, u[28], m[26]); + MONT_MAC(t0, t1, t2, u[29], m[25]); + MONT_MAC(v0, v1, v2, u[30], m[24]); + MONT_MAC(w0, w1, w2, u[31], m[23]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[22] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 55 */ + MONT_MAC(t0, t1, t2, a[24], b[31]); + MONT_MUL(v0, v1, v2, a[25], b[30]); + MONT_MUL(w0, w1, w2, a[26], b[29]); + MONT_MAC(t0, t1, t2, a[27], b[28]); + MONT_MAC(v0, v1, v2, a[28], b[27]); + MONT_MAC(w0, w1, w2, a[29], b[26]); + MONT_MAC(t0, t1, t2, a[30], b[25]); + MONT_MAC(v0, v1, v2, a[31], b[24]); + MONT_MAC(w0, w1, w2, u[24], m[31]); + MONT_MAC(t0, t1, t2, u[25], m[30]); + MONT_MAC(v0, v1, v2, u[26], m[29]); + MONT_MAC(w0, w1, w2, u[27], m[28]); + MONT_MAC(t0, t1, t2, u[28], m[27]); + MONT_MAC(v0, v1, v2, u[29], m[26]); + MONT_MAC(w0, w1, w2, u[30], m[25]); + MONT_MAC(t0, t1, t2, u[31], m[24]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[23] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 56 */ + MONT_MAC(t0, t1, t2, a[25], b[31]); + MONT_MUL(v0, v1, v2, a[26], b[30]); + MONT_MUL(w0, w1, w2, a[27], b[29]); + MONT_MAC(t0, t1, t2, a[28], b[28]); + MONT_MAC(v0, v1, v2, a[29], b[27]); + MONT_MAC(w0, w1, w2, a[30], b[26]); + MONT_MAC(t0, t1, t2, a[31], b[25]); + MONT_MAC(v0, v1, v2, u[25], m[31]); + MONT_MAC(w0, w1, w2, u[26], m[30]); + MONT_MAC(t0, t1, t2, u[27], m[29]); + MONT_MAC(v0, v1, v2, u[28], m[28]); + MONT_MAC(w0, w1, w2, u[29], m[27]); + MONT_MAC(t0, t1, t2, u[30], m[26]); + MONT_MAC(v0, v1, v2, u[31], m[25]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[24] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 57 */ + MONT_MAC(t0, t1, t2, a[26], b[31]); + MONT_MUL(v0, v1, v2, a[27], b[30]); + MONT_MUL(w0, w1, w2, a[28], b[29]); + MONT_MAC(t0, t1, t2, a[29], b[28]); + MONT_MAC(v0, v1, v2, a[30], b[27]); + MONT_MAC(w0, w1, w2, a[31], b[26]); + MONT_MAC(t0, t1, t2, u[26], m[31]); + MONT_MAC(v0, v1, v2, u[27], m[30]); + MONT_MAC(w0, w1, w2, u[28], m[29]); + MONT_MAC(t0, t1, t2, u[29], m[28]); + MONT_MAC(v0, v1, v2, u[30], m[27]); + MONT_MAC(w0, w1, w2, u[31], m[26]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[25] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 58 */ + MONT_MAC(t0, t1, t2, a[27], b[31]); + MONT_MUL(v0, v1, v2, a[28], b[30]); + MONT_MUL(w0, w1, w2, a[29], b[29]); + MONT_MAC(t0, t1, t2, a[30], b[28]); + MONT_MAC(v0, v1, v2, a[31], b[27]); + MONT_MAC(w0, w1, w2, u[27], m[31]); + MONT_MAC(t0, t1, t2, u[28], m[30]); + MONT_MAC(v0, v1, v2, u[29], m[29]); + MONT_MAC(w0, w1, w2, u[30], m[28]); + MONT_MAC(t0, t1, t2, u[31], m[27]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[26] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 59 */ + MONT_MAC(t0, t1, t2, a[28], b[31]); + MONT_MUL(v0, v1, v2, a[29], b[30]); + MONT_MUL(w0, w1, w2, a[30], b[29]); + MONT_MAC(t0, t1, t2, a[31], b[28]); + MONT_MAC(v0, v1, v2, u[28], m[31]); + MONT_MAC(w0, w1, w2, u[29], m[30]); + MONT_MAC(t0, t1, t2, u[30], m[29]); + MONT_MAC(v0, v1, v2, u[31], m[28]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[27] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 60 */ + MONT_MAC(t0, t1, t2, a[29], b[31]); + MONT_MUL(v0, v1, v2, a[30], b[30]); + MONT_MUL(w0, w1, w2, a[31], b[29]); + MONT_MAC(t0, t1, t2, u[29], m[31]); + MONT_MAC(v0, v1, v2, u[30], m[30]); + MONT_MAC(w0, w1, w2, u[31], m[29]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[28] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 61 */ + MONT_MAC(t0, t1, t2, a[30], b[31]); + MONT_MUL(v0, v1, v2, a[31], b[30]); + MONT_MUL(w0, w1, w2, u[30], m[31]); + MONT_MAC(t0, t1, t2, u[31], m[30]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[29] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 62 */ + MONT_MAC(t0, t1, t2, a[31], b[31]); + MONT_MUL(v0, v1, v2, u[31], m[31]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + z[30] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + z[31] = t0; + + /* under 2m, with t1 the bit above n limbs: at most one subtraction */ + borrow = sub_n(r, z, m, 32); + take = t1 | (borrow ^ 1); + select_n(r, r, z, take & 1, 32); +} + +/* r = a * a * R^-1 mod m, for a under m; t is 2n limbs of scratch */ +static void crypton_mont_sqr32(limb_t *r, const limb_t *a, + const limb_t *m, limb_t n0, limb_t *t) +{ + limb_t *u = t, *z = t + 32; + limb_t t0 = 0, t1 = 0, t2 = 0, v0, v1, v2, w0, w1, w2, borrow, take; + limb_t s0, s1, s2, c0, c1, c2; + + /* column 0 */ + MONT_MAC(t0, t1, t2, a[0], a[0]); + u[0] = t0 * n0; + MONT_MAC(t0, t1, t2, u[0], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 1 */ + MONT_MUL(s0, s1, s2, a[0], a[1]); + MONT_MAC(t0, t1, t2, u[0], m[1]); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[1] = t0 * n0; + MONT_MAC(t0, t1, t2, u[1], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 2 */ + MONT_MUL(s0, s1, s2, a[0], a[2]); + MONT_MAC(t0, t1, t2, a[1], a[1]); + MONT_MUL(v0, v1, v2, u[0], m[2]); + MONT_MUL(w0, w1, w2, u[1], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[2] = t0 * n0; + MONT_MAC(t0, t1, t2, u[2], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 3 */ + MONT_MUL(s0, s1, s2, a[0], a[3]); + MONT_MUL(c0, c1, c2, a[1], a[2]); + MONT_MAC(t0, t1, t2, u[0], m[3]); + MONT_MUL(v0, v1, v2, u[1], m[2]); + MONT_MUL(w0, w1, w2, u[2], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[3] = t0 * n0; + MONT_MAC(t0, t1, t2, u[3], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 4 */ + MONT_MUL(s0, s1, s2, a[0], a[4]); + MONT_MUL(c0, c1, c2, a[1], a[3]); + MONT_MAC(t0, t1, t2, a[2], a[2]); + MONT_MUL(v0, v1, v2, u[0], m[4]); + MONT_MUL(w0, w1, w2, u[1], m[3]); + MONT_MAC(t0, t1, t2, u[2], m[2]); + MONT_MAC(v0, v1, v2, u[3], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[4] = t0 * n0; + MONT_MAC(t0, t1, t2, u[4], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 5 */ + MONT_MUL(s0, s1, s2, a[0], a[5]); + MONT_MUL(c0, c1, c2, a[1], a[4]); + MONT_MAC(s0, s1, s2, a[2], a[3]); + MONT_MAC(t0, t1, t2, u[0], m[5]); + MONT_MUL(v0, v1, v2, u[1], m[4]); + MONT_MUL(w0, w1, w2, u[2], m[3]); + MONT_MAC(t0, t1, t2, u[3], m[2]); + MONT_MAC(v0, v1, v2, u[4], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[5] = t0 * n0; + MONT_MAC(t0, t1, t2, u[5], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 6 */ + MONT_MUL(s0, s1, s2, a[0], a[6]); + MONT_MUL(c0, c1, c2, a[1], a[5]); + MONT_MAC(s0, s1, s2, a[2], a[4]); + MONT_MAC(t0, t1, t2, a[3], a[3]); + MONT_MUL(v0, v1, v2, u[0], m[6]); + MONT_MUL(w0, w1, w2, u[1], m[5]); + MONT_MAC(t0, t1, t2, u[2], m[4]); + MONT_MAC(v0, v1, v2, u[3], m[3]); + MONT_MAC(w0, w1, w2, u[4], m[2]); + MONT_MAC(t0, t1, t2, u[5], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[6] = t0 * n0; + MONT_MAC(t0, t1, t2, u[6], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 7 */ + MONT_MUL(s0, s1, s2, a[0], a[7]); + MONT_MUL(c0, c1, c2, a[1], a[6]); + MONT_MAC(s0, s1, s2, a[2], a[5]); + MONT_MAC(c0, c1, c2, a[3], a[4]); + MONT_MAC(t0, t1, t2, u[0], m[7]); + MONT_MUL(v0, v1, v2, u[1], m[6]); + MONT_MUL(w0, w1, w2, u[2], m[5]); + MONT_MAC(t0, t1, t2, u[3], m[4]); + MONT_MAC(v0, v1, v2, u[4], m[3]); + MONT_MAC(w0, w1, w2, u[5], m[2]); + MONT_MAC(t0, t1, t2, u[6], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[7] = t0 * n0; + MONT_MAC(t0, t1, t2, u[7], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 8 */ + MONT_MUL(s0, s1, s2, a[0], a[8]); + MONT_MUL(c0, c1, c2, a[1], a[7]); + MONT_MAC(s0, s1, s2, a[2], a[6]); + MONT_MAC(c0, c1, c2, a[3], a[5]); + MONT_MAC(t0, t1, t2, a[4], a[4]); + MONT_MUL(v0, v1, v2, u[0], m[8]); + MONT_MUL(w0, w1, w2, u[1], m[7]); + MONT_MAC(t0, t1, t2, u[2], m[6]); + MONT_MAC(v0, v1, v2, u[3], m[5]); + MONT_MAC(w0, w1, w2, u[4], m[4]); + MONT_MAC(t0, t1, t2, u[5], m[3]); + MONT_MAC(v0, v1, v2, u[6], m[2]); + MONT_MAC(w0, w1, w2, u[7], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[8] = t0 * n0; + MONT_MAC(t0, t1, t2, u[8], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 9 */ + MONT_MUL(s0, s1, s2, a[0], a[9]); + MONT_MUL(c0, c1, c2, a[1], a[8]); + MONT_MAC(s0, s1, s2, a[2], a[7]); + MONT_MAC(c0, c1, c2, a[3], a[6]); + MONT_MAC(s0, s1, s2, a[4], a[5]); + MONT_MAC(t0, t1, t2, u[0], m[9]); + MONT_MUL(v0, v1, v2, u[1], m[8]); + MONT_MUL(w0, w1, w2, u[2], m[7]); + MONT_MAC(t0, t1, t2, u[3], m[6]); + MONT_MAC(v0, v1, v2, u[4], m[5]); + MONT_MAC(w0, w1, w2, u[5], m[4]); + MONT_MAC(t0, t1, t2, u[6], m[3]); + MONT_MAC(v0, v1, v2, u[7], m[2]); + MONT_MAC(w0, w1, w2, u[8], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[9] = t0 * n0; + MONT_MAC(t0, t1, t2, u[9], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 10 */ + MONT_MUL(s0, s1, s2, a[0], a[10]); + MONT_MUL(c0, c1, c2, a[1], a[9]); + MONT_MAC(s0, s1, s2, a[2], a[8]); + MONT_MAC(c0, c1, c2, a[3], a[7]); + MONT_MAC(s0, s1, s2, a[4], a[6]); + MONT_MAC(t0, t1, t2, a[5], a[5]); + MONT_MUL(v0, v1, v2, u[0], m[10]); + MONT_MUL(w0, w1, w2, u[1], m[9]); + MONT_MAC(t0, t1, t2, u[2], m[8]); + MONT_MAC(v0, v1, v2, u[3], m[7]); + MONT_MAC(w0, w1, w2, u[4], m[6]); + MONT_MAC(t0, t1, t2, u[5], m[5]); + MONT_MAC(v0, v1, v2, u[6], m[4]); + MONT_MAC(w0, w1, w2, u[7], m[3]); + MONT_MAC(t0, t1, t2, u[8], m[2]); + MONT_MAC(v0, v1, v2, u[9], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[10] = t0 * n0; + MONT_MAC(t0, t1, t2, u[10], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 11 */ + MONT_MUL(s0, s1, s2, a[0], a[11]); + MONT_MUL(c0, c1, c2, a[1], a[10]); + MONT_MAC(s0, s1, s2, a[2], a[9]); + MONT_MAC(c0, c1, c2, a[3], a[8]); + MONT_MAC(s0, s1, s2, a[4], a[7]); + MONT_MAC(c0, c1, c2, a[5], a[6]); + MONT_MAC(t0, t1, t2, u[0], m[11]); + MONT_MUL(v0, v1, v2, u[1], m[10]); + MONT_MUL(w0, w1, w2, u[2], m[9]); + MONT_MAC(t0, t1, t2, u[3], m[8]); + MONT_MAC(v0, v1, v2, u[4], m[7]); + MONT_MAC(w0, w1, w2, u[5], m[6]); + MONT_MAC(t0, t1, t2, u[6], m[5]); + MONT_MAC(v0, v1, v2, u[7], m[4]); + MONT_MAC(w0, w1, w2, u[8], m[3]); + MONT_MAC(t0, t1, t2, u[9], m[2]); + MONT_MAC(v0, v1, v2, u[10], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[11] = t0 * n0; + MONT_MAC(t0, t1, t2, u[11], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 12 */ + MONT_MUL(s0, s1, s2, a[0], a[12]); + MONT_MUL(c0, c1, c2, a[1], a[11]); + MONT_MAC(s0, s1, s2, a[2], a[10]); + MONT_MAC(c0, c1, c2, a[3], a[9]); + MONT_MAC(s0, s1, s2, a[4], a[8]); + MONT_MAC(c0, c1, c2, a[5], a[7]); + MONT_MAC(t0, t1, t2, a[6], a[6]); + MONT_MUL(v0, v1, v2, u[0], m[12]); + MONT_MUL(w0, w1, w2, u[1], m[11]); + MONT_MAC(t0, t1, t2, u[2], m[10]); + MONT_MAC(v0, v1, v2, u[3], m[9]); + MONT_MAC(w0, w1, w2, u[4], m[8]); + MONT_MAC(t0, t1, t2, u[5], m[7]); + MONT_MAC(v0, v1, v2, u[6], m[6]); + MONT_MAC(w0, w1, w2, u[7], m[5]); + MONT_MAC(t0, t1, t2, u[8], m[4]); + MONT_MAC(v0, v1, v2, u[9], m[3]); + MONT_MAC(w0, w1, w2, u[10], m[2]); + MONT_MAC(t0, t1, t2, u[11], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[12] = t0 * n0; + MONT_MAC(t0, t1, t2, u[12], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 13 */ + MONT_MUL(s0, s1, s2, a[0], a[13]); + MONT_MUL(c0, c1, c2, a[1], a[12]); + MONT_MAC(s0, s1, s2, a[2], a[11]); + MONT_MAC(c0, c1, c2, a[3], a[10]); + MONT_MAC(s0, s1, s2, a[4], a[9]); + MONT_MAC(c0, c1, c2, a[5], a[8]); + MONT_MAC(s0, s1, s2, a[6], a[7]); + MONT_MAC(t0, t1, t2, u[0], m[13]); + MONT_MUL(v0, v1, v2, u[1], m[12]); + MONT_MUL(w0, w1, w2, u[2], m[11]); + MONT_MAC(t0, t1, t2, u[3], m[10]); + MONT_MAC(v0, v1, v2, u[4], m[9]); + MONT_MAC(w0, w1, w2, u[5], m[8]); + MONT_MAC(t0, t1, t2, u[6], m[7]); + MONT_MAC(v0, v1, v2, u[7], m[6]); + MONT_MAC(w0, w1, w2, u[8], m[5]); + MONT_MAC(t0, t1, t2, u[9], m[4]); + MONT_MAC(v0, v1, v2, u[10], m[3]); + MONT_MAC(w0, w1, w2, u[11], m[2]); + MONT_MAC(t0, t1, t2, u[12], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[13] = t0 * n0; + MONT_MAC(t0, t1, t2, u[13], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 14 */ + MONT_MUL(s0, s1, s2, a[0], a[14]); + MONT_MUL(c0, c1, c2, a[1], a[13]); + MONT_MAC(s0, s1, s2, a[2], a[12]); + MONT_MAC(c0, c1, c2, a[3], a[11]); + MONT_MAC(s0, s1, s2, a[4], a[10]); + MONT_MAC(c0, c1, c2, a[5], a[9]); + MONT_MAC(s0, s1, s2, a[6], a[8]); + MONT_MAC(t0, t1, t2, a[7], a[7]); + MONT_MUL(v0, v1, v2, u[0], m[14]); + MONT_MUL(w0, w1, w2, u[1], m[13]); + MONT_MAC(t0, t1, t2, u[2], m[12]); + MONT_MAC(v0, v1, v2, u[3], m[11]); + MONT_MAC(w0, w1, w2, u[4], m[10]); + MONT_MAC(t0, t1, t2, u[5], m[9]); + MONT_MAC(v0, v1, v2, u[6], m[8]); + MONT_MAC(w0, w1, w2, u[7], m[7]); + MONT_MAC(t0, t1, t2, u[8], m[6]); + MONT_MAC(v0, v1, v2, u[9], m[5]); + MONT_MAC(w0, w1, w2, u[10], m[4]); + MONT_MAC(t0, t1, t2, u[11], m[3]); + MONT_MAC(v0, v1, v2, u[12], m[2]); + MONT_MAC(w0, w1, w2, u[13], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[14] = t0 * n0; + MONT_MAC(t0, t1, t2, u[14], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 15 */ + MONT_MUL(s0, s1, s2, a[0], a[15]); + MONT_MUL(c0, c1, c2, a[1], a[14]); + MONT_MAC(s0, s1, s2, a[2], a[13]); + MONT_MAC(c0, c1, c2, a[3], a[12]); + MONT_MAC(s0, s1, s2, a[4], a[11]); + MONT_MAC(c0, c1, c2, a[5], a[10]); + MONT_MAC(s0, s1, s2, a[6], a[9]); + MONT_MAC(c0, c1, c2, a[7], a[8]); + MONT_MAC(t0, t1, t2, u[0], m[15]); + MONT_MUL(v0, v1, v2, u[1], m[14]); + MONT_MUL(w0, w1, w2, u[2], m[13]); + MONT_MAC(t0, t1, t2, u[3], m[12]); + MONT_MAC(v0, v1, v2, u[4], m[11]); + MONT_MAC(w0, w1, w2, u[5], m[10]); + MONT_MAC(t0, t1, t2, u[6], m[9]); + MONT_MAC(v0, v1, v2, u[7], m[8]); + MONT_MAC(w0, w1, w2, u[8], m[7]); + MONT_MAC(t0, t1, t2, u[9], m[6]); + MONT_MAC(v0, v1, v2, u[10], m[5]); + MONT_MAC(w0, w1, w2, u[11], m[4]); + MONT_MAC(t0, t1, t2, u[12], m[3]); + MONT_MAC(v0, v1, v2, u[13], m[2]); + MONT_MAC(w0, w1, w2, u[14], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[15] = t0 * n0; + MONT_MAC(t0, t1, t2, u[15], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 16 */ + MONT_MUL(s0, s1, s2, a[0], a[16]); + MONT_MUL(c0, c1, c2, a[1], a[15]); + MONT_MAC(s0, s1, s2, a[2], a[14]); + MONT_MAC(c0, c1, c2, a[3], a[13]); + MONT_MAC(s0, s1, s2, a[4], a[12]); + MONT_MAC(c0, c1, c2, a[5], a[11]); + MONT_MAC(s0, s1, s2, a[6], a[10]); + MONT_MAC(c0, c1, c2, a[7], a[9]); + MONT_MAC(t0, t1, t2, a[8], a[8]); + MONT_MUL(v0, v1, v2, u[0], m[16]); + MONT_MUL(w0, w1, w2, u[1], m[15]); + MONT_MAC(t0, t1, t2, u[2], m[14]); + MONT_MAC(v0, v1, v2, u[3], m[13]); + MONT_MAC(w0, w1, w2, u[4], m[12]); + MONT_MAC(t0, t1, t2, u[5], m[11]); + MONT_MAC(v0, v1, v2, u[6], m[10]); + MONT_MAC(w0, w1, w2, u[7], m[9]); + MONT_MAC(t0, t1, t2, u[8], m[8]); + MONT_MAC(v0, v1, v2, u[9], m[7]); + MONT_MAC(w0, w1, w2, u[10], m[6]); + MONT_MAC(t0, t1, t2, u[11], m[5]); + MONT_MAC(v0, v1, v2, u[12], m[4]); + MONT_MAC(w0, w1, w2, u[13], m[3]); + MONT_MAC(t0, t1, t2, u[14], m[2]); + MONT_MAC(v0, v1, v2, u[15], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[16] = t0 * n0; + MONT_MAC(t0, t1, t2, u[16], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 17 */ + MONT_MUL(s0, s1, s2, a[0], a[17]); + MONT_MUL(c0, c1, c2, a[1], a[16]); + MONT_MAC(s0, s1, s2, a[2], a[15]); + MONT_MAC(c0, c1, c2, a[3], a[14]); + MONT_MAC(s0, s1, s2, a[4], a[13]); + MONT_MAC(c0, c1, c2, a[5], a[12]); + MONT_MAC(s0, s1, s2, a[6], a[11]); + MONT_MAC(c0, c1, c2, a[7], a[10]); + MONT_MAC(s0, s1, s2, a[8], a[9]); + MONT_MAC(t0, t1, t2, u[0], m[17]); + MONT_MUL(v0, v1, v2, u[1], m[16]); + MONT_MUL(w0, w1, w2, u[2], m[15]); + MONT_MAC(t0, t1, t2, u[3], m[14]); + MONT_MAC(v0, v1, v2, u[4], m[13]); + MONT_MAC(w0, w1, w2, u[5], m[12]); + MONT_MAC(t0, t1, t2, u[6], m[11]); + MONT_MAC(v0, v1, v2, u[7], m[10]); + MONT_MAC(w0, w1, w2, u[8], m[9]); + MONT_MAC(t0, t1, t2, u[9], m[8]); + MONT_MAC(v0, v1, v2, u[10], m[7]); + MONT_MAC(w0, w1, w2, u[11], m[6]); + MONT_MAC(t0, t1, t2, u[12], m[5]); + MONT_MAC(v0, v1, v2, u[13], m[4]); + MONT_MAC(w0, w1, w2, u[14], m[3]); + MONT_MAC(t0, t1, t2, u[15], m[2]); + MONT_MAC(v0, v1, v2, u[16], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[17] = t0 * n0; + MONT_MAC(t0, t1, t2, u[17], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 18 */ + MONT_MUL(s0, s1, s2, a[0], a[18]); + MONT_MUL(c0, c1, c2, a[1], a[17]); + MONT_MAC(s0, s1, s2, a[2], a[16]); + MONT_MAC(c0, c1, c2, a[3], a[15]); + MONT_MAC(s0, s1, s2, a[4], a[14]); + MONT_MAC(c0, c1, c2, a[5], a[13]); + MONT_MAC(s0, s1, s2, a[6], a[12]); + MONT_MAC(c0, c1, c2, a[7], a[11]); + MONT_MAC(s0, s1, s2, a[8], a[10]); + MONT_MAC(t0, t1, t2, a[9], a[9]); + MONT_MUL(v0, v1, v2, u[0], m[18]); + MONT_MUL(w0, w1, w2, u[1], m[17]); + MONT_MAC(t0, t1, t2, u[2], m[16]); + MONT_MAC(v0, v1, v2, u[3], m[15]); + MONT_MAC(w0, w1, w2, u[4], m[14]); + MONT_MAC(t0, t1, t2, u[5], m[13]); + MONT_MAC(v0, v1, v2, u[6], m[12]); + MONT_MAC(w0, w1, w2, u[7], m[11]); + MONT_MAC(t0, t1, t2, u[8], m[10]); + MONT_MAC(v0, v1, v2, u[9], m[9]); + MONT_MAC(w0, w1, w2, u[10], m[8]); + MONT_MAC(t0, t1, t2, u[11], m[7]); + MONT_MAC(v0, v1, v2, u[12], m[6]); + MONT_MAC(w0, w1, w2, u[13], m[5]); + MONT_MAC(t0, t1, t2, u[14], m[4]); + MONT_MAC(v0, v1, v2, u[15], m[3]); + MONT_MAC(w0, w1, w2, u[16], m[2]); + MONT_MAC(t0, t1, t2, u[17], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[18] = t0 * n0; + MONT_MAC(t0, t1, t2, u[18], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 19 */ + MONT_MUL(s0, s1, s2, a[0], a[19]); + MONT_MUL(c0, c1, c2, a[1], a[18]); + MONT_MAC(s0, s1, s2, a[2], a[17]); + MONT_MAC(c0, c1, c2, a[3], a[16]); + MONT_MAC(s0, s1, s2, a[4], a[15]); + MONT_MAC(c0, c1, c2, a[5], a[14]); + MONT_MAC(s0, s1, s2, a[6], a[13]); + MONT_MAC(c0, c1, c2, a[7], a[12]); + MONT_MAC(s0, s1, s2, a[8], a[11]); + MONT_MAC(c0, c1, c2, a[9], a[10]); + MONT_MAC(t0, t1, t2, u[0], m[19]); + MONT_MUL(v0, v1, v2, u[1], m[18]); + MONT_MUL(w0, w1, w2, u[2], m[17]); + MONT_MAC(t0, t1, t2, u[3], m[16]); + MONT_MAC(v0, v1, v2, u[4], m[15]); + MONT_MAC(w0, w1, w2, u[5], m[14]); + MONT_MAC(t0, t1, t2, u[6], m[13]); + MONT_MAC(v0, v1, v2, u[7], m[12]); + MONT_MAC(w0, w1, w2, u[8], m[11]); + MONT_MAC(t0, t1, t2, u[9], m[10]); + MONT_MAC(v0, v1, v2, u[10], m[9]); + MONT_MAC(w0, w1, w2, u[11], m[8]); + MONT_MAC(t0, t1, t2, u[12], m[7]); + MONT_MAC(v0, v1, v2, u[13], m[6]); + MONT_MAC(w0, w1, w2, u[14], m[5]); + MONT_MAC(t0, t1, t2, u[15], m[4]); + MONT_MAC(v0, v1, v2, u[16], m[3]); + MONT_MAC(w0, w1, w2, u[17], m[2]); + MONT_MAC(t0, t1, t2, u[18], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[19] = t0 * n0; + MONT_MAC(t0, t1, t2, u[19], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 20 */ + MONT_MUL(s0, s1, s2, a[0], a[20]); + MONT_MUL(c0, c1, c2, a[1], a[19]); + MONT_MAC(s0, s1, s2, a[2], a[18]); + MONT_MAC(c0, c1, c2, a[3], a[17]); + MONT_MAC(s0, s1, s2, a[4], a[16]); + MONT_MAC(c0, c1, c2, a[5], a[15]); + MONT_MAC(s0, s1, s2, a[6], a[14]); + MONT_MAC(c0, c1, c2, a[7], a[13]); + MONT_MAC(s0, s1, s2, a[8], a[12]); + MONT_MAC(c0, c1, c2, a[9], a[11]); + MONT_MAC(t0, t1, t2, a[10], a[10]); + MONT_MUL(v0, v1, v2, u[0], m[20]); + MONT_MUL(w0, w1, w2, u[1], m[19]); + MONT_MAC(t0, t1, t2, u[2], m[18]); + MONT_MAC(v0, v1, v2, u[3], m[17]); + MONT_MAC(w0, w1, w2, u[4], m[16]); + MONT_MAC(t0, t1, t2, u[5], m[15]); + MONT_MAC(v0, v1, v2, u[6], m[14]); + MONT_MAC(w0, w1, w2, u[7], m[13]); + MONT_MAC(t0, t1, t2, u[8], m[12]); + MONT_MAC(v0, v1, v2, u[9], m[11]); + MONT_MAC(w0, w1, w2, u[10], m[10]); + MONT_MAC(t0, t1, t2, u[11], m[9]); + MONT_MAC(v0, v1, v2, u[12], m[8]); + MONT_MAC(w0, w1, w2, u[13], m[7]); + MONT_MAC(t0, t1, t2, u[14], m[6]); + MONT_MAC(v0, v1, v2, u[15], m[5]); + MONT_MAC(w0, w1, w2, u[16], m[4]); + MONT_MAC(t0, t1, t2, u[17], m[3]); + MONT_MAC(v0, v1, v2, u[18], m[2]); + MONT_MAC(w0, w1, w2, u[19], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[20] = t0 * n0; + MONT_MAC(t0, t1, t2, u[20], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 21 */ + MONT_MUL(s0, s1, s2, a[0], a[21]); + MONT_MUL(c0, c1, c2, a[1], a[20]); + MONT_MAC(s0, s1, s2, a[2], a[19]); + MONT_MAC(c0, c1, c2, a[3], a[18]); + MONT_MAC(s0, s1, s2, a[4], a[17]); + MONT_MAC(c0, c1, c2, a[5], a[16]); + MONT_MAC(s0, s1, s2, a[6], a[15]); + MONT_MAC(c0, c1, c2, a[7], a[14]); + MONT_MAC(s0, s1, s2, a[8], a[13]); + MONT_MAC(c0, c1, c2, a[9], a[12]); + MONT_MAC(s0, s1, s2, a[10], a[11]); + MONT_MAC(t0, t1, t2, u[0], m[21]); + MONT_MUL(v0, v1, v2, u[1], m[20]); + MONT_MUL(w0, w1, w2, u[2], m[19]); + MONT_MAC(t0, t1, t2, u[3], m[18]); + MONT_MAC(v0, v1, v2, u[4], m[17]); + MONT_MAC(w0, w1, w2, u[5], m[16]); + MONT_MAC(t0, t1, t2, u[6], m[15]); + MONT_MAC(v0, v1, v2, u[7], m[14]); + MONT_MAC(w0, w1, w2, u[8], m[13]); + MONT_MAC(t0, t1, t2, u[9], m[12]); + MONT_MAC(v0, v1, v2, u[10], m[11]); + MONT_MAC(w0, w1, w2, u[11], m[10]); + MONT_MAC(t0, t1, t2, u[12], m[9]); + MONT_MAC(v0, v1, v2, u[13], m[8]); + MONT_MAC(w0, w1, w2, u[14], m[7]); + MONT_MAC(t0, t1, t2, u[15], m[6]); + MONT_MAC(v0, v1, v2, u[16], m[5]); + MONT_MAC(w0, w1, w2, u[17], m[4]); + MONT_MAC(t0, t1, t2, u[18], m[3]); + MONT_MAC(v0, v1, v2, u[19], m[2]); + MONT_MAC(w0, w1, w2, u[20], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[21] = t0 * n0; + MONT_MAC(t0, t1, t2, u[21], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 22 */ + MONT_MUL(s0, s1, s2, a[0], a[22]); + MONT_MUL(c0, c1, c2, a[1], a[21]); + MONT_MAC(s0, s1, s2, a[2], a[20]); + MONT_MAC(c0, c1, c2, a[3], a[19]); + MONT_MAC(s0, s1, s2, a[4], a[18]); + MONT_MAC(c0, c1, c2, a[5], a[17]); + MONT_MAC(s0, s1, s2, a[6], a[16]); + MONT_MAC(c0, c1, c2, a[7], a[15]); + MONT_MAC(s0, s1, s2, a[8], a[14]); + MONT_MAC(c0, c1, c2, a[9], a[13]); + MONT_MAC(s0, s1, s2, a[10], a[12]); + MONT_MAC(t0, t1, t2, a[11], a[11]); + MONT_MUL(v0, v1, v2, u[0], m[22]); + MONT_MUL(w0, w1, w2, u[1], m[21]); + MONT_MAC(t0, t1, t2, u[2], m[20]); + MONT_MAC(v0, v1, v2, u[3], m[19]); + MONT_MAC(w0, w1, w2, u[4], m[18]); + MONT_MAC(t0, t1, t2, u[5], m[17]); + MONT_MAC(v0, v1, v2, u[6], m[16]); + MONT_MAC(w0, w1, w2, u[7], m[15]); + MONT_MAC(t0, t1, t2, u[8], m[14]); + MONT_MAC(v0, v1, v2, u[9], m[13]); + MONT_MAC(w0, w1, w2, u[10], m[12]); + MONT_MAC(t0, t1, t2, u[11], m[11]); + MONT_MAC(v0, v1, v2, u[12], m[10]); + MONT_MAC(w0, w1, w2, u[13], m[9]); + MONT_MAC(t0, t1, t2, u[14], m[8]); + MONT_MAC(v0, v1, v2, u[15], m[7]); + MONT_MAC(w0, w1, w2, u[16], m[6]); + MONT_MAC(t0, t1, t2, u[17], m[5]); + MONT_MAC(v0, v1, v2, u[18], m[4]); + MONT_MAC(w0, w1, w2, u[19], m[3]); + MONT_MAC(t0, t1, t2, u[20], m[2]); + MONT_MAC(v0, v1, v2, u[21], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[22] = t0 * n0; + MONT_MAC(t0, t1, t2, u[22], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 23 */ + MONT_MUL(s0, s1, s2, a[0], a[23]); + MONT_MUL(c0, c1, c2, a[1], a[22]); + MONT_MAC(s0, s1, s2, a[2], a[21]); + MONT_MAC(c0, c1, c2, a[3], a[20]); + MONT_MAC(s0, s1, s2, a[4], a[19]); + MONT_MAC(c0, c1, c2, a[5], a[18]); + MONT_MAC(s0, s1, s2, a[6], a[17]); + MONT_MAC(c0, c1, c2, a[7], a[16]); + MONT_MAC(s0, s1, s2, a[8], a[15]); + MONT_MAC(c0, c1, c2, a[9], a[14]); + MONT_MAC(s0, s1, s2, a[10], a[13]); + MONT_MAC(c0, c1, c2, a[11], a[12]); + MONT_MAC(t0, t1, t2, u[0], m[23]); + MONT_MUL(v0, v1, v2, u[1], m[22]); + MONT_MUL(w0, w1, w2, u[2], m[21]); + MONT_MAC(t0, t1, t2, u[3], m[20]); + MONT_MAC(v0, v1, v2, u[4], m[19]); + MONT_MAC(w0, w1, w2, u[5], m[18]); + MONT_MAC(t0, t1, t2, u[6], m[17]); + MONT_MAC(v0, v1, v2, u[7], m[16]); + MONT_MAC(w0, w1, w2, u[8], m[15]); + MONT_MAC(t0, t1, t2, u[9], m[14]); + MONT_MAC(v0, v1, v2, u[10], m[13]); + MONT_MAC(w0, w1, w2, u[11], m[12]); + MONT_MAC(t0, t1, t2, u[12], m[11]); + MONT_MAC(v0, v1, v2, u[13], m[10]); + MONT_MAC(w0, w1, w2, u[14], m[9]); + MONT_MAC(t0, t1, t2, u[15], m[8]); + MONT_MAC(v0, v1, v2, u[16], m[7]); + MONT_MAC(w0, w1, w2, u[17], m[6]); + MONT_MAC(t0, t1, t2, u[18], m[5]); + MONT_MAC(v0, v1, v2, u[19], m[4]); + MONT_MAC(w0, w1, w2, u[20], m[3]); + MONT_MAC(t0, t1, t2, u[21], m[2]); + MONT_MAC(v0, v1, v2, u[22], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[23] = t0 * n0; + MONT_MAC(t0, t1, t2, u[23], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 24 */ + MONT_MUL(s0, s1, s2, a[0], a[24]); + MONT_MUL(c0, c1, c2, a[1], a[23]); + MONT_MAC(s0, s1, s2, a[2], a[22]); + MONT_MAC(c0, c1, c2, a[3], a[21]); + MONT_MAC(s0, s1, s2, a[4], a[20]); + MONT_MAC(c0, c1, c2, a[5], a[19]); + MONT_MAC(s0, s1, s2, a[6], a[18]); + MONT_MAC(c0, c1, c2, a[7], a[17]); + MONT_MAC(s0, s1, s2, a[8], a[16]); + MONT_MAC(c0, c1, c2, a[9], a[15]); + MONT_MAC(s0, s1, s2, a[10], a[14]); + MONT_MAC(c0, c1, c2, a[11], a[13]); + MONT_MAC(t0, t1, t2, a[12], a[12]); + MONT_MUL(v0, v1, v2, u[0], m[24]); + MONT_MUL(w0, w1, w2, u[1], m[23]); + MONT_MAC(t0, t1, t2, u[2], m[22]); + MONT_MAC(v0, v1, v2, u[3], m[21]); + MONT_MAC(w0, w1, w2, u[4], m[20]); + MONT_MAC(t0, t1, t2, u[5], m[19]); + MONT_MAC(v0, v1, v2, u[6], m[18]); + MONT_MAC(w0, w1, w2, u[7], m[17]); + MONT_MAC(t0, t1, t2, u[8], m[16]); + MONT_MAC(v0, v1, v2, u[9], m[15]); + MONT_MAC(w0, w1, w2, u[10], m[14]); + MONT_MAC(t0, t1, t2, u[11], m[13]); + MONT_MAC(v0, v1, v2, u[12], m[12]); + MONT_MAC(w0, w1, w2, u[13], m[11]); + MONT_MAC(t0, t1, t2, u[14], m[10]); + MONT_MAC(v0, v1, v2, u[15], m[9]); + MONT_MAC(w0, w1, w2, u[16], m[8]); + MONT_MAC(t0, t1, t2, u[17], m[7]); + MONT_MAC(v0, v1, v2, u[18], m[6]); + MONT_MAC(w0, w1, w2, u[19], m[5]); + MONT_MAC(t0, t1, t2, u[20], m[4]); + MONT_MAC(v0, v1, v2, u[21], m[3]); + MONT_MAC(w0, w1, w2, u[22], m[2]); + MONT_MAC(t0, t1, t2, u[23], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[24] = t0 * n0; + MONT_MAC(t0, t1, t2, u[24], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 25 */ + MONT_MUL(s0, s1, s2, a[0], a[25]); + MONT_MUL(c0, c1, c2, a[1], a[24]); + MONT_MAC(s0, s1, s2, a[2], a[23]); + MONT_MAC(c0, c1, c2, a[3], a[22]); + MONT_MAC(s0, s1, s2, a[4], a[21]); + MONT_MAC(c0, c1, c2, a[5], a[20]); + MONT_MAC(s0, s1, s2, a[6], a[19]); + MONT_MAC(c0, c1, c2, a[7], a[18]); + MONT_MAC(s0, s1, s2, a[8], a[17]); + MONT_MAC(c0, c1, c2, a[9], a[16]); + MONT_MAC(s0, s1, s2, a[10], a[15]); + MONT_MAC(c0, c1, c2, a[11], a[14]); + MONT_MAC(s0, s1, s2, a[12], a[13]); + MONT_MAC(t0, t1, t2, u[0], m[25]); + MONT_MUL(v0, v1, v2, u[1], m[24]); + MONT_MUL(w0, w1, w2, u[2], m[23]); + MONT_MAC(t0, t1, t2, u[3], m[22]); + MONT_MAC(v0, v1, v2, u[4], m[21]); + MONT_MAC(w0, w1, w2, u[5], m[20]); + MONT_MAC(t0, t1, t2, u[6], m[19]); + MONT_MAC(v0, v1, v2, u[7], m[18]); + MONT_MAC(w0, w1, w2, u[8], m[17]); + MONT_MAC(t0, t1, t2, u[9], m[16]); + MONT_MAC(v0, v1, v2, u[10], m[15]); + MONT_MAC(w0, w1, w2, u[11], m[14]); + MONT_MAC(t0, t1, t2, u[12], m[13]); + MONT_MAC(v0, v1, v2, u[13], m[12]); + MONT_MAC(w0, w1, w2, u[14], m[11]); + MONT_MAC(t0, t1, t2, u[15], m[10]); + MONT_MAC(v0, v1, v2, u[16], m[9]); + MONT_MAC(w0, w1, w2, u[17], m[8]); + MONT_MAC(t0, t1, t2, u[18], m[7]); + MONT_MAC(v0, v1, v2, u[19], m[6]); + MONT_MAC(w0, w1, w2, u[20], m[5]); + MONT_MAC(t0, t1, t2, u[21], m[4]); + MONT_MAC(v0, v1, v2, u[22], m[3]); + MONT_MAC(w0, w1, w2, u[23], m[2]); + MONT_MAC(t0, t1, t2, u[24], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[25] = t0 * n0; + MONT_MAC(t0, t1, t2, u[25], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 26 */ + MONT_MUL(s0, s1, s2, a[0], a[26]); + MONT_MUL(c0, c1, c2, a[1], a[25]); + MONT_MAC(s0, s1, s2, a[2], a[24]); + MONT_MAC(c0, c1, c2, a[3], a[23]); + MONT_MAC(s0, s1, s2, a[4], a[22]); + MONT_MAC(c0, c1, c2, a[5], a[21]); + MONT_MAC(s0, s1, s2, a[6], a[20]); + MONT_MAC(c0, c1, c2, a[7], a[19]); + MONT_MAC(s0, s1, s2, a[8], a[18]); + MONT_MAC(c0, c1, c2, a[9], a[17]); + MONT_MAC(s0, s1, s2, a[10], a[16]); + MONT_MAC(c0, c1, c2, a[11], a[15]); + MONT_MAC(s0, s1, s2, a[12], a[14]); + MONT_MAC(t0, t1, t2, a[13], a[13]); + MONT_MUL(v0, v1, v2, u[0], m[26]); + MONT_MUL(w0, w1, w2, u[1], m[25]); + MONT_MAC(t0, t1, t2, u[2], m[24]); + MONT_MAC(v0, v1, v2, u[3], m[23]); + MONT_MAC(w0, w1, w2, u[4], m[22]); + MONT_MAC(t0, t1, t2, u[5], m[21]); + MONT_MAC(v0, v1, v2, u[6], m[20]); + MONT_MAC(w0, w1, w2, u[7], m[19]); + MONT_MAC(t0, t1, t2, u[8], m[18]); + MONT_MAC(v0, v1, v2, u[9], m[17]); + MONT_MAC(w0, w1, w2, u[10], m[16]); + MONT_MAC(t0, t1, t2, u[11], m[15]); + MONT_MAC(v0, v1, v2, u[12], m[14]); + MONT_MAC(w0, w1, w2, u[13], m[13]); + MONT_MAC(t0, t1, t2, u[14], m[12]); + MONT_MAC(v0, v1, v2, u[15], m[11]); + MONT_MAC(w0, w1, w2, u[16], m[10]); + MONT_MAC(t0, t1, t2, u[17], m[9]); + MONT_MAC(v0, v1, v2, u[18], m[8]); + MONT_MAC(w0, w1, w2, u[19], m[7]); + MONT_MAC(t0, t1, t2, u[20], m[6]); + MONT_MAC(v0, v1, v2, u[21], m[5]); + MONT_MAC(w0, w1, w2, u[22], m[4]); + MONT_MAC(t0, t1, t2, u[23], m[3]); + MONT_MAC(v0, v1, v2, u[24], m[2]); + MONT_MAC(w0, w1, w2, u[25], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[26] = t0 * n0; + MONT_MAC(t0, t1, t2, u[26], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 27 */ + MONT_MUL(s0, s1, s2, a[0], a[27]); + MONT_MUL(c0, c1, c2, a[1], a[26]); + MONT_MAC(s0, s1, s2, a[2], a[25]); + MONT_MAC(c0, c1, c2, a[3], a[24]); + MONT_MAC(s0, s1, s2, a[4], a[23]); + MONT_MAC(c0, c1, c2, a[5], a[22]); + MONT_MAC(s0, s1, s2, a[6], a[21]); + MONT_MAC(c0, c1, c2, a[7], a[20]); + MONT_MAC(s0, s1, s2, a[8], a[19]); + MONT_MAC(c0, c1, c2, a[9], a[18]); + MONT_MAC(s0, s1, s2, a[10], a[17]); + MONT_MAC(c0, c1, c2, a[11], a[16]); + MONT_MAC(s0, s1, s2, a[12], a[15]); + MONT_MAC(c0, c1, c2, a[13], a[14]); + MONT_MAC(t0, t1, t2, u[0], m[27]); + MONT_MUL(v0, v1, v2, u[1], m[26]); + MONT_MUL(w0, w1, w2, u[2], m[25]); + MONT_MAC(t0, t1, t2, u[3], m[24]); + MONT_MAC(v0, v1, v2, u[4], m[23]); + MONT_MAC(w0, w1, w2, u[5], m[22]); + MONT_MAC(t0, t1, t2, u[6], m[21]); + MONT_MAC(v0, v1, v2, u[7], m[20]); + MONT_MAC(w0, w1, w2, u[8], m[19]); + MONT_MAC(t0, t1, t2, u[9], m[18]); + MONT_MAC(v0, v1, v2, u[10], m[17]); + MONT_MAC(w0, w1, w2, u[11], m[16]); + MONT_MAC(t0, t1, t2, u[12], m[15]); + MONT_MAC(v0, v1, v2, u[13], m[14]); + MONT_MAC(w0, w1, w2, u[14], m[13]); + MONT_MAC(t0, t1, t2, u[15], m[12]); + MONT_MAC(v0, v1, v2, u[16], m[11]); + MONT_MAC(w0, w1, w2, u[17], m[10]); + MONT_MAC(t0, t1, t2, u[18], m[9]); + MONT_MAC(v0, v1, v2, u[19], m[8]); + MONT_MAC(w0, w1, w2, u[20], m[7]); + MONT_MAC(t0, t1, t2, u[21], m[6]); + MONT_MAC(v0, v1, v2, u[22], m[5]); + MONT_MAC(w0, w1, w2, u[23], m[4]); + MONT_MAC(t0, t1, t2, u[24], m[3]); + MONT_MAC(v0, v1, v2, u[25], m[2]); + MONT_MAC(w0, w1, w2, u[26], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[27] = t0 * n0; + MONT_MAC(t0, t1, t2, u[27], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 28 */ + MONT_MUL(s0, s1, s2, a[0], a[28]); + MONT_MUL(c0, c1, c2, a[1], a[27]); + MONT_MAC(s0, s1, s2, a[2], a[26]); + MONT_MAC(c0, c1, c2, a[3], a[25]); + MONT_MAC(s0, s1, s2, a[4], a[24]); + MONT_MAC(c0, c1, c2, a[5], a[23]); + MONT_MAC(s0, s1, s2, a[6], a[22]); + MONT_MAC(c0, c1, c2, a[7], a[21]); + MONT_MAC(s0, s1, s2, a[8], a[20]); + MONT_MAC(c0, c1, c2, a[9], a[19]); + MONT_MAC(s0, s1, s2, a[10], a[18]); + MONT_MAC(c0, c1, c2, a[11], a[17]); + MONT_MAC(s0, s1, s2, a[12], a[16]); + MONT_MAC(c0, c1, c2, a[13], a[15]); + MONT_MAC(t0, t1, t2, a[14], a[14]); + MONT_MUL(v0, v1, v2, u[0], m[28]); + MONT_MUL(w0, w1, w2, u[1], m[27]); + MONT_MAC(t0, t1, t2, u[2], m[26]); + MONT_MAC(v0, v1, v2, u[3], m[25]); + MONT_MAC(w0, w1, w2, u[4], m[24]); + MONT_MAC(t0, t1, t2, u[5], m[23]); + MONT_MAC(v0, v1, v2, u[6], m[22]); + MONT_MAC(w0, w1, w2, u[7], m[21]); + MONT_MAC(t0, t1, t2, u[8], m[20]); + MONT_MAC(v0, v1, v2, u[9], m[19]); + MONT_MAC(w0, w1, w2, u[10], m[18]); + MONT_MAC(t0, t1, t2, u[11], m[17]); + MONT_MAC(v0, v1, v2, u[12], m[16]); + MONT_MAC(w0, w1, w2, u[13], m[15]); + MONT_MAC(t0, t1, t2, u[14], m[14]); + MONT_MAC(v0, v1, v2, u[15], m[13]); + MONT_MAC(w0, w1, w2, u[16], m[12]); + MONT_MAC(t0, t1, t2, u[17], m[11]); + MONT_MAC(v0, v1, v2, u[18], m[10]); + MONT_MAC(w0, w1, w2, u[19], m[9]); + MONT_MAC(t0, t1, t2, u[20], m[8]); + MONT_MAC(v0, v1, v2, u[21], m[7]); + MONT_MAC(w0, w1, w2, u[22], m[6]); + MONT_MAC(t0, t1, t2, u[23], m[5]); + MONT_MAC(v0, v1, v2, u[24], m[4]); + MONT_MAC(w0, w1, w2, u[25], m[3]); + MONT_MAC(t0, t1, t2, u[26], m[2]); + MONT_MAC(v0, v1, v2, u[27], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[28] = t0 * n0; + MONT_MAC(t0, t1, t2, u[28], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 29 */ + MONT_MUL(s0, s1, s2, a[0], a[29]); + MONT_MUL(c0, c1, c2, a[1], a[28]); + MONT_MAC(s0, s1, s2, a[2], a[27]); + MONT_MAC(c0, c1, c2, a[3], a[26]); + MONT_MAC(s0, s1, s2, a[4], a[25]); + MONT_MAC(c0, c1, c2, a[5], a[24]); + MONT_MAC(s0, s1, s2, a[6], a[23]); + MONT_MAC(c0, c1, c2, a[7], a[22]); + MONT_MAC(s0, s1, s2, a[8], a[21]); + MONT_MAC(c0, c1, c2, a[9], a[20]); + MONT_MAC(s0, s1, s2, a[10], a[19]); + MONT_MAC(c0, c1, c2, a[11], a[18]); + MONT_MAC(s0, s1, s2, a[12], a[17]); + MONT_MAC(c0, c1, c2, a[13], a[16]); + MONT_MAC(s0, s1, s2, a[14], a[15]); + MONT_MAC(t0, t1, t2, u[0], m[29]); + MONT_MUL(v0, v1, v2, u[1], m[28]); + MONT_MUL(w0, w1, w2, u[2], m[27]); + MONT_MAC(t0, t1, t2, u[3], m[26]); + MONT_MAC(v0, v1, v2, u[4], m[25]); + MONT_MAC(w0, w1, w2, u[5], m[24]); + MONT_MAC(t0, t1, t2, u[6], m[23]); + MONT_MAC(v0, v1, v2, u[7], m[22]); + MONT_MAC(w0, w1, w2, u[8], m[21]); + MONT_MAC(t0, t1, t2, u[9], m[20]); + MONT_MAC(v0, v1, v2, u[10], m[19]); + MONT_MAC(w0, w1, w2, u[11], m[18]); + MONT_MAC(t0, t1, t2, u[12], m[17]); + MONT_MAC(v0, v1, v2, u[13], m[16]); + MONT_MAC(w0, w1, w2, u[14], m[15]); + MONT_MAC(t0, t1, t2, u[15], m[14]); + MONT_MAC(v0, v1, v2, u[16], m[13]); + MONT_MAC(w0, w1, w2, u[17], m[12]); + MONT_MAC(t0, t1, t2, u[18], m[11]); + MONT_MAC(v0, v1, v2, u[19], m[10]); + MONT_MAC(w0, w1, w2, u[20], m[9]); + MONT_MAC(t0, t1, t2, u[21], m[8]); + MONT_MAC(v0, v1, v2, u[22], m[7]); + MONT_MAC(w0, w1, w2, u[23], m[6]); + MONT_MAC(t0, t1, t2, u[24], m[5]); + MONT_MAC(v0, v1, v2, u[25], m[4]); + MONT_MAC(w0, w1, w2, u[26], m[3]); + MONT_MAC(t0, t1, t2, u[27], m[2]); + MONT_MAC(v0, v1, v2, u[28], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[29] = t0 * n0; + MONT_MAC(t0, t1, t2, u[29], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 30 */ + MONT_MUL(s0, s1, s2, a[0], a[30]); + MONT_MUL(c0, c1, c2, a[1], a[29]); + MONT_MAC(s0, s1, s2, a[2], a[28]); + MONT_MAC(c0, c1, c2, a[3], a[27]); + MONT_MAC(s0, s1, s2, a[4], a[26]); + MONT_MAC(c0, c1, c2, a[5], a[25]); + MONT_MAC(s0, s1, s2, a[6], a[24]); + MONT_MAC(c0, c1, c2, a[7], a[23]); + MONT_MAC(s0, s1, s2, a[8], a[22]); + MONT_MAC(c0, c1, c2, a[9], a[21]); + MONT_MAC(s0, s1, s2, a[10], a[20]); + MONT_MAC(c0, c1, c2, a[11], a[19]); + MONT_MAC(s0, s1, s2, a[12], a[18]); + MONT_MAC(c0, c1, c2, a[13], a[17]); + MONT_MAC(s0, s1, s2, a[14], a[16]); + MONT_MAC(t0, t1, t2, a[15], a[15]); + MONT_MUL(v0, v1, v2, u[0], m[30]); + MONT_MUL(w0, w1, w2, u[1], m[29]); + MONT_MAC(t0, t1, t2, u[2], m[28]); + MONT_MAC(v0, v1, v2, u[3], m[27]); + MONT_MAC(w0, w1, w2, u[4], m[26]); + MONT_MAC(t0, t1, t2, u[5], m[25]); + MONT_MAC(v0, v1, v2, u[6], m[24]); + MONT_MAC(w0, w1, w2, u[7], m[23]); + MONT_MAC(t0, t1, t2, u[8], m[22]); + MONT_MAC(v0, v1, v2, u[9], m[21]); + MONT_MAC(w0, w1, w2, u[10], m[20]); + MONT_MAC(t0, t1, t2, u[11], m[19]); + MONT_MAC(v0, v1, v2, u[12], m[18]); + MONT_MAC(w0, w1, w2, u[13], m[17]); + MONT_MAC(t0, t1, t2, u[14], m[16]); + MONT_MAC(v0, v1, v2, u[15], m[15]); + MONT_MAC(w0, w1, w2, u[16], m[14]); + MONT_MAC(t0, t1, t2, u[17], m[13]); + MONT_MAC(v0, v1, v2, u[18], m[12]); + MONT_MAC(w0, w1, w2, u[19], m[11]); + MONT_MAC(t0, t1, t2, u[20], m[10]); + MONT_MAC(v0, v1, v2, u[21], m[9]); + MONT_MAC(w0, w1, w2, u[22], m[8]); + MONT_MAC(t0, t1, t2, u[23], m[7]); + MONT_MAC(v0, v1, v2, u[24], m[6]); + MONT_MAC(w0, w1, w2, u[25], m[5]); + MONT_MAC(t0, t1, t2, u[26], m[4]); + MONT_MAC(v0, v1, v2, u[27], m[3]); + MONT_MAC(w0, w1, w2, u[28], m[2]); + MONT_MAC(t0, t1, t2, u[29], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[30] = t0 * n0; + MONT_MAC(t0, t1, t2, u[30], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 31 */ + MONT_MUL(s0, s1, s2, a[0], a[31]); + MONT_MUL(c0, c1, c2, a[1], a[30]); + MONT_MAC(s0, s1, s2, a[2], a[29]); + MONT_MAC(c0, c1, c2, a[3], a[28]); + MONT_MAC(s0, s1, s2, a[4], a[27]); + MONT_MAC(c0, c1, c2, a[5], a[26]); + MONT_MAC(s0, s1, s2, a[6], a[25]); + MONT_MAC(c0, c1, c2, a[7], a[24]); + MONT_MAC(s0, s1, s2, a[8], a[23]); + MONT_MAC(c0, c1, c2, a[9], a[22]); + MONT_MAC(s0, s1, s2, a[10], a[21]); + MONT_MAC(c0, c1, c2, a[11], a[20]); + MONT_MAC(s0, s1, s2, a[12], a[19]); + MONT_MAC(c0, c1, c2, a[13], a[18]); + MONT_MAC(s0, s1, s2, a[14], a[17]); + MONT_MAC(c0, c1, c2, a[15], a[16]); + MONT_MAC(t0, t1, t2, u[0], m[31]); + MONT_MUL(v0, v1, v2, u[1], m[30]); + MONT_MUL(w0, w1, w2, u[2], m[29]); + MONT_MAC(t0, t1, t2, u[3], m[28]); + MONT_MAC(v0, v1, v2, u[4], m[27]); + MONT_MAC(w0, w1, w2, u[5], m[26]); + MONT_MAC(t0, t1, t2, u[6], m[25]); + MONT_MAC(v0, v1, v2, u[7], m[24]); + MONT_MAC(w0, w1, w2, u[8], m[23]); + MONT_MAC(t0, t1, t2, u[9], m[22]); + MONT_MAC(v0, v1, v2, u[10], m[21]); + MONT_MAC(w0, w1, w2, u[11], m[20]); + MONT_MAC(t0, t1, t2, u[12], m[19]); + MONT_MAC(v0, v1, v2, u[13], m[18]); + MONT_MAC(w0, w1, w2, u[14], m[17]); + MONT_MAC(t0, t1, t2, u[15], m[16]); + MONT_MAC(v0, v1, v2, u[16], m[15]); + MONT_MAC(w0, w1, w2, u[17], m[14]); + MONT_MAC(t0, t1, t2, u[18], m[13]); + MONT_MAC(v0, v1, v2, u[19], m[12]); + MONT_MAC(w0, w1, w2, u[20], m[11]); + MONT_MAC(t0, t1, t2, u[21], m[10]); + MONT_MAC(v0, v1, v2, u[22], m[9]); + MONT_MAC(w0, w1, w2, u[23], m[8]); + MONT_MAC(t0, t1, t2, u[24], m[7]); + MONT_MAC(v0, v1, v2, u[25], m[6]); + MONT_MAC(w0, w1, w2, u[26], m[5]); + MONT_MAC(t0, t1, t2, u[27], m[4]); + MONT_MAC(v0, v1, v2, u[28], m[3]); + MONT_MAC(w0, w1, w2, u[29], m[2]); + MONT_MAC(t0, t1, t2, u[30], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[31] = t0 * n0; + MONT_MAC(t0, t1, t2, u[31], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 32 */ + MONT_MUL(s0, s1, s2, a[1], a[31]); + MONT_MUL(c0, c1, c2, a[2], a[30]); + MONT_MAC(s0, s1, s2, a[3], a[29]); + MONT_MAC(c0, c1, c2, a[4], a[28]); + MONT_MAC(s0, s1, s2, a[5], a[27]); + MONT_MAC(c0, c1, c2, a[6], a[26]); + MONT_MAC(s0, s1, s2, a[7], a[25]); + MONT_MAC(c0, c1, c2, a[8], a[24]); + MONT_MAC(s0, s1, s2, a[9], a[23]); + MONT_MAC(c0, c1, c2, a[10], a[22]); + MONT_MAC(s0, s1, s2, a[11], a[21]); + MONT_MAC(c0, c1, c2, a[12], a[20]); + MONT_MAC(s0, s1, s2, a[13], a[19]); + MONT_MAC(c0, c1, c2, a[14], a[18]); + MONT_MAC(s0, s1, s2, a[15], a[17]); + MONT_MAC(t0, t1, t2, a[16], a[16]); + MONT_MUL(v0, v1, v2, u[1], m[31]); + MONT_MUL(w0, w1, w2, u[2], m[30]); + MONT_MAC(t0, t1, t2, u[3], m[29]); + MONT_MAC(v0, v1, v2, u[4], m[28]); + MONT_MAC(w0, w1, w2, u[5], m[27]); + MONT_MAC(t0, t1, t2, u[6], m[26]); + MONT_MAC(v0, v1, v2, u[7], m[25]); + MONT_MAC(w0, w1, w2, u[8], m[24]); + MONT_MAC(t0, t1, t2, u[9], m[23]); + MONT_MAC(v0, v1, v2, u[10], m[22]); + MONT_MAC(w0, w1, w2, u[11], m[21]); + MONT_MAC(t0, t1, t2, u[12], m[20]); + MONT_MAC(v0, v1, v2, u[13], m[19]); + MONT_MAC(w0, w1, w2, u[14], m[18]); + MONT_MAC(t0, t1, t2, u[15], m[17]); + MONT_MAC(v0, v1, v2, u[16], m[16]); + MONT_MAC(w0, w1, w2, u[17], m[15]); + MONT_MAC(t0, t1, t2, u[18], m[14]); + MONT_MAC(v0, v1, v2, u[19], m[13]); + MONT_MAC(w0, w1, w2, u[20], m[12]); + MONT_MAC(t0, t1, t2, u[21], m[11]); + MONT_MAC(v0, v1, v2, u[22], m[10]); + MONT_MAC(w0, w1, w2, u[23], m[9]); + MONT_MAC(t0, t1, t2, u[24], m[8]); + MONT_MAC(v0, v1, v2, u[25], m[7]); + MONT_MAC(w0, w1, w2, u[26], m[6]); + MONT_MAC(t0, t1, t2, u[27], m[5]); + MONT_MAC(v0, v1, v2, u[28], m[4]); + MONT_MAC(w0, w1, w2, u[29], m[3]); + MONT_MAC(t0, t1, t2, u[30], m[2]); + MONT_MAC(v0, v1, v2, u[31], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[0] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 33 */ + MONT_MUL(s0, s1, s2, a[2], a[31]); + MONT_MUL(c0, c1, c2, a[3], a[30]); + MONT_MAC(s0, s1, s2, a[4], a[29]); + MONT_MAC(c0, c1, c2, a[5], a[28]); + MONT_MAC(s0, s1, s2, a[6], a[27]); + MONT_MAC(c0, c1, c2, a[7], a[26]); + MONT_MAC(s0, s1, s2, a[8], a[25]); + MONT_MAC(c0, c1, c2, a[9], a[24]); + MONT_MAC(s0, s1, s2, a[10], a[23]); + MONT_MAC(c0, c1, c2, a[11], a[22]); + MONT_MAC(s0, s1, s2, a[12], a[21]); + MONT_MAC(c0, c1, c2, a[13], a[20]); + MONT_MAC(s0, s1, s2, a[14], a[19]); + MONT_MAC(c0, c1, c2, a[15], a[18]); + MONT_MAC(s0, s1, s2, a[16], a[17]); + MONT_MAC(t0, t1, t2, u[2], m[31]); + MONT_MUL(v0, v1, v2, u[3], m[30]); + MONT_MUL(w0, w1, w2, u[4], m[29]); + MONT_MAC(t0, t1, t2, u[5], m[28]); + MONT_MAC(v0, v1, v2, u[6], m[27]); + MONT_MAC(w0, w1, w2, u[7], m[26]); + MONT_MAC(t0, t1, t2, u[8], m[25]); + MONT_MAC(v0, v1, v2, u[9], m[24]); + MONT_MAC(w0, w1, w2, u[10], m[23]); + MONT_MAC(t0, t1, t2, u[11], m[22]); + MONT_MAC(v0, v1, v2, u[12], m[21]); + MONT_MAC(w0, w1, w2, u[13], m[20]); + MONT_MAC(t0, t1, t2, u[14], m[19]); + MONT_MAC(v0, v1, v2, u[15], m[18]); + MONT_MAC(w0, w1, w2, u[16], m[17]); + MONT_MAC(t0, t1, t2, u[17], m[16]); + MONT_MAC(v0, v1, v2, u[18], m[15]); + MONT_MAC(w0, w1, w2, u[19], m[14]); + MONT_MAC(t0, t1, t2, u[20], m[13]); + MONT_MAC(v0, v1, v2, u[21], m[12]); + MONT_MAC(w0, w1, w2, u[22], m[11]); + MONT_MAC(t0, t1, t2, u[23], m[10]); + MONT_MAC(v0, v1, v2, u[24], m[9]); + MONT_MAC(w0, w1, w2, u[25], m[8]); + MONT_MAC(t0, t1, t2, u[26], m[7]); + MONT_MAC(v0, v1, v2, u[27], m[6]); + MONT_MAC(w0, w1, w2, u[28], m[5]); + MONT_MAC(t0, t1, t2, u[29], m[4]); + MONT_MAC(v0, v1, v2, u[30], m[3]); + MONT_MAC(w0, w1, w2, u[31], m[2]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[1] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 34 */ + MONT_MUL(s0, s1, s2, a[3], a[31]); + MONT_MUL(c0, c1, c2, a[4], a[30]); + MONT_MAC(s0, s1, s2, a[5], a[29]); + MONT_MAC(c0, c1, c2, a[6], a[28]); + MONT_MAC(s0, s1, s2, a[7], a[27]); + MONT_MAC(c0, c1, c2, a[8], a[26]); + MONT_MAC(s0, s1, s2, a[9], a[25]); + MONT_MAC(c0, c1, c2, a[10], a[24]); + MONT_MAC(s0, s1, s2, a[11], a[23]); + MONT_MAC(c0, c1, c2, a[12], a[22]); + MONT_MAC(s0, s1, s2, a[13], a[21]); + MONT_MAC(c0, c1, c2, a[14], a[20]); + MONT_MAC(s0, s1, s2, a[15], a[19]); + MONT_MAC(c0, c1, c2, a[16], a[18]); + MONT_MAC(t0, t1, t2, a[17], a[17]); + MONT_MUL(v0, v1, v2, u[3], m[31]); + MONT_MUL(w0, w1, w2, u[4], m[30]); + MONT_MAC(t0, t1, t2, u[5], m[29]); + MONT_MAC(v0, v1, v2, u[6], m[28]); + MONT_MAC(w0, w1, w2, u[7], m[27]); + MONT_MAC(t0, t1, t2, u[8], m[26]); + MONT_MAC(v0, v1, v2, u[9], m[25]); + MONT_MAC(w0, w1, w2, u[10], m[24]); + MONT_MAC(t0, t1, t2, u[11], m[23]); + MONT_MAC(v0, v1, v2, u[12], m[22]); + MONT_MAC(w0, w1, w2, u[13], m[21]); + MONT_MAC(t0, t1, t2, u[14], m[20]); + MONT_MAC(v0, v1, v2, u[15], m[19]); + MONT_MAC(w0, w1, w2, u[16], m[18]); + MONT_MAC(t0, t1, t2, u[17], m[17]); + MONT_MAC(v0, v1, v2, u[18], m[16]); + MONT_MAC(w0, w1, w2, u[19], m[15]); + MONT_MAC(t0, t1, t2, u[20], m[14]); + MONT_MAC(v0, v1, v2, u[21], m[13]); + MONT_MAC(w0, w1, w2, u[22], m[12]); + MONT_MAC(t0, t1, t2, u[23], m[11]); + MONT_MAC(v0, v1, v2, u[24], m[10]); + MONT_MAC(w0, w1, w2, u[25], m[9]); + MONT_MAC(t0, t1, t2, u[26], m[8]); + MONT_MAC(v0, v1, v2, u[27], m[7]); + MONT_MAC(w0, w1, w2, u[28], m[6]); + MONT_MAC(t0, t1, t2, u[29], m[5]); + MONT_MAC(v0, v1, v2, u[30], m[4]); + MONT_MAC(w0, w1, w2, u[31], m[3]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[2] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 35 */ + MONT_MUL(s0, s1, s2, a[4], a[31]); + MONT_MUL(c0, c1, c2, a[5], a[30]); + MONT_MAC(s0, s1, s2, a[6], a[29]); + MONT_MAC(c0, c1, c2, a[7], a[28]); + MONT_MAC(s0, s1, s2, a[8], a[27]); + MONT_MAC(c0, c1, c2, a[9], a[26]); + MONT_MAC(s0, s1, s2, a[10], a[25]); + MONT_MAC(c0, c1, c2, a[11], a[24]); + MONT_MAC(s0, s1, s2, a[12], a[23]); + MONT_MAC(c0, c1, c2, a[13], a[22]); + MONT_MAC(s0, s1, s2, a[14], a[21]); + MONT_MAC(c0, c1, c2, a[15], a[20]); + MONT_MAC(s0, s1, s2, a[16], a[19]); + MONT_MAC(c0, c1, c2, a[17], a[18]); + MONT_MAC(t0, t1, t2, u[4], m[31]); + MONT_MUL(v0, v1, v2, u[5], m[30]); + MONT_MUL(w0, w1, w2, u[6], m[29]); + MONT_MAC(t0, t1, t2, u[7], m[28]); + MONT_MAC(v0, v1, v2, u[8], m[27]); + MONT_MAC(w0, w1, w2, u[9], m[26]); + MONT_MAC(t0, t1, t2, u[10], m[25]); + MONT_MAC(v0, v1, v2, u[11], m[24]); + MONT_MAC(w0, w1, w2, u[12], m[23]); + MONT_MAC(t0, t1, t2, u[13], m[22]); + MONT_MAC(v0, v1, v2, u[14], m[21]); + MONT_MAC(w0, w1, w2, u[15], m[20]); + MONT_MAC(t0, t1, t2, u[16], m[19]); + MONT_MAC(v0, v1, v2, u[17], m[18]); + MONT_MAC(w0, w1, w2, u[18], m[17]); + MONT_MAC(t0, t1, t2, u[19], m[16]); + MONT_MAC(v0, v1, v2, u[20], m[15]); + MONT_MAC(w0, w1, w2, u[21], m[14]); + MONT_MAC(t0, t1, t2, u[22], m[13]); + MONT_MAC(v0, v1, v2, u[23], m[12]); + MONT_MAC(w0, w1, w2, u[24], m[11]); + MONT_MAC(t0, t1, t2, u[25], m[10]); + MONT_MAC(v0, v1, v2, u[26], m[9]); + MONT_MAC(w0, w1, w2, u[27], m[8]); + MONT_MAC(t0, t1, t2, u[28], m[7]); + MONT_MAC(v0, v1, v2, u[29], m[6]); + MONT_MAC(w0, w1, w2, u[30], m[5]); + MONT_MAC(t0, t1, t2, u[31], m[4]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[3] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 36 */ + MONT_MUL(s0, s1, s2, a[5], a[31]); + MONT_MUL(c0, c1, c2, a[6], a[30]); + MONT_MAC(s0, s1, s2, a[7], a[29]); + MONT_MAC(c0, c1, c2, a[8], a[28]); + MONT_MAC(s0, s1, s2, a[9], a[27]); + MONT_MAC(c0, c1, c2, a[10], a[26]); + MONT_MAC(s0, s1, s2, a[11], a[25]); + MONT_MAC(c0, c1, c2, a[12], a[24]); + MONT_MAC(s0, s1, s2, a[13], a[23]); + MONT_MAC(c0, c1, c2, a[14], a[22]); + MONT_MAC(s0, s1, s2, a[15], a[21]); + MONT_MAC(c0, c1, c2, a[16], a[20]); + MONT_MAC(s0, s1, s2, a[17], a[19]); + MONT_MAC(t0, t1, t2, a[18], a[18]); + MONT_MUL(v0, v1, v2, u[5], m[31]); + MONT_MUL(w0, w1, w2, u[6], m[30]); + MONT_MAC(t0, t1, t2, u[7], m[29]); + MONT_MAC(v0, v1, v2, u[8], m[28]); + MONT_MAC(w0, w1, w2, u[9], m[27]); + MONT_MAC(t0, t1, t2, u[10], m[26]); + MONT_MAC(v0, v1, v2, u[11], m[25]); + MONT_MAC(w0, w1, w2, u[12], m[24]); + MONT_MAC(t0, t1, t2, u[13], m[23]); + MONT_MAC(v0, v1, v2, u[14], m[22]); + MONT_MAC(w0, w1, w2, u[15], m[21]); + MONT_MAC(t0, t1, t2, u[16], m[20]); + MONT_MAC(v0, v1, v2, u[17], m[19]); + MONT_MAC(w0, w1, w2, u[18], m[18]); + MONT_MAC(t0, t1, t2, u[19], m[17]); + MONT_MAC(v0, v1, v2, u[20], m[16]); + MONT_MAC(w0, w1, w2, u[21], m[15]); + MONT_MAC(t0, t1, t2, u[22], m[14]); + MONT_MAC(v0, v1, v2, u[23], m[13]); + MONT_MAC(w0, w1, w2, u[24], m[12]); + MONT_MAC(t0, t1, t2, u[25], m[11]); + MONT_MAC(v0, v1, v2, u[26], m[10]); + MONT_MAC(w0, w1, w2, u[27], m[9]); + MONT_MAC(t0, t1, t2, u[28], m[8]); + MONT_MAC(v0, v1, v2, u[29], m[7]); + MONT_MAC(w0, w1, w2, u[30], m[6]); + MONT_MAC(t0, t1, t2, u[31], m[5]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[4] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 37 */ + MONT_MUL(s0, s1, s2, a[6], a[31]); + MONT_MUL(c0, c1, c2, a[7], a[30]); + MONT_MAC(s0, s1, s2, a[8], a[29]); + MONT_MAC(c0, c1, c2, a[9], a[28]); + MONT_MAC(s0, s1, s2, a[10], a[27]); + MONT_MAC(c0, c1, c2, a[11], a[26]); + MONT_MAC(s0, s1, s2, a[12], a[25]); + MONT_MAC(c0, c1, c2, a[13], a[24]); + MONT_MAC(s0, s1, s2, a[14], a[23]); + MONT_MAC(c0, c1, c2, a[15], a[22]); + MONT_MAC(s0, s1, s2, a[16], a[21]); + MONT_MAC(c0, c1, c2, a[17], a[20]); + MONT_MAC(s0, s1, s2, a[18], a[19]); + MONT_MAC(t0, t1, t2, u[6], m[31]); + MONT_MUL(v0, v1, v2, u[7], m[30]); + MONT_MUL(w0, w1, w2, u[8], m[29]); + MONT_MAC(t0, t1, t2, u[9], m[28]); + MONT_MAC(v0, v1, v2, u[10], m[27]); + MONT_MAC(w0, w1, w2, u[11], m[26]); + MONT_MAC(t0, t1, t2, u[12], m[25]); + MONT_MAC(v0, v1, v2, u[13], m[24]); + MONT_MAC(w0, w1, w2, u[14], m[23]); + MONT_MAC(t0, t1, t2, u[15], m[22]); + MONT_MAC(v0, v1, v2, u[16], m[21]); + MONT_MAC(w0, w1, w2, u[17], m[20]); + MONT_MAC(t0, t1, t2, u[18], m[19]); + MONT_MAC(v0, v1, v2, u[19], m[18]); + MONT_MAC(w0, w1, w2, u[20], m[17]); + MONT_MAC(t0, t1, t2, u[21], m[16]); + MONT_MAC(v0, v1, v2, u[22], m[15]); + MONT_MAC(w0, w1, w2, u[23], m[14]); + MONT_MAC(t0, t1, t2, u[24], m[13]); + MONT_MAC(v0, v1, v2, u[25], m[12]); + MONT_MAC(w0, w1, w2, u[26], m[11]); + MONT_MAC(t0, t1, t2, u[27], m[10]); + MONT_MAC(v0, v1, v2, u[28], m[9]); + MONT_MAC(w0, w1, w2, u[29], m[8]); + MONT_MAC(t0, t1, t2, u[30], m[7]); + MONT_MAC(v0, v1, v2, u[31], m[6]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[5] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 38 */ + MONT_MUL(s0, s1, s2, a[7], a[31]); + MONT_MUL(c0, c1, c2, a[8], a[30]); + MONT_MAC(s0, s1, s2, a[9], a[29]); + MONT_MAC(c0, c1, c2, a[10], a[28]); + MONT_MAC(s0, s1, s2, a[11], a[27]); + MONT_MAC(c0, c1, c2, a[12], a[26]); + MONT_MAC(s0, s1, s2, a[13], a[25]); + MONT_MAC(c0, c1, c2, a[14], a[24]); + MONT_MAC(s0, s1, s2, a[15], a[23]); + MONT_MAC(c0, c1, c2, a[16], a[22]); + MONT_MAC(s0, s1, s2, a[17], a[21]); + MONT_MAC(c0, c1, c2, a[18], a[20]); + MONT_MAC(t0, t1, t2, a[19], a[19]); + MONT_MUL(v0, v1, v2, u[7], m[31]); + MONT_MUL(w0, w1, w2, u[8], m[30]); + MONT_MAC(t0, t1, t2, u[9], m[29]); + MONT_MAC(v0, v1, v2, u[10], m[28]); + MONT_MAC(w0, w1, w2, u[11], m[27]); + MONT_MAC(t0, t1, t2, u[12], m[26]); + MONT_MAC(v0, v1, v2, u[13], m[25]); + MONT_MAC(w0, w1, w2, u[14], m[24]); + MONT_MAC(t0, t1, t2, u[15], m[23]); + MONT_MAC(v0, v1, v2, u[16], m[22]); + MONT_MAC(w0, w1, w2, u[17], m[21]); + MONT_MAC(t0, t1, t2, u[18], m[20]); + MONT_MAC(v0, v1, v2, u[19], m[19]); + MONT_MAC(w0, w1, w2, u[20], m[18]); + MONT_MAC(t0, t1, t2, u[21], m[17]); + MONT_MAC(v0, v1, v2, u[22], m[16]); + MONT_MAC(w0, w1, w2, u[23], m[15]); + MONT_MAC(t0, t1, t2, u[24], m[14]); + MONT_MAC(v0, v1, v2, u[25], m[13]); + MONT_MAC(w0, w1, w2, u[26], m[12]); + MONT_MAC(t0, t1, t2, u[27], m[11]); + MONT_MAC(v0, v1, v2, u[28], m[10]); + MONT_MAC(w0, w1, w2, u[29], m[9]); + MONT_MAC(t0, t1, t2, u[30], m[8]); + MONT_MAC(v0, v1, v2, u[31], m[7]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[6] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 39 */ + MONT_MUL(s0, s1, s2, a[8], a[31]); + MONT_MUL(c0, c1, c2, a[9], a[30]); + MONT_MAC(s0, s1, s2, a[10], a[29]); + MONT_MAC(c0, c1, c2, a[11], a[28]); + MONT_MAC(s0, s1, s2, a[12], a[27]); + MONT_MAC(c0, c1, c2, a[13], a[26]); + MONT_MAC(s0, s1, s2, a[14], a[25]); + MONT_MAC(c0, c1, c2, a[15], a[24]); + MONT_MAC(s0, s1, s2, a[16], a[23]); + MONT_MAC(c0, c1, c2, a[17], a[22]); + MONT_MAC(s0, s1, s2, a[18], a[21]); + MONT_MAC(c0, c1, c2, a[19], a[20]); + MONT_MAC(t0, t1, t2, u[8], m[31]); + MONT_MUL(v0, v1, v2, u[9], m[30]); + MONT_MUL(w0, w1, w2, u[10], m[29]); + MONT_MAC(t0, t1, t2, u[11], m[28]); + MONT_MAC(v0, v1, v2, u[12], m[27]); + MONT_MAC(w0, w1, w2, u[13], m[26]); + MONT_MAC(t0, t1, t2, u[14], m[25]); + MONT_MAC(v0, v1, v2, u[15], m[24]); + MONT_MAC(w0, w1, w2, u[16], m[23]); + MONT_MAC(t0, t1, t2, u[17], m[22]); + MONT_MAC(v0, v1, v2, u[18], m[21]); + MONT_MAC(w0, w1, w2, u[19], m[20]); + MONT_MAC(t0, t1, t2, u[20], m[19]); + MONT_MAC(v0, v1, v2, u[21], m[18]); + MONT_MAC(w0, w1, w2, u[22], m[17]); + MONT_MAC(t0, t1, t2, u[23], m[16]); + MONT_MAC(v0, v1, v2, u[24], m[15]); + MONT_MAC(w0, w1, w2, u[25], m[14]); + MONT_MAC(t0, t1, t2, u[26], m[13]); + MONT_MAC(v0, v1, v2, u[27], m[12]); + MONT_MAC(w0, w1, w2, u[28], m[11]); + MONT_MAC(t0, t1, t2, u[29], m[10]); + MONT_MAC(v0, v1, v2, u[30], m[9]); + MONT_MAC(w0, w1, w2, u[31], m[8]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[7] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 40 */ + MONT_MUL(s0, s1, s2, a[9], a[31]); + MONT_MUL(c0, c1, c2, a[10], a[30]); + MONT_MAC(s0, s1, s2, a[11], a[29]); + MONT_MAC(c0, c1, c2, a[12], a[28]); + MONT_MAC(s0, s1, s2, a[13], a[27]); + MONT_MAC(c0, c1, c2, a[14], a[26]); + MONT_MAC(s0, s1, s2, a[15], a[25]); + MONT_MAC(c0, c1, c2, a[16], a[24]); + MONT_MAC(s0, s1, s2, a[17], a[23]); + MONT_MAC(c0, c1, c2, a[18], a[22]); + MONT_MAC(s0, s1, s2, a[19], a[21]); + MONT_MAC(t0, t1, t2, a[20], a[20]); + MONT_MUL(v0, v1, v2, u[9], m[31]); + MONT_MUL(w0, w1, w2, u[10], m[30]); + MONT_MAC(t0, t1, t2, u[11], m[29]); + MONT_MAC(v0, v1, v2, u[12], m[28]); + MONT_MAC(w0, w1, w2, u[13], m[27]); + MONT_MAC(t0, t1, t2, u[14], m[26]); + MONT_MAC(v0, v1, v2, u[15], m[25]); + MONT_MAC(w0, w1, w2, u[16], m[24]); + MONT_MAC(t0, t1, t2, u[17], m[23]); + MONT_MAC(v0, v1, v2, u[18], m[22]); + MONT_MAC(w0, w1, w2, u[19], m[21]); + MONT_MAC(t0, t1, t2, u[20], m[20]); + MONT_MAC(v0, v1, v2, u[21], m[19]); + MONT_MAC(w0, w1, w2, u[22], m[18]); + MONT_MAC(t0, t1, t2, u[23], m[17]); + MONT_MAC(v0, v1, v2, u[24], m[16]); + MONT_MAC(w0, w1, w2, u[25], m[15]); + MONT_MAC(t0, t1, t2, u[26], m[14]); + MONT_MAC(v0, v1, v2, u[27], m[13]); + MONT_MAC(w0, w1, w2, u[28], m[12]); + MONT_MAC(t0, t1, t2, u[29], m[11]); + MONT_MAC(v0, v1, v2, u[30], m[10]); + MONT_MAC(w0, w1, w2, u[31], m[9]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[8] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 41 */ + MONT_MUL(s0, s1, s2, a[10], a[31]); + MONT_MUL(c0, c1, c2, a[11], a[30]); + MONT_MAC(s0, s1, s2, a[12], a[29]); + MONT_MAC(c0, c1, c2, a[13], a[28]); + MONT_MAC(s0, s1, s2, a[14], a[27]); + MONT_MAC(c0, c1, c2, a[15], a[26]); + MONT_MAC(s0, s1, s2, a[16], a[25]); + MONT_MAC(c0, c1, c2, a[17], a[24]); + MONT_MAC(s0, s1, s2, a[18], a[23]); + MONT_MAC(c0, c1, c2, a[19], a[22]); + MONT_MAC(s0, s1, s2, a[20], a[21]); + MONT_MAC(t0, t1, t2, u[10], m[31]); + MONT_MUL(v0, v1, v2, u[11], m[30]); + MONT_MUL(w0, w1, w2, u[12], m[29]); + MONT_MAC(t0, t1, t2, u[13], m[28]); + MONT_MAC(v0, v1, v2, u[14], m[27]); + MONT_MAC(w0, w1, w2, u[15], m[26]); + MONT_MAC(t0, t1, t2, u[16], m[25]); + MONT_MAC(v0, v1, v2, u[17], m[24]); + MONT_MAC(w0, w1, w2, u[18], m[23]); + MONT_MAC(t0, t1, t2, u[19], m[22]); + MONT_MAC(v0, v1, v2, u[20], m[21]); + MONT_MAC(w0, w1, w2, u[21], m[20]); + MONT_MAC(t0, t1, t2, u[22], m[19]); + MONT_MAC(v0, v1, v2, u[23], m[18]); + MONT_MAC(w0, w1, w2, u[24], m[17]); + MONT_MAC(t0, t1, t2, u[25], m[16]); + MONT_MAC(v0, v1, v2, u[26], m[15]); + MONT_MAC(w0, w1, w2, u[27], m[14]); + MONT_MAC(t0, t1, t2, u[28], m[13]); + MONT_MAC(v0, v1, v2, u[29], m[12]); + MONT_MAC(w0, w1, w2, u[30], m[11]); + MONT_MAC(t0, t1, t2, u[31], m[10]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[9] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 42 */ + MONT_MUL(s0, s1, s2, a[11], a[31]); + MONT_MUL(c0, c1, c2, a[12], a[30]); + MONT_MAC(s0, s1, s2, a[13], a[29]); + MONT_MAC(c0, c1, c2, a[14], a[28]); + MONT_MAC(s0, s1, s2, a[15], a[27]); + MONT_MAC(c0, c1, c2, a[16], a[26]); + MONT_MAC(s0, s1, s2, a[17], a[25]); + MONT_MAC(c0, c1, c2, a[18], a[24]); + MONT_MAC(s0, s1, s2, a[19], a[23]); + MONT_MAC(c0, c1, c2, a[20], a[22]); + MONT_MAC(t0, t1, t2, a[21], a[21]); + MONT_MUL(v0, v1, v2, u[11], m[31]); + MONT_MUL(w0, w1, w2, u[12], m[30]); + MONT_MAC(t0, t1, t2, u[13], m[29]); + MONT_MAC(v0, v1, v2, u[14], m[28]); + MONT_MAC(w0, w1, w2, u[15], m[27]); + MONT_MAC(t0, t1, t2, u[16], m[26]); + MONT_MAC(v0, v1, v2, u[17], m[25]); + MONT_MAC(w0, w1, w2, u[18], m[24]); + MONT_MAC(t0, t1, t2, u[19], m[23]); + MONT_MAC(v0, v1, v2, u[20], m[22]); + MONT_MAC(w0, w1, w2, u[21], m[21]); + MONT_MAC(t0, t1, t2, u[22], m[20]); + MONT_MAC(v0, v1, v2, u[23], m[19]); + MONT_MAC(w0, w1, w2, u[24], m[18]); + MONT_MAC(t0, t1, t2, u[25], m[17]); + MONT_MAC(v0, v1, v2, u[26], m[16]); + MONT_MAC(w0, w1, w2, u[27], m[15]); + MONT_MAC(t0, t1, t2, u[28], m[14]); + MONT_MAC(v0, v1, v2, u[29], m[13]); + MONT_MAC(w0, w1, w2, u[30], m[12]); + MONT_MAC(t0, t1, t2, u[31], m[11]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[10] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 43 */ + MONT_MUL(s0, s1, s2, a[12], a[31]); + MONT_MUL(c0, c1, c2, a[13], a[30]); + MONT_MAC(s0, s1, s2, a[14], a[29]); + MONT_MAC(c0, c1, c2, a[15], a[28]); + MONT_MAC(s0, s1, s2, a[16], a[27]); + MONT_MAC(c0, c1, c2, a[17], a[26]); + MONT_MAC(s0, s1, s2, a[18], a[25]); + MONT_MAC(c0, c1, c2, a[19], a[24]); + MONT_MAC(s0, s1, s2, a[20], a[23]); + MONT_MAC(c0, c1, c2, a[21], a[22]); + MONT_MAC(t0, t1, t2, u[12], m[31]); + MONT_MUL(v0, v1, v2, u[13], m[30]); + MONT_MUL(w0, w1, w2, u[14], m[29]); + MONT_MAC(t0, t1, t2, u[15], m[28]); + MONT_MAC(v0, v1, v2, u[16], m[27]); + MONT_MAC(w0, w1, w2, u[17], m[26]); + MONT_MAC(t0, t1, t2, u[18], m[25]); + MONT_MAC(v0, v1, v2, u[19], m[24]); + MONT_MAC(w0, w1, w2, u[20], m[23]); + MONT_MAC(t0, t1, t2, u[21], m[22]); + MONT_MAC(v0, v1, v2, u[22], m[21]); + MONT_MAC(w0, w1, w2, u[23], m[20]); + MONT_MAC(t0, t1, t2, u[24], m[19]); + MONT_MAC(v0, v1, v2, u[25], m[18]); + MONT_MAC(w0, w1, w2, u[26], m[17]); + MONT_MAC(t0, t1, t2, u[27], m[16]); + MONT_MAC(v0, v1, v2, u[28], m[15]); + MONT_MAC(w0, w1, w2, u[29], m[14]); + MONT_MAC(t0, t1, t2, u[30], m[13]); + MONT_MAC(v0, v1, v2, u[31], m[12]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[11] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 44 */ + MONT_MUL(s0, s1, s2, a[13], a[31]); + MONT_MUL(c0, c1, c2, a[14], a[30]); + MONT_MAC(s0, s1, s2, a[15], a[29]); + MONT_MAC(c0, c1, c2, a[16], a[28]); + MONT_MAC(s0, s1, s2, a[17], a[27]); + MONT_MAC(c0, c1, c2, a[18], a[26]); + MONT_MAC(s0, s1, s2, a[19], a[25]); + MONT_MAC(c0, c1, c2, a[20], a[24]); + MONT_MAC(s0, s1, s2, a[21], a[23]); + MONT_MAC(t0, t1, t2, a[22], a[22]); + MONT_MUL(v0, v1, v2, u[13], m[31]); + MONT_MUL(w0, w1, w2, u[14], m[30]); + MONT_MAC(t0, t1, t2, u[15], m[29]); + MONT_MAC(v0, v1, v2, u[16], m[28]); + MONT_MAC(w0, w1, w2, u[17], m[27]); + MONT_MAC(t0, t1, t2, u[18], m[26]); + MONT_MAC(v0, v1, v2, u[19], m[25]); + MONT_MAC(w0, w1, w2, u[20], m[24]); + MONT_MAC(t0, t1, t2, u[21], m[23]); + MONT_MAC(v0, v1, v2, u[22], m[22]); + MONT_MAC(w0, w1, w2, u[23], m[21]); + MONT_MAC(t0, t1, t2, u[24], m[20]); + MONT_MAC(v0, v1, v2, u[25], m[19]); + MONT_MAC(w0, w1, w2, u[26], m[18]); + MONT_MAC(t0, t1, t2, u[27], m[17]); + MONT_MAC(v0, v1, v2, u[28], m[16]); + MONT_MAC(w0, w1, w2, u[29], m[15]); + MONT_MAC(t0, t1, t2, u[30], m[14]); + MONT_MAC(v0, v1, v2, u[31], m[13]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[12] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 45 */ + MONT_MUL(s0, s1, s2, a[14], a[31]); + MONT_MUL(c0, c1, c2, a[15], a[30]); + MONT_MAC(s0, s1, s2, a[16], a[29]); + MONT_MAC(c0, c1, c2, a[17], a[28]); + MONT_MAC(s0, s1, s2, a[18], a[27]); + MONT_MAC(c0, c1, c2, a[19], a[26]); + MONT_MAC(s0, s1, s2, a[20], a[25]); + MONT_MAC(c0, c1, c2, a[21], a[24]); + MONT_MAC(s0, s1, s2, a[22], a[23]); + MONT_MAC(t0, t1, t2, u[14], m[31]); + MONT_MUL(v0, v1, v2, u[15], m[30]); + MONT_MUL(w0, w1, w2, u[16], m[29]); + MONT_MAC(t0, t1, t2, u[17], m[28]); + MONT_MAC(v0, v1, v2, u[18], m[27]); + MONT_MAC(w0, w1, w2, u[19], m[26]); + MONT_MAC(t0, t1, t2, u[20], m[25]); + MONT_MAC(v0, v1, v2, u[21], m[24]); + MONT_MAC(w0, w1, w2, u[22], m[23]); + MONT_MAC(t0, t1, t2, u[23], m[22]); + MONT_MAC(v0, v1, v2, u[24], m[21]); + MONT_MAC(w0, w1, w2, u[25], m[20]); + MONT_MAC(t0, t1, t2, u[26], m[19]); + MONT_MAC(v0, v1, v2, u[27], m[18]); + MONT_MAC(w0, w1, w2, u[28], m[17]); + MONT_MAC(t0, t1, t2, u[29], m[16]); + MONT_MAC(v0, v1, v2, u[30], m[15]); + MONT_MAC(w0, w1, w2, u[31], m[14]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[13] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 46 */ + MONT_MUL(s0, s1, s2, a[15], a[31]); + MONT_MUL(c0, c1, c2, a[16], a[30]); + MONT_MAC(s0, s1, s2, a[17], a[29]); + MONT_MAC(c0, c1, c2, a[18], a[28]); + MONT_MAC(s0, s1, s2, a[19], a[27]); + MONT_MAC(c0, c1, c2, a[20], a[26]); + MONT_MAC(s0, s1, s2, a[21], a[25]); + MONT_MAC(c0, c1, c2, a[22], a[24]); + MONT_MAC(t0, t1, t2, a[23], a[23]); + MONT_MUL(v0, v1, v2, u[15], m[31]); + MONT_MUL(w0, w1, w2, u[16], m[30]); + MONT_MAC(t0, t1, t2, u[17], m[29]); + MONT_MAC(v0, v1, v2, u[18], m[28]); + MONT_MAC(w0, w1, w2, u[19], m[27]); + MONT_MAC(t0, t1, t2, u[20], m[26]); + MONT_MAC(v0, v1, v2, u[21], m[25]); + MONT_MAC(w0, w1, w2, u[22], m[24]); + MONT_MAC(t0, t1, t2, u[23], m[23]); + MONT_MAC(v0, v1, v2, u[24], m[22]); + MONT_MAC(w0, w1, w2, u[25], m[21]); + MONT_MAC(t0, t1, t2, u[26], m[20]); + MONT_MAC(v0, v1, v2, u[27], m[19]); + MONT_MAC(w0, w1, w2, u[28], m[18]); + MONT_MAC(t0, t1, t2, u[29], m[17]); + MONT_MAC(v0, v1, v2, u[30], m[16]); + MONT_MAC(w0, w1, w2, u[31], m[15]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[14] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 47 */ + MONT_MUL(s0, s1, s2, a[16], a[31]); + MONT_MUL(c0, c1, c2, a[17], a[30]); + MONT_MAC(s0, s1, s2, a[18], a[29]); + MONT_MAC(c0, c1, c2, a[19], a[28]); + MONT_MAC(s0, s1, s2, a[20], a[27]); + MONT_MAC(c0, c1, c2, a[21], a[26]); + MONT_MAC(s0, s1, s2, a[22], a[25]); + MONT_MAC(c0, c1, c2, a[23], a[24]); + MONT_MAC(t0, t1, t2, u[16], m[31]); + MONT_MUL(v0, v1, v2, u[17], m[30]); + MONT_MUL(w0, w1, w2, u[18], m[29]); + MONT_MAC(t0, t1, t2, u[19], m[28]); + MONT_MAC(v0, v1, v2, u[20], m[27]); + MONT_MAC(w0, w1, w2, u[21], m[26]); + MONT_MAC(t0, t1, t2, u[22], m[25]); + MONT_MAC(v0, v1, v2, u[23], m[24]); + MONT_MAC(w0, w1, w2, u[24], m[23]); + MONT_MAC(t0, t1, t2, u[25], m[22]); + MONT_MAC(v0, v1, v2, u[26], m[21]); + MONT_MAC(w0, w1, w2, u[27], m[20]); + MONT_MAC(t0, t1, t2, u[28], m[19]); + MONT_MAC(v0, v1, v2, u[29], m[18]); + MONT_MAC(w0, w1, w2, u[30], m[17]); + MONT_MAC(t0, t1, t2, u[31], m[16]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[15] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 48 */ + MONT_MUL(s0, s1, s2, a[17], a[31]); + MONT_MUL(c0, c1, c2, a[18], a[30]); + MONT_MAC(s0, s1, s2, a[19], a[29]); + MONT_MAC(c0, c1, c2, a[20], a[28]); + MONT_MAC(s0, s1, s2, a[21], a[27]); + MONT_MAC(c0, c1, c2, a[22], a[26]); + MONT_MAC(s0, s1, s2, a[23], a[25]); + MONT_MAC(t0, t1, t2, a[24], a[24]); + MONT_MUL(v0, v1, v2, u[17], m[31]); + MONT_MUL(w0, w1, w2, u[18], m[30]); + MONT_MAC(t0, t1, t2, u[19], m[29]); + MONT_MAC(v0, v1, v2, u[20], m[28]); + MONT_MAC(w0, w1, w2, u[21], m[27]); + MONT_MAC(t0, t1, t2, u[22], m[26]); + MONT_MAC(v0, v1, v2, u[23], m[25]); + MONT_MAC(w0, w1, w2, u[24], m[24]); + MONT_MAC(t0, t1, t2, u[25], m[23]); + MONT_MAC(v0, v1, v2, u[26], m[22]); + MONT_MAC(w0, w1, w2, u[27], m[21]); + MONT_MAC(t0, t1, t2, u[28], m[20]); + MONT_MAC(v0, v1, v2, u[29], m[19]); + MONT_MAC(w0, w1, w2, u[30], m[18]); + MONT_MAC(t0, t1, t2, u[31], m[17]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[16] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 49 */ + MONT_MUL(s0, s1, s2, a[18], a[31]); + MONT_MUL(c0, c1, c2, a[19], a[30]); + MONT_MAC(s0, s1, s2, a[20], a[29]); + MONT_MAC(c0, c1, c2, a[21], a[28]); + MONT_MAC(s0, s1, s2, a[22], a[27]); + MONT_MAC(c0, c1, c2, a[23], a[26]); + MONT_MAC(s0, s1, s2, a[24], a[25]); + MONT_MAC(t0, t1, t2, u[18], m[31]); + MONT_MUL(v0, v1, v2, u[19], m[30]); + MONT_MUL(w0, w1, w2, u[20], m[29]); + MONT_MAC(t0, t1, t2, u[21], m[28]); + MONT_MAC(v0, v1, v2, u[22], m[27]); + MONT_MAC(w0, w1, w2, u[23], m[26]); + MONT_MAC(t0, t1, t2, u[24], m[25]); + MONT_MAC(v0, v1, v2, u[25], m[24]); + MONT_MAC(w0, w1, w2, u[26], m[23]); + MONT_MAC(t0, t1, t2, u[27], m[22]); + MONT_MAC(v0, v1, v2, u[28], m[21]); + MONT_MAC(w0, w1, w2, u[29], m[20]); + MONT_MAC(t0, t1, t2, u[30], m[19]); + MONT_MAC(v0, v1, v2, u[31], m[18]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[17] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 50 */ + MONT_MUL(s0, s1, s2, a[19], a[31]); + MONT_MUL(c0, c1, c2, a[20], a[30]); + MONT_MAC(s0, s1, s2, a[21], a[29]); + MONT_MAC(c0, c1, c2, a[22], a[28]); + MONT_MAC(s0, s1, s2, a[23], a[27]); + MONT_MAC(c0, c1, c2, a[24], a[26]); + MONT_MAC(t0, t1, t2, a[25], a[25]); + MONT_MUL(v0, v1, v2, u[19], m[31]); + MONT_MUL(w0, w1, w2, u[20], m[30]); + MONT_MAC(t0, t1, t2, u[21], m[29]); + MONT_MAC(v0, v1, v2, u[22], m[28]); + MONT_MAC(w0, w1, w2, u[23], m[27]); + MONT_MAC(t0, t1, t2, u[24], m[26]); + MONT_MAC(v0, v1, v2, u[25], m[25]); + MONT_MAC(w0, w1, w2, u[26], m[24]); + MONT_MAC(t0, t1, t2, u[27], m[23]); + MONT_MAC(v0, v1, v2, u[28], m[22]); + MONT_MAC(w0, w1, w2, u[29], m[21]); + MONT_MAC(t0, t1, t2, u[30], m[20]); + MONT_MAC(v0, v1, v2, u[31], m[19]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[18] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 51 */ + MONT_MUL(s0, s1, s2, a[20], a[31]); + MONT_MUL(c0, c1, c2, a[21], a[30]); + MONT_MAC(s0, s1, s2, a[22], a[29]); + MONT_MAC(c0, c1, c2, a[23], a[28]); + MONT_MAC(s0, s1, s2, a[24], a[27]); + MONT_MAC(c0, c1, c2, a[25], a[26]); + MONT_MAC(t0, t1, t2, u[20], m[31]); + MONT_MUL(v0, v1, v2, u[21], m[30]); + MONT_MUL(w0, w1, w2, u[22], m[29]); + MONT_MAC(t0, t1, t2, u[23], m[28]); + MONT_MAC(v0, v1, v2, u[24], m[27]); + MONT_MAC(w0, w1, w2, u[25], m[26]); + MONT_MAC(t0, t1, t2, u[26], m[25]); + MONT_MAC(v0, v1, v2, u[27], m[24]); + MONT_MAC(w0, w1, w2, u[28], m[23]); + MONT_MAC(t0, t1, t2, u[29], m[22]); + MONT_MAC(v0, v1, v2, u[30], m[21]); + MONT_MAC(w0, w1, w2, u[31], m[20]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[19] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 52 */ + MONT_MUL(s0, s1, s2, a[21], a[31]); + MONT_MUL(c0, c1, c2, a[22], a[30]); + MONT_MAC(s0, s1, s2, a[23], a[29]); + MONT_MAC(c0, c1, c2, a[24], a[28]); + MONT_MAC(s0, s1, s2, a[25], a[27]); + MONT_MAC(t0, t1, t2, a[26], a[26]); + MONT_MUL(v0, v1, v2, u[21], m[31]); + MONT_MUL(w0, w1, w2, u[22], m[30]); + MONT_MAC(t0, t1, t2, u[23], m[29]); + MONT_MAC(v0, v1, v2, u[24], m[28]); + MONT_MAC(w0, w1, w2, u[25], m[27]); + MONT_MAC(t0, t1, t2, u[26], m[26]); + MONT_MAC(v0, v1, v2, u[27], m[25]); + MONT_MAC(w0, w1, w2, u[28], m[24]); + MONT_MAC(t0, t1, t2, u[29], m[23]); + MONT_MAC(v0, v1, v2, u[30], m[22]); + MONT_MAC(w0, w1, w2, u[31], m[21]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[20] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 53 */ + MONT_MUL(s0, s1, s2, a[22], a[31]); + MONT_MUL(c0, c1, c2, a[23], a[30]); + MONT_MAC(s0, s1, s2, a[24], a[29]); + MONT_MAC(c0, c1, c2, a[25], a[28]); + MONT_MAC(s0, s1, s2, a[26], a[27]); + MONT_MAC(t0, t1, t2, u[22], m[31]); + MONT_MUL(v0, v1, v2, u[23], m[30]); + MONT_MUL(w0, w1, w2, u[24], m[29]); + MONT_MAC(t0, t1, t2, u[25], m[28]); + MONT_MAC(v0, v1, v2, u[26], m[27]); + MONT_MAC(w0, w1, w2, u[27], m[26]); + MONT_MAC(t0, t1, t2, u[28], m[25]); + MONT_MAC(v0, v1, v2, u[29], m[24]); + MONT_MAC(w0, w1, w2, u[30], m[23]); + MONT_MAC(t0, t1, t2, u[31], m[22]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[21] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 54 */ + MONT_MUL(s0, s1, s2, a[23], a[31]); + MONT_MUL(c0, c1, c2, a[24], a[30]); + MONT_MAC(s0, s1, s2, a[25], a[29]); + MONT_MAC(c0, c1, c2, a[26], a[28]); + MONT_MAC(t0, t1, t2, a[27], a[27]); + MONT_MUL(v0, v1, v2, u[23], m[31]); + MONT_MUL(w0, w1, w2, u[24], m[30]); + MONT_MAC(t0, t1, t2, u[25], m[29]); + MONT_MAC(v0, v1, v2, u[26], m[28]); + MONT_MAC(w0, w1, w2, u[27], m[27]); + MONT_MAC(t0, t1, t2, u[28], m[26]); + MONT_MAC(v0, v1, v2, u[29], m[25]); + MONT_MAC(w0, w1, w2, u[30], m[24]); + MONT_MAC(t0, t1, t2, u[31], m[23]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[22] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 55 */ + MONT_MUL(s0, s1, s2, a[24], a[31]); + MONT_MUL(c0, c1, c2, a[25], a[30]); + MONT_MAC(s0, s1, s2, a[26], a[29]); + MONT_MAC(c0, c1, c2, a[27], a[28]); + MONT_MAC(t0, t1, t2, u[24], m[31]); + MONT_MUL(v0, v1, v2, u[25], m[30]); + MONT_MUL(w0, w1, w2, u[26], m[29]); + MONT_MAC(t0, t1, t2, u[27], m[28]); + MONT_MAC(v0, v1, v2, u[28], m[27]); + MONT_MAC(w0, w1, w2, u[29], m[26]); + MONT_MAC(t0, t1, t2, u[30], m[25]); + MONT_MAC(v0, v1, v2, u[31], m[24]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[23] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 56 */ + MONT_MUL(s0, s1, s2, a[25], a[31]); + MONT_MUL(c0, c1, c2, a[26], a[30]); + MONT_MAC(s0, s1, s2, a[27], a[29]); + MONT_MAC(t0, t1, t2, a[28], a[28]); + MONT_MUL(v0, v1, v2, u[25], m[31]); + MONT_MUL(w0, w1, w2, u[26], m[30]); + MONT_MAC(t0, t1, t2, u[27], m[29]); + MONT_MAC(v0, v1, v2, u[28], m[28]); + MONT_MAC(w0, w1, w2, u[29], m[27]); + MONT_MAC(t0, t1, t2, u[30], m[26]); + MONT_MAC(v0, v1, v2, u[31], m[25]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[24] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 57 */ + MONT_MUL(s0, s1, s2, a[26], a[31]); + MONT_MUL(c0, c1, c2, a[27], a[30]); + MONT_MAC(s0, s1, s2, a[28], a[29]); + MONT_MAC(t0, t1, t2, u[26], m[31]); + MONT_MUL(v0, v1, v2, u[27], m[30]); + MONT_MUL(w0, w1, w2, u[28], m[29]); + MONT_MAC(t0, t1, t2, u[29], m[28]); + MONT_MAC(v0, v1, v2, u[30], m[27]); + MONT_MAC(w0, w1, w2, u[31], m[26]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[25] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 58 */ + MONT_MUL(s0, s1, s2, a[27], a[31]); + MONT_MUL(c0, c1, c2, a[28], a[30]); + MONT_MAC(t0, t1, t2, a[29], a[29]); + MONT_MUL(v0, v1, v2, u[27], m[31]); + MONT_MUL(w0, w1, w2, u[28], m[30]); + MONT_MAC(t0, t1, t2, u[29], m[29]); + MONT_MAC(v0, v1, v2, u[30], m[28]); + MONT_MAC(w0, w1, w2, u[31], m[27]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[26] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 59 */ + MONT_MUL(s0, s1, s2, a[28], a[31]); + MONT_MUL(c0, c1, c2, a[29], a[30]); + MONT_MAC(t0, t1, t2, u[28], m[31]); + MONT_MUL(v0, v1, v2, u[29], m[30]); + MONT_MUL(w0, w1, w2, u[30], m[29]); + MONT_MAC(t0, t1, t2, u[31], m[28]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[27] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 60 */ + MONT_MUL(s0, s1, s2, a[29], a[31]); + MONT_MAC(t0, t1, t2, a[30], a[30]); + MONT_MUL(v0, v1, v2, u[29], m[31]); + MONT_MUL(w0, w1, w2, u[30], m[30]); + MONT_MAC(t0, t1, t2, u[31], m[29]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[28] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 61 */ + MONT_MUL(s0, s1, s2, a[30], a[31]); + MONT_MAC(t0, t1, t2, u[30], m[31]); + MONT_MUL(v0, v1, v2, u[31], m[30]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[29] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 62 */ + MONT_MAC(t0, t1, t2, a[31], a[31]); + MONT_MUL(v0, v1, v2, u[31], m[31]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + z[30] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + z[31] = t0; + + /* under 2m, with t1 the bit above n limbs: at most one subtraction */ + borrow = sub_n(r, z, m, 32); + take = t1 | (borrow ^ 1); + select_n(r, r, z, take & 1, 32); +} + +/* r = a * a * R^-1 mod m, for a under m; t is 2n limbs of scratch */ +static void crypton_mont_sqr48(limb_t *r, const limb_t *a, + const limb_t *m, limb_t n0, limb_t *t) +{ + limb_t *u = t, *z = t + 48; + limb_t t0 = 0, t1 = 0, t2 = 0, v0, v1, v2, w0, w1, w2, borrow, take; + limb_t s0, s1, s2, c0, c1, c2; + + /* column 0 */ + MONT_MAC(t0, t1, t2, a[0], a[0]); + u[0] = t0 * n0; + MONT_MAC(t0, t1, t2, u[0], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 1 */ + MONT_MUL(s0, s1, s2, a[0], a[1]); + MONT_MAC(t0, t1, t2, u[0], m[1]); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[1] = t0 * n0; + MONT_MAC(t0, t1, t2, u[1], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 2 */ + MONT_MUL(s0, s1, s2, a[0], a[2]); + MONT_MAC(t0, t1, t2, a[1], a[1]); + MONT_MUL(v0, v1, v2, u[0], m[2]); + MONT_MUL(w0, w1, w2, u[1], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[2] = t0 * n0; + MONT_MAC(t0, t1, t2, u[2], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 3 */ + MONT_MUL(s0, s1, s2, a[0], a[3]); + MONT_MUL(c0, c1, c2, a[1], a[2]); + MONT_MAC(t0, t1, t2, u[0], m[3]); + MONT_MUL(v0, v1, v2, u[1], m[2]); + MONT_MUL(w0, w1, w2, u[2], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[3] = t0 * n0; + MONT_MAC(t0, t1, t2, u[3], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 4 */ + MONT_MUL(s0, s1, s2, a[0], a[4]); + MONT_MUL(c0, c1, c2, a[1], a[3]); + MONT_MAC(t0, t1, t2, a[2], a[2]); + MONT_MUL(v0, v1, v2, u[0], m[4]); + MONT_MUL(w0, w1, w2, u[1], m[3]); + MONT_MAC(t0, t1, t2, u[2], m[2]); + MONT_MAC(v0, v1, v2, u[3], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[4] = t0 * n0; + MONT_MAC(t0, t1, t2, u[4], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 5 */ + MONT_MUL(s0, s1, s2, a[0], a[5]); + MONT_MUL(c0, c1, c2, a[1], a[4]); + MONT_MAC(s0, s1, s2, a[2], a[3]); + MONT_MAC(t0, t1, t2, u[0], m[5]); + MONT_MUL(v0, v1, v2, u[1], m[4]); + MONT_MUL(w0, w1, w2, u[2], m[3]); + MONT_MAC(t0, t1, t2, u[3], m[2]); + MONT_MAC(v0, v1, v2, u[4], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[5] = t0 * n0; + MONT_MAC(t0, t1, t2, u[5], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 6 */ + MONT_MUL(s0, s1, s2, a[0], a[6]); + MONT_MUL(c0, c1, c2, a[1], a[5]); + MONT_MAC(s0, s1, s2, a[2], a[4]); + MONT_MAC(t0, t1, t2, a[3], a[3]); + MONT_MUL(v0, v1, v2, u[0], m[6]); + MONT_MUL(w0, w1, w2, u[1], m[5]); + MONT_MAC(t0, t1, t2, u[2], m[4]); + MONT_MAC(v0, v1, v2, u[3], m[3]); + MONT_MAC(w0, w1, w2, u[4], m[2]); + MONT_MAC(t0, t1, t2, u[5], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[6] = t0 * n0; + MONT_MAC(t0, t1, t2, u[6], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 7 */ + MONT_MUL(s0, s1, s2, a[0], a[7]); + MONT_MUL(c0, c1, c2, a[1], a[6]); + MONT_MAC(s0, s1, s2, a[2], a[5]); + MONT_MAC(c0, c1, c2, a[3], a[4]); + MONT_MAC(t0, t1, t2, u[0], m[7]); + MONT_MUL(v0, v1, v2, u[1], m[6]); + MONT_MUL(w0, w1, w2, u[2], m[5]); + MONT_MAC(t0, t1, t2, u[3], m[4]); + MONT_MAC(v0, v1, v2, u[4], m[3]); + MONT_MAC(w0, w1, w2, u[5], m[2]); + MONT_MAC(t0, t1, t2, u[6], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[7] = t0 * n0; + MONT_MAC(t0, t1, t2, u[7], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 8 */ + MONT_MUL(s0, s1, s2, a[0], a[8]); + MONT_MUL(c0, c1, c2, a[1], a[7]); + MONT_MAC(s0, s1, s2, a[2], a[6]); + MONT_MAC(c0, c1, c2, a[3], a[5]); + MONT_MAC(t0, t1, t2, a[4], a[4]); + MONT_MUL(v0, v1, v2, u[0], m[8]); + MONT_MUL(w0, w1, w2, u[1], m[7]); + MONT_MAC(t0, t1, t2, u[2], m[6]); + MONT_MAC(v0, v1, v2, u[3], m[5]); + MONT_MAC(w0, w1, w2, u[4], m[4]); + MONT_MAC(t0, t1, t2, u[5], m[3]); + MONT_MAC(v0, v1, v2, u[6], m[2]); + MONT_MAC(w0, w1, w2, u[7], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[8] = t0 * n0; + MONT_MAC(t0, t1, t2, u[8], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 9 */ + MONT_MUL(s0, s1, s2, a[0], a[9]); + MONT_MUL(c0, c1, c2, a[1], a[8]); + MONT_MAC(s0, s1, s2, a[2], a[7]); + MONT_MAC(c0, c1, c2, a[3], a[6]); + MONT_MAC(s0, s1, s2, a[4], a[5]); + MONT_MAC(t0, t1, t2, u[0], m[9]); + MONT_MUL(v0, v1, v2, u[1], m[8]); + MONT_MUL(w0, w1, w2, u[2], m[7]); + MONT_MAC(t0, t1, t2, u[3], m[6]); + MONT_MAC(v0, v1, v2, u[4], m[5]); + MONT_MAC(w0, w1, w2, u[5], m[4]); + MONT_MAC(t0, t1, t2, u[6], m[3]); + MONT_MAC(v0, v1, v2, u[7], m[2]); + MONT_MAC(w0, w1, w2, u[8], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[9] = t0 * n0; + MONT_MAC(t0, t1, t2, u[9], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 10 */ + MONT_MUL(s0, s1, s2, a[0], a[10]); + MONT_MUL(c0, c1, c2, a[1], a[9]); + MONT_MAC(s0, s1, s2, a[2], a[8]); + MONT_MAC(c0, c1, c2, a[3], a[7]); + MONT_MAC(s0, s1, s2, a[4], a[6]); + MONT_MAC(t0, t1, t2, a[5], a[5]); + MONT_MUL(v0, v1, v2, u[0], m[10]); + MONT_MUL(w0, w1, w2, u[1], m[9]); + MONT_MAC(t0, t1, t2, u[2], m[8]); + MONT_MAC(v0, v1, v2, u[3], m[7]); + MONT_MAC(w0, w1, w2, u[4], m[6]); + MONT_MAC(t0, t1, t2, u[5], m[5]); + MONT_MAC(v0, v1, v2, u[6], m[4]); + MONT_MAC(w0, w1, w2, u[7], m[3]); + MONT_MAC(t0, t1, t2, u[8], m[2]); + MONT_MAC(v0, v1, v2, u[9], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[10] = t0 * n0; + MONT_MAC(t0, t1, t2, u[10], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 11 */ + MONT_MUL(s0, s1, s2, a[0], a[11]); + MONT_MUL(c0, c1, c2, a[1], a[10]); + MONT_MAC(s0, s1, s2, a[2], a[9]); + MONT_MAC(c0, c1, c2, a[3], a[8]); + MONT_MAC(s0, s1, s2, a[4], a[7]); + MONT_MAC(c0, c1, c2, a[5], a[6]); + MONT_MAC(t0, t1, t2, u[0], m[11]); + MONT_MUL(v0, v1, v2, u[1], m[10]); + MONT_MUL(w0, w1, w2, u[2], m[9]); + MONT_MAC(t0, t1, t2, u[3], m[8]); + MONT_MAC(v0, v1, v2, u[4], m[7]); + MONT_MAC(w0, w1, w2, u[5], m[6]); + MONT_MAC(t0, t1, t2, u[6], m[5]); + MONT_MAC(v0, v1, v2, u[7], m[4]); + MONT_MAC(w0, w1, w2, u[8], m[3]); + MONT_MAC(t0, t1, t2, u[9], m[2]); + MONT_MAC(v0, v1, v2, u[10], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[11] = t0 * n0; + MONT_MAC(t0, t1, t2, u[11], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 12 */ + MONT_MUL(s0, s1, s2, a[0], a[12]); + MONT_MUL(c0, c1, c2, a[1], a[11]); + MONT_MAC(s0, s1, s2, a[2], a[10]); + MONT_MAC(c0, c1, c2, a[3], a[9]); + MONT_MAC(s0, s1, s2, a[4], a[8]); + MONT_MAC(c0, c1, c2, a[5], a[7]); + MONT_MAC(t0, t1, t2, a[6], a[6]); + MONT_MUL(v0, v1, v2, u[0], m[12]); + MONT_MUL(w0, w1, w2, u[1], m[11]); + MONT_MAC(t0, t1, t2, u[2], m[10]); + MONT_MAC(v0, v1, v2, u[3], m[9]); + MONT_MAC(w0, w1, w2, u[4], m[8]); + MONT_MAC(t0, t1, t2, u[5], m[7]); + MONT_MAC(v0, v1, v2, u[6], m[6]); + MONT_MAC(w0, w1, w2, u[7], m[5]); + MONT_MAC(t0, t1, t2, u[8], m[4]); + MONT_MAC(v0, v1, v2, u[9], m[3]); + MONT_MAC(w0, w1, w2, u[10], m[2]); + MONT_MAC(t0, t1, t2, u[11], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[12] = t0 * n0; + MONT_MAC(t0, t1, t2, u[12], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 13 */ + MONT_MUL(s0, s1, s2, a[0], a[13]); + MONT_MUL(c0, c1, c2, a[1], a[12]); + MONT_MAC(s0, s1, s2, a[2], a[11]); + MONT_MAC(c0, c1, c2, a[3], a[10]); + MONT_MAC(s0, s1, s2, a[4], a[9]); + MONT_MAC(c0, c1, c2, a[5], a[8]); + MONT_MAC(s0, s1, s2, a[6], a[7]); + MONT_MAC(t0, t1, t2, u[0], m[13]); + MONT_MUL(v0, v1, v2, u[1], m[12]); + MONT_MUL(w0, w1, w2, u[2], m[11]); + MONT_MAC(t0, t1, t2, u[3], m[10]); + MONT_MAC(v0, v1, v2, u[4], m[9]); + MONT_MAC(w0, w1, w2, u[5], m[8]); + MONT_MAC(t0, t1, t2, u[6], m[7]); + MONT_MAC(v0, v1, v2, u[7], m[6]); + MONT_MAC(w0, w1, w2, u[8], m[5]); + MONT_MAC(t0, t1, t2, u[9], m[4]); + MONT_MAC(v0, v1, v2, u[10], m[3]); + MONT_MAC(w0, w1, w2, u[11], m[2]); + MONT_MAC(t0, t1, t2, u[12], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[13] = t0 * n0; + MONT_MAC(t0, t1, t2, u[13], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 14 */ + MONT_MUL(s0, s1, s2, a[0], a[14]); + MONT_MUL(c0, c1, c2, a[1], a[13]); + MONT_MAC(s0, s1, s2, a[2], a[12]); + MONT_MAC(c0, c1, c2, a[3], a[11]); + MONT_MAC(s0, s1, s2, a[4], a[10]); + MONT_MAC(c0, c1, c2, a[5], a[9]); + MONT_MAC(s0, s1, s2, a[6], a[8]); + MONT_MAC(t0, t1, t2, a[7], a[7]); + MONT_MUL(v0, v1, v2, u[0], m[14]); + MONT_MUL(w0, w1, w2, u[1], m[13]); + MONT_MAC(t0, t1, t2, u[2], m[12]); + MONT_MAC(v0, v1, v2, u[3], m[11]); + MONT_MAC(w0, w1, w2, u[4], m[10]); + MONT_MAC(t0, t1, t2, u[5], m[9]); + MONT_MAC(v0, v1, v2, u[6], m[8]); + MONT_MAC(w0, w1, w2, u[7], m[7]); + MONT_MAC(t0, t1, t2, u[8], m[6]); + MONT_MAC(v0, v1, v2, u[9], m[5]); + MONT_MAC(w0, w1, w2, u[10], m[4]); + MONT_MAC(t0, t1, t2, u[11], m[3]); + MONT_MAC(v0, v1, v2, u[12], m[2]); + MONT_MAC(w0, w1, w2, u[13], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[14] = t0 * n0; + MONT_MAC(t0, t1, t2, u[14], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 15 */ + MONT_MUL(s0, s1, s2, a[0], a[15]); + MONT_MUL(c0, c1, c2, a[1], a[14]); + MONT_MAC(s0, s1, s2, a[2], a[13]); + MONT_MAC(c0, c1, c2, a[3], a[12]); + MONT_MAC(s0, s1, s2, a[4], a[11]); + MONT_MAC(c0, c1, c2, a[5], a[10]); + MONT_MAC(s0, s1, s2, a[6], a[9]); + MONT_MAC(c0, c1, c2, a[7], a[8]); + MONT_MAC(t0, t1, t2, u[0], m[15]); + MONT_MUL(v0, v1, v2, u[1], m[14]); + MONT_MUL(w0, w1, w2, u[2], m[13]); + MONT_MAC(t0, t1, t2, u[3], m[12]); + MONT_MAC(v0, v1, v2, u[4], m[11]); + MONT_MAC(w0, w1, w2, u[5], m[10]); + MONT_MAC(t0, t1, t2, u[6], m[9]); + MONT_MAC(v0, v1, v2, u[7], m[8]); + MONT_MAC(w0, w1, w2, u[8], m[7]); + MONT_MAC(t0, t1, t2, u[9], m[6]); + MONT_MAC(v0, v1, v2, u[10], m[5]); + MONT_MAC(w0, w1, w2, u[11], m[4]); + MONT_MAC(t0, t1, t2, u[12], m[3]); + MONT_MAC(v0, v1, v2, u[13], m[2]); + MONT_MAC(w0, w1, w2, u[14], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[15] = t0 * n0; + MONT_MAC(t0, t1, t2, u[15], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 16 */ + MONT_MUL(s0, s1, s2, a[0], a[16]); + MONT_MUL(c0, c1, c2, a[1], a[15]); + MONT_MAC(s0, s1, s2, a[2], a[14]); + MONT_MAC(c0, c1, c2, a[3], a[13]); + MONT_MAC(s0, s1, s2, a[4], a[12]); + MONT_MAC(c0, c1, c2, a[5], a[11]); + MONT_MAC(s0, s1, s2, a[6], a[10]); + MONT_MAC(c0, c1, c2, a[7], a[9]); + MONT_MAC(t0, t1, t2, a[8], a[8]); + MONT_MUL(v0, v1, v2, u[0], m[16]); + MONT_MUL(w0, w1, w2, u[1], m[15]); + MONT_MAC(t0, t1, t2, u[2], m[14]); + MONT_MAC(v0, v1, v2, u[3], m[13]); + MONT_MAC(w0, w1, w2, u[4], m[12]); + MONT_MAC(t0, t1, t2, u[5], m[11]); + MONT_MAC(v0, v1, v2, u[6], m[10]); + MONT_MAC(w0, w1, w2, u[7], m[9]); + MONT_MAC(t0, t1, t2, u[8], m[8]); + MONT_MAC(v0, v1, v2, u[9], m[7]); + MONT_MAC(w0, w1, w2, u[10], m[6]); + MONT_MAC(t0, t1, t2, u[11], m[5]); + MONT_MAC(v0, v1, v2, u[12], m[4]); + MONT_MAC(w0, w1, w2, u[13], m[3]); + MONT_MAC(t0, t1, t2, u[14], m[2]); + MONT_MAC(v0, v1, v2, u[15], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[16] = t0 * n0; + MONT_MAC(t0, t1, t2, u[16], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 17 */ + MONT_MUL(s0, s1, s2, a[0], a[17]); + MONT_MUL(c0, c1, c2, a[1], a[16]); + MONT_MAC(s0, s1, s2, a[2], a[15]); + MONT_MAC(c0, c1, c2, a[3], a[14]); + MONT_MAC(s0, s1, s2, a[4], a[13]); + MONT_MAC(c0, c1, c2, a[5], a[12]); + MONT_MAC(s0, s1, s2, a[6], a[11]); + MONT_MAC(c0, c1, c2, a[7], a[10]); + MONT_MAC(s0, s1, s2, a[8], a[9]); + MONT_MAC(t0, t1, t2, u[0], m[17]); + MONT_MUL(v0, v1, v2, u[1], m[16]); + MONT_MUL(w0, w1, w2, u[2], m[15]); + MONT_MAC(t0, t1, t2, u[3], m[14]); + MONT_MAC(v0, v1, v2, u[4], m[13]); + MONT_MAC(w0, w1, w2, u[5], m[12]); + MONT_MAC(t0, t1, t2, u[6], m[11]); + MONT_MAC(v0, v1, v2, u[7], m[10]); + MONT_MAC(w0, w1, w2, u[8], m[9]); + MONT_MAC(t0, t1, t2, u[9], m[8]); + MONT_MAC(v0, v1, v2, u[10], m[7]); + MONT_MAC(w0, w1, w2, u[11], m[6]); + MONT_MAC(t0, t1, t2, u[12], m[5]); + MONT_MAC(v0, v1, v2, u[13], m[4]); + MONT_MAC(w0, w1, w2, u[14], m[3]); + MONT_MAC(t0, t1, t2, u[15], m[2]); + MONT_MAC(v0, v1, v2, u[16], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[17] = t0 * n0; + MONT_MAC(t0, t1, t2, u[17], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 18 */ + MONT_MUL(s0, s1, s2, a[0], a[18]); + MONT_MUL(c0, c1, c2, a[1], a[17]); + MONT_MAC(s0, s1, s2, a[2], a[16]); + MONT_MAC(c0, c1, c2, a[3], a[15]); + MONT_MAC(s0, s1, s2, a[4], a[14]); + MONT_MAC(c0, c1, c2, a[5], a[13]); + MONT_MAC(s0, s1, s2, a[6], a[12]); + MONT_MAC(c0, c1, c2, a[7], a[11]); + MONT_MAC(s0, s1, s2, a[8], a[10]); + MONT_MAC(t0, t1, t2, a[9], a[9]); + MONT_MUL(v0, v1, v2, u[0], m[18]); + MONT_MUL(w0, w1, w2, u[1], m[17]); + MONT_MAC(t0, t1, t2, u[2], m[16]); + MONT_MAC(v0, v1, v2, u[3], m[15]); + MONT_MAC(w0, w1, w2, u[4], m[14]); + MONT_MAC(t0, t1, t2, u[5], m[13]); + MONT_MAC(v0, v1, v2, u[6], m[12]); + MONT_MAC(w0, w1, w2, u[7], m[11]); + MONT_MAC(t0, t1, t2, u[8], m[10]); + MONT_MAC(v0, v1, v2, u[9], m[9]); + MONT_MAC(w0, w1, w2, u[10], m[8]); + MONT_MAC(t0, t1, t2, u[11], m[7]); + MONT_MAC(v0, v1, v2, u[12], m[6]); + MONT_MAC(w0, w1, w2, u[13], m[5]); + MONT_MAC(t0, t1, t2, u[14], m[4]); + MONT_MAC(v0, v1, v2, u[15], m[3]); + MONT_MAC(w0, w1, w2, u[16], m[2]); + MONT_MAC(t0, t1, t2, u[17], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[18] = t0 * n0; + MONT_MAC(t0, t1, t2, u[18], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 19 */ + MONT_MUL(s0, s1, s2, a[0], a[19]); + MONT_MUL(c0, c1, c2, a[1], a[18]); + MONT_MAC(s0, s1, s2, a[2], a[17]); + MONT_MAC(c0, c1, c2, a[3], a[16]); + MONT_MAC(s0, s1, s2, a[4], a[15]); + MONT_MAC(c0, c1, c2, a[5], a[14]); + MONT_MAC(s0, s1, s2, a[6], a[13]); + MONT_MAC(c0, c1, c2, a[7], a[12]); + MONT_MAC(s0, s1, s2, a[8], a[11]); + MONT_MAC(c0, c1, c2, a[9], a[10]); + MONT_MAC(t0, t1, t2, u[0], m[19]); + MONT_MUL(v0, v1, v2, u[1], m[18]); + MONT_MUL(w0, w1, w2, u[2], m[17]); + MONT_MAC(t0, t1, t2, u[3], m[16]); + MONT_MAC(v0, v1, v2, u[4], m[15]); + MONT_MAC(w0, w1, w2, u[5], m[14]); + MONT_MAC(t0, t1, t2, u[6], m[13]); + MONT_MAC(v0, v1, v2, u[7], m[12]); + MONT_MAC(w0, w1, w2, u[8], m[11]); + MONT_MAC(t0, t1, t2, u[9], m[10]); + MONT_MAC(v0, v1, v2, u[10], m[9]); + MONT_MAC(w0, w1, w2, u[11], m[8]); + MONT_MAC(t0, t1, t2, u[12], m[7]); + MONT_MAC(v0, v1, v2, u[13], m[6]); + MONT_MAC(w0, w1, w2, u[14], m[5]); + MONT_MAC(t0, t1, t2, u[15], m[4]); + MONT_MAC(v0, v1, v2, u[16], m[3]); + MONT_MAC(w0, w1, w2, u[17], m[2]); + MONT_MAC(t0, t1, t2, u[18], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[19] = t0 * n0; + MONT_MAC(t0, t1, t2, u[19], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 20 */ + MONT_MUL(s0, s1, s2, a[0], a[20]); + MONT_MUL(c0, c1, c2, a[1], a[19]); + MONT_MAC(s0, s1, s2, a[2], a[18]); + MONT_MAC(c0, c1, c2, a[3], a[17]); + MONT_MAC(s0, s1, s2, a[4], a[16]); + MONT_MAC(c0, c1, c2, a[5], a[15]); + MONT_MAC(s0, s1, s2, a[6], a[14]); + MONT_MAC(c0, c1, c2, a[7], a[13]); + MONT_MAC(s0, s1, s2, a[8], a[12]); + MONT_MAC(c0, c1, c2, a[9], a[11]); + MONT_MAC(t0, t1, t2, a[10], a[10]); + MONT_MUL(v0, v1, v2, u[0], m[20]); + MONT_MUL(w0, w1, w2, u[1], m[19]); + MONT_MAC(t0, t1, t2, u[2], m[18]); + MONT_MAC(v0, v1, v2, u[3], m[17]); + MONT_MAC(w0, w1, w2, u[4], m[16]); + MONT_MAC(t0, t1, t2, u[5], m[15]); + MONT_MAC(v0, v1, v2, u[6], m[14]); + MONT_MAC(w0, w1, w2, u[7], m[13]); + MONT_MAC(t0, t1, t2, u[8], m[12]); + MONT_MAC(v0, v1, v2, u[9], m[11]); + MONT_MAC(w0, w1, w2, u[10], m[10]); + MONT_MAC(t0, t1, t2, u[11], m[9]); + MONT_MAC(v0, v1, v2, u[12], m[8]); + MONT_MAC(w0, w1, w2, u[13], m[7]); + MONT_MAC(t0, t1, t2, u[14], m[6]); + MONT_MAC(v0, v1, v2, u[15], m[5]); + MONT_MAC(w0, w1, w2, u[16], m[4]); + MONT_MAC(t0, t1, t2, u[17], m[3]); + MONT_MAC(v0, v1, v2, u[18], m[2]); + MONT_MAC(w0, w1, w2, u[19], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[20] = t0 * n0; + MONT_MAC(t0, t1, t2, u[20], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 21 */ + MONT_MUL(s0, s1, s2, a[0], a[21]); + MONT_MUL(c0, c1, c2, a[1], a[20]); + MONT_MAC(s0, s1, s2, a[2], a[19]); + MONT_MAC(c0, c1, c2, a[3], a[18]); + MONT_MAC(s0, s1, s2, a[4], a[17]); + MONT_MAC(c0, c1, c2, a[5], a[16]); + MONT_MAC(s0, s1, s2, a[6], a[15]); + MONT_MAC(c0, c1, c2, a[7], a[14]); + MONT_MAC(s0, s1, s2, a[8], a[13]); + MONT_MAC(c0, c1, c2, a[9], a[12]); + MONT_MAC(s0, s1, s2, a[10], a[11]); + MONT_MAC(t0, t1, t2, u[0], m[21]); + MONT_MUL(v0, v1, v2, u[1], m[20]); + MONT_MUL(w0, w1, w2, u[2], m[19]); + MONT_MAC(t0, t1, t2, u[3], m[18]); + MONT_MAC(v0, v1, v2, u[4], m[17]); + MONT_MAC(w0, w1, w2, u[5], m[16]); + MONT_MAC(t0, t1, t2, u[6], m[15]); + MONT_MAC(v0, v1, v2, u[7], m[14]); + MONT_MAC(w0, w1, w2, u[8], m[13]); + MONT_MAC(t0, t1, t2, u[9], m[12]); + MONT_MAC(v0, v1, v2, u[10], m[11]); + MONT_MAC(w0, w1, w2, u[11], m[10]); + MONT_MAC(t0, t1, t2, u[12], m[9]); + MONT_MAC(v0, v1, v2, u[13], m[8]); + MONT_MAC(w0, w1, w2, u[14], m[7]); + MONT_MAC(t0, t1, t2, u[15], m[6]); + MONT_MAC(v0, v1, v2, u[16], m[5]); + MONT_MAC(w0, w1, w2, u[17], m[4]); + MONT_MAC(t0, t1, t2, u[18], m[3]); + MONT_MAC(v0, v1, v2, u[19], m[2]); + MONT_MAC(w0, w1, w2, u[20], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[21] = t0 * n0; + MONT_MAC(t0, t1, t2, u[21], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 22 */ + MONT_MUL(s0, s1, s2, a[0], a[22]); + MONT_MUL(c0, c1, c2, a[1], a[21]); + MONT_MAC(s0, s1, s2, a[2], a[20]); + MONT_MAC(c0, c1, c2, a[3], a[19]); + MONT_MAC(s0, s1, s2, a[4], a[18]); + MONT_MAC(c0, c1, c2, a[5], a[17]); + MONT_MAC(s0, s1, s2, a[6], a[16]); + MONT_MAC(c0, c1, c2, a[7], a[15]); + MONT_MAC(s0, s1, s2, a[8], a[14]); + MONT_MAC(c0, c1, c2, a[9], a[13]); + MONT_MAC(s0, s1, s2, a[10], a[12]); + MONT_MAC(t0, t1, t2, a[11], a[11]); + MONT_MUL(v0, v1, v2, u[0], m[22]); + MONT_MUL(w0, w1, w2, u[1], m[21]); + MONT_MAC(t0, t1, t2, u[2], m[20]); + MONT_MAC(v0, v1, v2, u[3], m[19]); + MONT_MAC(w0, w1, w2, u[4], m[18]); + MONT_MAC(t0, t1, t2, u[5], m[17]); + MONT_MAC(v0, v1, v2, u[6], m[16]); + MONT_MAC(w0, w1, w2, u[7], m[15]); + MONT_MAC(t0, t1, t2, u[8], m[14]); + MONT_MAC(v0, v1, v2, u[9], m[13]); + MONT_MAC(w0, w1, w2, u[10], m[12]); + MONT_MAC(t0, t1, t2, u[11], m[11]); + MONT_MAC(v0, v1, v2, u[12], m[10]); + MONT_MAC(w0, w1, w2, u[13], m[9]); + MONT_MAC(t0, t1, t2, u[14], m[8]); + MONT_MAC(v0, v1, v2, u[15], m[7]); + MONT_MAC(w0, w1, w2, u[16], m[6]); + MONT_MAC(t0, t1, t2, u[17], m[5]); + MONT_MAC(v0, v1, v2, u[18], m[4]); + MONT_MAC(w0, w1, w2, u[19], m[3]); + MONT_MAC(t0, t1, t2, u[20], m[2]); + MONT_MAC(v0, v1, v2, u[21], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[22] = t0 * n0; + MONT_MAC(t0, t1, t2, u[22], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 23 */ + MONT_MUL(s0, s1, s2, a[0], a[23]); + MONT_MUL(c0, c1, c2, a[1], a[22]); + MONT_MAC(s0, s1, s2, a[2], a[21]); + MONT_MAC(c0, c1, c2, a[3], a[20]); + MONT_MAC(s0, s1, s2, a[4], a[19]); + MONT_MAC(c0, c1, c2, a[5], a[18]); + MONT_MAC(s0, s1, s2, a[6], a[17]); + MONT_MAC(c0, c1, c2, a[7], a[16]); + MONT_MAC(s0, s1, s2, a[8], a[15]); + MONT_MAC(c0, c1, c2, a[9], a[14]); + MONT_MAC(s0, s1, s2, a[10], a[13]); + MONT_MAC(c0, c1, c2, a[11], a[12]); + MONT_MAC(t0, t1, t2, u[0], m[23]); + MONT_MUL(v0, v1, v2, u[1], m[22]); + MONT_MUL(w0, w1, w2, u[2], m[21]); + MONT_MAC(t0, t1, t2, u[3], m[20]); + MONT_MAC(v0, v1, v2, u[4], m[19]); + MONT_MAC(w0, w1, w2, u[5], m[18]); + MONT_MAC(t0, t1, t2, u[6], m[17]); + MONT_MAC(v0, v1, v2, u[7], m[16]); + MONT_MAC(w0, w1, w2, u[8], m[15]); + MONT_MAC(t0, t1, t2, u[9], m[14]); + MONT_MAC(v0, v1, v2, u[10], m[13]); + MONT_MAC(w0, w1, w2, u[11], m[12]); + MONT_MAC(t0, t1, t2, u[12], m[11]); + MONT_MAC(v0, v1, v2, u[13], m[10]); + MONT_MAC(w0, w1, w2, u[14], m[9]); + MONT_MAC(t0, t1, t2, u[15], m[8]); + MONT_MAC(v0, v1, v2, u[16], m[7]); + MONT_MAC(w0, w1, w2, u[17], m[6]); + MONT_MAC(t0, t1, t2, u[18], m[5]); + MONT_MAC(v0, v1, v2, u[19], m[4]); + MONT_MAC(w0, w1, w2, u[20], m[3]); + MONT_MAC(t0, t1, t2, u[21], m[2]); + MONT_MAC(v0, v1, v2, u[22], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[23] = t0 * n0; + MONT_MAC(t0, t1, t2, u[23], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 24 */ + MONT_MUL(s0, s1, s2, a[0], a[24]); + MONT_MUL(c0, c1, c2, a[1], a[23]); + MONT_MAC(s0, s1, s2, a[2], a[22]); + MONT_MAC(c0, c1, c2, a[3], a[21]); + MONT_MAC(s0, s1, s2, a[4], a[20]); + MONT_MAC(c0, c1, c2, a[5], a[19]); + MONT_MAC(s0, s1, s2, a[6], a[18]); + MONT_MAC(c0, c1, c2, a[7], a[17]); + MONT_MAC(s0, s1, s2, a[8], a[16]); + MONT_MAC(c0, c1, c2, a[9], a[15]); + MONT_MAC(s0, s1, s2, a[10], a[14]); + MONT_MAC(c0, c1, c2, a[11], a[13]); + MONT_MAC(t0, t1, t2, a[12], a[12]); + MONT_MUL(v0, v1, v2, u[0], m[24]); + MONT_MUL(w0, w1, w2, u[1], m[23]); + MONT_MAC(t0, t1, t2, u[2], m[22]); + MONT_MAC(v0, v1, v2, u[3], m[21]); + MONT_MAC(w0, w1, w2, u[4], m[20]); + MONT_MAC(t0, t1, t2, u[5], m[19]); + MONT_MAC(v0, v1, v2, u[6], m[18]); + MONT_MAC(w0, w1, w2, u[7], m[17]); + MONT_MAC(t0, t1, t2, u[8], m[16]); + MONT_MAC(v0, v1, v2, u[9], m[15]); + MONT_MAC(w0, w1, w2, u[10], m[14]); + MONT_MAC(t0, t1, t2, u[11], m[13]); + MONT_MAC(v0, v1, v2, u[12], m[12]); + MONT_MAC(w0, w1, w2, u[13], m[11]); + MONT_MAC(t0, t1, t2, u[14], m[10]); + MONT_MAC(v0, v1, v2, u[15], m[9]); + MONT_MAC(w0, w1, w2, u[16], m[8]); + MONT_MAC(t0, t1, t2, u[17], m[7]); + MONT_MAC(v0, v1, v2, u[18], m[6]); + MONT_MAC(w0, w1, w2, u[19], m[5]); + MONT_MAC(t0, t1, t2, u[20], m[4]); + MONT_MAC(v0, v1, v2, u[21], m[3]); + MONT_MAC(w0, w1, w2, u[22], m[2]); + MONT_MAC(t0, t1, t2, u[23], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[24] = t0 * n0; + MONT_MAC(t0, t1, t2, u[24], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 25 */ + MONT_MUL(s0, s1, s2, a[0], a[25]); + MONT_MUL(c0, c1, c2, a[1], a[24]); + MONT_MAC(s0, s1, s2, a[2], a[23]); + MONT_MAC(c0, c1, c2, a[3], a[22]); + MONT_MAC(s0, s1, s2, a[4], a[21]); + MONT_MAC(c0, c1, c2, a[5], a[20]); + MONT_MAC(s0, s1, s2, a[6], a[19]); + MONT_MAC(c0, c1, c2, a[7], a[18]); + MONT_MAC(s0, s1, s2, a[8], a[17]); + MONT_MAC(c0, c1, c2, a[9], a[16]); + MONT_MAC(s0, s1, s2, a[10], a[15]); + MONT_MAC(c0, c1, c2, a[11], a[14]); + MONT_MAC(s0, s1, s2, a[12], a[13]); + MONT_MAC(t0, t1, t2, u[0], m[25]); + MONT_MUL(v0, v1, v2, u[1], m[24]); + MONT_MUL(w0, w1, w2, u[2], m[23]); + MONT_MAC(t0, t1, t2, u[3], m[22]); + MONT_MAC(v0, v1, v2, u[4], m[21]); + MONT_MAC(w0, w1, w2, u[5], m[20]); + MONT_MAC(t0, t1, t2, u[6], m[19]); + MONT_MAC(v0, v1, v2, u[7], m[18]); + MONT_MAC(w0, w1, w2, u[8], m[17]); + MONT_MAC(t0, t1, t2, u[9], m[16]); + MONT_MAC(v0, v1, v2, u[10], m[15]); + MONT_MAC(w0, w1, w2, u[11], m[14]); + MONT_MAC(t0, t1, t2, u[12], m[13]); + MONT_MAC(v0, v1, v2, u[13], m[12]); + MONT_MAC(w0, w1, w2, u[14], m[11]); + MONT_MAC(t0, t1, t2, u[15], m[10]); + MONT_MAC(v0, v1, v2, u[16], m[9]); + MONT_MAC(w0, w1, w2, u[17], m[8]); + MONT_MAC(t0, t1, t2, u[18], m[7]); + MONT_MAC(v0, v1, v2, u[19], m[6]); + MONT_MAC(w0, w1, w2, u[20], m[5]); + MONT_MAC(t0, t1, t2, u[21], m[4]); + MONT_MAC(v0, v1, v2, u[22], m[3]); + MONT_MAC(w0, w1, w2, u[23], m[2]); + MONT_MAC(t0, t1, t2, u[24], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[25] = t0 * n0; + MONT_MAC(t0, t1, t2, u[25], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 26 */ + MONT_MUL(s0, s1, s2, a[0], a[26]); + MONT_MUL(c0, c1, c2, a[1], a[25]); + MONT_MAC(s0, s1, s2, a[2], a[24]); + MONT_MAC(c0, c1, c2, a[3], a[23]); + MONT_MAC(s0, s1, s2, a[4], a[22]); + MONT_MAC(c0, c1, c2, a[5], a[21]); + MONT_MAC(s0, s1, s2, a[6], a[20]); + MONT_MAC(c0, c1, c2, a[7], a[19]); + MONT_MAC(s0, s1, s2, a[8], a[18]); + MONT_MAC(c0, c1, c2, a[9], a[17]); + MONT_MAC(s0, s1, s2, a[10], a[16]); + MONT_MAC(c0, c1, c2, a[11], a[15]); + MONT_MAC(s0, s1, s2, a[12], a[14]); + MONT_MAC(t0, t1, t2, a[13], a[13]); + MONT_MUL(v0, v1, v2, u[0], m[26]); + MONT_MUL(w0, w1, w2, u[1], m[25]); + MONT_MAC(t0, t1, t2, u[2], m[24]); + MONT_MAC(v0, v1, v2, u[3], m[23]); + MONT_MAC(w0, w1, w2, u[4], m[22]); + MONT_MAC(t0, t1, t2, u[5], m[21]); + MONT_MAC(v0, v1, v2, u[6], m[20]); + MONT_MAC(w0, w1, w2, u[7], m[19]); + MONT_MAC(t0, t1, t2, u[8], m[18]); + MONT_MAC(v0, v1, v2, u[9], m[17]); + MONT_MAC(w0, w1, w2, u[10], m[16]); + MONT_MAC(t0, t1, t2, u[11], m[15]); + MONT_MAC(v0, v1, v2, u[12], m[14]); + MONT_MAC(w0, w1, w2, u[13], m[13]); + MONT_MAC(t0, t1, t2, u[14], m[12]); + MONT_MAC(v0, v1, v2, u[15], m[11]); + MONT_MAC(w0, w1, w2, u[16], m[10]); + MONT_MAC(t0, t1, t2, u[17], m[9]); + MONT_MAC(v0, v1, v2, u[18], m[8]); + MONT_MAC(w0, w1, w2, u[19], m[7]); + MONT_MAC(t0, t1, t2, u[20], m[6]); + MONT_MAC(v0, v1, v2, u[21], m[5]); + MONT_MAC(w0, w1, w2, u[22], m[4]); + MONT_MAC(t0, t1, t2, u[23], m[3]); + MONT_MAC(v0, v1, v2, u[24], m[2]); + MONT_MAC(w0, w1, w2, u[25], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[26] = t0 * n0; + MONT_MAC(t0, t1, t2, u[26], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 27 */ + MONT_MUL(s0, s1, s2, a[0], a[27]); + MONT_MUL(c0, c1, c2, a[1], a[26]); + MONT_MAC(s0, s1, s2, a[2], a[25]); + MONT_MAC(c0, c1, c2, a[3], a[24]); + MONT_MAC(s0, s1, s2, a[4], a[23]); + MONT_MAC(c0, c1, c2, a[5], a[22]); + MONT_MAC(s0, s1, s2, a[6], a[21]); + MONT_MAC(c0, c1, c2, a[7], a[20]); + MONT_MAC(s0, s1, s2, a[8], a[19]); + MONT_MAC(c0, c1, c2, a[9], a[18]); + MONT_MAC(s0, s1, s2, a[10], a[17]); + MONT_MAC(c0, c1, c2, a[11], a[16]); + MONT_MAC(s0, s1, s2, a[12], a[15]); + MONT_MAC(c0, c1, c2, a[13], a[14]); + MONT_MAC(t0, t1, t2, u[0], m[27]); + MONT_MUL(v0, v1, v2, u[1], m[26]); + MONT_MUL(w0, w1, w2, u[2], m[25]); + MONT_MAC(t0, t1, t2, u[3], m[24]); + MONT_MAC(v0, v1, v2, u[4], m[23]); + MONT_MAC(w0, w1, w2, u[5], m[22]); + MONT_MAC(t0, t1, t2, u[6], m[21]); + MONT_MAC(v0, v1, v2, u[7], m[20]); + MONT_MAC(w0, w1, w2, u[8], m[19]); + MONT_MAC(t0, t1, t2, u[9], m[18]); + MONT_MAC(v0, v1, v2, u[10], m[17]); + MONT_MAC(w0, w1, w2, u[11], m[16]); + MONT_MAC(t0, t1, t2, u[12], m[15]); + MONT_MAC(v0, v1, v2, u[13], m[14]); + MONT_MAC(w0, w1, w2, u[14], m[13]); + MONT_MAC(t0, t1, t2, u[15], m[12]); + MONT_MAC(v0, v1, v2, u[16], m[11]); + MONT_MAC(w0, w1, w2, u[17], m[10]); + MONT_MAC(t0, t1, t2, u[18], m[9]); + MONT_MAC(v0, v1, v2, u[19], m[8]); + MONT_MAC(w0, w1, w2, u[20], m[7]); + MONT_MAC(t0, t1, t2, u[21], m[6]); + MONT_MAC(v0, v1, v2, u[22], m[5]); + MONT_MAC(w0, w1, w2, u[23], m[4]); + MONT_MAC(t0, t1, t2, u[24], m[3]); + MONT_MAC(v0, v1, v2, u[25], m[2]); + MONT_MAC(w0, w1, w2, u[26], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[27] = t0 * n0; + MONT_MAC(t0, t1, t2, u[27], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 28 */ + MONT_MUL(s0, s1, s2, a[0], a[28]); + MONT_MUL(c0, c1, c2, a[1], a[27]); + MONT_MAC(s0, s1, s2, a[2], a[26]); + MONT_MAC(c0, c1, c2, a[3], a[25]); + MONT_MAC(s0, s1, s2, a[4], a[24]); + MONT_MAC(c0, c1, c2, a[5], a[23]); + MONT_MAC(s0, s1, s2, a[6], a[22]); + MONT_MAC(c0, c1, c2, a[7], a[21]); + MONT_MAC(s0, s1, s2, a[8], a[20]); + MONT_MAC(c0, c1, c2, a[9], a[19]); + MONT_MAC(s0, s1, s2, a[10], a[18]); + MONT_MAC(c0, c1, c2, a[11], a[17]); + MONT_MAC(s0, s1, s2, a[12], a[16]); + MONT_MAC(c0, c1, c2, a[13], a[15]); + MONT_MAC(t0, t1, t2, a[14], a[14]); + MONT_MUL(v0, v1, v2, u[0], m[28]); + MONT_MUL(w0, w1, w2, u[1], m[27]); + MONT_MAC(t0, t1, t2, u[2], m[26]); + MONT_MAC(v0, v1, v2, u[3], m[25]); + MONT_MAC(w0, w1, w2, u[4], m[24]); + MONT_MAC(t0, t1, t2, u[5], m[23]); + MONT_MAC(v0, v1, v2, u[6], m[22]); + MONT_MAC(w0, w1, w2, u[7], m[21]); + MONT_MAC(t0, t1, t2, u[8], m[20]); + MONT_MAC(v0, v1, v2, u[9], m[19]); + MONT_MAC(w0, w1, w2, u[10], m[18]); + MONT_MAC(t0, t1, t2, u[11], m[17]); + MONT_MAC(v0, v1, v2, u[12], m[16]); + MONT_MAC(w0, w1, w2, u[13], m[15]); + MONT_MAC(t0, t1, t2, u[14], m[14]); + MONT_MAC(v0, v1, v2, u[15], m[13]); + MONT_MAC(w0, w1, w2, u[16], m[12]); + MONT_MAC(t0, t1, t2, u[17], m[11]); + MONT_MAC(v0, v1, v2, u[18], m[10]); + MONT_MAC(w0, w1, w2, u[19], m[9]); + MONT_MAC(t0, t1, t2, u[20], m[8]); + MONT_MAC(v0, v1, v2, u[21], m[7]); + MONT_MAC(w0, w1, w2, u[22], m[6]); + MONT_MAC(t0, t1, t2, u[23], m[5]); + MONT_MAC(v0, v1, v2, u[24], m[4]); + MONT_MAC(w0, w1, w2, u[25], m[3]); + MONT_MAC(t0, t1, t2, u[26], m[2]); + MONT_MAC(v0, v1, v2, u[27], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[28] = t0 * n0; + MONT_MAC(t0, t1, t2, u[28], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 29 */ + MONT_MUL(s0, s1, s2, a[0], a[29]); + MONT_MUL(c0, c1, c2, a[1], a[28]); + MONT_MAC(s0, s1, s2, a[2], a[27]); + MONT_MAC(c0, c1, c2, a[3], a[26]); + MONT_MAC(s0, s1, s2, a[4], a[25]); + MONT_MAC(c0, c1, c2, a[5], a[24]); + MONT_MAC(s0, s1, s2, a[6], a[23]); + MONT_MAC(c0, c1, c2, a[7], a[22]); + MONT_MAC(s0, s1, s2, a[8], a[21]); + MONT_MAC(c0, c1, c2, a[9], a[20]); + MONT_MAC(s0, s1, s2, a[10], a[19]); + MONT_MAC(c0, c1, c2, a[11], a[18]); + MONT_MAC(s0, s1, s2, a[12], a[17]); + MONT_MAC(c0, c1, c2, a[13], a[16]); + MONT_MAC(s0, s1, s2, a[14], a[15]); + MONT_MAC(t0, t1, t2, u[0], m[29]); + MONT_MUL(v0, v1, v2, u[1], m[28]); + MONT_MUL(w0, w1, w2, u[2], m[27]); + MONT_MAC(t0, t1, t2, u[3], m[26]); + MONT_MAC(v0, v1, v2, u[4], m[25]); + MONT_MAC(w0, w1, w2, u[5], m[24]); + MONT_MAC(t0, t1, t2, u[6], m[23]); + MONT_MAC(v0, v1, v2, u[7], m[22]); + MONT_MAC(w0, w1, w2, u[8], m[21]); + MONT_MAC(t0, t1, t2, u[9], m[20]); + MONT_MAC(v0, v1, v2, u[10], m[19]); + MONT_MAC(w0, w1, w2, u[11], m[18]); + MONT_MAC(t0, t1, t2, u[12], m[17]); + MONT_MAC(v0, v1, v2, u[13], m[16]); + MONT_MAC(w0, w1, w2, u[14], m[15]); + MONT_MAC(t0, t1, t2, u[15], m[14]); + MONT_MAC(v0, v1, v2, u[16], m[13]); + MONT_MAC(w0, w1, w2, u[17], m[12]); + MONT_MAC(t0, t1, t2, u[18], m[11]); + MONT_MAC(v0, v1, v2, u[19], m[10]); + MONT_MAC(w0, w1, w2, u[20], m[9]); + MONT_MAC(t0, t1, t2, u[21], m[8]); + MONT_MAC(v0, v1, v2, u[22], m[7]); + MONT_MAC(w0, w1, w2, u[23], m[6]); + MONT_MAC(t0, t1, t2, u[24], m[5]); + MONT_MAC(v0, v1, v2, u[25], m[4]); + MONT_MAC(w0, w1, w2, u[26], m[3]); + MONT_MAC(t0, t1, t2, u[27], m[2]); + MONT_MAC(v0, v1, v2, u[28], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[29] = t0 * n0; + MONT_MAC(t0, t1, t2, u[29], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 30 */ + MONT_MUL(s0, s1, s2, a[0], a[30]); + MONT_MUL(c0, c1, c2, a[1], a[29]); + MONT_MAC(s0, s1, s2, a[2], a[28]); + MONT_MAC(c0, c1, c2, a[3], a[27]); + MONT_MAC(s0, s1, s2, a[4], a[26]); + MONT_MAC(c0, c1, c2, a[5], a[25]); + MONT_MAC(s0, s1, s2, a[6], a[24]); + MONT_MAC(c0, c1, c2, a[7], a[23]); + MONT_MAC(s0, s1, s2, a[8], a[22]); + MONT_MAC(c0, c1, c2, a[9], a[21]); + MONT_MAC(s0, s1, s2, a[10], a[20]); + MONT_MAC(c0, c1, c2, a[11], a[19]); + MONT_MAC(s0, s1, s2, a[12], a[18]); + MONT_MAC(c0, c1, c2, a[13], a[17]); + MONT_MAC(s0, s1, s2, a[14], a[16]); + MONT_MAC(t0, t1, t2, a[15], a[15]); + MONT_MUL(v0, v1, v2, u[0], m[30]); + MONT_MUL(w0, w1, w2, u[1], m[29]); + MONT_MAC(t0, t1, t2, u[2], m[28]); + MONT_MAC(v0, v1, v2, u[3], m[27]); + MONT_MAC(w0, w1, w2, u[4], m[26]); + MONT_MAC(t0, t1, t2, u[5], m[25]); + MONT_MAC(v0, v1, v2, u[6], m[24]); + MONT_MAC(w0, w1, w2, u[7], m[23]); + MONT_MAC(t0, t1, t2, u[8], m[22]); + MONT_MAC(v0, v1, v2, u[9], m[21]); + MONT_MAC(w0, w1, w2, u[10], m[20]); + MONT_MAC(t0, t1, t2, u[11], m[19]); + MONT_MAC(v0, v1, v2, u[12], m[18]); + MONT_MAC(w0, w1, w2, u[13], m[17]); + MONT_MAC(t0, t1, t2, u[14], m[16]); + MONT_MAC(v0, v1, v2, u[15], m[15]); + MONT_MAC(w0, w1, w2, u[16], m[14]); + MONT_MAC(t0, t1, t2, u[17], m[13]); + MONT_MAC(v0, v1, v2, u[18], m[12]); + MONT_MAC(w0, w1, w2, u[19], m[11]); + MONT_MAC(t0, t1, t2, u[20], m[10]); + MONT_MAC(v0, v1, v2, u[21], m[9]); + MONT_MAC(w0, w1, w2, u[22], m[8]); + MONT_MAC(t0, t1, t2, u[23], m[7]); + MONT_MAC(v0, v1, v2, u[24], m[6]); + MONT_MAC(w0, w1, w2, u[25], m[5]); + MONT_MAC(t0, t1, t2, u[26], m[4]); + MONT_MAC(v0, v1, v2, u[27], m[3]); + MONT_MAC(w0, w1, w2, u[28], m[2]); + MONT_MAC(t0, t1, t2, u[29], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[30] = t0 * n0; + MONT_MAC(t0, t1, t2, u[30], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 31 */ + MONT_MUL(s0, s1, s2, a[0], a[31]); + MONT_MUL(c0, c1, c2, a[1], a[30]); + MONT_MAC(s0, s1, s2, a[2], a[29]); + MONT_MAC(c0, c1, c2, a[3], a[28]); + MONT_MAC(s0, s1, s2, a[4], a[27]); + MONT_MAC(c0, c1, c2, a[5], a[26]); + MONT_MAC(s0, s1, s2, a[6], a[25]); + MONT_MAC(c0, c1, c2, a[7], a[24]); + MONT_MAC(s0, s1, s2, a[8], a[23]); + MONT_MAC(c0, c1, c2, a[9], a[22]); + MONT_MAC(s0, s1, s2, a[10], a[21]); + MONT_MAC(c0, c1, c2, a[11], a[20]); + MONT_MAC(s0, s1, s2, a[12], a[19]); + MONT_MAC(c0, c1, c2, a[13], a[18]); + MONT_MAC(s0, s1, s2, a[14], a[17]); + MONT_MAC(c0, c1, c2, a[15], a[16]); + MONT_MAC(t0, t1, t2, u[0], m[31]); + MONT_MUL(v0, v1, v2, u[1], m[30]); + MONT_MUL(w0, w1, w2, u[2], m[29]); + MONT_MAC(t0, t1, t2, u[3], m[28]); + MONT_MAC(v0, v1, v2, u[4], m[27]); + MONT_MAC(w0, w1, w2, u[5], m[26]); + MONT_MAC(t0, t1, t2, u[6], m[25]); + MONT_MAC(v0, v1, v2, u[7], m[24]); + MONT_MAC(w0, w1, w2, u[8], m[23]); + MONT_MAC(t0, t1, t2, u[9], m[22]); + MONT_MAC(v0, v1, v2, u[10], m[21]); + MONT_MAC(w0, w1, w2, u[11], m[20]); + MONT_MAC(t0, t1, t2, u[12], m[19]); + MONT_MAC(v0, v1, v2, u[13], m[18]); + MONT_MAC(w0, w1, w2, u[14], m[17]); + MONT_MAC(t0, t1, t2, u[15], m[16]); + MONT_MAC(v0, v1, v2, u[16], m[15]); + MONT_MAC(w0, w1, w2, u[17], m[14]); + MONT_MAC(t0, t1, t2, u[18], m[13]); + MONT_MAC(v0, v1, v2, u[19], m[12]); + MONT_MAC(w0, w1, w2, u[20], m[11]); + MONT_MAC(t0, t1, t2, u[21], m[10]); + MONT_MAC(v0, v1, v2, u[22], m[9]); + MONT_MAC(w0, w1, w2, u[23], m[8]); + MONT_MAC(t0, t1, t2, u[24], m[7]); + MONT_MAC(v0, v1, v2, u[25], m[6]); + MONT_MAC(w0, w1, w2, u[26], m[5]); + MONT_MAC(t0, t1, t2, u[27], m[4]); + MONT_MAC(v0, v1, v2, u[28], m[3]); + MONT_MAC(w0, w1, w2, u[29], m[2]); + MONT_MAC(t0, t1, t2, u[30], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[31] = t0 * n0; + MONT_MAC(t0, t1, t2, u[31], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 32 */ + MONT_MUL(s0, s1, s2, a[0], a[32]); + MONT_MUL(c0, c1, c2, a[1], a[31]); + MONT_MAC(s0, s1, s2, a[2], a[30]); + MONT_MAC(c0, c1, c2, a[3], a[29]); + MONT_MAC(s0, s1, s2, a[4], a[28]); + MONT_MAC(c0, c1, c2, a[5], a[27]); + MONT_MAC(s0, s1, s2, a[6], a[26]); + MONT_MAC(c0, c1, c2, a[7], a[25]); + MONT_MAC(s0, s1, s2, a[8], a[24]); + MONT_MAC(c0, c1, c2, a[9], a[23]); + MONT_MAC(s0, s1, s2, a[10], a[22]); + MONT_MAC(c0, c1, c2, a[11], a[21]); + MONT_MAC(s0, s1, s2, a[12], a[20]); + MONT_MAC(c0, c1, c2, a[13], a[19]); + MONT_MAC(s0, s1, s2, a[14], a[18]); + MONT_MAC(c0, c1, c2, a[15], a[17]); + MONT_MAC(t0, t1, t2, a[16], a[16]); + MONT_MUL(v0, v1, v2, u[0], m[32]); + MONT_MUL(w0, w1, w2, u[1], m[31]); + MONT_MAC(t0, t1, t2, u[2], m[30]); + MONT_MAC(v0, v1, v2, u[3], m[29]); + MONT_MAC(w0, w1, w2, u[4], m[28]); + MONT_MAC(t0, t1, t2, u[5], m[27]); + MONT_MAC(v0, v1, v2, u[6], m[26]); + MONT_MAC(w0, w1, w2, u[7], m[25]); + MONT_MAC(t0, t1, t2, u[8], m[24]); + MONT_MAC(v0, v1, v2, u[9], m[23]); + MONT_MAC(w0, w1, w2, u[10], m[22]); + MONT_MAC(t0, t1, t2, u[11], m[21]); + MONT_MAC(v0, v1, v2, u[12], m[20]); + MONT_MAC(w0, w1, w2, u[13], m[19]); + MONT_MAC(t0, t1, t2, u[14], m[18]); + MONT_MAC(v0, v1, v2, u[15], m[17]); + MONT_MAC(w0, w1, w2, u[16], m[16]); + MONT_MAC(t0, t1, t2, u[17], m[15]); + MONT_MAC(v0, v1, v2, u[18], m[14]); + MONT_MAC(w0, w1, w2, u[19], m[13]); + MONT_MAC(t0, t1, t2, u[20], m[12]); + MONT_MAC(v0, v1, v2, u[21], m[11]); + MONT_MAC(w0, w1, w2, u[22], m[10]); + MONT_MAC(t0, t1, t2, u[23], m[9]); + MONT_MAC(v0, v1, v2, u[24], m[8]); + MONT_MAC(w0, w1, w2, u[25], m[7]); + MONT_MAC(t0, t1, t2, u[26], m[6]); + MONT_MAC(v0, v1, v2, u[27], m[5]); + MONT_MAC(w0, w1, w2, u[28], m[4]); + MONT_MAC(t0, t1, t2, u[29], m[3]); + MONT_MAC(v0, v1, v2, u[30], m[2]); + MONT_MAC(w0, w1, w2, u[31], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[32] = t0 * n0; + MONT_MAC(t0, t1, t2, u[32], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 33 */ + MONT_MUL(s0, s1, s2, a[0], a[33]); + MONT_MUL(c0, c1, c2, a[1], a[32]); + MONT_MAC(s0, s1, s2, a[2], a[31]); + MONT_MAC(c0, c1, c2, a[3], a[30]); + MONT_MAC(s0, s1, s2, a[4], a[29]); + MONT_MAC(c0, c1, c2, a[5], a[28]); + MONT_MAC(s0, s1, s2, a[6], a[27]); + MONT_MAC(c0, c1, c2, a[7], a[26]); + MONT_MAC(s0, s1, s2, a[8], a[25]); + MONT_MAC(c0, c1, c2, a[9], a[24]); + MONT_MAC(s0, s1, s2, a[10], a[23]); + MONT_MAC(c0, c1, c2, a[11], a[22]); + MONT_MAC(s0, s1, s2, a[12], a[21]); + MONT_MAC(c0, c1, c2, a[13], a[20]); + MONT_MAC(s0, s1, s2, a[14], a[19]); + MONT_MAC(c0, c1, c2, a[15], a[18]); + MONT_MAC(s0, s1, s2, a[16], a[17]); + MONT_MAC(t0, t1, t2, u[0], m[33]); + MONT_MUL(v0, v1, v2, u[1], m[32]); + MONT_MUL(w0, w1, w2, u[2], m[31]); + MONT_MAC(t0, t1, t2, u[3], m[30]); + MONT_MAC(v0, v1, v2, u[4], m[29]); + MONT_MAC(w0, w1, w2, u[5], m[28]); + MONT_MAC(t0, t1, t2, u[6], m[27]); + MONT_MAC(v0, v1, v2, u[7], m[26]); + MONT_MAC(w0, w1, w2, u[8], m[25]); + MONT_MAC(t0, t1, t2, u[9], m[24]); + MONT_MAC(v0, v1, v2, u[10], m[23]); + MONT_MAC(w0, w1, w2, u[11], m[22]); + MONT_MAC(t0, t1, t2, u[12], m[21]); + MONT_MAC(v0, v1, v2, u[13], m[20]); + MONT_MAC(w0, w1, w2, u[14], m[19]); + MONT_MAC(t0, t1, t2, u[15], m[18]); + MONT_MAC(v0, v1, v2, u[16], m[17]); + MONT_MAC(w0, w1, w2, u[17], m[16]); + MONT_MAC(t0, t1, t2, u[18], m[15]); + MONT_MAC(v0, v1, v2, u[19], m[14]); + MONT_MAC(w0, w1, w2, u[20], m[13]); + MONT_MAC(t0, t1, t2, u[21], m[12]); + MONT_MAC(v0, v1, v2, u[22], m[11]); + MONT_MAC(w0, w1, w2, u[23], m[10]); + MONT_MAC(t0, t1, t2, u[24], m[9]); + MONT_MAC(v0, v1, v2, u[25], m[8]); + MONT_MAC(w0, w1, w2, u[26], m[7]); + MONT_MAC(t0, t1, t2, u[27], m[6]); + MONT_MAC(v0, v1, v2, u[28], m[5]); + MONT_MAC(w0, w1, w2, u[29], m[4]); + MONT_MAC(t0, t1, t2, u[30], m[3]); + MONT_MAC(v0, v1, v2, u[31], m[2]); + MONT_MAC(w0, w1, w2, u[32], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[33] = t0 * n0; + MONT_MAC(t0, t1, t2, u[33], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 34 */ + MONT_MUL(s0, s1, s2, a[0], a[34]); + MONT_MUL(c0, c1, c2, a[1], a[33]); + MONT_MAC(s0, s1, s2, a[2], a[32]); + MONT_MAC(c0, c1, c2, a[3], a[31]); + MONT_MAC(s0, s1, s2, a[4], a[30]); + MONT_MAC(c0, c1, c2, a[5], a[29]); + MONT_MAC(s0, s1, s2, a[6], a[28]); + MONT_MAC(c0, c1, c2, a[7], a[27]); + MONT_MAC(s0, s1, s2, a[8], a[26]); + MONT_MAC(c0, c1, c2, a[9], a[25]); + MONT_MAC(s0, s1, s2, a[10], a[24]); + MONT_MAC(c0, c1, c2, a[11], a[23]); + MONT_MAC(s0, s1, s2, a[12], a[22]); + MONT_MAC(c0, c1, c2, a[13], a[21]); + MONT_MAC(s0, s1, s2, a[14], a[20]); + MONT_MAC(c0, c1, c2, a[15], a[19]); + MONT_MAC(s0, s1, s2, a[16], a[18]); + MONT_MAC(t0, t1, t2, a[17], a[17]); + MONT_MUL(v0, v1, v2, u[0], m[34]); + MONT_MUL(w0, w1, w2, u[1], m[33]); + MONT_MAC(t0, t1, t2, u[2], m[32]); + MONT_MAC(v0, v1, v2, u[3], m[31]); + MONT_MAC(w0, w1, w2, u[4], m[30]); + MONT_MAC(t0, t1, t2, u[5], m[29]); + MONT_MAC(v0, v1, v2, u[6], m[28]); + MONT_MAC(w0, w1, w2, u[7], m[27]); + MONT_MAC(t0, t1, t2, u[8], m[26]); + MONT_MAC(v0, v1, v2, u[9], m[25]); + MONT_MAC(w0, w1, w2, u[10], m[24]); + MONT_MAC(t0, t1, t2, u[11], m[23]); + MONT_MAC(v0, v1, v2, u[12], m[22]); + MONT_MAC(w0, w1, w2, u[13], m[21]); + MONT_MAC(t0, t1, t2, u[14], m[20]); + MONT_MAC(v0, v1, v2, u[15], m[19]); + MONT_MAC(w0, w1, w2, u[16], m[18]); + MONT_MAC(t0, t1, t2, u[17], m[17]); + MONT_MAC(v0, v1, v2, u[18], m[16]); + MONT_MAC(w0, w1, w2, u[19], m[15]); + MONT_MAC(t0, t1, t2, u[20], m[14]); + MONT_MAC(v0, v1, v2, u[21], m[13]); + MONT_MAC(w0, w1, w2, u[22], m[12]); + MONT_MAC(t0, t1, t2, u[23], m[11]); + MONT_MAC(v0, v1, v2, u[24], m[10]); + MONT_MAC(w0, w1, w2, u[25], m[9]); + MONT_MAC(t0, t1, t2, u[26], m[8]); + MONT_MAC(v0, v1, v2, u[27], m[7]); + MONT_MAC(w0, w1, w2, u[28], m[6]); + MONT_MAC(t0, t1, t2, u[29], m[5]); + MONT_MAC(v0, v1, v2, u[30], m[4]); + MONT_MAC(w0, w1, w2, u[31], m[3]); + MONT_MAC(t0, t1, t2, u[32], m[2]); + MONT_MAC(v0, v1, v2, u[33], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[34] = t0 * n0; + MONT_MAC(t0, t1, t2, u[34], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 35 */ + MONT_MUL(s0, s1, s2, a[0], a[35]); + MONT_MUL(c0, c1, c2, a[1], a[34]); + MONT_MAC(s0, s1, s2, a[2], a[33]); + MONT_MAC(c0, c1, c2, a[3], a[32]); + MONT_MAC(s0, s1, s2, a[4], a[31]); + MONT_MAC(c0, c1, c2, a[5], a[30]); + MONT_MAC(s0, s1, s2, a[6], a[29]); + MONT_MAC(c0, c1, c2, a[7], a[28]); + MONT_MAC(s0, s1, s2, a[8], a[27]); + MONT_MAC(c0, c1, c2, a[9], a[26]); + MONT_MAC(s0, s1, s2, a[10], a[25]); + MONT_MAC(c0, c1, c2, a[11], a[24]); + MONT_MAC(s0, s1, s2, a[12], a[23]); + MONT_MAC(c0, c1, c2, a[13], a[22]); + MONT_MAC(s0, s1, s2, a[14], a[21]); + MONT_MAC(c0, c1, c2, a[15], a[20]); + MONT_MAC(s0, s1, s2, a[16], a[19]); + MONT_MAC(c0, c1, c2, a[17], a[18]); + MONT_MAC(t0, t1, t2, u[0], m[35]); + MONT_MUL(v0, v1, v2, u[1], m[34]); + MONT_MUL(w0, w1, w2, u[2], m[33]); + MONT_MAC(t0, t1, t2, u[3], m[32]); + MONT_MAC(v0, v1, v2, u[4], m[31]); + MONT_MAC(w0, w1, w2, u[5], m[30]); + MONT_MAC(t0, t1, t2, u[6], m[29]); + MONT_MAC(v0, v1, v2, u[7], m[28]); + MONT_MAC(w0, w1, w2, u[8], m[27]); + MONT_MAC(t0, t1, t2, u[9], m[26]); + MONT_MAC(v0, v1, v2, u[10], m[25]); + MONT_MAC(w0, w1, w2, u[11], m[24]); + MONT_MAC(t0, t1, t2, u[12], m[23]); + MONT_MAC(v0, v1, v2, u[13], m[22]); + MONT_MAC(w0, w1, w2, u[14], m[21]); + MONT_MAC(t0, t1, t2, u[15], m[20]); + MONT_MAC(v0, v1, v2, u[16], m[19]); + MONT_MAC(w0, w1, w2, u[17], m[18]); + MONT_MAC(t0, t1, t2, u[18], m[17]); + MONT_MAC(v0, v1, v2, u[19], m[16]); + MONT_MAC(w0, w1, w2, u[20], m[15]); + MONT_MAC(t0, t1, t2, u[21], m[14]); + MONT_MAC(v0, v1, v2, u[22], m[13]); + MONT_MAC(w0, w1, w2, u[23], m[12]); + MONT_MAC(t0, t1, t2, u[24], m[11]); + MONT_MAC(v0, v1, v2, u[25], m[10]); + MONT_MAC(w0, w1, w2, u[26], m[9]); + MONT_MAC(t0, t1, t2, u[27], m[8]); + MONT_MAC(v0, v1, v2, u[28], m[7]); + MONT_MAC(w0, w1, w2, u[29], m[6]); + MONT_MAC(t0, t1, t2, u[30], m[5]); + MONT_MAC(v0, v1, v2, u[31], m[4]); + MONT_MAC(w0, w1, w2, u[32], m[3]); + MONT_MAC(t0, t1, t2, u[33], m[2]); + MONT_MAC(v0, v1, v2, u[34], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[35] = t0 * n0; + MONT_MAC(t0, t1, t2, u[35], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 36 */ + MONT_MUL(s0, s1, s2, a[0], a[36]); + MONT_MUL(c0, c1, c2, a[1], a[35]); + MONT_MAC(s0, s1, s2, a[2], a[34]); + MONT_MAC(c0, c1, c2, a[3], a[33]); + MONT_MAC(s0, s1, s2, a[4], a[32]); + MONT_MAC(c0, c1, c2, a[5], a[31]); + MONT_MAC(s0, s1, s2, a[6], a[30]); + MONT_MAC(c0, c1, c2, a[7], a[29]); + MONT_MAC(s0, s1, s2, a[8], a[28]); + MONT_MAC(c0, c1, c2, a[9], a[27]); + MONT_MAC(s0, s1, s2, a[10], a[26]); + MONT_MAC(c0, c1, c2, a[11], a[25]); + MONT_MAC(s0, s1, s2, a[12], a[24]); + MONT_MAC(c0, c1, c2, a[13], a[23]); + MONT_MAC(s0, s1, s2, a[14], a[22]); + MONT_MAC(c0, c1, c2, a[15], a[21]); + MONT_MAC(s0, s1, s2, a[16], a[20]); + MONT_MAC(c0, c1, c2, a[17], a[19]); + MONT_MAC(t0, t1, t2, a[18], a[18]); + MONT_MUL(v0, v1, v2, u[0], m[36]); + MONT_MUL(w0, w1, w2, u[1], m[35]); + MONT_MAC(t0, t1, t2, u[2], m[34]); + MONT_MAC(v0, v1, v2, u[3], m[33]); + MONT_MAC(w0, w1, w2, u[4], m[32]); + MONT_MAC(t0, t1, t2, u[5], m[31]); + MONT_MAC(v0, v1, v2, u[6], m[30]); + MONT_MAC(w0, w1, w2, u[7], m[29]); + MONT_MAC(t0, t1, t2, u[8], m[28]); + MONT_MAC(v0, v1, v2, u[9], m[27]); + MONT_MAC(w0, w1, w2, u[10], m[26]); + MONT_MAC(t0, t1, t2, u[11], m[25]); + MONT_MAC(v0, v1, v2, u[12], m[24]); + MONT_MAC(w0, w1, w2, u[13], m[23]); + MONT_MAC(t0, t1, t2, u[14], m[22]); + MONT_MAC(v0, v1, v2, u[15], m[21]); + MONT_MAC(w0, w1, w2, u[16], m[20]); + MONT_MAC(t0, t1, t2, u[17], m[19]); + MONT_MAC(v0, v1, v2, u[18], m[18]); + MONT_MAC(w0, w1, w2, u[19], m[17]); + MONT_MAC(t0, t1, t2, u[20], m[16]); + MONT_MAC(v0, v1, v2, u[21], m[15]); + MONT_MAC(w0, w1, w2, u[22], m[14]); + MONT_MAC(t0, t1, t2, u[23], m[13]); + MONT_MAC(v0, v1, v2, u[24], m[12]); + MONT_MAC(w0, w1, w2, u[25], m[11]); + MONT_MAC(t0, t1, t2, u[26], m[10]); + MONT_MAC(v0, v1, v2, u[27], m[9]); + MONT_MAC(w0, w1, w2, u[28], m[8]); + MONT_MAC(t0, t1, t2, u[29], m[7]); + MONT_MAC(v0, v1, v2, u[30], m[6]); + MONT_MAC(w0, w1, w2, u[31], m[5]); + MONT_MAC(t0, t1, t2, u[32], m[4]); + MONT_MAC(v0, v1, v2, u[33], m[3]); + MONT_MAC(w0, w1, w2, u[34], m[2]); + MONT_MAC(t0, t1, t2, u[35], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[36] = t0 * n0; + MONT_MAC(t0, t1, t2, u[36], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 37 */ + MONT_MUL(s0, s1, s2, a[0], a[37]); + MONT_MUL(c0, c1, c2, a[1], a[36]); + MONT_MAC(s0, s1, s2, a[2], a[35]); + MONT_MAC(c0, c1, c2, a[3], a[34]); + MONT_MAC(s0, s1, s2, a[4], a[33]); + MONT_MAC(c0, c1, c2, a[5], a[32]); + MONT_MAC(s0, s1, s2, a[6], a[31]); + MONT_MAC(c0, c1, c2, a[7], a[30]); + MONT_MAC(s0, s1, s2, a[8], a[29]); + MONT_MAC(c0, c1, c2, a[9], a[28]); + MONT_MAC(s0, s1, s2, a[10], a[27]); + MONT_MAC(c0, c1, c2, a[11], a[26]); + MONT_MAC(s0, s1, s2, a[12], a[25]); + MONT_MAC(c0, c1, c2, a[13], a[24]); + MONT_MAC(s0, s1, s2, a[14], a[23]); + MONT_MAC(c0, c1, c2, a[15], a[22]); + MONT_MAC(s0, s1, s2, a[16], a[21]); + MONT_MAC(c0, c1, c2, a[17], a[20]); + MONT_MAC(s0, s1, s2, a[18], a[19]); + MONT_MAC(t0, t1, t2, u[0], m[37]); + MONT_MUL(v0, v1, v2, u[1], m[36]); + MONT_MUL(w0, w1, w2, u[2], m[35]); + MONT_MAC(t0, t1, t2, u[3], m[34]); + MONT_MAC(v0, v1, v2, u[4], m[33]); + MONT_MAC(w0, w1, w2, u[5], m[32]); + MONT_MAC(t0, t1, t2, u[6], m[31]); + MONT_MAC(v0, v1, v2, u[7], m[30]); + MONT_MAC(w0, w1, w2, u[8], m[29]); + MONT_MAC(t0, t1, t2, u[9], m[28]); + MONT_MAC(v0, v1, v2, u[10], m[27]); + MONT_MAC(w0, w1, w2, u[11], m[26]); + MONT_MAC(t0, t1, t2, u[12], m[25]); + MONT_MAC(v0, v1, v2, u[13], m[24]); + MONT_MAC(w0, w1, w2, u[14], m[23]); + MONT_MAC(t0, t1, t2, u[15], m[22]); + MONT_MAC(v0, v1, v2, u[16], m[21]); + MONT_MAC(w0, w1, w2, u[17], m[20]); + MONT_MAC(t0, t1, t2, u[18], m[19]); + MONT_MAC(v0, v1, v2, u[19], m[18]); + MONT_MAC(w0, w1, w2, u[20], m[17]); + MONT_MAC(t0, t1, t2, u[21], m[16]); + MONT_MAC(v0, v1, v2, u[22], m[15]); + MONT_MAC(w0, w1, w2, u[23], m[14]); + MONT_MAC(t0, t1, t2, u[24], m[13]); + MONT_MAC(v0, v1, v2, u[25], m[12]); + MONT_MAC(w0, w1, w2, u[26], m[11]); + MONT_MAC(t0, t1, t2, u[27], m[10]); + MONT_MAC(v0, v1, v2, u[28], m[9]); + MONT_MAC(w0, w1, w2, u[29], m[8]); + MONT_MAC(t0, t1, t2, u[30], m[7]); + MONT_MAC(v0, v1, v2, u[31], m[6]); + MONT_MAC(w0, w1, w2, u[32], m[5]); + MONT_MAC(t0, t1, t2, u[33], m[4]); + MONT_MAC(v0, v1, v2, u[34], m[3]); + MONT_MAC(w0, w1, w2, u[35], m[2]); + MONT_MAC(t0, t1, t2, u[36], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[37] = t0 * n0; + MONT_MAC(t0, t1, t2, u[37], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 38 */ + MONT_MUL(s0, s1, s2, a[0], a[38]); + MONT_MUL(c0, c1, c2, a[1], a[37]); + MONT_MAC(s0, s1, s2, a[2], a[36]); + MONT_MAC(c0, c1, c2, a[3], a[35]); + MONT_MAC(s0, s1, s2, a[4], a[34]); + MONT_MAC(c0, c1, c2, a[5], a[33]); + MONT_MAC(s0, s1, s2, a[6], a[32]); + MONT_MAC(c0, c1, c2, a[7], a[31]); + MONT_MAC(s0, s1, s2, a[8], a[30]); + MONT_MAC(c0, c1, c2, a[9], a[29]); + MONT_MAC(s0, s1, s2, a[10], a[28]); + MONT_MAC(c0, c1, c2, a[11], a[27]); + MONT_MAC(s0, s1, s2, a[12], a[26]); + MONT_MAC(c0, c1, c2, a[13], a[25]); + MONT_MAC(s0, s1, s2, a[14], a[24]); + MONT_MAC(c0, c1, c2, a[15], a[23]); + MONT_MAC(s0, s1, s2, a[16], a[22]); + MONT_MAC(c0, c1, c2, a[17], a[21]); + MONT_MAC(s0, s1, s2, a[18], a[20]); + MONT_MAC(t0, t1, t2, a[19], a[19]); + MONT_MUL(v0, v1, v2, u[0], m[38]); + MONT_MUL(w0, w1, w2, u[1], m[37]); + MONT_MAC(t0, t1, t2, u[2], m[36]); + MONT_MAC(v0, v1, v2, u[3], m[35]); + MONT_MAC(w0, w1, w2, u[4], m[34]); + MONT_MAC(t0, t1, t2, u[5], m[33]); + MONT_MAC(v0, v1, v2, u[6], m[32]); + MONT_MAC(w0, w1, w2, u[7], m[31]); + MONT_MAC(t0, t1, t2, u[8], m[30]); + MONT_MAC(v0, v1, v2, u[9], m[29]); + MONT_MAC(w0, w1, w2, u[10], m[28]); + MONT_MAC(t0, t1, t2, u[11], m[27]); + MONT_MAC(v0, v1, v2, u[12], m[26]); + MONT_MAC(w0, w1, w2, u[13], m[25]); + MONT_MAC(t0, t1, t2, u[14], m[24]); + MONT_MAC(v0, v1, v2, u[15], m[23]); + MONT_MAC(w0, w1, w2, u[16], m[22]); + MONT_MAC(t0, t1, t2, u[17], m[21]); + MONT_MAC(v0, v1, v2, u[18], m[20]); + MONT_MAC(w0, w1, w2, u[19], m[19]); + MONT_MAC(t0, t1, t2, u[20], m[18]); + MONT_MAC(v0, v1, v2, u[21], m[17]); + MONT_MAC(w0, w1, w2, u[22], m[16]); + MONT_MAC(t0, t1, t2, u[23], m[15]); + MONT_MAC(v0, v1, v2, u[24], m[14]); + MONT_MAC(w0, w1, w2, u[25], m[13]); + MONT_MAC(t0, t1, t2, u[26], m[12]); + MONT_MAC(v0, v1, v2, u[27], m[11]); + MONT_MAC(w0, w1, w2, u[28], m[10]); + MONT_MAC(t0, t1, t2, u[29], m[9]); + MONT_MAC(v0, v1, v2, u[30], m[8]); + MONT_MAC(w0, w1, w2, u[31], m[7]); + MONT_MAC(t0, t1, t2, u[32], m[6]); + MONT_MAC(v0, v1, v2, u[33], m[5]); + MONT_MAC(w0, w1, w2, u[34], m[4]); + MONT_MAC(t0, t1, t2, u[35], m[3]); + MONT_MAC(v0, v1, v2, u[36], m[2]); + MONT_MAC(w0, w1, w2, u[37], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[38] = t0 * n0; + MONT_MAC(t0, t1, t2, u[38], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 39 */ + MONT_MUL(s0, s1, s2, a[0], a[39]); + MONT_MUL(c0, c1, c2, a[1], a[38]); + MONT_MAC(s0, s1, s2, a[2], a[37]); + MONT_MAC(c0, c1, c2, a[3], a[36]); + MONT_MAC(s0, s1, s2, a[4], a[35]); + MONT_MAC(c0, c1, c2, a[5], a[34]); + MONT_MAC(s0, s1, s2, a[6], a[33]); + MONT_MAC(c0, c1, c2, a[7], a[32]); + MONT_MAC(s0, s1, s2, a[8], a[31]); + MONT_MAC(c0, c1, c2, a[9], a[30]); + MONT_MAC(s0, s1, s2, a[10], a[29]); + MONT_MAC(c0, c1, c2, a[11], a[28]); + MONT_MAC(s0, s1, s2, a[12], a[27]); + MONT_MAC(c0, c1, c2, a[13], a[26]); + MONT_MAC(s0, s1, s2, a[14], a[25]); + MONT_MAC(c0, c1, c2, a[15], a[24]); + MONT_MAC(s0, s1, s2, a[16], a[23]); + MONT_MAC(c0, c1, c2, a[17], a[22]); + MONT_MAC(s0, s1, s2, a[18], a[21]); + MONT_MAC(c0, c1, c2, a[19], a[20]); + MONT_MAC(t0, t1, t2, u[0], m[39]); + MONT_MUL(v0, v1, v2, u[1], m[38]); + MONT_MUL(w0, w1, w2, u[2], m[37]); + MONT_MAC(t0, t1, t2, u[3], m[36]); + MONT_MAC(v0, v1, v2, u[4], m[35]); + MONT_MAC(w0, w1, w2, u[5], m[34]); + MONT_MAC(t0, t1, t2, u[6], m[33]); + MONT_MAC(v0, v1, v2, u[7], m[32]); + MONT_MAC(w0, w1, w2, u[8], m[31]); + MONT_MAC(t0, t1, t2, u[9], m[30]); + MONT_MAC(v0, v1, v2, u[10], m[29]); + MONT_MAC(w0, w1, w2, u[11], m[28]); + MONT_MAC(t0, t1, t2, u[12], m[27]); + MONT_MAC(v0, v1, v2, u[13], m[26]); + MONT_MAC(w0, w1, w2, u[14], m[25]); + MONT_MAC(t0, t1, t2, u[15], m[24]); + MONT_MAC(v0, v1, v2, u[16], m[23]); + MONT_MAC(w0, w1, w2, u[17], m[22]); + MONT_MAC(t0, t1, t2, u[18], m[21]); + MONT_MAC(v0, v1, v2, u[19], m[20]); + MONT_MAC(w0, w1, w2, u[20], m[19]); + MONT_MAC(t0, t1, t2, u[21], m[18]); + MONT_MAC(v0, v1, v2, u[22], m[17]); + MONT_MAC(w0, w1, w2, u[23], m[16]); + MONT_MAC(t0, t1, t2, u[24], m[15]); + MONT_MAC(v0, v1, v2, u[25], m[14]); + MONT_MAC(w0, w1, w2, u[26], m[13]); + MONT_MAC(t0, t1, t2, u[27], m[12]); + MONT_MAC(v0, v1, v2, u[28], m[11]); + MONT_MAC(w0, w1, w2, u[29], m[10]); + MONT_MAC(t0, t1, t2, u[30], m[9]); + MONT_MAC(v0, v1, v2, u[31], m[8]); + MONT_MAC(w0, w1, w2, u[32], m[7]); + MONT_MAC(t0, t1, t2, u[33], m[6]); + MONT_MAC(v0, v1, v2, u[34], m[5]); + MONT_MAC(w0, w1, w2, u[35], m[4]); + MONT_MAC(t0, t1, t2, u[36], m[3]); + MONT_MAC(v0, v1, v2, u[37], m[2]); + MONT_MAC(w0, w1, w2, u[38], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[39] = t0 * n0; + MONT_MAC(t0, t1, t2, u[39], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 40 */ + MONT_MUL(s0, s1, s2, a[0], a[40]); + MONT_MUL(c0, c1, c2, a[1], a[39]); + MONT_MAC(s0, s1, s2, a[2], a[38]); + MONT_MAC(c0, c1, c2, a[3], a[37]); + MONT_MAC(s0, s1, s2, a[4], a[36]); + MONT_MAC(c0, c1, c2, a[5], a[35]); + MONT_MAC(s0, s1, s2, a[6], a[34]); + MONT_MAC(c0, c1, c2, a[7], a[33]); + MONT_MAC(s0, s1, s2, a[8], a[32]); + MONT_MAC(c0, c1, c2, a[9], a[31]); + MONT_MAC(s0, s1, s2, a[10], a[30]); + MONT_MAC(c0, c1, c2, a[11], a[29]); + MONT_MAC(s0, s1, s2, a[12], a[28]); + MONT_MAC(c0, c1, c2, a[13], a[27]); + MONT_MAC(s0, s1, s2, a[14], a[26]); + MONT_MAC(c0, c1, c2, a[15], a[25]); + MONT_MAC(s0, s1, s2, a[16], a[24]); + MONT_MAC(c0, c1, c2, a[17], a[23]); + MONT_MAC(s0, s1, s2, a[18], a[22]); + MONT_MAC(c0, c1, c2, a[19], a[21]); + MONT_MAC(t0, t1, t2, a[20], a[20]); + MONT_MUL(v0, v1, v2, u[0], m[40]); + MONT_MUL(w0, w1, w2, u[1], m[39]); + MONT_MAC(t0, t1, t2, u[2], m[38]); + MONT_MAC(v0, v1, v2, u[3], m[37]); + MONT_MAC(w0, w1, w2, u[4], m[36]); + MONT_MAC(t0, t1, t2, u[5], m[35]); + MONT_MAC(v0, v1, v2, u[6], m[34]); + MONT_MAC(w0, w1, w2, u[7], m[33]); + MONT_MAC(t0, t1, t2, u[8], m[32]); + MONT_MAC(v0, v1, v2, u[9], m[31]); + MONT_MAC(w0, w1, w2, u[10], m[30]); + MONT_MAC(t0, t1, t2, u[11], m[29]); + MONT_MAC(v0, v1, v2, u[12], m[28]); + MONT_MAC(w0, w1, w2, u[13], m[27]); + MONT_MAC(t0, t1, t2, u[14], m[26]); + MONT_MAC(v0, v1, v2, u[15], m[25]); + MONT_MAC(w0, w1, w2, u[16], m[24]); + MONT_MAC(t0, t1, t2, u[17], m[23]); + MONT_MAC(v0, v1, v2, u[18], m[22]); + MONT_MAC(w0, w1, w2, u[19], m[21]); + MONT_MAC(t0, t1, t2, u[20], m[20]); + MONT_MAC(v0, v1, v2, u[21], m[19]); + MONT_MAC(w0, w1, w2, u[22], m[18]); + MONT_MAC(t0, t1, t2, u[23], m[17]); + MONT_MAC(v0, v1, v2, u[24], m[16]); + MONT_MAC(w0, w1, w2, u[25], m[15]); + MONT_MAC(t0, t1, t2, u[26], m[14]); + MONT_MAC(v0, v1, v2, u[27], m[13]); + MONT_MAC(w0, w1, w2, u[28], m[12]); + MONT_MAC(t0, t1, t2, u[29], m[11]); + MONT_MAC(v0, v1, v2, u[30], m[10]); + MONT_MAC(w0, w1, w2, u[31], m[9]); + MONT_MAC(t0, t1, t2, u[32], m[8]); + MONT_MAC(v0, v1, v2, u[33], m[7]); + MONT_MAC(w0, w1, w2, u[34], m[6]); + MONT_MAC(t0, t1, t2, u[35], m[5]); + MONT_MAC(v0, v1, v2, u[36], m[4]); + MONT_MAC(w0, w1, w2, u[37], m[3]); + MONT_MAC(t0, t1, t2, u[38], m[2]); + MONT_MAC(v0, v1, v2, u[39], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[40] = t0 * n0; + MONT_MAC(t0, t1, t2, u[40], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 41 */ + MONT_MUL(s0, s1, s2, a[0], a[41]); + MONT_MUL(c0, c1, c2, a[1], a[40]); + MONT_MAC(s0, s1, s2, a[2], a[39]); + MONT_MAC(c0, c1, c2, a[3], a[38]); + MONT_MAC(s0, s1, s2, a[4], a[37]); + MONT_MAC(c0, c1, c2, a[5], a[36]); + MONT_MAC(s0, s1, s2, a[6], a[35]); + MONT_MAC(c0, c1, c2, a[7], a[34]); + MONT_MAC(s0, s1, s2, a[8], a[33]); + MONT_MAC(c0, c1, c2, a[9], a[32]); + MONT_MAC(s0, s1, s2, a[10], a[31]); + MONT_MAC(c0, c1, c2, a[11], a[30]); + MONT_MAC(s0, s1, s2, a[12], a[29]); + MONT_MAC(c0, c1, c2, a[13], a[28]); + MONT_MAC(s0, s1, s2, a[14], a[27]); + MONT_MAC(c0, c1, c2, a[15], a[26]); + MONT_MAC(s0, s1, s2, a[16], a[25]); + MONT_MAC(c0, c1, c2, a[17], a[24]); + MONT_MAC(s0, s1, s2, a[18], a[23]); + MONT_MAC(c0, c1, c2, a[19], a[22]); + MONT_MAC(s0, s1, s2, a[20], a[21]); + MONT_MAC(t0, t1, t2, u[0], m[41]); + MONT_MUL(v0, v1, v2, u[1], m[40]); + MONT_MUL(w0, w1, w2, u[2], m[39]); + MONT_MAC(t0, t1, t2, u[3], m[38]); + MONT_MAC(v0, v1, v2, u[4], m[37]); + MONT_MAC(w0, w1, w2, u[5], m[36]); + MONT_MAC(t0, t1, t2, u[6], m[35]); + MONT_MAC(v0, v1, v2, u[7], m[34]); + MONT_MAC(w0, w1, w2, u[8], m[33]); + MONT_MAC(t0, t1, t2, u[9], m[32]); + MONT_MAC(v0, v1, v2, u[10], m[31]); + MONT_MAC(w0, w1, w2, u[11], m[30]); + MONT_MAC(t0, t1, t2, u[12], m[29]); + MONT_MAC(v0, v1, v2, u[13], m[28]); + MONT_MAC(w0, w1, w2, u[14], m[27]); + MONT_MAC(t0, t1, t2, u[15], m[26]); + MONT_MAC(v0, v1, v2, u[16], m[25]); + MONT_MAC(w0, w1, w2, u[17], m[24]); + MONT_MAC(t0, t1, t2, u[18], m[23]); + MONT_MAC(v0, v1, v2, u[19], m[22]); + MONT_MAC(w0, w1, w2, u[20], m[21]); + MONT_MAC(t0, t1, t2, u[21], m[20]); + MONT_MAC(v0, v1, v2, u[22], m[19]); + MONT_MAC(w0, w1, w2, u[23], m[18]); + MONT_MAC(t0, t1, t2, u[24], m[17]); + MONT_MAC(v0, v1, v2, u[25], m[16]); + MONT_MAC(w0, w1, w2, u[26], m[15]); + MONT_MAC(t0, t1, t2, u[27], m[14]); + MONT_MAC(v0, v1, v2, u[28], m[13]); + MONT_MAC(w0, w1, w2, u[29], m[12]); + MONT_MAC(t0, t1, t2, u[30], m[11]); + MONT_MAC(v0, v1, v2, u[31], m[10]); + MONT_MAC(w0, w1, w2, u[32], m[9]); + MONT_MAC(t0, t1, t2, u[33], m[8]); + MONT_MAC(v0, v1, v2, u[34], m[7]); + MONT_MAC(w0, w1, w2, u[35], m[6]); + MONT_MAC(t0, t1, t2, u[36], m[5]); + MONT_MAC(v0, v1, v2, u[37], m[4]); + MONT_MAC(w0, w1, w2, u[38], m[3]); + MONT_MAC(t0, t1, t2, u[39], m[2]); + MONT_MAC(v0, v1, v2, u[40], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[41] = t0 * n0; + MONT_MAC(t0, t1, t2, u[41], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 42 */ + MONT_MUL(s0, s1, s2, a[0], a[42]); + MONT_MUL(c0, c1, c2, a[1], a[41]); + MONT_MAC(s0, s1, s2, a[2], a[40]); + MONT_MAC(c0, c1, c2, a[3], a[39]); + MONT_MAC(s0, s1, s2, a[4], a[38]); + MONT_MAC(c0, c1, c2, a[5], a[37]); + MONT_MAC(s0, s1, s2, a[6], a[36]); + MONT_MAC(c0, c1, c2, a[7], a[35]); + MONT_MAC(s0, s1, s2, a[8], a[34]); + MONT_MAC(c0, c1, c2, a[9], a[33]); + MONT_MAC(s0, s1, s2, a[10], a[32]); + MONT_MAC(c0, c1, c2, a[11], a[31]); + MONT_MAC(s0, s1, s2, a[12], a[30]); + MONT_MAC(c0, c1, c2, a[13], a[29]); + MONT_MAC(s0, s1, s2, a[14], a[28]); + MONT_MAC(c0, c1, c2, a[15], a[27]); + MONT_MAC(s0, s1, s2, a[16], a[26]); + MONT_MAC(c0, c1, c2, a[17], a[25]); + MONT_MAC(s0, s1, s2, a[18], a[24]); + MONT_MAC(c0, c1, c2, a[19], a[23]); + MONT_MAC(s0, s1, s2, a[20], a[22]); + MONT_MAC(t0, t1, t2, a[21], a[21]); + MONT_MUL(v0, v1, v2, u[0], m[42]); + MONT_MUL(w0, w1, w2, u[1], m[41]); + MONT_MAC(t0, t1, t2, u[2], m[40]); + MONT_MAC(v0, v1, v2, u[3], m[39]); + MONT_MAC(w0, w1, w2, u[4], m[38]); + MONT_MAC(t0, t1, t2, u[5], m[37]); + MONT_MAC(v0, v1, v2, u[6], m[36]); + MONT_MAC(w0, w1, w2, u[7], m[35]); + MONT_MAC(t0, t1, t2, u[8], m[34]); + MONT_MAC(v0, v1, v2, u[9], m[33]); + MONT_MAC(w0, w1, w2, u[10], m[32]); + MONT_MAC(t0, t1, t2, u[11], m[31]); + MONT_MAC(v0, v1, v2, u[12], m[30]); + MONT_MAC(w0, w1, w2, u[13], m[29]); + MONT_MAC(t0, t1, t2, u[14], m[28]); + MONT_MAC(v0, v1, v2, u[15], m[27]); + MONT_MAC(w0, w1, w2, u[16], m[26]); + MONT_MAC(t0, t1, t2, u[17], m[25]); + MONT_MAC(v0, v1, v2, u[18], m[24]); + MONT_MAC(w0, w1, w2, u[19], m[23]); + MONT_MAC(t0, t1, t2, u[20], m[22]); + MONT_MAC(v0, v1, v2, u[21], m[21]); + MONT_MAC(w0, w1, w2, u[22], m[20]); + MONT_MAC(t0, t1, t2, u[23], m[19]); + MONT_MAC(v0, v1, v2, u[24], m[18]); + MONT_MAC(w0, w1, w2, u[25], m[17]); + MONT_MAC(t0, t1, t2, u[26], m[16]); + MONT_MAC(v0, v1, v2, u[27], m[15]); + MONT_MAC(w0, w1, w2, u[28], m[14]); + MONT_MAC(t0, t1, t2, u[29], m[13]); + MONT_MAC(v0, v1, v2, u[30], m[12]); + MONT_MAC(w0, w1, w2, u[31], m[11]); + MONT_MAC(t0, t1, t2, u[32], m[10]); + MONT_MAC(v0, v1, v2, u[33], m[9]); + MONT_MAC(w0, w1, w2, u[34], m[8]); + MONT_MAC(t0, t1, t2, u[35], m[7]); + MONT_MAC(v0, v1, v2, u[36], m[6]); + MONT_MAC(w0, w1, w2, u[37], m[5]); + MONT_MAC(t0, t1, t2, u[38], m[4]); + MONT_MAC(v0, v1, v2, u[39], m[3]); + MONT_MAC(w0, w1, w2, u[40], m[2]); + MONT_MAC(t0, t1, t2, u[41], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[42] = t0 * n0; + MONT_MAC(t0, t1, t2, u[42], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 43 */ + MONT_MUL(s0, s1, s2, a[0], a[43]); + MONT_MUL(c0, c1, c2, a[1], a[42]); + MONT_MAC(s0, s1, s2, a[2], a[41]); + MONT_MAC(c0, c1, c2, a[3], a[40]); + MONT_MAC(s0, s1, s2, a[4], a[39]); + MONT_MAC(c0, c1, c2, a[5], a[38]); + MONT_MAC(s0, s1, s2, a[6], a[37]); + MONT_MAC(c0, c1, c2, a[7], a[36]); + MONT_MAC(s0, s1, s2, a[8], a[35]); + MONT_MAC(c0, c1, c2, a[9], a[34]); + MONT_MAC(s0, s1, s2, a[10], a[33]); + MONT_MAC(c0, c1, c2, a[11], a[32]); + MONT_MAC(s0, s1, s2, a[12], a[31]); + MONT_MAC(c0, c1, c2, a[13], a[30]); + MONT_MAC(s0, s1, s2, a[14], a[29]); + MONT_MAC(c0, c1, c2, a[15], a[28]); + MONT_MAC(s0, s1, s2, a[16], a[27]); + MONT_MAC(c0, c1, c2, a[17], a[26]); + MONT_MAC(s0, s1, s2, a[18], a[25]); + MONT_MAC(c0, c1, c2, a[19], a[24]); + MONT_MAC(s0, s1, s2, a[20], a[23]); + MONT_MAC(c0, c1, c2, a[21], a[22]); + MONT_MAC(t0, t1, t2, u[0], m[43]); + MONT_MUL(v0, v1, v2, u[1], m[42]); + MONT_MUL(w0, w1, w2, u[2], m[41]); + MONT_MAC(t0, t1, t2, u[3], m[40]); + MONT_MAC(v0, v1, v2, u[4], m[39]); + MONT_MAC(w0, w1, w2, u[5], m[38]); + MONT_MAC(t0, t1, t2, u[6], m[37]); + MONT_MAC(v0, v1, v2, u[7], m[36]); + MONT_MAC(w0, w1, w2, u[8], m[35]); + MONT_MAC(t0, t1, t2, u[9], m[34]); + MONT_MAC(v0, v1, v2, u[10], m[33]); + MONT_MAC(w0, w1, w2, u[11], m[32]); + MONT_MAC(t0, t1, t2, u[12], m[31]); + MONT_MAC(v0, v1, v2, u[13], m[30]); + MONT_MAC(w0, w1, w2, u[14], m[29]); + MONT_MAC(t0, t1, t2, u[15], m[28]); + MONT_MAC(v0, v1, v2, u[16], m[27]); + MONT_MAC(w0, w1, w2, u[17], m[26]); + MONT_MAC(t0, t1, t2, u[18], m[25]); + MONT_MAC(v0, v1, v2, u[19], m[24]); + MONT_MAC(w0, w1, w2, u[20], m[23]); + MONT_MAC(t0, t1, t2, u[21], m[22]); + MONT_MAC(v0, v1, v2, u[22], m[21]); + MONT_MAC(w0, w1, w2, u[23], m[20]); + MONT_MAC(t0, t1, t2, u[24], m[19]); + MONT_MAC(v0, v1, v2, u[25], m[18]); + MONT_MAC(w0, w1, w2, u[26], m[17]); + MONT_MAC(t0, t1, t2, u[27], m[16]); + MONT_MAC(v0, v1, v2, u[28], m[15]); + MONT_MAC(w0, w1, w2, u[29], m[14]); + MONT_MAC(t0, t1, t2, u[30], m[13]); + MONT_MAC(v0, v1, v2, u[31], m[12]); + MONT_MAC(w0, w1, w2, u[32], m[11]); + MONT_MAC(t0, t1, t2, u[33], m[10]); + MONT_MAC(v0, v1, v2, u[34], m[9]); + MONT_MAC(w0, w1, w2, u[35], m[8]); + MONT_MAC(t0, t1, t2, u[36], m[7]); + MONT_MAC(v0, v1, v2, u[37], m[6]); + MONT_MAC(w0, w1, w2, u[38], m[5]); + MONT_MAC(t0, t1, t2, u[39], m[4]); + MONT_MAC(v0, v1, v2, u[40], m[3]); + MONT_MAC(w0, w1, w2, u[41], m[2]); + MONT_MAC(t0, t1, t2, u[42], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[43] = t0 * n0; + MONT_MAC(t0, t1, t2, u[43], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 44 */ + MONT_MUL(s0, s1, s2, a[0], a[44]); + MONT_MUL(c0, c1, c2, a[1], a[43]); + MONT_MAC(s0, s1, s2, a[2], a[42]); + MONT_MAC(c0, c1, c2, a[3], a[41]); + MONT_MAC(s0, s1, s2, a[4], a[40]); + MONT_MAC(c0, c1, c2, a[5], a[39]); + MONT_MAC(s0, s1, s2, a[6], a[38]); + MONT_MAC(c0, c1, c2, a[7], a[37]); + MONT_MAC(s0, s1, s2, a[8], a[36]); + MONT_MAC(c0, c1, c2, a[9], a[35]); + MONT_MAC(s0, s1, s2, a[10], a[34]); + MONT_MAC(c0, c1, c2, a[11], a[33]); + MONT_MAC(s0, s1, s2, a[12], a[32]); + MONT_MAC(c0, c1, c2, a[13], a[31]); + MONT_MAC(s0, s1, s2, a[14], a[30]); + MONT_MAC(c0, c1, c2, a[15], a[29]); + MONT_MAC(s0, s1, s2, a[16], a[28]); + MONT_MAC(c0, c1, c2, a[17], a[27]); + MONT_MAC(s0, s1, s2, a[18], a[26]); + MONT_MAC(c0, c1, c2, a[19], a[25]); + MONT_MAC(s0, s1, s2, a[20], a[24]); + MONT_MAC(c0, c1, c2, a[21], a[23]); + MONT_MAC(t0, t1, t2, a[22], a[22]); + MONT_MUL(v0, v1, v2, u[0], m[44]); + MONT_MUL(w0, w1, w2, u[1], m[43]); + MONT_MAC(t0, t1, t2, u[2], m[42]); + MONT_MAC(v0, v1, v2, u[3], m[41]); + MONT_MAC(w0, w1, w2, u[4], m[40]); + MONT_MAC(t0, t1, t2, u[5], m[39]); + MONT_MAC(v0, v1, v2, u[6], m[38]); + MONT_MAC(w0, w1, w2, u[7], m[37]); + MONT_MAC(t0, t1, t2, u[8], m[36]); + MONT_MAC(v0, v1, v2, u[9], m[35]); + MONT_MAC(w0, w1, w2, u[10], m[34]); + MONT_MAC(t0, t1, t2, u[11], m[33]); + MONT_MAC(v0, v1, v2, u[12], m[32]); + MONT_MAC(w0, w1, w2, u[13], m[31]); + MONT_MAC(t0, t1, t2, u[14], m[30]); + MONT_MAC(v0, v1, v2, u[15], m[29]); + MONT_MAC(w0, w1, w2, u[16], m[28]); + MONT_MAC(t0, t1, t2, u[17], m[27]); + MONT_MAC(v0, v1, v2, u[18], m[26]); + MONT_MAC(w0, w1, w2, u[19], m[25]); + MONT_MAC(t0, t1, t2, u[20], m[24]); + MONT_MAC(v0, v1, v2, u[21], m[23]); + MONT_MAC(w0, w1, w2, u[22], m[22]); + MONT_MAC(t0, t1, t2, u[23], m[21]); + MONT_MAC(v0, v1, v2, u[24], m[20]); + MONT_MAC(w0, w1, w2, u[25], m[19]); + MONT_MAC(t0, t1, t2, u[26], m[18]); + MONT_MAC(v0, v1, v2, u[27], m[17]); + MONT_MAC(w0, w1, w2, u[28], m[16]); + MONT_MAC(t0, t1, t2, u[29], m[15]); + MONT_MAC(v0, v1, v2, u[30], m[14]); + MONT_MAC(w0, w1, w2, u[31], m[13]); + MONT_MAC(t0, t1, t2, u[32], m[12]); + MONT_MAC(v0, v1, v2, u[33], m[11]); + MONT_MAC(w0, w1, w2, u[34], m[10]); + MONT_MAC(t0, t1, t2, u[35], m[9]); + MONT_MAC(v0, v1, v2, u[36], m[8]); + MONT_MAC(w0, w1, w2, u[37], m[7]); + MONT_MAC(t0, t1, t2, u[38], m[6]); + MONT_MAC(v0, v1, v2, u[39], m[5]); + MONT_MAC(w0, w1, w2, u[40], m[4]); + MONT_MAC(t0, t1, t2, u[41], m[3]); + MONT_MAC(v0, v1, v2, u[42], m[2]); + MONT_MAC(w0, w1, w2, u[43], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[44] = t0 * n0; + MONT_MAC(t0, t1, t2, u[44], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 45 */ + MONT_MUL(s0, s1, s2, a[0], a[45]); + MONT_MUL(c0, c1, c2, a[1], a[44]); + MONT_MAC(s0, s1, s2, a[2], a[43]); + MONT_MAC(c0, c1, c2, a[3], a[42]); + MONT_MAC(s0, s1, s2, a[4], a[41]); + MONT_MAC(c0, c1, c2, a[5], a[40]); + MONT_MAC(s0, s1, s2, a[6], a[39]); + MONT_MAC(c0, c1, c2, a[7], a[38]); + MONT_MAC(s0, s1, s2, a[8], a[37]); + MONT_MAC(c0, c1, c2, a[9], a[36]); + MONT_MAC(s0, s1, s2, a[10], a[35]); + MONT_MAC(c0, c1, c2, a[11], a[34]); + MONT_MAC(s0, s1, s2, a[12], a[33]); + MONT_MAC(c0, c1, c2, a[13], a[32]); + MONT_MAC(s0, s1, s2, a[14], a[31]); + MONT_MAC(c0, c1, c2, a[15], a[30]); + MONT_MAC(s0, s1, s2, a[16], a[29]); + MONT_MAC(c0, c1, c2, a[17], a[28]); + MONT_MAC(s0, s1, s2, a[18], a[27]); + MONT_MAC(c0, c1, c2, a[19], a[26]); + MONT_MAC(s0, s1, s2, a[20], a[25]); + MONT_MAC(c0, c1, c2, a[21], a[24]); + MONT_MAC(s0, s1, s2, a[22], a[23]); + MONT_MAC(t0, t1, t2, u[0], m[45]); + MONT_MUL(v0, v1, v2, u[1], m[44]); + MONT_MUL(w0, w1, w2, u[2], m[43]); + MONT_MAC(t0, t1, t2, u[3], m[42]); + MONT_MAC(v0, v1, v2, u[4], m[41]); + MONT_MAC(w0, w1, w2, u[5], m[40]); + MONT_MAC(t0, t1, t2, u[6], m[39]); + MONT_MAC(v0, v1, v2, u[7], m[38]); + MONT_MAC(w0, w1, w2, u[8], m[37]); + MONT_MAC(t0, t1, t2, u[9], m[36]); + MONT_MAC(v0, v1, v2, u[10], m[35]); + MONT_MAC(w0, w1, w2, u[11], m[34]); + MONT_MAC(t0, t1, t2, u[12], m[33]); + MONT_MAC(v0, v1, v2, u[13], m[32]); + MONT_MAC(w0, w1, w2, u[14], m[31]); + MONT_MAC(t0, t1, t2, u[15], m[30]); + MONT_MAC(v0, v1, v2, u[16], m[29]); + MONT_MAC(w0, w1, w2, u[17], m[28]); + MONT_MAC(t0, t1, t2, u[18], m[27]); + MONT_MAC(v0, v1, v2, u[19], m[26]); + MONT_MAC(w0, w1, w2, u[20], m[25]); + MONT_MAC(t0, t1, t2, u[21], m[24]); + MONT_MAC(v0, v1, v2, u[22], m[23]); + MONT_MAC(w0, w1, w2, u[23], m[22]); + MONT_MAC(t0, t1, t2, u[24], m[21]); + MONT_MAC(v0, v1, v2, u[25], m[20]); + MONT_MAC(w0, w1, w2, u[26], m[19]); + MONT_MAC(t0, t1, t2, u[27], m[18]); + MONT_MAC(v0, v1, v2, u[28], m[17]); + MONT_MAC(w0, w1, w2, u[29], m[16]); + MONT_MAC(t0, t1, t2, u[30], m[15]); + MONT_MAC(v0, v1, v2, u[31], m[14]); + MONT_MAC(w0, w1, w2, u[32], m[13]); + MONT_MAC(t0, t1, t2, u[33], m[12]); + MONT_MAC(v0, v1, v2, u[34], m[11]); + MONT_MAC(w0, w1, w2, u[35], m[10]); + MONT_MAC(t0, t1, t2, u[36], m[9]); + MONT_MAC(v0, v1, v2, u[37], m[8]); + MONT_MAC(w0, w1, w2, u[38], m[7]); + MONT_MAC(t0, t1, t2, u[39], m[6]); + MONT_MAC(v0, v1, v2, u[40], m[5]); + MONT_MAC(w0, w1, w2, u[41], m[4]); + MONT_MAC(t0, t1, t2, u[42], m[3]); + MONT_MAC(v0, v1, v2, u[43], m[2]); + MONT_MAC(w0, w1, w2, u[44], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[45] = t0 * n0; + MONT_MAC(t0, t1, t2, u[45], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 46 */ + MONT_MUL(s0, s1, s2, a[0], a[46]); + MONT_MUL(c0, c1, c2, a[1], a[45]); + MONT_MAC(s0, s1, s2, a[2], a[44]); + MONT_MAC(c0, c1, c2, a[3], a[43]); + MONT_MAC(s0, s1, s2, a[4], a[42]); + MONT_MAC(c0, c1, c2, a[5], a[41]); + MONT_MAC(s0, s1, s2, a[6], a[40]); + MONT_MAC(c0, c1, c2, a[7], a[39]); + MONT_MAC(s0, s1, s2, a[8], a[38]); + MONT_MAC(c0, c1, c2, a[9], a[37]); + MONT_MAC(s0, s1, s2, a[10], a[36]); + MONT_MAC(c0, c1, c2, a[11], a[35]); + MONT_MAC(s0, s1, s2, a[12], a[34]); + MONT_MAC(c0, c1, c2, a[13], a[33]); + MONT_MAC(s0, s1, s2, a[14], a[32]); + MONT_MAC(c0, c1, c2, a[15], a[31]); + MONT_MAC(s0, s1, s2, a[16], a[30]); + MONT_MAC(c0, c1, c2, a[17], a[29]); + MONT_MAC(s0, s1, s2, a[18], a[28]); + MONT_MAC(c0, c1, c2, a[19], a[27]); + MONT_MAC(s0, s1, s2, a[20], a[26]); + MONT_MAC(c0, c1, c2, a[21], a[25]); + MONT_MAC(s0, s1, s2, a[22], a[24]); + MONT_MAC(t0, t1, t2, a[23], a[23]); + MONT_MUL(v0, v1, v2, u[0], m[46]); + MONT_MUL(w0, w1, w2, u[1], m[45]); + MONT_MAC(t0, t1, t2, u[2], m[44]); + MONT_MAC(v0, v1, v2, u[3], m[43]); + MONT_MAC(w0, w1, w2, u[4], m[42]); + MONT_MAC(t0, t1, t2, u[5], m[41]); + MONT_MAC(v0, v1, v2, u[6], m[40]); + MONT_MAC(w0, w1, w2, u[7], m[39]); + MONT_MAC(t0, t1, t2, u[8], m[38]); + MONT_MAC(v0, v1, v2, u[9], m[37]); + MONT_MAC(w0, w1, w2, u[10], m[36]); + MONT_MAC(t0, t1, t2, u[11], m[35]); + MONT_MAC(v0, v1, v2, u[12], m[34]); + MONT_MAC(w0, w1, w2, u[13], m[33]); + MONT_MAC(t0, t1, t2, u[14], m[32]); + MONT_MAC(v0, v1, v2, u[15], m[31]); + MONT_MAC(w0, w1, w2, u[16], m[30]); + MONT_MAC(t0, t1, t2, u[17], m[29]); + MONT_MAC(v0, v1, v2, u[18], m[28]); + MONT_MAC(w0, w1, w2, u[19], m[27]); + MONT_MAC(t0, t1, t2, u[20], m[26]); + MONT_MAC(v0, v1, v2, u[21], m[25]); + MONT_MAC(w0, w1, w2, u[22], m[24]); + MONT_MAC(t0, t1, t2, u[23], m[23]); + MONT_MAC(v0, v1, v2, u[24], m[22]); + MONT_MAC(w0, w1, w2, u[25], m[21]); + MONT_MAC(t0, t1, t2, u[26], m[20]); + MONT_MAC(v0, v1, v2, u[27], m[19]); + MONT_MAC(w0, w1, w2, u[28], m[18]); + MONT_MAC(t0, t1, t2, u[29], m[17]); + MONT_MAC(v0, v1, v2, u[30], m[16]); + MONT_MAC(w0, w1, w2, u[31], m[15]); + MONT_MAC(t0, t1, t2, u[32], m[14]); + MONT_MAC(v0, v1, v2, u[33], m[13]); + MONT_MAC(w0, w1, w2, u[34], m[12]); + MONT_MAC(t0, t1, t2, u[35], m[11]); + MONT_MAC(v0, v1, v2, u[36], m[10]); + MONT_MAC(w0, w1, w2, u[37], m[9]); + MONT_MAC(t0, t1, t2, u[38], m[8]); + MONT_MAC(v0, v1, v2, u[39], m[7]); + MONT_MAC(w0, w1, w2, u[40], m[6]); + MONT_MAC(t0, t1, t2, u[41], m[5]); + MONT_MAC(v0, v1, v2, u[42], m[4]); + MONT_MAC(w0, w1, w2, u[43], m[3]); + MONT_MAC(t0, t1, t2, u[44], m[2]); + MONT_MAC(v0, v1, v2, u[45], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[46] = t0 * n0; + MONT_MAC(t0, t1, t2, u[46], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 47 */ + MONT_MUL(s0, s1, s2, a[0], a[47]); + MONT_MUL(c0, c1, c2, a[1], a[46]); + MONT_MAC(s0, s1, s2, a[2], a[45]); + MONT_MAC(c0, c1, c2, a[3], a[44]); + MONT_MAC(s0, s1, s2, a[4], a[43]); + MONT_MAC(c0, c1, c2, a[5], a[42]); + MONT_MAC(s0, s1, s2, a[6], a[41]); + MONT_MAC(c0, c1, c2, a[7], a[40]); + MONT_MAC(s0, s1, s2, a[8], a[39]); + MONT_MAC(c0, c1, c2, a[9], a[38]); + MONT_MAC(s0, s1, s2, a[10], a[37]); + MONT_MAC(c0, c1, c2, a[11], a[36]); + MONT_MAC(s0, s1, s2, a[12], a[35]); + MONT_MAC(c0, c1, c2, a[13], a[34]); + MONT_MAC(s0, s1, s2, a[14], a[33]); + MONT_MAC(c0, c1, c2, a[15], a[32]); + MONT_MAC(s0, s1, s2, a[16], a[31]); + MONT_MAC(c0, c1, c2, a[17], a[30]); + MONT_MAC(s0, s1, s2, a[18], a[29]); + MONT_MAC(c0, c1, c2, a[19], a[28]); + MONT_MAC(s0, s1, s2, a[20], a[27]); + MONT_MAC(c0, c1, c2, a[21], a[26]); + MONT_MAC(s0, s1, s2, a[22], a[25]); + MONT_MAC(c0, c1, c2, a[23], a[24]); + MONT_MAC(t0, t1, t2, u[0], m[47]); + MONT_MUL(v0, v1, v2, u[1], m[46]); + MONT_MUL(w0, w1, w2, u[2], m[45]); + MONT_MAC(t0, t1, t2, u[3], m[44]); + MONT_MAC(v0, v1, v2, u[4], m[43]); + MONT_MAC(w0, w1, w2, u[5], m[42]); + MONT_MAC(t0, t1, t2, u[6], m[41]); + MONT_MAC(v0, v1, v2, u[7], m[40]); + MONT_MAC(w0, w1, w2, u[8], m[39]); + MONT_MAC(t0, t1, t2, u[9], m[38]); + MONT_MAC(v0, v1, v2, u[10], m[37]); + MONT_MAC(w0, w1, w2, u[11], m[36]); + MONT_MAC(t0, t1, t2, u[12], m[35]); + MONT_MAC(v0, v1, v2, u[13], m[34]); + MONT_MAC(w0, w1, w2, u[14], m[33]); + MONT_MAC(t0, t1, t2, u[15], m[32]); + MONT_MAC(v0, v1, v2, u[16], m[31]); + MONT_MAC(w0, w1, w2, u[17], m[30]); + MONT_MAC(t0, t1, t2, u[18], m[29]); + MONT_MAC(v0, v1, v2, u[19], m[28]); + MONT_MAC(w0, w1, w2, u[20], m[27]); + MONT_MAC(t0, t1, t2, u[21], m[26]); + MONT_MAC(v0, v1, v2, u[22], m[25]); + MONT_MAC(w0, w1, w2, u[23], m[24]); + MONT_MAC(t0, t1, t2, u[24], m[23]); + MONT_MAC(v0, v1, v2, u[25], m[22]); + MONT_MAC(w0, w1, w2, u[26], m[21]); + MONT_MAC(t0, t1, t2, u[27], m[20]); + MONT_MAC(v0, v1, v2, u[28], m[19]); + MONT_MAC(w0, w1, w2, u[29], m[18]); + MONT_MAC(t0, t1, t2, u[30], m[17]); + MONT_MAC(v0, v1, v2, u[31], m[16]); + MONT_MAC(w0, w1, w2, u[32], m[15]); + MONT_MAC(t0, t1, t2, u[33], m[14]); + MONT_MAC(v0, v1, v2, u[34], m[13]); + MONT_MAC(w0, w1, w2, u[35], m[12]); + MONT_MAC(t0, t1, t2, u[36], m[11]); + MONT_MAC(v0, v1, v2, u[37], m[10]); + MONT_MAC(w0, w1, w2, u[38], m[9]); + MONT_MAC(t0, t1, t2, u[39], m[8]); + MONT_MAC(v0, v1, v2, u[40], m[7]); + MONT_MAC(w0, w1, w2, u[41], m[6]); + MONT_MAC(t0, t1, t2, u[42], m[5]); + MONT_MAC(v0, v1, v2, u[43], m[4]); + MONT_MAC(w0, w1, w2, u[44], m[3]); + MONT_MAC(t0, t1, t2, u[45], m[2]); + MONT_MAC(v0, v1, v2, u[46], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[47] = t0 * n0; + MONT_MAC(t0, t1, t2, u[47], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 48 */ + MONT_MUL(s0, s1, s2, a[1], a[47]); + MONT_MUL(c0, c1, c2, a[2], a[46]); + MONT_MAC(s0, s1, s2, a[3], a[45]); + MONT_MAC(c0, c1, c2, a[4], a[44]); + MONT_MAC(s0, s1, s2, a[5], a[43]); + MONT_MAC(c0, c1, c2, a[6], a[42]); + MONT_MAC(s0, s1, s2, a[7], a[41]); + MONT_MAC(c0, c1, c2, a[8], a[40]); + MONT_MAC(s0, s1, s2, a[9], a[39]); + MONT_MAC(c0, c1, c2, a[10], a[38]); + MONT_MAC(s0, s1, s2, a[11], a[37]); + MONT_MAC(c0, c1, c2, a[12], a[36]); + MONT_MAC(s0, s1, s2, a[13], a[35]); + MONT_MAC(c0, c1, c2, a[14], a[34]); + MONT_MAC(s0, s1, s2, a[15], a[33]); + MONT_MAC(c0, c1, c2, a[16], a[32]); + MONT_MAC(s0, s1, s2, a[17], a[31]); + MONT_MAC(c0, c1, c2, a[18], a[30]); + MONT_MAC(s0, s1, s2, a[19], a[29]); + MONT_MAC(c0, c1, c2, a[20], a[28]); + MONT_MAC(s0, s1, s2, a[21], a[27]); + MONT_MAC(c0, c1, c2, a[22], a[26]); + MONT_MAC(s0, s1, s2, a[23], a[25]); + MONT_MAC(t0, t1, t2, a[24], a[24]); + MONT_MUL(v0, v1, v2, u[1], m[47]); + MONT_MUL(w0, w1, w2, u[2], m[46]); + MONT_MAC(t0, t1, t2, u[3], m[45]); + MONT_MAC(v0, v1, v2, u[4], m[44]); + MONT_MAC(w0, w1, w2, u[5], m[43]); + MONT_MAC(t0, t1, t2, u[6], m[42]); + MONT_MAC(v0, v1, v2, u[7], m[41]); + MONT_MAC(w0, w1, w2, u[8], m[40]); + MONT_MAC(t0, t1, t2, u[9], m[39]); + MONT_MAC(v0, v1, v2, u[10], m[38]); + MONT_MAC(w0, w1, w2, u[11], m[37]); + MONT_MAC(t0, t1, t2, u[12], m[36]); + MONT_MAC(v0, v1, v2, u[13], m[35]); + MONT_MAC(w0, w1, w2, u[14], m[34]); + MONT_MAC(t0, t1, t2, u[15], m[33]); + MONT_MAC(v0, v1, v2, u[16], m[32]); + MONT_MAC(w0, w1, w2, u[17], m[31]); + MONT_MAC(t0, t1, t2, u[18], m[30]); + MONT_MAC(v0, v1, v2, u[19], m[29]); + MONT_MAC(w0, w1, w2, u[20], m[28]); + MONT_MAC(t0, t1, t2, u[21], m[27]); + MONT_MAC(v0, v1, v2, u[22], m[26]); + MONT_MAC(w0, w1, w2, u[23], m[25]); + MONT_MAC(t0, t1, t2, u[24], m[24]); + MONT_MAC(v0, v1, v2, u[25], m[23]); + MONT_MAC(w0, w1, w2, u[26], m[22]); + MONT_MAC(t0, t1, t2, u[27], m[21]); + MONT_MAC(v0, v1, v2, u[28], m[20]); + MONT_MAC(w0, w1, w2, u[29], m[19]); + MONT_MAC(t0, t1, t2, u[30], m[18]); + MONT_MAC(v0, v1, v2, u[31], m[17]); + MONT_MAC(w0, w1, w2, u[32], m[16]); + MONT_MAC(t0, t1, t2, u[33], m[15]); + MONT_MAC(v0, v1, v2, u[34], m[14]); + MONT_MAC(w0, w1, w2, u[35], m[13]); + MONT_MAC(t0, t1, t2, u[36], m[12]); + MONT_MAC(v0, v1, v2, u[37], m[11]); + MONT_MAC(w0, w1, w2, u[38], m[10]); + MONT_MAC(t0, t1, t2, u[39], m[9]); + MONT_MAC(v0, v1, v2, u[40], m[8]); + MONT_MAC(w0, w1, w2, u[41], m[7]); + MONT_MAC(t0, t1, t2, u[42], m[6]); + MONT_MAC(v0, v1, v2, u[43], m[5]); + MONT_MAC(w0, w1, w2, u[44], m[4]); + MONT_MAC(t0, t1, t2, u[45], m[3]); + MONT_MAC(v0, v1, v2, u[46], m[2]); + MONT_MAC(w0, w1, w2, u[47], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[0] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 49 */ + MONT_MUL(s0, s1, s2, a[2], a[47]); + MONT_MUL(c0, c1, c2, a[3], a[46]); + MONT_MAC(s0, s1, s2, a[4], a[45]); + MONT_MAC(c0, c1, c2, a[5], a[44]); + MONT_MAC(s0, s1, s2, a[6], a[43]); + MONT_MAC(c0, c1, c2, a[7], a[42]); + MONT_MAC(s0, s1, s2, a[8], a[41]); + MONT_MAC(c0, c1, c2, a[9], a[40]); + MONT_MAC(s0, s1, s2, a[10], a[39]); + MONT_MAC(c0, c1, c2, a[11], a[38]); + MONT_MAC(s0, s1, s2, a[12], a[37]); + MONT_MAC(c0, c1, c2, a[13], a[36]); + MONT_MAC(s0, s1, s2, a[14], a[35]); + MONT_MAC(c0, c1, c2, a[15], a[34]); + MONT_MAC(s0, s1, s2, a[16], a[33]); + MONT_MAC(c0, c1, c2, a[17], a[32]); + MONT_MAC(s0, s1, s2, a[18], a[31]); + MONT_MAC(c0, c1, c2, a[19], a[30]); + MONT_MAC(s0, s1, s2, a[20], a[29]); + MONT_MAC(c0, c1, c2, a[21], a[28]); + MONT_MAC(s0, s1, s2, a[22], a[27]); + MONT_MAC(c0, c1, c2, a[23], a[26]); + MONT_MAC(s0, s1, s2, a[24], a[25]); + MONT_MAC(t0, t1, t2, u[2], m[47]); + MONT_MUL(v0, v1, v2, u[3], m[46]); + MONT_MUL(w0, w1, w2, u[4], m[45]); + MONT_MAC(t0, t1, t2, u[5], m[44]); + MONT_MAC(v0, v1, v2, u[6], m[43]); + MONT_MAC(w0, w1, w2, u[7], m[42]); + MONT_MAC(t0, t1, t2, u[8], m[41]); + MONT_MAC(v0, v1, v2, u[9], m[40]); + MONT_MAC(w0, w1, w2, u[10], m[39]); + MONT_MAC(t0, t1, t2, u[11], m[38]); + MONT_MAC(v0, v1, v2, u[12], m[37]); + MONT_MAC(w0, w1, w2, u[13], m[36]); + MONT_MAC(t0, t1, t2, u[14], m[35]); + MONT_MAC(v0, v1, v2, u[15], m[34]); + MONT_MAC(w0, w1, w2, u[16], m[33]); + MONT_MAC(t0, t1, t2, u[17], m[32]); + MONT_MAC(v0, v1, v2, u[18], m[31]); + MONT_MAC(w0, w1, w2, u[19], m[30]); + MONT_MAC(t0, t1, t2, u[20], m[29]); + MONT_MAC(v0, v1, v2, u[21], m[28]); + MONT_MAC(w0, w1, w2, u[22], m[27]); + MONT_MAC(t0, t1, t2, u[23], m[26]); + MONT_MAC(v0, v1, v2, u[24], m[25]); + MONT_MAC(w0, w1, w2, u[25], m[24]); + MONT_MAC(t0, t1, t2, u[26], m[23]); + MONT_MAC(v0, v1, v2, u[27], m[22]); + MONT_MAC(w0, w1, w2, u[28], m[21]); + MONT_MAC(t0, t1, t2, u[29], m[20]); + MONT_MAC(v0, v1, v2, u[30], m[19]); + MONT_MAC(w0, w1, w2, u[31], m[18]); + MONT_MAC(t0, t1, t2, u[32], m[17]); + MONT_MAC(v0, v1, v2, u[33], m[16]); + MONT_MAC(w0, w1, w2, u[34], m[15]); + MONT_MAC(t0, t1, t2, u[35], m[14]); + MONT_MAC(v0, v1, v2, u[36], m[13]); + MONT_MAC(w0, w1, w2, u[37], m[12]); + MONT_MAC(t0, t1, t2, u[38], m[11]); + MONT_MAC(v0, v1, v2, u[39], m[10]); + MONT_MAC(w0, w1, w2, u[40], m[9]); + MONT_MAC(t0, t1, t2, u[41], m[8]); + MONT_MAC(v0, v1, v2, u[42], m[7]); + MONT_MAC(w0, w1, w2, u[43], m[6]); + MONT_MAC(t0, t1, t2, u[44], m[5]); + MONT_MAC(v0, v1, v2, u[45], m[4]); + MONT_MAC(w0, w1, w2, u[46], m[3]); + MONT_MAC(t0, t1, t2, u[47], m[2]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[1] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 50 */ + MONT_MUL(s0, s1, s2, a[3], a[47]); + MONT_MUL(c0, c1, c2, a[4], a[46]); + MONT_MAC(s0, s1, s2, a[5], a[45]); + MONT_MAC(c0, c1, c2, a[6], a[44]); + MONT_MAC(s0, s1, s2, a[7], a[43]); + MONT_MAC(c0, c1, c2, a[8], a[42]); + MONT_MAC(s0, s1, s2, a[9], a[41]); + MONT_MAC(c0, c1, c2, a[10], a[40]); + MONT_MAC(s0, s1, s2, a[11], a[39]); + MONT_MAC(c0, c1, c2, a[12], a[38]); + MONT_MAC(s0, s1, s2, a[13], a[37]); + MONT_MAC(c0, c1, c2, a[14], a[36]); + MONT_MAC(s0, s1, s2, a[15], a[35]); + MONT_MAC(c0, c1, c2, a[16], a[34]); + MONT_MAC(s0, s1, s2, a[17], a[33]); + MONT_MAC(c0, c1, c2, a[18], a[32]); + MONT_MAC(s0, s1, s2, a[19], a[31]); + MONT_MAC(c0, c1, c2, a[20], a[30]); + MONT_MAC(s0, s1, s2, a[21], a[29]); + MONT_MAC(c0, c1, c2, a[22], a[28]); + MONT_MAC(s0, s1, s2, a[23], a[27]); + MONT_MAC(c0, c1, c2, a[24], a[26]); + MONT_MAC(t0, t1, t2, a[25], a[25]); + MONT_MUL(v0, v1, v2, u[3], m[47]); + MONT_MUL(w0, w1, w2, u[4], m[46]); + MONT_MAC(t0, t1, t2, u[5], m[45]); + MONT_MAC(v0, v1, v2, u[6], m[44]); + MONT_MAC(w0, w1, w2, u[7], m[43]); + MONT_MAC(t0, t1, t2, u[8], m[42]); + MONT_MAC(v0, v1, v2, u[9], m[41]); + MONT_MAC(w0, w1, w2, u[10], m[40]); + MONT_MAC(t0, t1, t2, u[11], m[39]); + MONT_MAC(v0, v1, v2, u[12], m[38]); + MONT_MAC(w0, w1, w2, u[13], m[37]); + MONT_MAC(t0, t1, t2, u[14], m[36]); + MONT_MAC(v0, v1, v2, u[15], m[35]); + MONT_MAC(w0, w1, w2, u[16], m[34]); + MONT_MAC(t0, t1, t2, u[17], m[33]); + MONT_MAC(v0, v1, v2, u[18], m[32]); + MONT_MAC(w0, w1, w2, u[19], m[31]); + MONT_MAC(t0, t1, t2, u[20], m[30]); + MONT_MAC(v0, v1, v2, u[21], m[29]); + MONT_MAC(w0, w1, w2, u[22], m[28]); + MONT_MAC(t0, t1, t2, u[23], m[27]); + MONT_MAC(v0, v1, v2, u[24], m[26]); + MONT_MAC(w0, w1, w2, u[25], m[25]); + MONT_MAC(t0, t1, t2, u[26], m[24]); + MONT_MAC(v0, v1, v2, u[27], m[23]); + MONT_MAC(w0, w1, w2, u[28], m[22]); + MONT_MAC(t0, t1, t2, u[29], m[21]); + MONT_MAC(v0, v1, v2, u[30], m[20]); + MONT_MAC(w0, w1, w2, u[31], m[19]); + MONT_MAC(t0, t1, t2, u[32], m[18]); + MONT_MAC(v0, v1, v2, u[33], m[17]); + MONT_MAC(w0, w1, w2, u[34], m[16]); + MONT_MAC(t0, t1, t2, u[35], m[15]); + MONT_MAC(v0, v1, v2, u[36], m[14]); + MONT_MAC(w0, w1, w2, u[37], m[13]); + MONT_MAC(t0, t1, t2, u[38], m[12]); + MONT_MAC(v0, v1, v2, u[39], m[11]); + MONT_MAC(w0, w1, w2, u[40], m[10]); + MONT_MAC(t0, t1, t2, u[41], m[9]); + MONT_MAC(v0, v1, v2, u[42], m[8]); + MONT_MAC(w0, w1, w2, u[43], m[7]); + MONT_MAC(t0, t1, t2, u[44], m[6]); + MONT_MAC(v0, v1, v2, u[45], m[5]); + MONT_MAC(w0, w1, w2, u[46], m[4]); + MONT_MAC(t0, t1, t2, u[47], m[3]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[2] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 51 */ + MONT_MUL(s0, s1, s2, a[4], a[47]); + MONT_MUL(c0, c1, c2, a[5], a[46]); + MONT_MAC(s0, s1, s2, a[6], a[45]); + MONT_MAC(c0, c1, c2, a[7], a[44]); + MONT_MAC(s0, s1, s2, a[8], a[43]); + MONT_MAC(c0, c1, c2, a[9], a[42]); + MONT_MAC(s0, s1, s2, a[10], a[41]); + MONT_MAC(c0, c1, c2, a[11], a[40]); + MONT_MAC(s0, s1, s2, a[12], a[39]); + MONT_MAC(c0, c1, c2, a[13], a[38]); + MONT_MAC(s0, s1, s2, a[14], a[37]); + MONT_MAC(c0, c1, c2, a[15], a[36]); + MONT_MAC(s0, s1, s2, a[16], a[35]); + MONT_MAC(c0, c1, c2, a[17], a[34]); + MONT_MAC(s0, s1, s2, a[18], a[33]); + MONT_MAC(c0, c1, c2, a[19], a[32]); + MONT_MAC(s0, s1, s2, a[20], a[31]); + MONT_MAC(c0, c1, c2, a[21], a[30]); + MONT_MAC(s0, s1, s2, a[22], a[29]); + MONT_MAC(c0, c1, c2, a[23], a[28]); + MONT_MAC(s0, s1, s2, a[24], a[27]); + MONT_MAC(c0, c1, c2, a[25], a[26]); + MONT_MAC(t0, t1, t2, u[4], m[47]); + MONT_MUL(v0, v1, v2, u[5], m[46]); + MONT_MUL(w0, w1, w2, u[6], m[45]); + MONT_MAC(t0, t1, t2, u[7], m[44]); + MONT_MAC(v0, v1, v2, u[8], m[43]); + MONT_MAC(w0, w1, w2, u[9], m[42]); + MONT_MAC(t0, t1, t2, u[10], m[41]); + MONT_MAC(v0, v1, v2, u[11], m[40]); + MONT_MAC(w0, w1, w2, u[12], m[39]); + MONT_MAC(t0, t1, t2, u[13], m[38]); + MONT_MAC(v0, v1, v2, u[14], m[37]); + MONT_MAC(w0, w1, w2, u[15], m[36]); + MONT_MAC(t0, t1, t2, u[16], m[35]); + MONT_MAC(v0, v1, v2, u[17], m[34]); + MONT_MAC(w0, w1, w2, u[18], m[33]); + MONT_MAC(t0, t1, t2, u[19], m[32]); + MONT_MAC(v0, v1, v2, u[20], m[31]); + MONT_MAC(w0, w1, w2, u[21], m[30]); + MONT_MAC(t0, t1, t2, u[22], m[29]); + MONT_MAC(v0, v1, v2, u[23], m[28]); + MONT_MAC(w0, w1, w2, u[24], m[27]); + MONT_MAC(t0, t1, t2, u[25], m[26]); + MONT_MAC(v0, v1, v2, u[26], m[25]); + MONT_MAC(w0, w1, w2, u[27], m[24]); + MONT_MAC(t0, t1, t2, u[28], m[23]); + MONT_MAC(v0, v1, v2, u[29], m[22]); + MONT_MAC(w0, w1, w2, u[30], m[21]); + MONT_MAC(t0, t1, t2, u[31], m[20]); + MONT_MAC(v0, v1, v2, u[32], m[19]); + MONT_MAC(w0, w1, w2, u[33], m[18]); + MONT_MAC(t0, t1, t2, u[34], m[17]); + MONT_MAC(v0, v1, v2, u[35], m[16]); + MONT_MAC(w0, w1, w2, u[36], m[15]); + MONT_MAC(t0, t1, t2, u[37], m[14]); + MONT_MAC(v0, v1, v2, u[38], m[13]); + MONT_MAC(w0, w1, w2, u[39], m[12]); + MONT_MAC(t0, t1, t2, u[40], m[11]); + MONT_MAC(v0, v1, v2, u[41], m[10]); + MONT_MAC(w0, w1, w2, u[42], m[9]); + MONT_MAC(t0, t1, t2, u[43], m[8]); + MONT_MAC(v0, v1, v2, u[44], m[7]); + MONT_MAC(w0, w1, w2, u[45], m[6]); + MONT_MAC(t0, t1, t2, u[46], m[5]); + MONT_MAC(v0, v1, v2, u[47], m[4]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[3] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 52 */ + MONT_MUL(s0, s1, s2, a[5], a[47]); + MONT_MUL(c0, c1, c2, a[6], a[46]); + MONT_MAC(s0, s1, s2, a[7], a[45]); + MONT_MAC(c0, c1, c2, a[8], a[44]); + MONT_MAC(s0, s1, s2, a[9], a[43]); + MONT_MAC(c0, c1, c2, a[10], a[42]); + MONT_MAC(s0, s1, s2, a[11], a[41]); + MONT_MAC(c0, c1, c2, a[12], a[40]); + MONT_MAC(s0, s1, s2, a[13], a[39]); + MONT_MAC(c0, c1, c2, a[14], a[38]); + MONT_MAC(s0, s1, s2, a[15], a[37]); + MONT_MAC(c0, c1, c2, a[16], a[36]); + MONT_MAC(s0, s1, s2, a[17], a[35]); + MONT_MAC(c0, c1, c2, a[18], a[34]); + MONT_MAC(s0, s1, s2, a[19], a[33]); + MONT_MAC(c0, c1, c2, a[20], a[32]); + MONT_MAC(s0, s1, s2, a[21], a[31]); + MONT_MAC(c0, c1, c2, a[22], a[30]); + MONT_MAC(s0, s1, s2, a[23], a[29]); + MONT_MAC(c0, c1, c2, a[24], a[28]); + MONT_MAC(s0, s1, s2, a[25], a[27]); + MONT_MAC(t0, t1, t2, a[26], a[26]); + MONT_MUL(v0, v1, v2, u[5], m[47]); + MONT_MUL(w0, w1, w2, u[6], m[46]); + MONT_MAC(t0, t1, t2, u[7], m[45]); + MONT_MAC(v0, v1, v2, u[8], m[44]); + MONT_MAC(w0, w1, w2, u[9], m[43]); + MONT_MAC(t0, t1, t2, u[10], m[42]); + MONT_MAC(v0, v1, v2, u[11], m[41]); + MONT_MAC(w0, w1, w2, u[12], m[40]); + MONT_MAC(t0, t1, t2, u[13], m[39]); + MONT_MAC(v0, v1, v2, u[14], m[38]); + MONT_MAC(w0, w1, w2, u[15], m[37]); + MONT_MAC(t0, t1, t2, u[16], m[36]); + MONT_MAC(v0, v1, v2, u[17], m[35]); + MONT_MAC(w0, w1, w2, u[18], m[34]); + MONT_MAC(t0, t1, t2, u[19], m[33]); + MONT_MAC(v0, v1, v2, u[20], m[32]); + MONT_MAC(w0, w1, w2, u[21], m[31]); + MONT_MAC(t0, t1, t2, u[22], m[30]); + MONT_MAC(v0, v1, v2, u[23], m[29]); + MONT_MAC(w0, w1, w2, u[24], m[28]); + MONT_MAC(t0, t1, t2, u[25], m[27]); + MONT_MAC(v0, v1, v2, u[26], m[26]); + MONT_MAC(w0, w1, w2, u[27], m[25]); + MONT_MAC(t0, t1, t2, u[28], m[24]); + MONT_MAC(v0, v1, v2, u[29], m[23]); + MONT_MAC(w0, w1, w2, u[30], m[22]); + MONT_MAC(t0, t1, t2, u[31], m[21]); + MONT_MAC(v0, v1, v2, u[32], m[20]); + MONT_MAC(w0, w1, w2, u[33], m[19]); + MONT_MAC(t0, t1, t2, u[34], m[18]); + MONT_MAC(v0, v1, v2, u[35], m[17]); + MONT_MAC(w0, w1, w2, u[36], m[16]); + MONT_MAC(t0, t1, t2, u[37], m[15]); + MONT_MAC(v0, v1, v2, u[38], m[14]); + MONT_MAC(w0, w1, w2, u[39], m[13]); + MONT_MAC(t0, t1, t2, u[40], m[12]); + MONT_MAC(v0, v1, v2, u[41], m[11]); + MONT_MAC(w0, w1, w2, u[42], m[10]); + MONT_MAC(t0, t1, t2, u[43], m[9]); + MONT_MAC(v0, v1, v2, u[44], m[8]); + MONT_MAC(w0, w1, w2, u[45], m[7]); + MONT_MAC(t0, t1, t2, u[46], m[6]); + MONT_MAC(v0, v1, v2, u[47], m[5]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[4] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 53 */ + MONT_MUL(s0, s1, s2, a[6], a[47]); + MONT_MUL(c0, c1, c2, a[7], a[46]); + MONT_MAC(s0, s1, s2, a[8], a[45]); + MONT_MAC(c0, c1, c2, a[9], a[44]); + MONT_MAC(s0, s1, s2, a[10], a[43]); + MONT_MAC(c0, c1, c2, a[11], a[42]); + MONT_MAC(s0, s1, s2, a[12], a[41]); + MONT_MAC(c0, c1, c2, a[13], a[40]); + MONT_MAC(s0, s1, s2, a[14], a[39]); + MONT_MAC(c0, c1, c2, a[15], a[38]); + MONT_MAC(s0, s1, s2, a[16], a[37]); + MONT_MAC(c0, c1, c2, a[17], a[36]); + MONT_MAC(s0, s1, s2, a[18], a[35]); + MONT_MAC(c0, c1, c2, a[19], a[34]); + MONT_MAC(s0, s1, s2, a[20], a[33]); + MONT_MAC(c0, c1, c2, a[21], a[32]); + MONT_MAC(s0, s1, s2, a[22], a[31]); + MONT_MAC(c0, c1, c2, a[23], a[30]); + MONT_MAC(s0, s1, s2, a[24], a[29]); + MONT_MAC(c0, c1, c2, a[25], a[28]); + MONT_MAC(s0, s1, s2, a[26], a[27]); + MONT_MAC(t0, t1, t2, u[6], m[47]); + MONT_MUL(v0, v1, v2, u[7], m[46]); + MONT_MUL(w0, w1, w2, u[8], m[45]); + MONT_MAC(t0, t1, t2, u[9], m[44]); + MONT_MAC(v0, v1, v2, u[10], m[43]); + MONT_MAC(w0, w1, w2, u[11], m[42]); + MONT_MAC(t0, t1, t2, u[12], m[41]); + MONT_MAC(v0, v1, v2, u[13], m[40]); + MONT_MAC(w0, w1, w2, u[14], m[39]); + MONT_MAC(t0, t1, t2, u[15], m[38]); + MONT_MAC(v0, v1, v2, u[16], m[37]); + MONT_MAC(w0, w1, w2, u[17], m[36]); + MONT_MAC(t0, t1, t2, u[18], m[35]); + MONT_MAC(v0, v1, v2, u[19], m[34]); + MONT_MAC(w0, w1, w2, u[20], m[33]); + MONT_MAC(t0, t1, t2, u[21], m[32]); + MONT_MAC(v0, v1, v2, u[22], m[31]); + MONT_MAC(w0, w1, w2, u[23], m[30]); + MONT_MAC(t0, t1, t2, u[24], m[29]); + MONT_MAC(v0, v1, v2, u[25], m[28]); + MONT_MAC(w0, w1, w2, u[26], m[27]); + MONT_MAC(t0, t1, t2, u[27], m[26]); + MONT_MAC(v0, v1, v2, u[28], m[25]); + MONT_MAC(w0, w1, w2, u[29], m[24]); + MONT_MAC(t0, t1, t2, u[30], m[23]); + MONT_MAC(v0, v1, v2, u[31], m[22]); + MONT_MAC(w0, w1, w2, u[32], m[21]); + MONT_MAC(t0, t1, t2, u[33], m[20]); + MONT_MAC(v0, v1, v2, u[34], m[19]); + MONT_MAC(w0, w1, w2, u[35], m[18]); + MONT_MAC(t0, t1, t2, u[36], m[17]); + MONT_MAC(v0, v1, v2, u[37], m[16]); + MONT_MAC(w0, w1, w2, u[38], m[15]); + MONT_MAC(t0, t1, t2, u[39], m[14]); + MONT_MAC(v0, v1, v2, u[40], m[13]); + MONT_MAC(w0, w1, w2, u[41], m[12]); + MONT_MAC(t0, t1, t2, u[42], m[11]); + MONT_MAC(v0, v1, v2, u[43], m[10]); + MONT_MAC(w0, w1, w2, u[44], m[9]); + MONT_MAC(t0, t1, t2, u[45], m[8]); + MONT_MAC(v0, v1, v2, u[46], m[7]); + MONT_MAC(w0, w1, w2, u[47], m[6]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[5] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 54 */ + MONT_MUL(s0, s1, s2, a[7], a[47]); + MONT_MUL(c0, c1, c2, a[8], a[46]); + MONT_MAC(s0, s1, s2, a[9], a[45]); + MONT_MAC(c0, c1, c2, a[10], a[44]); + MONT_MAC(s0, s1, s2, a[11], a[43]); + MONT_MAC(c0, c1, c2, a[12], a[42]); + MONT_MAC(s0, s1, s2, a[13], a[41]); + MONT_MAC(c0, c1, c2, a[14], a[40]); + MONT_MAC(s0, s1, s2, a[15], a[39]); + MONT_MAC(c0, c1, c2, a[16], a[38]); + MONT_MAC(s0, s1, s2, a[17], a[37]); + MONT_MAC(c0, c1, c2, a[18], a[36]); + MONT_MAC(s0, s1, s2, a[19], a[35]); + MONT_MAC(c0, c1, c2, a[20], a[34]); + MONT_MAC(s0, s1, s2, a[21], a[33]); + MONT_MAC(c0, c1, c2, a[22], a[32]); + MONT_MAC(s0, s1, s2, a[23], a[31]); + MONT_MAC(c0, c1, c2, a[24], a[30]); + MONT_MAC(s0, s1, s2, a[25], a[29]); + MONT_MAC(c0, c1, c2, a[26], a[28]); + MONT_MAC(t0, t1, t2, a[27], a[27]); + MONT_MUL(v0, v1, v2, u[7], m[47]); + MONT_MUL(w0, w1, w2, u[8], m[46]); + MONT_MAC(t0, t1, t2, u[9], m[45]); + MONT_MAC(v0, v1, v2, u[10], m[44]); + MONT_MAC(w0, w1, w2, u[11], m[43]); + MONT_MAC(t0, t1, t2, u[12], m[42]); + MONT_MAC(v0, v1, v2, u[13], m[41]); + MONT_MAC(w0, w1, w2, u[14], m[40]); + MONT_MAC(t0, t1, t2, u[15], m[39]); + MONT_MAC(v0, v1, v2, u[16], m[38]); + MONT_MAC(w0, w1, w2, u[17], m[37]); + MONT_MAC(t0, t1, t2, u[18], m[36]); + MONT_MAC(v0, v1, v2, u[19], m[35]); + MONT_MAC(w0, w1, w2, u[20], m[34]); + MONT_MAC(t0, t1, t2, u[21], m[33]); + MONT_MAC(v0, v1, v2, u[22], m[32]); + MONT_MAC(w0, w1, w2, u[23], m[31]); + MONT_MAC(t0, t1, t2, u[24], m[30]); + MONT_MAC(v0, v1, v2, u[25], m[29]); + MONT_MAC(w0, w1, w2, u[26], m[28]); + MONT_MAC(t0, t1, t2, u[27], m[27]); + MONT_MAC(v0, v1, v2, u[28], m[26]); + MONT_MAC(w0, w1, w2, u[29], m[25]); + MONT_MAC(t0, t1, t2, u[30], m[24]); + MONT_MAC(v0, v1, v2, u[31], m[23]); + MONT_MAC(w0, w1, w2, u[32], m[22]); + MONT_MAC(t0, t1, t2, u[33], m[21]); + MONT_MAC(v0, v1, v2, u[34], m[20]); + MONT_MAC(w0, w1, w2, u[35], m[19]); + MONT_MAC(t0, t1, t2, u[36], m[18]); + MONT_MAC(v0, v1, v2, u[37], m[17]); + MONT_MAC(w0, w1, w2, u[38], m[16]); + MONT_MAC(t0, t1, t2, u[39], m[15]); + MONT_MAC(v0, v1, v2, u[40], m[14]); + MONT_MAC(w0, w1, w2, u[41], m[13]); + MONT_MAC(t0, t1, t2, u[42], m[12]); + MONT_MAC(v0, v1, v2, u[43], m[11]); + MONT_MAC(w0, w1, w2, u[44], m[10]); + MONT_MAC(t0, t1, t2, u[45], m[9]); + MONT_MAC(v0, v1, v2, u[46], m[8]); + MONT_MAC(w0, w1, w2, u[47], m[7]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[6] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 55 */ + MONT_MUL(s0, s1, s2, a[8], a[47]); + MONT_MUL(c0, c1, c2, a[9], a[46]); + MONT_MAC(s0, s1, s2, a[10], a[45]); + MONT_MAC(c0, c1, c2, a[11], a[44]); + MONT_MAC(s0, s1, s2, a[12], a[43]); + MONT_MAC(c0, c1, c2, a[13], a[42]); + MONT_MAC(s0, s1, s2, a[14], a[41]); + MONT_MAC(c0, c1, c2, a[15], a[40]); + MONT_MAC(s0, s1, s2, a[16], a[39]); + MONT_MAC(c0, c1, c2, a[17], a[38]); + MONT_MAC(s0, s1, s2, a[18], a[37]); + MONT_MAC(c0, c1, c2, a[19], a[36]); + MONT_MAC(s0, s1, s2, a[20], a[35]); + MONT_MAC(c0, c1, c2, a[21], a[34]); + MONT_MAC(s0, s1, s2, a[22], a[33]); + MONT_MAC(c0, c1, c2, a[23], a[32]); + MONT_MAC(s0, s1, s2, a[24], a[31]); + MONT_MAC(c0, c1, c2, a[25], a[30]); + MONT_MAC(s0, s1, s2, a[26], a[29]); + MONT_MAC(c0, c1, c2, a[27], a[28]); + MONT_MAC(t0, t1, t2, u[8], m[47]); + MONT_MUL(v0, v1, v2, u[9], m[46]); + MONT_MUL(w0, w1, w2, u[10], m[45]); + MONT_MAC(t0, t1, t2, u[11], m[44]); + MONT_MAC(v0, v1, v2, u[12], m[43]); + MONT_MAC(w0, w1, w2, u[13], m[42]); + MONT_MAC(t0, t1, t2, u[14], m[41]); + MONT_MAC(v0, v1, v2, u[15], m[40]); + MONT_MAC(w0, w1, w2, u[16], m[39]); + MONT_MAC(t0, t1, t2, u[17], m[38]); + MONT_MAC(v0, v1, v2, u[18], m[37]); + MONT_MAC(w0, w1, w2, u[19], m[36]); + MONT_MAC(t0, t1, t2, u[20], m[35]); + MONT_MAC(v0, v1, v2, u[21], m[34]); + MONT_MAC(w0, w1, w2, u[22], m[33]); + MONT_MAC(t0, t1, t2, u[23], m[32]); + MONT_MAC(v0, v1, v2, u[24], m[31]); + MONT_MAC(w0, w1, w2, u[25], m[30]); + MONT_MAC(t0, t1, t2, u[26], m[29]); + MONT_MAC(v0, v1, v2, u[27], m[28]); + MONT_MAC(w0, w1, w2, u[28], m[27]); + MONT_MAC(t0, t1, t2, u[29], m[26]); + MONT_MAC(v0, v1, v2, u[30], m[25]); + MONT_MAC(w0, w1, w2, u[31], m[24]); + MONT_MAC(t0, t1, t2, u[32], m[23]); + MONT_MAC(v0, v1, v2, u[33], m[22]); + MONT_MAC(w0, w1, w2, u[34], m[21]); + MONT_MAC(t0, t1, t2, u[35], m[20]); + MONT_MAC(v0, v1, v2, u[36], m[19]); + MONT_MAC(w0, w1, w2, u[37], m[18]); + MONT_MAC(t0, t1, t2, u[38], m[17]); + MONT_MAC(v0, v1, v2, u[39], m[16]); + MONT_MAC(w0, w1, w2, u[40], m[15]); + MONT_MAC(t0, t1, t2, u[41], m[14]); + MONT_MAC(v0, v1, v2, u[42], m[13]); + MONT_MAC(w0, w1, w2, u[43], m[12]); + MONT_MAC(t0, t1, t2, u[44], m[11]); + MONT_MAC(v0, v1, v2, u[45], m[10]); + MONT_MAC(w0, w1, w2, u[46], m[9]); + MONT_MAC(t0, t1, t2, u[47], m[8]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[7] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 56 */ + MONT_MUL(s0, s1, s2, a[9], a[47]); + MONT_MUL(c0, c1, c2, a[10], a[46]); + MONT_MAC(s0, s1, s2, a[11], a[45]); + MONT_MAC(c0, c1, c2, a[12], a[44]); + MONT_MAC(s0, s1, s2, a[13], a[43]); + MONT_MAC(c0, c1, c2, a[14], a[42]); + MONT_MAC(s0, s1, s2, a[15], a[41]); + MONT_MAC(c0, c1, c2, a[16], a[40]); + MONT_MAC(s0, s1, s2, a[17], a[39]); + MONT_MAC(c0, c1, c2, a[18], a[38]); + MONT_MAC(s0, s1, s2, a[19], a[37]); + MONT_MAC(c0, c1, c2, a[20], a[36]); + MONT_MAC(s0, s1, s2, a[21], a[35]); + MONT_MAC(c0, c1, c2, a[22], a[34]); + MONT_MAC(s0, s1, s2, a[23], a[33]); + MONT_MAC(c0, c1, c2, a[24], a[32]); + MONT_MAC(s0, s1, s2, a[25], a[31]); + MONT_MAC(c0, c1, c2, a[26], a[30]); + MONT_MAC(s0, s1, s2, a[27], a[29]); + MONT_MAC(t0, t1, t2, a[28], a[28]); + MONT_MUL(v0, v1, v2, u[9], m[47]); + MONT_MUL(w0, w1, w2, u[10], m[46]); + MONT_MAC(t0, t1, t2, u[11], m[45]); + MONT_MAC(v0, v1, v2, u[12], m[44]); + MONT_MAC(w0, w1, w2, u[13], m[43]); + MONT_MAC(t0, t1, t2, u[14], m[42]); + MONT_MAC(v0, v1, v2, u[15], m[41]); + MONT_MAC(w0, w1, w2, u[16], m[40]); + MONT_MAC(t0, t1, t2, u[17], m[39]); + MONT_MAC(v0, v1, v2, u[18], m[38]); + MONT_MAC(w0, w1, w2, u[19], m[37]); + MONT_MAC(t0, t1, t2, u[20], m[36]); + MONT_MAC(v0, v1, v2, u[21], m[35]); + MONT_MAC(w0, w1, w2, u[22], m[34]); + MONT_MAC(t0, t1, t2, u[23], m[33]); + MONT_MAC(v0, v1, v2, u[24], m[32]); + MONT_MAC(w0, w1, w2, u[25], m[31]); + MONT_MAC(t0, t1, t2, u[26], m[30]); + MONT_MAC(v0, v1, v2, u[27], m[29]); + MONT_MAC(w0, w1, w2, u[28], m[28]); + MONT_MAC(t0, t1, t2, u[29], m[27]); + MONT_MAC(v0, v1, v2, u[30], m[26]); + MONT_MAC(w0, w1, w2, u[31], m[25]); + MONT_MAC(t0, t1, t2, u[32], m[24]); + MONT_MAC(v0, v1, v2, u[33], m[23]); + MONT_MAC(w0, w1, w2, u[34], m[22]); + MONT_MAC(t0, t1, t2, u[35], m[21]); + MONT_MAC(v0, v1, v2, u[36], m[20]); + MONT_MAC(w0, w1, w2, u[37], m[19]); + MONT_MAC(t0, t1, t2, u[38], m[18]); + MONT_MAC(v0, v1, v2, u[39], m[17]); + MONT_MAC(w0, w1, w2, u[40], m[16]); + MONT_MAC(t0, t1, t2, u[41], m[15]); + MONT_MAC(v0, v1, v2, u[42], m[14]); + MONT_MAC(w0, w1, w2, u[43], m[13]); + MONT_MAC(t0, t1, t2, u[44], m[12]); + MONT_MAC(v0, v1, v2, u[45], m[11]); + MONT_MAC(w0, w1, w2, u[46], m[10]); + MONT_MAC(t0, t1, t2, u[47], m[9]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[8] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 57 */ + MONT_MUL(s0, s1, s2, a[10], a[47]); + MONT_MUL(c0, c1, c2, a[11], a[46]); + MONT_MAC(s0, s1, s2, a[12], a[45]); + MONT_MAC(c0, c1, c2, a[13], a[44]); + MONT_MAC(s0, s1, s2, a[14], a[43]); + MONT_MAC(c0, c1, c2, a[15], a[42]); + MONT_MAC(s0, s1, s2, a[16], a[41]); + MONT_MAC(c0, c1, c2, a[17], a[40]); + MONT_MAC(s0, s1, s2, a[18], a[39]); + MONT_MAC(c0, c1, c2, a[19], a[38]); + MONT_MAC(s0, s1, s2, a[20], a[37]); + MONT_MAC(c0, c1, c2, a[21], a[36]); + MONT_MAC(s0, s1, s2, a[22], a[35]); + MONT_MAC(c0, c1, c2, a[23], a[34]); + MONT_MAC(s0, s1, s2, a[24], a[33]); + MONT_MAC(c0, c1, c2, a[25], a[32]); + MONT_MAC(s0, s1, s2, a[26], a[31]); + MONT_MAC(c0, c1, c2, a[27], a[30]); + MONT_MAC(s0, s1, s2, a[28], a[29]); + MONT_MAC(t0, t1, t2, u[10], m[47]); + MONT_MUL(v0, v1, v2, u[11], m[46]); + MONT_MUL(w0, w1, w2, u[12], m[45]); + MONT_MAC(t0, t1, t2, u[13], m[44]); + MONT_MAC(v0, v1, v2, u[14], m[43]); + MONT_MAC(w0, w1, w2, u[15], m[42]); + MONT_MAC(t0, t1, t2, u[16], m[41]); + MONT_MAC(v0, v1, v2, u[17], m[40]); + MONT_MAC(w0, w1, w2, u[18], m[39]); + MONT_MAC(t0, t1, t2, u[19], m[38]); + MONT_MAC(v0, v1, v2, u[20], m[37]); + MONT_MAC(w0, w1, w2, u[21], m[36]); + MONT_MAC(t0, t1, t2, u[22], m[35]); + MONT_MAC(v0, v1, v2, u[23], m[34]); + MONT_MAC(w0, w1, w2, u[24], m[33]); + MONT_MAC(t0, t1, t2, u[25], m[32]); + MONT_MAC(v0, v1, v2, u[26], m[31]); + MONT_MAC(w0, w1, w2, u[27], m[30]); + MONT_MAC(t0, t1, t2, u[28], m[29]); + MONT_MAC(v0, v1, v2, u[29], m[28]); + MONT_MAC(w0, w1, w2, u[30], m[27]); + MONT_MAC(t0, t1, t2, u[31], m[26]); + MONT_MAC(v0, v1, v2, u[32], m[25]); + MONT_MAC(w0, w1, w2, u[33], m[24]); + MONT_MAC(t0, t1, t2, u[34], m[23]); + MONT_MAC(v0, v1, v2, u[35], m[22]); + MONT_MAC(w0, w1, w2, u[36], m[21]); + MONT_MAC(t0, t1, t2, u[37], m[20]); + MONT_MAC(v0, v1, v2, u[38], m[19]); + MONT_MAC(w0, w1, w2, u[39], m[18]); + MONT_MAC(t0, t1, t2, u[40], m[17]); + MONT_MAC(v0, v1, v2, u[41], m[16]); + MONT_MAC(w0, w1, w2, u[42], m[15]); + MONT_MAC(t0, t1, t2, u[43], m[14]); + MONT_MAC(v0, v1, v2, u[44], m[13]); + MONT_MAC(w0, w1, w2, u[45], m[12]); + MONT_MAC(t0, t1, t2, u[46], m[11]); + MONT_MAC(v0, v1, v2, u[47], m[10]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[9] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 58 */ + MONT_MUL(s0, s1, s2, a[11], a[47]); + MONT_MUL(c0, c1, c2, a[12], a[46]); + MONT_MAC(s0, s1, s2, a[13], a[45]); + MONT_MAC(c0, c1, c2, a[14], a[44]); + MONT_MAC(s0, s1, s2, a[15], a[43]); + MONT_MAC(c0, c1, c2, a[16], a[42]); + MONT_MAC(s0, s1, s2, a[17], a[41]); + MONT_MAC(c0, c1, c2, a[18], a[40]); + MONT_MAC(s0, s1, s2, a[19], a[39]); + MONT_MAC(c0, c1, c2, a[20], a[38]); + MONT_MAC(s0, s1, s2, a[21], a[37]); + MONT_MAC(c0, c1, c2, a[22], a[36]); + MONT_MAC(s0, s1, s2, a[23], a[35]); + MONT_MAC(c0, c1, c2, a[24], a[34]); + MONT_MAC(s0, s1, s2, a[25], a[33]); + MONT_MAC(c0, c1, c2, a[26], a[32]); + MONT_MAC(s0, s1, s2, a[27], a[31]); + MONT_MAC(c0, c1, c2, a[28], a[30]); + MONT_MAC(t0, t1, t2, a[29], a[29]); + MONT_MUL(v0, v1, v2, u[11], m[47]); + MONT_MUL(w0, w1, w2, u[12], m[46]); + MONT_MAC(t0, t1, t2, u[13], m[45]); + MONT_MAC(v0, v1, v2, u[14], m[44]); + MONT_MAC(w0, w1, w2, u[15], m[43]); + MONT_MAC(t0, t1, t2, u[16], m[42]); + MONT_MAC(v0, v1, v2, u[17], m[41]); + MONT_MAC(w0, w1, w2, u[18], m[40]); + MONT_MAC(t0, t1, t2, u[19], m[39]); + MONT_MAC(v0, v1, v2, u[20], m[38]); + MONT_MAC(w0, w1, w2, u[21], m[37]); + MONT_MAC(t0, t1, t2, u[22], m[36]); + MONT_MAC(v0, v1, v2, u[23], m[35]); + MONT_MAC(w0, w1, w2, u[24], m[34]); + MONT_MAC(t0, t1, t2, u[25], m[33]); + MONT_MAC(v0, v1, v2, u[26], m[32]); + MONT_MAC(w0, w1, w2, u[27], m[31]); + MONT_MAC(t0, t1, t2, u[28], m[30]); + MONT_MAC(v0, v1, v2, u[29], m[29]); + MONT_MAC(w0, w1, w2, u[30], m[28]); + MONT_MAC(t0, t1, t2, u[31], m[27]); + MONT_MAC(v0, v1, v2, u[32], m[26]); + MONT_MAC(w0, w1, w2, u[33], m[25]); + MONT_MAC(t0, t1, t2, u[34], m[24]); + MONT_MAC(v0, v1, v2, u[35], m[23]); + MONT_MAC(w0, w1, w2, u[36], m[22]); + MONT_MAC(t0, t1, t2, u[37], m[21]); + MONT_MAC(v0, v1, v2, u[38], m[20]); + MONT_MAC(w0, w1, w2, u[39], m[19]); + MONT_MAC(t0, t1, t2, u[40], m[18]); + MONT_MAC(v0, v1, v2, u[41], m[17]); + MONT_MAC(w0, w1, w2, u[42], m[16]); + MONT_MAC(t0, t1, t2, u[43], m[15]); + MONT_MAC(v0, v1, v2, u[44], m[14]); + MONT_MAC(w0, w1, w2, u[45], m[13]); + MONT_MAC(t0, t1, t2, u[46], m[12]); + MONT_MAC(v0, v1, v2, u[47], m[11]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[10] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 59 */ + MONT_MUL(s0, s1, s2, a[12], a[47]); + MONT_MUL(c0, c1, c2, a[13], a[46]); + MONT_MAC(s0, s1, s2, a[14], a[45]); + MONT_MAC(c0, c1, c2, a[15], a[44]); + MONT_MAC(s0, s1, s2, a[16], a[43]); + MONT_MAC(c0, c1, c2, a[17], a[42]); + MONT_MAC(s0, s1, s2, a[18], a[41]); + MONT_MAC(c0, c1, c2, a[19], a[40]); + MONT_MAC(s0, s1, s2, a[20], a[39]); + MONT_MAC(c0, c1, c2, a[21], a[38]); + MONT_MAC(s0, s1, s2, a[22], a[37]); + MONT_MAC(c0, c1, c2, a[23], a[36]); + MONT_MAC(s0, s1, s2, a[24], a[35]); + MONT_MAC(c0, c1, c2, a[25], a[34]); + MONT_MAC(s0, s1, s2, a[26], a[33]); + MONT_MAC(c0, c1, c2, a[27], a[32]); + MONT_MAC(s0, s1, s2, a[28], a[31]); + MONT_MAC(c0, c1, c2, a[29], a[30]); + MONT_MAC(t0, t1, t2, u[12], m[47]); + MONT_MUL(v0, v1, v2, u[13], m[46]); + MONT_MUL(w0, w1, w2, u[14], m[45]); + MONT_MAC(t0, t1, t2, u[15], m[44]); + MONT_MAC(v0, v1, v2, u[16], m[43]); + MONT_MAC(w0, w1, w2, u[17], m[42]); + MONT_MAC(t0, t1, t2, u[18], m[41]); + MONT_MAC(v0, v1, v2, u[19], m[40]); + MONT_MAC(w0, w1, w2, u[20], m[39]); + MONT_MAC(t0, t1, t2, u[21], m[38]); + MONT_MAC(v0, v1, v2, u[22], m[37]); + MONT_MAC(w0, w1, w2, u[23], m[36]); + MONT_MAC(t0, t1, t2, u[24], m[35]); + MONT_MAC(v0, v1, v2, u[25], m[34]); + MONT_MAC(w0, w1, w2, u[26], m[33]); + MONT_MAC(t0, t1, t2, u[27], m[32]); + MONT_MAC(v0, v1, v2, u[28], m[31]); + MONT_MAC(w0, w1, w2, u[29], m[30]); + MONT_MAC(t0, t1, t2, u[30], m[29]); + MONT_MAC(v0, v1, v2, u[31], m[28]); + MONT_MAC(w0, w1, w2, u[32], m[27]); + MONT_MAC(t0, t1, t2, u[33], m[26]); + MONT_MAC(v0, v1, v2, u[34], m[25]); + MONT_MAC(w0, w1, w2, u[35], m[24]); + MONT_MAC(t0, t1, t2, u[36], m[23]); + MONT_MAC(v0, v1, v2, u[37], m[22]); + MONT_MAC(w0, w1, w2, u[38], m[21]); + MONT_MAC(t0, t1, t2, u[39], m[20]); + MONT_MAC(v0, v1, v2, u[40], m[19]); + MONT_MAC(w0, w1, w2, u[41], m[18]); + MONT_MAC(t0, t1, t2, u[42], m[17]); + MONT_MAC(v0, v1, v2, u[43], m[16]); + MONT_MAC(w0, w1, w2, u[44], m[15]); + MONT_MAC(t0, t1, t2, u[45], m[14]); + MONT_MAC(v0, v1, v2, u[46], m[13]); + MONT_MAC(w0, w1, w2, u[47], m[12]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[11] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 60 */ + MONT_MUL(s0, s1, s2, a[13], a[47]); + MONT_MUL(c0, c1, c2, a[14], a[46]); + MONT_MAC(s0, s1, s2, a[15], a[45]); + MONT_MAC(c0, c1, c2, a[16], a[44]); + MONT_MAC(s0, s1, s2, a[17], a[43]); + MONT_MAC(c0, c1, c2, a[18], a[42]); + MONT_MAC(s0, s1, s2, a[19], a[41]); + MONT_MAC(c0, c1, c2, a[20], a[40]); + MONT_MAC(s0, s1, s2, a[21], a[39]); + MONT_MAC(c0, c1, c2, a[22], a[38]); + MONT_MAC(s0, s1, s2, a[23], a[37]); + MONT_MAC(c0, c1, c2, a[24], a[36]); + MONT_MAC(s0, s1, s2, a[25], a[35]); + MONT_MAC(c0, c1, c2, a[26], a[34]); + MONT_MAC(s0, s1, s2, a[27], a[33]); + MONT_MAC(c0, c1, c2, a[28], a[32]); + MONT_MAC(s0, s1, s2, a[29], a[31]); + MONT_MAC(t0, t1, t2, a[30], a[30]); + MONT_MUL(v0, v1, v2, u[13], m[47]); + MONT_MUL(w0, w1, w2, u[14], m[46]); + MONT_MAC(t0, t1, t2, u[15], m[45]); + MONT_MAC(v0, v1, v2, u[16], m[44]); + MONT_MAC(w0, w1, w2, u[17], m[43]); + MONT_MAC(t0, t1, t2, u[18], m[42]); + MONT_MAC(v0, v1, v2, u[19], m[41]); + MONT_MAC(w0, w1, w2, u[20], m[40]); + MONT_MAC(t0, t1, t2, u[21], m[39]); + MONT_MAC(v0, v1, v2, u[22], m[38]); + MONT_MAC(w0, w1, w2, u[23], m[37]); + MONT_MAC(t0, t1, t2, u[24], m[36]); + MONT_MAC(v0, v1, v2, u[25], m[35]); + MONT_MAC(w0, w1, w2, u[26], m[34]); + MONT_MAC(t0, t1, t2, u[27], m[33]); + MONT_MAC(v0, v1, v2, u[28], m[32]); + MONT_MAC(w0, w1, w2, u[29], m[31]); + MONT_MAC(t0, t1, t2, u[30], m[30]); + MONT_MAC(v0, v1, v2, u[31], m[29]); + MONT_MAC(w0, w1, w2, u[32], m[28]); + MONT_MAC(t0, t1, t2, u[33], m[27]); + MONT_MAC(v0, v1, v2, u[34], m[26]); + MONT_MAC(w0, w1, w2, u[35], m[25]); + MONT_MAC(t0, t1, t2, u[36], m[24]); + MONT_MAC(v0, v1, v2, u[37], m[23]); + MONT_MAC(w0, w1, w2, u[38], m[22]); + MONT_MAC(t0, t1, t2, u[39], m[21]); + MONT_MAC(v0, v1, v2, u[40], m[20]); + MONT_MAC(w0, w1, w2, u[41], m[19]); + MONT_MAC(t0, t1, t2, u[42], m[18]); + MONT_MAC(v0, v1, v2, u[43], m[17]); + MONT_MAC(w0, w1, w2, u[44], m[16]); + MONT_MAC(t0, t1, t2, u[45], m[15]); + MONT_MAC(v0, v1, v2, u[46], m[14]); + MONT_MAC(w0, w1, w2, u[47], m[13]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[12] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 61 */ + MONT_MUL(s0, s1, s2, a[14], a[47]); + MONT_MUL(c0, c1, c2, a[15], a[46]); + MONT_MAC(s0, s1, s2, a[16], a[45]); + MONT_MAC(c0, c1, c2, a[17], a[44]); + MONT_MAC(s0, s1, s2, a[18], a[43]); + MONT_MAC(c0, c1, c2, a[19], a[42]); + MONT_MAC(s0, s1, s2, a[20], a[41]); + MONT_MAC(c0, c1, c2, a[21], a[40]); + MONT_MAC(s0, s1, s2, a[22], a[39]); + MONT_MAC(c0, c1, c2, a[23], a[38]); + MONT_MAC(s0, s1, s2, a[24], a[37]); + MONT_MAC(c0, c1, c2, a[25], a[36]); + MONT_MAC(s0, s1, s2, a[26], a[35]); + MONT_MAC(c0, c1, c2, a[27], a[34]); + MONT_MAC(s0, s1, s2, a[28], a[33]); + MONT_MAC(c0, c1, c2, a[29], a[32]); + MONT_MAC(s0, s1, s2, a[30], a[31]); + MONT_MAC(t0, t1, t2, u[14], m[47]); + MONT_MUL(v0, v1, v2, u[15], m[46]); + MONT_MUL(w0, w1, w2, u[16], m[45]); + MONT_MAC(t0, t1, t2, u[17], m[44]); + MONT_MAC(v0, v1, v2, u[18], m[43]); + MONT_MAC(w0, w1, w2, u[19], m[42]); + MONT_MAC(t0, t1, t2, u[20], m[41]); + MONT_MAC(v0, v1, v2, u[21], m[40]); + MONT_MAC(w0, w1, w2, u[22], m[39]); + MONT_MAC(t0, t1, t2, u[23], m[38]); + MONT_MAC(v0, v1, v2, u[24], m[37]); + MONT_MAC(w0, w1, w2, u[25], m[36]); + MONT_MAC(t0, t1, t2, u[26], m[35]); + MONT_MAC(v0, v1, v2, u[27], m[34]); + MONT_MAC(w0, w1, w2, u[28], m[33]); + MONT_MAC(t0, t1, t2, u[29], m[32]); + MONT_MAC(v0, v1, v2, u[30], m[31]); + MONT_MAC(w0, w1, w2, u[31], m[30]); + MONT_MAC(t0, t1, t2, u[32], m[29]); + MONT_MAC(v0, v1, v2, u[33], m[28]); + MONT_MAC(w0, w1, w2, u[34], m[27]); + MONT_MAC(t0, t1, t2, u[35], m[26]); + MONT_MAC(v0, v1, v2, u[36], m[25]); + MONT_MAC(w0, w1, w2, u[37], m[24]); + MONT_MAC(t0, t1, t2, u[38], m[23]); + MONT_MAC(v0, v1, v2, u[39], m[22]); + MONT_MAC(w0, w1, w2, u[40], m[21]); + MONT_MAC(t0, t1, t2, u[41], m[20]); + MONT_MAC(v0, v1, v2, u[42], m[19]); + MONT_MAC(w0, w1, w2, u[43], m[18]); + MONT_MAC(t0, t1, t2, u[44], m[17]); + MONT_MAC(v0, v1, v2, u[45], m[16]); + MONT_MAC(w0, w1, w2, u[46], m[15]); + MONT_MAC(t0, t1, t2, u[47], m[14]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[13] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 62 */ + MONT_MUL(s0, s1, s2, a[15], a[47]); + MONT_MUL(c0, c1, c2, a[16], a[46]); + MONT_MAC(s0, s1, s2, a[17], a[45]); + MONT_MAC(c0, c1, c2, a[18], a[44]); + MONT_MAC(s0, s1, s2, a[19], a[43]); + MONT_MAC(c0, c1, c2, a[20], a[42]); + MONT_MAC(s0, s1, s2, a[21], a[41]); + MONT_MAC(c0, c1, c2, a[22], a[40]); + MONT_MAC(s0, s1, s2, a[23], a[39]); + MONT_MAC(c0, c1, c2, a[24], a[38]); + MONT_MAC(s0, s1, s2, a[25], a[37]); + MONT_MAC(c0, c1, c2, a[26], a[36]); + MONT_MAC(s0, s1, s2, a[27], a[35]); + MONT_MAC(c0, c1, c2, a[28], a[34]); + MONT_MAC(s0, s1, s2, a[29], a[33]); + MONT_MAC(c0, c1, c2, a[30], a[32]); + MONT_MAC(t0, t1, t2, a[31], a[31]); + MONT_MUL(v0, v1, v2, u[15], m[47]); + MONT_MUL(w0, w1, w2, u[16], m[46]); + MONT_MAC(t0, t1, t2, u[17], m[45]); + MONT_MAC(v0, v1, v2, u[18], m[44]); + MONT_MAC(w0, w1, w2, u[19], m[43]); + MONT_MAC(t0, t1, t2, u[20], m[42]); + MONT_MAC(v0, v1, v2, u[21], m[41]); + MONT_MAC(w0, w1, w2, u[22], m[40]); + MONT_MAC(t0, t1, t2, u[23], m[39]); + MONT_MAC(v0, v1, v2, u[24], m[38]); + MONT_MAC(w0, w1, w2, u[25], m[37]); + MONT_MAC(t0, t1, t2, u[26], m[36]); + MONT_MAC(v0, v1, v2, u[27], m[35]); + MONT_MAC(w0, w1, w2, u[28], m[34]); + MONT_MAC(t0, t1, t2, u[29], m[33]); + MONT_MAC(v0, v1, v2, u[30], m[32]); + MONT_MAC(w0, w1, w2, u[31], m[31]); + MONT_MAC(t0, t1, t2, u[32], m[30]); + MONT_MAC(v0, v1, v2, u[33], m[29]); + MONT_MAC(w0, w1, w2, u[34], m[28]); + MONT_MAC(t0, t1, t2, u[35], m[27]); + MONT_MAC(v0, v1, v2, u[36], m[26]); + MONT_MAC(w0, w1, w2, u[37], m[25]); + MONT_MAC(t0, t1, t2, u[38], m[24]); + MONT_MAC(v0, v1, v2, u[39], m[23]); + MONT_MAC(w0, w1, w2, u[40], m[22]); + MONT_MAC(t0, t1, t2, u[41], m[21]); + MONT_MAC(v0, v1, v2, u[42], m[20]); + MONT_MAC(w0, w1, w2, u[43], m[19]); + MONT_MAC(t0, t1, t2, u[44], m[18]); + MONT_MAC(v0, v1, v2, u[45], m[17]); + MONT_MAC(w0, w1, w2, u[46], m[16]); + MONT_MAC(t0, t1, t2, u[47], m[15]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[14] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 63 */ + MONT_MUL(s0, s1, s2, a[16], a[47]); + MONT_MUL(c0, c1, c2, a[17], a[46]); + MONT_MAC(s0, s1, s2, a[18], a[45]); + MONT_MAC(c0, c1, c2, a[19], a[44]); + MONT_MAC(s0, s1, s2, a[20], a[43]); + MONT_MAC(c0, c1, c2, a[21], a[42]); + MONT_MAC(s0, s1, s2, a[22], a[41]); + MONT_MAC(c0, c1, c2, a[23], a[40]); + MONT_MAC(s0, s1, s2, a[24], a[39]); + MONT_MAC(c0, c1, c2, a[25], a[38]); + MONT_MAC(s0, s1, s2, a[26], a[37]); + MONT_MAC(c0, c1, c2, a[27], a[36]); + MONT_MAC(s0, s1, s2, a[28], a[35]); + MONT_MAC(c0, c1, c2, a[29], a[34]); + MONT_MAC(s0, s1, s2, a[30], a[33]); + MONT_MAC(c0, c1, c2, a[31], a[32]); + MONT_MAC(t0, t1, t2, u[16], m[47]); + MONT_MUL(v0, v1, v2, u[17], m[46]); + MONT_MUL(w0, w1, w2, u[18], m[45]); + MONT_MAC(t0, t1, t2, u[19], m[44]); + MONT_MAC(v0, v1, v2, u[20], m[43]); + MONT_MAC(w0, w1, w2, u[21], m[42]); + MONT_MAC(t0, t1, t2, u[22], m[41]); + MONT_MAC(v0, v1, v2, u[23], m[40]); + MONT_MAC(w0, w1, w2, u[24], m[39]); + MONT_MAC(t0, t1, t2, u[25], m[38]); + MONT_MAC(v0, v1, v2, u[26], m[37]); + MONT_MAC(w0, w1, w2, u[27], m[36]); + MONT_MAC(t0, t1, t2, u[28], m[35]); + MONT_MAC(v0, v1, v2, u[29], m[34]); + MONT_MAC(w0, w1, w2, u[30], m[33]); + MONT_MAC(t0, t1, t2, u[31], m[32]); + MONT_MAC(v0, v1, v2, u[32], m[31]); + MONT_MAC(w0, w1, w2, u[33], m[30]); + MONT_MAC(t0, t1, t2, u[34], m[29]); + MONT_MAC(v0, v1, v2, u[35], m[28]); + MONT_MAC(w0, w1, w2, u[36], m[27]); + MONT_MAC(t0, t1, t2, u[37], m[26]); + MONT_MAC(v0, v1, v2, u[38], m[25]); + MONT_MAC(w0, w1, w2, u[39], m[24]); + MONT_MAC(t0, t1, t2, u[40], m[23]); + MONT_MAC(v0, v1, v2, u[41], m[22]); + MONT_MAC(w0, w1, w2, u[42], m[21]); + MONT_MAC(t0, t1, t2, u[43], m[20]); + MONT_MAC(v0, v1, v2, u[44], m[19]); + MONT_MAC(w0, w1, w2, u[45], m[18]); + MONT_MAC(t0, t1, t2, u[46], m[17]); + MONT_MAC(v0, v1, v2, u[47], m[16]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[15] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 64 */ + MONT_MUL(s0, s1, s2, a[17], a[47]); + MONT_MUL(c0, c1, c2, a[18], a[46]); + MONT_MAC(s0, s1, s2, a[19], a[45]); + MONT_MAC(c0, c1, c2, a[20], a[44]); + MONT_MAC(s0, s1, s2, a[21], a[43]); + MONT_MAC(c0, c1, c2, a[22], a[42]); + MONT_MAC(s0, s1, s2, a[23], a[41]); + MONT_MAC(c0, c1, c2, a[24], a[40]); + MONT_MAC(s0, s1, s2, a[25], a[39]); + MONT_MAC(c0, c1, c2, a[26], a[38]); + MONT_MAC(s0, s1, s2, a[27], a[37]); + MONT_MAC(c0, c1, c2, a[28], a[36]); + MONT_MAC(s0, s1, s2, a[29], a[35]); + MONT_MAC(c0, c1, c2, a[30], a[34]); + MONT_MAC(s0, s1, s2, a[31], a[33]); + MONT_MAC(t0, t1, t2, a[32], a[32]); + MONT_MUL(v0, v1, v2, u[17], m[47]); + MONT_MUL(w0, w1, w2, u[18], m[46]); + MONT_MAC(t0, t1, t2, u[19], m[45]); + MONT_MAC(v0, v1, v2, u[20], m[44]); + MONT_MAC(w0, w1, w2, u[21], m[43]); + MONT_MAC(t0, t1, t2, u[22], m[42]); + MONT_MAC(v0, v1, v2, u[23], m[41]); + MONT_MAC(w0, w1, w2, u[24], m[40]); + MONT_MAC(t0, t1, t2, u[25], m[39]); + MONT_MAC(v0, v1, v2, u[26], m[38]); + MONT_MAC(w0, w1, w2, u[27], m[37]); + MONT_MAC(t0, t1, t2, u[28], m[36]); + MONT_MAC(v0, v1, v2, u[29], m[35]); + MONT_MAC(w0, w1, w2, u[30], m[34]); + MONT_MAC(t0, t1, t2, u[31], m[33]); + MONT_MAC(v0, v1, v2, u[32], m[32]); + MONT_MAC(w0, w1, w2, u[33], m[31]); + MONT_MAC(t0, t1, t2, u[34], m[30]); + MONT_MAC(v0, v1, v2, u[35], m[29]); + MONT_MAC(w0, w1, w2, u[36], m[28]); + MONT_MAC(t0, t1, t2, u[37], m[27]); + MONT_MAC(v0, v1, v2, u[38], m[26]); + MONT_MAC(w0, w1, w2, u[39], m[25]); + MONT_MAC(t0, t1, t2, u[40], m[24]); + MONT_MAC(v0, v1, v2, u[41], m[23]); + MONT_MAC(w0, w1, w2, u[42], m[22]); + MONT_MAC(t0, t1, t2, u[43], m[21]); + MONT_MAC(v0, v1, v2, u[44], m[20]); + MONT_MAC(w0, w1, w2, u[45], m[19]); + MONT_MAC(t0, t1, t2, u[46], m[18]); + MONT_MAC(v0, v1, v2, u[47], m[17]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[16] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 65 */ + MONT_MUL(s0, s1, s2, a[18], a[47]); + MONT_MUL(c0, c1, c2, a[19], a[46]); + MONT_MAC(s0, s1, s2, a[20], a[45]); + MONT_MAC(c0, c1, c2, a[21], a[44]); + MONT_MAC(s0, s1, s2, a[22], a[43]); + MONT_MAC(c0, c1, c2, a[23], a[42]); + MONT_MAC(s0, s1, s2, a[24], a[41]); + MONT_MAC(c0, c1, c2, a[25], a[40]); + MONT_MAC(s0, s1, s2, a[26], a[39]); + MONT_MAC(c0, c1, c2, a[27], a[38]); + MONT_MAC(s0, s1, s2, a[28], a[37]); + MONT_MAC(c0, c1, c2, a[29], a[36]); + MONT_MAC(s0, s1, s2, a[30], a[35]); + MONT_MAC(c0, c1, c2, a[31], a[34]); + MONT_MAC(s0, s1, s2, a[32], a[33]); + MONT_MAC(t0, t1, t2, u[18], m[47]); + MONT_MUL(v0, v1, v2, u[19], m[46]); + MONT_MUL(w0, w1, w2, u[20], m[45]); + MONT_MAC(t0, t1, t2, u[21], m[44]); + MONT_MAC(v0, v1, v2, u[22], m[43]); + MONT_MAC(w0, w1, w2, u[23], m[42]); + MONT_MAC(t0, t1, t2, u[24], m[41]); + MONT_MAC(v0, v1, v2, u[25], m[40]); + MONT_MAC(w0, w1, w2, u[26], m[39]); + MONT_MAC(t0, t1, t2, u[27], m[38]); + MONT_MAC(v0, v1, v2, u[28], m[37]); + MONT_MAC(w0, w1, w2, u[29], m[36]); + MONT_MAC(t0, t1, t2, u[30], m[35]); + MONT_MAC(v0, v1, v2, u[31], m[34]); + MONT_MAC(w0, w1, w2, u[32], m[33]); + MONT_MAC(t0, t1, t2, u[33], m[32]); + MONT_MAC(v0, v1, v2, u[34], m[31]); + MONT_MAC(w0, w1, w2, u[35], m[30]); + MONT_MAC(t0, t1, t2, u[36], m[29]); + MONT_MAC(v0, v1, v2, u[37], m[28]); + MONT_MAC(w0, w1, w2, u[38], m[27]); + MONT_MAC(t0, t1, t2, u[39], m[26]); + MONT_MAC(v0, v1, v2, u[40], m[25]); + MONT_MAC(w0, w1, w2, u[41], m[24]); + MONT_MAC(t0, t1, t2, u[42], m[23]); + MONT_MAC(v0, v1, v2, u[43], m[22]); + MONT_MAC(w0, w1, w2, u[44], m[21]); + MONT_MAC(t0, t1, t2, u[45], m[20]); + MONT_MAC(v0, v1, v2, u[46], m[19]); + MONT_MAC(w0, w1, w2, u[47], m[18]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[17] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 66 */ + MONT_MUL(s0, s1, s2, a[19], a[47]); + MONT_MUL(c0, c1, c2, a[20], a[46]); + MONT_MAC(s0, s1, s2, a[21], a[45]); + MONT_MAC(c0, c1, c2, a[22], a[44]); + MONT_MAC(s0, s1, s2, a[23], a[43]); + MONT_MAC(c0, c1, c2, a[24], a[42]); + MONT_MAC(s0, s1, s2, a[25], a[41]); + MONT_MAC(c0, c1, c2, a[26], a[40]); + MONT_MAC(s0, s1, s2, a[27], a[39]); + MONT_MAC(c0, c1, c2, a[28], a[38]); + MONT_MAC(s0, s1, s2, a[29], a[37]); + MONT_MAC(c0, c1, c2, a[30], a[36]); + MONT_MAC(s0, s1, s2, a[31], a[35]); + MONT_MAC(c0, c1, c2, a[32], a[34]); + MONT_MAC(t0, t1, t2, a[33], a[33]); + MONT_MUL(v0, v1, v2, u[19], m[47]); + MONT_MUL(w0, w1, w2, u[20], m[46]); + MONT_MAC(t0, t1, t2, u[21], m[45]); + MONT_MAC(v0, v1, v2, u[22], m[44]); + MONT_MAC(w0, w1, w2, u[23], m[43]); + MONT_MAC(t0, t1, t2, u[24], m[42]); + MONT_MAC(v0, v1, v2, u[25], m[41]); + MONT_MAC(w0, w1, w2, u[26], m[40]); + MONT_MAC(t0, t1, t2, u[27], m[39]); + MONT_MAC(v0, v1, v2, u[28], m[38]); + MONT_MAC(w0, w1, w2, u[29], m[37]); + MONT_MAC(t0, t1, t2, u[30], m[36]); + MONT_MAC(v0, v1, v2, u[31], m[35]); + MONT_MAC(w0, w1, w2, u[32], m[34]); + MONT_MAC(t0, t1, t2, u[33], m[33]); + MONT_MAC(v0, v1, v2, u[34], m[32]); + MONT_MAC(w0, w1, w2, u[35], m[31]); + MONT_MAC(t0, t1, t2, u[36], m[30]); + MONT_MAC(v0, v1, v2, u[37], m[29]); + MONT_MAC(w0, w1, w2, u[38], m[28]); + MONT_MAC(t0, t1, t2, u[39], m[27]); + MONT_MAC(v0, v1, v2, u[40], m[26]); + MONT_MAC(w0, w1, w2, u[41], m[25]); + MONT_MAC(t0, t1, t2, u[42], m[24]); + MONT_MAC(v0, v1, v2, u[43], m[23]); + MONT_MAC(w0, w1, w2, u[44], m[22]); + MONT_MAC(t0, t1, t2, u[45], m[21]); + MONT_MAC(v0, v1, v2, u[46], m[20]); + MONT_MAC(w0, w1, w2, u[47], m[19]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[18] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 67 */ + MONT_MUL(s0, s1, s2, a[20], a[47]); + MONT_MUL(c0, c1, c2, a[21], a[46]); + MONT_MAC(s0, s1, s2, a[22], a[45]); + MONT_MAC(c0, c1, c2, a[23], a[44]); + MONT_MAC(s0, s1, s2, a[24], a[43]); + MONT_MAC(c0, c1, c2, a[25], a[42]); + MONT_MAC(s0, s1, s2, a[26], a[41]); + MONT_MAC(c0, c1, c2, a[27], a[40]); + MONT_MAC(s0, s1, s2, a[28], a[39]); + MONT_MAC(c0, c1, c2, a[29], a[38]); + MONT_MAC(s0, s1, s2, a[30], a[37]); + MONT_MAC(c0, c1, c2, a[31], a[36]); + MONT_MAC(s0, s1, s2, a[32], a[35]); + MONT_MAC(c0, c1, c2, a[33], a[34]); + MONT_MAC(t0, t1, t2, u[20], m[47]); + MONT_MUL(v0, v1, v2, u[21], m[46]); + MONT_MUL(w0, w1, w2, u[22], m[45]); + MONT_MAC(t0, t1, t2, u[23], m[44]); + MONT_MAC(v0, v1, v2, u[24], m[43]); + MONT_MAC(w0, w1, w2, u[25], m[42]); + MONT_MAC(t0, t1, t2, u[26], m[41]); + MONT_MAC(v0, v1, v2, u[27], m[40]); + MONT_MAC(w0, w1, w2, u[28], m[39]); + MONT_MAC(t0, t1, t2, u[29], m[38]); + MONT_MAC(v0, v1, v2, u[30], m[37]); + MONT_MAC(w0, w1, w2, u[31], m[36]); + MONT_MAC(t0, t1, t2, u[32], m[35]); + MONT_MAC(v0, v1, v2, u[33], m[34]); + MONT_MAC(w0, w1, w2, u[34], m[33]); + MONT_MAC(t0, t1, t2, u[35], m[32]); + MONT_MAC(v0, v1, v2, u[36], m[31]); + MONT_MAC(w0, w1, w2, u[37], m[30]); + MONT_MAC(t0, t1, t2, u[38], m[29]); + MONT_MAC(v0, v1, v2, u[39], m[28]); + MONT_MAC(w0, w1, w2, u[40], m[27]); + MONT_MAC(t0, t1, t2, u[41], m[26]); + MONT_MAC(v0, v1, v2, u[42], m[25]); + MONT_MAC(w0, w1, w2, u[43], m[24]); + MONT_MAC(t0, t1, t2, u[44], m[23]); + MONT_MAC(v0, v1, v2, u[45], m[22]); + MONT_MAC(w0, w1, w2, u[46], m[21]); + MONT_MAC(t0, t1, t2, u[47], m[20]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[19] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 68 */ + MONT_MUL(s0, s1, s2, a[21], a[47]); + MONT_MUL(c0, c1, c2, a[22], a[46]); + MONT_MAC(s0, s1, s2, a[23], a[45]); + MONT_MAC(c0, c1, c2, a[24], a[44]); + MONT_MAC(s0, s1, s2, a[25], a[43]); + MONT_MAC(c0, c1, c2, a[26], a[42]); + MONT_MAC(s0, s1, s2, a[27], a[41]); + MONT_MAC(c0, c1, c2, a[28], a[40]); + MONT_MAC(s0, s1, s2, a[29], a[39]); + MONT_MAC(c0, c1, c2, a[30], a[38]); + MONT_MAC(s0, s1, s2, a[31], a[37]); + MONT_MAC(c0, c1, c2, a[32], a[36]); + MONT_MAC(s0, s1, s2, a[33], a[35]); + MONT_MAC(t0, t1, t2, a[34], a[34]); + MONT_MUL(v0, v1, v2, u[21], m[47]); + MONT_MUL(w0, w1, w2, u[22], m[46]); + MONT_MAC(t0, t1, t2, u[23], m[45]); + MONT_MAC(v0, v1, v2, u[24], m[44]); + MONT_MAC(w0, w1, w2, u[25], m[43]); + MONT_MAC(t0, t1, t2, u[26], m[42]); + MONT_MAC(v0, v1, v2, u[27], m[41]); + MONT_MAC(w0, w1, w2, u[28], m[40]); + MONT_MAC(t0, t1, t2, u[29], m[39]); + MONT_MAC(v0, v1, v2, u[30], m[38]); + MONT_MAC(w0, w1, w2, u[31], m[37]); + MONT_MAC(t0, t1, t2, u[32], m[36]); + MONT_MAC(v0, v1, v2, u[33], m[35]); + MONT_MAC(w0, w1, w2, u[34], m[34]); + MONT_MAC(t0, t1, t2, u[35], m[33]); + MONT_MAC(v0, v1, v2, u[36], m[32]); + MONT_MAC(w0, w1, w2, u[37], m[31]); + MONT_MAC(t0, t1, t2, u[38], m[30]); + MONT_MAC(v0, v1, v2, u[39], m[29]); + MONT_MAC(w0, w1, w2, u[40], m[28]); + MONT_MAC(t0, t1, t2, u[41], m[27]); + MONT_MAC(v0, v1, v2, u[42], m[26]); + MONT_MAC(w0, w1, w2, u[43], m[25]); + MONT_MAC(t0, t1, t2, u[44], m[24]); + MONT_MAC(v0, v1, v2, u[45], m[23]); + MONT_MAC(w0, w1, w2, u[46], m[22]); + MONT_MAC(t0, t1, t2, u[47], m[21]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[20] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 69 */ + MONT_MUL(s0, s1, s2, a[22], a[47]); + MONT_MUL(c0, c1, c2, a[23], a[46]); + MONT_MAC(s0, s1, s2, a[24], a[45]); + MONT_MAC(c0, c1, c2, a[25], a[44]); + MONT_MAC(s0, s1, s2, a[26], a[43]); + MONT_MAC(c0, c1, c2, a[27], a[42]); + MONT_MAC(s0, s1, s2, a[28], a[41]); + MONT_MAC(c0, c1, c2, a[29], a[40]); + MONT_MAC(s0, s1, s2, a[30], a[39]); + MONT_MAC(c0, c1, c2, a[31], a[38]); + MONT_MAC(s0, s1, s2, a[32], a[37]); + MONT_MAC(c0, c1, c2, a[33], a[36]); + MONT_MAC(s0, s1, s2, a[34], a[35]); + MONT_MAC(t0, t1, t2, u[22], m[47]); + MONT_MUL(v0, v1, v2, u[23], m[46]); + MONT_MUL(w0, w1, w2, u[24], m[45]); + MONT_MAC(t0, t1, t2, u[25], m[44]); + MONT_MAC(v0, v1, v2, u[26], m[43]); + MONT_MAC(w0, w1, w2, u[27], m[42]); + MONT_MAC(t0, t1, t2, u[28], m[41]); + MONT_MAC(v0, v1, v2, u[29], m[40]); + MONT_MAC(w0, w1, w2, u[30], m[39]); + MONT_MAC(t0, t1, t2, u[31], m[38]); + MONT_MAC(v0, v1, v2, u[32], m[37]); + MONT_MAC(w0, w1, w2, u[33], m[36]); + MONT_MAC(t0, t1, t2, u[34], m[35]); + MONT_MAC(v0, v1, v2, u[35], m[34]); + MONT_MAC(w0, w1, w2, u[36], m[33]); + MONT_MAC(t0, t1, t2, u[37], m[32]); + MONT_MAC(v0, v1, v2, u[38], m[31]); + MONT_MAC(w0, w1, w2, u[39], m[30]); + MONT_MAC(t0, t1, t2, u[40], m[29]); + MONT_MAC(v0, v1, v2, u[41], m[28]); + MONT_MAC(w0, w1, w2, u[42], m[27]); + MONT_MAC(t0, t1, t2, u[43], m[26]); + MONT_MAC(v0, v1, v2, u[44], m[25]); + MONT_MAC(w0, w1, w2, u[45], m[24]); + MONT_MAC(t0, t1, t2, u[46], m[23]); + MONT_MAC(v0, v1, v2, u[47], m[22]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[21] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 70 */ + MONT_MUL(s0, s1, s2, a[23], a[47]); + MONT_MUL(c0, c1, c2, a[24], a[46]); + MONT_MAC(s0, s1, s2, a[25], a[45]); + MONT_MAC(c0, c1, c2, a[26], a[44]); + MONT_MAC(s0, s1, s2, a[27], a[43]); + MONT_MAC(c0, c1, c2, a[28], a[42]); + MONT_MAC(s0, s1, s2, a[29], a[41]); + MONT_MAC(c0, c1, c2, a[30], a[40]); + MONT_MAC(s0, s1, s2, a[31], a[39]); + MONT_MAC(c0, c1, c2, a[32], a[38]); + MONT_MAC(s0, s1, s2, a[33], a[37]); + MONT_MAC(c0, c1, c2, a[34], a[36]); + MONT_MAC(t0, t1, t2, a[35], a[35]); + MONT_MUL(v0, v1, v2, u[23], m[47]); + MONT_MUL(w0, w1, w2, u[24], m[46]); + MONT_MAC(t0, t1, t2, u[25], m[45]); + MONT_MAC(v0, v1, v2, u[26], m[44]); + MONT_MAC(w0, w1, w2, u[27], m[43]); + MONT_MAC(t0, t1, t2, u[28], m[42]); + MONT_MAC(v0, v1, v2, u[29], m[41]); + MONT_MAC(w0, w1, w2, u[30], m[40]); + MONT_MAC(t0, t1, t2, u[31], m[39]); + MONT_MAC(v0, v1, v2, u[32], m[38]); + MONT_MAC(w0, w1, w2, u[33], m[37]); + MONT_MAC(t0, t1, t2, u[34], m[36]); + MONT_MAC(v0, v1, v2, u[35], m[35]); + MONT_MAC(w0, w1, w2, u[36], m[34]); + MONT_MAC(t0, t1, t2, u[37], m[33]); + MONT_MAC(v0, v1, v2, u[38], m[32]); + MONT_MAC(w0, w1, w2, u[39], m[31]); + MONT_MAC(t0, t1, t2, u[40], m[30]); + MONT_MAC(v0, v1, v2, u[41], m[29]); + MONT_MAC(w0, w1, w2, u[42], m[28]); + MONT_MAC(t0, t1, t2, u[43], m[27]); + MONT_MAC(v0, v1, v2, u[44], m[26]); + MONT_MAC(w0, w1, w2, u[45], m[25]); + MONT_MAC(t0, t1, t2, u[46], m[24]); + MONT_MAC(v0, v1, v2, u[47], m[23]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[22] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 71 */ + MONT_MUL(s0, s1, s2, a[24], a[47]); + MONT_MUL(c0, c1, c2, a[25], a[46]); + MONT_MAC(s0, s1, s2, a[26], a[45]); + MONT_MAC(c0, c1, c2, a[27], a[44]); + MONT_MAC(s0, s1, s2, a[28], a[43]); + MONT_MAC(c0, c1, c2, a[29], a[42]); + MONT_MAC(s0, s1, s2, a[30], a[41]); + MONT_MAC(c0, c1, c2, a[31], a[40]); + MONT_MAC(s0, s1, s2, a[32], a[39]); + MONT_MAC(c0, c1, c2, a[33], a[38]); + MONT_MAC(s0, s1, s2, a[34], a[37]); + MONT_MAC(c0, c1, c2, a[35], a[36]); + MONT_MAC(t0, t1, t2, u[24], m[47]); + MONT_MUL(v0, v1, v2, u[25], m[46]); + MONT_MUL(w0, w1, w2, u[26], m[45]); + MONT_MAC(t0, t1, t2, u[27], m[44]); + MONT_MAC(v0, v1, v2, u[28], m[43]); + MONT_MAC(w0, w1, w2, u[29], m[42]); + MONT_MAC(t0, t1, t2, u[30], m[41]); + MONT_MAC(v0, v1, v2, u[31], m[40]); + MONT_MAC(w0, w1, w2, u[32], m[39]); + MONT_MAC(t0, t1, t2, u[33], m[38]); + MONT_MAC(v0, v1, v2, u[34], m[37]); + MONT_MAC(w0, w1, w2, u[35], m[36]); + MONT_MAC(t0, t1, t2, u[36], m[35]); + MONT_MAC(v0, v1, v2, u[37], m[34]); + MONT_MAC(w0, w1, w2, u[38], m[33]); + MONT_MAC(t0, t1, t2, u[39], m[32]); + MONT_MAC(v0, v1, v2, u[40], m[31]); + MONT_MAC(w0, w1, w2, u[41], m[30]); + MONT_MAC(t0, t1, t2, u[42], m[29]); + MONT_MAC(v0, v1, v2, u[43], m[28]); + MONT_MAC(w0, w1, w2, u[44], m[27]); + MONT_MAC(t0, t1, t2, u[45], m[26]); + MONT_MAC(v0, v1, v2, u[46], m[25]); + MONT_MAC(w0, w1, w2, u[47], m[24]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[23] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 72 */ + MONT_MUL(s0, s1, s2, a[25], a[47]); + MONT_MUL(c0, c1, c2, a[26], a[46]); + MONT_MAC(s0, s1, s2, a[27], a[45]); + MONT_MAC(c0, c1, c2, a[28], a[44]); + MONT_MAC(s0, s1, s2, a[29], a[43]); + MONT_MAC(c0, c1, c2, a[30], a[42]); + MONT_MAC(s0, s1, s2, a[31], a[41]); + MONT_MAC(c0, c1, c2, a[32], a[40]); + MONT_MAC(s0, s1, s2, a[33], a[39]); + MONT_MAC(c0, c1, c2, a[34], a[38]); + MONT_MAC(s0, s1, s2, a[35], a[37]); + MONT_MAC(t0, t1, t2, a[36], a[36]); + MONT_MUL(v0, v1, v2, u[25], m[47]); + MONT_MUL(w0, w1, w2, u[26], m[46]); + MONT_MAC(t0, t1, t2, u[27], m[45]); + MONT_MAC(v0, v1, v2, u[28], m[44]); + MONT_MAC(w0, w1, w2, u[29], m[43]); + MONT_MAC(t0, t1, t2, u[30], m[42]); + MONT_MAC(v0, v1, v2, u[31], m[41]); + MONT_MAC(w0, w1, w2, u[32], m[40]); + MONT_MAC(t0, t1, t2, u[33], m[39]); + MONT_MAC(v0, v1, v2, u[34], m[38]); + MONT_MAC(w0, w1, w2, u[35], m[37]); + MONT_MAC(t0, t1, t2, u[36], m[36]); + MONT_MAC(v0, v1, v2, u[37], m[35]); + MONT_MAC(w0, w1, w2, u[38], m[34]); + MONT_MAC(t0, t1, t2, u[39], m[33]); + MONT_MAC(v0, v1, v2, u[40], m[32]); + MONT_MAC(w0, w1, w2, u[41], m[31]); + MONT_MAC(t0, t1, t2, u[42], m[30]); + MONT_MAC(v0, v1, v2, u[43], m[29]); + MONT_MAC(w0, w1, w2, u[44], m[28]); + MONT_MAC(t0, t1, t2, u[45], m[27]); + MONT_MAC(v0, v1, v2, u[46], m[26]); + MONT_MAC(w0, w1, w2, u[47], m[25]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[24] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 73 */ + MONT_MUL(s0, s1, s2, a[26], a[47]); + MONT_MUL(c0, c1, c2, a[27], a[46]); + MONT_MAC(s0, s1, s2, a[28], a[45]); + MONT_MAC(c0, c1, c2, a[29], a[44]); + MONT_MAC(s0, s1, s2, a[30], a[43]); + MONT_MAC(c0, c1, c2, a[31], a[42]); + MONT_MAC(s0, s1, s2, a[32], a[41]); + MONT_MAC(c0, c1, c2, a[33], a[40]); + MONT_MAC(s0, s1, s2, a[34], a[39]); + MONT_MAC(c0, c1, c2, a[35], a[38]); + MONT_MAC(s0, s1, s2, a[36], a[37]); + MONT_MAC(t0, t1, t2, u[26], m[47]); + MONT_MUL(v0, v1, v2, u[27], m[46]); + MONT_MUL(w0, w1, w2, u[28], m[45]); + MONT_MAC(t0, t1, t2, u[29], m[44]); + MONT_MAC(v0, v1, v2, u[30], m[43]); + MONT_MAC(w0, w1, w2, u[31], m[42]); + MONT_MAC(t0, t1, t2, u[32], m[41]); + MONT_MAC(v0, v1, v2, u[33], m[40]); + MONT_MAC(w0, w1, w2, u[34], m[39]); + MONT_MAC(t0, t1, t2, u[35], m[38]); + MONT_MAC(v0, v1, v2, u[36], m[37]); + MONT_MAC(w0, w1, w2, u[37], m[36]); + MONT_MAC(t0, t1, t2, u[38], m[35]); + MONT_MAC(v0, v1, v2, u[39], m[34]); + MONT_MAC(w0, w1, w2, u[40], m[33]); + MONT_MAC(t0, t1, t2, u[41], m[32]); + MONT_MAC(v0, v1, v2, u[42], m[31]); + MONT_MAC(w0, w1, w2, u[43], m[30]); + MONT_MAC(t0, t1, t2, u[44], m[29]); + MONT_MAC(v0, v1, v2, u[45], m[28]); + MONT_MAC(w0, w1, w2, u[46], m[27]); + MONT_MAC(t0, t1, t2, u[47], m[26]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[25] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 74 */ + MONT_MUL(s0, s1, s2, a[27], a[47]); + MONT_MUL(c0, c1, c2, a[28], a[46]); + MONT_MAC(s0, s1, s2, a[29], a[45]); + MONT_MAC(c0, c1, c2, a[30], a[44]); + MONT_MAC(s0, s1, s2, a[31], a[43]); + MONT_MAC(c0, c1, c2, a[32], a[42]); + MONT_MAC(s0, s1, s2, a[33], a[41]); + MONT_MAC(c0, c1, c2, a[34], a[40]); + MONT_MAC(s0, s1, s2, a[35], a[39]); + MONT_MAC(c0, c1, c2, a[36], a[38]); + MONT_MAC(t0, t1, t2, a[37], a[37]); + MONT_MUL(v0, v1, v2, u[27], m[47]); + MONT_MUL(w0, w1, w2, u[28], m[46]); + MONT_MAC(t0, t1, t2, u[29], m[45]); + MONT_MAC(v0, v1, v2, u[30], m[44]); + MONT_MAC(w0, w1, w2, u[31], m[43]); + MONT_MAC(t0, t1, t2, u[32], m[42]); + MONT_MAC(v0, v1, v2, u[33], m[41]); + MONT_MAC(w0, w1, w2, u[34], m[40]); + MONT_MAC(t0, t1, t2, u[35], m[39]); + MONT_MAC(v0, v1, v2, u[36], m[38]); + MONT_MAC(w0, w1, w2, u[37], m[37]); + MONT_MAC(t0, t1, t2, u[38], m[36]); + MONT_MAC(v0, v1, v2, u[39], m[35]); + MONT_MAC(w0, w1, w2, u[40], m[34]); + MONT_MAC(t0, t1, t2, u[41], m[33]); + MONT_MAC(v0, v1, v2, u[42], m[32]); + MONT_MAC(w0, w1, w2, u[43], m[31]); + MONT_MAC(t0, t1, t2, u[44], m[30]); + MONT_MAC(v0, v1, v2, u[45], m[29]); + MONT_MAC(w0, w1, w2, u[46], m[28]); + MONT_MAC(t0, t1, t2, u[47], m[27]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[26] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 75 */ + MONT_MUL(s0, s1, s2, a[28], a[47]); + MONT_MUL(c0, c1, c2, a[29], a[46]); + MONT_MAC(s0, s1, s2, a[30], a[45]); + MONT_MAC(c0, c1, c2, a[31], a[44]); + MONT_MAC(s0, s1, s2, a[32], a[43]); + MONT_MAC(c0, c1, c2, a[33], a[42]); + MONT_MAC(s0, s1, s2, a[34], a[41]); + MONT_MAC(c0, c1, c2, a[35], a[40]); + MONT_MAC(s0, s1, s2, a[36], a[39]); + MONT_MAC(c0, c1, c2, a[37], a[38]); + MONT_MAC(t0, t1, t2, u[28], m[47]); + MONT_MUL(v0, v1, v2, u[29], m[46]); + MONT_MUL(w0, w1, w2, u[30], m[45]); + MONT_MAC(t0, t1, t2, u[31], m[44]); + MONT_MAC(v0, v1, v2, u[32], m[43]); + MONT_MAC(w0, w1, w2, u[33], m[42]); + MONT_MAC(t0, t1, t2, u[34], m[41]); + MONT_MAC(v0, v1, v2, u[35], m[40]); + MONT_MAC(w0, w1, w2, u[36], m[39]); + MONT_MAC(t0, t1, t2, u[37], m[38]); + MONT_MAC(v0, v1, v2, u[38], m[37]); + MONT_MAC(w0, w1, w2, u[39], m[36]); + MONT_MAC(t0, t1, t2, u[40], m[35]); + MONT_MAC(v0, v1, v2, u[41], m[34]); + MONT_MAC(w0, w1, w2, u[42], m[33]); + MONT_MAC(t0, t1, t2, u[43], m[32]); + MONT_MAC(v0, v1, v2, u[44], m[31]); + MONT_MAC(w0, w1, w2, u[45], m[30]); + MONT_MAC(t0, t1, t2, u[46], m[29]); + MONT_MAC(v0, v1, v2, u[47], m[28]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[27] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 76 */ + MONT_MUL(s0, s1, s2, a[29], a[47]); + MONT_MUL(c0, c1, c2, a[30], a[46]); + MONT_MAC(s0, s1, s2, a[31], a[45]); + MONT_MAC(c0, c1, c2, a[32], a[44]); + MONT_MAC(s0, s1, s2, a[33], a[43]); + MONT_MAC(c0, c1, c2, a[34], a[42]); + MONT_MAC(s0, s1, s2, a[35], a[41]); + MONT_MAC(c0, c1, c2, a[36], a[40]); + MONT_MAC(s0, s1, s2, a[37], a[39]); + MONT_MAC(t0, t1, t2, a[38], a[38]); + MONT_MUL(v0, v1, v2, u[29], m[47]); + MONT_MUL(w0, w1, w2, u[30], m[46]); + MONT_MAC(t0, t1, t2, u[31], m[45]); + MONT_MAC(v0, v1, v2, u[32], m[44]); + MONT_MAC(w0, w1, w2, u[33], m[43]); + MONT_MAC(t0, t1, t2, u[34], m[42]); + MONT_MAC(v0, v1, v2, u[35], m[41]); + MONT_MAC(w0, w1, w2, u[36], m[40]); + MONT_MAC(t0, t1, t2, u[37], m[39]); + MONT_MAC(v0, v1, v2, u[38], m[38]); + MONT_MAC(w0, w1, w2, u[39], m[37]); + MONT_MAC(t0, t1, t2, u[40], m[36]); + MONT_MAC(v0, v1, v2, u[41], m[35]); + MONT_MAC(w0, w1, w2, u[42], m[34]); + MONT_MAC(t0, t1, t2, u[43], m[33]); + MONT_MAC(v0, v1, v2, u[44], m[32]); + MONT_MAC(w0, w1, w2, u[45], m[31]); + MONT_MAC(t0, t1, t2, u[46], m[30]); + MONT_MAC(v0, v1, v2, u[47], m[29]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[28] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 77 */ + MONT_MUL(s0, s1, s2, a[30], a[47]); + MONT_MUL(c0, c1, c2, a[31], a[46]); + MONT_MAC(s0, s1, s2, a[32], a[45]); + MONT_MAC(c0, c1, c2, a[33], a[44]); + MONT_MAC(s0, s1, s2, a[34], a[43]); + MONT_MAC(c0, c1, c2, a[35], a[42]); + MONT_MAC(s0, s1, s2, a[36], a[41]); + MONT_MAC(c0, c1, c2, a[37], a[40]); + MONT_MAC(s0, s1, s2, a[38], a[39]); + MONT_MAC(t0, t1, t2, u[30], m[47]); + MONT_MUL(v0, v1, v2, u[31], m[46]); + MONT_MUL(w0, w1, w2, u[32], m[45]); + MONT_MAC(t0, t1, t2, u[33], m[44]); + MONT_MAC(v0, v1, v2, u[34], m[43]); + MONT_MAC(w0, w1, w2, u[35], m[42]); + MONT_MAC(t0, t1, t2, u[36], m[41]); + MONT_MAC(v0, v1, v2, u[37], m[40]); + MONT_MAC(w0, w1, w2, u[38], m[39]); + MONT_MAC(t0, t1, t2, u[39], m[38]); + MONT_MAC(v0, v1, v2, u[40], m[37]); + MONT_MAC(w0, w1, w2, u[41], m[36]); + MONT_MAC(t0, t1, t2, u[42], m[35]); + MONT_MAC(v0, v1, v2, u[43], m[34]); + MONT_MAC(w0, w1, w2, u[44], m[33]); + MONT_MAC(t0, t1, t2, u[45], m[32]); + MONT_MAC(v0, v1, v2, u[46], m[31]); + MONT_MAC(w0, w1, w2, u[47], m[30]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[29] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 78 */ + MONT_MUL(s0, s1, s2, a[31], a[47]); + MONT_MUL(c0, c1, c2, a[32], a[46]); + MONT_MAC(s0, s1, s2, a[33], a[45]); + MONT_MAC(c0, c1, c2, a[34], a[44]); + MONT_MAC(s0, s1, s2, a[35], a[43]); + MONT_MAC(c0, c1, c2, a[36], a[42]); + MONT_MAC(s0, s1, s2, a[37], a[41]); + MONT_MAC(c0, c1, c2, a[38], a[40]); + MONT_MAC(t0, t1, t2, a[39], a[39]); + MONT_MUL(v0, v1, v2, u[31], m[47]); + MONT_MUL(w0, w1, w2, u[32], m[46]); + MONT_MAC(t0, t1, t2, u[33], m[45]); + MONT_MAC(v0, v1, v2, u[34], m[44]); + MONT_MAC(w0, w1, w2, u[35], m[43]); + MONT_MAC(t0, t1, t2, u[36], m[42]); + MONT_MAC(v0, v1, v2, u[37], m[41]); + MONT_MAC(w0, w1, w2, u[38], m[40]); + MONT_MAC(t0, t1, t2, u[39], m[39]); + MONT_MAC(v0, v1, v2, u[40], m[38]); + MONT_MAC(w0, w1, w2, u[41], m[37]); + MONT_MAC(t0, t1, t2, u[42], m[36]); + MONT_MAC(v0, v1, v2, u[43], m[35]); + MONT_MAC(w0, w1, w2, u[44], m[34]); + MONT_MAC(t0, t1, t2, u[45], m[33]); + MONT_MAC(v0, v1, v2, u[46], m[32]); + MONT_MAC(w0, w1, w2, u[47], m[31]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[30] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 79 */ + MONT_MUL(s0, s1, s2, a[32], a[47]); + MONT_MUL(c0, c1, c2, a[33], a[46]); + MONT_MAC(s0, s1, s2, a[34], a[45]); + MONT_MAC(c0, c1, c2, a[35], a[44]); + MONT_MAC(s0, s1, s2, a[36], a[43]); + MONT_MAC(c0, c1, c2, a[37], a[42]); + MONT_MAC(s0, s1, s2, a[38], a[41]); + MONT_MAC(c0, c1, c2, a[39], a[40]); + MONT_MAC(t0, t1, t2, u[32], m[47]); + MONT_MUL(v0, v1, v2, u[33], m[46]); + MONT_MUL(w0, w1, w2, u[34], m[45]); + MONT_MAC(t0, t1, t2, u[35], m[44]); + MONT_MAC(v0, v1, v2, u[36], m[43]); + MONT_MAC(w0, w1, w2, u[37], m[42]); + MONT_MAC(t0, t1, t2, u[38], m[41]); + MONT_MAC(v0, v1, v2, u[39], m[40]); + MONT_MAC(w0, w1, w2, u[40], m[39]); + MONT_MAC(t0, t1, t2, u[41], m[38]); + MONT_MAC(v0, v1, v2, u[42], m[37]); + MONT_MAC(w0, w1, w2, u[43], m[36]); + MONT_MAC(t0, t1, t2, u[44], m[35]); + MONT_MAC(v0, v1, v2, u[45], m[34]); + MONT_MAC(w0, w1, w2, u[46], m[33]); + MONT_MAC(t0, t1, t2, u[47], m[32]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[31] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 80 */ + MONT_MUL(s0, s1, s2, a[33], a[47]); + MONT_MUL(c0, c1, c2, a[34], a[46]); + MONT_MAC(s0, s1, s2, a[35], a[45]); + MONT_MAC(c0, c1, c2, a[36], a[44]); + MONT_MAC(s0, s1, s2, a[37], a[43]); + MONT_MAC(c0, c1, c2, a[38], a[42]); + MONT_MAC(s0, s1, s2, a[39], a[41]); + MONT_MAC(t0, t1, t2, a[40], a[40]); + MONT_MUL(v0, v1, v2, u[33], m[47]); + MONT_MUL(w0, w1, w2, u[34], m[46]); + MONT_MAC(t0, t1, t2, u[35], m[45]); + MONT_MAC(v0, v1, v2, u[36], m[44]); + MONT_MAC(w0, w1, w2, u[37], m[43]); + MONT_MAC(t0, t1, t2, u[38], m[42]); + MONT_MAC(v0, v1, v2, u[39], m[41]); + MONT_MAC(w0, w1, w2, u[40], m[40]); + MONT_MAC(t0, t1, t2, u[41], m[39]); + MONT_MAC(v0, v1, v2, u[42], m[38]); + MONT_MAC(w0, w1, w2, u[43], m[37]); + MONT_MAC(t0, t1, t2, u[44], m[36]); + MONT_MAC(v0, v1, v2, u[45], m[35]); + MONT_MAC(w0, w1, w2, u[46], m[34]); + MONT_MAC(t0, t1, t2, u[47], m[33]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[32] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 81 */ + MONT_MUL(s0, s1, s2, a[34], a[47]); + MONT_MUL(c0, c1, c2, a[35], a[46]); + MONT_MAC(s0, s1, s2, a[36], a[45]); + MONT_MAC(c0, c1, c2, a[37], a[44]); + MONT_MAC(s0, s1, s2, a[38], a[43]); + MONT_MAC(c0, c1, c2, a[39], a[42]); + MONT_MAC(s0, s1, s2, a[40], a[41]); + MONT_MAC(t0, t1, t2, u[34], m[47]); + MONT_MUL(v0, v1, v2, u[35], m[46]); + MONT_MUL(w0, w1, w2, u[36], m[45]); + MONT_MAC(t0, t1, t2, u[37], m[44]); + MONT_MAC(v0, v1, v2, u[38], m[43]); + MONT_MAC(w0, w1, w2, u[39], m[42]); + MONT_MAC(t0, t1, t2, u[40], m[41]); + MONT_MAC(v0, v1, v2, u[41], m[40]); + MONT_MAC(w0, w1, w2, u[42], m[39]); + MONT_MAC(t0, t1, t2, u[43], m[38]); + MONT_MAC(v0, v1, v2, u[44], m[37]); + MONT_MAC(w0, w1, w2, u[45], m[36]); + MONT_MAC(t0, t1, t2, u[46], m[35]); + MONT_MAC(v0, v1, v2, u[47], m[34]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[33] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 82 */ + MONT_MUL(s0, s1, s2, a[35], a[47]); + MONT_MUL(c0, c1, c2, a[36], a[46]); + MONT_MAC(s0, s1, s2, a[37], a[45]); + MONT_MAC(c0, c1, c2, a[38], a[44]); + MONT_MAC(s0, s1, s2, a[39], a[43]); + MONT_MAC(c0, c1, c2, a[40], a[42]); + MONT_MAC(t0, t1, t2, a[41], a[41]); + MONT_MUL(v0, v1, v2, u[35], m[47]); + MONT_MUL(w0, w1, w2, u[36], m[46]); + MONT_MAC(t0, t1, t2, u[37], m[45]); + MONT_MAC(v0, v1, v2, u[38], m[44]); + MONT_MAC(w0, w1, w2, u[39], m[43]); + MONT_MAC(t0, t1, t2, u[40], m[42]); + MONT_MAC(v0, v1, v2, u[41], m[41]); + MONT_MAC(w0, w1, w2, u[42], m[40]); + MONT_MAC(t0, t1, t2, u[43], m[39]); + MONT_MAC(v0, v1, v2, u[44], m[38]); + MONT_MAC(w0, w1, w2, u[45], m[37]); + MONT_MAC(t0, t1, t2, u[46], m[36]); + MONT_MAC(v0, v1, v2, u[47], m[35]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[34] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 83 */ + MONT_MUL(s0, s1, s2, a[36], a[47]); + MONT_MUL(c0, c1, c2, a[37], a[46]); + MONT_MAC(s0, s1, s2, a[38], a[45]); + MONT_MAC(c0, c1, c2, a[39], a[44]); + MONT_MAC(s0, s1, s2, a[40], a[43]); + MONT_MAC(c0, c1, c2, a[41], a[42]); + MONT_MAC(t0, t1, t2, u[36], m[47]); + MONT_MUL(v0, v1, v2, u[37], m[46]); + MONT_MUL(w0, w1, w2, u[38], m[45]); + MONT_MAC(t0, t1, t2, u[39], m[44]); + MONT_MAC(v0, v1, v2, u[40], m[43]); + MONT_MAC(w0, w1, w2, u[41], m[42]); + MONT_MAC(t0, t1, t2, u[42], m[41]); + MONT_MAC(v0, v1, v2, u[43], m[40]); + MONT_MAC(w0, w1, w2, u[44], m[39]); + MONT_MAC(t0, t1, t2, u[45], m[38]); + MONT_MAC(v0, v1, v2, u[46], m[37]); + MONT_MAC(w0, w1, w2, u[47], m[36]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[35] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 84 */ + MONT_MUL(s0, s1, s2, a[37], a[47]); + MONT_MUL(c0, c1, c2, a[38], a[46]); + MONT_MAC(s0, s1, s2, a[39], a[45]); + MONT_MAC(c0, c1, c2, a[40], a[44]); + MONT_MAC(s0, s1, s2, a[41], a[43]); + MONT_MAC(t0, t1, t2, a[42], a[42]); + MONT_MUL(v0, v1, v2, u[37], m[47]); + MONT_MUL(w0, w1, w2, u[38], m[46]); + MONT_MAC(t0, t1, t2, u[39], m[45]); + MONT_MAC(v0, v1, v2, u[40], m[44]); + MONT_MAC(w0, w1, w2, u[41], m[43]); + MONT_MAC(t0, t1, t2, u[42], m[42]); + MONT_MAC(v0, v1, v2, u[43], m[41]); + MONT_MAC(w0, w1, w2, u[44], m[40]); + MONT_MAC(t0, t1, t2, u[45], m[39]); + MONT_MAC(v0, v1, v2, u[46], m[38]); + MONT_MAC(w0, w1, w2, u[47], m[37]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[36] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 85 */ + MONT_MUL(s0, s1, s2, a[38], a[47]); + MONT_MUL(c0, c1, c2, a[39], a[46]); + MONT_MAC(s0, s1, s2, a[40], a[45]); + MONT_MAC(c0, c1, c2, a[41], a[44]); + MONT_MAC(s0, s1, s2, a[42], a[43]); + MONT_MAC(t0, t1, t2, u[38], m[47]); + MONT_MUL(v0, v1, v2, u[39], m[46]); + MONT_MUL(w0, w1, w2, u[40], m[45]); + MONT_MAC(t0, t1, t2, u[41], m[44]); + MONT_MAC(v0, v1, v2, u[42], m[43]); + MONT_MAC(w0, w1, w2, u[43], m[42]); + MONT_MAC(t0, t1, t2, u[44], m[41]); + MONT_MAC(v0, v1, v2, u[45], m[40]); + MONT_MAC(w0, w1, w2, u[46], m[39]); + MONT_MAC(t0, t1, t2, u[47], m[38]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[37] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 86 */ + MONT_MUL(s0, s1, s2, a[39], a[47]); + MONT_MUL(c0, c1, c2, a[40], a[46]); + MONT_MAC(s0, s1, s2, a[41], a[45]); + MONT_MAC(c0, c1, c2, a[42], a[44]); + MONT_MAC(t0, t1, t2, a[43], a[43]); + MONT_MUL(v0, v1, v2, u[39], m[47]); + MONT_MUL(w0, w1, w2, u[40], m[46]); + MONT_MAC(t0, t1, t2, u[41], m[45]); + MONT_MAC(v0, v1, v2, u[42], m[44]); + MONT_MAC(w0, w1, w2, u[43], m[43]); + MONT_MAC(t0, t1, t2, u[44], m[42]); + MONT_MAC(v0, v1, v2, u[45], m[41]); + MONT_MAC(w0, w1, w2, u[46], m[40]); + MONT_MAC(t0, t1, t2, u[47], m[39]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[38] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 87 */ + MONT_MUL(s0, s1, s2, a[40], a[47]); + MONT_MUL(c0, c1, c2, a[41], a[46]); + MONT_MAC(s0, s1, s2, a[42], a[45]); + MONT_MAC(c0, c1, c2, a[43], a[44]); + MONT_MAC(t0, t1, t2, u[40], m[47]); + MONT_MUL(v0, v1, v2, u[41], m[46]); + MONT_MUL(w0, w1, w2, u[42], m[45]); + MONT_MAC(t0, t1, t2, u[43], m[44]); + MONT_MAC(v0, v1, v2, u[44], m[43]); + MONT_MAC(w0, w1, w2, u[45], m[42]); + MONT_MAC(t0, t1, t2, u[46], m[41]); + MONT_MAC(v0, v1, v2, u[47], m[40]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[39] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 88 */ + MONT_MUL(s0, s1, s2, a[41], a[47]); + MONT_MUL(c0, c1, c2, a[42], a[46]); + MONT_MAC(s0, s1, s2, a[43], a[45]); + MONT_MAC(t0, t1, t2, a[44], a[44]); + MONT_MUL(v0, v1, v2, u[41], m[47]); + MONT_MUL(w0, w1, w2, u[42], m[46]); + MONT_MAC(t0, t1, t2, u[43], m[45]); + MONT_MAC(v0, v1, v2, u[44], m[44]); + MONT_MAC(w0, w1, w2, u[45], m[43]); + MONT_MAC(t0, t1, t2, u[46], m[42]); + MONT_MAC(v0, v1, v2, u[47], m[41]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[40] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 89 */ + MONT_MUL(s0, s1, s2, a[42], a[47]); + MONT_MUL(c0, c1, c2, a[43], a[46]); + MONT_MAC(s0, s1, s2, a[44], a[45]); + MONT_MAC(t0, t1, t2, u[42], m[47]); + MONT_MUL(v0, v1, v2, u[43], m[46]); + MONT_MUL(w0, w1, w2, u[44], m[45]); + MONT_MAC(t0, t1, t2, u[45], m[44]); + MONT_MAC(v0, v1, v2, u[46], m[43]); + MONT_MAC(w0, w1, w2, u[47], m[42]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[41] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 90 */ + MONT_MUL(s0, s1, s2, a[43], a[47]); + MONT_MUL(c0, c1, c2, a[44], a[46]); + MONT_MAC(t0, t1, t2, a[45], a[45]); + MONT_MUL(v0, v1, v2, u[43], m[47]); + MONT_MUL(w0, w1, w2, u[44], m[46]); + MONT_MAC(t0, t1, t2, u[45], m[45]); + MONT_MAC(v0, v1, v2, u[46], m[44]); + MONT_MAC(w0, w1, w2, u[47], m[43]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[42] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 91 */ + MONT_MUL(s0, s1, s2, a[44], a[47]); + MONT_MUL(c0, c1, c2, a[45], a[46]); + MONT_MAC(t0, t1, t2, u[44], m[47]); + MONT_MUL(v0, v1, v2, u[45], m[46]); + MONT_MUL(w0, w1, w2, u[46], m[45]); + MONT_MAC(t0, t1, t2, u[47], m[44]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[43] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 92 */ + MONT_MUL(s0, s1, s2, a[45], a[47]); + MONT_MAC(t0, t1, t2, a[46], a[46]); + MONT_MUL(v0, v1, v2, u[45], m[47]); + MONT_MUL(w0, w1, w2, u[46], m[46]); + MONT_MAC(t0, t1, t2, u[47], m[45]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[44] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 93 */ + MONT_MUL(s0, s1, s2, a[46], a[47]); + MONT_MAC(t0, t1, t2, u[46], m[47]); + MONT_MUL(v0, v1, v2, u[47], m[46]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[45] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 94 */ + MONT_MAC(t0, t1, t2, a[47], a[47]); + MONT_MUL(v0, v1, v2, u[47], m[47]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + z[46] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + z[47] = t0; + + /* under 2m, with t1 the bit above n limbs: at most one subtraction */ + borrow = sub_n(r, z, m, 48); + take = t1 | (borrow ^ 1); + select_n(r, r, z, take & 1, 48); +} + +#endif diff --git a/cbits/crypton_mont.py b/cbits/crypton_mont.py new file mode 100644 --- /dev/null +++ b/cbits/crypton_mont.py @@ -0,0 +1,173 @@ +#!/usr/bin/env python3 +"""Generate cbits/crypton_mont.h: Montgomery multiplication and squaring for +AArch64 at the limb counts of RSA's CRT halves, fully unrolled and +product-scanning. + +Usage: python3 cbits/crypton_mont.py > cbits/crypton_mont.h + +The result is a column at a time, with the reduction folded into the same +scan (the FIPS method), so the running sum stays in registers instead of +going through memory a row at a time. A column's products are spread over +three accumulators and added together at its end, since one accumulator's +carry chain is what holds the core back, not its multipliers. In a square, +each product off the diagonal is summed once, doubled, and added in. +""" + +SIZES = (16, 24, 32) +# Squaring at 48 too, RSA-3072's modulus, for the verification that is +# seventeen squarings. Its multiplication, at twice the code, is left out. +SQR_ONLY = (48,) + +HEADER = """\ +/* Generated by cbits/crypton_mont.py; do not edit. + * + * Montgomery multiplication and squaring for AArch64 at 16, 24 and 32 limbs, + * the halves of RSA-2048, -3072 and -4096, and squaring at 48, RSA-3072's + * modulus. Each answers what mont_mul and mont_sqr answer. */ +#ifndef CRYPTON_MONT_H +#define CRYPTON_MONT_H + +#include "crypton_bignum.h" + +/* (t2:t1:t0) += x * y */ +#define MONT_MAC(t0, t1, t2, x, y) \\ + do { \\ + limb_t lo_, hi_; \\ + __asm__("mul\\t%0, %5, %6\\n\\t" \\ + "umulh\\t%1, %5, %6\\n\\t" \\ + "adds\\t%2, %2, %0\\n\\t" \\ + "adcs\\t%3, %3, %1\\n\\t" \\ + "adc\\t%4, %4, xzr" \\ + : "=&r"(lo_), "=&r"(hi_), "+r"(t0), "+r"(t1), "+r"(t2) \\ + : "r"(x), "r"(y) \\ + : "cc"); \\ + } while (0) + +/* (s2:s1:s0) = x * y */ +#define MONT_MUL(s0, s1, s2, x, y) \\ + __asm__("mul\\t%0, %3, %4\\n\\t" \\ + "umulh\\t%1, %3, %4\\n\\t" \\ + "mov\\t%2, xzr" \\ + : "=&r"(s0), "=&r"(s1), "=r"(s2) \\ + : "r"(x), "r"(y)) + +/* (t2:t1:t0) += (s2:s1:s0) */ +#define MONT_ADD(t0, t1, t2, s0, s1, s2) \\ + __asm__("adds\\t%0, %0, %3\\n\\t" \\ + "adcs\\t%1, %1, %4\\n\\t" \\ + "adc\\t%2, %2, %5" \\ + : "+r"(t0), "+r"(t1), "+r"(t2) \\ + : "r"(s0), "r"(s1), "r"(s2) \\ + : "cc") + +/* (s2:s1:s0) *= 2 */ +#define MONT_DBL(s0, s1, s2) \\ + do { \\ + s2 = (s2 << 1) | (s1 >> 63); \\ + s1 = (s1 << 1) | (s0 >> 63); \\ + s0 <<= 1; \\ + } while (0) +""" + +FOOTER = """\ +#endif +""" + + +class Column: + """Products added to accumulators, each set up by its first product.""" + + def __init__(self, w): + self.w = w + self.started = set() + + def add(self, acc, x, y): + if acc == "t" or acc in self.started: + self.w(f"\tMONT_MAC({acc}0, {acc}1, {acc}2, {x}, {y});") + else: + self.w(f"\tMONT_MUL({acc}0, {acc}1, {acc}2, {x}, {y});") + self.started.add(acc) + + def fold(self, into, acc): + if acc in self.started: + self.w(f"\tMONT_ADD({into}0, {into}1, {into}2, {acc}0, {acc}1, {acc}2);") + + +def column(w, singles, cross): + """singles are added once, cross twice: s and c take the cross products, + t, v and w take the rest, and all of them end up in t.""" + col = Column(w) + for k, (x, y) in enumerate(cross): + col.add("s" if k % 2 == 0 else "c", x, y) + for k, (x, y) in enumerate(singles): + col.add("tvw"[k % 3], x, y) + col.fold("t", "v") + col.fold("t", "w") + if cross: + col.fold("s", "c") + w("\tMONT_DBL(s0, s1, s2);") + w("\tMONT_ADD(t0, t1, t2, s0, s1, s2);") + + +def gen(n, square): + out = [] + w = out.append + name = f"crypton_mont_{'sqr' if square else 'mul'}{n}" + args = "const limb_t *a," if square else "const limb_t *a, const limb_t *b," + if square: + w(f"/* r = a * a * R^-1 mod m, for a under m; t is 2n limbs of scratch */") + else: + w(f"/* r = a * b * R^-1 mod m, for a and b under m; t is 2n limbs of scratch */") + w(f"static void {name}(limb_t *r, {args}") + w(f"{' ' * len(f'static void {name}(')}const limb_t *m, limb_t n0, limb_t *t)") + w("{") + w(f"\tlimb_t *u = t, *z = t + {n};") + w("\tlimb_t t0 = 0, t1 = 0, t2 = 0, v0, v1, v2, w0, w1, w2, borrow, take;") + if square: + w("\tlimb_t s0, s1, s2, c0, c1, c2;") + w("") + for i in range(2 * n - 1): + w(f"\t/* column {i} */") + singles, cross = [], [] + for j in range(max(0, i - n + 1), min(i, n - 1) + 1): + k = i - j + if not square: + singles.append((f"a[{j}]", f"b[{k}]")) + elif j < k: + cross.append((f"a[{j}]", f"a[{k}]")) + elif j == k: + singles.append((f"a[{j}]", f"a[{j}]")) + for j in range(max(0, i - n + 1), min(i, n)): + singles.append((f"u[{j}]", f"m[{i - j}]")) + column(w, singles, cross) + if i < n: + w(f"\tu[{i}] = t0 * n0;") + w(f"\tMONT_MAC(t0, t1, t2, u[{i}], m[0]);") + else: + w(f"\tz[{i - n}] = t0;") + w("\tt0 = t1;") + w("\tt1 = t2;") + w("\tt2 = 0;") + w(f"\tz[{n - 1}] = t0;") + w("") + w("\t/* under 2m, with t1 the bit above n limbs: at most one subtraction */") + w(f"\tborrow = sub_n(r, z, m, {n});") + w("\ttake = t1 | (borrow ^ 1);") + w(f"\tselect_n(r, r, z, take & 1, {n});") + w("}") + w("") + return "\n".join(out) + + +def main(): + print(HEADER) + for n in SIZES: + print(gen(n, False)) + print(gen(n, True)) + for n in SQR_ONLY: + print(gen(n, True)) + print(FOOTER, end="") + + +if __name__ == "__main__": + main() diff --git a/cbits/crypton_powm.c b/cbits/crypton_powm.c --- a/cbits/crypton_powm.c +++ b/cbits/crypton_powm.c @@ -22,6 +22,16 @@ #include #include +/* On AArch64, multiplication and squaring at the RSA half sizes go through + * generated code that scans a column at a time with the reduction folded in; + * see cbits/crypton_mont.py. CRYPTON_NO_MONT_FIXED keeps mont_mul and + * mont_sqr, to compare. */ +#if defined(__aarch64__) && LIMB_BITS == 64 && !defined(CRYPTON_NO_MONT_FIXED) \ + && (defined(__GNUC__) || defined(__clang__)) +#define CRYPTON_MONT_FIXED 1 +#include +#endif + /* * At RSA sizes on x86-64, the Montgomery multiplication below is the whole * cost, and s2n-bignum's is twice as fast because the C cannot form the two @@ -32,7 +42,9 @@ * Not on AArch64, where the C measures 5% faster than the assembly. * See cbits/s2n/README.md. */ -#if defined(CRYPTON_S2N_BIGNUM) && defined(__x86_64__) +/* Not on macOS, where crypton.cabal leaves these files out: Apple's assembler + * cannot assemble bignum_emontredc_8n. See the note beside them there. */ +#if defined(CRYPTON_S2N_BIGNUM) && defined(__x86_64__) && !defined(__APPLE__) #define CRYPTON_POWM_S2N 1 #include @@ -44,20 +56,56 @@ extern void bignum_ksqr_32_64(uint64_t *z, const uint64_t *x, uint64_t *t); extern uint64_t bignum_emontredc_8n(uint64_t k, uint64_t *z, const uint64_t *m, uint64_t w); +extern void bignum_mul(uint64_t k, uint64_t *z, uint64_t m, const uint64_t *x, + uint64_t n, const uint64_t *y); +extern void bignum_sqr(uint64_t k, uint64_t *z, uint64_t n, const uint64_t *x); +extern void bignum_sqr_8_16(uint64_t *z, const uint64_t *x); +extern void bignum_mul_8_16(uint64_t *z, const uint64_t *x, const uint64_t *y); /* 16 limbs is 1024 bits and 32 is 2048: the halves a CRT exponentiation - * works in for RSA-2048 and RSA-4096, and the whole thing without CRT. The - * reduction wants ADX, so the answer is a run-time one. */ + * works in for RSA-2048 and RSA-4096, and the whole thing without CRT. 24 + * is 1536, the half of RSA-3072, which has no Karatsuba routine of its own: + * its multiplication is the 32-limb one with the operands padded, and its + * square is built from 8-limb pieces, which measured faster on a whole + * exponentiation than the padded square. Both are still nearly twice the + * C, since the C cannot form ADCX and ADOX's two carry chains at any size. + * + * 48 and 64 are 3072 and 4096 bits: finite-field Diffie-Hellman over the + * RFC 7919 groups of those sizes, and RSA without CRT. There is no + * Karatsuba routine at either, and padding to the next would be wasteful, + * so the multiplication is s2n-bignum's schoolbook one of any size -- + * plain MUL and ADC, no ADX -- and the reduction is the same ADX one. Not + * the gain the Karatsuba sizes have, and still well ahead of the C. + * + * The reduction wants ADX, so the answer is a run-time one. */ static int powm_s2n_usable(uint32_t n) { - return (n == 16 || n == 32) + return (n == 16 || n == 24 || n == 32 || n == 48 || n == 64) && (crypton_x86_simd_features() & CRYPTON_X86_ADX) != 0; } -/* The scratch the widest of them asks for, in multiples of n: kmul_32_64 - * wants 96 limbs for n = 32. */ -#define POWM_S2N_SCRATCH 3 +/* The scratch the routines ask for, in limbs. kmul_16_32 and kmul_32_64 + * want three times their operand. A 24-limb operand padded to 32 wants the + * 32-limb routine's 64-limb product and its 96 limbs of its own, and the two + * padded operands besides. */ +#define POWM_S2N_PAD_Z 0 +#define POWM_S2N_PAD_T 64 +#define POWM_S2N_PAD_X (64 + 96) +#define POWM_S2N_PAD_Y (64 + 96 + 32) +static uint32_t powm_s2n_scratch(uint32_t n) +{ + return n == 24 ? 64 + 96 + 32 + 32 : 3 * n; +} +/* a 24-limb operand in the low limbs of 32, the rest zero: zeroed every + * time rather than once, so that nothing else writing the scratch can + * leave a high limb behind */ +static void powm_s2n_pad(limb_t *to, const limb_t *from) +{ + memcpy(to, from, 24 * sizeof(limb_t)); + memset(to + 24, 0, 8 * sizeof(limb_t)); +} + /* bignum_emontredc_8n leaves the result in the top half of z with one more * bit as its return value, and what is there is under twice the modulus -- * the same place mont_reduce ends up, and finished the same way. */ @@ -74,24 +122,72 @@ const limb_t *m, limb_t n0, uint32_t n, limb_t *z, limb_t *scratch) { - if (n == 16) + if (n == 24) { + /* the product of two 24-limb numbers is 48 limbs, so the top + * sixteen of the 64 written are zero and the reduction reads + * only what it should */ + z = scratch + POWM_S2N_PAD_Z; + powm_s2n_pad(scratch + POWM_S2N_PAD_X, a); + powm_s2n_pad(scratch + POWM_S2N_PAD_Y, b); + bignum_kmul_32_64(z, scratch + POWM_S2N_PAD_X, + scratch + POWM_S2N_PAD_Y, + scratch + POWM_S2N_PAD_T); + } else if (n == 48 || n == 64) + bignum_mul(2 * n, z, n, a, n, b); + else if (n == 16) bignum_kmul_16_32(z, a, b, scratch); else bignum_kmul_32_64(z, a, b, scratch); powm_s2n_finish(r, z, m, bignum_emontredc_8n(n, z, m, n0), n); } +/* a^2 for a of 24 limbs, from three 8-limb squares and three 8-limb + * products: a = a0 + a1 X + a2 X^2 with X = 2^512, and the cross terms, at + * X to X^3 in c, go in doubled. c is 48 limbs of scratch. */ +static void powm_s2n_sqr24(limb_t *z, const limb_t *a, limb_t *c) +{ + limb_t k; + uint32_t i; + + bignum_sqr_8_16(z, a); + bignum_sqr_8_16(z + 16, a + 8); + bignum_sqr_8_16(z + 32, a + 16); + bignum_mul_8_16(c, a, a + 8); + bignum_mul_8_16(c + 16, a + 8, a + 16); + bignum_mul_8_16(c + 32, a, a + 16); + k = add_n(c + 8, c + 8, c + 32, 16); + for (i = 24; i < 32; i++) { + c[i] += k; + k = c[i] < k; + } + k = shl1(c, 32); + k += add_n(z + 8, z + 8, c, 32); + for (i = 40; i < 48; i++) { + z[i] += k; + k = z[i] < k; + } +} + static void powm_s2n_sqr(limb_t *r, const limb_t *a, const limb_t *m, limb_t n0, uint32_t n, limb_t *z, limb_t *scratch) { - if (n == 16) + if (n == 24) { + z = scratch + POWM_S2N_PAD_Z; + powm_s2n_sqr24(z, a, scratch + POWM_S2N_PAD_T); + } else if (n == 48 || n == 64) + bignum_sqr(2 * n, z, n, a); + else if (n == 16) bignum_ksqr_16_32(z, a, scratch); else bignum_ksqr_32_64(z, a, scratch); powm_s2n_finish(r, z, m, bignum_emontredc_8n(n, z, m, n0), n); } #else -#define POWM_S2N_SCRATCH 0 +static uint32_t powm_s2n_scratch(uint32_t n) +{ + (void) n; + return 0; +} #endif /* One or the other, decided once per call */ @@ -108,6 +204,13 @@ (void)scratch; (void)s2n; #endif +#ifdef CRYPTON_MONT_FIXED + switch (n) { + case 16: crypton_mont_mul16(r, a, b, m, n0, t); return; + case 24: crypton_mont_mul24(r, a, b, m, n0, t); return; + case 32: crypton_mont_mul32(r, a, b, m, n0, t); return; + } +#endif mont_mul(r, a, b, m, n0, n, t); } @@ -123,24 +226,71 @@ (void)scratch; (void)s2n; #endif +#ifdef CRYPTON_MONT_FIXED + switch (n) { + case 16: crypton_mont_sqr16(r, a, m, n0, t); return; + case 24: crypton_mont_sqr24(r, a, m, n0, t); return; + case 32: crypton_mont_sqr32(r, a, m, n0, t); return; + case 48: crypton_mont_sqr48(r, a, m, n0, t); return; + } +#endif mont_sqr(r, a, m, n0, n, t); } -/* Four bits of exponent per window, so a table of sixteen and no leftover - * bits: a byte holds exactly two windows. +/* Four bits of exponent per window, so a table of sixteen. * - * Five was written and measured, and is not here. A wider window saves + * Five was written and measured and is not here. A wider window saves * multiplications -- 205 of them against 256 at 1024 bits, with the same * 1024 squarings -- and pays for it in the masked scan of a table twice as - * long, and which way that comes out depends on the machine and on which - * multiplication is running: 3.5% better on an Apple M4, about 1% worse on - * an older x86-64, and 7% worse anywhere s2n-bignum's multiplication is - * used, since that makes the scan the expensive half. Six measured level - * with five on the M4 and seven worse. What would make a wider window pay - * everywhere is a cheaper scan, not a wider window. */ -#define WINDOW_BITS 4 + * long. When that was first measured the scan was the scalar one, and five + * came out 3.5% better on an Apple M4, about 1% worse on an older x86-64 and + * 7% worse where s2n-bignum's multiplication runs, since that makes the scan + * the expensive half. The conclusion written here was that what would make + * a wider window pay is a cheaper scan. + * + * The cheaper scan arrived twenty-one minutes later, in the AVX2 form above, + * and nobody measured the window again. Measured since, on the machine the + * x86-64 table is measured on and with both builds raced in one run: five is + * 0.6% worse than four, beside 0.5% between two builds that differ in + * nothing. So the cheaper scan did not turn it round -- the window stays at + * four, and now for a reason that was checked after the scan rather than + * before it. + * + * The width is POWM_WINDOW_BITS so that it can be measured rather than + * argued about. It had to be renamed to be settable at all: WINDOW_BITS was + * defined here, in cbits/crypton_ecc.c and in cbits/crypton_bignum.h, all + * three to four, and a redefinition to the same value is silent -- so + * -DWINDOW_BITS=5 was overridden by the header and three builds meant to + * differ did not. cbits/tests/powm builds the walk at four, five and six + * and requires the same answers of all three. */ +#ifndef POWM_WINDOW_BITS +#define POWM_WINDOW_BITS 4 +#endif +#define WINDOW_BITS POWM_WINDOW_BITS #define TABLE_SIZE (1 << WINDOW_BITS) +/* Window number k counting from the bottom, where the bottom window holds + * the exponent's least significant WINDOW_BITS bits. The exponent is + * big-endian bytes, and a window may straddle a byte boundary once the width + * stops dividing eight, so this reads the two bytes it can fall across. + * + * Which window is being read is the loop counter, not the exponent, so the + * shifts and the index here are public. + */ +static limb_t exp_window(const uint8_t *exp, uint32_t explen, uint32_t k) +{ + uint32_t lo = k * WINDOW_BITS; /* bit offset from the bottom */ + uint32_t byte = lo / 8, off = lo % 8; + uint32_t v; + + if (byte >= explen) + return 0; + v = exp[explen - 1 - byte]; + if (off + WINDOW_BITS > 8 && byte + 1 < explen) + v |= (uint32_t) exp[explen - 2 - byte] << 8; + return (v >> off) & (TABLE_SIZE - 1); +} + /* The masked scan of the table: every entry is read and a mask keeps the one * wanted, so that the address stream does not follow the exponent. At * RSA-2048's CRT size that is two kilobytes read per window, and the window @@ -205,13 +355,195 @@ } #endif +/* Modular exponentiation for an exponent that is not a secret. + * + * RSA verification raises a signature to the public exponent, which is in + * the public key and is usually 65537. crypton_powm_sec below hides the + * exponent and pays for it twice: a table of sixteen powers built before the + * walk begins, and every entry of that table read at every window. For a + * seventeen-bit exponent the table alone costs more than the exponentiation. + * + * So this one reads the exponent's bits and branches on them, squaring at + * every step and multiplying where a bit is set. It is for public exponents + * only, and nothing in this library may call it with a secret. + * + * It declines -- returning 1 without touching out -- in three cases, so a + * caller must have something to fall back to. + * + * The first is a long exponent. Walking it a bit at a time costs about one + * and a half multiplications per bit where a windowed walk costs a quarter + * less, and the table that buys the difference pays for itself after some + * sixty-odd bits. Past that the caller's own exponentiation is the better + * algorithm and this one would have to grow a table to compete, which is + * the thing it exists not to have. RSA public exponents are seventeen + * bits; the bound is set well above them and well below where a Diffie- + * Hellman or DSA exponent starts. + * + * The second and third are a machine and a size where GMP, which the + * caller has, was measured the faster; see powm_pub_wanted. + * + * Define CRYPTON_POWM_PUB_ALWAYS to make it accept anyway: the differential + * test builds that way, so that this arithmetic is exercised on the + * machines the assembly never reaches. + */ +#define POWM_PUB_MAX_EXP_BYTES 8 + +/* Which sizes this is worth taking at is not which sizes the assembly runs + * at. Against GMP, which is what the caller has, on an EPYC 7763 with ADX + * on both sides: ahead at 2048 bits, and behind everywhere else measured -- + * slightly at 1024, by more at 1536, and by more again at 3072 and 4096, + * which is the size where GMP's own setup and multiplication are the better + * pair. So 2048 alone, which is RSA-2048's modulus and the size nearly + * every RSA public key has. It once took every size the assembly ran at, + * on the strength of 2048 alone, and was slower than GMP at 1024. */ +static int powm_pub_wanted(uint32_t n) +{ +#if defined(CRYPTON_POWM_PUB_ALWAYS) + return n > 0; +#elif defined(CRYPTON_POWM_S2N) + return n == 32 && powm_s2n_usable(n); +#elif defined(CRYPTON_MONT_FIXED) + /* AArch64: the generated code is ahead of GMP at 2048 bits. At the + * other RSA sizes Linux's GMP is level with the C or ahead of it. The + * GMP GHC builds into itself on macOS is compiled without -O, and is + * behind the C from 1024 to 4096 bits on the same M4 */ +#if defined(__APPLE__) + return n >= 16 && n <= 64; +#else + return n == 32; +#endif +#else + (void) n; + return 0; +#endif +} + +int crypton_powm_window_bits(void) +{ + return WINDOW_BITS; +} + +int crypton_powm_pub(uint8_t *out, + const uint8_t *base, uint32_t baselen, + const uint8_t *exp, uint32_t explen, + const uint8_t *mod, uint32_t modlen) +{ + return crypton_powm_pub_r2(out, base, baselen, exp, explen, mod, modlen, NULL); +} + +int crypton_powm_r2(uint8_t *out, const uint8_t *mod, uint32_t modlen) +{ + uint32_t n = (modlen + LIMB_BYTES - 1) / LIMB_BYTES; + limb_t *space, *m, *r2, *t; + + if (modlen == 0 || n == 0 || (mod[modlen - 1] & 1) == 0 || !powm_pub_wanted(n)) + return 1; + space = calloc(5 * n, sizeof(limb_t)); + if (space == NULL) + return 1; + m = space; + r2 = m + n; + t = r2 + n; + if (from_be(m, n, mod, modlen) != 0) { + free(space); + return 1; + } + mont_r2(r2, m, mont_n0(m[0]), n, t); + to_be(out, modlen, r2, n); + free(space); + return 0; +} + +int crypton_powm_pub_r2(uint8_t *out, + const uint8_t *base, uint32_t baselen, + const uint8_t *exp, uint32_t explen, + const uint8_t *mod, uint32_t modlen, + const uint8_t *r2be) +{ + uint32_t n = (modlen + LIMB_BYTES - 1) / LIMB_BYTES; + uint32_t words = (6 + 2) * n + powm_s2n_scratch(n); + limb_t *space, *m, *r2, *acc, *tmp, *bm, *one, *t, *scratch, n0; + uint32_t top, i; + int s2n = 0; + + if (modlen == 0 || n == 0 || (mod[modlen - 1] & 1) == 0) + return 1; + if (explen > POWM_PUB_MAX_EXP_BYTES) + return 1; + if (!powm_pub_wanted(n)) + return 1; +#ifdef CRYPTON_POWM_S2N + s2n = powm_s2n_usable(n); +#endif + + /* the exponent's top set bit, counted from the bottom. An exponent of + * zero is left to the caller, whose answer for it is already right. */ + top = explen * 8; + while (top > 0 && + ((exp[explen - 1 - (top - 1) / 8] >> ((top - 1) % 8)) & 1) == 0) + top--; + if (top == 0) + return 1; + + space = calloc(words, sizeof(limb_t)); + if (space == NULL) + return 1; + m = space; + r2 = m + n; + acc = r2 + n; + tmp = acc + n; + bm = tmp + n; + one = bm + n; + t = one + n; + scratch = t + 2 * n; + + if (from_be(m, n, mod, modlen) != 0) + goto fail; + n0 = mont_n0(m[0]); + if (r2be == NULL) + mont_r2(r2, m, n0, n, t); + else if (from_be(r2, n, r2be, modlen) != 0) + goto fail; + + /* the base into Montgomery form. The accumulator starts there rather + * than at one, because the exponent's top bit is a one by + * construction and the first squaring would be of one. */ + if (from_be(tmp, n, base, baselen) != 0) + goto fail; + powm_mul(bm, tmp, r2, m, n0, n, t, scratch, s2n); + memcpy(acc, bm, n * sizeof(limb_t)); + + for (i = top - 1; i > 0; i--) { + limb_t *swap; + + powm_sqr(tmp, acc, m, n0, n, t, scratch, s2n); + swap = acc; acc = tmp; tmp = swap; + if ((exp[explen - 1 - (i - 1) / 8] >> ((i - 1) % 8)) & 1) { + powm_mul(tmp, acc, bm, m, n0, n, t, scratch, s2n); + swap = acc; acc = tmp; tmp = swap; + } + } + + /* out of Montgomery form */ + one[0] = 1; + powm_mul(tmp, acc, one, m, n0, n, t, scratch, s2n); + to_be(out, modlen, tmp, n); + + free(space); + return 0; + +fail: + free(space); + return 1; +} + int crypton_powm_sec(uint8_t *out, const uint8_t *base, uint32_t baselen, const uint8_t *exp, uint32_t explen, const uint8_t *mod, uint32_t modlen) { uint32_t n = (modlen + LIMB_BYTES - 1) / LIMB_BYTES; - uint32_t words = (TABLE_SIZE + 7 + POWM_S2N_SCRATCH) * n; + uint32_t words = (TABLE_SIZE + 7) * n + powm_s2n_scratch(n); limb_t *space, *m, *r2, *acc, *sel, *prod, *table, *t, *scratch, n0; uint32_t i, j, k; int s2n = 0; @@ -259,9 +591,8 @@ memcpy(acc, table, n * sizeof(limb_t)); - for (i = explen * 2; i > 0; i--) { - uint32_t nib = i - 1; - limb_t w = (exp[explen - 1 - nib / 2] >> (4 * (nib % 2))) & 0xf; + for (i = (explen * 8 + WINDOW_BITS - 1) / WINDOW_BITS; i > 0; i--) { + limb_t w = exp_window(exp, explen, i - 1); /* squaring into the other buffer and swapping the two saves a * copy of the modulus' width every time; which of the three diff --git a/cbits/crypton_powm.h b/cbits/crypton_powm.h --- a/cbits/crypton_powm.h +++ b/cbits/crypton_powm.h @@ -20,4 +20,53 @@ const uint8_t *exp, uint32_t explen, const uint8_t *mod, uint32_t modlen); +/* The same, for an exponent that is not a secret: it reads the exponent's + * bits and branches on them. Nothing in this library may call it with a + * secret exponent. + * + * It declines -- returning 1 with out untouched -- whenever it is not the + * faster of the two things the caller could do: for an exponent of more + * than eight bytes, where a windowed walk is the better algorithm; + * everywhere the vendored assembly's Montgomery multiplication does not + * run; and at every modulus size but 2048 bits, where GMP was measured the + * faster even with the assembly. So a caller must have something to fall + * back to. + */ +int crypton_powm_pub(uint8_t *out, + const uint8_t *base, uint32_t baselen, + const uint8_t *exp, uint32_t explen, + const uint8_t *mod, uint32_t modlen); + +/* crypton_powm_pub with R^2 mod m given, big-endian in modlen bytes, as + * crypton_powm_r2 computes it; NULL computes it here as crypton_powm_pub does. + * A caller verifying under one key many times keeps it. */ +int crypton_powm_pub_r2(uint8_t *out, + const uint8_t *base, uint32_t baselen, + const uint8_t *exp, uint32_t explen, + const uint8_t *mod, uint32_t modlen, + const uint8_t *r2be); + +/* R^2 mod m for crypton_powm_pub_r2, or nonzero where crypton_powm_pub would + * decline this modulus, so that nothing is kept for it. */ +int crypton_powm_r2(uint8_t *out, const uint8_t *mod, uint32_t modlen); + +/* Two secure exponentiations at once, as the halves of an RSA private-key + * operation by CRT are: each as crypton_powm_sec, with the same contract. + * + * Only on x86-64 with AVX-512 IFMA, for two moduli of 128, 192 or 256 bytes + * and two exponents of the same length. It declines everywhere else -- + * returning nonzero with both outputs untouched -- so a caller must have + * crypton_powm_sec to fall back to. */ +int crypton_powm_sec_x2(uint8_t *out1, const uint8_t *base1, uint32_t baselen1, + const uint8_t *exp1, uint32_t explen1, + const uint8_t *mod1, uint32_t modlen1, + uint8_t *out2, const uint8_t *base2, uint32_t baselen2, + const uint8_t *exp2, uint32_t explen2, + const uint8_t *mod2, uint32_t modlen2); + +/* The window the secure exponentiation walks the exponent in, so that a + * build which did not take -DPOWM_WINDOW_BITS cannot pass for one that + * did. */ +int crypton_powm_window_bits(void); + #endif diff --git a/cbits/crypton_powm_ifma.c b/cbits/crypton_powm_ifma.c new file mode 100644 --- /dev/null +++ b/cbits/crypton_powm_ifma.c @@ -0,0 +1,187 @@ +/* Two secure exponentiations at once with AVX-512 IFMA, the CRT halves of + * RSA-2048, -3072 and -4096: crypton_powm_sec's walk with a five-bit window, + * over the multiplication generated by cbits/crypton_powm_ifma.py. */ +#include +#include +#include +#include +#include + +#if defined(__x86_64__) && (defined(__GNUC__) || defined(__clang__)) +#include +#include "crypton_powm_ifma.h" + +#define WIN 5 + +typedef void (*amm_fn)(uint64_t *, const uint64_t *, const uint64_t *, const uint64_t *, + uint64_t, uint64_t *, const uint64_t *, const uint64_t *, + const uint64_t *, uint64_t); +typedef void (*sel_fn)(uint64_t *, const uint64_t *, uint64_t, uint64_t *, + const uint64_t *, uint64_t); + +/* limbs of 64 bits to digits of 52, d of them; the rest of the w words zero */ +static void to_digits(uint64_t *dg, uint32_t w, const limb_t *l, uint32_t n, uint32_t d) +{ + uint32_t i; + + memset(dg, 0, w * sizeof(uint64_t)); + for (i = 0; i < d; i++) { + uint32_t bit = 52 * i, li = bit / 64, sh = bit % 64; + uint64_t v = li < n ? l[li] >> sh : 0; + + if (sh > 12 && li + 1 < n) + v |= l[li + 1] << (64 - sh); + dg[i] = v & IFMA_MASK52; + } +} + +static void from_digits(limb_t *l, uint32_t n, const uint64_t *dg, uint32_t d) +{ + uint32_t i; + + memset(l, 0, n * sizeof(limb_t)); + for (i = 0; i < d; i++) { + uint32_t bit = 52 * i, li = bit / 64, sh = bit % 64; + + if (li < n) + l[li] |= dg[i] << sh; + if (sh > 12 && li + 1 < n) + l[li + 1] |= dg[i] >> (64 - sh); + } +} + +/* Window k from the bottom of a big-endian exponent of len bytes. Which + * window is read is the loop counter's business, not the exponent's. */ +static uint64_t window(const uint8_t *e, uint32_t len, uint32_t k) +{ + uint64_t v = 0; + uint32_t j; + + for (j = 0; j < WIN; j++) { + uint32_t bit = k * WIN + j; + + if (bit < 8 * len) + v |= (uint64_t) ((e[len - 1 - bit / 8] >> (bit % 8)) & 1) << j; + } + return v; +} + +/* m and k0 in digits, and R^2 mod m for R = 2^(52d): crypton's R^2 is for + * 2^(64n), so it is doubled the rest of the way */ +static int setup(const uint8_t *mod, uint32_t modlen, uint32_t n, uint32_t d, uint32_t w, + limb_t *ml, uint64_t *md, uint64_t *rr, uint64_t *k0, limb_t *t) +{ + limb_t *r2 = t + 4 * n; + uint32_t j; + + if (from_be(ml, n, mod, modlen) != 0 || (ml[0] & 1) == 0) + return 1; + *k0 = mont_n0(ml[0]); + mont_r2(r2, ml, *k0, n, t); + for (j = 0; j < 2 * (52 * d - 64 * n); j++) + dbl_mod(r2, ml, n, t); + to_digits(md, w, ml, n, d); + to_digits(rr, w, r2, n, d); + return 0; +} + +int crypton_powm_sec_x2(uint8_t *out1, const uint8_t *base1, uint32_t baselen1, + const uint8_t *exp1, uint32_t explen1, + const uint8_t *mod1, uint32_t modlen1, + uint8_t *out2, const uint8_t *base2, uint32_t baselen2, + const uint8_t *exp2, uint32_t explen2, + const uint8_t *mod2, uint32_t modlen2) +{ + uint32_t n, d, w, nwin, k, j; + amm_fn amm; + sel_fn sel; + uint64_t *space, *m1, *m2, *rr1, *rr2, *x1, *x2, *s1, *s2, *one, *t1, *t2, k1, k2; + limb_t *l1, *l2, *lt, *lt2; + size_t words; + + if (modlen1 != modlen2 || explen1 != explen2) + return 1; + if ((crypton_x86_simd_features() & CRYPTON_X86_IFMA) == 0) + return 1; + switch (modlen1) { + case 128: d = 20; amm = ifma_amm20_x2; sel = ifma_sel5; break; + case 192: d = 30; amm = ifma_amm30_x2; sel = ifma_sel8; break; + case 256: d = 40; amm = ifma_amm40_x2; sel = ifma_sel10; break; + default: return 1; + } + n = modlen1 / 8; + w = 4 * ((d + 3) / 4); + + /* nine numbers in digits, two tables of 32, and limbs for the setup */ + words = (size_t) w * (9 + 2 * 32) + (size_t) n * 12; + space = calloc(words, sizeof(uint64_t)); + if (space == NULL) + return 1; + m1 = space; m2 = m1 + w; rr1 = m2 + w; rr2 = rr1 + w; + x1 = rr2 + w; x2 = x1 + w; s1 = x2 + w; s2 = s1 + w; one = s2 + w; + t1 = one + w; t2 = t1 + 32 * w; + l1 = (limb_t *)(t2 + 32 * w); l2 = l1 + n; lt = l2 + n; lt2 = lt + n; + + if (setup(mod1, modlen1, n, d, w, l1, m1, rr1, &k1, lt) != 0 + || setup(mod2, modlen2, n, d, w, l2, m2, rr2, &k2, lt) != 0) + goto fail; + + /* the bases, under their moduli as the contract says, into digits */ + if (from_be(lt, n, base1, baselen1) != 0 || from_be(lt2, n, base2, baselen2) != 0) + goto fail; + to_digits(s1, w, lt, n, d); + to_digits(s2, w, lt2, n, d); + + /* table[0] = R and table[1] = b R in Montgomery form, table[k] = b^k R */ + one[0] = 1; + amm(t1, one, rr1, m1, k1, t2, one, rr2, m2, k2); + amm(t1 + w, s1, rr1, m1, k1, t2 + w, s2, rr2, m2, k2); + for (k = 2; k < 32; k++) + amm(t1 + k * w, t1 + (k - 1) * w, t1 + w, m1, k1, + t2 + k * w, t2 + (k - 1) * w, t2 + w, m2, k2); + + memcpy(x1, t1, w * sizeof(uint64_t)); + memcpy(x2, t2, w * sizeof(uint64_t)); + nwin = (8 * explen1 + WIN - 1) / WIN; + for (k = nwin; k > 0; k--) { + for (j = 0; j < WIN; j++) + amm(x1, x1, x1, m1, k1, x2, x2, x2, m2, k2); + sel(s1, t1, window(exp1, explen1, k - 1), s2, t2, window(exp2, explen2, k - 1)); + amm(x1, x1, s1, m1, k1, x2, x2, s2, m2, k2); + } + + /* out of Montgomery form, then under m with one masked subtraction */ + amm(x1, x1, one, m1, k1, x2, x2, one, m2, k2); + from_digits(lt, n, x1, d); + select_n(lt, lt2, lt, sub_n(lt2, lt, l1, n) ^ 1, n); + to_be(out1, modlen1, lt, n); + from_digits(lt, n, x2, d); + select_n(lt, lt2, lt, sub_n(lt2, lt, l2, n) ^ 1, n); + to_be(out2, modlen2, lt, n); + + crypton_bzero(space, words * sizeof(uint64_t)); + free(space); + return 0; + +fail: + crypton_bzero(space, words * sizeof(uint64_t)); + free(space); + return 1; +} + +#else + +int crypton_powm_sec_x2(uint8_t *out1, const uint8_t *base1, uint32_t baselen1, + const uint8_t *exp1, uint32_t explen1, + const uint8_t *mod1, uint32_t modlen1, + uint8_t *out2, const uint8_t *base2, uint32_t baselen2, + const uint8_t *exp2, uint32_t explen2, + const uint8_t *mod2, uint32_t modlen2) +{ + (void) out1; (void) base1; (void) baselen1; (void) exp1; (void) explen1; + (void) mod1; (void) modlen1; (void) out2; (void) base2; (void) baselen2; + (void) exp2; (void) explen2; (void) mod2; (void) modlen2; + return 1; +} + +#endif diff --git a/cbits/crypton_powm_ifma.h b/cbits/crypton_powm_ifma.h new file mode 100644 --- /dev/null +++ b/cbits/crypton_powm_ifma.h @@ -0,0 +1,513 @@ +/* Generated by cbits/crypton_powm_ifma.py; do not edit. + * + * Almost-Montgomery multiplication in radix 2^52, two at once: r = a b + * 2^-52D mod m, under 2m for a and b under 2m. Digits are 52 bits in 64-bit + * words, each number padded with zero words to a whole number of 256-bit + * registers. m is read from memory; at 20 digits a is held in registers, + * above that it is read from memory too. */ +#ifndef CRYPTON_POWM_IFMA_H +#define CRYPTON_POWM_IFMA_H + +#include +#include + +#define IFMA_MASK52 ((UINT64_C(1) << 52) - 1) +#define IFMA_TARGET __attribute__((target("avx2,avx512f,avx512vl,avx512ifma"))) + +/* 20 lanes to 52-bit digits: what is above 52 bits moves up a lane, + * and the one-bit carries that can then remain are resolved together */ +IFMA_TARGET static inline void ifma_norm5(uint64_t *r, __m256i X0, __m256i X1, __m256i X2, __m256i X3, __m256i X4) +{ + const __m256i mk = _mm256_set1_epi64x((long long) IFMA_MASK52), one = _mm256_set1_epi64x(1); + const __m256i zero = _mm256_setzero_si256(); + uint64_t g = 0, p = 0, c; + __m256i C0 = _mm256_srli_epi64(X0, 52); + __m256i C1 = _mm256_srli_epi64(X1, 52); + __m256i C2 = _mm256_srli_epi64(X2, 52); + __m256i C3 = _mm256_srli_epi64(X3, 52); + __m256i C4 = _mm256_srli_epi64(X4, 52); + X0 = _mm256_add_epi64(_mm256_and_si256(X0, mk), _mm256_alignr_epi64(C0, zero, 3)); + X1 = _mm256_add_epi64(_mm256_and_si256(X1, mk), _mm256_alignr_epi64(C1, C0, 3)); + X2 = _mm256_add_epi64(_mm256_and_si256(X2, mk), _mm256_alignr_epi64(C2, C1, 3)); + X3 = _mm256_add_epi64(_mm256_and_si256(X3, mk), _mm256_alignr_epi64(C3, C2, 3)); + X4 = _mm256_add_epi64(_mm256_and_si256(X4, mk), _mm256_alignr_epi64(C4, C3, 3)); + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X0, mk) << 0; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X0, mk) << 0; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X1, mk) << 4; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X1, mk) << 4; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X2, mk) << 8; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X2, mk) << 8; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X3, mk) << 12; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X3, mk) << 12; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X4, mk) << 16; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X4, mk) << 16; + c = ((g << 1) + p) ^ p; + X0 = _mm256_and_si256(_mm256_mask_add_epi64(X0, (__mmask8)(c >> 0 & 15), X0, one), mk); + _mm256_storeu_si256((__m256i *)(r + 0), X0); + X1 = _mm256_and_si256(_mm256_mask_add_epi64(X1, (__mmask8)(c >> 4 & 15), X1, one), mk); + _mm256_storeu_si256((__m256i *)(r + 4), X1); + X2 = _mm256_and_si256(_mm256_mask_add_epi64(X2, (__mmask8)(c >> 8 & 15), X2, one), mk); + _mm256_storeu_si256((__m256i *)(r + 8), X2); + X3 = _mm256_and_si256(_mm256_mask_add_epi64(X3, (__mmask8)(c >> 12 & 15), X3, one), mk); + _mm256_storeu_si256((__m256i *)(r + 12), X3); + X4 = _mm256_and_si256(_mm256_mask_add_epi64(X4, (__mmask8)(c >> 16 & 15), X4, one), mk); + _mm256_storeu_si256((__m256i *)(r + 16), X4); +} + +/* both tables' entries k1 and k2 of 32, every entry read and a mask keeping + * one; entries are 20 words apart */ +IFMA_TARGET static void ifma_sel5(uint64_t *o1, const uint64_t *t1, uint64_t k1, + uint64_t *o2, const uint64_t *t2, uint64_t k2) +{ + const __m256i one = _mm256_set1_epi64x(1); + __m256i v1 = _mm256_set1_epi64x((long long) k1), v2 = _mm256_set1_epi64x((long long) k2); + __m256i kv = _mm256_setzero_si256(); + __m256i P0 = kv, Q0 = kv, P1 = kv, Q1 = kv, P2 = kv, Q2 = kv, P3 = kv, Q3 = kv, P4 = kv, Q4 = kv; + int k; + for (k = 0; k < 32; k++) { + __mmask8 s1 = _mm256_cmpeq_epi64_mask(v1, kv), s2 = _mm256_cmpeq_epi64_mask(v2, kv); + const uint64_t *e1 = t1 + 20 * k, *e2 = t2 + 20 * k; + P0 = _mm256_mask_loadu_epi64(P0, s1, e1 + 0); Q0 = _mm256_mask_loadu_epi64(Q0, s2, e2 + 0); + P1 = _mm256_mask_loadu_epi64(P1, s1, e1 + 4); Q1 = _mm256_mask_loadu_epi64(Q1, s2, e2 + 4); + P2 = _mm256_mask_loadu_epi64(P2, s1, e1 + 8); Q2 = _mm256_mask_loadu_epi64(Q2, s2, e2 + 8); + P3 = _mm256_mask_loadu_epi64(P3, s1, e1 + 12); Q3 = _mm256_mask_loadu_epi64(Q3, s2, e2 + 12); + P4 = _mm256_mask_loadu_epi64(P4, s1, e1 + 16); Q4 = _mm256_mask_loadu_epi64(Q4, s2, e2 + 16); + kv = _mm256_add_epi64(kv, one); + } + _mm256_storeu_si256((__m256i *)(o1 + 0), P0); _mm256_storeu_si256((__m256i *)(o2 + 0), Q0); + _mm256_storeu_si256((__m256i *)(o1 + 4), P1); _mm256_storeu_si256((__m256i *)(o2 + 4), Q1); + _mm256_storeu_si256((__m256i *)(o1 + 8), P2); _mm256_storeu_si256((__m256i *)(o2 + 8), Q2); + _mm256_storeu_si256((__m256i *)(o1 + 12), P3); _mm256_storeu_si256((__m256i *)(o2 + 12), Q3); + _mm256_storeu_si256((__m256i *)(o1 + 16), P4); _mm256_storeu_si256((__m256i *)(o2 + 16), Q4); +} + +/* 32 lanes to 52-bit digits: what is above 52 bits moves up a lane, + * and the one-bit carries that can then remain are resolved together */ +IFMA_TARGET static inline void ifma_norm8(uint64_t *r, __m256i X0, __m256i X1, __m256i X2, __m256i X3, __m256i X4, __m256i X5, __m256i X6, __m256i X7) +{ + const __m256i mk = _mm256_set1_epi64x((long long) IFMA_MASK52), one = _mm256_set1_epi64x(1); + const __m256i zero = _mm256_setzero_si256(); + uint64_t g = 0, p = 0, c; + __m256i C0 = _mm256_srli_epi64(X0, 52); + __m256i C1 = _mm256_srli_epi64(X1, 52); + __m256i C2 = _mm256_srli_epi64(X2, 52); + __m256i C3 = _mm256_srli_epi64(X3, 52); + __m256i C4 = _mm256_srli_epi64(X4, 52); + __m256i C5 = _mm256_srli_epi64(X5, 52); + __m256i C6 = _mm256_srli_epi64(X6, 52); + __m256i C7 = _mm256_srli_epi64(X7, 52); + X0 = _mm256_add_epi64(_mm256_and_si256(X0, mk), _mm256_alignr_epi64(C0, zero, 3)); + X1 = _mm256_add_epi64(_mm256_and_si256(X1, mk), _mm256_alignr_epi64(C1, C0, 3)); + X2 = _mm256_add_epi64(_mm256_and_si256(X2, mk), _mm256_alignr_epi64(C2, C1, 3)); + X3 = _mm256_add_epi64(_mm256_and_si256(X3, mk), _mm256_alignr_epi64(C3, C2, 3)); + X4 = _mm256_add_epi64(_mm256_and_si256(X4, mk), _mm256_alignr_epi64(C4, C3, 3)); + X5 = _mm256_add_epi64(_mm256_and_si256(X5, mk), _mm256_alignr_epi64(C5, C4, 3)); + X6 = _mm256_add_epi64(_mm256_and_si256(X6, mk), _mm256_alignr_epi64(C6, C5, 3)); + X7 = _mm256_add_epi64(_mm256_and_si256(X7, mk), _mm256_alignr_epi64(C7, C6, 3)); + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X0, mk) << 0; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X0, mk) << 0; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X1, mk) << 4; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X1, mk) << 4; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X2, mk) << 8; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X2, mk) << 8; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X3, mk) << 12; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X3, mk) << 12; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X4, mk) << 16; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X4, mk) << 16; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X5, mk) << 20; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X5, mk) << 20; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X6, mk) << 24; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X6, mk) << 24; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X7, mk) << 28; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X7, mk) << 28; + c = ((g << 1) + p) ^ p; + X0 = _mm256_and_si256(_mm256_mask_add_epi64(X0, (__mmask8)(c >> 0 & 15), X0, one), mk); + _mm256_storeu_si256((__m256i *)(r + 0), X0); + X1 = _mm256_and_si256(_mm256_mask_add_epi64(X1, (__mmask8)(c >> 4 & 15), X1, one), mk); + _mm256_storeu_si256((__m256i *)(r + 4), X1); + X2 = _mm256_and_si256(_mm256_mask_add_epi64(X2, (__mmask8)(c >> 8 & 15), X2, one), mk); + _mm256_storeu_si256((__m256i *)(r + 8), X2); + X3 = _mm256_and_si256(_mm256_mask_add_epi64(X3, (__mmask8)(c >> 12 & 15), X3, one), mk); + _mm256_storeu_si256((__m256i *)(r + 12), X3); + X4 = _mm256_and_si256(_mm256_mask_add_epi64(X4, (__mmask8)(c >> 16 & 15), X4, one), mk); + _mm256_storeu_si256((__m256i *)(r + 16), X4); + X5 = _mm256_and_si256(_mm256_mask_add_epi64(X5, (__mmask8)(c >> 20 & 15), X5, one), mk); + _mm256_storeu_si256((__m256i *)(r + 20), X5); + X6 = _mm256_and_si256(_mm256_mask_add_epi64(X6, (__mmask8)(c >> 24 & 15), X6, one), mk); + _mm256_storeu_si256((__m256i *)(r + 24), X6); + X7 = _mm256_and_si256(_mm256_mask_add_epi64(X7, (__mmask8)(c >> 28 & 15), X7, one), mk); + _mm256_storeu_si256((__m256i *)(r + 28), X7); +} + +/* both tables' entries k1 and k2 of 32, every entry read and a mask keeping + * one; entries are 32 words apart */ +IFMA_TARGET static void ifma_sel8(uint64_t *o1, const uint64_t *t1, uint64_t k1, + uint64_t *o2, const uint64_t *t2, uint64_t k2) +{ + const __m256i one = _mm256_set1_epi64x(1); + __m256i v1 = _mm256_set1_epi64x((long long) k1), v2 = _mm256_set1_epi64x((long long) k2); + __m256i kv = _mm256_setzero_si256(); + __m256i P0 = kv, Q0 = kv, P1 = kv, Q1 = kv, P2 = kv, Q2 = kv, P3 = kv, Q3 = kv, P4 = kv, Q4 = kv, P5 = kv, Q5 = kv, P6 = kv, Q6 = kv, P7 = kv, Q7 = kv; + int k; + for (k = 0; k < 32; k++) { + __mmask8 s1 = _mm256_cmpeq_epi64_mask(v1, kv), s2 = _mm256_cmpeq_epi64_mask(v2, kv); + const uint64_t *e1 = t1 + 32 * k, *e2 = t2 + 32 * k; + P0 = _mm256_mask_loadu_epi64(P0, s1, e1 + 0); Q0 = _mm256_mask_loadu_epi64(Q0, s2, e2 + 0); + P1 = _mm256_mask_loadu_epi64(P1, s1, e1 + 4); Q1 = _mm256_mask_loadu_epi64(Q1, s2, e2 + 4); + P2 = _mm256_mask_loadu_epi64(P2, s1, e1 + 8); Q2 = _mm256_mask_loadu_epi64(Q2, s2, e2 + 8); + P3 = _mm256_mask_loadu_epi64(P3, s1, e1 + 12); Q3 = _mm256_mask_loadu_epi64(Q3, s2, e2 + 12); + P4 = _mm256_mask_loadu_epi64(P4, s1, e1 + 16); Q4 = _mm256_mask_loadu_epi64(Q4, s2, e2 + 16); + P5 = _mm256_mask_loadu_epi64(P5, s1, e1 + 20); Q5 = _mm256_mask_loadu_epi64(Q5, s2, e2 + 20); + P6 = _mm256_mask_loadu_epi64(P6, s1, e1 + 24); Q6 = _mm256_mask_loadu_epi64(Q6, s2, e2 + 24); + P7 = _mm256_mask_loadu_epi64(P7, s1, e1 + 28); Q7 = _mm256_mask_loadu_epi64(Q7, s2, e2 + 28); + kv = _mm256_add_epi64(kv, one); + } + _mm256_storeu_si256((__m256i *)(o1 + 0), P0); _mm256_storeu_si256((__m256i *)(o2 + 0), Q0); + _mm256_storeu_si256((__m256i *)(o1 + 4), P1); _mm256_storeu_si256((__m256i *)(o2 + 4), Q1); + _mm256_storeu_si256((__m256i *)(o1 + 8), P2); _mm256_storeu_si256((__m256i *)(o2 + 8), Q2); + _mm256_storeu_si256((__m256i *)(o1 + 12), P3); _mm256_storeu_si256((__m256i *)(o2 + 12), Q3); + _mm256_storeu_si256((__m256i *)(o1 + 16), P4); _mm256_storeu_si256((__m256i *)(o2 + 16), Q4); + _mm256_storeu_si256((__m256i *)(o1 + 20), P5); _mm256_storeu_si256((__m256i *)(o2 + 20), Q5); + _mm256_storeu_si256((__m256i *)(o1 + 24), P6); _mm256_storeu_si256((__m256i *)(o2 + 24), Q6); + _mm256_storeu_si256((__m256i *)(o1 + 28), P7); _mm256_storeu_si256((__m256i *)(o2 + 28), Q7); +} + +/* 40 lanes to 52-bit digits: what is above 52 bits moves up a lane, + * and the one-bit carries that can then remain are resolved together */ +IFMA_TARGET static inline void ifma_norm10(uint64_t *r, __m256i X0, __m256i X1, __m256i X2, __m256i X3, __m256i X4, __m256i X5, __m256i X6, __m256i X7, __m256i X8, __m256i X9) +{ + const __m256i mk = _mm256_set1_epi64x((long long) IFMA_MASK52), one = _mm256_set1_epi64x(1); + const __m256i zero = _mm256_setzero_si256(); + uint64_t g = 0, p = 0, c; + __m256i C0 = _mm256_srli_epi64(X0, 52); + __m256i C1 = _mm256_srli_epi64(X1, 52); + __m256i C2 = _mm256_srli_epi64(X2, 52); + __m256i C3 = _mm256_srli_epi64(X3, 52); + __m256i C4 = _mm256_srli_epi64(X4, 52); + __m256i C5 = _mm256_srli_epi64(X5, 52); + __m256i C6 = _mm256_srli_epi64(X6, 52); + __m256i C7 = _mm256_srli_epi64(X7, 52); + __m256i C8 = _mm256_srli_epi64(X8, 52); + __m256i C9 = _mm256_srli_epi64(X9, 52); + X0 = _mm256_add_epi64(_mm256_and_si256(X0, mk), _mm256_alignr_epi64(C0, zero, 3)); + X1 = _mm256_add_epi64(_mm256_and_si256(X1, mk), _mm256_alignr_epi64(C1, C0, 3)); + X2 = _mm256_add_epi64(_mm256_and_si256(X2, mk), _mm256_alignr_epi64(C2, C1, 3)); + X3 = _mm256_add_epi64(_mm256_and_si256(X3, mk), _mm256_alignr_epi64(C3, C2, 3)); + X4 = _mm256_add_epi64(_mm256_and_si256(X4, mk), _mm256_alignr_epi64(C4, C3, 3)); + X5 = _mm256_add_epi64(_mm256_and_si256(X5, mk), _mm256_alignr_epi64(C5, C4, 3)); + X6 = _mm256_add_epi64(_mm256_and_si256(X6, mk), _mm256_alignr_epi64(C6, C5, 3)); + X7 = _mm256_add_epi64(_mm256_and_si256(X7, mk), _mm256_alignr_epi64(C7, C6, 3)); + X8 = _mm256_add_epi64(_mm256_and_si256(X8, mk), _mm256_alignr_epi64(C8, C7, 3)); + X9 = _mm256_add_epi64(_mm256_and_si256(X9, mk), _mm256_alignr_epi64(C9, C8, 3)); + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X0, mk) << 0; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X0, mk) << 0; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X1, mk) << 4; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X1, mk) << 4; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X2, mk) << 8; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X2, mk) << 8; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X3, mk) << 12; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X3, mk) << 12; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X4, mk) << 16; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X4, mk) << 16; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X5, mk) << 20; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X5, mk) << 20; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X6, mk) << 24; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X6, mk) << 24; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X7, mk) << 28; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X7, mk) << 28; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X8, mk) << 32; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X8, mk) << 32; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X9, mk) << 36; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X9, mk) << 36; + c = ((g << 1) + p) ^ p; + X0 = _mm256_and_si256(_mm256_mask_add_epi64(X0, (__mmask8)(c >> 0 & 15), X0, one), mk); + _mm256_storeu_si256((__m256i *)(r + 0), X0); + X1 = _mm256_and_si256(_mm256_mask_add_epi64(X1, (__mmask8)(c >> 4 & 15), X1, one), mk); + _mm256_storeu_si256((__m256i *)(r + 4), X1); + X2 = _mm256_and_si256(_mm256_mask_add_epi64(X2, (__mmask8)(c >> 8 & 15), X2, one), mk); + _mm256_storeu_si256((__m256i *)(r + 8), X2); + X3 = _mm256_and_si256(_mm256_mask_add_epi64(X3, (__mmask8)(c >> 12 & 15), X3, one), mk); + _mm256_storeu_si256((__m256i *)(r + 12), X3); + X4 = _mm256_and_si256(_mm256_mask_add_epi64(X4, (__mmask8)(c >> 16 & 15), X4, one), mk); + _mm256_storeu_si256((__m256i *)(r + 16), X4); + X5 = _mm256_and_si256(_mm256_mask_add_epi64(X5, (__mmask8)(c >> 20 & 15), X5, one), mk); + _mm256_storeu_si256((__m256i *)(r + 20), X5); + X6 = _mm256_and_si256(_mm256_mask_add_epi64(X6, (__mmask8)(c >> 24 & 15), X6, one), mk); + _mm256_storeu_si256((__m256i *)(r + 24), X6); + X7 = _mm256_and_si256(_mm256_mask_add_epi64(X7, (__mmask8)(c >> 28 & 15), X7, one), mk); + _mm256_storeu_si256((__m256i *)(r + 28), X7); + X8 = _mm256_and_si256(_mm256_mask_add_epi64(X8, (__mmask8)(c >> 32 & 15), X8, one), mk); + _mm256_storeu_si256((__m256i *)(r + 32), X8); + X9 = _mm256_and_si256(_mm256_mask_add_epi64(X9, (__mmask8)(c >> 36 & 15), X9, one), mk); + _mm256_storeu_si256((__m256i *)(r + 36), X9); +} + +/* both tables' entries k1 and k2 of 32, every entry read and a mask keeping + * one; entries are 40 words apart */ +IFMA_TARGET static void ifma_sel10(uint64_t *o1, const uint64_t *t1, uint64_t k1, + uint64_t *o2, const uint64_t *t2, uint64_t k2) +{ + const __m256i one = _mm256_set1_epi64x(1); + __m256i v1 = _mm256_set1_epi64x((long long) k1), v2 = _mm256_set1_epi64x((long long) k2); + __m256i kv = _mm256_setzero_si256(); + __m256i P0 = kv, Q0 = kv, P1 = kv, Q1 = kv, P2 = kv, Q2 = kv, P3 = kv, Q3 = kv, P4 = kv, Q4 = kv, P5 = kv, Q5 = kv, P6 = kv, Q6 = kv, P7 = kv, Q7 = kv, P8 = kv, Q8 = kv, P9 = kv, Q9 = kv; + int k; + for (k = 0; k < 32; k++) { + __mmask8 s1 = _mm256_cmpeq_epi64_mask(v1, kv), s2 = _mm256_cmpeq_epi64_mask(v2, kv); + const uint64_t *e1 = t1 + 40 * k, *e2 = t2 + 40 * k; + P0 = _mm256_mask_loadu_epi64(P0, s1, e1 + 0); Q0 = _mm256_mask_loadu_epi64(Q0, s2, e2 + 0); + P1 = _mm256_mask_loadu_epi64(P1, s1, e1 + 4); Q1 = _mm256_mask_loadu_epi64(Q1, s2, e2 + 4); + P2 = _mm256_mask_loadu_epi64(P2, s1, e1 + 8); Q2 = _mm256_mask_loadu_epi64(Q2, s2, e2 + 8); + P3 = _mm256_mask_loadu_epi64(P3, s1, e1 + 12); Q3 = _mm256_mask_loadu_epi64(Q3, s2, e2 + 12); + P4 = _mm256_mask_loadu_epi64(P4, s1, e1 + 16); Q4 = _mm256_mask_loadu_epi64(Q4, s2, e2 + 16); + P5 = _mm256_mask_loadu_epi64(P5, s1, e1 + 20); Q5 = _mm256_mask_loadu_epi64(Q5, s2, e2 + 20); + P6 = _mm256_mask_loadu_epi64(P6, s1, e1 + 24); Q6 = _mm256_mask_loadu_epi64(Q6, s2, e2 + 24); + P7 = _mm256_mask_loadu_epi64(P7, s1, e1 + 28); Q7 = _mm256_mask_loadu_epi64(Q7, s2, e2 + 28); + P8 = _mm256_mask_loadu_epi64(P8, s1, e1 + 32); Q8 = _mm256_mask_loadu_epi64(Q8, s2, e2 + 32); + P9 = _mm256_mask_loadu_epi64(P9, s1, e1 + 36); Q9 = _mm256_mask_loadu_epi64(Q9, s2, e2 + 36); + kv = _mm256_add_epi64(kv, one); + } + _mm256_storeu_si256((__m256i *)(o1 + 0), P0); _mm256_storeu_si256((__m256i *)(o2 + 0), Q0); + _mm256_storeu_si256((__m256i *)(o1 + 4), P1); _mm256_storeu_si256((__m256i *)(o2 + 4), Q1); + _mm256_storeu_si256((__m256i *)(o1 + 8), P2); _mm256_storeu_si256((__m256i *)(o2 + 8), Q2); + _mm256_storeu_si256((__m256i *)(o1 + 12), P3); _mm256_storeu_si256((__m256i *)(o2 + 12), Q3); + _mm256_storeu_si256((__m256i *)(o1 + 16), P4); _mm256_storeu_si256((__m256i *)(o2 + 16), Q4); + _mm256_storeu_si256((__m256i *)(o1 + 20), P5); _mm256_storeu_si256((__m256i *)(o2 + 20), Q5); + _mm256_storeu_si256((__m256i *)(o1 + 24), P6); _mm256_storeu_si256((__m256i *)(o2 + 24), Q6); + _mm256_storeu_si256((__m256i *)(o1 + 28), P7); _mm256_storeu_si256((__m256i *)(o2 + 28), Q7); + _mm256_storeu_si256((__m256i *)(o1 + 32), P8); _mm256_storeu_si256((__m256i *)(o2 + 32), Q8); + _mm256_storeu_si256((__m256i *)(o1 + 36), P9); _mm256_storeu_si256((__m256i *)(o2 + 36), Q9); +} + +/* 20 digits, 5 registers a number */ +IFMA_TARGET static void ifma_amm20_x2(uint64_t *r1, const uint64_t *a1, const uint64_t *b1, + const uint64_t *m1, uint64_t k1, + uint64_t *r2, const uint64_t *a2, const uint64_t *b2, + const uint64_t *m2, uint64_t k2) +{ + const __m256i zero = _mm256_setzero_si256(); +#define L(p, k) _mm256_loadu_si256((const __m256i *)((p) + 4 * (k))) + __m256i X0 = zero, X1 = zero, X2 = zero, X3 = zero, X4 = zero, Y0 = zero, Y1 = zero, Y2 = zero, Y3 = zero, Y4 = zero; + __m256i A0 = L(a1, 0), A1 = L(a1, 1), A2 = L(a1, 2), A3 = L(a1, 3), A4 = L(a1, 4); + __m256i B0 = L(a2, 0), B1 = L(a2, 1), B2 = L(a2, 2), B3 = L(a2, 3), B4 = L(a2, 4); + uint64_t x1 = 0, x2 = 0; + int i; + + for (i = 0; i < 20; i++) { + uint64_t bi = b1[i], ci = b2[i], y, z; + /* lane 0 in full: the products' high halves, which are lane 1's, too */ + unsigned __int128 s1 = (unsigned __int128) a1[0] * bi + x1; + unsigned __int128 s2 = (unsigned __int128) a2[0] * ci + x2; + __m256i bv = _mm256_set1_epi64x((long long) bi), cv = _mm256_set1_epi64x((long long) ci); + __m256i yv, zv; + + y = ((uint64_t) s1 * k1) & IFMA_MASK52; + z = ((uint64_t) s2 * k2) & IFMA_MASK52; + s1 += (unsigned __int128) m1[0] * y; + s2 += (unsigned __int128) m2[0] * z; + yv = _mm256_set1_epi64x((long long) y); + zv = _mm256_set1_epi64x((long long) z); + X0 = _mm256_madd52lo_epu64(X0, A0, bv); Y0 = _mm256_madd52lo_epu64(Y0, B0, cv); + X1 = _mm256_madd52lo_epu64(X1, A1, bv); Y1 = _mm256_madd52lo_epu64(Y1, B1, cv); + X2 = _mm256_madd52lo_epu64(X2, A2, bv); Y2 = _mm256_madd52lo_epu64(Y2, B2, cv); + X3 = _mm256_madd52lo_epu64(X3, A3, bv); Y3 = _mm256_madd52lo_epu64(Y3, B3, cv); + X4 = _mm256_madd52lo_epu64(X4, A4, bv); Y4 = _mm256_madd52lo_epu64(Y4, B4, cv); + X0 = _mm256_madd52lo_epu64(X0, L(m1, 0), yv); Y0 = _mm256_madd52lo_epu64(Y0, L(m2, 0), zv); + X1 = _mm256_madd52lo_epu64(X1, L(m1, 1), yv); Y1 = _mm256_madd52lo_epu64(Y1, L(m2, 1), zv); + X2 = _mm256_madd52lo_epu64(X2, L(m1, 2), yv); Y2 = _mm256_madd52lo_epu64(Y2, L(m2, 2), zv); + X3 = _mm256_madd52lo_epu64(X3, L(m1, 3), yv); Y3 = _mm256_madd52lo_epu64(Y3, L(m2, 3), zv); + X4 = _mm256_madd52lo_epu64(X4, L(m1, 4), yv); Y4 = _mm256_madd52lo_epu64(Y4, L(m2, 4), zv); + /* down a lane: lane 0 is a multiple of 2^52, and s carries it */ + X0 = _mm256_alignr_epi64(X1, X0, 1); Y0 = _mm256_alignr_epi64(Y1, Y0, 1); + X1 = _mm256_alignr_epi64(X2, X1, 1); Y1 = _mm256_alignr_epi64(Y2, Y1, 1); + X2 = _mm256_alignr_epi64(X3, X2, 1); Y2 = _mm256_alignr_epi64(Y3, Y2, 1); + X3 = _mm256_alignr_epi64(X4, X3, 1); Y3 = _mm256_alignr_epi64(Y4, Y3, 1); + X4 = _mm256_alignr_epi64(zero, X4, 1); Y4 = _mm256_alignr_epi64(zero, Y4, 1); + x1 = (uint64_t) _mm_cvtsi128_si64(_mm256_castsi256_si128(X0)) + (uint64_t)(s1 >> 52); + x2 = (uint64_t) _mm_cvtsi128_si64(_mm256_castsi256_si128(Y0)) + (uint64_t)(s2 >> 52); + X0 = _mm256_madd52hi_epu64(X0, A0, bv); Y0 = _mm256_madd52hi_epu64(Y0, B0, cv); + X1 = _mm256_madd52hi_epu64(X1, A1, bv); Y1 = _mm256_madd52hi_epu64(Y1, B1, cv); + X2 = _mm256_madd52hi_epu64(X2, A2, bv); Y2 = _mm256_madd52hi_epu64(Y2, B2, cv); + X3 = _mm256_madd52hi_epu64(X3, A3, bv); Y3 = _mm256_madd52hi_epu64(Y3, B3, cv); + X4 = _mm256_madd52hi_epu64(X4, A4, bv); Y4 = _mm256_madd52hi_epu64(Y4, B4, cv); + X0 = _mm256_madd52hi_epu64(X0, L(m1, 0), yv); Y0 = _mm256_madd52hi_epu64(Y0, L(m2, 0), zv); + X1 = _mm256_madd52hi_epu64(X1, L(m1, 1), yv); Y1 = _mm256_madd52hi_epu64(Y1, L(m2, 1), zv); + X2 = _mm256_madd52hi_epu64(X2, L(m1, 2), yv); Y2 = _mm256_madd52hi_epu64(Y2, L(m2, 2), zv); + X3 = _mm256_madd52hi_epu64(X3, L(m1, 3), yv); Y3 = _mm256_madd52hi_epu64(Y3, L(m2, 3), zv); + X4 = _mm256_madd52hi_epu64(X4, L(m1, 4), yv); Y4 = _mm256_madd52hi_epu64(Y4, L(m2, 4), zv); + } +#undef L + /* lane 0, with the carry it was never given, is what the scalar holds */ + X0 = _mm256_mask_set1_epi64(X0, 1, (long long) x1); + Y0 = _mm256_mask_set1_epi64(Y0, 1, (long long) x2); + ifma_norm5(r1, X0, X1, X2, X3, X4); + ifma_norm5(r2, Y0, Y1, Y2, Y3, Y4); +} + +/* 30 digits, 8 registers a number */ +IFMA_TARGET static void ifma_amm30_x2(uint64_t *r1, const uint64_t *a1, const uint64_t *b1, + const uint64_t *m1, uint64_t k1, + uint64_t *r2, const uint64_t *a2, const uint64_t *b2, + const uint64_t *m2, uint64_t k2) +{ + const __m256i zero = _mm256_setzero_si256(); +#define L(p, k) _mm256_loadu_si256((const __m256i *)((p) + 4 * (k))) + __m256i X0 = zero, X1 = zero, X2 = zero, X3 = zero, X4 = zero, X5 = zero, X6 = zero, X7 = zero, Y0 = zero, Y1 = zero, Y2 = zero, Y3 = zero, Y4 = zero, Y5 = zero, Y6 = zero, Y7 = zero; + uint64_t x1 = 0, x2 = 0; + int i; + + for (i = 0; i < 30; i++) { + uint64_t bi = b1[i], ci = b2[i], y, z; + /* lane 0 in full: the products' high halves, which are lane 1's, too */ + unsigned __int128 s1 = (unsigned __int128) a1[0] * bi + x1; + unsigned __int128 s2 = (unsigned __int128) a2[0] * ci + x2; + __m256i bv = _mm256_set1_epi64x((long long) bi), cv = _mm256_set1_epi64x((long long) ci); + __m256i yv, zv; + + y = ((uint64_t) s1 * k1) & IFMA_MASK52; + z = ((uint64_t) s2 * k2) & IFMA_MASK52; + s1 += (unsigned __int128) m1[0] * y; + s2 += (unsigned __int128) m2[0] * z; + yv = _mm256_set1_epi64x((long long) y); + zv = _mm256_set1_epi64x((long long) z); + X0 = _mm256_madd52lo_epu64(X0, L(a1, 0), bv); Y0 = _mm256_madd52lo_epu64(Y0, L(a2, 0), cv); + X1 = _mm256_madd52lo_epu64(X1, L(a1, 1), bv); Y1 = _mm256_madd52lo_epu64(Y1, L(a2, 1), cv); + X2 = _mm256_madd52lo_epu64(X2, L(a1, 2), bv); Y2 = _mm256_madd52lo_epu64(Y2, L(a2, 2), cv); + X3 = _mm256_madd52lo_epu64(X3, L(a1, 3), bv); Y3 = _mm256_madd52lo_epu64(Y3, L(a2, 3), cv); + X4 = _mm256_madd52lo_epu64(X4, L(a1, 4), bv); Y4 = _mm256_madd52lo_epu64(Y4, L(a2, 4), cv); + X5 = _mm256_madd52lo_epu64(X5, L(a1, 5), bv); Y5 = _mm256_madd52lo_epu64(Y5, L(a2, 5), cv); + X6 = _mm256_madd52lo_epu64(X6, L(a1, 6), bv); Y6 = _mm256_madd52lo_epu64(Y6, L(a2, 6), cv); + X7 = _mm256_madd52lo_epu64(X7, L(a1, 7), bv); Y7 = _mm256_madd52lo_epu64(Y7, L(a2, 7), cv); + X0 = _mm256_madd52lo_epu64(X0, L(m1, 0), yv); Y0 = _mm256_madd52lo_epu64(Y0, L(m2, 0), zv); + X1 = _mm256_madd52lo_epu64(X1, L(m1, 1), yv); Y1 = _mm256_madd52lo_epu64(Y1, L(m2, 1), zv); + X2 = _mm256_madd52lo_epu64(X2, L(m1, 2), yv); Y2 = _mm256_madd52lo_epu64(Y2, L(m2, 2), zv); + X3 = _mm256_madd52lo_epu64(X3, L(m1, 3), yv); Y3 = _mm256_madd52lo_epu64(Y3, L(m2, 3), zv); + X4 = _mm256_madd52lo_epu64(X4, L(m1, 4), yv); Y4 = _mm256_madd52lo_epu64(Y4, L(m2, 4), zv); + X5 = _mm256_madd52lo_epu64(X5, L(m1, 5), yv); Y5 = _mm256_madd52lo_epu64(Y5, L(m2, 5), zv); + X6 = _mm256_madd52lo_epu64(X6, L(m1, 6), yv); Y6 = _mm256_madd52lo_epu64(Y6, L(m2, 6), zv); + X7 = _mm256_madd52lo_epu64(X7, L(m1, 7), yv); Y7 = _mm256_madd52lo_epu64(Y7, L(m2, 7), zv); + /* down a lane: lane 0 is a multiple of 2^52, and s carries it */ + X0 = _mm256_alignr_epi64(X1, X0, 1); Y0 = _mm256_alignr_epi64(Y1, Y0, 1); + X1 = _mm256_alignr_epi64(X2, X1, 1); Y1 = _mm256_alignr_epi64(Y2, Y1, 1); + X2 = _mm256_alignr_epi64(X3, X2, 1); Y2 = _mm256_alignr_epi64(Y3, Y2, 1); + X3 = _mm256_alignr_epi64(X4, X3, 1); Y3 = _mm256_alignr_epi64(Y4, Y3, 1); + X4 = _mm256_alignr_epi64(X5, X4, 1); Y4 = _mm256_alignr_epi64(Y5, Y4, 1); + X5 = _mm256_alignr_epi64(X6, X5, 1); Y5 = _mm256_alignr_epi64(Y6, Y5, 1); + X6 = _mm256_alignr_epi64(X7, X6, 1); Y6 = _mm256_alignr_epi64(Y7, Y6, 1); + X7 = _mm256_alignr_epi64(zero, X7, 1); Y7 = _mm256_alignr_epi64(zero, Y7, 1); + x1 = (uint64_t) _mm_cvtsi128_si64(_mm256_castsi256_si128(X0)) + (uint64_t)(s1 >> 52); + x2 = (uint64_t) _mm_cvtsi128_si64(_mm256_castsi256_si128(Y0)) + (uint64_t)(s2 >> 52); + X0 = _mm256_madd52hi_epu64(X0, L(a1, 0), bv); Y0 = _mm256_madd52hi_epu64(Y0, L(a2, 0), cv); + X1 = _mm256_madd52hi_epu64(X1, L(a1, 1), bv); Y1 = _mm256_madd52hi_epu64(Y1, L(a2, 1), cv); + X2 = _mm256_madd52hi_epu64(X2, L(a1, 2), bv); Y2 = _mm256_madd52hi_epu64(Y2, L(a2, 2), cv); + X3 = _mm256_madd52hi_epu64(X3, L(a1, 3), bv); Y3 = _mm256_madd52hi_epu64(Y3, L(a2, 3), cv); + X4 = _mm256_madd52hi_epu64(X4, L(a1, 4), bv); Y4 = _mm256_madd52hi_epu64(Y4, L(a2, 4), cv); + X5 = _mm256_madd52hi_epu64(X5, L(a1, 5), bv); Y5 = _mm256_madd52hi_epu64(Y5, L(a2, 5), cv); + X6 = _mm256_madd52hi_epu64(X6, L(a1, 6), bv); Y6 = _mm256_madd52hi_epu64(Y6, L(a2, 6), cv); + X7 = _mm256_madd52hi_epu64(X7, L(a1, 7), bv); Y7 = _mm256_madd52hi_epu64(Y7, L(a2, 7), cv); + X0 = _mm256_madd52hi_epu64(X0, L(m1, 0), yv); Y0 = _mm256_madd52hi_epu64(Y0, L(m2, 0), zv); + X1 = _mm256_madd52hi_epu64(X1, L(m1, 1), yv); Y1 = _mm256_madd52hi_epu64(Y1, L(m2, 1), zv); + X2 = _mm256_madd52hi_epu64(X2, L(m1, 2), yv); Y2 = _mm256_madd52hi_epu64(Y2, L(m2, 2), zv); + X3 = _mm256_madd52hi_epu64(X3, L(m1, 3), yv); Y3 = _mm256_madd52hi_epu64(Y3, L(m2, 3), zv); + X4 = _mm256_madd52hi_epu64(X4, L(m1, 4), yv); Y4 = _mm256_madd52hi_epu64(Y4, L(m2, 4), zv); + X5 = _mm256_madd52hi_epu64(X5, L(m1, 5), yv); Y5 = _mm256_madd52hi_epu64(Y5, L(m2, 5), zv); + X6 = _mm256_madd52hi_epu64(X6, L(m1, 6), yv); Y6 = _mm256_madd52hi_epu64(Y6, L(m2, 6), zv); + X7 = _mm256_madd52hi_epu64(X7, L(m1, 7), yv); Y7 = _mm256_madd52hi_epu64(Y7, L(m2, 7), zv); + } +#undef L + /* lane 0, with the carry it was never given, is what the scalar holds */ + X0 = _mm256_mask_set1_epi64(X0, 1, (long long) x1); + Y0 = _mm256_mask_set1_epi64(Y0, 1, (long long) x2); + ifma_norm8(r1, X0, X1, X2, X3, X4, X5, X6, X7); + ifma_norm8(r2, Y0, Y1, Y2, Y3, Y4, Y5, Y6, Y7); +} + +/* 40 digits, 10 registers a number */ +IFMA_TARGET static void ifma_amm40_x2(uint64_t *r1, const uint64_t *a1, const uint64_t *b1, + const uint64_t *m1, uint64_t k1, + uint64_t *r2, const uint64_t *a2, const uint64_t *b2, + const uint64_t *m2, uint64_t k2) +{ + const __m256i zero = _mm256_setzero_si256(); +#define L(p, k) _mm256_loadu_si256((const __m256i *)((p) + 4 * (k))) + __m256i X0 = zero, X1 = zero, X2 = zero, X3 = zero, X4 = zero, X5 = zero, X6 = zero, X7 = zero, X8 = zero, X9 = zero, Y0 = zero, Y1 = zero, Y2 = zero, Y3 = zero, Y4 = zero, Y5 = zero, Y6 = zero, Y7 = zero, Y8 = zero, Y9 = zero; + uint64_t x1 = 0, x2 = 0; + int i; + + for (i = 0; i < 40; i++) { + uint64_t bi = b1[i], ci = b2[i], y, z; + /* lane 0 in full: the products' high halves, which are lane 1's, too */ + unsigned __int128 s1 = (unsigned __int128) a1[0] * bi + x1; + unsigned __int128 s2 = (unsigned __int128) a2[0] * ci + x2; + __m256i bv = _mm256_set1_epi64x((long long) bi), cv = _mm256_set1_epi64x((long long) ci); + __m256i yv, zv; + + y = ((uint64_t) s1 * k1) & IFMA_MASK52; + z = ((uint64_t) s2 * k2) & IFMA_MASK52; + s1 += (unsigned __int128) m1[0] * y; + s2 += (unsigned __int128) m2[0] * z; + yv = _mm256_set1_epi64x((long long) y); + zv = _mm256_set1_epi64x((long long) z); + X0 = _mm256_madd52lo_epu64(X0, L(a1, 0), bv); Y0 = _mm256_madd52lo_epu64(Y0, L(a2, 0), cv); + X1 = _mm256_madd52lo_epu64(X1, L(a1, 1), bv); Y1 = _mm256_madd52lo_epu64(Y1, L(a2, 1), cv); + X2 = _mm256_madd52lo_epu64(X2, L(a1, 2), bv); Y2 = _mm256_madd52lo_epu64(Y2, L(a2, 2), cv); + X3 = _mm256_madd52lo_epu64(X3, L(a1, 3), bv); Y3 = _mm256_madd52lo_epu64(Y3, L(a2, 3), cv); + X4 = _mm256_madd52lo_epu64(X4, L(a1, 4), bv); Y4 = _mm256_madd52lo_epu64(Y4, L(a2, 4), cv); + X5 = _mm256_madd52lo_epu64(X5, L(a1, 5), bv); Y5 = _mm256_madd52lo_epu64(Y5, L(a2, 5), cv); + X6 = _mm256_madd52lo_epu64(X6, L(a1, 6), bv); Y6 = _mm256_madd52lo_epu64(Y6, L(a2, 6), cv); + X7 = _mm256_madd52lo_epu64(X7, L(a1, 7), bv); Y7 = _mm256_madd52lo_epu64(Y7, L(a2, 7), cv); + X8 = _mm256_madd52lo_epu64(X8, L(a1, 8), bv); Y8 = _mm256_madd52lo_epu64(Y8, L(a2, 8), cv); + X9 = _mm256_madd52lo_epu64(X9, L(a1, 9), bv); Y9 = _mm256_madd52lo_epu64(Y9, L(a2, 9), cv); + X0 = _mm256_madd52lo_epu64(X0, L(m1, 0), yv); Y0 = _mm256_madd52lo_epu64(Y0, L(m2, 0), zv); + X1 = _mm256_madd52lo_epu64(X1, L(m1, 1), yv); Y1 = _mm256_madd52lo_epu64(Y1, L(m2, 1), zv); + X2 = _mm256_madd52lo_epu64(X2, L(m1, 2), yv); Y2 = _mm256_madd52lo_epu64(Y2, L(m2, 2), zv); + X3 = _mm256_madd52lo_epu64(X3, L(m1, 3), yv); Y3 = _mm256_madd52lo_epu64(Y3, L(m2, 3), zv); + X4 = _mm256_madd52lo_epu64(X4, L(m1, 4), yv); Y4 = _mm256_madd52lo_epu64(Y4, L(m2, 4), zv); + X5 = _mm256_madd52lo_epu64(X5, L(m1, 5), yv); Y5 = _mm256_madd52lo_epu64(Y5, L(m2, 5), zv); + X6 = _mm256_madd52lo_epu64(X6, L(m1, 6), yv); Y6 = _mm256_madd52lo_epu64(Y6, L(m2, 6), zv); + X7 = _mm256_madd52lo_epu64(X7, L(m1, 7), yv); Y7 = _mm256_madd52lo_epu64(Y7, L(m2, 7), zv); + X8 = _mm256_madd52lo_epu64(X8, L(m1, 8), yv); Y8 = _mm256_madd52lo_epu64(Y8, L(m2, 8), zv); + X9 = _mm256_madd52lo_epu64(X9, L(m1, 9), yv); Y9 = _mm256_madd52lo_epu64(Y9, L(m2, 9), zv); + /* down a lane: lane 0 is a multiple of 2^52, and s carries it */ + X0 = _mm256_alignr_epi64(X1, X0, 1); Y0 = _mm256_alignr_epi64(Y1, Y0, 1); + X1 = _mm256_alignr_epi64(X2, X1, 1); Y1 = _mm256_alignr_epi64(Y2, Y1, 1); + X2 = _mm256_alignr_epi64(X3, X2, 1); Y2 = _mm256_alignr_epi64(Y3, Y2, 1); + X3 = _mm256_alignr_epi64(X4, X3, 1); Y3 = _mm256_alignr_epi64(Y4, Y3, 1); + X4 = _mm256_alignr_epi64(X5, X4, 1); Y4 = _mm256_alignr_epi64(Y5, Y4, 1); + X5 = _mm256_alignr_epi64(X6, X5, 1); Y5 = _mm256_alignr_epi64(Y6, Y5, 1); + X6 = _mm256_alignr_epi64(X7, X6, 1); Y6 = _mm256_alignr_epi64(Y7, Y6, 1); + X7 = _mm256_alignr_epi64(X8, X7, 1); Y7 = _mm256_alignr_epi64(Y8, Y7, 1); + X8 = _mm256_alignr_epi64(X9, X8, 1); Y8 = _mm256_alignr_epi64(Y9, Y8, 1); + X9 = _mm256_alignr_epi64(zero, X9, 1); Y9 = _mm256_alignr_epi64(zero, Y9, 1); + x1 = (uint64_t) _mm_cvtsi128_si64(_mm256_castsi256_si128(X0)) + (uint64_t)(s1 >> 52); + x2 = (uint64_t) _mm_cvtsi128_si64(_mm256_castsi256_si128(Y0)) + (uint64_t)(s2 >> 52); + X0 = _mm256_madd52hi_epu64(X0, L(a1, 0), bv); Y0 = _mm256_madd52hi_epu64(Y0, L(a2, 0), cv); + X1 = _mm256_madd52hi_epu64(X1, L(a1, 1), bv); Y1 = _mm256_madd52hi_epu64(Y1, L(a2, 1), cv); + X2 = _mm256_madd52hi_epu64(X2, L(a1, 2), bv); Y2 = _mm256_madd52hi_epu64(Y2, L(a2, 2), cv); + X3 = _mm256_madd52hi_epu64(X3, L(a1, 3), bv); Y3 = _mm256_madd52hi_epu64(Y3, L(a2, 3), cv); + X4 = _mm256_madd52hi_epu64(X4, L(a1, 4), bv); Y4 = _mm256_madd52hi_epu64(Y4, L(a2, 4), cv); + X5 = _mm256_madd52hi_epu64(X5, L(a1, 5), bv); Y5 = _mm256_madd52hi_epu64(Y5, L(a2, 5), cv); + X6 = _mm256_madd52hi_epu64(X6, L(a1, 6), bv); Y6 = _mm256_madd52hi_epu64(Y6, L(a2, 6), cv); + X7 = _mm256_madd52hi_epu64(X7, L(a1, 7), bv); Y7 = _mm256_madd52hi_epu64(Y7, L(a2, 7), cv); + X8 = _mm256_madd52hi_epu64(X8, L(a1, 8), bv); Y8 = _mm256_madd52hi_epu64(Y8, L(a2, 8), cv); + X9 = _mm256_madd52hi_epu64(X9, L(a1, 9), bv); Y9 = _mm256_madd52hi_epu64(Y9, L(a2, 9), cv); + X0 = _mm256_madd52hi_epu64(X0, L(m1, 0), yv); Y0 = _mm256_madd52hi_epu64(Y0, L(m2, 0), zv); + X1 = _mm256_madd52hi_epu64(X1, L(m1, 1), yv); Y1 = _mm256_madd52hi_epu64(Y1, L(m2, 1), zv); + X2 = _mm256_madd52hi_epu64(X2, L(m1, 2), yv); Y2 = _mm256_madd52hi_epu64(Y2, L(m2, 2), zv); + X3 = _mm256_madd52hi_epu64(X3, L(m1, 3), yv); Y3 = _mm256_madd52hi_epu64(Y3, L(m2, 3), zv); + X4 = _mm256_madd52hi_epu64(X4, L(m1, 4), yv); Y4 = _mm256_madd52hi_epu64(Y4, L(m2, 4), zv); + X5 = _mm256_madd52hi_epu64(X5, L(m1, 5), yv); Y5 = _mm256_madd52hi_epu64(Y5, L(m2, 5), zv); + X6 = _mm256_madd52hi_epu64(X6, L(m1, 6), yv); Y6 = _mm256_madd52hi_epu64(Y6, L(m2, 6), zv); + X7 = _mm256_madd52hi_epu64(X7, L(m1, 7), yv); Y7 = _mm256_madd52hi_epu64(Y7, L(m2, 7), zv); + X8 = _mm256_madd52hi_epu64(X8, L(m1, 8), yv); Y8 = _mm256_madd52hi_epu64(Y8, L(m2, 8), zv); + X9 = _mm256_madd52hi_epu64(X9, L(m1, 9), yv); Y9 = _mm256_madd52hi_epu64(Y9, L(m2, 9), zv); + } +#undef L + /* lane 0, with the carry it was never given, is what the scalar holds */ + X0 = _mm256_mask_set1_epi64(X0, 1, (long long) x1); + Y0 = _mm256_mask_set1_epi64(Y0, 1, (long long) x2); + ifma_norm10(r1, X0, X1, X2, X3, X4, X5, X6, X7, X8, X9); + ifma_norm10(r2, Y0, Y1, Y2, Y3, Y4, Y5, Y6, Y7, Y8, Y9); +} + +#endif diff --git a/cbits/crypton_powm_ifma.py b/cbits/crypton_powm_ifma.py new file mode 100644 --- /dev/null +++ b/cbits/crypton_powm_ifma.py @@ -0,0 +1,178 @@ +#!/usr/bin/env python3 +"""Generate cbits/crypton_powm_ifma.h: two almost-Montgomery multiplications +at once in radix 2^52 with AVX-512 IFMA on 256-bit registers, at 20, 30 and +40 digits. Usage: python3 cbits/crypton_powm_ifma.py > cbits/crypton_powm_ifma.h + +Gueron and Krasnov's method, a digit of b at a time. Lane 0 is followed in +full in a scalar, so it is read before the high halves go in and they stay +off the chain the next digit waits on. +""" + +SIZES = (20, 30, 40) + +HEADER = """\ +/* Generated by cbits/crypton_powm_ifma.py; do not edit. + * + * Almost-Montgomery multiplication in radix 2^52, two at once: r = a b + * 2^-52D mod m, under 2m for a and b under 2m. Digits are 52 bits in 64-bit + * words, each number padded with zero words to a whole number of 256-bit + * registers. m is read from memory; at 20 digits a is held in registers, + * above that it is read from memory too. */ +#ifndef CRYPTON_POWM_IFMA_H +#define CRYPTON_POWM_IFMA_H + +#include +#include + +#define IFMA_MASK52 ((UINT64_C(1) << 52) - 1) +#define IFMA_TARGET __attribute__((target("avx2,avx512f,avx512vl,avx512ifma"))) +""" + +FOOTER = """\ +#endif +""" + + +def regs(d): + return (d + 3) // 4 + + +def gen_norm(R): + o = [] + w = o.append + xs = [f"X{j}" for j in range(R)] + w(f"/* {4 * R} lanes to 52-bit digits: what is above 52 bits moves up a lane,") + w(f" * and the one-bit carries that can then remain are resolved together */") + w(f"IFMA_TARGET static inline void ifma_norm{R}(uint64_t *r, " + + ", ".join(f"__m256i {x}" for x in xs) + ")") + w("{") + w("\tconst __m256i mk = _mm256_set1_epi64x((long long) IFMA_MASK52), one = _mm256_set1_epi64x(1);") + w("\tconst __m256i zero = _mm256_setzero_si256();") + w("\tuint64_t g = 0, p = 0, c;") + for j in range(R): + w(f"\t__m256i C{j} = _mm256_srli_epi64(X{j}, 52);") + for j in range(R): + prev = f"C{j - 1}" if j else "zero" + w(f"\tX{j} = _mm256_add_epi64(_mm256_and_si256(X{j}, mk), _mm256_alignr_epi64(C{j}, {prev}, 3));") + for j in range(R): + w(f"\tg |= (uint64_t) _mm256_cmpgt_epu64_mask(X{j}, mk) << {4 * j};") + w(f"\tp |= (uint64_t) _mm256_cmpeq_epu64_mask(X{j}, mk) << {4 * j};") + w("\tc = ((g << 1) + p) ^ p;") + for j in range(R): + w(f"\tX{j} = _mm256_and_si256(_mm256_mask_add_epi64(X{j}, (__mmask8)(c >> {4 * j} & 15), X{j}, one), mk);") + w(f"\t_mm256_storeu_si256((__m256i *)(r + {4 * j}), X{j});") + w("}") + w("") + return "\n".join(o) + + +def gen_amm(d): + R = regs(d) + in_regs = R <= 5 + o = [] + w = o.append + w(f"/* {d} digits, {R} registers a number */") + w(f"IFMA_TARGET static void ifma_amm{d}_x2(uint64_t *r1, const uint64_t *a1, const uint64_t *b1,") + w(f" const uint64_t *m1, uint64_t k1,") + w(f" uint64_t *r2, const uint64_t *a2, const uint64_t *b2,") + w(f" const uint64_t *m2, uint64_t k2)") + w("{") + w("\tconst __m256i zero = _mm256_setzero_si256();") + w("#define L(p, k) _mm256_loadu_si256((const __m256i *)((p) + 4 * (k)))") + X = [f"X{j}" for j in range(R)] + Y = [f"Y{j}" for j in range(R)] + w("\t__m256i " + ", ".join(f"{v} = zero" for v in X + Y) + ";") + if in_regs: + w("\t__m256i " + ", ".join(f"A{j} = L(a1, {j})" for j in range(R)) + ";") + w("\t__m256i " + ", ".join(f"B{j} = L(a2, {j})" for j in range(R)) + ";") + A = [f"A{j}" for j in range(R)] + B = [f"B{j}" for j in range(R)] + else: + A = [f"L(a1, {j})" for j in range(R)] + B = [f"L(a2, {j})" for j in range(R)] + M = [f"L(m1, {j})" for j in range(R)] + N = [f"L(m2, {j})" for j in range(R)] + w("\tuint64_t x1 = 0, x2 = 0;") + w("\tint i;") + w("") + w(f"\tfor (i = 0; i < {d}; i++) {{") + w("\t\tuint64_t bi = b1[i], ci = b2[i], y, z;") + w("\t\t/* lane 0 in full: the products' high halves, which are lane 1's, too */") + w("\t\tunsigned __int128 s1 = (unsigned __int128) a1[0] * bi + x1;") + w("\t\tunsigned __int128 s2 = (unsigned __int128) a2[0] * ci + x2;") + w("\t\t__m256i bv = _mm256_set1_epi64x((long long) bi), cv = _mm256_set1_epi64x((long long) ci);") + w("\t\t__m256i yv, zv;") + w("") + w("\t\ty = ((uint64_t) s1 * k1) & IFMA_MASK52;") + w("\t\tz = ((uint64_t) s2 * k2) & IFMA_MASK52;") + w("\t\ts1 += (unsigned __int128) m1[0] * y;") + w("\t\ts2 += (unsigned __int128) m2[0] * z;") + w("\t\tyv = _mm256_set1_epi64x((long long) y);") + w("\t\tzv = _mm256_set1_epi64x((long long) z);") + for j in range(R): + w(f"\t\tX{j} = _mm256_madd52lo_epu64(X{j}, {A[j]}, bv); Y{j} = _mm256_madd52lo_epu64(Y{j}, {B[j]}, cv);") + for j in range(R): + w(f"\t\tX{j} = _mm256_madd52lo_epu64(X{j}, {M[j]}, yv); Y{j} = _mm256_madd52lo_epu64(Y{j}, {N[j]}, zv);") + w("\t\t/* down a lane: lane 0 is a multiple of 2^52, and s carries it */") + for j in range(R): + nx = f"X{j + 1}" if j + 1 < R else "zero" + ny = f"Y{j + 1}" if j + 1 < R else "zero" + w(f"\t\tX{j} = _mm256_alignr_epi64({nx}, X{j}, 1); Y{j} = _mm256_alignr_epi64({ny}, Y{j}, 1);") + w("\t\tx1 = (uint64_t) _mm_cvtsi128_si64(_mm256_castsi256_si128(X0)) + (uint64_t)(s1 >> 52);") + w("\t\tx2 = (uint64_t) _mm_cvtsi128_si64(_mm256_castsi256_si128(Y0)) + (uint64_t)(s2 >> 52);") + for j in range(R): + w(f"\t\tX{j} = _mm256_madd52hi_epu64(X{j}, {A[j]}, bv); Y{j} = _mm256_madd52hi_epu64(Y{j}, {B[j]}, cv);") + for j in range(R): + w(f"\t\tX{j} = _mm256_madd52hi_epu64(X{j}, {M[j]}, yv); Y{j} = _mm256_madd52hi_epu64(Y{j}, {N[j]}, zv);") + w("\t}") + w("#undef L") + w("\t/* lane 0, with the carry it was never given, is what the scalar holds */") + w("\tX0 = _mm256_mask_set1_epi64(X0, 1, (long long) x1);") + w("\tY0 = _mm256_mask_set1_epi64(Y0, 1, (long long) x2);") + w(f"\tifma_norm{R}(r1, " + ", ".join(X) + ");") + w(f"\tifma_norm{R}(r2, " + ", ".join(Y) + ");") + w("}") + w("") + return "\n".join(o) + + +def gen_sel(R): + o = [] + w = o.append + w(f"/* both tables' entries k1 and k2 of 32, every entry read and a mask keeping") + w(f" * one; entries are {4 * R} words apart */") + w(f"IFMA_TARGET static void ifma_sel{R}(uint64_t *o1, const uint64_t *t1, uint64_t k1,") + w(f" uint64_t *o2, const uint64_t *t2, uint64_t k2)") + w("{") + w("\tconst __m256i one = _mm256_set1_epi64x(1);") + w("\t__m256i v1 = _mm256_set1_epi64x((long long) k1), v2 = _mm256_set1_epi64x((long long) k2);") + w("\t__m256i kv = _mm256_setzero_si256();") + w("\t__m256i " + ", ".join(f"P{j} = kv, Q{j} = kv" for j in range(R)) + ";") + w("\tint k;") + w("\tfor (k = 0; k < 32; k++) {") + w("\t\t__mmask8 s1 = _mm256_cmpeq_epi64_mask(v1, kv), s2 = _mm256_cmpeq_epi64_mask(v2, kv);") + w(f"\t\tconst uint64_t *e1 = t1 + {4 * R} * k, *e2 = t2 + {4 * R} * k;") + for j in range(R): + w(f"\t\tP{j} = _mm256_mask_loadu_epi64(P{j}, s1, e1 + {4 * j}); Q{j} = _mm256_mask_loadu_epi64(Q{j}, s2, e2 + {4 * j});") + w("\t\tkv = _mm256_add_epi64(kv, one);") + w("\t}") + for j in range(R): + w(f"\t_mm256_storeu_si256((__m256i *)(o1 + {4 * j}), P{j}); _mm256_storeu_si256((__m256i *)(o2 + {4 * j}), Q{j});") + w("}") + w("") + return "\n".join(o) + + +def main(): + print(HEADER) + rs = sorted({regs(d) for d in SIZES}) + for R in rs: + print(gen_norm(R)) + print(gen_sel(R)) + for d in SIZES: + print(gen_amm(d)) + print(FOOTER, end="") + + +if __name__ == "__main__": + main() diff --git a/cbits/crypton_rdrand.c b/cbits/crypton_rdrand.c deleted file mode 100644 --- a/cbits/crypton_rdrand.c +++ /dev/null @@ -1,126 +0,0 @@ -/* - * Copyright (C) Thomas DuBuisson - * Copyright (C) 2013 Vincent Hanquez - * - * All rights reserved. - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions - * are met: - * 1. Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * 2. Redistributions in binary form must reproduce the above copyright - * notice, this list of conditions and the following disclaimer in the - * documentation and/or other materials provided with the distribution. - * 3. Neither the name of the author nor the names of his contributors - * may be used to endorse or promote products derived from this software - * without specific prior written permission. - * - * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND - * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE - * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE - * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL - * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS - * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) - * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT - * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY - * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF - * SUCH DAMAGE. - */ - -#include -#include -#include -#include - -int crypton_cpu_has_rdrand() -{ - uint32_t ax,bx,cx,dx,func=1; -#if defined(__PIC__) && defined(__i386__) - __asm__ volatile ("mov %%ebx, %%edi;" "cpuid;" "xchgl %%ebx, %%edi;" - : "=a" (ax), "=D" (bx), "=c" (cx), "=d" (dx) : "a" (func)); -#else - __asm__ volatile ("cpuid": "=a" (ax), "=b" (bx), "=c" (cx), "=d" (dx) : "a" (func)); -#endif - return (cx & 0x40000000); -} - -/* inline encoding of 'rdrand %rax' to cover old binutils - * - no inputs - * - 'cc' to the clobber list as we modify condition code. - * - output of rdrand in rax and have a 8 bit error condition - */ -#define inline_rdrand_rax(val, err) \ - asm(".byte 0x48,0x0f,0xc7,0xf0; setc %1" \ - : "=a" (val), "=q" (err) \ - : \ - : "cc") - -/* inline encoding of 'rdrand %eax' to cover old binutils - * - no inputs - * - 'cc' to the clobber list as we modify condition code. - * - output of rdrand in eax and have a 8 bit error condition - */ -#define inline_rdrand_eax(val, err) \ - asm(".byte 0x0f,0xc7,0xf0; setc %1" \ - : "=a" (val), "=q" (err) \ - : \ - : "cc") - -#ifdef __x86_64__ -# define RDRAND_SZ 8 -# define RDRAND_T uint64_t -#define inline_rdrand(val, err) err = crypton_rdrand_step(&val) -#else -# define RDRAND_SZ 4 -# define RDRAND_T uint32_t -#define inline_rdrand(val, err) err = crypton_rdrand_step(&val) -#endif - -/* sadly many people are still using an old binutils, - * leading to report that instruction is not recognized. - */ -#if 1 -/* Returns 1 on success */ -static inline int crypton_rdrand_step(RDRAND_T *buffer) -{ - unsigned char err; - asm volatile ("rdrand %0; setc %1" : "=r" (*buffer), "=qm" (err)); - return (int) err; -} -#endif - -/* Returns the number of bytes successfully generated */ -int crypton_get_rand_bytes(uint8_t *buffer, size_t len) -{ - RDRAND_T tmp; - int aligned = (intptr_t) buffer % RDRAND_SZ; - int orig_len = len; - int to_alignment = RDRAND_SZ - aligned; - uint8_t ok; - - if (aligned != 0) { - inline_rdrand(tmp, ok); - if (!ok) - return 0; - memcpy(buffer, (uint8_t *) &tmp, to_alignment); - buffer += to_alignment; - len -= to_alignment; - } - - for (; len >= RDRAND_SZ; buffer += RDRAND_SZ, len -= RDRAND_SZ) { - inline_rdrand(tmp, ok); - if (!ok) - return (orig_len - len); - *((RDRAND_T *) buffer) = tmp; - } - - if (len > 0) { - inline_rdrand(tmp, ok); - if (!ok) - return (orig_len - len); - memcpy(buffer, (uint8_t *) &tmp, len); - } - return orig_len; -} diff --git a/cbits/crypton_sysdrg.c b/cbits/crypton_sysdrg.c new file mode 100644 --- /dev/null +++ b/cbits/crypton_sysdrg.c @@ -0,0 +1,388 @@ +/* + * The generator behind MonadRandom IO. + * + * A ChaCha20 DRBG per operating system thread, seeded from a process-wide + * DRBG, which is itself seeded from the system entropy pool. This is the + * shape RFC 9180's neighbours and the other libraries have settled on, and + * it was asked for in #298. + * + * Per operating system thread and not per Haskell thread: a forkIO thread + * moves between capabilities, so state kept against it would be shared by + * threads running at the same time. That is why the state is here and + * reached through pthread_getspecific rather than held in Haskell. + * + * Three things force a reseed: + * + * - a thread has produced CRYPTON_THREAD_RESEED bytes, + * - the process DRBG has issued CRYPTON_GLOBAL_RESEED bytes of seed, + * - the process has forked. + * + * The last one is the one that bites. A child inherits its parent's state + * and would otherwise produce the same stream; the generation counter below + * is bumped in the child by a pthread_atfork handler, and every generator + * compares against it before it answers. + */ + +#include +#include +#include + +#include "crypton_chacha.h" +#include "crypton_sha512.h" + +#ifdef _WIN32 +#include +#else +#include +#include +#endif + +/* from crypton_sysrandom.c */ +int crypton_sysrandom_available(void); +int crypton_sysrandom_bytes(uint8_t *buf, int len); + +#define CHACHA_ROUNDS 20 +#define SEED_KEY 32 +#define SEED_IV 8 +#define SEED_LEN (SEED_KEY + SEED_IV) + +#define CRYPTON_THREAD_RESEED (1u << 20) +#define CRYPTON_GLOBAL_RESEED (1u << 20) + +typedef struct { + crypton_chacha_context ctx; + uint64_t used; + uint32_t generation; + int seeded; +} drg_t; + +static drg_t global_drg; +static volatile uint32_t fork_generation = 0; + +#ifdef _WIN32 +static CRITICAL_SECTION global_lock; +/* Fls and not Tls: TlsAlloc has no destructor, so the state of every thread + * that ever drew a byte would be left allocated and unscrubbed when the + * thread ended. FlsAlloc takes the callback that pthread_key_create does. */ +static DWORD thread_slot; +static INIT_ONCE init_once = INIT_ONCE_STATIC_INIT; +#else +static pthread_mutex_t global_lock = PTHREAD_MUTEX_INITIALIZER; +static pthread_key_t thread_slot; +static pthread_once_t init_once = PTHREAD_ONCE_INIT; +#endif + +static void scrub(void *p, size_t n) +{ + volatile uint8_t *q = (volatile uint8_t *) p; + while (n--) *q++ = 0; +} + +/* + * Seed material for the process DRBG. + * + * What the system gives goes through SHA-512 rather than into the key + * directly, so that the key is a fixed size whatever the call returns, and + * so that a second source could be added without any of this changing shape. + * There is one source today and the note below says why. + */ +static int seed_from_system(uint8_t out[SEED_LEN]) +{ + struct sha512_ctx h; + uint8_t buf[64]; + uint8_t digest[64]; + int got; + + crypton_sha512_init(&h); + + /* + * The system call only. Where there is none -- an old kernel, a BSD + * this does not know -- seeding fails and the caller keeps the path + * it has, rather than this file growing a second copy of the device + * reading that Crypto.Random.Entropy.Unix already does. + * + * And RDRAND is not mixed in beside it, which it was until + * @vdukhovni pointed out that it cannot add anything: every system + * this code runs on already feeds RDRAND into the pool the call below + * draws from. Linux does that whatever random.trust_cpu says -- that + * setting decides whether the contribution is *credited*, not whether + * it is mixed -- so the instruction's output is in this buffer + * already. + * + * The argument for keeping it was defence in depth against that pool + * having gone wrong. It does not survive the above: a pool that has + * gone wrong has gone wrong with RDRAND already in it, so asking the + * instruction a second time covers only the case where the kernel's + * mixing is broken and the instruction is not. That is not worth a + * second code path, a flag, and the standing invitation to read this + * as a second source when it is the same one twice. + */ + got = crypton_sysrandom_available() ? crypton_sysrandom_bytes(buf, 64) : 0; + if (got <= 0) { + /* Nothing else here is a seed on its own, so there is nothing to + * go on with: return before drawing anything that would only be + * thrown away. */ + scrub(&h, sizeof h); + scrub(buf, sizeof buf); + return 0; + } + crypton_sha512_update(&h, buf, (uint32_t) got); + + + crypton_sha512_finalize(&h, digest); + memcpy(out, digest, SEED_LEN); + + scrub(&h, sizeof h); + scrub(buf, sizeof buf); + scrub(digest, sizeof digest); + return 1; +} + +static void drg_seed(drg_t *d, const uint8_t seed[SEED_LEN]) +{ + crypton_chacha_init(&d->ctx, CHACHA_ROUNDS, SEED_KEY, seed, + SEED_IV, seed + SEED_KEY); + d->used = 0; + d->generation = fork_generation; + d->seeded = 1; +} + +/* + * Forget the key that produced the bytes just handed out. + * + * Without this the key stands until the next reseed, and anyone who reads a + * generator's state can wind the counter back and reproduce everything it + * has issued since -- up to CRYPTON_THREAD_RESEED bytes that were meant to + * be secret. Taking the next forty bytes of keystream as the new key and + * nonce, and dropping the old ones, puts that out of reach one step after + * it is issued: ChaCha20 does not run backwards, and the key that would + * have been needed is gone. It is what arc4random does. + * + * crypton_chacha_init memsets the whole context, so the counter returns to + * zero and the tail of a part-used block goes with the old key rather than + * being handed out under the new one. + * + * d->used is not touched. It counts what callers were given, which is what + * the reseed interval is written in terms of; these forty bytes are the + * cost of the rekey and not an answer to anybody. + */ +static void drg_rekey(drg_t *d) +{ + uint8_t next[SEED_LEN]; + + crypton_chacha_generate(next, &d->ctx, SEED_LEN); + crypton_chacha_init(&d->ctx, CHACHA_ROUNDS, SEED_KEY, next, + SEED_IV, next + SEED_KEY); + scrub(next, sizeof next); +} + +static int drg_stale(const drg_t *d, uint64_t limit) +{ + return !d->seeded || d->used >= limit || d->generation != fork_generation; +} + +/* Bytes from the process DRBG, which is only ever asked for seed material. */ +static int global_bytes(uint8_t *out, uint32_t len) +{ + int ok = 1; + +#ifdef _WIN32 + EnterCriticalSection(&global_lock); +#else + pthread_mutex_lock(&global_lock); +#endif + if (drg_stale(&global_drg, CRYPTON_GLOBAL_RESEED)) { + uint8_t seed[SEED_LEN]; + ok = seed_from_system(seed); + if (ok) + drg_seed(&global_drg, seed); + scrub(seed, sizeof seed); + } + if (ok) { + crypton_chacha_generate(out, &global_drg.ctx, len); + global_drg.used += len; + drg_rekey(&global_drg); + } +#ifdef _WIN32 + LeaveCriticalSection(&global_lock); +#else + pthread_mutex_unlock(&global_lock); +#endif + return ok; +} + +#ifdef _WIN32 +static void WINAPI thread_free(void *p) +#else +static void thread_free(void *p) +#endif +{ + if (p) { + scrub(p, sizeof(drg_t)); + free(p); + } +} + +#ifndef _WIN32 +/* All three handlers, not just the child's. The child's first draw has to + * reseed -- the generation has changed -- and reseeding takes global_lock. + * A fork made while another thread held it would hand the child a mutex + * locked by a thread that did not come across, and the child would wait on + * it for ever. So the lock is taken before the fork and released on both + * sides of it, which is the state the child needs it in. */ +static void before_fork(void) +{ + pthread_mutex_lock(&global_lock); +} + +static void after_fork_in_parent(void) +{ + pthread_mutex_unlock(&global_lock); +} + +static void after_fork_in_child(void) +{ + pthread_mutex_unlock(&global_lock); + fork_generation++; +} +#endif + +#ifdef CRYPTON_SYSDRG_TESTING +/* For cbits/tests/sysdrg and nothing else: hold and release the process + * generator's lock, so that a fork can be made to happen while another + * thread holds it. Nothing outside that test declares these, and the + * library is never built with this defined. */ +void crypton_sysdrg_test_lock(void); +void crypton_sysdrg_test_unlock(void); + +static drg_t *this_thread(void); + +/* The calling thread's ChaCha key, which is d[4..11] of the state. A test + * uses it to ask whether the key that produced a draw is still there + * afterwards; see "a draw replaces the key that made it" in + * cbits/tests/sysdrg. */ +void crypton_sysdrg_test_key(uint8_t out[32]); + +void crypton_sysdrg_test_key(uint8_t out[32]) +{ + drg_t *d = this_thread(); + int i; + + if (!d) + return; + for (i = 0; i < 8; i++) { + uint32_t w = d->ctx.st.d[4 + i]; + out[i * 4 + 0] = (uint8_t) (w); + out[i * 4 + 1] = (uint8_t) (w >> 8); + out[i * 4 + 2] = (uint8_t) (w >> 16); + out[i * 4 + 3] = (uint8_t) (w >> 24); + } +} + +void crypton_sysdrg_test_lock(void) +{ +#ifndef _WIN32 + pthread_mutex_lock(&global_lock); +#endif +} + +void crypton_sysdrg_test_unlock(void) +{ +#ifndef _WIN32 + pthread_mutex_unlock(&global_lock); +#endif +} +#endif + +#ifdef _WIN32 +static BOOL CALLBACK init_slot(PINIT_ONCE o, PVOID p, PVOID *c) +{ + (void) o; (void) p; (void) c; + InitializeCriticalSection(&global_lock); + thread_slot = FlsAlloc(thread_free); + return TRUE; +} +#else +static void init_slot(void) +{ + pthread_key_create(&thread_slot, thread_free); + pthread_atfork(before_fork, after_fork_in_parent, after_fork_in_child); +} +#endif + +static drg_t *this_thread(void) +{ + drg_t *d; + +#ifdef _WIN32 + InitOnceExecuteOnce(&init_once, init_slot, NULL, NULL); + if (thread_slot == FLS_OUT_OF_INDEXES) + return NULL; + d = (drg_t *) FlsGetValue(thread_slot); +#else + pthread_once(&init_once, init_slot); + d = (drg_t *) pthread_getspecific(thread_slot); +#endif + if (!d) { + d = (drg_t *) calloc(1, sizeof(drg_t)); + if (!d) + return NULL; +#ifdef _WIN32 + if (!FlsSetValue(thread_slot, d)) { + free(d); + return NULL; + } +#else + if (pthread_setspecific(thread_slot, d) != 0) { + free(d); + return NULL; + } +#endif + } + return d; +} + +/* Returns the number of bytes written, which is len unless there was no + * seed to be had -- and then it is 0, so that the caller can say so rather + * than hand back a buffer it cannot vouch for. */ +int crypton_sysdrg_bytes(uint8_t *out, int len) +{ + drg_t *d; + + if (len < 0) + return 0; + d = this_thread(); + if (!d) + return 0; + + if (drg_stale(d, CRYPTON_THREAD_RESEED)) { + uint8_t seed[SEED_LEN]; + int ok = global_bytes(seed, SEED_LEN); + if (ok) + drg_seed(d, seed); + scrub(seed, sizeof seed); + if (!ok) + return 0; + } + + crypton_chacha_generate(out, &d->ctx, (uint32_t) len); + d->used += (uint64_t) len; + drg_rekey(d); + return len; +} + +/* For the tests: how many times the process has been seen to fork. */ +uint32_t crypton_sysdrg_generation(void) +{ + return fork_generation; +} + +/* For the tests: bytes this thread's generator has produced since it was + * last seeded. A generator shared between threads would carry the first + * thread's count into the second; a per-thread one starts again at zero, + * and that is the only way from outside to tell the two apart. */ +uint64_t crypton_sysdrg_thread_used(void) +{ + drg_t *d = this_thread(); + return d ? d->used : 0; +} diff --git a/cbits/crypton_sysrandom.c b/cbits/crypton_sysrandom.c new file mode 100644 --- /dev/null +++ b/cbits/crypton_sysrandom.c @@ -0,0 +1,119 @@ +/* + * The kernel's own random number generator, reached without a file + * descriptor: getrandom(2) on Linux and FreeBSD, getentropy(3) on the + * systems that have that instead. + * + * This is preferred over reading /dev/urandom because it needs no path and + * no descriptor, so it still works where /dev is not mounted or not + * populated -- a minimal container, a chroot, a sandbox -- and it cannot be + * defeated by exhausting the descriptor table. + * + * Availability is decided at run time as well as at compile time: a binary + * built against headers that declare getrandom can still run on a kernel + * that does not implement it, and says so with ENOSYS. + */ + +#include +#include +#include + +#if defined(__linux__) +#include +#include +#ifdef SYS_getrandom +#define CRYPTON_SYSRANDOM_GETRANDOM 1 +#endif +#elif defined(__FreeBSD__) +#include +#if __FreeBSD_version >= 1200000 +#include +#define CRYPTON_SYSRANDOM_GETRANDOM 1 +#endif +#elif defined(__APPLE__) +/* getentropy(3) is declared in sys/random.h on Darwin, since 10.12 -- but + * macOS only. Crypto.Random says iOS does not allow it, which is why that + * platform builds with INSECURE_ENTROPY; until someone can try it there, + * iOS keeps the path it has rather than gaining an untested one. */ +#include +#if defined(TARGET_OS_OSX) && TARGET_OS_OSX +#include +#define CRYPTON_SYSRANDOM_GETENTROPY 1 +#endif +#elif defined(__OpenBSD__) +#include +#define CRYPTON_SYSRANDOM_GETENTROPY 1 +#elif defined(__NetBSD__) +#include +#if __NetBSD_Version__ >= 1000000000 +#include +#define CRYPTON_SYSRANDOM_GETENTROPY 1 +#endif +#endif + +/* getentropy(3) refuses more than 256 bytes in one call. */ +#define CRYPTON_GETENTROPY_MAX 256 + +#if defined(CRYPTON_SYSRANDOM_GETRANDOM) + +static int sysrandom_once(uint8_t *buf, size_t n) +{ +#if defined(__linux__) + return (int) syscall(SYS_getrandom, buf, n, 0); +#else + return (int) getrandom(buf, n, 0); +#endif +} + +#elif defined(CRYPTON_SYSRANDOM_GETENTROPY) + +static int sysrandom_once(uint8_t *buf, size_t n) +{ + if (n > CRYPTON_GETENTROPY_MAX) + n = CRYPTON_GETENTROPY_MAX; + if (getentropy(buf, n) != 0) + return -1; + return (int) n; +} + +#else + +static int sysrandom_once(uint8_t *buf, size_t n) +{ + (void) buf; (void) n; + errno = ENOSYS; + return -1; +} + +#endif + +/* Is the call there, on this kernel, right now? A zero-length request + * answers that without consuming anything. */ +int crypton_sysrandom_available(void) +{ + uint8_t b; + int r = sysrandom_once(&b, 0); + return r < 0 ? 0 : 1; +} + +/* Fill the buffer. Returns the number of bytes written, which is n unless + * the call failed for a reason other than being interrupted. */ +int crypton_sysrandom_bytes(uint8_t *buf, int len) +{ + size_t n = (size_t) len; + size_t done = 0; + + if (len < 0) + return 0; + while (done < n) { + int r = sysrandom_once(buf + done, n - done); + if (r < 0) { + if (errno == EINTR) + continue; + break; + } + if (r == 0) + break; + done += (size_t) r; + } + return (int) done; +} diff --git a/cbits/mldsa/crypton_mldsa.h b/cbits/mldsa/crypton_mldsa.h --- a/cbits/mldsa/crypton_mldsa.h +++ b/cbits/mldsa/crypton_mldsa.h @@ -4,7 +4,8 @@ * choice crypton makes is made here instead of by editing it. * * Included first by both crypton_mldsa.c and crypton_mldsa_asm.S, so it must - * hold nothing but preprocessor directives. + * hold nothing the assembler cannot read: directives, and C behind + * __ASSEMBLER__. */ #ifndef CRYPTON_MLDSA_H #define CRYPTON_MLDSA_H @@ -36,6 +37,43 @@ #ifdef CRYPTON_MLDSA_NATIVE_BACKEND #define MLD_CONFIG_USE_NATIVE_BACKEND_ARITH #define MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202 + +/* + * Which of the backend's instructions this processor actually has. + * + * Without this both packages decide at compile time: the backend is built + * because the architecture has the instructions in principle, and is then + * called with no check. Upstream says so plainly -- if the machine that + * runs the binary may differ from the one that built it, this function + * MUST be provided -- and crypton is a library whose builds are copied + * between machines and shipped by distributions. Without it a binary + * built on a host with AVX2 dies with SIGILL on one without (#316). + * + * The dispatch sites are upstream's; each falls back to the portable C + * when this answers no. The answers come from crypton's own detection, + * the same that AES-NI, PCLMUL and the s2n-bignum variants use. + */ +#define MLD_CONFIG_CUSTOM_CAPABILITY_FUNC +#ifndef __ASSEMBLER__ +#include "src/sys.h" +#include "crypton_cpu.h" + +static MLD_INLINE int mld_sys_check_capability(mld_sys_cap cap) +{ +#if defined(__x86_64__) + if (cap == MLD_SYS_CAP_X86_64_AVX2) + return (crypton_x86_simd_features() & CRYPTON_X86_AVX2) != 0; +#elif defined(__aarch64__) + /* NEON is not optional on AArch64; the Keccak instructions are. */ + if (cap == MLD_SYS_CAP_AARCH64_NEON) + return 1; + if (cap == MLD_SYS_CAP_AARCH64_SHA3) + return (crypton_arm_features() & CRYPTON_ARM_SHA3) != 0; +#endif + (void)cap; + return 0; +} +#endif /* !__ASSEMBLER__ */ #endif #endif /* CRYPTON_MLDSA_H */ diff --git a/cbits/mlkem/crypton_mlkem.h b/cbits/mlkem/crypton_mlkem.h --- a/cbits/mlkem/crypton_mlkem.h +++ b/cbits/mlkem/crypton_mlkem.h @@ -4,7 +4,8 @@ * choice crypton makes is made here instead of by editing it. * * Included first by both crypton_mlkem.c and crypton_mlkem_asm.S, so it must - * hold nothing but preprocessor directives. + * hold nothing the assembler cannot read: directives, and C behind + * __ASSEMBLER__. */ #ifndef CRYPTON_MLKEM_H #define CRYPTON_MLKEM_H @@ -36,6 +37,43 @@ #ifdef CRYPTON_MLKEM_NATIVE_BACKEND #define MLK_CONFIG_USE_NATIVE_BACKEND_ARITH #define MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202 + +/* + * Which of the backend's instructions this processor actually has. + * + * Without this both packages decide at compile time: the backend is built + * because the architecture has the instructions in principle, and is then + * called with no check. Upstream says so plainly -- if the machine that + * runs the binary may differ from the one that built it, this function + * MUST be provided -- and crypton is a library whose builds are copied + * between machines and shipped by distributions. Without it a binary + * built on a host with AVX2 dies with SIGILL on one without (#316). + * + * The dispatch sites are upstream's; each falls back to the portable C + * when this answers no. The answers come from crypton's own detection, + * the same that AES-NI, PCLMUL and the s2n-bignum variants use. + */ +#define MLK_CONFIG_CUSTOM_CAPABILITY_FUNC +#ifndef __ASSEMBLER__ +#include "src/sys.h" +#include "crypton_cpu.h" + +static MLK_INLINE int mlk_sys_check_capability(mlk_sys_cap cap) +{ +#if defined(__x86_64__) + if (cap == MLK_SYS_CAP_X86_64_AVX2) + return (crypton_x86_simd_features() & CRYPTON_X86_AVX2) != 0; +#elif defined(__aarch64__) + /* NEON is not optional on AArch64; the Keccak instructions are. */ + if (cap == MLK_SYS_CAP_AARCH64_NEON) + return 1; + if (cap == MLK_SYS_CAP_AARCH64_SHA3) + return (crypton_arm_features() & CRYPTON_ARM_SHA3) != 0; +#endif + (void)cap; + return 0; +} +#endif /* !__ASSEMBLER__ */ #endif #endif /* CRYPTON_MLKEM_H */ diff --git a/cbits/s2n/README.md b/cbits/s2n/README.md --- a/cbits/s2n/README.md +++ b/cbits/s2n/README.md @@ -53,7 +53,8 @@ `crypton_powm.c`'s modular exponentiation also borrows from here, but only its innermost step and only on x86-64: `bignum_kmul_16_32`, `bignum_ksqr_16_32`, `bignum_kmul_32_64`, `bignum_ksqr_32_64` and `bignum_emontredc_8n` replace the -C's Montgomery multiplication and square. The window, the table and its masked +C's Montgomery multiplication and square, and at RSA-3072's 24 limbs the square +is built from `bignum_sqr_8_16` and `bignum_mul_8_16`. The window, the table and its masked scan are crypton's throughout. Sixteen limbs is 1024 bits and thirty-two is 2048: the halves a CRT exponentiation works in for RSA-2048 and RSA-4096, and the whole thing without CRT. The reduction wants ADX, so the choice is made at @@ -67,7 +68,34 @@ | multiplication | 0.4832 us | **0.2479** | | square | 0.3984 | **0.1994** | -**Nothing is vendored for AArch64**, where the same five routines measure level +### The Montgomery constant + +`bignum_montifier` is vendored for **both** architectures. It is the one +routine here that is not about a particular curve or a particular size: every +modular exponentiation and every setup in `crypton_ecc.c` starts by computing +R^2 mod m, and this computes it. crypton's C climbs to it by a +binary ladder of Montgomery squarings -- log2 of the modulus' bits of them, +eleven at 2048 -- and the assembly is between two and three and a half times +faster, the gap being the carry chains again: + +| | C | s2n-bignum | +| --- | ---: | ---: | +| M4, 1024 bits | 1.853 us | **0.771** | +| M4, 2048 bits | 5.835 | **2.407** | +| an Intel x86-64, 1024 bits | 6.145 | **1.995** | +| an Intel x86-64, 2048 bits | 23.332 | **6.852** | + +A CRT signature computes it twice, so an RSA-2048 private operation on the +EPYC 7763 goes from 613.5 to 606.4 microseconds -- a little over one per +cent, which is all a constant computed twice out of some two and a half +thousand multiplications can be. It is worth much more than that to anything +whose exponent is short, which is why it is here: see `crypton_powm.c`. + +The two agree bit for bit; `cbits/tests/powm` requires it at six widths, with +moduli that fill their limbs and moduli that do not, and a build without the +assembly (`-DCRYPTON_NO_MONTIFIER`) is what it is checked against. + +**None of the five above is vendored for AArch64**, where they measure level with the C. That took three attempts to establish, and the first two were wrong in opposite directions: a reading when the x86-64 routines went in put the C 5% ahead, and one on 2026-09-26 put the assembly 4% ahead. Both were @@ -91,10 +119,66 @@ the window is worth, and why it is four and not five, is in `cbits/crypton_powm.c` beside the constant. +## Windows + +x86-64 Windows takes the assembly as every other x86-64 does. Two things +stood in the way, and neither is fixed in a vendored file: + +* **The ELF directives.** Each file ends its symbols with `.type` and + `.size`, which the COFF assemblers refuse -- mingw's gas says "junk at end + of line" after a warning, clang says "expected absolute expression". + There is nothing in a PE object for them to say, so on `_WIN32` they are + emptied, as upstream already does for Mach-O. +* **The calling convention.** Each file carries a prologue that moves + Microsoft's argument registers into System V's, under `WINDOWS_ABI`, which + upstream leaves to the build to define. On `_WIN32` it is defined. + +Both are done in `x86_att/_internal_s2n_bignum_x86_att.h`, which is crypton's +and not upstream's. Every vendored file includes that name in quotes, and a +quoted include looks first beside the file that asks for it, so this is the +one each gets; it hands on to upstream's in `include/` by a relative path. +**It must not be removed as an upstream file that went missing**: without it +Windows does not assemble. On Linux and macOS it changes nothing -- the +objects are byte for byte what they were without it. + +Not `#include_next`, which is the obvious way to hand on: clang, finding this +header beside the file that asked for it, starts that search from the top +and finds this header again, without end. + +Windows on ARM keeps the C. The AArch64 files have the same directives and +would want the same header, and there is no runner to check one. + ## What is not here s2n-bignum has only x86-64 and AArch64, so crypton's C stays and is what -every other architecture uses. `p384_montjscalarmul` and `p521_jscalarmul` +every other architecture uses. + +Its `generic` directory holds a complete variable-size bignum library on both +architectures -- `bignum_montmul`, `bignum_montredc`, `bignum_emontredc`, +`bignum_modinv` and a whole `bignum_modexp` -- and only `bignum_montifier` is +taken from it. **The rest was measured on an M4 and is slower than what is +here**, which is why it stays out; they agree with crypton's answers in every +case first, so both were computing the same thing. + +| nanoseconds, AArch64 | C | s2n-bignum | +| --- | ---: | ---: | +| Montgomery multiply, 1024 bits | **171.7** | 227.9 | +| Montgomery multiply, 2048 bits | **649.6** | 870.3 | +| Montgomery square, 1024 bits | **157.3** | 202.6 | +| Montgomery square, 2048 bits | **539.2** | 870.8 | + +That is the inline assembly above doing its job: four limbs to an iteration +with two carry chains, where a size-parametric routine takes them one at a +time. `bignum_modexp`, which would have replaced `crypton_powm_sec` outright, +loses by more again -- 421.0 microseconds against 246.4 at 1024 bits and 3578 +against 1534 at 2048 -- because it has that multiplication inside it and no +window, so it pays about twice the multiplications as well. + +The README there says every function is written in a constant-time style, with +one named exception that is none of these, though it also says that unlike +functional correctness this is not machine-checked. + +`p384_montjscalarmul` and `p521_jscalarmul` have no affine wrapper upstream; crypton's is in `cbits/p256/p256_s2n.c`'s sibling for those curves. There is no fixed-base routine for P-384 or P-521 at all, so signing on those curves keeps crypton's comb. diff --git a/cbits/s2n/arm/bignum_montifier.S b/cbits/s2n/arm/bignum_montifier.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/bignum_montifier.S @@ -0,0 +1,462 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Compute "montification" constant z := 2^{128k} mod m +// Input m[k]; output z[k]; temporary buffer t[>=k] +// +// extern void bignum_montifier(uint64_t k, uint64_t *z, const uint64_t *m, +// uint64_t *t); +// +// The last argument points to a temporary buffer t that should have size >= k. +// This is called "montifier" because given any other k-digit number x, +// whether or not it's reduced modulo m, it can be mapped to its Montgomery +// representation (2^{64k} * x) mod m just by Montgomery multiplication by z. +// +// Standard ARM ABI: X0 = k, X1 = z, X2 = m, X3 = t +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_arm.h" + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_montifier) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_montifier) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_montifier) + .text + .balign 4 + +#define k x0 +#define z x1 +#define m x2 +#define t x3 + +// Some variables +// Modular inverse w is aliased to i, but we never use them together + +#define i x4 +#define w x4 +#define j x5 +#define h x6 +#define a x7 +#define l x8 +#define c x9 +#define b x10 +#define d x11 + +// Some aliases for the values b and d + +#define r x10 +#define q x11 + + +S2N_BN_SYMBOL(bignum_montifier): + CFI_START + +// If k = 0 the whole operation is trivial + + cbz k, Lbignum_montifier_end + +// Copy the input m into the temporary buffer t. The temporary register +// c matters since we want it to hold the highest digit, ready for the +// normalization phase. + + mov i, xzr +Lbignum_montifier_copyinloop: + ldr c, [m, i, lsl #3] + str c, [t, i, lsl #3] + add i, i, #1 + cmp i, k + bcc Lbignum_montifier_copyinloop + +// Do a rather stupid but constant-time digit normalization, conditionally +// shifting left (k-1) times based on whether the top word is zero. +// With careful binary striding this could be O(k*log(k)) instead of O(k^2) +// while still retaining the constant-time style. +// The "cmp c, xzr" sets the zeroness predicate (ZF) for the entire inner loop + + subs i, k, #1 + beq Lbignum_montifier_normalized +Lbignum_montifier_normloop: + mov j, xzr + cmp c, xzr + mov a, xzr +Lbignum_montifier_shufloop: + mov c, a + ldr a, [t, j, lsl #3] + csel c, c, a, eq + str c, [t, j, lsl #3] + add j, j, #1 + sub d, j, k + cbnz d, Lbignum_montifier_shufloop + subs i, i, #1 + bne Lbignum_montifier_normloop + +// We now have the top digit nonzero, assuming the input was nonzero, +// and as per the invariant of the loop above, c holds that digit. So +// now just count c's leading zeros and shift t bitwise that many bits. + +Lbignum_montifier_normalized: + clz c, c + + mov b, xzr + mov i, xzr + ands xzr, c, #63 + csetm l, ne + neg d, c +Lbignum_montifier_bitloop: + ldr j, [t, i, lsl #3] + lsl a, j, c + orr a, a, b + lsr b, j, d + and b, b, l + str a, [t, i, lsl #3] + add i, i, #1 + cmp i, k + bcc Lbignum_montifier_bitloop + +// Let h be the high word of n, which in all the in-scope cases is >= 2^63. +// Now successively form q = 2^i div h and r = 2^i mod h as i goes from +// 64 to 126. We avoid just using division out of constant-time concerns +// (at the least we would need to fix up h = 0 for out-of-scope inputs) and +// don't bother with Newton-Raphson, since this stupid simple loop doesn't +// contribute much of the overall runtime at typical sizes. + + sub h, k, #1 + ldr h, [t, h, lsl #3] + mov q, #1 + neg r, h + mov i, #62 +Lbignum_montifier_estloop: + add q, q, q + mov a, h + sub a, a, r + cmp r, a // CF <=> r >= h - r <=> 2 * r >= h + csetm a, cs + sub q, q, a + add r, r, r + and a, a, h + sub r, r, a + subs i, i, #1 + bne Lbignum_montifier_estloop + +// Strictly speaking the above loop doesn't quite give the true remainder +// and quotient in the special case r = h = 2^63, so fix it up. We get +// q = 2^63 - 1 and r = 2^63 and really want q = 2^63 and r = 0. This is +// supererogatory, because the main property of q used below still holds +// in this case unless the initial m = 1, and then anyway the overall +// specification (congruence modulo m) holds degenerately. But it seems +// nicer to get a "true" quotient and remainder. + + cmp r, h + csinc q, q, q, ne + +// So now we have q and r with 2^126 = q * h + r (imagining r = 0 in the +// fixed-up case above: note that we never actually use the computed +// value of r below and so didn't adjust it). And we can assume the ranges +// q <= 2^63 and r < h < 2^64. +// +// The idea is to use q as a first quotient estimate for a remainder +// of 2^{p+62} mod n, where p = 64 * k. We have, splitting n into the +// high and low parts h and l: +// +// 2^{p+62} - q * n = 2^{p+62} - q * (2^{p-64} * h + l) +// = 2^{p+62} - (2^{p-64} * (q * h) + q * l) +// = 2^{p+62} - 2^{p-64} * (2^126 - r) - q * l +// = 2^{p-64} * r - q * l +// +// Note that 2^{p-64} * r < 2^{p-64} * h <= n +// and also q * l < 2^63 * 2^{p-64} = 2^{p-1} <= n +// so |diff| = |2^{p-64} * r - q * l| < n. +// +// If in fact diff >= 0 then it is already 2^{p+62} mod n. +// otherwise diff + n is the right answer. +// +// To (maybe?) make the computation slightly easier we actually flip +// the sign and compute d = q * n - 2^{p+62}. Then the answer is either +// -d (when negative) or n - d; in either case we effectively negate d. +// This negating tweak in fact spoils the result for cases where +// 2^{p+62} mod n = 0, when we get n instead. However the only case +// where this can happen is m = 1, when the whole spec holds trivially, +// and actually the remainder of the logic below works anyway since +// the latter part of the code only needs a congruence for the k-digit +// result, not strict modular reduction (the doublings will maintain +// the non-strict inequality). + + mov c, xzr + adds i, xzr, xzr +Lbignum_montifier_mulloop: + ldr a, [t, i, lsl #3] + mul l, q, a + adcs l, l, c + umulh c, q, a + str l, [z, i, lsl #3] + add i, i, #1 + sub a, i, k + cbnz a, Lbignum_montifier_mulloop + + adc c, c, xzr + mov a, #0x4000000000000000 + subs c, c, a + csetm q, cs + +// Now do [c] * n - d for our final answer + + subs i, xzr, xzr +Lbignum_montifier_remloop: + ldr a, [t, i, lsl #3] + ldr b, [z, i, lsl #3] + and a, a, q + sbcs a, a, b + str a, [z, i, lsl #3] + add i, i, #1 + sub a, i, k + cbnz a, Lbignum_montifier_remloop + +// Now still need to do a couple of modular doublings to get us all the +// way up to 2^{p+64} == r from the initial 2^{p+62} == r (mod n). + + mov c, xzr + subs j, xzr, xzr +Lbignum_montifier_dubloop1: + ldr a, [z, j, lsl #3] + extr c, a, c, #63 + ldr b, [t, j, lsl #3] + sbcs c, c, b + str c, [z, j, lsl #3] + mov c, a + add j, j, #1 + sub a, j, k + cbnz a, Lbignum_montifier_dubloop1 + lsr c, c, #63 + sbc c, c, xzr + adds j, xzr, xzr +Lbignum_montifier_corrloop1: + ldr a, [z, j, lsl #3] + ldr b, [t, j, lsl #3] + and b, b, c + adcs a, a, b + str a, [z, j, lsl #3] + add j, j, #1 + sub a, j, k + cbnz a, Lbignum_montifier_corrloop1 + +// This is not exactly the same: we also copy output to t giving the +// initialization t_1 = r == 2^{p+64} mod n for the main loop next. + + mov c, xzr + subs j, xzr, xzr +Lbignum_montifier_dubloop2: + ldr a, [z, j, lsl #3] + extr c, a, c, #63 + ldr b, [t, j, lsl #3] + sbcs c, c, b + str c, [z, j, lsl #3] + mov c, a + add j, j, #1 + sub a, j, k + cbnz a, Lbignum_montifier_dubloop2 + lsr c, c, #63 + sbc c, c, xzr + adds j, xzr, xzr +Lbignum_montifier_corrloop2: + ldr a, [z, j, lsl #3] + ldr b, [t, j, lsl #3] + and b, b, c + adcs a, a, b + str a, [z, j, lsl #3] + str a, [t, j, lsl #3] + add j, j, #1 + sub a, j, k + cbnz a, Lbignum_montifier_corrloop2 + +// We then successively generate (k+1)-digit values satisfying +// t_i == 2^{p+64*i} mod n, each of which is stored in h::t. Finish +// initialization by zeroing h initially + + mov h, xzr + +// Then if t_i = 2^{p} * h + l +// we have t_{i+1} == 2^64 * t_i +// = (2^{p+64} * h) + (2^64 * l) +// == r * h + l<<64 +// Do this 2*k more times so we end up == 2^{192*k+64}, one more than we want +// +// Writing B = 2^{64k}, the possible correction of adding r, which for +// a (k+1)-digit result is equivalent to subtracting q = 2^{64*(k+1)} - r +// would give the overall worst-case value minus q of +// [ B * (B^k - 1) + (B - 1) * r ] - [B^{k+1} - r] +// = B * (r - 1) < B^{k+1} so we keep inside k+1 digits as required. +// +// This implementation makes the shift implicit by starting b with the +// "previous" digit (initially 0) to offset things by 1. + + add i, k, k +Lbignum_montifier_modloop: + mov j, xzr + mov b, xzr + adds c, xzr, xzr +Lbignum_montifier_cmaloop: + ldr a, [z, j, lsl #3] + mul l, h, a + adcs b, b, c + umulh c, h, a + adc c, c, xzr + adds l, b, l + ldr b, [t, j, lsl #3] + str l, [t, j, lsl #3] + add j, j, #1 + sub a, j, k + cbnz a, Lbignum_montifier_cmaloop + + adcs h, b, c + + csetm l, cs + + adds j, xzr, xzr +Lbignum_montifier_oaloop: + ldr a, [t, j, lsl #3] + ldr b, [z, j, lsl #3] + and b, b, l + adcs a, a, b + str a, [t, j, lsl #3] + add j, j, #1 + sub a, j, k + cbnz a, Lbignum_montifier_oaloop + adc h, h, xzr + + subs i, i, #1 + bne Lbignum_montifier_modloop + +// Compute the negated modular inverse w (same register as i, not used again). + + ldr a, [m] + lsl w, a, #2 + sub w, a, w + eor w, w, #2 + mov l, #1 + madd c, a, w, l + mul b, c, c + madd w, c, w, w + mul c, b, b + madd w, b, w, w + mul b, c, c + madd w, c, w, w + madd w, b, w, w + +// Now do one almost-Montgomery reduction w.r.t. the original m +// which lops off one 2^64 from the congruence and, with the usual +// almost-Montgomery correction, gets us back inside k digits for +// the end result. + + ldr b, [t] + mul d, b, w + + mul l, d, a + umulh c, d, a + mov j, #1 + sub a, k, #1 + adds xzr, b, l + cbz a, Lbignum_montifier_amontend + +Lbignum_montifier_amontloop: + ldr a, [m, j, lsl #3] + ldr b, [t, j, lsl #3] + mul l, d, a + adcs b, b, c + umulh c, d, a + adc c, c, xzr + adds b, b, l + sub a, j, #1 + str b, [t, a, lsl #3] + add j, j, #1 + sub a, j, k + cbnz a, Lbignum_montifier_amontloop +Lbignum_montifier_amontend: + adcs h, h, c + csetm l, cs + sub a, k, #1 + str h, [t, a, lsl #3] + + subs j, xzr, xzr +Lbignum_montifier_osloop: + ldr a, [t, j, lsl #3] + ldr b, [m, j, lsl #3] + and b, b, l + sbcs a, a, b + str a, [z, j, lsl #3] + add j, j, #1 + sub a, j, k + cbnz a, Lbignum_montifier_osloop + +// So far, the code(basically a variant of bignum_amontifier) has produced +// a k-digit value z == 2^{192k} (mod m), not necessarily fully reduced mod m. +// We now do a short Montgomery reduction (similar to bignum_demont) so that +// we achieve full reduction mod m while lopping 2^{64k} off the congruence. +// We recycle h as the somewhat strangely-named outer loop counter. + + mov h, k + +Lbignum_montifier_montouterloop: + ldr b, [z] + mul d, b, w + ldr a, [m] + mul l, d, a + umulh c, d, a + mov j, #1 + sub a, k, #1 + adds xzr, b, l + cbz a, Lbignum_montifier_montend +Lbignum_montifier_montloop: + ldr a, [m, j, lsl #3] + ldr b, [z, j, lsl #3] + mul l, d, a + adcs b, b, c + umulh c, d, a + adc c, c, xzr + adds b, b, l + sub a, j, #1 + str b, [z, a, lsl #3] + add j, j, #1 + sub a, j, k + cbnz a, Lbignum_montifier_montloop +Lbignum_montifier_montend: + adc c, c, xzr + sub a, k, #1 + str c, [z, a, lsl #3] + + subs h, h, #1 + bne Lbignum_montifier_montouterloop + +// Now do a comparison of z with m to set a final correction mask +// indicating that z >= m and so we need to subtract m. + + subs j, xzr, xzr +Lbignum_montifier_cmploop: + ldr a, [z, j, lsl #3] + ldr b, [m, j, lsl #3] + sbcs xzr, a, b + add j, j, #1 + sub a, j, k + cbnz a, Lbignum_montifier_cmploop + csetm h, cs + +// Now do a masked subtraction of m for the final reduced result. + + subs j, xzr, xzr +Lbignum_montifier_corrloop: + ldr a, [z, j, lsl #3] + ldr b, [m, j, lsl #3] + and b, b, h + sbcs a, a, b + str a, [z, j, lsl #3] + add j, j, #1 + sub a, j, k + cbnz a, Lbignum_montifier_corrloop + +Lbignum_montifier_end: + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_montifier) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/import.sh b/cbits/s2n/import.sh --- a/cbits/s2n/import.sh +++ b/cbits/s2n/import.sh @@ -20,7 +20,10 @@ git clone -q "$REPO" "$TMP/s2n" git -C "$TMP/s2n" checkout -q "$REV" -# The headers every vendored file includes. +# The headers every vendored file includes. Not x86_att/'s own header of +# the same name, which is crypton's -- it makes the x86-64 files assemble on +# Windows and hands on to this one; see README.md. Nothing here copies into +# that name, and nothing should. for h in _internal_s2n_bignum_arm.h _internal_s2n_bignum_x86_att.h; do cp "$TMP/s2n/include/$h" "$HERE/include/$h" done @@ -107,6 +110,22 @@ take_x86 fastmul bignum_kmul_32_64 take_x86 fastmul bignum_ksqr_32_64 take_x86 fastmul bignum_emontredc_8n +# RSA-3072's 24-limb square, built from these where the 32-limb Karatsuba +# would take its operands padded +take_x86 fastmul bignum_sqr_8_16 +take_x86 fastmul bignum_mul_8_16 +# and, for the sizes there is no Karatsuba routine for, the schoolbook +# multiplication and squaring of any size, which bignum_emontredc_8n then +# reduces as it reduces the others +take_x86 generic bignum_mul +take_x86 generic bignum_sqr + +# R^2 mod m, which every exponentiation and every setup in crypton_ecc.c +# begins with. Both architectures: the C climbs to it by a ladder of +# Montgomery squarings and this is faster on both. Taken from upstream's +# main when it went in rather than by this script, which is why it was +# missing here; it is the same file at the COMMIT above. +take generic bignum_montifier git -C "$TMP/s2n" rev-parse HEAD > "$HERE/COMMIT" echo "imported $(cat "$HERE/COMMIT")" diff --git a/cbits/s2n/x86_att/_internal_s2n_bignum_x86_att.h b/cbits/s2n/x86_att/_internal_s2n_bignum_x86_att.h new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/_internal_s2n_bignum_x86_att.h @@ -0,0 +1,33 @@ +/* + * crypton's, not s2n-bignum's: what the vendored assembly needs on Windows, + * kept out of the vendored files so that import.sh can replace them as it + * always has. + * + * Every file in this directory includes "_internal_s2n_bignum_x86_att.h", + * and a quoted include looks first in the directory of the file that asks + * for it -- so this is the one each of them gets, and it hands on to + * upstream's own in cbits/s2n/include. On Linux and macOS it changes + * nothing. + * + * On Windows, two things. The ELF directives .type and .size are refused by + * the COFF assemblers -- gcc's says "junk at end of line" and clang's + * "expected absolute expression" -- and there is nothing in a PE object for + * them to say, so they are emptied, as upstream does for Mach-O. And the + * calling convention is Microsoft's, not System V's: each file carries a + * prologue that moves its arguments into the registers it was written + * for, under WINDOWS_ABI, which upstream leaves to the build to define. + */ +/* By a path from this file rather than by #include_next: clang, finding + * this file next to the one that asked for it, starts #include_next from + * the top of the search and finds this file again, without end. */ +#include "../include/_internal_s2n_bignum_x86_att.h" + +#if defined(_WIN32) +# undef S2N_BN_FUNCTION_TYPE_DIRECTIVE +# define S2N_BN_FUNCTION_TYPE_DIRECTIVE(name) /* not used in COFF */ +# undef S2N_BN_SIZE_DIRECTIVE +# define S2N_BN_SIZE_DIRECTIVE(name) /* not used in COFF */ +# ifndef WINDOWS_ABI +# define WINDOWS_ABI 1 +# endif +#endif diff --git a/cbits/s2n/x86_att/bignum_montifier.S b/cbits/s2n/x86_att/bignum_montifier.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/bignum_montifier.S @@ -0,0 +1,545 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Compute "montification" constant z := 2^{128k} mod m +// Input m[k]; output z[k]; temporary buffer t[>=k] +// +// extern void bignum_montifier(uint64_t k, uint64_t *z, const uint64_t *m, +// uint64_t *t); +// +// The last argument points to a temporary buffer t that should have size >= k. +// This is called "montifier" because given any other k-digit number x, +// whether or not it's reduced modulo m, it can be mapped to its Montgomery +// representation (2^{64k} * x) mod m just by Montgomery multiplication by z. +// +// Standard x86-64 ABI: RDI = k, RSI = z, RDX = m, RCX = t +// Microsoft x64 ABI: RCX = k, RDX = z, R8 = m, R9 = t +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_montifier) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_montifier) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_montifier) + .text + .balign 32 + +#define k %rdi +#define z %rsi + +// These two inputs get moved to different places since RCX and RDX are special + +#define m %r12 +#define t %r13 + +// Other variables + +#define i %rbx +// Modular inverse; aliased to i, but we never use them together +#define w %rbx +#define j %rbp +// Matters that this is RAX for special use in multiplies +#define a %rax +// Matters that this is RDX for special use in multiplies +#define d %rdx +// Matters that this is RCX as CL=lo(c) is assumed in shifts +#define c %rcx +#define h %r11 +#define l %r10 +#define b %r9 +#define n %r8 + +// Some aliases for the values b and n + +#define q %r8 +#define r %r9 + +#define ashort %eax +#define ishort %ebx +#define jshort %ebp +#define qshort %r8d + + +S2N_BN_SYMBOL(bignum_montifier): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi + movq %r8, %rdx + movq %r9, %rcx +#endif + +// Save some additional registers for use, copy args out of RCX and RDX + + CFI_PUSH(%rbp) + CFI_PUSH(%rbx) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + + movq %rdx, m + movq %rcx, t + +// If k = 0 the whole operation is trivial + + testq k, k + jz Lbignum_montifier_end + +// Copy the input m into the temporary buffer t. The temporary register +// c matters since we want it to hold the highest digit, ready for the +// normalization phase. + + xorq i, i +Lbignum_montifier_copyinloop: + movq (m,i,8), c + movq c, (t,i,8) + incq i + cmpq k, i + jc Lbignum_montifier_copyinloop + +// Do a rather stupid but constant-time digit normalization, conditionally +// shifting left (k-1) times based on whether the top word is zero. +// With careful binary striding this could be O(k*log(k)) instead of O(k^2) +// while still retaining the constant-time style. +// The "neg c" sets the zeroness predicate (~CF) for the entire inner loop + + movq k, i + decq i + jz Lbignum_montifier_normalized +Lbignum_montifier_normloop: + xorq j, j + movq k, h + negq c + movl $0, ashort +Lbignum_montifier_shufloop: + movq a, c + movq (t,j,8), a + cmovcq a, c + movq c, (t,j,8) + incq j + decq h + jnz Lbignum_montifier_shufloop + decq i + jnz Lbignum_montifier_normloop + +// We now have the top digit nonzero, assuming the input was nonzero, +// and as per the invariant of the loop above, c holds that digit. So +// now just count c's leading zeros and shift t bitwise that many bits. +// Note that we don't care about the result of bsr for zero inputs so +// the simple xor-ing with 63 is safe. + +Lbignum_montifier_normalized: + + bsrq c, c + xorq $63, c + + xorq b, b + xorq i, i +Lbignum_montifier_bitloop: + movq (t,i,8), a + movq a, j + shldq %cl, b, a + movq a, (t,i,8) + movq j, b + incq i + cmpq k, i + jc Lbignum_montifier_bitloop + +// Let h be the high word of n, which in all the in-scope cases is >= 2^63. +// Now successively form q = 2^i div h and r = 2^i mod h as i goes from +// 64 to 126. We avoid just using division out of constant-time concerns +// (at the least we would need to fix up h = 0 for out-of-scope inputs) and +// don't bother with Newton-Raphson, since this stupid simple loop doesn't +// contribute much of the overall runtime at typical sizes. + + movq -8(t,k,8), h + movl $1, qshort + movq h, r + negq r + movl $62, ishort +Lbignum_montifier_estloop: + + addq q, q + movq h, a + subq r, a + cmpq a, r // CF <=> r < h - r <=> 2 * r < h + sbbq a, a + notq a // a = bitmask(2 * r >= h) + subq a, q + addq r, r + andq h, a + subq a, r + decq i + jnz Lbignum_montifier_estloop + +// Strictly speaking the above loop doesn't quite give the true remainder +// and quotient in the special case r = h = 2^63, so fix it up. We get +// q = 2^63 - 1 and r = 2^63 and really want q = 2^63 and r = 0. This is +// supererogatory, because the main property of q used below still holds +// in this case unless the initial m = 1, and then anyway the overall +// specification (congruence modulo m) holds degenerately. But it seems +// nicer to get a "true" quotient and remainder. + + incq r + cmpq r, h + adcq $0, q + +// So now we have q and r with 2^126 = q * h + r (imagining r = 0 in the +// fixed-up case above: note that we never actually use the computed +// value of r below and so didn't adjust it). And we can assume the ranges +// q <= 2^63 and r < h < 2^64. +// +// The idea is to use q as a first quotient estimate for a remainder +// of 2^{p+62} mod n, where p = 64 * k. We have, splitting n into the +// high and low parts h and l: +// +// 2^{p+62} - q * n = 2^{p+62} - q * (2^{p-64} * h + l) +// = 2^{p+62} - (2^{p-64} * (q * h) + q * l) +// = 2^{p+62} - 2^{p-64} * (2^126 - r) - q * l +// = 2^{p-64} * r - q * l +// +// Note that 2^{p-64} * r < 2^{p-64} * h <= n +// and also q * l < 2^63 * 2^{p-64} = 2^{p-1} <= n +// so |diff| = |2^{p-64} * r - q * l| < n. +// +// If in fact diff >= 0 then it is already 2^{p+62} mod n. +// otherwise diff + n is the right answer. +// +// To (maybe?) make the computation slightly easier we actually flip +// the sign and compute d = q * n - 2^{p+62}. Then the answer is either +// -d (when negative) or n - d; in either case we effectively negate d. +// This negating tweak in fact spoils the result for cases where +// 2^{p+62} mod n = 0, when we get n instead. However the only case +// where this can happen is m = 1, when the whole spec holds trivially, +// and actually the remainder of the logic below works anyway since +// the latter part of the code only needs a congruence for the k-digit +// result, not strict modular reduction (the doublings will maintain +// the non-strict inequality). + + xorq c, c + xorq i, i +Lbignum_montifier_mulloop: + movq (t,i,8), a + mulq q + addq c, a + adcq $0, d + movq a, (z,i,8) + movq d, c + incq i + cmpq k, i + jc Lbignum_montifier_mulloop + +// Now c is the high word of the product, so subtract 2^62 +// and then turn it into a bitmask in q = h + + movq $0x4000000000000000, a + subq a, c + sbbq q, q + notq q + +// Now do [c] * n - d for our final answer + + xorq c, c + xorq i, i +Lbignum_montifier_remloop: + movq (t,i,8), a + andq q, a + negq c + sbbq (z,i,8), a + sbbq c, c + movq a, (z,i,8) + incq i + cmpq k, i + jc Lbignum_montifier_remloop + +// Now still need to do a couple of modular doublings to get us all the +// way up to 2^{p+64} == r from initial 2^{p+62} == r (mod n). + + xorq c, c + xorq j, j + xorq b, b +Lbignum_montifier_dubloop1: + movq (z,j,8), a + shrdq $63, a, c + negq b + sbbq (t,j,8), c + sbbq b, b + movq c, (z,j,8) + movq a, c + incq j + cmpq k, j + jc Lbignum_montifier_dubloop1 + shrq $63, c + addq b, c + xorq j, j + xorq b, b +Lbignum_montifier_corrloop1: + movq (t,j,8), a + andq c, a + negq b + adcq (z,j,8), a + sbbq b, b + movq a, (z,j,8) + incq j + cmpq k, j + jc Lbignum_montifier_corrloop1 + +// This is not exactly the same: we also copy output to t giving the +// initialization t_1 = r == 2^{p+64} mod n for the main loop next. + + xorq c, c + xorq j, j + xorq b, b +Lbignum_montifier_dubloop2: + movq (z,j,8), a + shrdq $63, a, c + negq b + sbbq (t,j,8), c + sbbq b, b + movq c, (z,j,8) + movq a, c + incq j + cmpq k, j + jc Lbignum_montifier_dubloop2 + shrq $63, c + addq b, c + xorq j, j + xorq b, b +Lbignum_montifier_corrloop2: + movq (t,j,8), a + andq c, a + negq b + adcq (z,j,8), a + sbbq b, b + movq a, (z,j,8) + movq a, (t,j,8) + incq j + cmpq k, j + jc Lbignum_montifier_corrloop2 + +// We then successively generate (k+1)-digit values satisfying +// t_i == 2^{p+64*i} mod n, each of which is stored in h::t. Finish +// initialization by zeroing h initially + + xorq h, h + +// Then if t_i = 2^{p} * h + l +// we have t_{i+1} == 2^64 * t_i +// = (2^{p+64} * h) + (2^64 * l) +// == r * h + l<<64 +// Do this 2*k more times so we end up == 2^{192*k+64}, one more than we want +// +// Writing B = 2^{64k}, the possible correction of adding r, which for +// a (k+1)-digit result is equivalent to subtracting q = 2^{64*(k+1)} - r +// would give the overall worst-case value minus q of +// [ B * (B^k - 1) + (B - 1) * r ] - [B^{k+1} - r] +// = B * (r - 1) < B^{k+1} so we keep inside k+1 digits as required. +// +// This implementation makes the shift implicit by starting b with the +// "previous" digit (initially 0) to offset things by 1. + + leaq (k,k), i +Lbignum_montifier_modloop: + xorq b, b + movq k, n + xorq j, j + xorq c, c +Lbignum_montifier_cmaloop: + adcq b, c + sbbq l, l + movq (z,j,8), a + mulq h + subq l, d + addq c, a + movq (t,j,8), b + movq a, (t,j,8) + movq d, c + incq j + decq n + jnz Lbignum_montifier_cmaloop + adcq c, b + movq b, h + + sbbq l, l + + xorq j, j + xorq c, c +Lbignum_montifier_oaloop: + movq (t,j,8), a + movq (z,j,8), b + andq l, b + negq c + adcq b, a + sbbq c, c + movq a, (t,j,8) + incq j + cmpq k, j + jc Lbignum_montifier_oaloop + subq c, h + + decq i + jnz Lbignum_montifier_modloop + +// Compute the negated modular inverse w (same register as i, not used again). + + movq (m), a + movq a, c + movq a, w + shlq $2, c + subq c, w + xorq $2, w + movq w, c + imulq a, c + movl $2, ashort + addq c, a + addq $1, c + imulq a, w + imulq c, c + movl $1, ashort + addq c, a + imulq a, w + imulq c, c + movl $1, ashort + addq c, a + imulq a, w + imulq c, c + movl $1, ashort + addq c, a + imulq a, w + +// Now do one almost-Montgomery reduction w.r.t. the original m +// which lops off one 2^64 from the congruence and, with the usual +// almost-Montgomery correction, gets us back inside k digits + + movq (t), c + movq w, b + imulq c, b + + movq (m), a + mulq b + addq c, a + movq d, c + movl $1, jshort + movq k, n + decq n + jz Lbignum_montifier_amontend +Lbignum_montifier_amontloop: + adcq (t,j,8), c + sbbq l, l + movq (m,j,8), a + mulq b + subq l, d + addq c, a + movq a, -8(t,j,8) + movq d, c + incq j + decq n + jnz Lbignum_montifier_amontloop +Lbignum_montifier_amontend: + adcq c, h + sbbq l, l + movq h, -8(t,k,8) + + xorq j, j + xorq c, c +Lbignum_montifier_aosloop: + movq (t,j,8), a + movq (m,j,8), b + andq l, b + negq c + sbbq b, a + sbbq c, c + movq a, (z,j,8) + incq j + cmpq k, j + jc Lbignum_montifier_aosloop + +// So far, the code (basically a variant of bignum_amontifier) has produced +// a k-digit value z == 2^{192k} (mod m), not necessarily fully reduced mod m. +// We now do a short Montgomery reduction (similar to bignum_demont) so that +// we achieve full reduction mod m while lopping 2^{64k} off the congruence. +// We recycle h as the somewhat strangely-named outer loop counter. + + movq k, h + +Lbignum_montifier_montouterloop: + movq (z), c + movq w, b + imulq c, b + movq (m), a + mulq b + addq c, a + movq d, c + movl $1, jshort + movq k, n + decq n + jz Lbignum_montifier_montend +Lbignum_montifier_montloop: + adcq (z,j,8), c + sbbq l, l + movq (m,j,8), a + mulq b + subq l, d + addq c, a + movq a, -8(z,j,8) + movq d, c + incq j + decq n + jnz Lbignum_montifier_montloop +Lbignum_montifier_montend: + adcq $0, c + movq c, -8(z,k,8) + + decq h + jnz Lbignum_montifier_montouterloop + +// Now do a comparison of z with m to set a final correction mask +// indicating that z >= m and so we need to subtract m. + + xorq j, j + movq k, n +Lbignum_montifier_cmploop: + movq (z,j,8), a + sbbq (m,j,8), a + incq j + decq n + jnz Lbignum_montifier_cmploop + sbbq d, d + notq d + +// Now do a masked subtraction of m for the final reduced result. + + xorq l, l + xorq j, j +Lbignum_montifier_corrloop: + movq (m,j,8), a + andq d, a + negq l + sbbq a, (z,j,8) + sbbq l, l + incq j + cmpq k, j + jc Lbignum_montifier_corrloop + +Lbignum_montifier_end: + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbx) + CFI_POP(%rbp) + +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_montifier) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/x86_att/bignum_mul.S b/cbits/s2n/x86_att/bignum_mul.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/bignum_mul.S @@ -0,0 +1,160 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Multiply z := x * y +// Inputs x[m], y[n]; output z[k] +// +// extern void bignum_mul(uint64_t k, uint64_t *z, uint64_t m, const uint64_t *x, +// uint64_t n, const uint64_t *y); +// +// Does the "z := x * y" operation where x is m digits, y is n, result z is k. +// Truncates the result in general unless k >= m + n +// +// Standard x86-64 ABI: RDI = k, RSI = z, RDX = m, RCX = x, R8 = n, R9 = y +// Microsoft x64 ABI: RCX = k, RDX = z, R8 = m, R9 = x, [RSP+40] = n, [RSP+48] = y +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_mul) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_mul) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_mul) + .text + .balign 32 + +// These are actually right + +#define p %rdi +#define z %rsi +#define n %r8 + +// These are not + +#define c %r15 +#define h %r14 +#define l %r13 +#define x %r12 +#define y %r11 +#define i %rbx +#define k %r10 +#define m %rbp + +// These are always local scratch since multiplier result is in these + +#define a %rax +#define d %rdx + + + +S2N_BN_SYMBOL(bignum_mul): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi + movq %r8, %rdx + movq %r9, %rcx + movq 56(%rsp), %r8 + movq 64(%rsp), %r9 +#endif + +// We use too many registers, and also we need %rax:%rdx for multiplications + + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + movq %rdx, m + +// If the result size is zero, do nothing +// Note that even if either or both inputs has size zero, we can't +// just give up because we at least need to zero the output array +// If we did a multiply-add variant, however, then we could + + testq p, p + jz Lbignum_mul_end + +// Set initial 2-part sum to zero (we zero c inside the body) + + xorq h, h + xorq l, l + +// Otherwise do outer loop k = 0 ... k = p - 1 + + xorq k, k + +Lbignum_mul_outerloop: + +// Zero our carry term first; we eventually want it and a zero is useful now +// Set a = max 0 (k + 1 - n), i = min (k + 1) m +// This defines the range a <= j < i for the inner summation +// Note that since k < p < 2^64 we can assume k + 1 doesn't overflow +// And since we want to increment it anyway, we might as well do it now + + xorq c, c // c = 0 + incq k // k = k + 1 + + movq k, a // a = k + 1 + subq n, a // a = k + 1 - n + cmovcq c, a // a = max 0 (k + 1 - n) + + movq m, i // i = m + cmpq m, k // CF <=> k + 1 < m + cmovcq k, i // i = min (k + 1) m + +// Turn i into a loop count, and skip things if it's <= 0 +// Otherwise set up initial pointers x -> x0[a] and y -> y0[k - a] +// and then launch into the main inner loop, postdecrementing i + + movq k, d + subq i, d + subq a, i + jbe Lbignum_mul_innerend + leaq (%rcx,a,8), x + leaq -8(%r9,d,8), y + +Lbignum_mul_innerloop: + movq (y,i,8), %rax + mulq (x) + addq $8, x + addq %rax, l + adcq %rdx, h + adcq $0, c + decq i + jnz Lbignum_mul_innerloop + +Lbignum_mul_innerend: + + movq l, (z) + movq h, l + movq c, h + addq $8, z + + cmpq p, k + jc Lbignum_mul_outerloop + +Lbignum_mul_end: + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_mul) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/x86_att/bignum_mul_8_16.S b/cbits/s2n/x86_att/bignum_mul_8_16.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/bignum_mul_8_16.S @@ -0,0 +1,264 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Multiply z := x * y +// Inputs x[8], y[8]; output z[16] +// +// extern void bignum_mul_8_16(uint64_t z[static 16], const uint64_t x[static 8], +// const uint64_t y[static 8]); +// +// Standard x86-64 ABI: RDI = z, RSI = x, RDX = y +// Microsoft x64 ABI: RCX = z, RDX = x, R8 = y +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_mul_8_16) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_mul_8_16) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_mul_8_16) + .text + +// These are actually right + +#define z %rdi +#define x %rsi + +// Copied in or set up + +#define y %rcx + +// A zero register + +#define zero %rbp +#define zeroe %ebp + +// mulpadd i, j adds x[i] * rdx (now assumed = y[j]) into the window at i+j + +.macro mulpadd arg1,arg2 + mulxq 8*\arg1(x), %rax, %rbx +.if ((\arg1 + \arg2) % 8 == 0) + adcxq %rax, %r8 + adoxq %rbx, %r9 +.elseif ((\arg1 + \arg2) % 8 == 1) + adcxq %rax, %r9 + adoxq %rbx, %r10 +.elseif ((\arg1 + \arg2) % 8 == 2) + adcxq %rax, %r10 + adoxq %rbx, %r11 +.elseif ((\arg1 + \arg2) % 8 == 3) + adcxq %rax, %r11 + adoxq %rbx, %r12 +.elseif ((\arg1 + \arg2) % 8 == 4) + adcxq %rax, %r12 + adoxq %rbx, %r13 +.elseif ((\arg1 + \arg2) % 8 == 5) + adcxq %rax, %r13 + adoxq %rbx, %r14 +.elseif ((\arg1 + \arg2) % 8 == 6) + adcxq %rax, %r14 + adoxq %rbx, %r15 +.elseif ((\arg1 + \arg2) % 8 == 7) + adcxq %rax, %r15 + adoxq %rbx, %r8 +.endif + +.endm + +// mulpade i, j adds x[i] * rdx (now assumed = y[j]) into the window at i+j +// but re-creates the top word assuming nothing to add there + +.macro mulpade arg1,arg2 +.if ((\arg1 + \arg2) % 8 == 0) + mulxq 8*\arg1(x), %rax, %r9 + adcxq %rax, %r8 + adoxq zero, %r9 +.elseif ((\arg1 + \arg2) % 8 == 1) + mulxq 8*\arg1(x), %rax, %r10 + adcxq %rax, %r9 + adoxq zero, %r10 +.elseif ((\arg1 + \arg2) % 8 == 2) + mulxq 8*\arg1(x), %rax, %r11 + adcxq %rax, %r10 + adoxq zero, %r11 +.elseif ((\arg1 + \arg2) % 8 == 3) + mulxq 8*\arg1(x), %rax, %r12 + adcxq %rax, %r11 + adoxq zero, %r12 +.elseif ((\arg1 + \arg2) % 8 == 4) + mulxq 8*\arg1(x), %rax, %r13 + adcxq %rax, %r12 + adoxq zero, %r13 +.elseif ((\arg1 + \arg2) % 8 == 5) + mulxq 8*\arg1(x), %rax, %r14 + adcxq %rax, %r13 + adoxq zero, %r14 +.elseif ((\arg1 + \arg2) % 8 == 6) + mulxq 8*\arg1(x), %rax, %r15 + adcxq %rax, %r14 + adoxq zero, %r15 +.elseif ((\arg1 + \arg2) % 8 == 7) + mulxq 8*\arg1(x), %rax, %r8 + adcxq %rax, %r15 + adoxq zero, %r8 +.endif + +.endm + +// Add in the whole j'th row + +.macro addrow arg1 + movq 8*\arg1(y), %rdx + xorl zeroe, zeroe + + mulpadd 0, \arg1 + +.if (\arg1 % 8 == 0) + movq %r8, 8*\arg1(z) +.elseif (\arg1 % 8 == 1) + movq %r9, 8*\arg1(z) +.elseif (\arg1 % 8 == 2) + movq %r10, 8*\arg1(z) +.elseif (\arg1 % 8 == 3) + movq %r11, 8*\arg1(z) +.elseif (\arg1 % 8 == 4) + movq %r12, 8*\arg1(z) +.elseif (\arg1 % 8 == 5) + movq %r13, 8*\arg1(z) +.elseif (\arg1 % 8 == 6) + movq %r14, 8*\arg1(z) +.elseif (\arg1 % 8 == 7) + movq %r15, 8*\arg1(z) +.endif + + mulpadd 1, \arg1 + mulpadd 2, \arg1 + mulpadd 3, \arg1 + mulpadd 4, \arg1 + mulpadd 5, \arg1 + mulpadd 6, \arg1 + mulpade 7, \arg1 + +.if (\arg1 % 8 == 0) + adcq zero, %r8 +.elseif (\arg1 % 8 == 1) + adcq zero, %r9 +.elseif (\arg1 % 8 == 2) + adcq zero, %r10 +.elseif (\arg1 % 8 == 3) + adcq zero, %r11 +.elseif (\arg1 % 8 == 4) + adcq zero, %r12 +.elseif (\arg1 % 8 == 5) + adcq zero, %r13 +.elseif (\arg1 % 8 == 6) + adcq zero, %r14 +.elseif (\arg1 % 8 == 7) + adcq zero, %r15 +.endif + +.endm + + +S2N_BN_SYMBOL(bignum_mul_8_16): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi + movq %r8, %rdx +#endif + +// Save more registers to play with + + CFI_PUSH(%rbp) + CFI_PUSH(%rbx) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + +// Copy y into a safe register to start with + + movq %rdx, y + +// Zero a register, which also makes sure we don't get a fake carry-in + + xorl zeroe, zeroe + +// Do the zeroth row, which is a bit different +// Write back the zero-zero product and then accumulate +// %r8,%r15,%r14,%r13,%r12,%r11,%r10,%r9 as y[0] * x from 1..8 + + movq (y), %rdx + + mulxq (x), %r8, %r9 + movq %r8, (z) + + mulxq 8(x), %rbx, %r10 + adcq %rbx, %r9 + + mulxq 16(x), %rbx, %r11 + adcq %rbx, %r10 + + mulxq 24(x), %rbx, %r12 + adcq %rbx, %r11 + + mulxq 32(x), %rbx, %r13 + adcq %rbx, %r12 + + mulxq 40(x), %rbx, %r14 + adcq %rbx, %r13 + + mulxq 48(x), %rbx, %r15 + adcq %rbx, %r14 + + mulxq 56(x), %rbx, %r8 + adcq %rbx, %r15 + adcq zero, %r8 + +// Now all the other rows in a uniform pattern + + addrow 1 + addrow 2 + addrow 3 + addrow 4 + addrow 5 + addrow 6 + addrow 7 + +// Now write back the additional columns + + movq %r8, 64(z) + movq %r9, 72(z) + movq %r10, 80(z) + movq %r11, 88(z) + movq %r12, 96(z) + movq %r13, 104(z) + movq %r14, 112(z) + movq %r15, 120(z) + +// Real epilog + + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbx) + CFI_POP(%rbp) + +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_mul_8_16) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/x86_att/bignum_sqr.S b/cbits/s2n/x86_att/bignum_sqr.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/bignum_sqr.S @@ -0,0 +1,190 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Square z := x^2 +// Input x[n]; output z[k] +// +// extern void bignum_sqr(uint64_t k, uint64_t *z, uint64_t n, const uint64_t *x); +// +// Does the "z := x^2" operation where x is n digits and result z is k. +// Truncates the result in general unless k >= 2 * n +// +// Standard x86-64 ABI: RDI = k, RSI = z, RDX = n, RCX = x +// Microsoft x64 ABI: RCX = k, RDX = z, R8 = n, R9 = x +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_sqr) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_sqr) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_sqr) + .text + .balign 32 + +// First three are where arguments come in, but n is moved. + +#define p %rdi +#define z %rsi +#define x %rcx +#define n %r8 + +// These are always local scratch since multiplier result is in these + +#define a %rax +#define d %rdx + +// Other variables + +#define i %rbx +#define ll %rbp +#define hh %r9 +#define k %r10 +#define y %r11 +#define htop %r12 +#define l %r13 +#define h %r14 +#define c %r15 + +// Short versions + +#define llshort %ebp + +S2N_BN_SYMBOL(bignum_sqr): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi + movq %r8, %rdx + movq %r9, %rcx +#endif + +// We use too many registers, and also we need %rax:%rdx for multiplications + + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + movq %rdx, n + +// If p = 0 the result is trivial and nothing needs doing + + testq p, p + jz Lbignum_sqr_end + +// initialize (hh,ll) = 0 + + xorl llshort, llshort + xorq hh, hh + +// Iterate outer loop from k = 0 ... k = p - 1 producing result digits + + xorq k, k + +Lbignum_sqr_outerloop: + +// First let bot = MAX 0 (k + 1 - n) and top = MIN (k + 1) n +// We want to accumulate all x[i] * x[k - i] for bot <= i < top +// For the optimization of squaring we avoid duplication and do +// 2 * x[i] * x[k - i] for i < htop, where htop = MIN ((k+1)/2) n +// Initialize i = bot; in fact just compute bot as i directly. + + xorq c, c + leaq 1(k), i + movq i, htop + shrq $1, htop + subq n, i + cmovcq c, i + cmpq n, htop + cmovncq n, htop + +// Initialize the three-part local sum (c,h,l); c was already done above + + xorq l, l + xorq h, h + +// If htop <= bot then main doubled part of the sum is empty + + cmpq htop, i + jnc Lbignum_sqr_nosumming + +// Use a moving pointer for [y] = x[k-i] for the cofactor + + movq k, a + subq i, a + leaq (x,a,8), y + +// Do the main part of the sum x[i] * x[k - i] for 2 * i < k + +Lbignum_sqr_innerloop: + movq (x,i,8), a + mulq (y) + addq a, l + adcq d, h + adcq $0, c + subq $8, y + incq i + cmpq htop, i + jc Lbignum_sqr_innerloop + +// Now double it + + addq l, l + adcq h, h + adcq c, c + +// If k is even (which means 2 * i = k) and i < n add the extra x[i]^2 term + +Lbignum_sqr_nosumming: + testq $1, k + jnz Lbignum_sqr_innerend + cmpq n, i + jnc Lbignum_sqr_innerend + + movq (x,i,8), a + mulq a + addq a, l + adcq d, h + adcq $0, c + +// Now add the local sum into the global sum, store and shift + +Lbignum_sqr_innerend: + addq ll, l + movq l, (z,k,8) + adcq hh, h + movq h, ll + adcq $0, c + movq c, hh + + incq k + cmpq p, k + jc Lbignum_sqr_outerloop + +// Restore registers and return + +Lbignum_sqr_end: + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_sqr) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/x86_att/bignum_sqr_8_16.S b/cbits/s2n/x86_att/bignum_sqr_8_16.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/bignum_sqr_8_16.S @@ -0,0 +1,302 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Square, z := x^2 +// Input x[8]; output z[16] +// +// extern void bignum_sqr_8_16(uint64_t z[static 16], const uint64_t x[static 8]); +// +// Standard x86-64 ABI: RDI = z, RSI = x +// Microsoft x64 ABI: RCX = z, RDX = x +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_sqr_8_16) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_sqr_8_16) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_sqr_8_16) + .text + +// These are actually right + +#define z %rdi +#define x %rsi + +// A zero register + +#define zero %rbp +#define zeroe %ebp + +// mulpadd i, j adds rdx * x[i] into the window at the i+j point + +.macro mulpadd arg1,arg2 + mulxq 8*\arg1(x), %rax, %rcx +.if ((\arg1 + \arg2) % 8 == 0) + adcxq %rax, %r8 + adoxq %rcx, %r9 +.elseif ((\arg1 + \arg2) % 8 == 1) + adcxq %rax, %r9 + adoxq %rcx, %r10 +.elseif ((\arg1 + \arg2) % 8 == 2) + adcxq %rax, %r10 + adoxq %rcx, %r11 +.elseif ((\arg1 + \arg2) % 8 == 3) + adcxq %rax, %r11 + adoxq %rcx, %r12 +.elseif ((\arg1 + \arg2) % 8 == 4) + adcxq %rax, %r12 + adoxq %rcx, %r13 +.elseif ((\arg1 + \arg2) % 8 == 5) + adcxq %rax, %r13 + adoxq %rcx, %r14 +.elseif ((\arg1 + \arg2) % 8 == 6) + adcxq %rax, %r14 + adoxq %rcx, %r15 +.elseif ((\arg1 + \arg2) % 8 == 7) + adcxq %rax, %r15 + adoxq %rcx, %r8 +.endif + +.endm + +// mulpade i, j adds rdx * x[i] into the window at i+j +// but re-creates the top word assuming nothing to add there + +.macro mulpade arg1,arg2 +.if ((\arg1 + \arg2) % 8 == 0) + mulxq 8*\arg1(x), %rax, %r9 + adcxq %rax, %r8 + adoxq zero, %r9 +.elseif ((\arg1 + \arg2) % 8 == 1) + mulxq 8*\arg1(x), %rax, %r10 + adcxq %rax, %r9 + adoxq zero, %r10 +.elseif ((\arg1 + \arg2) % 8 == 2) + mulxq 8*\arg1(x), %rax, %r11 + adcxq %rax, %r10 + adoxq zero, %r11 +.elseif ((\arg1 + \arg2) % 8 == 3) + mulxq 8*\arg1(x), %rax, %r12 + adcxq %rax, %r11 + adoxq zero, %r12 +.elseif ((\arg1 + \arg2) % 8 == 4) + mulxq 8*\arg1(x), %rax, %r13 + adcxq %rax, %r12 + adoxq zero, %r13 +.elseif ((\arg1 + \arg2) % 8 == 5) + mulxq 8*\arg1(x), %rax, %r14 + adcxq %rax, %r13 + adoxq zero, %r14 +.elseif ((\arg1 + \arg2) % 8 == 6) + mulxq 8*\arg1(x), %rax, %r15 + adcxq %rax, %r14 + adoxq zero, %r15 +.elseif ((\arg1 + \arg2) % 8 == 7) + mulxq 8*\arg1(x), %rax, %r8 + adcxq %rax, %r15 + adoxq zero, %r8 +.endif + +.endm + +.macro diagonals + + xorl zeroe, zeroe + +// Set initial window [%r8..%r10] + 2 wb = 10 + 20 + 30 + 40 + 50 + 60 + 70 + + movq (x), %rdx + mulxq 8(x), %r9, %rax + movq %r9, 8(z) + mulxq 16(x), %r10, %rcx + adcxq %rax, %r10 + movq %r10, 16(z) + mulxq 24(x), %r11, %rax + adcxq %rcx, %r11 + mulxq 32(x), %r12, %rcx + adcxq %rax, %r12 + mulxq 40(x), %r13, %rax + adcxq %rcx, %r13 + mulxq 48(x), %r14, %rcx + adcxq %rax, %r14 + mulxq 56(x), %r15, %r8 + adcxq %rcx, %r15 + adcxq zero, %r8 + +// Add in the next diagonal = 21 + 31 + 41 + 51 + 61 + 71 + 54 + + xorl zeroe, zeroe + movq 8(x), %rdx + mulpadd 2, 1 + movq %r11, 24(z) + mulpadd 3, 1 + movq %r12, 32(z) + mulpadd 4, 1 + mulpadd 5, 1 + mulpadd 6, 1 + mulpade 7, 1 + movq 32(x), %rdx + mulpade 5, 4 + adcxq zero, %r10 + +// And the next one = 32 + 42 + 52 + 62 + 72 + 64 + 65 + + xorl zeroe, zeroe + movq 16(x), %rdx + mulpadd 3, 2 + movq %r13, 40(z) + mulpadd 4, 2 + movq %r14, 48(z) + mulpadd 5, 2 + mulpadd 6, 2 + mulpadd 7, 2 + movq 48(x), %rdx + mulpade 4, 6 + mulpade 5, 6 + adcxq zero, %r12 + +// And the final one = 43 + 53 + 63 + 73 + 74 + 75 + 76 + + xorl zeroe, zeroe + movq 24(x), %rdx + mulpadd 4, 3 + movq %r15, 56(z) + mulpadd 5, 3 + movq %r8, 64(z) + mulpadd 6, 3 + mulpadd 7, 3 + movq 56(x), %rdx + mulpadd 4, 7 + mulpade 5, 7 + mulpade 6, 7 + adcxq zero, %r14 + +// Double and add things; use z[1]..z[8] and thereafter the registers +// %r9..%r15 which haven't been written back yet + + xorl zeroe, zeroe + movq (x), %rdx + mulxq %rdx, %rax, %rcx + movq %rax, (z) + movq 8(z), %rax + adcxq %rax, %rax + adoxq %rcx, %rax + movq %rax, 8(z) + + movq 16(z), %rax + movq 8(x), %rdx + mulxq %rdx, %rdx, %rcx + adcxq %rax, %rax + adoxq %rdx, %rax + movq %rax, 16(z) + movq 24(z), %rax + adcxq %rax, %rax + adoxq %rcx, %rax + movq %rax, 24(z) + + movq 32(z), %rax + movq 16(x), %rdx + mulxq %rdx, %rdx, %rcx + adcxq %rax, %rax + adoxq %rdx, %rax + movq %rax, 32(z) + movq 40(z), %rax + adcxq %rax, %rax + adoxq %rcx, %rax + movq %rax, 40(z) + + movq 48(z), %rax + movq 24(x), %rdx + mulxq %rdx, %rdx, %rcx + adcxq %rax, %rax + adoxq %rdx, %rax + movq %rax, 48(z) + movq 56(z), %rax + adcxq %rax, %rax + adoxq %rcx, %rax + movq %rax, 56(z) + + movq 64(z), %rax + movq 32(x), %rdx + mulxq %rdx, %rdx, %rcx + adcxq %rax, %rax + adoxq %rdx, %rax + movq %rax, 64(z) + adcxq %r9, %r9 + adoxq %rcx, %r9 + movq %r9, 72(z) + + movq 40(x), %rdx + mulxq %rdx, %rdx, %rcx + adcxq %r10, %r10 + adoxq %rdx, %r10 + movq %r10, 80(z) + adcxq %r11, %r11 + adoxq %rcx, %r11 + movq %r11, 88(z) + + movq 48(x), %rdx + mulxq %rdx, %rdx, %rcx + adcxq %r12, %r12 + adoxq %rdx, %r12 + movq %r12, 96(z) + adcxq %r13, %r13 + adoxq %rcx, %r13 + movq %r13, 104(z) + + movq 56(x), %rdx + mulxq %rdx, %rdx, %r15 + adcxq %r14, %r14 + adoxq %rdx, %r14 + movq %r14, 112(z) + adcxq zero, %r15 + adoxq zero, %r15 + movq %r15, 120(z) + +.endm + + +S2N_BN_SYMBOL(bignum_sqr_8_16): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi +#endif + +// Save more registers to play with + + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + +// Do the multiplication + + diagonals + +// Real epilog + + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_sqr_8_16) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/tests/bearssl_diff.c b/cbits/tests/bearssl_diff.c new file mode 100644 --- /dev/null +++ b/cbits/tests/bearssl_diff.c @@ -0,0 +1,432 @@ +/* + * crypton's portable AES and GHASH against the BearSSL they are written on. + * + * This began as the migration check: for one commit, cbits/aes/generic.c and + * cbits/aes/gf.c were still the table-driven implementations, and this said + * the vendored code computed what they computed before they were replaced. + * + * Since the replacement the two sides are no longer independent, and what is + * left is still worth checking: everything in generic.c and gf.c is now + * crypton's own glue -- a schedule kept compressed and carried through + * memcpy, the interleave-and-ortho idiom around a single block, three of four + * lanes left idle, and a GHASH entry that reaches the same multiply by handing + * it a block of zeros. Each of those is somewhere a mistake would live, and + * each is compared here against calling BearSSL directly. + * + * FIPS-197's own vectors come first either way, so that agreement means AES + * rather than two callers agreeing on something that is not. + * + * Run with an argument to corrupt results on purpose and see the comparison + * notice: a differential test that cannot fail has said nothing. + */ +#include +#include +#include +#include + +#include "bearssl/inner.h" +#include "crypton_aes.h" +#include "aes/generic.h" +#include "aes/gf.h" +#include "aes/block128.h" + +/* + * crypton_aes.c's table, which is a global, and the three key sizes of the + * two CTR entries this looks for in it. Searching rather than indexing + * because the index is an enum private to that file. + */ +#define BRANCH_TABLE_SEARCH 64 +extern void *crypton_aes_branch_table[]; + +/* the two GCM implementations, which crypton_aes.c declares but no header + * does: in this build the generic one reaches the portable block function + * too, so the two have to agree exactly, tag and all */ +void crypton_aes_generic_gcm_encrypt(uint8_t *, aes_gcm *, aes_key *, uint8_t *, uint32_t); +void crypton_aes_generic_gcm_decrypt(uint8_t *, aes_gcm *, aes_key *, uint8_t *, uint32_t); +void crypton_aes_bitsliced_gcm_encrypt(uint8_t *, aes_gcm *, aes_key *, uint8_t *, uint32_t); +void crypton_aes_bitsliced_gcm_decrypt(uint8_t *, aes_gcm *, aes_key *, uint8_t *, uint32_t); + +/* ---- the vendored code, one block at a time, as aes_ct64_cbcenc.c does ---- */ + +static void bear_key(uint64_t *comp_skey, unsigned *nr, + const uint8_t *key, size_t len) +{ + *nr = br_aes_ct64_keysched(comp_skey, key, len); +} + +static void bear_block(uint8_t *out, const uint64_t *comp_skey, unsigned nr, + const uint8_t *in, int decrypt) +{ + uint64_t sk_exp[120]; + uint32_t w[4]; + uint64_t q[8]; + + br_aes_ct64_skey_expand(sk_exp, nr, comp_skey); + w[0] = br_dec32le(in); + w[1] = br_dec32le(in + 4); + w[2] = br_dec32le(in + 8); + w[3] = br_dec32le(in + 12); + memset(q, 0, sizeof q); + br_aes_ct64_interleave_in(&q[0], &q[4], w); + br_aes_ct64_ortho(q); + if (decrypt) + br_aes_ct64_bitslice_decrypt(nr, sk_exp, q); + else + br_aes_ct64_bitslice_encrypt(nr, sk_exp, q); + br_aes_ct64_ortho(q); + br_aes_ct64_interleave_out(w, q[0], q[4]); + br_enc32le(out, w[0]); + br_enc32le(out + 4, w[1]); + br_enc32le(out + 8, w[2]); + br_enc32le(out + 12, w[3]); +} + +/* ---- crypton's GHASH, driven the way crypton_aes.c drives it ---- */ + +static void crypton_ghash(uint8_t *y, const uint8_t *h, + const uint8_t *data, size_t len) +{ + table_4bit ht; + block128 acc; + size_t i; + + crypton_aes_generic_hinit(ht, (const block128 *) h); + block128_zero(&acc); + for (i = 0; i < len; i += 16) { + block128_xor_bytes(&acc, data + i, 16); + crypton_aes_generic_gf_mul(&acc, ht); + } + memcpy(y, &acc, 16); +} + +/* ---- the comparison ---- */ + +static int failures; +static int sabotage; + +static void same(const char *what, const uint8_t *a, const uint8_t *b, size_t n) +{ + if (memcmp(a, b, n) != 0) { + size_t i; + + printf(" MISMATCH %s\n bearssl ", what); + for (i = 0; i < n; i++) printf("%02x", a[i]); + printf("\n crypton "); + for (i = 0; i < n; i++) printf("%02x", b[i]); + printf("\n"); + failures++; + } +} + +static uint32_t rnd_state = 1; +static uint8_t rnd(void) +{ + rnd_state = rnd_state * 1103515245u + 12345u; + return (uint8_t)(rnd_state >> 16); +} +static void rnd_fill(uint8_t *p, size_t n) +{ + while (n--) *p++ = rnd(); +} + +int main(int argc, char **argv) +{ + /* FIPS-197 C.1, C.2, C.3 */ + static const uint8_t pt[16] = { + 0x00,0x11,0x22,0x33,0x44,0x55,0x66,0x77, + 0x88,0x99,0xaa,0xbb,0xcc,0xdd,0xee,0xff }; + static const uint8_t k128[16] = { + 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15 }; + static const uint8_t k192[24] = { + 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23 }; + static const uint8_t k256[32] = { + 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15, + 16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31 }; + static const uint8_t c128[16] = { + 0x69,0xc4,0xe0,0xd8,0x6a,0x7b,0x04,0x30, + 0xd8,0xcd,0xb7,0x80,0x70,0xb4,0xc5,0x5a }; + static const uint8_t c192[16] = { + 0xdd,0xa9,0x7c,0xa4,0x86,0x4c,0xdf,0xe0, + 0x6e,0xaf,0x70,0xa0,0xec,0x0d,0x71,0x91 }; + static const uint8_t c256[16] = { + 0x8e,0xa2,0xb7,0xca,0x51,0x67,0x45,0xbf, + 0xea,0xfc,0x49,0x90,0x4b,0x49,0x60,0x89 }; + const uint8_t *keys[3] = { k128, k192, k256 }; + const uint8_t *cts[3] = { c128, c192, c256 }; + size_t klens[3] = { 16, 24, 32 }; + uint64_t comp[30]; + unsigned nr; + uint8_t a[16], b[16]; + int i, round; + + sabotage = argc > 1; + printf("== FIPS-197, so that agreement means AES ==\n"); + for (i = 0; i < 3; i++) { + bear_key(comp, &nr, keys[i], klens[i]); + bear_block(a, comp, nr, pt, 0); + if (sabotage && i == 1) a[0] ^= 1; + same("bearssl vs FIPS-197", a, cts[i], 16); + bear_block(b, comp, nr, cts[i], 1); + same("bearssl decrypt vs plaintext", b, pt, 16); + } + + printf("== crypton's glue vs BearSSL direct, 2000 random keys and blocks ==\n"); + for (round = 0; round < 2000; round++) { + uint8_t key[32], in[16], e1[16], e2[16], d1[16], d2[16]; + size_t kl = klens[round % 3]; + aes_key ck; + + rnd_fill(key, kl); + rnd_fill(in, 16); + + bear_key(comp, &nr, key, kl); + bear_block(e1, comp, nr, in, 0); + bear_block(d1, comp, nr, in, 1); + + crypton_aes_generic_init(&ck, key, (uint8_t) kl); + crypton_aes_generic_encrypt_block((aes_block *) e2, &ck, + (aes_block *) in); + crypton_aes_generic_decrypt_block((aes_block *) d2, &ck, + (aes_block *) in); + if (sabotage && round == 7) e1[3] ^= 0x10; + same("encrypt", e1, e2, 16); + same("decrypt", d1, d2, 16); + } + + printf("== GHASH: crypton's entries vs br_ghash_ctmul64, 500 messages ==\n"); + for (round = 0; round < 500; round++) { + uint8_t h[16], data[256], y1[16], y2[16]; + size_t len = 16u * (size_t)(1 + (round % 16)); + + rnd_fill(h, 16); + rnd_fill(data, len); + + memset(y1, 0, 16); + br_ghash_ctmul64(y1, h, data, len); + crypton_ghash(y2, h, data, len); + if (sabotage && round == 3) y1[15] ^= 0x80; + same("ghash", y1, y2, 16); + } + + printf("== gf_mul4: the four-block entry against the same four blocks ==\n"); + for (round = 0; round < 500; round++) { + uint8_t h[16], data[64], y1[16]; + table_4bit ht; + block128 acc; + + rnd_fill(h, 16); + rnd_fill(data, sizeof data); + + memset(y1, 0, 16); + br_ghash_ctmul64(y1, h, data, sizeof data); + + crypton_aes_generic_hinit(ht, (const block128 *) h); + block128_zero(&acc); + crypton_aes_generic_gf_mul4(&acc, (const block128 *) data, ht); + if (sabotage && round == 11) y1[0] ^= 0x40; + same("gf_mul4", y1, (const uint8_t *) &acc, 16); + } + + /* + * And that the wide entries are the ones installed. Nothing else + * here would notice if they were not: the entries they replace are + * correct too, just a block at a time, so every answer above would + * be the same and only the speed would be gone. + */ + /* + * The Haskell suite reaches the four-block pass, but thinly: its + * vectors are mostly a block or three long, and breaking that pass + * alone fails sixteen of its examples where breaking every pass + * fails seven hundred. So the lane logic and the counter are + * covered here instead, where the lengths can be chosen. + */ + printf("== many blocks at a pass against one at a time ==\n"); + for (round = 0; round < 200; round++) { + uint8_t key[32], in[16 * 9], wide[16 * 9], single[16 * 9]; + size_t kl = klens[round % 3]; + uint32_t nb = 1 + (round % 9); + aes_sched sched; + aes_key ck; + uint32_t i; + int dec = round & 1; + + rnd_fill(key, kl); + rnd_fill(in, nb * 16); + crypton_aes_generic_init(&ck, key, (uint8_t) kl); + + crypton_aes_generic_schedule(&sched, &ck); + crypton_aes_generic_blocks(wide, in, nb, &sched, dec); + + for (i = 0; i < nb; i++) { + if (dec) + crypton_aes_generic_decrypt_block( + (aes_block *) (single + 16 * i), &ck, + (aes_block *) (in + 16 * i)); + else + crypton_aes_generic_encrypt_block( + (aes_block *) (single + 16 * i), &ck, + (aes_block *) (in + 16 * i)); + } + if (sabotage && round == 5) wide[16] ^= 2; + same("wide pass", wide, single, nb * 16); + } + + printf("== the four-block CTR against one block at a time ==\n"); + for (round = 0; round < 200; round++) { + uint8_t key[32], iv[16], in[200], got[200], want[200]; + size_t kl = klens[round % 3]; + /* lengths that land on, before and after a group of four */ + uint32_t len = 1 + (round % 200); + aes_key ck; + aes_block counter, ks; + uint32_t done, n, i; + int c32 = round & 1; + + rnd_fill(key, kl); + rnd_fill(iv, 16); + rnd_fill(in, len); + crypton_aes_generic_init(&ck, key, (uint8_t) kl); + + if (c32) + crypton_aes_bitsliced_encrypt_c32(got, &ck, + (aes_block *) iv, in, len); + else + crypton_aes_bitsliced_encrypt_ctr(got, &ck, + (aes_block *) iv, in, len); + + block128_copy(&counter, (block128 *) iv); + for (done = 0; done < len; done += 16) { + crypton_aes_generic_encrypt_block(&ks, &ck, &counter); + n = len - done < 16 ? len - done : 16; + for (i = 0; i < n; i++) + want[done + i] = ((uint8_t *) &ks)[i] + ^ in[done + i]; + if (c32) + block128_inc32_le(&counter); + else + block128_inc_be(&counter); + } + if (sabotage && round == 9) got[len - 1] ^= 4; + same("ctr", got, want, len); + } + + printf("== XTS four at a pass against one at a time ==\n"); + for (round = 0; round < 200; round++) { + uint8_t key[32], key2[32], du[16], in[16 * 11]; + uint8_t got[16 * 11], want[16 * 11]; + size_t kl = klens[round % 3]; + uint32_t nb = 1 + (round % 11); + uint32_t spoint = round % 3; + aes_key k1, k2; + aes_block tweak; + uint32_t i, j; + int dec = round & 1; + + rnd_fill(key, kl); + rnd_fill(key2, kl); + rnd_fill(du, 16); + rnd_fill(in, nb * 16); + crypton_aes_initkey(&k1, key, (uint8_t) kl); + crypton_aes_initkey(&k2, key2, (uint8_t) kl); + + { + aes_block d; + memcpy(&d, du, 16); + if (dec) + crypton_aes_decrypt_xts((aes_block *) got, &k1, &k2, + &d, spoint, (aes_block *) in, nb); + else + crypton_aes_encrypt_xts((aes_block *) got, &k1, &k2, + &d, spoint, (aes_block *) in, nb); + } + + /* the same thing a block at a time */ + memcpy(&tweak, du, 16); + crypton_aes_generic_encrypt_block(&tweak, &k2, &tweak); + for (j = 0; j < spoint; j++) + crypton_aes_generic_gf_mulx((block128 *) &tweak); + for (i = 0; i < nb; i++) { + aes_block t; + + block128_vxor(&t, (block128 *) (in + 16 * i), &tweak); + if (dec) + crypton_aes_generic_decrypt_block(&t, &k1, &t); + else + crypton_aes_generic_encrypt_block(&t, &k1, &t); + block128_vxor((block128 *) (want + 16 * i), &t, &tweak); + crypton_aes_generic_gf_mulx((block128 *) &tweak); + } + if (sabotage && round == 17) got[16] ^= 8; + same("xts", got, want, nb * 16); + } + + printf("== the four-block GCM against the one-block GCM ==\n"); + for (round = 0; round < 200; round++) { + uint8_t key[32], iv[12], in[300], ga[300], gb[300]; + size_t kl = klens[round % 3]; + uint32_t len = 1 + (round % 300); + aes_key ck; + aes_gcm g1, g2; + int dec = round & 1; + + rnd_fill(key, kl); + rnd_fill(iv, sizeof iv); + rnd_fill(in, len); + crypton_aes_initkey(&ck, key, (uint8_t) kl); + crypton_aes_gcm_init(&g1, &ck, iv, sizeof iv); + memcpy(&g2, &g1, sizeof g1); + + if (dec) { + crypton_aes_generic_gcm_decrypt(ga, &g1, &ck, in, len); + crypton_aes_bitsliced_gcm_decrypt(gb, &g2, &ck, in, len); + } else { + crypton_aes_generic_gcm_encrypt(ga, &g1, &ck, in, len); + crypton_aes_bitsliced_gcm_encrypt(gb, &g2, &ck, in, len); + } + if (sabotage && round == 13) gb[0] ^= 1; + same("gcm text", ga, gb, len); + /* the running GHASH and counter, which the tag is made from */ + same("gcm state", (const uint8_t *) &g1, (const uint8_t *) &g2, + sizeof g1); + } + + printf("== which implementation the build chose ==\n"); +#if !defined(WITH_AESNI) && !defined(WITH_ARMV8_CRYPTO) + /* + * With no accelerator compiled in, crypton_aes.c does not read the + * branch table at all -- its GET_ macros name the portable entries + * directly -- so there is nothing here to look at, and which + * implementation runs is settled by the preprocessor. Scanning the + * table in this build was a check that passed while telling nothing, + * which is how the four-block CTR came to be written, installed, and + * never called. + */ + printf(" named at compile time; the table is not read in this build\n"); + (void) crypton_aes_branch_table; + if (sabotage) { /* nothing to corrupt here */ } +#else + { + int i, ctr = 0, c32 = 0; + + for (i = 0; i < BRANCH_TABLE_SEARCH; i++) { + if (crypton_aes_branch_table[i] == + (void *) crypton_aes_bitsliced_encrypt_ctr) + ctr++; + if (crypton_aes_branch_table[i] == + (void *) crypton_aes_bitsliced_encrypt_c32) + c32++; + } + printf(" CTR entries %d, C32 entries %d\n", ctr, c32); + if (sabotage) { ctr = 0; } + if (ctr != 3 || c32 != 3) { + printf(" MISMATCH the portable CTR entries were not installed\n"); + failures++; + } + } +#endif + + printf("%s: %d mismatch(es)%s\n", + failures ? "FAIL" : "ok", failures, + sabotage ? " (sabotage was asked for)" : ""); + return failures != 0; +} diff --git a/cbits/tests/ct/README b/cbits/tests/ct/README --- a/cbits/tests/ct/README +++ b/cbits/tests/ct/README @@ -26,29 +26,33 @@ chapoly the ChaCha20 key, the plaintext, and the Poly1305 key aes the AES key and the plaintext, through ECB and GCM aes_armv8 the same driver again, on AArch64, against the instructions + aes_x86ni the same driver again, on x86-64, against AES-NI -The aes driver is built against cbits/aes/generic.c and cbits/aes/gf.c on -purpose, rather than whatever the machine offers. AES-NI and the ARMv8 -instructions do not look anything up and would report nothing, which would say -nothing about the table-driven code every other machine runs. That code is -variable-time by construction -- a table index is a byte of the state -- and -so is the table-driven GHASH beside it. A report from the aes driver is -therefore expected and is a property of those implementations, not a defect -found in them; it is here so that the size of it is written down rather than -assumed. Everything else is expected to be silent. +One driver, built three times, against the portable C and against each of +the two instruction sets. All three must report nothing at all -- not +"nothing outside known.txt", nothing. The portable AES and GHASH are +bitsliced and table-free; AESE, AESMC, PMULL, AESENC and PCLMULQDQ look +nothing up. None of the three has anywhere for a secret to decide a branch +or an address. -On AArch64 the same driver is then built a second time, as aes_armv8, against -cbits/aes/armv8.c with the crypto extension turned on. AESE, AESMC and PMULL -look nothing up and branch on nothing, so that run must report nothing at all --- not "nothing outside known.txt", nothing; a table site appearing there -would mean the dispatch had not picked the instructions. The two runs keep -each other honest: the table-driven one has to report and the instruction one -has to be silent, and either going the wrong way says the run is not -measuring what it claims to. +This entry used to read the other way round. The portable implementation +was table-driven -- an S-box indexed by a byte of the state, and Shoup's +4-bit table for the GHASH -- so the aes driver was *required* to report, and +known.txt carried four entries saying so. That requirement was also what +kept the pair honest: if the portable build had quietly taken an accelerated +path it would have fallen silent, and the silence would have failed it. -Until that was added the harness ran only on x86-64, so crypton's AArch64 AES -and GHASH had never been put to it -- which was noticed when the GHASH was -rewritten. +With the tables gone that check goes too, so the question is now asked +outright rather than read off a leak. Each build of the driver prints which +implementation its dispatch took, and run.sh requires the answer it expects +before it looks at anything else. A build that takes the wrong path fails +on that line rather than passing by saying nothing. + +The x86-64 build is new with the rewrite. Before it, AES-NI and PCLMULQDQ +had never been put to this harness at all: their being constant time was an +inference from the instruction specifications rather than a measurement of +crypton's code. The AArch64 build came earlier, when the GHASH was +rewritten and it was noticed that the harness had only ever run on x86-64. What round eight found ---------------------- diff --git a/cbits/tests/ct/cpu_stub.c b/cbits/tests/ct/cpu_stub.c new file mode 100644 --- /dev/null +++ b/cbits/tests/ct/cpu_stub.c @@ -0,0 +1,19 @@ +/* + * One symbol, so that cbits/crypton_cpu.c can be linked on its own. + * + * The ML-KEM backend asks crypton_cpu.c whether this processor has AVX2. + * That file also carries crypton_cpu_option, which the Haskell side calls + * and which reaches into cbits/crypton_aes.c, so the linker wants + * crypton_aes_cpu_init as well -- and behind it the whole AES build, which + * has nothing to do with what is being measured here. This stands in for + * it. Nothing in an ML-KEM driver calls it; if anything ever did, zeroes + * say "no AES instructions", which is the safe answer. + */ +#include +#include "crypton_cpu.h" + +uint8_t *crypton_aes_cpu_init(void) +{ + static uint8_t none[CPU_OPTION_COUNT]; + return none; +} diff --git a/cbits/tests/ct/ct_aes.c b/cbits/tests/ct/ct_aes.c --- a/cbits/tests/ct/ct_aes.c +++ b/cbits/tests/ct/ct_aes.c @@ -5,13 +5,24 @@ * is table-driven and is variable-time by construction, which is a property * of that code rather than a defect in it. See cbits/tests/ct/README. */ #include "tests/ct/ct.h" +#include #include #include "crypton_aes.h" +#include "crypton_cpu.h" +/* crypton_aes.c fills this; run.sh reads the line below to tell the three + * builds of this driver apart. They are all silent now, so which one took + * which implementation cannot be read off a leak any more. */ +uint8_t *crypton_aes_cpu_init(void); + int main(void) { aes_key k; aes_gcm_key gk; uint8_t key[32], pt[256], ct[256 + 16], iv[12]; + + printf("dispatch aes=%d pclmul=%d\n", + crypton_aes_cpu_init()[CPU_AESNI] != 0, + crypton_aes_cpu_init()[CPU_PCLMUL] != 0); ct_fill(key, sizeof key); ct_fill(pt, sizeof pt); diff --git a/cbits/tests/ct/ct_powm.c b/cbits/tests/ct/ct_powm.c --- a/cbits/tests/ct/ct_powm.c +++ b/cbits/tests/ct/ct_powm.c @@ -4,8 +4,34 @@ #include #include "crypton_powm.h" +/* A 2048-bit modulus unless the build says otherwise: run.sh builds it at + * the other sizes the vendored multiplication takes as well, since those + * are different code. */ +#ifndef CT_POWM_BYTES +#define CT_POWM_BYTES 256 +#endif + +#ifdef CT_POWM_COUNT_S2N +/* run.sh links this build with -Wl,--wrap=bignum_emontredc_8n, so every + * reduction the s2n-bignum path does comes through here and is counted. + * The build then says which implementation it ran, rather than leaving it + * to be inferred: under valgrind the processor's answers are valgrind's, + * and a build that quietly took the C would pass here while checking what + * the plain powm build already has. */ +#include +uint64_t __real_bignum_emontredc_8n(uint64_t k, uint64_t *z, + const uint64_t *m, uint64_t w); +static unsigned long s2n_reductions; +uint64_t __wrap_bignum_emontredc_8n(uint64_t k, uint64_t *z, + const uint64_t *m, uint64_t w) +{ + s2n_reductions++; + return __real_bignum_emontredc_8n(k, z, m, w); +} +#endif + int main(void) { - enum { LEN = 256 }; /* a 2048-bit modulus */ + enum { LEN = CT_POWM_BYTES }; uint8_t out[LEN], base[LEN], mod[LEN], exp[LEN]; ct_fill(base, sizeof base); @@ -25,5 +51,8 @@ } CT_PUBLIC(out, sizeof out); ct_sink(out, sizeof out); +#ifdef CT_POWM_COUNT_S2N + printf("dispatch powm_s2n=%lu\n", s2n_reductions); +#endif return 0; } diff --git a/cbits/tests/ct/known.txt b/cbits/tests/ct/known.txt --- a/cbits/tests/ct/known.txt +++ b/cbits/tests/ct/known.txt @@ -22,15 +22,6 @@ # arithmetic, and it too goes the same way on every valid input. decaf.c:136 assert(ret) in crypton_gf_invert -# The table-driven AES and the table-driven GHASH index with a byte of the -# state, which is what makes them fast and what makes them variable-time. -# That is a property of those implementations rather than a defect in them; -# a machine with AES-NI or the ARMv8 instructions runs neither. -generic.c the AES tables, in key expansion and in the rounds -gf.c the GHASH table -crypton_aes.c the same tables, attributed to the code that inlines them -block128.h likewise - # AArch64 only. gcc keeps a carry in the flags and takes it out with `cset` # or `cinc`, where on x86-64 it uses `adc` and the carry never leaves the # data path. memcheck calls `cset` a conditional move and reports it, and @@ -49,6 +40,27 @@ # 400f64 b.ne 400f3c <- what memcheck names # 404c6c cmp x3, x6 <- public: j against n_table # 404c70 b.ne 404c30 <- what memcheck names -p256.c:147 addM's loop; the carry is taken with cset and cinc -p256.c:149 the same +addM@p256.c the carry of addM's loop, which is taken without a +# branch. Named by function rather than by line because +# the line is the compiler's choice and the function is +# not. Three compilers were asked: AArch64 names 147 and +# 149, gcc 11.5 on x86-64 names 147, 148 and 149, and the +# gcc 14 the CI uses names none of them -- and all three +# name addM. +# +# On AArch64 the only instructions consuming the secret's +# flags are cset and cinc, which do not branch. On x86-64, +# disassembled in a -no-pie build, the body between the +# loop label and its closing branch is +# +# 401ad6 mov (%rax,%rdx,1),%rbx +# 401aeb and %rcx,%rbx <- the secret, as a mask +# 401af1 add %r12,%r8 +# 401af4 adc %r13,%r9 <- carried, not branched on +# 401afd mov %rsi,0x120(%rsp,%rdx,1) +# 401b0e cmp $0x20,%rdx <- public: the loop offset +# 401b12 jne 401ad6 <- the only branch +# +# one branch, on a public counter; no conditional move, and +# no address the secret reaches. constant_time.h:150 decaf's constant-time lookup, over j < n_table diff --git a/cbits/tests/ct/run.sh b/cbits/tests/ct/run.sh --- a/cbits/tests/ct/run.sh +++ b/cbits/tests/ct/run.sh @@ -27,17 +27,25 @@ decaf_inc="-DCRYPTON_DECAF_WORD_BITS=64 -I$D/include -I$D/p448 -I$D/include/arch_ref64 -I$D/p448/arch_ref64" -# The generic C, not whatever the machine happens to offer. A build that -# takes AES-NI reports nothing from the AES driver and says nothing about the -# table-driven code every other machine runs. -aes_src="cbits/crypton_aes.c cbits/aes/generic.c cbits/aes/gf.c" +# The portable C, not whatever the machine happens to offer, and the BearSSL +# it is written on. All three builds below have to be silent, so what tells +# them apart is which implementation each one's dispatch chose -- which the +# driver prints and run_one checks, since otherwise a build that quietly took +# the wrong path would pass by saying nothing. +aes_src="cbits/crypton_aes.c cbits/aes/generic.c cbits/aes/gf.c + cbits/bearssl/aes_ct64.c cbits/bearssl/aes_ct64_enc.c + cbits/bearssl/aes_ct64_dec.c cbits/bearssl/ghash_ctmul64.c + cbits/bearssl/dec32le.c" -# And, on AArch64, the same driver again against the instructions. That one -# has to be silent; this one has to report. Either going the wrong way says -# the run is not measuring what it claims to. +# The same driver again against the instructions, on each architecture that +# has them. armv8_src="$aes_src cbits/aes/armv8.c cbits/crypton_cpu.c" armv8_inc="-DWITH_ARMV8_CRYPTO -march=armv8-a+crypto -Icbits/aes" +x86ni_src="$aes_src cbits/aes/x86ni.c cbits/crypton_cpu.c + cbits/aes/gcm_vaes_x86.c cbits/aes/gcm_vaes512_x86.c" +x86ni_inc="-DWITH_AESNI -DWITH_PCLMUL -maes -mpclmul -mssse3 -msse4.1 -Icbits/aes" + status=0 have_valgrind=no ct_define= @@ -63,10 +71,42 @@ # And the hand-written backend, where the architecture has one. Both builds # have to be silent: unlike the AES pair below there is no variable-time # implementation here for one of them to expose. -mlkem_native_src="$mlkem_src cbits/mlkem/crypton_mlkem_asm.S" +# +# crypton_cpu.c because the backend asks it whether this processor has the +# instructions before using them; without it the link fails on the +# capability function in cbits/mlkem/crypton_mlkem.h. cpu_stub.c is the one +# symbol that file needs from the AES build, which is not wanted here. +mlkem_native_src="$mlkem_src cbits/mlkem/crypton_mlkem_asm.S cbits/crypton_cpu.c \ + cbits/tests/ct/cpu_stub.c" mlkem_native_inc="$mlkem_inc -DCRYPTON_MLKEM_NATIVE_BACKEND" +# Does this processor have AES instructions at all? +# +# Asked of the system rather than of crypton, because the question being +# settled is whether crypton found what is there -- and a witness that comes +# from the code under test cannot answer that. "Cannot tell" counts as +# having them, so an unfamiliar system turns into a loud failure rather than +# a silent skip. +# +# CRYPTON_CT_CPUINFO is for testing this function itself; nothing else sets +# it. +machine_has_aes() { + cpuinfo=${CRYPTON_CT_CPUINFO:-/proc/cpuinfo} + if [ -r "$cpuinfo" ]; then + # "aes" in Features on ARM, in flags on x86; -w so that a model + # name containing the letters does not answer for the flags + grep -qw aes "$cpuinfo" + return + fi + if [ "$(uname -s)" = Darwin ]; then + [ "$(sysctl -n hw.optional.arm.FEAT_AES 2>/dev/null)" = 1 ] && return 0 + [ "$(sysctl -n hw.optional.aes 2>/dev/null)" = 1 ] && return 0 + return 1 + fi + return 0 +} + # run_one [driver] # # The driver defaults to ct_.c. It is given separately where one @@ -79,13 +119,76 @@ -o "$out/$name" "cbits/tests/ct/ct_$drv.c" $srcs 2> "$out/$name.cc" || { echo "FAIL $name did not build"; sed -n '1,12p' "$out/$name.cc"; status=1; return } + # Which implementation did this build's dispatch actually take? The + # driver says, and the three AES builds want different answers. This + # used to be inferred: the portable build was required to report, + # because the tables leaked, and silence meant it had taken an + # accelerated path instead. The tables are gone and all three are + # silent now, so the question is asked outright rather than read off a + # leak that no longer happens. + case $name in + aes | aes_armv8 | aes_x86ni) + want=0 + [ "$name" = aes ] || want=1 + got=$("$out/$name" 2>/dev/null | sed -n 's/^dispatch aes=\([01]\).*/\1/p') + if [ "$got" != "$want" ]; then + # A processor with no AES instructions is not a failure of + # this driver; there is simply nothing here to measure, and + # the portable one above has already measured what does run. + # Boards in this position are common -- the Raspberry Pi 2, + # 3 and 4 among them, on either word size. + if [ "$want" = 1 ] && [ "$got" = 0 ] && ! machine_has_aes; then + echo "skip $name: this processor has no AES instructions," + echo " so there is nothing here to measure" + return + fi + echo "FAIL $name: dispatch took aes=$got, wanted aes=$want --" + echo " this build is not running the implementation it is named for" + status=1 + return + fi + ;; + powm_s2n_*) + got=$("$out/$name" 2>/dev/null | sed -n 's/^dispatch powm_s2n=\([0-9]*\).*/\1/p') + if [ -z "$got" ]; then + echo "FAIL $name: it did not run to the end on this processor" + status=1 + return + elif [ "$got" -eq 0 ]; then + echo "FAIL $name: the s2n-bignum reduction ran 0 times --" + echo " this build checked the C again, not the path it is named for" + status=1 + return + fi + ;; + esac if [ "$have_valgrind" = no ]; then "$out/$name" > /dev/null 2>&1 && echo "built $name (no valgrind here; nothing checked)" \ || { echo "FAIL $name did not run"; status=1; } return fi - valgrind --error-exitcode=0 --track-origins=yes --num-callers=20 \ - --log-file="$out/$name.log" "$out/$name" > /dev/null 2>&1 || true + # in a subshell, so that a driver valgrind kills is not announced by + # this shell as well -- the log says what happened + ( valgrind --error-exitcode=0 --track-origins=yes --num-callers=20 \ + --log-file="$out/$name.log" "$out/$name" > /dev/null 2>&1 ) \ + 2> /dev/null || true + # A driver valgrind could not run to the end reports nothing, and nothing + # used to read as a pass. An instruction valgrind cannot decode is the + # likely one: the vendored ADX assembly needs a valgrind that knows it. + if grep -q "unhandled instruction\|Illegal instruction\|SIGILL" "$out/$name.log"; then + case $name in + powm_s2n_*) + echo "skip $name: this valgrind cannot decode the vendored assembly's" + echo " instructions, so it cannot run the path this build is for" + ;; + *) + echo "FAIL $name: valgrind could not run it to the end" + grep -m3 "unhandled instruction\|Illegal instruction\|SIGILL" "$out/$name.log" | sed 's/^/ /' + status=1 + ;; + esac + return + fi n=$(grep -c "^==[0-9]*== \(Conditional jump\|Use of uninitialised\)" "$out/$name.log" || true) # Which places did it name? Only the frame the report is against -- the # first "at" line under the complaint -- is the place; the "by" lines @@ -93,18 +196,28 @@ # on anything. Nor is the "at" line under "Uninitialised value was # created by", which --track-origins prints to say where the value came # from: that frame is a stack allocation, not a branch, and taking it - # for one put a function's opening brace on the list. A site is - # "file:line", and the ones listed in known.txt are understood. + # for one put a function's opening brace on the list. + # + # A site is kept as "function@file:line", and known.txt may name any of + # the three widths: that exact line, the function, or the whole file. + # The function is there because the line is the compiler's choice and + # not the code's: gcc 14 names no line of addM's loop at all, gcc 11.5 + # on x86-64 names three, and clang on AArch64 named two of those three. + # What the disassembly establishes is a property of the function, so + # that is what an entry should be able to say. sites=$(awk ' /^==[0-9]*== (Conditional jump|Use of uninitialised)/ { want = 1; next } want && /^==[0-9]*== at 0x/ { print; want = 0 } ' "$out/$name.log" | - sed -n 's/^==[0-9]*== at 0x[0-9A-Fa-f]*: [A-Za-z_0-9]* (\([^)]*\))$/\1/p' | - grep -v '^ct_' | sort -u) + sed -n 's/^==[0-9]*== at 0x[0-9A-Fa-f]*: \([A-Za-z_0-9.]*\) (\([^)]*\))$/\1@\2/p' | + grep -v '@ct_' | sort -u) unknown= for site in $sites; do - file=${site%%:*} - if grep -q "^$site[[:space:]]" cbits/tests/ct/known.txt || + funcfile=${site%%:*} + fileline=${site#*@} + file=${fileline%%:*} + if grep -q "^$fileline[[:space:]]" cbits/tests/ct/known.txt || + grep -q "^$funcfile[[:space:]]" cbits/tests/ct/known.txt || grep -q "^$file[[:space:]]" cbits/tests/ct/known.txt; then continue fi @@ -123,27 +236,17 @@ echo "ok canary: reported $n, so the marking works" fi ;; - aes) - # Silence would mean the build took an accelerated path and so - # measured nothing; the tables reporting is the point. - if [ "$n" -eq 0 ]; then - echo "FAIL aes: reported nothing, so this build did not take the" - echo " table-driven code the driver exists to measure" - status=1 - else - echo "note aes: $n report(s), from $(echo "$sites" | tr '\n' ' ')" - fi - ;; - aes_armv8) - # The opposite demand, and known.txt does not apply: the entries in - # it are for the tables, and this build is not supposed to reach - # them. Anything at all here is a finding, including a table site, - # which would mean the dispatch did not pick the instructions. + aes | aes_armv8 | aes_x86ni) + # Nothing here looks anything up: the portable AES and GHASH are + # bitsliced, and the two instruction sets branch on nothing. So + # anything at all is a finding and known.txt does not apply -- + # until the tables went this entry read the other way round, with + # the portable build required to report. if [ "$n" -eq 0 ]; then - echo "ok aes_armv8: the instructions decided nothing" + echo "ok $name: the secret decided nothing" else - echo "FAIL aes_armv8: $n report(s) from the AArch64 AES or GHASH," - echo " which look nothing up and should branch on nothing:" + echo "FAIL $name: $n report(s) from AES or GHASH," + echo " which should branch on nothing:" for site in $sites; do echo " $site"; done sed -n '/Conditional jump\|Use of uninitialised/,/^==[0-9]*== $/p' \ "$out/$name.log" | head -30 | sed 's/^/ /' @@ -171,6 +274,50 @@ run_one canary "" "" run_one powm "cbits/crypton_powm.c" "" + +# The same exponentiation through s2n-bignum, which is what RSA signing runs +# on x86-64 with ADX. The build above is the C: it does not define +# CRYPTON_S2N_BIGNUM, so from the release the assembly went in until this +# was added, the path that actually runs was never put through this +# harness. Each size is a different routine -- Karatsuba at 1024 and 2048, +# a padded operand through the 2048-bit one at 1536, schoolbook at 3072 and +# 4096 -- so each is built. +# +# ADX is assumed by CRYPTON_CPU_TESTING_ADX, and only where the processor +# reports it: under valgrind the processor's answers are valgrind's, and +# what this is for is the assembly, not valgrind's opinion of the +# processor. The driver counts the reductions it did through the assembly, +# and a count of nothing fails the build rather than passing it. +case $(uname -m) in +x86_64 | amd64) + if grep -qw adx "${CRYPTON_CT_CPUINFO:-/proc/cpuinfo}" 2>/dev/null; then + s2n=cbits/s2n/x86_att + powm_s2n_src="cbits/crypton_powm.c cbits/crypton_cpu.c cbits/tests/ct/cpu_stub.c + $s2n/bignum_montifier.S $s2n/bignum_emontredc_8n.S + $s2n/bignum_kmul_16_32.S $s2n/bignum_ksqr_16_32.S + $s2n/bignum_kmul_32_64.S $s2n/bignum_ksqr_32_64.S + $s2n/bignum_sqr_8_16.S $s2n/bignum_mul_8_16.S + $s2n/bignum_mul.S $s2n/bignum_sqr.S" + powm_s2n_inc="-Icbits/s2n/include -DCRYPTON_S2N_BIGNUM + -DCRYPTON_CPU_TESTING_ADX -DCT_POWM_COUNT_S2N + -Wl,--wrap=bignum_emontredc_8n" + for bytes in 128 192 256 384 512; do + run_one "powm_s2n_$((bytes * 8))" "$powm_s2n_src" \ + "$powm_s2n_inc -DCT_POWM_BYTES=$bytes" powm + done + else + echo "skip powm_s2n: this processor reports no ADX, so the vendored" + echo " multiplication cannot run here" + fi + ;; +aarch64 | arm64) + # the generated squaring is separate code at each size; 2048 is above + for bytes in 128 192; do + run_one "powm_$((bytes * 8))" "cbits/crypton_powm.c" \ + "-DCT_POWM_BYTES=$bytes" powm + done + ;; +esac run_one p256 "cbits/p256/p256.c cbits/p256/p256_ec.c" "" run_one x25519 "cbits/curve25519/curve25519-donna-c64.c" "" run_one ed25519 "cbits/ed25519/ed25519.c cbits/crypton_sha512.c" "-Icbits/ed25519" @@ -183,18 +330,31 @@ # and the build would not even compile. case $(uname -m) in aarch64 | arm64) - run_one aes_armv8 "$armv8_src" "$armv8_inc" + run_one aes_armv8 "$armv8_src" "$armv8_inc" aes ;; +x86_64 | amd64) + run_one aes_x86ni "$x86ni_src" "$x86ni_inc" aes + ;; esac # The ML-KEM backend, on both architectures that have one. x86-64 wants the # flags its own build system passes, or the AVX2 sources do not assemble. +# +# The backend now asks the processor before it runs, so on x86-64 without +# AVX2 this build measures the portable C under a name that says otherwise. +# Say so and skip, rather than pass while checking the thing the plain +# mlkem build above has already checked. case $(uname -m) in aarch64 | arm64) run_one mlkem_native "$mlkem_native_src" "$mlkem_native_inc" mlkem ;; x86_64 | amd64) - run_one mlkem_native "$mlkem_native_src" "$mlkem_native_inc -mavx2 -mbmi2" mlkem + if grep -qw avx2 "${CRYPTON_CT_CPUINFO:-/proc/cpuinfo}" 2>/dev/null; then + run_one mlkem_native "$mlkem_native_src" "$mlkem_native_inc -mavx2 -mbmi2" mlkem + else + echo "skip mlkem_native: this processor has no AVX2, so the backend" + echo " would fall back to the C the mlkem build above measured" + fi ;; esac diff --git a/cbits/tests/endian/endian.c b/cbits/tests/endian/endian.c --- a/cbits/tests/endian/endian.c +++ b/cbits/tests/endian/endian.c @@ -25,6 +25,9 @@ #include "crypton_chacha.h" #include "crypton_salsa.h" #include "crypton_poly1305.h" +#include "crypton_aes.h" +#include "aes/gf.h" +#include "aes/block128.h" /* The skein headers spell the prefix "cryponite", which nothing defines. */ void crypton_skein256_init(struct skein256_ctx *ctx, uint32_t hashlen); @@ -92,6 +95,131 @@ } \ } while (0) +/* + * AES, which reaches further into the byte order than the hashes above do. + * + * The portable implementation keeps its schedule as 64-bit words and reads + * its input through br_dec32le; the GHASH beside it reads H and the + * accumulator as big-endian words; the counter modes carry a counter that is + * incremented big-endian and stored little-endian in one case and the other + * way round in another; and XTS doubles its tweak in GF(2^128) through + * cpu_to_le64. Every one of those is a place where a big-endian machine can + * differ, and none of them was asked about here until now. + * + * The entries called are the public ones, so this is whichever + * implementation the build installed -- which, for the build this harness + * makes, is the portable one. That is the one a big-endian machine runs: + * crypton has no AES instructions for s390x, and the POWER8 ones are + * little-endian only. + */ +static void aes_answers(void) { + static const uint8_t keylens[] = {16, 24, 32}; + /* multiples of the block, for the modes that take whole blocks */ + static const uint32_t blocks[] = {1, 2, 4, 7}; + /* and byte counts, including a partial block, for the ones that do not */ + static const uint32_t bytes[] = {0, 1, 15, 16, 17, 64, 100}; + uint8_t key[32], key2[32], iv[16], out[128], tmp[128]; + char nm[128]; + size_t ki, li; + uint32_t i; + + for (i = 0; i < 32; i++) { key[i] = (uint8_t)(i * 3 + 1); + key2[i] = (uint8_t)(i * 5 + 2); } + for (i = 0; i < 16; i++) iv[i] = (uint8_t)(i * 11 + 7); + + for (ki = 0; ki < sizeof keylens / sizeof *keylens; ki++) { + uint8_t kl = keylens[ki]; + aes_key k, k2; + aes_gcm_key gk; + + crypton_aes_initkey(&k, key, kl); + crypton_aes_initkey(&k2, key2, kl); + crypton_aes_gcm_key_init(&gk, &k); + + for (li = 0; li < sizeof blocks / sizeof *blocks; li++) { + uint32_t nb = blocks[li]; + aes_block ivb; + + crypton_aes_encrypt_ecb((aes_block *)out, &k, (aes_block *)buf, nb); + snprintf(nm, sizeof nm, "aes%u-ecb/%u", kl * 8, nb); + answer(nm, out, nb * 16); + + crypton_aes_decrypt_ecb((aes_block *)out, &k, (aes_block *)buf, nb); + snprintf(nm, sizeof nm, "aes%u-ecbd/%u", kl * 8, nb); + answer(nm, out, nb * 16); + + memcpy(&ivb, iv, 16); + crypton_aes_encrypt_cbc((aes_block *)out, &k, &ivb, (aes_block *)buf, nb); + snprintf(nm, sizeof nm, "aes%u-cbc/%u", kl * 8, nb); + answer(nm, out, nb * 16); + + memcpy(&ivb, iv, 16); + crypton_aes_decrypt_cbc((aes_block *)out, &k, &ivb, (aes_block *)buf, nb); + snprintf(nm, sizeof nm, "aes%u-cbcd/%u", kl * 8, nb); + answer(nm, out, nb * 16); + + memcpy(&ivb, iv, 16); + crypton_aes_encrypt_xts((aes_block *)out, &k, &k2, &ivb, 0, + (aes_block *)buf, nb); + snprintf(nm, sizeof nm, "aes%u-xts/%u", kl * 8, nb); + answer(nm, out, nb * 16); + + memcpy(&ivb, iv, 16); + crypton_aes_decrypt_xts((aes_block *)out, &k, &k2, &ivb, 0, + (aes_block *)buf, nb); + snprintf(nm, sizeof nm, "aes%u-xtsd/%u", kl * 8, nb); + answer(nm, out, nb * 16); + + /* a starting point too, since that is extra tweak doubling */ + memcpy(&ivb, iv, 16); + crypton_aes_encrypt_xts((aes_block *)out, &k, &k2, &ivb, 3, + (aes_block *)buf, nb); + snprintf(nm, sizeof nm, "aes%u-xts-sp3/%u", kl * 8, nb); + answer(nm, out, nb * 16); + } + + for (li = 0; li < sizeof bytes / sizeof *bytes; li++) { + uint32_t n = bytes[li]; + aes_block ivb; + + memcpy(&ivb, iv, 16); + crypton_aes_encrypt_ctr(out, &k, &ivb, buf, n); + snprintf(nm, sizeof nm, "aes%u-ctr/%u", kl * 8, n); + answer(nm, out, n); + + /* the tag goes after the ciphertext, so this answers for both */ + crypton_aes_gcm_full_encrypt(out, &gk, &k, iv, 12, buf, 13, + buf, n, 16); + snprintf(nm, sizeof nm, "aes%u-gcm/%u", kl * 8, n); + answer(nm, out, n + 16); + } + } + + /* GHASH and POLYVAL on their own, which the modes above reach only + * through whatever length they were given */ + { + table_4bit ht; + block128 acc; + aes_polyval pv; + + crypton_aes_generic_hinit(ht, (const block128 *)buf); + memcpy(&acc, buf + 16, 16); + crypton_aes_generic_gf_mul(&acc, ht); + answer("ghash-mul", (const uint8_t *)&acc, 16); + + crypton_aes_generic_hinit(ht, (const block128 *)buf); + memcpy(&acc, buf + 16, 16); + crypton_aes_generic_gf_mul4(&acc, (const block128 *)(buf + 32), ht); + answer("ghash-mul4", (const uint8_t *)&acc, 16); + + memcpy(tmp, buf, 16); + crypton_aes_polyval_init(&pv, (const aes_block *)tmp); + crypton_aes_polyval_update(&pv, buf + 16, 64); + crypton_aes_polyval_finalize(&pv, (aes_block *)out); + answer("polyval", out, 16); + } +} + int main(int argc, char **argv) { generating = (argc > 1 && strcmp(argv[1], "generate") == 0); vf = fopen(argc > 2 ? argv[2] : "cbits/tests/endian/vectors.txt", @@ -182,6 +310,8 @@ answer(nmbuf, mac, sizeof mac); } } + + aes_answers(); if (!generating && failures == 0) printf("ok %d answers match the little-endian ones\n", checked); diff --git a/cbits/tests/endian/run.sh b/cbits/tests/endian/run.sh --- a/cbits/tests/endian/run.sh +++ b/cbits/tests/endian/run.sh @@ -6,6 +6,14 @@ # loaders in crypton_align.h -- rewritten from word-typed casts to memcpy in # #256 -- and eight more decide something from the byte order themselves. # +# AES is here as well, and it reaches further into the byte order than the +# hashes do: a schedule kept as 64-bit words, a GHASH that reads H and its +# accumulator as big-endian words, counters incremented one way and stored +# the other, and an XTS tweak doubled through cpu_to_le64. It is the +# portable implementation that answers, which is what a big-endian machine +# runs: crypton has no AES instructions for s390x, and the POWER8 ones are +# little-endian only. +# # The two sides cannot be compared in one run the way the 32-bit harness # compares two builds, because the machine doing the comparing has only one # byte order. So the answers are frozen: vectors.txt is what this code gives @@ -26,7 +34,11 @@ cbits/crypton_sha256.c cbits/crypton_sha512.c cbits/crypton_sha3.c cbits/crypton_ripemd.c cbits/crypton_skein256.c cbits/crypton_skein512.c cbits/crypton_tiger.c cbits/crypton_whirlpool.c - cbits/crypton_chacha.c cbits/crypton_salsa.c cbits/crypton_poly1305.c" + cbits/crypton_chacha.c cbits/crypton_salsa.c cbits/crypton_poly1305.c + cbits/crypton_aes.c cbits/aes/generic.c cbits/aes/gf.c + cbits/bearssl/aes_ct64.c cbits/bearssl/aes_ct64_enc.c + cbits/bearssl/aes_ct64_dec.c cbits/bearssl/ghash_ctmul64.c + cbits/bearssl/dec32le.c" # Generating is done under the sanitizers, since a driver that writes out of # bounds would otherwise freeze whatever it happened to leave behind. That @@ -38,7 +50,7 @@ fi # shellcheck disable=SC2086 -$cc -O2 -g $san -Icbits -Icbits/include64 -o "$out/endian" \ +$cc -O2 -g $san -Icbits -Icbits/aes -Icbits/include64 -o "$out/endian" \ cbits/tests/endian/endian.c $srcs if [ "$mode" = generate ]; then diff --git a/cbits/tests/endian/vectors.txt b/cbits/tests/endian/vectors.txt --- a/cbits/tests/endian/vectors.txt +++ b/cbits/tests/endian/vectors.txt @@ -250,3 +250,132 @@ chacha20/1000 064429ccbd6ef85054af07f8dee3a283e75e844df9a59b1b76addd7d41fb554946f5dd12b49b97f4a72f05b0b41c557e8c69da034ae9dddc8a54dbf245d2bcff salsa20/1000 9ee7f7b5774db507d8e78fe7398594de9b40444a5301c14dfbc675e6310f9aa659e9cca9892563df9386a60607a2e18781d35d4897ea2b029db7b8819cafae17 poly1305/1000 c403a63071ea2f2e732472db8e7b79c0 +aes128-ecb/1 51d0767be0142b54f38422f098a979c1 +aes128-ecbd/1 0772f11dd1e53677251c6d2b42c12a84 +aes128-cbc/1 53423fee1b0b9dda9006158bb0e3c6e6 +aes128-cbcd/1 0060ec35e2db7f237a7618abc9578b28 +aes128-xts/1 2634e65251d6fed878a1b3546061cd8f +aes128-xtsd/1 9d22db2b665e917344f60d1561481854 +aes128-xts-sp3/1 1d40e186532c4338b08ebb40039c49b6 +aes128-ecb/2 51d0767be0142b54f38422f098a979c1f5e0ee3fe17beef5b33b72d8f60ec17c +aes128-ecbd/2 0772f11dd1e53677251c6d2b42c12a843d8abf116e0bd219a05d2c1d89e8faee +aes128-cbc/2 53423fee1b0b9dda9006158bb0e3c6e6b1acc41a9ff16d75c2170e3242346760 +aes128-cbcd/2 0060ec35e2db7f237a7618abc9578b283d8db1047228f82898626a50ddb39887 +aes128-xts/2 2634e65251d6fed878a1b3546061cd8ff2e82679d8d7ade78233b89a76958855 +aes128-xtsd/2 9d22db2b665e917344f60d1561481854c8d8c3fe1890b52db7ca720cd253192d +aes128-xts-sp3/2 1d40e186532c4338b08ebb40039c49b6cc2b2b42b37e9412005737634804ad97 +aes128-ecb/4 51d0767be0142b54f38422f098a979c1f5e0ee3fe17beef5b33b72d8f60ec17c527ee4edb0bd6e9346d1148c3763613b5112f9a67b30c06fd511e725279957b9 +aes128-ecbd/4 0772f11dd1e53677251c6d2b42c12a843d8abf116e0bd219a05d2c1d89e8faeed21af82debb90e26d30fd338766a0ff47234606001901e7888e61a64468495be +aes128-cbc/4 53423fee1b0b9dda9006158bb0e3c6e6b1acc41a9ff16d75c2170e324234676075d60f392793fd0623859fffb40b23585955ff09a2c20c02b9c2baf084d6943b +aes128-cbcd/4 0060ec35e2db7f237a7618abc9578b283d8db1047228f82898626a50ddb39887a26d86a8672a94877ba06585b2a1dd2d8d326d741ab23748bfd85f2815def4d6 +aes128-xts/4 2634e65251d6fed878a1b3546061cd8ff2e82679d8d7ade78233b89a76958855da809daab8adf228232779804848093c5aa478a2d7e0c11b58dfbfd19b94ea36 +aes128-xtsd/4 9d22db2b665e917344f60d1561481854c8d8c3fe1890b52db7ca720cd253192dedd68b0854e3340afb2d8a34942bcd5479bed9f04ee52769479c2192544eaa2f +aes128-xts-sp3/4 1d40e186532c4338b08ebb40039c49b6cc2b2b42b37e9412005737634804ad970343c410aa37b63b6595f3e6e8bbcb8bc8581ec02491cb1e3df920165a8d05c7 +aes128-ecb/7 51d0767be0142b54f38422f098a979c1f5e0ee3fe17beef5b33b72d8f60ec17c527ee4edb0bd6e9346d1148c3763613b5112f9a67b30c06fd511e725279957b977a15ad7bbc6a54c141a2d07c4b233000adedca5ef085da03709e978ed1aca3500e2bd1f0ad7a29ade429446d2ddfb93 +aes128-ecbd/7 0772f11dd1e53677251c6d2b42c12a843d8abf116e0bd219a05d2c1d89e8faeed21af82debb90e26d30fd338766a0ff47234606001901e7888e61a64468495be2de8cb16ce03066f65758235de9386de764ddfa85c70fae9b4c80eed9b53a216d5c34dcea96fa27e88fe002306a3548d +aes128-cbc/7 53423fee1b0b9dda9006158bb0e3c6e6b1acc41a9ff16d75c2170e324234676075d60f392793fd0623859fffb40b23585955ff09a2c20c02b9c2baf084d6943b84e8cef5ab838dc0864175cab14a1026f6230f8df62be8d6794dfb9aba0622e62e6a358aab88832f2b18d4819eca4f91 +aes128-cbcd/7 0060ec35e2db7f237a7618abc9578b283d8db1047228f82898626a50ddb39887a26d86a8672a94877ba06585b2a1dd2d8d326d741ab23748bfd85f2815def4d6429eb69245919fcfc2db37891d5957068848d3bb4651d2c682f54aa6c90ac271bbb6314d23fe3ae12e53b498c46a845a +aes128-xts/7 2634e65251d6fed878a1b3546061cd8ff2e82679d8d7ade78233b89a76958855da809daab8adf228232779804848093c5aa478a2d7e0c11b58dfbfd19b94ea3651d66c7adda6172d3e8672cc65cce7dbd8d9d6f32bf8f087867527f0a8128fdd80ed3a97398f66577f2041342c585e72 +aes128-xtsd/7 9d22db2b665e917344f60d1561481854c8d8c3fe1890b52db7ca720cd253192dedd68b0854e3340afb2d8a34942bcd5479bed9f04ee52769479c2192544eaa2fa426c177c9b397f0bb380e3991c69479759294d63723eeb04aae18e48f84fa213dcf254eb1d532252c6328a8fdd8976f +aes128-xts-sp3/7 1d40e186532c4338b08ebb40039c49b6cc2b2b42b37e9412005737634804ad970343c410aa37b63b6595f3e6e8bbcb8bc8581ec02491cb1e3df920165a8d05c7fa1c26910201c2ac91dab7ffcb1995c57cfc19a1a8ffbb3e11debe1e9de63147bd99f1f0f9af66d827b18a427ed02d32 +aes128-ctr/0 - +aes128-gcm/0 91527d0ffa250047dc10c74aaf7dc630 +aes128-ctr/1 f3 +aes128-gcm/1 110be12ced502543dc2b098ce214c704a9 +aes128-ctr/15 f3e218ff08f7128f6c9e088df658bf +aes128-gcm/15 11e696e7a098883e54b19200262e669d12ac829a6a3eed6933b267f13690e7 +aes128-ctr/16 f3e218ff08f7128f6c9e088df658bfcc +aes128-gcm/16 11e696e7a098883e54b19200262e6622968a6e12ea8c1d162967acbccc339980 +aes128-ctr/17 f3e218ff08f7128f6c9e088df658bfccea +aes128-gcm/17 11e696e7a098883e54b19200262e6622dd540e6e9683dd7026b3b24579826c0593 +aes128-ctr/64 f3e218ff08f7128f6c9e088df658bfccea245fbfae1cb12ed69d5f01a5a6d7f161f8ae682ba4555a088d89fe91fdfb2890935fe9d0ae19f0ac4f73e19d26d1cc +aes128-gcm/64 11e696e7a098883e54b19200262e6622dd7f1ac8145981a493fdaea4641bf49da46e0d311c3daf1665fd5fdf025965d9312f324d4c646d3f3f85ab13d236716fdb10935bd1c334f13b56a98aa09ef476 +aes128-ctr/100 f3e218ff08f7128f6c9e088df658bfccea245fbfae1cb12ed69d5f01a5a6d7f161f8ae682ba4555a088d89fe91fdfb2890935fe9d0ae19f0ac4f73e19d26d1cc8c30a3d0749027d7cff23bb8dbccb1a8540feb89f5ca9447a5cdef585d59125c58b73b22 +aes128-gcm/100 11e696e7a098883e54b19200262e6622dd7f1ac8145981a493fdaea4641bf49da46e0d311c3daf1665fd5fdf025965d9312f324d4c646d3f3f85ab13d236716fc40f23dfc154e04b18dfc89ae50a386551eab4dd02434c526f9fc45a278b21a485347381adbe43ff0386975d584a98adacbde34a +aes192-ecb/1 b7bd8a4e647f413adc38c4464e4434b6 +aes192-ecbd/1 4308adb49827c2741aabb7c40c5b63a5 +aes192-cbc/1 9fdd2c700fd7e916fd89805f18439a27 +aes192-cbcd/1 441ab09cab198b2045c1c24487cdc209 +aes192-xts/1 c2eea1b9f5f0fd170168a3e2af1d059a +aes192-xtsd/1 5a3a19588f9700d5482eae9dd316051d +aes192-xts-sp3/1 f1dba9fc3e3d43d3a46608690e98b0e8 +aes192-ecb/2 b7bd8a4e647f413adc38c4464e4434b67d307330822566b908545f9e8de18c1a +aes192-ecbd/2 4308adb49827c2741aabb7c40c5b63a5e6015605e3e6c8df51f2de769db588e2 +aes192-cbc/2 9fdd2c700fd7e916fd89805f18439a2754e55a17706f184fe6b2185ef87eb075 +aes192-cbcd/2 441ab09cab198b2045c1c24487cdc209e6065810ffc5e2ee69cd983bc9eeea8b +aes192-xts/2 c2eea1b9f5f0fd170168a3e2af1d059af4c8308582684da17d5b6d631459d6d5 +aes192-xtsd/2 5a3a19588f9700d5482eae9dd316051dab8c3b6155002903c90664413bd96653 +aes192-xts-sp3/2 f1dba9fc3e3d43d3a46608690e98b0e823173a551464b1a3ef6095560902a48c +aes192-ecb/4 b7bd8a4e647f413adc38c4464e4434b67d307330822566b908545f9e8de18c1ac8d6dd6f7bee6e99e4682aef343f7cd0ea985b24e2368b8d1acd3b0bf164a30b +aes192-ecbd/4 4308adb49827c2741aabb7c40c5b63a5e6015605e3e6c8df51f2de769db588e2f4a1ce5088ed9eee6f721f73986dcf23ffc534ac5d16cb7010015dc1b50330ce +aes192-cbc/4 9fdd2c700fd7e916fd89805f18439a2754e55a17706f184fe6b2185ef87eb075061361ddd895a18991e3068ef563dcea087fc1517810a4ac222a391215026095 +aes192-cbcd/4 441ab09cab198b2045c1c24487cdc209e6065810ffc5e2ee69cd983bc9eeea8b84d6b0d5047e044fc7dda9ce5ca61dfa00c339b84634e240273f188de65951a6 +aes192-xts/4 c2eea1b9f5f0fd170168a3e2af1d059af4c8308582684da17d5b6d631459d6d52e43efd39bfc3ebbbbf189a4bfb1071bbc3851a67841dc4206c4f0f86fca0fed +aes192-xtsd/4 5a3a19588f9700d5482eae9dd316051dab8c3b6155002903c90664413bd9665351b41f0a6b8b2afd38a19274800e86739bb50da065cc97f9818895e062c6be6c +aes192-xts-sp3/4 f1dba9fc3e3d43d3a46608690e98b0e823173a551464b1a3ef6095560902a48cdc267dd87440e9a910bedc98758b512799ad24644966748404717b701ad70804 +aes192-ecb/7 b7bd8a4e647f413adc38c4464e4434b67d307330822566b908545f9e8de18c1ac8d6dd6f7bee6e99e4682aef343f7cd0ea985b24e2368b8d1acd3b0bf164a30b225dbf0a85eb782bbaf30c02286d91d271bbfb08e60fc12977e7eea142abc72bed4ade399f048bfa5d5a48f02ac47e22 +aes192-ecbd/7 4308adb49827c2741aabb7c40c5b63a5e6015605e3e6c8df51f2de769db588e2f4a1ce5088ed9eee6f721f73986dcf23ffc534ac5d16cb7010015dc1b50330ce1b0b2582b28d8f6174ee21d4cefdc31632f8ca413704961b7da50fdc2de61e730ab37b8b3148660f36335f6d4a01d85a +aes192-cbc/7 9fdd2c700fd7e916fd89805f18439a2754e55a17706f184fe6b2185ef87eb075061361ddd895a18991e3068ef563dcea087fc1517810a4ac222a391215026095582b2b9c34d2b9b40ef88f9ec4459fc671f5ecb0d23defc6ea99f6075174381b861dec529ac5a49c9c55efae85aca707 +aes192-cbcd/7 441ab09cab198b2045c1c24487cdc209e6065810ffc5e2ee69cd983bc9eeea8b84d6b0d5047e044fc7dda9ce5ca61dfa00c339b84634e240273f188de65951a6747d5806391f16c1d34094680d3712ceccfdc6522d25be344b984b977fbf7e1464c60708bbd9fe90909eebd688c8088d +aes192-xts/7 c2eea1b9f5f0fd170168a3e2af1d059af4c8308582684da17d5b6d631459d6d52e43efd39bfc3ebbbbf189a4bfb1071bbc3851a67841dc4206c4f0f86fca0fedf90c2617695920f2f5611aab587198b0263b555f0a93d1580c22822b6fe3451e31e5f67c9526f5128d87bc586e61a255 +aes192-xtsd/7 5a3a19588f9700d5482eae9dd316051dab8c3b6155002903c90664413bd9665351b41f0a6b8b2afd38a19274800e86739bb50da065cc97f9818895e062c6be6cfa00838607d2c4a55642a4973aac6fbdd5e9c514e69e8569d1559acde57fc207a5709b8f0eff9a93ad7dc6c170fda2c0 +aes192-xts-sp3/7 f1dba9fc3e3d43d3a46608690e98b0e823173a551464b1a3ef6095560902a48cdc267dd87440e9a910bedc98758b512799ad24644966748404717b701ad70804506ba3f939ed54c43dc721d3532f83cf267aa4c32d9543edb71de9548dacd5171e76bef732fdad2c49ebb781d639a445 +aes192-ctr/0 - +aes192-gcm/0 b03daf3f9346095b3083d09391d47cee +aes192-ctr/1 72 +aes192-gcm/1 bdc01f6ed1a9e3dc9be2c4d9e8d8d3b584 +aes192-ctr/15 72138e46596ca4b6de00c2aeadc2c1 +aes192-gcm/15 bd19761039873f6946e29ba643ee41c6ca3791dc7497012fce9f48631f599c +aes192-ctr/16 72138e46596ca4b6de00c2aeadc2c17c +aes192-gcm/16 bd19761039873f6946e29ba643ee4175c85f71850653f05f2410354e2e93cdc9 +aes192-ctr/17 72138e46596ca4b6de00c2aeadc2c17c02 +aes192-gcm/17 bd19761039873f6946e29ba643ee4175b3e87ccbca9af385c12d31382f6e721151 +aes192-ctr/64 72138e46596ca4b6de00c2aeadc2c17c02df852e3c7abed9e3cd5a94d24c71074945c7835771862ec94dbc9c8ba47ef305131b7f3933a150025d8a796c5889b2 +aes192-gcm/64 bd19761039873f6946e29ba643ee4175b3c8f8e942762e364f67333b85f6c220999554b2d6e78a47fc789581a39e89a4b6a5539491894c85fbba17f03e183e3b27736f65aaba8f86ba884618e2c75b83 +aes192-ctr/100 72138e46596ca4b6de00c2aeadc2c17c02df852e3c7abed9e3cd5a94d24c71074945c7835771862ec94dbc9c8ba47ef305131b7f3933a150025d8a796c5889b2aea7d364add8934a993f8532bae1b59ec0c4d75bf59dca8e265de412ed73052188706192 +aes192-gcm/100 bd19761039873f6946e29ba643ee4175b3c8f8e942762e364f67333b85f6c220999554b2d6e78a47fc789581a39e89a4b6a5539491894c85fbba17f03e183e3bb42b4d0d362fc35b7719b76c299e857e82b8b127e9cbc732559c26b6338ef92f153271feb0462a5c14552476a3a41aae5c275596 +aes256-ecb/1 791925c38818304f047d756d78ef8f13 +aes256-ecbd/1 831e330b93ed108f70c15c463224b5c6 +aes256-cbc/1 6b1dfef4ab6d3fd0a32a3c3a7c9603d5 +aes256-cbcd/1 840c2e23a0d359db2fab29c6b9b2146a +aes256-xts/1 b90a69094799c4e679fc829084d41055 +aes256-xtsd/1 18db4b08618f8573974e5c85d0954302 +aes256-xts-sp3/1 d5f9ef046dd7e688938aab18d3d68ce5 +aes256-ecb/2 791925c38818304f047d756d78ef8f139da36087c3b26165970cf0987eee7d70 +aes256-ecbd/2 831e330b93ed108f70c15c463224b5c6f2f7d848bb39a82dd65b4db54a6f54ff +aes256-cbc/2 6b1dfef4ab6d3fd0a32a3c3a7c9603d5090fa2b707cd96fea1132c143d5683fa +aes256-cbcd/2 840c2e23a0d359db2fab29c6b9b2146af2f0d65da71a821cee640bf81e343696 +aes256-xts/2 b90a69094799c4e679fc829084d4105597001399d1f01a458a64913345271c27 +aes256-xtsd/2 18db4b08618f8573974e5c85d0954302511a3cdde7128329882a30bd28169980 +aes256-xts-sp3/2 d5f9ef046dd7e688938aab18d3d68ce5724313716ca706a5b7a914024b26cfc1 +aes256-ecb/4 791925c38818304f047d756d78ef8f139da36087c3b26165970cf0987eee7d70aff88ab2d21b91927e27057d2f91f923cd1e35c75f2b3111adc0fd520ae368b4 +aes256-ecbd/4 831e330b93ed108f70c15c463224b5c6f2f7d848bb39a82dd65b4db54a6f54ff6c4c59c5bd048efc1c7cf8353e29eae59fd397d041edd80e0c4a41824a3b7bc1 +aes256-cbc/4 6b1dfef4ab6d3fd0a32a3c3a7c9603d5090fa2b707cd96fea1132c143d5683fae5e8441b18737ef3581adef0e182aea3b94a6e5e4925464a95ddb6b409f19e88 +aes256-cbcd/4 840c2e23a0d359db2fab29c6b9b2146af2f0d65da71a821cee640bf81e3436961c3b27403197145db4d34e88fae2383c60d59ac45acff13e3b7404ce19611aa9 +aes256-xts/4 b90a69094799c4e679fc829084d4105597001399d1f01a458a64913345271c27261700fa455603c6d5370d3867bf3213717c7241850cc16b414cf4598472f9e7 +aes256-xtsd/4 18db4b08618f8573974e5c85d0954302511a3cdde7128329882a30bd28169980f52cc7d5d434263c1c22188e735a49d6130e0683a55e636739ba7abb743e7b09 +aes256-xts-sp3/4 d5f9ef046dd7e688938aab18d3d68ce5724313716ca706a5b7a914024b26cfc1b47c048f8548cc7a80a22a9dcaf25065efd935f4730020a085cfd489b8a98cab +aes256-ecb/7 791925c38818304f047d756d78ef8f139da36087c3b26165970cf0987eee7d70aff88ab2d21b91927e27057d2f91f923cd1e35c75f2b3111adc0fd520ae368b4b6d87226c642479229919a9c47bfca56ca5607a049cd7f423fb0556511f5db4bb8f7165904e83e4da028f316252e9f5d +aes256-ecbd/7 831e330b93ed108f70c15c463224b5c6f2f7d848bb39a82dd65b4db54a6f54ff6c4c59c5bd048efc1c7cf8353e29eae59fd397d041edd80e0c4a41824a3b7bc1eaa8daa1a6f40e115cfd93684f1b7ad8b66b09a303b2488f93eaac36762e571d4f00c320962c9dc1607e8d884579f306 +aes256-cbc/7 6b1dfef4ab6d3fd0a32a3c3a7c9603d5090fa2b707cd96fea1132c143d5683fae5e8441b18737ef3581adef0e182aea3b94a6e5e4925464a95ddb6b409f19e88f104a783b152e599e90b85fc60f57aef7ee95c1608290490adf2688b4683b2d70a6b16291cb5a49008dd74dbf8c16191 +aes256-cbcd/7 840c2e23a0d359db2fab29c6b9b2146af2f0d65da71a821cee640bf81e3436961c3b27403197145db4d34e88fae2383c60d59ac45acff13e3b7404ce19611aa985dea7252d6697b1fb5326d48cd1ab00486e05b0199360a0a5d7e87d2477377a2175bfa31cbd055ec6d3393387b023d1 +aes256-xts/7 b90a69094799c4e679fc829084d4105597001399d1f01a458a64913345271c27261700fa455603c6d5370d3867bf3213717c7241850cc16b414cf4598472f9e77778636d8d94ec90c8bc220a7075ab56b4758635686c3985c243224397844894defd980efc5ce05ed451cec1af9583e9 +aes256-xtsd/7 18db4b08618f8573974e5c85d0954302511a3cdde7128329882a30bd28169980f52cc7d5d434263c1c22188e735a49d6130e0683a55e636739ba7abb743e7b09cdc794b975bdb545b6a5a69fde322ceb0f3da5d8a1dd3103dee190f9fb65d8a0e86ddffed9358c0872862ae921e7ba75 +aes256-xts-sp3/7 d5f9ef046dd7e688938aab18d3d68ce5724313716ca706a5b7a914024b26cfc1b47c048f8548cc7a80a22a9dcaf25065efd935f4730020a085cfd489b8a98cab5e7eb266eb52f17b6f78bcb0996b96fc7ca3a671856b32fe2e8c9341c50a70c2d40906b8d450fed98e6a5f6a4d5f84ab +aes256-ctr/0 - +aes256-gcm/0 8316aabedf3935666f9d8b2dd6b5b356 +aes256-ctr/1 21 +aes256-gcm/1 7b8fcd0cf59d4bccd66dd123de4c1c7aa4 +aes256-ctr/15 219430a80fe1a08ed512ff1c9cee61 +aes256-gcm/15 7bfc609c1f245b464a8e0763c5e8b7a1dba82dac1ce33ecb0350b7d9e582df +aes256-ctr/16 219430a80fe1a08ed512ff1c9cee61dc +aes256-gcm/16 7bfc609c1f245b464a8e0763c5e8b7f9ae001617c759469bd602cd92dd4bf7e9 +aes256-ctr/17 219430a80fe1a08ed512ff1c9cee61dc52 +aes256-gcm/17 7bfc609c1f245b464a8e0763c5e8b7f96b70cf1b59a4a56949e6437f0b9b7b257b +aes256-ctr/64 219430a80fe1a08ed512ff1c9cee61dc52867258efbdf3943f48d30ef7da12c7bfb56c293beeddc92b1c270b7110c0e2db805ddad464f0eac9d46e7109001374 +aes256-gcm/64 7bfc609c1f245b464a8e0763c5e8b7f96bc3d807f39fd255cbf14006baf6bbccb5d2509a2b3cba5a4c7ee680ee09979e810f94882ff5c0b6e572d397aef389ec72e6a3f5eb4be76004c10d8c5881492a +aes256-ctr/100 219430a80fe1a08ed512ff1c9cee61dc52867258efbdf3943f48d30ef7da12c7bfb56c293beeddc92b1c270b7110c0e2db805ddad464f0eac9d46e71090013745c52cea313ba7efea2eac9da963dcf1bc484a1c1924fd8f91fd8bf774dc77b472356c522 +aes256-gcm/100 7bfc609c1f245b464a8e0763c5e8b7f96bc3d807f39fd255cbf14006baf6bbccb5d2509a2b3cba5a4c7ee680ee09979e810f94882ff5c0b6e572d397aef389ecb7bd0a0a1f4d638d3e2557ca92beb2bf4228499ac6cc38d20e0539cd4658ed2a7d24f4ab32d4b6ea69814b4c2718fd1dcdb7e16a +ghash-mul c65b128f165974f311df03df3486ce71 +ghash-mul4 6bfd46cd3d314cac2d75ead0e96092c7 +polyval bbc31d0761373f0b2737904b9a76fb27 diff --git a/cbits/tests/fuzz/run.sh b/cbits/tests/fuzz/run.sh --- a/cbits/tests/fuzz/run.sh +++ b/cbits/tests/fuzz/run.sh @@ -31,7 +31,10 @@ canary) echo "" ;; ed25519) echo "cbits/ed25519/ed25519.c cbits/crypton_sha512.c" ;; p256) echo "cbits/p256/p256.c cbits/p256/p256_ec.c" ;; - aead) echo "cbits/crypton_aes.c cbits/aes/generic.c cbits/aes/gf.c" ;; + aead) echo "cbits/crypton_aes.c cbits/aes/generic.c cbits/aes/gf.c + cbits/bearssl/aes_ct64.c cbits/bearssl/aes_ct64_enc.c + cbits/bearssl/aes_ct64_dec.c cbits/bearssl/ghash_ctmul64.c + cbits/bearssl/dec32le.c" ;; decaf) echo "$D/ed448goldilocks/decaf_all.c $D/ed448goldilocks/eddsa.c $D/ed448goldilocks/scalar.c $D/p448/f_arithmetic.c $D/p448/f_generic.c $D/utils.c diff --git a/cbits/tests/ppc8_diff.c b/cbits/tests/ppc8_diff.c new file mode 100644 --- /dev/null +++ b/cbits/tests/ppc8_diff.c @@ -0,0 +1,270 @@ +/* + * The POWER8 path against the portable one. + * + * crypton's own test suite exercises whichever implementation the machine + * installed, so on a POWER8 it never runs the portable code and nothing + * compares the two. CI has no POWER8 either. This does the comparison in + * one process, with both compiled in, and runs under qemu-ppc64le. + * + * powerpc64le-linux-gnu-gcc -O2 -static -Icbits -Icbits/aes \ + * -DWITH_PPC8_CRYPTO -o ppc8_diff \ + * cbits/tests/ppc8_diff.c cbits/crypton_aes.c cbits/aes/generic.c \ + * cbits/aes/gf.c cbits/aes/ppc8.c cbits/crypton_cpu.c \ + * cbits/asm/aesp8-ppc-linux64le.S cbits/asm/ghashp8-ppc-linux64le.S \ + * cbits/bearssl/*.c + * qemu-ppc64le-static ./ppc8_diff + * + * Given any argument it corrupts a result in each section, so that the + * comparison can be seen to notice: one that cannot fail has said nothing. + * + * One trap in writing this, worth naming because it looks like a crash in + * the code under test. Several of the generic mode loops reach the block + * function through the branch table, which in this process holds the POWER8 + * entries -- so calling them as the reference hands a portable key to the + * POWER8 block function. The references below are therefore the entries + * that reach cbits/aes/generic.c directly, and XTS, whose generic loop takes + * its tweak through the table, is written out here instead. + */ +#include +#include +#include +#include + +#include "crypton_aes.h" +#include "aes/generic.h" +#include "aes/gf.h" +#include "aes/block128.h" +#include "aes/ppc8.h" + +/* The portable mode loops this compares against. crypton_aes.c defines + * them and declares them only to itself, at the top of that file, so they + * are declared again here. They were once in a header; when they left it, + * this file -- built by hand and by nobody since -- went on calling them + * undeclared, which C has not allowed since C99 and gcc 14 refuses. */ +void crypton_aes_generic_encrypt_ecb(aes_block *output, aes_key *key, + aes_block *input, uint32_t nb_blocks); +void crypton_aes_generic_decrypt_ecb(aes_block *output, aes_key *key, + aes_block *input, uint32_t nb_blocks); +void crypton_aes_generic_encrypt_cbc(aes_block *output, aes_key *key, + aes_block *iv, aes_block *input, + uint32_t nb_blocks); +void crypton_aes_generic_decrypt_cbc(aes_block *output, aes_key *key, + aes_block *iv, aes_block *input, + uint32_t nb_blocks); + +/* the portable CTR, which reaches generic.c rather than the branch table */ +void crypton_aes_bitsliced_encrypt_ctr(uint8_t *output, aes_key *key, + aes_block *iv, uint8_t *input, + uint32_t len); + +static int failures; +static int sabotage; + +static void same(const char *what, const uint8_t *a, const uint8_t *b, size_t n) +{ + size_t i; + + if (memcmp(a, b, n) == 0) + return; + printf(" MISMATCH %s\n ppc8 ", what); + for (i = 0; i < n && i < 32; i++) printf("%02x", a[i]); + printf("\n portable "); + for (i = 0; i < n && i < 32; i++) printf("%02x", b[i]); + printf("\n"); + failures++; +} + +static uint32_t rnd_state = 1; +static uint8_t rnd(void) +{ + rnd_state = rnd_state * 1103515245u + 12345u; + return (uint8_t) (rnd_state >> 16); +} +static void rnd_fill(uint8_t *p, size_t n) +{ + while (n--) *p++ = rnd(); +} + +/* + * XTS with the portable block function, written out because the generic + * loop takes its tweak through the branch table. + */ +static void xts_reference(uint8_t *out, aes_key *k1, aes_key *k2, + const uint8_t *dataunit, uint32_t spoint, + const uint8_t *in, uint32_t nb, int decrypt) +{ + block128 tweak, t; + uint32_t i; + + memcpy(&tweak, dataunit, 16); + crypton_aes_generic_encrypt_block(&tweak, k2, &tweak); + while (spoint-- > 0) + crypton_aes_generic_gf_mulx(&tweak); + + for (i = 0; i < nb; i++) { + block128_vxor(&t, (const block128 *) (in + 16 * i), &tweak); + if (decrypt) + crypton_aes_generic_decrypt_block(&t, k1, &t); + else + crypton_aes_generic_encrypt_block(&t, k1, &t); + block128_vxor((block128 *) (out + 16 * i), &t, &tweak); + crypton_aes_generic_gf_mulx(&tweak); + } +} + +/* the two inits write different things into the same aes_key, so each side + * gets its own */ +static void keys(aes_key *p8, aes_key *gen, const uint8_t *k, uint8_t len) +{ + memset(p8, 0, sizeof *p8); + memset(gen, 0, sizeof *gen); + p8->nbr = gen->nbr = len == 16 ? 10 : len == 24 ? 12 : 14; + crypton_aes_ppc8_init(p8, (uint8_t *) k, len); + crypton_aes_generic_init(gen, (uint8_t *) k, len); +} + +int main(int argc, char **argv) +{ + static const uint8_t klens[3] = { 16, 24, 32 }; + int round; + + setvbuf(stdout, NULL, _IONBF, 0); /* so a crash does not eat the section it was in */ + sabotage = argc > 1; + + printf("== the dispatch would take: aes=%d ==\n", + crypton_aes_ppc8_available()); + + printf("== ECB and CBC, both directions ==\n"); + for (round = 0; round < 300; round++) { + uint8_t key[32], in[16 * 9], a[16 * 9], b[16 * 9], iv[16]; + uint8_t len = klens[round % 3]; + uint32_t nb = 1 + (round % 9); + aes_key kp, kg; + + rnd_fill(key, len); + rnd_fill(in, nb * 16); + rnd_fill(iv, 16); + keys(&kp, &kg, key, len); + + crypton_aes_ppc8_encrypt_ecb((aes_block *) a, &kp, (aes_block *) in, nb); + crypton_aes_generic_encrypt_ecb((aes_block *) b, &kg, (aes_block *) in, nb); + if (sabotage && round == 3) a[0] ^= 1; + same("ecb encrypt", a, b, nb * 16); + + crypton_aes_ppc8_decrypt_ecb((aes_block *) a, &kp, (aes_block *) in, nb); + crypton_aes_generic_decrypt_ecb((aes_block *) b, &kg, (aes_block *) in, nb); + same("ecb decrypt", a, b, nb * 16); + + { + aes_block iv1, iv2; + memcpy(&iv1, iv, 16); memcpy(&iv2, iv, 16); + crypton_aes_ppc8_encrypt_cbc((aes_block *) a, &kp, &iv1, (aes_block *) in, nb); + crypton_aes_generic_encrypt_cbc((aes_block *) b, &kg, &iv2, (aes_block *) in, nb); + same("cbc encrypt", a, b, nb * 16); + + memcpy(&iv1, iv, 16); memcpy(&iv2, iv, 16); + crypton_aes_ppc8_decrypt_cbc((aes_block *) a, &kp, &iv1, (aes_block *) in, nb); + crypton_aes_generic_decrypt_cbc((aes_block *) b, &kg, &iv2, (aes_block *) in, nb); + same("cbc decrypt", a, b, nb * 16); + } + } + + /* + * CTR, and the one place the two counters have to be reconciled by + * hand. crypton counts over all 128 bits and the assembly over the + * low 32, so the work is handed over in runs that stop where that word + * wraps. A buffer that never reaches the wrap exercises none of that, + * so the IV here is placed a few blocks before it on purpose. + */ + printf("== CTR, including the 32-bit counter wrapping ==\n"); + for (round = 0; round < 200; round++) { + uint8_t key[32], in[600], a[600], b[600], iv[16]; + uint8_t len = klens[round % 3]; + uint32_t n = 1 + (round % 600); + aes_key kp, kg; + aes_block iv1, iv2; + uint32_t before = round % 5; /* blocks left before the wrap */ + + rnd_fill(key, len); + rnd_fill(in, n); + rnd_fill(iv, 16); + /* put the low word within a few blocks of wrapping, and for a + * third of the rounds exactly on it */ + iv[12] = 0xff; iv[13] = 0xff; iv[14] = 0xff; + iv[15] = (uint8_t) (0x100 - before - 1); + keys(&kp, &kg, key, len); + + memcpy(&iv1, iv, 16); memcpy(&iv2, iv, 16); + crypton_aes_ppc8_encrypt_ctr(a, &kp, &iv1, in, n); + crypton_aes_bitsliced_encrypt_ctr(b, &kg, &iv2, in, n); + if (sabotage && round == 7) a[n - 1] ^= 2; + same("ctr across the wrap", a, b, n); + } + + printf("== XTS, both directions and a starting point ==\n"); + for (round = 0; round < 200; round++) { + uint8_t k1[32], k2[32], in[16 * 11], a[16 * 11], b[16 * 11], du[16]; + uint8_t len = klens[round % 3]; + uint32_t nb = 1 + (round % 11); + uint32_t spoint = round % 4; + aes_key p1, g1, p2, g2; + aes_block d1; + + rnd_fill(k1, len); + rnd_fill(k2, len); + rnd_fill(in, nb * 16); + rnd_fill(du, 16); + keys(&p1, &g1, k1, len); + keys(&p2, &g2, k2, len); + + memcpy(&d1, du, 16); + crypton_aes_ppc8_encrypt_xts((aes_block *) a, &p1, &p2, &d1, spoint, + (aes_block *) in, nb); + xts_reference(b, &g1, &g2, du, spoint, in, nb, 0); + /* a[0], not a further block: at this round nb is 1 and anything + * past the first block is outside what same() is given */ + if (sabotage && round == 11) a[0] ^= 4; + same("xts encrypt", a, b, nb * 16); + + memcpy(&d1, du, 16); + crypton_aes_ppc8_decrypt_xts((aes_block *) a, &p1, &p2, &d1, spoint, + (aes_block *) in, nb); + xts_reference(b, &g1, &g2, du, spoint, in, nb, 1); + same("xts decrypt", a, b, nb * 16); + } + + /* + * GHASH, where the two arguments do not take the same convention and + * ppc8.c has to swap one of them. Two different H and a non-zero + * accumulator, because a symmetric H or a zero start can hide a wrong + * convention. + */ + printf("== GHASH ==\n"); + for (round = 0; round < 300; round++) { + uint8_t h[16], data[64], start[16]; + table_4bit tp, tg; + block128 ap, ag; + + rnd_fill(h, 16); + rnd_fill(data, sizeof data); + rnd_fill(start, 16); + + crypton_aes_ppc8_hinit(tp, (const block128 *) h); + crypton_aes_generic_hinit(tg, (const block128 *) h); + + memcpy(&ap, start, 16); memcpy(&ag, start, 16); + crypton_aes_ppc8_gf_mul4(&ap, (const block128 *) data, tp); + crypton_aes_generic_gf_mul4(&ag, (const block128 *) data, tg); + if (sabotage && round == 5) ((uint8_t *) &ap)[0] ^= 8; + same("gf_mul4", (const uint8_t *) &ap, (const uint8_t *) &ag, 16); + + memcpy(&ap, start, 16); memcpy(&ag, start, 16); + crypton_aes_ppc8_gf_mul(&ap, tp); + crypton_aes_generic_gf_mul(&ag, tg); + same("gf_mul", (const uint8_t *) &ap, (const uint8_t *) &ag, 16); + } + + printf("%s: %d mismatch(es)%s\n", failures ? "FAIL" : "ok", failures, + sabotage ? " (sabotage was asked for)" : ""); + return failures != 0; +} diff --git a/cbits/tests/sysdrg/run.sh b/cbits/tests/sysdrg/run.sh new file mode 100644 --- /dev/null +++ b/cbits/tests/sysdrg/run.sh @@ -0,0 +1,22 @@ +#!/bin/sh +# Does the generator behind MonadRandom IO survive the things that break a +# generator: a second thread, a reseed, and a fork? +# +# None of this can be asked from Haskell alone. A forkIO thread is not an +# operating system thread, and the Haskell test suite cannot fork the way a +# child process must for the question to mean anything. +# +# Usage: cbits/tests/sysdrg/run.sh [build-dir] +set -eu + +cbits=$(cd "$(dirname "$0")/../.." && pwd) +out=${1:-$(mktemp -d)} + +${CC:-cc} -O2 -Wall -Wextra -DCRYPTON_SYSDRG_TESTING -I"$cbits" -o "$out/sysdrg" \ + "$cbits/tests/sysdrg/sysdrg.c" \ + "$cbits/crypton_sysdrg.c" \ + "$cbits/crypton_sysrandom.c" \ + "$cbits/crypton_chacha.c" \ + "$cbits/crypton_sha512.c" + +"$out/sysdrg" diff --git a/cbits/tests/sysdrg/sysdrg.c b/cbits/tests/sysdrg/sysdrg.c new file mode 100644 --- /dev/null +++ b/cbits/tests/sysdrg/sysdrg.c @@ -0,0 +1,140 @@ +#include +#include +#include +#include +#include +#include +#include + +void crypton_sysdrg_test_key(uint8_t out[32]); +void crypton_sysdrg_test_lock(void); +void crypton_sysdrg_test_unlock(void); +int crypton_sysdrg_bytes(uint8_t *out, int len); +uint32_t crypton_sysdrg_generation(void); +uint64_t crypton_sysdrg_thread_used(void); +static uint64_t used_in_thread; + +static int fail = 0; +static void check(const char *what, int ok) { + printf("%-52s %s\n", what, ok ? "ok" : "FAIL"); + if (!ok) fail = 1; +} + +/* Holds the process generator's lock for a moment, so that a fork can be + * made to happen while it is held. Releasing after a delay rather than on + * demand is deliberate: a prepare handler has to be able to take the lock, + * and a holder that waited to be told would stop the fork instead of the + * child. */ +static pthread_mutex_t gate = PTHREAD_MUTEX_INITIALIZER; +static pthread_cond_t gate_cv = PTHREAD_COND_INITIALIZER; +static int holding = 0; + +static void *lock_holder(void *arg) { + (void) arg; + crypton_sysdrg_test_lock(); + pthread_mutex_lock(&gate); + holding = 1; + pthread_cond_broadcast(&gate_cv); + pthread_mutex_unlock(&gate); + usleep(200000); + crypton_sysdrg_test_unlock(); + return NULL; +} + +static void *thread_body(void *arg) { + uint8_t *out = (uint8_t *) arg; + check("a second OS thread gets bytes", crypton_sysdrg_bytes(out, 32) == 32); + used_in_thread = crypton_sysdrg_thread_used(); + return NULL; +} + +int main(void) { + uint8_t a[32], b[32], big[4096]; + memset(a, 0, 32); memset(b, 0, 32); + + check("asking for 32 bytes gives 32", crypton_sysdrg_bytes(a, 32) == 32); + check("asking again gives something else", + crypton_sysdrg_bytes(b, 32) == 32 && memcmp(a, b, 32) != 0); + check("a 4096-byte request is filled", crypton_sysdrg_bytes(big, 4096) == 4096); + check("zero length is fine", crypton_sysdrg_bytes(a, 0) == 0); + + /* two OS threads must not share a stream. The main thread has already + * produced over 4 KiB by here, so a shared generator would show it. */ + uint8_t t1[32], t2[32]; + pthread_t p1, p2; + pthread_create(&p1, NULL, thread_body, t1); + pthread_join(p1, NULL); + pthread_create(&p2, NULL, thread_body, t2); + pthread_join(p2, NULL); + check("two OS threads do not produce the same bytes", memcmp(t1, t2, 32) != 0); + check("a new OS thread starts its own stream, not the main one's", + used_in_thread == 32); + + /* crossing the per-thread reseed limit (1 MiB) must not repeat */ + uint8_t before[32], after[32]; + crypton_sysdrg_bytes(before, 32); + for (int i = 0; i < 1100; i++) { uint8_t junk[1024]; crypton_sysdrg_bytes(junk, 1024); } + crypton_sysdrg_bytes(after, 32); + check("output after a reseed differs from before", memcmp(before, after, 32) != 0); + + /* fork: parent and child must diverge */ + int fds[2]; + if (pipe(fds) != 0) { perror("pipe"); return 2; } + uint8_t parent[32], child[32]; + pid_t pid = fork(); + if (pid == 0) { + close(fds[0]); + crypton_sysdrg_bytes(child, 32); + ssize_t w = write(fds[1], child, 32); + _exit(w == 32 ? 0 : 1); + } + close(fds[1]); + crypton_sysdrg_bytes(parent, 32); + ssize_t r = read(fds[0], child, 32); + int status = 0; waitpid(pid, &status, 0); + check("the child was able to produce bytes", r == 32 && status == 0); + check("parent and child do not produce the same bytes", memcmp(parent, child, 32) != 0); + check("the fork was noticed", crypton_sysdrg_generation() == 0); + + /* Backtracking resistance. The key that produced a draw is replaced + * once the bytes are out, so a state read afterwards is not the state + * that made them and cannot be wound back to remake them. Comparing + * the key before and against the key after is the whole of it: without + * the rekey it is the same key and the counter alone says where to + * start. */ + uint8_t key_before[32], key_after[32], drawn[32]; + memset(key_before, 0, 32); memset(key_after, 0, 32); + crypton_sysdrg_test_key(key_before); + crypton_sysdrg_bytes(drawn, 32); + crypton_sysdrg_test_key(key_after); + check("a draw replaces the key that made it", + memcmp(key_before, key_after, 32) != 0); + + /* A fork while another thread holds the process generator's lock. The + * child's first draw has to reseed, the generation having changed, and + * reseeding takes that lock. With no prepare and parent handlers the + * child inherits it locked and waits on it for ever -- the thread that + * would release it did not come across the fork. The alarm is what + * turns that into a failure rather than a hung test run. */ + pthread_t holder; + if (pthread_create(&holder, NULL, lock_holder, NULL) != 0) { + perror("pthread_create"); return 2; + } + pthread_mutex_lock(&gate); + while (!holding) pthread_cond_wait(&gate_cv, &gate); + pthread_mutex_unlock(&gate); + + pid_t locked_pid = fork(); + if (locked_pid == 0) { + uint8_t c[32]; + alarm(5); + _exit(crypton_sysdrg_bytes(c, 32) == 32 ? 0 : 1); + } + pthread_join(holder, NULL); + int locked_status = 0; + waitpid(locked_pid, &locked_status, 0); + check("a child forked while the lock was held can draw", + WIFEXITED(locked_status) && WEXITSTATUS(locked_status) == 0); + + return fail; +} diff --git a/crypton.cabal b/crypton.cabal --- a/crypton.cabal +++ b/crypton.cabal @@ -1,10 +1,11 @@ cabal-version: 3.0 name: crypton -version: 2.1.10 +version: 2.2.5 -- crypton's own code is BSD-3-Clause. The parts of -- cbits/aes/gcm_fused_x86.c that follow picotls's fusion are MIT, and the --- vendored s2n-bignum assembly in cbits/s2n is taken under ISC; each has --- its licence beside it, and they are listed below. The CRYPTOGAMS +-- vendored s2n-bignum assembly in cbits/s2n is taken under ISC, and the +-- constant-time AES and GHASH in cbits/bearssl are BearSSL's under MIT; +-- each has its licence beside it, and they are listed below. The CRYPTOGAMS -- assembly in cbits/asm is taken under its BSD-3-Clause option, and the -- AArch64 multiply-accumulate loop in cbits/crypton_bignum.h follows Go's, -- which is BSD-3-Clause too; the first term already covers both. @@ -15,6 +16,7 @@ cbits/aes/LICENSE.fusion cbits/asm/LICENSE.cryptogams cbits/s2n/LICENSE + cbits/bearssl/LICENSE cbits/mlkem/LICENSE cbits/mldsa/LICENSE copyright: @@ -46,6 +48,9 @@ cbits/asm/README.md cbits/asm/aesni-gcm-x86_64.pl cbits/asm/arm-xlate.pl + cbits/asm/aesp8-ppc.pl + cbits/asm/ghashp8-ppc.pl + cbits/asm/ppc-xlate.pl cbits/asm/arm_arch.h cbits/asm/chacha-armv8.pl cbits/asm/chacha-x86_64.pl @@ -64,6 +69,8 @@ cbits/blake2/ref/*.h cbits/blake2/sse/*.h cbits/crypton_hash_prefix.c + cbits/crypton_mont.py + cbits/crypton_powm_ifma.py cbits/mlkem/*.S cbits/mlkem/*.c cbits/mlkem/*.h @@ -146,6 +153,12 @@ cbits/s2n/import.sh cbits/s2n/include/*.h cbits/s2n/x86_att/*.S + cbits/s2n/x86_att/*.h + cbits/bearssl/README.md + cbits/bearssl/VERSION + cbits/bearssl/import.sh + cbits/bearssl/inner.h + cbits/tests/*.c cbits/tests/ct/*.c cbits/tests/ct/*.h cbits/tests/ct/known.txt @@ -167,9 +180,13 @@ cbits/tests/scrub/README cbits/tests/scrub/known.txt cbits/tests/scrub/run.sh + cbits/tests/sysdrg/*.c + cbits/tests/sysdrg/run.sh cbits/tests/width/*.c cbits/tests/width/run.sh tests/*.hs + tests/tutorial/extract.awk + tests/tutorial/run.sh extra-doc-files: CHANGELOG.md @@ -185,12 +202,6 @@ manual: True -flag support_rdrand - description: - allow compilation with RDRAND on system and architecture that supports it - - manual: True - flag support_pclmuldq description: Allow compilation with pclmuldq on architecture that supports it @@ -319,6 +330,7 @@ Crypto.PubKey.Rabin.RW Crypto.PubKey.Rabin.Types Crypto.PubKey.RSA + Crypto.PubKey.RSA.Blinded Crypto.PubKey.RSA.OAEP Crypto.PubKey.RSA.PKCS15 Crypto.PubKey.RSA.Prim @@ -355,6 +367,17 @@ cbits/crypton_chacha.c cbits/crypton_chachapoly.c cbits/crypton_cpu.c + cbits/crypton_sysdrg.c + cbits/crypton_sysrandom.c + -- BearSSL's constant-time AES and GHASH, which cbits/aes/generic.c + -- and cbits/aes/gf.c are written on top of. Unconditional because + -- those two are: every one of the three AES branches below names + -- them, accelerated or not. + cbits/bearssl/aes_ct64.c + cbits/bearssl/aes_ct64_dec.c + cbits/bearssl/aes_ct64_enc.c + cbits/bearssl/dec32le.c + cbits/bearssl/ghash_ctmul64.c cbits/crypton_des.c cbits/crypton_ecc.c cbits/crypton_f2m.c @@ -365,6 +388,7 @@ cbits/crypton_pbkdf2.c cbits/crypton_poly1305.c cbits/crypton_powm.c + cbits/crypton_powm_ifma.c cbits/crypton_rc4.c cbits/crypton_ripemd.c cbits/crypton_salsa.c @@ -384,6 +408,8 @@ cbits/p256/p256_ec.c other-modules: + Crypto.Number.ModArithmetic.Pub + Crypto.PubKey.RSA.Blinded.Internal Crypto.Cipher.AES.Primitive Crypto.Cipher.Blowfish.Primitive Crypto.Cipher.Camellia.Primitive @@ -441,6 +467,7 @@ Crypto.Random.Entropy.Source Crypto.Random.HmacDRG Crypto.Random.Probabilistic + Crypto.Random.SysDRG Crypto.Random.SystemDRG default-language: Haskell2010 @@ -576,11 +603,17 @@ -- unlike OpenSSL's and BoringSSL's ecp_nistz256 it can be used. Both -- variants of each routine are built and chosen between in -- cbits/p256/p256_s2n.c -- see cbits/s2n/README.md for why the question - -- is a different one on the two architectures. Windows is left out for - -- now: the vendored files carry ELF and Mach-O directives and nothing - -- for COFF, the same reason the CRYPTOGAMS AArch64 assembly above skips - -- it. x86-64 there wants -DWINDOWS_ABI=1 as well. - if (flag(support_s2n_bignum) && (arch(x86_64) || arch(aarch64)) && !os(windows)) + -- is a different one on the two architectures. + -- + -- On Windows, x86-64 only. The vendored files carry the ELF directives + -- .type and .size, which neither of Windows' assemblers accepts, and + -- assume the System V calling convention unless WINDOWS_ABI is set; + -- cbits/s2n/x86_att/_internal_s2n_bignum_x86_att.h, which every one of + -- them reaches first, empties the first and sets the second on _WIN32, + -- so the vendored files themselves stay as upstream wrote them. The + -- AArch64 files have the same directives and no such header yet, and no + -- runner to check one, so Windows on ARM keeps the C. + if flag(support_s2n_bignum) && (arch(x86_64) || (arch(aarch64) && !os(windows))) cc-options: -DCRYPTON_S2N_BIGNUM include-dirs: cbits/s2n/include c-sources: @@ -593,6 +626,7 @@ if arch(aarch64) asm-sources: cbits/s2n/arm/bignum_deamont_p384.S + cbits/s2n/arm/bignum_montifier.S cbits/s2n/arm/bignum_modinv.S cbits/s2n/arm/curve25519_x25519.S cbits/s2n/arm/curve25519_x25519_alt.S @@ -670,21 +704,31 @@ cbits/s2n/x86_att/p256_scalarmulbase.S cbits/s2n/x86_att/p256_scalarmulbase_alt.S - -- Modular exponentiation at RSA sizes, x86-64 only: on AArch64 - -- crypton's C measures faster than these, so nothing is taken - -- for it. All of them want ADX, which the run-time check in - -- cbits/crypton_powm.c asks about before using any. asm-sources: - cbits/s2n/x86_att/bignum_emontredc_8n.S - cbits/s2n/x86_att/bignum_kmul_16_32.S - cbits/s2n/x86_att/bignum_kmul_32_64.S - cbits/s2n/x86_att/bignum_ksqr_16_32.S - cbits/s2n/x86_att/bignum_ksqr_32_64.S + cbits/s2n/x86_att/bignum_montifier.S cbits/s2n/x86_att/p384_montjscalarmul.S cbits/s2n/x86_att/p384_montjscalarmul_alt.S cbits/s2n/x86_att/p521_jscalarmul.S cbits/s2n/x86_att/p521_jscalarmul_alt.S + -- Modular exponentiation at RSA sizes, x86-64 only: on AArch64 + -- crypton's C measures faster than these, so nothing is taken + -- for it. All of them want ADX, which the run-time check in + -- cbits/crypton_powm.c asks about before using any. + -- Not on macOS: Apple's assembler reads $0 in bignum_emontredc_8n's + -- macros as a macro argument and rejects the file. + if !os(osx) + asm-sources: + cbits/s2n/x86_att/bignum_emontredc_8n.S + cbits/s2n/x86_att/bignum_mul.S + cbits/s2n/x86_att/bignum_sqr.S + cbits/s2n/x86_att/bignum_kmul_16_32.S + cbits/s2n/x86_att/bignum_kmul_32_64.S + cbits/s2n/x86_att/bignum_ksqr_16_32.S + cbits/s2n/x86_att/bignum_ksqr_32_64.S + cbits/s2n/x86_att/bignum_sqr_8_16.S + cbits/s2n/x86_att/bignum_mul_8_16.S + -- The PQ Code Package's mlkem-native and mldsa-native, vendored under -- cbits/mlkem and cbits/mldsa. Apache-2.0 OR ISC OR MIT, the same -- three-way form as s2n-bignum above, and by the same authors, so @@ -716,18 +760,17 @@ cbits/mldsa/crypton_mldsa_asm.S -- What the two packages' own build systems pass for the x86-64 - -- backend; without them the AVX2 sources do not assemble. AArch64 - -- needs nothing beyond the baseline for the arithmetic, and takes - -- the armv8.4 Keccak from the -march the block below already sets. + -- backend, and only for the assembly: cc-options in a conditional + -- are the component's, not the block's, so -mavx2 here would raise + -- the baseline of every C file in the library -- Ed25519, Argon2, + -- P-256 and the rest -- and gcc would vectorise code that asks for + -- no such thing, with nothing to check at runtime. The backends' + -- own AVX2 is all in the .S. AArch64 needs nothing beyond the + -- baseline for the arithmetic, and takes the armv8.4 Keccak from + -- the -march the block below already sets. if arch(x86_64) - cc-options: -mavx2 -mbmi2 asm-options: -mavx2 -mbmi2 - if ((flag(support_rdrand) && (arch(i386) || arch(x86_64))) && !os(windows)) - cpp-options: -DSUPPORT_RDRAND - c-sources: cbits/crypton_rdrand.c - other-modules: Crypto.Random.Entropy.RDRand - if (flag(support_aesni) && arch(aarch64)) cc-options: -DWITH_ARMV8_CRYPTO c-sources: @@ -739,6 +782,52 @@ if !flag(use_target_attributes) cc-options: -march=armv8-a+crypto + -- The same instructions, reached from AArch32. cbits/aes/armv8.c needs + -- nothing changed to compile here: every intrinsic it uses exists in both + -- execution states, including the PMULL2 it takes through + -- vmull_high_p64, which the compiler lowers to a pair of VMULL.P64. What + -- differs is how the operating system reports the instructions -- + -- AArch32 has filled AT_HWCAP with older features and puts these in + -- AT_HWCAP2 -- which cbits/crypton_cpu.c now asks for. + -- + -- -mfpu is passed whatever use_target_attributes says, unlike the branch + -- above. AArch32 names an FPU rather than an architecture extension and + -- the two compilers spell the function attribute for it differently; a + -- global option is safe here, since a compiler emits these instructions + -- where an intrinsic asks for them and nowhere else. + -- + -- A soft-float ARM that cannot take -mfpu fails loudly at the first C + -- file rather than quietly, and -f-support_aesni is the way out. + if (flag(support_aesni) && arch(arm)) + cc-options: -DWITH_ARMV8_CRYPTO -mfpu=crypto-neon-fp-armv8 + c-sources: + cbits/aes/generic.c + cbits/aes/gf.c + cbits/aes/armv8.c + cbits/crypton_aes.c + + -- POWER8's vector AES and vector carry-less multiply, through the + -- CRYPTOGAMS assembly in cbits/asm. No cc-options beyond the define: + -- cbits/aes/ppc8.c calls that assembly rather than using intrinsics of + -- its own, and the assembly says .machine "any", so the assembler takes + -- the instructions without being told about the processor. + -- + -- Little-endian only. The generator emits a big-endian flavour from the + -- same source, and it is not checked in: nothing here has been able to + -- run it, and an AES path that has never been executed is not one to + -- ship. cbits/asm/generate.sh says what adding it would take. + if (flag(support_aesni) && arch(ppc64le)) + cc-options: -DWITH_PPC8_CRYPTO + c-sources: + cbits/aes/generic.c + cbits/aes/gf.c + cbits/aes/ppc8.c + cbits/crypton_aes.c + + asm-sources: + cbits/asm/aesp8-ppc-linux64le.S + cbits/asm/ghashp8-ppc-linux64le.S + if arch(aarch64) cc-options: -DWITH_ARMV8_SHA1 -DWITH_ARMV8_SHA2 -DWITH_ARMV8_SHA3 @@ -864,7 +953,7 @@ -- branch had already named every file it names. Cabal drops the repeats, -- so nothing was built twice, but the line read as the fallback for a -- platform with no AES instructions and was not one. - if !((flag(support_aesni) && arch(aarch64)) || (flag(support_aesni) && (arch(i386) || arch(x86_64)))) + if !((flag(support_aesni) && (arch(aarch64) || arch(arm))) || (flag(support_aesni) && (arch(i386) || arch(x86_64))) || (flag(support_aesni) && arch(ppc64le))) c-sources: cbits/aes/generic.c cbits/aes/gf.c @@ -901,7 +990,9 @@ build-depends: Win32 <2.15 else - other-modules: Crypto.Random.Entropy.Unix + other-modules: + Crypto.Random.Entropy.SysRandom + Crypto.Random.Entropy.Unix if (impl(ghc >=0) && flag(integer-gmp)) build-depends: integer-gmp <1.2 @@ -986,7 +1077,9 @@ PubKey.RabinSpec PubKey.SecrecySpec PubKey.RSASpec + EntropySpec RuntimeSpec + SysDRGSpec StreamCipher.ChaChaPoly1305Spec StreamCipher.ChaChaSpec StreamCipher.RC4Spec @@ -1007,6 +1100,45 @@ default-language: Haskell2010 ghc-options: -Wall -fno-warn-orphans -fno-warn-missing-signatures -rtsopts + -threaded "-with-rtsopts=-N2" + +-- The generator behind MonadRandom and a fork made from Haskell. Its own +-- executables because the suite above runs with -N2, where GHC does not +-- support forkProcess, and because the answer may differ between the +-- threaded runtime and the one without it -- so it is built for both. +-- Not on Windows, which has no fork and no unix package. +test-suite test-forkprocess-threaded + type: exitcode-stdio-1.0 + main-is: ForkProcess.hs + hs-source-dirs: tests/forkprocess + default-language: Haskell2010 + ghc-options: + -Wall -rtsopts -threaded "-with-rtsopts=-N1" + + if os(windows) + buildable: False + else + build-depends: + base >=4.13 && <5, + bytestring, + crypton, + unix + +test-suite test-forkprocess-unthreaded + type: exitcode-stdio-1.0 + main-is: ForkProcess.hs + hs-source-dirs: tests/forkprocess + default-language: Haskell2010 + ghc-options: -Wall -rtsopts + + if os(windows) + buildable: False + else + build-depends: + base >=4.13 && <5, + bytestring, + crypton, + unix benchmark bench-crypton type: exitcode-stdio-1.0 diff --git a/tests/EntropySpec.hs b/tests/EntropySpec.hs new file mode 100644 --- /dev/null +++ b/tests/EntropySpec.hs @@ -0,0 +1,58 @@ +-- | The system source of entropy. +-- +-- There is nothing here to check the bytes against -- they are supposed to +-- be unpredictable, and a test that said otherwise would be a test of the +-- kernel. What can be checked is that the right number of them comes back, +-- which is where an implementation of the call with a limit per request +-- goes wrong: @getentropy(3)@ refuses more than 256 bytes at a time, so a +-- backend that forgets to loop answers a short buffer for anything larger. +module EntropySpec (spec) where + +import Control.Exception (try) +import qualified Data.ByteString as BS +import Data.Maybe (catMaybes) +import Foreign.Marshal.Alloc (allocaBytes) +import Test.Hspec + +import Crypto.Random.Entropy (EntropyError (..), getEntropy) +import Crypto.Random.Entropy.Unsafe (gatherBackend, replenish, supportedBackends) + +spec :: Spec +spec = describe "the system entropy source" $ do + mapM_ lengthCase [0, 1, 31, 32, 255, 256, 257, 512, 1000] + + it "does not answer the same thing twice" $ do + a <- getEntropy 64 :: IO BS.ByteString + b <- getEntropy 64 :: IO BS.ByteString + a `shouldNotBe` b + + it "is not answering a constant" $ do + b <- getEntropy 1024 :: IO BS.ByteString + BS.length (BS.filter (== BS.head b) b) `shouldSatisfy` (< 64) + + -- getEntropy would not notice: replenish tops a short answer up from + -- the next backend, so the buffer comes back full either way and the + -- system call is quietly replaced by the device file. The backend has + -- to be asked on its own. + it "the best backend fills a buffer larger than one request on its own" $ do + bs <- catMaybes `fmap` sequence supportedBackends + case bs of + [] -> expectationFailure "no source of entropy on this system" + (b : _) -> do + n <- allocaBytes 1000 $ \ptr -> gatherBackend b ptr 1000 + n `shouldBe` 1000 + + -- A system with no source of entropy is the one failure this library + -- cannot answer, and it used to be an `error`, which a caller could not + -- tell from a bug. There is no way to take the sources away from a + -- running machine, but replenish can be handed the empty list, which is + -- the same question. + it "says so with an exception when there is no source at all" $ do + r <- allocaBytes 16 $ \ptr -> try (replenish 16 [] ptr) + r `shouldBe` Left NoEntropySource + +lengthCase :: Int -> Spec +lengthCase n = + it ("gives back the " ++ show n ++ " bytes asked for") $ do + b <- getEntropy n :: IO BS.ByteString + BS.length b `shouldBe` n diff --git a/tests/NumberSpec.hs b/tests/NumberSpec.hs --- a/tests/NumberSpec.hs +++ b/tests/NumberSpec.hs @@ -202,6 +202,21 @@ forAll (wideOf bytes) $ \e -> forAll (wideOf 128) $ \b -> expSafe b e modulus1024 === expFast b e modulus1024 + -- the C takes RSA's exponent at these sizes on some machines; plain + -- squaring shares no multiplication with it + prop "agrees with plain squaring on RSA's exponent at every key size" $ + forAll (elements [128, 192, 256, 384, 512]) $ \bytes -> + forAll (wideOf bytes) $ \m' -> + forAll (wideOf bytes) $ \b -> + let m = m' .|. bit (8 * bytes - 1) .|. 1 + in expFast b 65537 m === naivePow (b `mod` m) 65537 m + -- the fast one keeps R^2 for the last few moduli; asking under more of + -- them than it keeps, round and round, would show one used for another + it "keeps each modulus' constant to that modulus" $ do + let ms = [modulus2048 - 2 * 0x10001 * fromIntegral k | k <- [0 .. 11 :: Int]] + bs = [modulus2048 `div` (3 + fromIntegral k) | k <- [0 .. 11 :: Int]] + asked = concat (replicate 3 (zip bs ms)) + [(b, m) | (b, m) <- asked, expFast b 65537 m /= expSafe b 65537 m] `shouldBe` [] where safely (b, e, m) = expSafe b e m fastly (b, e, m) = expFast b e m diff --git a/tests/PubKey/MGF1Spec.hs b/tests/PubKey/MGF1Spec.hs --- a/tests/PubKey/MGF1Spec.hs +++ b/tests/PubKey/MGF1Spec.hs @@ -10,7 +10,11 @@ import Data.ByteString.Char8 () import Crypto.Hash +import qualified Data.ByteArray as BA +import Crypto.Number.Serialize (i2ospOf_) import Crypto.PubKey.MaskGenFunction +import Test.Hspec.QuickCheck (prop) +import Test.QuickCheck (choose, forAll) import Utils @@ -32,5 +36,26 @@ } ] +-- | RFC 8017's definition through the pure hash API, block by block, which +-- shares nothing with mgf1's in-place contexts or with the one-shot hash. +reference :: HashAlgorithm a => a -> ByteString -> Int -> ByteString +reference alg sd len = B.take len $ B.concat [block c | c <- [0 .. blocks - 1]] + where + blocks = (len + hashDigestSize alg - 1) `div` hashDigestSize alg + block c = BA.convert (hashFinalize (hashUpdates (hashInitWith alg) [sd, i2ospOf_ 4 (fromIntegral c)])) :: ByteString + spec :: Spec -spec = zipWithM_ doMGFTest [katZero ..] vectorsMGF +spec = do + zipWithM_ doMGFTest [katZero ..] vectorsMGF + describe "against the definition" $ + mapM_ agree + [ ("SHA1", \s n -> (mgf1 SHA1 s n, reference SHA1 s n)) + , ("SHA256", \s n -> (mgf1 SHA256 s n, reference SHA256 s n)) + , ("SHA512", \s n -> (mgf1 SHA512 s n, reference SHA512 s n)) + , ("SHA3_256", \s n -> (mgf1 SHA3_256 s n, reference SHA3_256 s n)) + , ("Blake2b_512", \s n -> (mgf1 Blake2b_512 s n, reference Blake2b_512 s n)) + ] + where + agree (name, f) = prop name $ + forAll (arbitraryBSof 0 100) $ \s -> forAll (choose (0, 300)) $ \n -> + let (got, want) = f s n in got == (want :: ByteString) diff --git a/tests/PubKey/RSASpec.hs b/tests/PubKey/RSASpec.hs --- a/tests/PubKey/RSASpec.hs +++ b/tests/PubKey/RSASpec.hs @@ -1,16 +1,26 @@ +-- The properties below hold the new entry points against sign, signSafer +-- and verify, which are deprecated as of this release. Comparing against +-- them is the point, so the warning is off here and nowhere else. +{-# OPTIONS_GHC -Wno-deprecations #-} {-# LANGUAGE ExistentialQuantification #-} {-# LANGUAGE OverloadedStrings #-} module PubKey.RSASpec (spec) where +import Control.Concurrent (forkIO) +import Control.Concurrent.MVar import Crypto.Hash import Crypto.Number.ModArithmetic (inverse) import Crypto.Number.Serialize (i2osp, i2ospOf_, os2ip) import qualified Crypto.PubKey.RSA as RSA +import qualified Crypto.PubKey.RSA.Blinded as RSA +import qualified Crypto.PubKey.RSA.OAEP as OAEP import qualified Crypto.PubKey.RSA.PKCS15 as RSA +import qualified Crypto.PubKey.RSA.PSS as PSS import Crypto.PubKey.RSA.Prim (ep) import qualified Crypto.PubKey.RSA.Prim as Prim import qualified Data.ByteString as B +import Data.Bits ((.|.)) import Data.Either import Imports @@ -232,6 +242,68 @@ , os2ip ct + modulus < 2 ^ (8 * k) ] +-- | The public-key operation takes bytes to the C and back where it can, and +-- the 'Integer' path where it cannot: a representative not below the +-- modulus, or one of another length. Either way the answer is what plain +-- squaring gives, which shares nothing with the C. +publicOperationTests :: Spec +publicOperationTests = describe "public-key operation" $ do + prop "agrees with plain squaring" $ + forAll (elements [128, 256, 384]) $ \k -> + forAll (arbitraryBS k) $ \nb -> + forAll (arbitraryBS k) $ \m -> + let pk = keyOf k nb in ep pk m === expected pk m + prop "agrees at the edges of the modulus and of the length" $ + forAll (elements [128, 256, 384]) $ \k -> + forAll (arbitraryBS k) $ \nb -> + let pk = keyOf k nb + nn = RSA.public_n pk + edges = + [i2ospOf_ k x | x <- [0, 1, nn - 1, nn, nn + 1], x < 2 ^ (8 * k)] + ++ [i2ospOf_ (k - 1) (nn `div` 256), B.cons 0 (i2ospOf_ k (nn - 1))] + in [m | m <- edges, ep pk m /= expected pk m] === [] + where + -- an odd modulus of exactly k bytes, which is all the operation needs + keyOf k nb = + RSA.PublicKey k (os2ip nb .|. (2 ^ (8 * k - 1)) .|. 1) 0x10001 + -- b^65537 as b squared sixteen times, times b + expected pk m = i2ospOf_ (RSA.public_size pk) (squared 16 b * b `mod` nn) :: ByteString + where + nn = RSA.public_n pk + b = os2ip m `mod` nn + squared :: Int -> Integer -> Integer + squared 0 x = x + squared i x = squared (i - 1) (x * x `mod` nn) + +-- | A key whose CRT values are off by one stands in for a fault in one half +-- of the computation, which would otherwise hand out a signature whose gcd +-- with n is a prime. Every private-key path has to answer as the sound key. +crtFaultTests :: Spec +crtFaultTests = describe "a fault in one CRT half" $ + forM_ faults $ \(name, bad) -> describe name $ do + it "signs what the sound key signs" $ + RSA.sign Nothing (Just SHA256) bad m `shouldBe` RSA.sign Nothing (Just SHA256) key m + prop "signs with a blinder what the sound key signs" $ \testDRG -> + let blinder = withTestDRG testDRG $ RSA.generateBlinder (RSA.public_n pub) + in RSA.sign (Just blinder) (Just SHA256) bad m + === RSA.sign Nothing (Just SHA256) key m + prop "signs through signSafer what the sound key signs" $ \testDRG -> + withTestDRG testDRG (RSA.signSafer (Just SHA256) bad m) + === RSA.sign Nothing (Just SHA256) key m + it "decrypts what it was given" $ do + c <- either (error . show) id <$> RSA.encrypt pub m + RSA.decrypt Nothing bad c `shouldBe` (Right m :: Either RSA.Error ByteString) + where + vector = firstVector vectorsSHA1 + key = vectorToPrivate vector + pub = vectorToPublic vector + m = "a message to sign" :: ByteString + faults = + [ ("dP", key{RSA.private_dP = RSA.private_dP key + 1}) + , ("dQ", key{RSA.private_dQ = RSA.private_dQ key + 1}) + , ("qinv", key{RSA.private_qinv = RSA.private_qinv key + 1}) + ] + -- | Building a key from its two primes has to arrive at the key the vectors -- carry -- the private exponent, both of its halves, and the inverse of one -- prime modulo the other, which is the part worked out without the extended @@ -423,6 +495,110 @@ key = vectorToPrivate vector pub = vectorToPublic vector +-- | A 'RSA.BlindedKey' carries one blinding pair, each operation returning +-- the key with it squared and a new one drawn every 32 uses; the @Safer@ +-- functions keep one per key for the last few keys. None of it may change +-- an answer, so every answer is held against the unblinded one, over more +-- uses than one pair lasts. +blindedKeyTests :: Spec +blindedKeyTests = describe "BlindedKey" $ do + it "signs what signDigest Nothing signs, across refreshes" $ + threaded 100 $ \bk i -> do + let dg = hashWith SHA256 (B.replicate i 0x61) + (got, bk') <- RSA.signDigestBlinded bk dg + got `shouldBe` RSA.signDigest Nothing key dg + return bk' + + it "signs a DigestInfo as signDigestInfo Nothing does" $ + threaded 40 $ \bk i -> do + let di = B.replicate (20 + i `mod` 10) (fromIntegral i) + (got, bk') <- RSA.signDigestInfoBlinded bk di + got `shouldBe` RSA.signDigestInfo Nothing key di + return bk' + + it "decrypts what it was given" $ + threaded 40 $ \bk i -> do + let m = B.replicate i 0x5a + c <- either (error . show) id <$> RSA.encrypt pub m + (got, bk') <- RSA.decryptBlinded bk c + got `shouldBe` (Right m :: Either RSA.Error ByteString) + return bk' + + it "refuses what decrypt refuses" $ do + let c = B.replicate (RSA.public_size pub + 1) 1 + (got, _) <- RSA.decryptBlinded (RSA.blindedKey key) c + got `shouldBe` (Left RSA.MessageSizeIncorrect :: Either RSA.Error ByteString) + + it "makes PSS signatures that verify" $ + threaded 40 $ \bk i -> do + let m = B.replicate i 0x33 + params = PSS.defaultPSSParams SHA256 + (s, bk') <- PSS.signBlinded params bk m + either (error . show) (PSS.verify params pub m) s `shouldBe` True + return bk' + + it "decrypts OAEP" $ + threaded 40 $ \bk i -> do + let m = B.replicate i 0x77 + params = OAEP.defaultOAEPParams SHA256 + c <- either (error . show) id <$> OAEP.encrypt params pub m + (got, bk') <- OAEP.decryptBlinded params bk c + got `shouldBe` Right m + return bk' + + it "works for a key without its CRT values" $ do + let bare = key {RSA.private_p = 0, RSA.private_q = 0} + go _ 41 = return () + go bk i = do + let dg = hashWith SHA256 (B.replicate i 0x62) + (got, bk') <- RSA.signDigestBlinded bk dg + got `shouldBe` RSA.signDigest Nothing key dg + go bk' (i + 1) + go (RSA.blindedKey bare) (1 :: Int) + + -- The randomness comes from the monad, so a pure generator will do. + prop "runs in a pure generator" $ \testDRG -> + let go bk i acc + | i > (40 :: Int) = return (reverse acc) + | otherwise = do + (s, bk') <- RSA.signDigestBlinded bk (hashWith SHA256 (B.replicate i 0x64)) + go bk' (i + 1) (s : acc) + sigs = withTestDRG testDRG (go (RSA.blindedKey key) 1 []) + in sigs === [RSA.signDigest Nothing key (hashWith SHA256 (B.replicate i 0x64)) | i <- [1 .. 40]] + + it "gives every thread a right answer through the Safer functions" $ do + dones <- forM [1 .. 8 :: Int] $ \t -> do + done <- newEmptyMVar + _ <- forkIO $ do + oks <- forM [1 .. 30 :: Int] $ \i -> do + let dg = hashWith SHA256 (B.pack [fromIntegral t, fromIntegral i]) + got <- RSA.signSaferDigest key dg + return $! got == RSA.signDigest Nothing key dg + putMVar done (and oks) + return done + results <- mapM takeMVar dones + results `shouldBe` replicate 8 True + + -- More keys than the Safer functions remember, interleaved, so that the + -- table both hits and evicts. + it "keeps the Safer functions right across more keys than it remembers" $ do + let keys = key : [snd (withTestDRG (TestDRG (s', 2, 3, 4, 5)) (RSA.generate 64 65537)) | s' <- [1 .. 5]] + forM_ [1 .. 12 :: Int] $ \i -> + forM_ keys $ \k -> do + let dg = hashWith SHA256 (B.replicate i 0x63) + got <- RSA.signSaferDigest k dg + got `shouldBe` RSA.signDigest Nothing k dg + where + vector = firstVector vectorsSHA1 + key = vectorToPrivate vector + pub = vectorToPublic vector + -- run count operations, each handed the key the one before returned + threaded count op = go (RSA.blindedKey key) (1 :: Int) + where + go bk i + | i > count = return () + | otherwise = op bk i >>= \bk' -> go bk' (i + 1) + spec :: Spec spec = do keyGenerationTests @@ -439,5 +615,8 @@ zipWith doMalleabilityTest [katZero ..] $ filter vectorHasSignature vectorsSHA1 digestOperationTests + blindedKeyTests unpadTests ciphertextRangeTests + publicOperationTests + crtFaultTests diff --git a/tests/RuntimeSpec.hs b/tests/RuntimeSpec.hs --- a/tests/RuntimeSpec.hs +++ b/tests/RuntimeSpec.hs @@ -1,7 +1,72 @@ +-- | What 'processorOptions' says about the machine it is running on. +-- +-- A test cannot know what processor it is on, so it cannot check that the +-- answer is right. What it can check is that the answer is well formed: +-- that every name is distinct, that nothing is reported under a name from +-- another architecture, and that the two questions which do not name one +-- agree with the list. The list is printed as well, because a CI log that +-- says what each runner reported is the only way the detection itself gets +-- looked at. module RuntimeSpec (spec) where -import Crypto.System.CPU +import Data.List (nub, sort) import Test.Hspec +import Crypto.System.CPU ( + ProcessorOption (..), + hasAESAcceleration, + hasGHASHAcceleration, + processorOptions, + ) + +-- | Every name this module gives, which is also the set 'Show' has to +-- cover. +x86Options :: [ProcessorOption] +x86Options = + [AESNI, PCLMUL, SSSE3, AVX, AVX2, SHANI, MOVBE, ADX, VAES, VAES512] + +armOptions :: [ProcessorOption] +armOptions = [NEON, ARMAES, ARMPMULL, ARMSHA1, ARMSHA2, ARMSHA512] + +ppcOptions :: [ProcessorOption] +ppcOptions = [PPCAES, PPCVPMSUM] + spec :: Spec -spec = it "CPU" $ putStrLn (show processorOptions) +spec = describe "processorOptions" $ do + it "CPU" $ putStrLn (show processorOptions) + + it "gives every name a number of its own" $ + -- two patterns sharing a number would make one of them unreachable + -- and the other print under the wrong name + length (nub (x86Options ++ armOptions ++ ppcOptions)) + `shouldBe` length (x86Options ++ armOptions ++ ppcOptions) + + it "has a name for every option it names" $ + -- Show falls back to "ProcessorOption n" for what it does not know, + -- which is for values from a newer release, not for these + filter (startsWith "ProcessorOption " . show) (x86Options ++ armOptions ++ ppcOptions) + `shouldBe` [] + + it "reports each option at most once, in order" $ do + processorOptions `shouldBe` sort processorOptions + nub processorOptions `shouldBe` processorOptions + + it "does not mix one architecture's names with another's" $ do + let reported g = any (`elem` g) processorOptions + -- a machine reporting two of these groups is the + -- AArch64-says-AESNI fault this replaced + length (filter reported [x86Options, armOptions, ppcOptions]) + `shouldSatisfy` (<= 1) + + -- Half a check, and which half depends on the machine: where the + -- processor has AES this fails if the answer is broken to False and + -- passes if it is broken to True, and the other way round on a machine + -- without it. Both were tried. + it "answers the architecture-free questions from the same list" $ do + hasAESAcceleration + `shouldBe` any (`elem` processorOptions) [AESNI, ARMAES, PPCAES] + hasGHASHAcceleration + `shouldBe` any (`elem` processorOptions) [PCLMUL, ARMPMULL, PPCVPMSUM] + +startsWith :: String -> String -> Bool +startsWith p s = take (length p) s == p diff --git a/tests/SysDRGSpec.hs b/tests/SysDRGSpec.hs new file mode 100644 --- /dev/null +++ b/tests/SysDRGSpec.hs @@ -0,0 +1,100 @@ +-- | The generator behind 'MonadRandom' for 'IO'. +-- +-- What can be asked from Haskell is narrow. A @forkIO@ thread is not an +-- operating system thread, and this process cannot fork, so the questions +-- that matter most -- does a new operating system thread start its own +-- stream, does a child after @fork@ diverge from its parent -- are in +-- @cbits\/tests\/sysdrg@ instead. +-- +-- What is left is still worth asking: the generator keeps a lock and a +-- thread-local slot, and it is reached through a @safe@ foreign call, so +-- many Haskell threads drawing at once is exactly the shape that deadlocks +-- or hands two of them the same bytes. That needs @-threaded@ and more +-- than one capability, which is why the suite has them. +module SysDRGSpec (spec) where + +import Control.Concurrent ( + forkIO, + getNumCapabilities, + setNumCapabilities, + yield, + ) +import Control.Concurrent.MVar (newEmptyMVar, putMVar, takeMVar) +import Control.Exception (finally) +import Control.Monad (forM, forM_) +import qualified Data.ByteString as BS +import Data.List (group, nub, sort) +import Test.Hspec + +import Crypto.Random (getRandomBytes) + +spec :: Spec +spec = describe "the generator behind MonadRandom IO" $ do + it "runs on more than one capability, or the rest proves little" $ do + n <- getNumCapabilities + n `shouldSatisfy` (> 1) + + mapM_ lengthCase [0, 1, 32, 1000, 4096] + + it "gives every thread something different" $ do + -- plainly, rather than through async, which is not a dependency here + boxes <- forM [1 .. 256 :: Int] $ \_ -> do + box <- newEmptyMVar + _ <- forkIO $ do + b <- getRandomBytes 32 :: IO BS.ByteString + putMVar box b + return box + bss <- mapM takeMVar boxes + length (nub bss) `shouldBe` 256 + + it "keeps the streams apart while setNumCapabilities changes underneath" $ do + -- The state is held against the operating system thread rather than + -- against the capability, which is what makes this safe: a forkIO + -- thread moves between capabilities, and a capability is served by + -- different worker threads over its life, so state held against one + -- would be shared by threads running at the same time. Raising the + -- count makes the runtime create worker threads, each of which has + -- to start its own stream; lowering it disables capabilities under + -- threads that are drawing. + n0 <- getNumCapabilities + let flips = concat (replicate 3 [1, 2, min 8 (n0 * 2), n0]) + threads = 64 :: Int + draws = 16 :: Int + bss <- + ( do + flipped <- newEmptyMVar + _ <- forkIO $ do + forM_ flips $ \n -> setNumCapabilities n >> yield + putMVar flipped () + boxes <- forM [1 .. threads] $ \_ -> do + box <- newEmptyMVar + _ <- forkIO $ do + bs <- forM [1 .. draws] $ \_ -> do + b <- getRandomBytes 32 :: IO BS.ByteString + yield + return b + putMVar box bs + return box + bss <- concat <$> mapM takeMVar boxes + takeMVar flipped + return bss + ) + `finally` setNumCapabilities n0 + -- sorted and grouped rather than nub, which is quadratic and this + -- list is long enough for that to show + length bss `shouldBe` threads * draws + length (group (sort bss)) `shouldBe` threads * draws + + it "does not repeat across a reseed" $ do + -- the per-thread generator reseeds after a mebibyte + first <- getRandomBytes 32 :: IO BS.ByteString + forM_ [1 .. 300 :: Int] $ \_ -> + (getRandomBytes 4096 :: IO BS.ByteString) >>= \b -> b `seq` return () + second <- getRandomBytes 32 :: IO BS.ByteString + second `shouldNotBe` first + +lengthCase :: Int -> Spec +lengthCase n = + it ("gives back the " ++ show n ++ " bytes asked for") $ do + b <- getRandomBytes n :: IO BS.ByteString + BS.length b `shouldBe` n diff --git a/tests/forkprocess/ForkProcess.hs b/tests/forkprocess/ForkProcess.hs new file mode 100644 --- /dev/null +++ b/tests/forkprocess/ForkProcess.hs @@ -0,0 +1,81 @@ +-- | Does the generator behind 'MonadRandom' notice a fork made from +-- Haskell? +-- +-- @cbits\/tests\/sysdrg@ asks the same question of @fork(2)@ called from C, +-- in a process with no runtime system in it at all. That leaves the case +-- anyone actually meets untested: 'forkProcess', with the runtime's own +-- threads about. The handler is registered with @pthread_atfork@, which +-- libc runs for every @fork(2)@ whoever calls it, so it should reach this +-- too -- should, which is why this is a test and not a comment. +-- +-- It cannot live in the main suite. That one runs with @-N2@, where GHC +-- says 'forkProcess' is not supported; this needs its own runtime options, +-- and is built twice, once threaded with one capability and once not +-- threaded at all. Both are configurations GHC supports 'forkProcess' in. +-- +-- The check is that parent and child disagree. Without fork detection the +-- child carries on the parent's stream, so the next block each of them +-- draws is the same block -- they would agree exactly, which is the fault. +module Main (main) where + +import Control.Concurrent (getNumCapabilities, rtsSupportsBoundThreads) +import Control.Monad (when) +import qualified Data.ByteString as B +import System.Exit (exitFailure) +import System.IO (hClose, hFlush, hPutStrLn, stderr, stdout) +import System.Posix.IO (closeFd, createPipe, fdToHandle) +import System.Posix.Process (ProcessStatus (..), forkProcess, getProcessStatus) + +import Crypto.Random (getRandomBytes) + +draw :: IO B.ByteString +draw = getRandomBytes 32 + +main :: IO () +main = do + caps <- getNumCapabilities + putStrLn $ + "threaded: " + ++ show rtsSupportsBoundThreads + ++ ", capabilities: " + ++ show caps + -- GHC supports forkProcess with -threaded only while one capability is + -- in use. Saying so here means a change to the runtime options shows + -- up as a failure rather than as a test that quietly means nothing. + when (rtsSupportsBoundThreads && caps /= 1) $ + die "this test needs one capability when threaded" + + -- Before the fork, or the child inherits whatever is still in the + -- buffer and writes it out again when it exits. + hFlush stdout + + -- Draw once first, so that both sides inherit a generator that has been + -- seeded. A child of an unseeded one would seed itself for the first + -- time and differ for that reason instead of this one. + _ <- draw + + (readEnd, writeEnd) <- createPipe + pid <- forkProcess $ do + closeFd readEnd + b <- draw + h <- fdToHandle writeEnd + B.hPut h b + hClose h + closeFd writeEnd + hr <- fdToHandle readEnd + fromChild <- B.hGet hr 32 + hClose hr + status <- getProcessStatus True False pid + + fromParent <- draw + + case status of + Just (Exited _) -> return () + other -> die ("the child did not exit cleanly: " ++ show other) + when (B.length fromChild /= 32) $ + die ("the child sent " ++ show (B.length fromChild) ++ " bytes, not 32") + when (fromChild == fromParent) $ + die "parent and child drew the same bytes: the fork went unnoticed" + putStrLn "parent and child drew different bytes" + where + die msg = hPutStrLn stderr ("FAIL: " ++ msg) >> exitFailure diff --git a/tests/tutorial/extract.awk b/tests/tutorial/extract.awk new file mode 100644 --- /dev/null +++ b/tests/tutorial/extract.awk @@ -0,0 +1,53 @@ +# Pull every code block out of a Haddock module and write each one as a +# compilable Haskell module. +# +# A block is a maximal run of lines beginning "-- >", which is what Haddock +# renders as a code sample. The run is ended by any line that is not one, +# so two samples separated by a line of prose are two modules and cannot +# refer to each other -- which is the point: each sample has to stand on its +# own, because that is how a reader will copy it. +# +# The name comes from the "-- $section" marker the block sits under, so a +# compiler message names the section of the tutorial that is wrong rather +# than a number. The module header goes after any LANGUAGE pragmas, since +# those have to come first. +# +# Writes one file per block into the directory given as -v out=, and prints +# each module name on stdout. + +/^-- \$[A-Za-z_][A-Za-z0-9_]*$/ { + section = substr($0, 5) + nth = 0 + next +} + +/^-- >/ { + if (!inblock) { + inblock = 1 + nlines = 0 + nth++ + mod = "Example_" section "_" nth + file = out "/" mod ".hs" + } + # "-- > foo" carries a space that is not part of the code; "-- >" alone + # is a blank line inside the block. + line = (length($0) > 5) ? substr($0, 6) : "" + lines[++nlines] = line + next +} + +{ if (inblock) flush() } + +END { if (inblock) flush() } + +function flush( i, k) { + # Pragmas, then the header, then the rest. + k = 1 + while (k <= nlines && (lines[k] ~ /^\{-#/ || lines[k] == "")) k++ + for (i = 1; i < k; i++) print lines[i] > file + print "module " mod " where" > file + for (i = k; i <= nlines; i++) print lines[i] > file + close(file) + print mod + inblock = 0 +} diff --git a/tests/tutorial/run.sh b/tests/tutorial/run.sh new file mode 100644 --- /dev/null +++ b/tests/tutorial/run.sh @@ -0,0 +1,126 @@ +#!/bin/sh +# Does the tutorial still compile? +# +# Crypto/Tutorial.hs is the one place in the package whose code the compiler +# never sees: every example in it is a Haddock block, so an API change +# silently leaves it wrong and the next reader copies something that does +# not build. That has happened -- taking a checked key in Poly1305 made the +# tutorial's crypto_box stop type-checking, and it was noticed by reading +# the module rather than by anything here. +# +# So the blocks are pulled out and type-checked. Each one becomes a module +# of its own, because each one is a thing a reader copies whole; a block +# that needs a definition from the block above it will fail here, which is +# the right answer. +# +# Checked against this tree rather than against whatever crypton is +# installed: cabal repl has the project's library as its home package, so a +# tutorial written for an API this working tree has not got cannot pass by +# finding the API in a released version on the machine. -fno-code because +# nothing is run -- the question is only whether it compiles. +# +# -Wall as well, and a warning in a tutorial counts as a failure: an unused +# import is three words a reader will paste into their own file. Only the +# extracted modules are held to it, not the library loaded beside them. +# +# $GHC names a compiler other than the one on the path, and $BUILDDIR a +# build directory to keep that compiler's products out of the usual one. +# CI has one compiler per job and sets neither; locally they are how an +# example is asked whether it also builds on the oldest GHC the package +# supports, which is the reader this is most likely to have failed. +# +# The first module is wrong on purpose and has to be reported, or the run +# proves nothing: a repl that failed to start, a ghci whose message format +# changed, or an extractor that produced no modules would otherwise all look +# like a tutorial that compiles. +set -eu + +cd "$(dirname "$0")/../.." +tutorial=Crypto/Tutorial.hs + +work=$(mktemp -d) +trap 'rm -rf "$work"' EXIT INT TERM + +modules=$(awk -v out="$work" -f tests/tutorial/extract.awk "$tutorial") +count=$(printf '%s\n' "$modules" | grep -c . || true) +if [ "$count" -eq 0 ]; then + echo "FAIL no code blocks found in $tutorial" + exit 1 +fi +echo "$count code blocks in $tutorial" + +# The deliberate one. hashWith returns a Digest, and has since the module +# was written, so this is the shape of every break this harness is for: the +# tutorial says something the library no longer agrees with. +cat > "$work/Calibration.hs" <<'HS' +module Calibration where +import Crypto.Hash (SHA1 (..), hashWith) +thisIsNotADigest :: Int +thisIsNotADigest = hashWith SHA1 "the harness has to report this" +HS + +{ + echo ':!echo @@ Calibration' + echo ":load $work/Calibration.hs" + for m in $modules; do + echo ":!echo @@ $m" + echo ":load $work/$m.hs" + done + echo ':quit' +} > "$work/script" + +# Not cabal's -v0: that reaches ghci too and takes the "Ok, N modules +# loaded." line with it, which is what is read below. +cabal repl crypton ${GHC:+-w "$GHC"} ${BUILDDIR:+--builddir="$BUILDDIR"} \ + --repl-options=-i"$work" --repl-options=-fno-code \ + --repl-options=-Wall --repl-options=-Wno-missing-home-modules \ + < "$work/script" > "$work/log" 2>&1 || true + +awk -v expected="$count" ' + # ghci writes its prompt before the echo, so the marker is at the end + # of the line rather than the start of it. + /@@ [A-Za-z_]/ { mod = $NF; verdict[mod] = "no answer"; order[++n] = mod; next } + mod == "" { next } + /^Ok, [0-9]+ modules? loaded\.$/ { verdict[mod] = "ok"; next } + /^Failed, [0-9]+ modules? loaded\.$/ { verdict[mod] = "failed"; next } + # Only the extracted modules are held to -Wall; the library is loaded + # beside them and is not what this is asking about. + /Example_[A-Za-z0-9_]*\.hs:[0-9]+:[0-9]+: warning:/ { warned[mod] = 1 } + # Everything ghci said about a module before it gave its verdict, minus + # the progress lines, which are the bulk of it and say nothing. + /^(ghci> )?\[ *[0-9]+ of [0-9]+\] Compiling/ { next } + { if (verdict[mod] == "no answer" && $0 != "") detail[mod] = detail[mod] $0 "\n" } + END { + bad = 0 + for (i = 1; i <= n; i++) { + m = order[i] + want = (m == "Calibration") ? "failed" : "ok" + got = verdict[m] + if (got == want && want == "ok" && warned[m]) got = "warned" + if (got == want) { + printf "ok %s%s\n", m, (m == "Calibration" ? " (reported, as it must be)" : "") + } else { + bad++ + printf "FAIL %s: expected %s, got %s\n", m, want, got + printf "%s", detail[m] + } + } + # Nothing answered, or not everything did: the run itself did not + # happen the way this script assumes, so the log is worth seeing. + if (n != expected + 1) { + printf "FAIL %d modules answered, %d expected\n", n - 1, expected + exit 2 + } + exit (bad > 0) + } +' "$work/log" || { + status=$? + if [ "$status" -eq 2 ]; then + echo + echo "--- the last of the log ---" + tail -60 "$work/log" + fi + exit 1 +} + +echo "every example in $tutorial compiles"